I was in the same boat with snowboarding... when I first started the people that snowboarded didn't want anything to do with me because they were afraid they would have to babysit me and the ones that didn't snowboard couldn't care less about it. So I started driving to the hill (3hrs one way) on my own. Then I started flying to CO on my own and then I found this website and now I have many friends that ride in the midwest and Colorado...
This wknd I didnt feel like driving anywhere and it dumped about a foot of snow, so I got in my car and drove 5 mins to the forest preserve where I hiked and met more people...
If you get out there, you'll eventually meet people.
I still don't have travel buddies though so I normally go to other states/countries by myself. Last year, I went to Europe and SE Asia for a few months; it was the best trip I ever went on. In a way, I like going by myself because I have a freedom to do whatever I want and I meet more people. I think if you go with friends you'll have a great time but you won't be so open to other people and you'll be locked in to your own group... not a bad thing just different.
Also, the dating pool has definitely shrunk since I figured out what makes my boat float but I go with the flow... being single definitely has it's perks :)