The great thing about living in VA is that you can carry on your hip, perfectly legal. Just the sight of the gun on a hip is detterent enough. I am not a small guy, but like another poster said, there are always some drunk campers, or poaching types (usually illegals in my neck of the woods) that just won't be happy unless they are messing you. I've had my tackle kicked over, been surrounded, had a guy repeatedly try to snap my rod tip while his buddies laughed, rocks chucked at me, trash and cigarettes thrown at me, etc. A few times I was able to just back away and get out of the situation, a few times I had to cast a rattle trap at somebody's face before being able to flee. Ever since I began carrying openly, I have not been in one situation, people just pass me by.
Another thing to remember, pulling your gun out and showing somebody (even if justified) can be considered 'brandishing a weapon' and it can get you arrested. Say you show your pistol at some threatening idiot, and he leaves. He can easily get to the parking lot, call the cops, give them his story (in which he did absolutely nothing wrong of course) and guess who gets arrested, gets a court date, and possibly even their firearm confiscated? YOU. So be careful, be smart, be alert and use self restraint. The idiot(s) that would mess with you when fishing are the same spineless toads that will not hesitate to lie to the police to get you back...
And we all know how knowledgeable, unbiased, and fair the courts are to people brought up on charges of assualt with a gun or brandishing or whatever, don't we...