Well we fish our community pond as do others, it really boils down to a few things. Where you are is under HOA rules, read the written rule book if it is not stated in there then you need to default to signage. If there are no signs or they are not clearly visible then you are ok for now(like with me I told them that one time, there are no signs and this is a community pond then the signs will show up if it that big a deal). Now when someone tells you to leave you can tell them to show you the signs, you read the rule book etc. as far as calling the cops it is an HOA property not a homeowner I dunno who actually would have to call them the HOA president or what, techincally your not tresspassing if it is your(firends whatever as long as one of you live there) community and in a HOA governed area. But once they post or clearly post the sign or write that rule then your done, they have ground to ask you to leave, fine you with an hoa fine, or call the cops. But I am not a lawyer just defaulting to common land development practices so take my words with caution, read the community rule book first before you go back out. When you go back out check for signs and stay out of peoples back yards.
Backyard you should be good but it can depend and they can get technical with it if they want. If there is an easement, the property ends prior to edge of water or a clear mow strip then they may/can say it is in there jurisdiction and no fishing.