That is absolutely ahorrent behavior, and I'm sorry you had to deal with that.
If they were engaged in illegal activity, I probably would have tried to take down their license plate numbers and reported them. Probably wouldn't do anything, but I'd be too morally outraged to not do anything.
As far as their bullying, intrusive behavior, I would just move. There are lots of fish. Lots of great fishing spots, probably many way better than even the one you were fishing. I don't know if I could keep myself from telling them what I thought of them, though. Try to find fishing locations removed from humanity. Hard to get to, off the beaten path. That tends to weed out a lot of the lazy, trashy people that frequent easily accessible public areas.
My favorite pond was completely surrounded by heavy brush and trees, and I could always count on fishing alone. Last year the city cleared all the brush out and turned it into a really nice, easily accessible water feature. And now there are always anglers at all times of day down there. Ruined it for me.
But, yeah, I just try to avoid people. If someone steals or crowds my spot, it's extremely rude behavior, but I don't own the water, the fish, or the land. So, I'll just mosey on over to another spot.