I'd say just fishing isn't what's only involved with having the boat. It also includes a different scenery and a chance to move away from others when you are stuck on the bank, among other things. Boating is a different experience in which I enjoy that comes with fishing and enhances it on certain levels.
So no, I wouldn't give up boating because I could.