My son is 4 almost 5 and he and I go fishing together when he wants to. Right now I do not bring any of my stuff when he and I go fishing together, I just help him. We also leave almost as soon as he starts to complain. I want him to enjoy the experience as much as possible so that when he gets older we can continue to fish together.
I also have made fun games in our backyard for him to practice casting, so he can continue to be more self reliant. I try to set those up whenever he mentions it. He also helps me pick out lures when I go shopping, again, just gives him a bit of ownership of the experience.
With my dad.