Believe it or not, there is an entire academic literature in applied ethics on the question of whether the choice to engage in catch and release fishing at all is ethically justifiable. The argument against boils down to what you say here: If I do it, I cause unnecessary harm and distress to fish and I risk mortality, whereas all I gain is personal enjoyment. That is, the net benefits do not outweigh the costs. Catch and keep fishing, however, like hunting, result in a harvest in which the benefits can outweigh the mortality cost (presumably, if you harvest for food, that means there is something else you are not eating instead, which is saved). Ergo, hunting and catch and keep fishing are actually more ethically justifiable than catch and release fishing. Note, it's not an argument about what should be legal, just an argument about whether one, at the moment of decision, is ethically justified in choosing to do it, compared to not doing it, or to doing something else.
There are many possible responses to this, some of which are good and some of which are bad. But my response is simply to shrug and point out that the range of alternative activities I could do for leisure instead would impose worse costs, without any greater benefit, simply by moving around more, using up more energy, and taking up more space, potentially creating more waste, and creating more lucrative markets for resource extraction and exploitation. In the face of all the myriad things humans do natural environments and creatures within them every day, even If I accept their argument, I'm quite sure my own catch and release fishing is nowhere near the most ethically problematic thing either I or they do every day.