Same here - which is why I went with a canoe. Last year it was the canoe, TM, battery that ate most of my Christmas money from the parents. This year it was the trailer and costs for the modifications I did...still have to get a new length of 2x3 to replace the trailer tongue piece to lengthen it - 14' canoe, 10' trailer as shipped.
The only debt I have is my mortgage - everything else I pay cash for. If you can handle being under debt for a boat for the next 10-15-20 years, go for it. I'll happily paddle/power along my lakes knowing that I'm not paying for something that sits idle for 5+ months of the year.