I am in the same boat as you are. I refuse to put a motor on my kayak, because my fat ass next the exercise. I also camp a lot, where I can't charge the battery. But even so, the cost of a motor and a battery with good life is about as much as I'll spend to upgrade to a pedal kayak and sell mine. So I demo'd several models, thought about going all the way with a Hobie Pro Angler 14 but ended up deciding to go with the Old Town Sportsman Bigwater 132PDL. I will lose the open hull layout and added rigability and horizontal rod storage of the PA14, but save myself a few thousand dollars, back strain loading it and gain a lot of reliability, speed and manuverability - the Biigwater was the fastest kayak I demo'd for sure, even faster than the Outback.
I will second the advice of others when I say buy go out and buy what you really want, since you know you'll be using it. Buy once, and cry once as they say.
The other piece of advice I will say is when you look at different models. Look for places that demo them - I had to take a few weeks and drive a few hours this summer/fall to demo all the models I was interested in but it was worth it - I never would have decided on the Bigwater if I didn't. Then figure out how you are going to rig it before you buy it. Learn your options. Go online, maybe Youtube and look at others setups. Sometimes searching something like "Old Town Sportsman Tournament setup" will pull up videos on youtube and you might get ideas from others. If you find a really good setup on one kayak, then maybe it's worth it. If you find a really good setup for the kayak you already have and just haven't put in the time or effort, then maybe it's time to go that route.
The Old Town Sportsman 120 is a very good budget friendly kayak. There are also pedal drive and motorized versions out there. You might be okay in the shorter 106 as well. There's a lot of cheaper models out there that aren't worth the savings. The Native SlayerMax 12.5 also isn't bad, but I found the added features weren't really done in a usable way and for the extra $500, I could add back tracks to the 120 and save $400 and I can actually use the slots in the 120 they give you to lay down rods where if you use the rails on SlayerMax, the rod holders are worthless (the edges of the plastic are also very sharp and will cut your line).