Being a power fisherman, I will say this:
If you only caught two small fish all day, you don't have much to complain about. Doesn't sound like they were on to me. It would be different if, say, you caught those two fish back to back.
If you want a good bait to use in this situation, go to a swimbait on a 3/16 to a 1/5 ounce weighted hook. Cast a little ahead of the boat centerline.
Some swimbaits don't like fast retrieves, so make sure you select on that does. I have two molds for my swimbaits: a fast retrieve and a slow retrieve. they are the same size and look almost identicle, but they each need different speeds for the same action.