I always toss mine and reel it just fast enough so I can feel the vibration in my rod. I might burn it sometimes or even let sink and try to work the bottom with it. What color chatter did you get and what brand? some say knocks isn't always good. Chatterbaits are always my favorite bank attack when coming to small ponds. You might just have to experiment with your retrieve and different trailers as well, from flukes, Magic Shad, curl tails, craws, to split tail.
I always had luck on white or white/chartreuse with a matching Live Magic Shad. Lately been tossing out black/blue with no luck but might not be in the right conditions for that one. I never tried around the lilies just the outside of them with no luck.
Also don't be alarm if have a catfish on the end of your line!!!