It probably saw you on that last twitch and spooked. I have fish follow baits all the time and not strike. They might strike that same bait on the next cast or may become completely disinterested. Usually switching baits gets them interested again. If I can just pick up another rigged rod and follow-up as a test that usually tells me what they're biting. If they come up to the boat or shore and see me moving, they swim away fast.
I had an interesting experience on some semi-captive fish in a creek last weekend. The creek is really low and the fish are trapped in the deeper holes. I snuck down there and got a few casts in with a small barbless crank and they were all over it at first, but as soon as they saw me, they retreated to the other side of the hole and wouldn't bite again.
So I moved about 20 yards down to the next hole and started catching them on the Ned rig. After catching and releasing a few, they wouldn't hit the Ned any more, so I switched to a plastic swimbait. Caught a few more and then stopped again, so I switched to a jig. Seems like in that captive environment, they figured out what was a trap pretty quick, so I had to keep switching due to the low population and close proximity of the fish.
If they don't like the popping action but are hitting on top, you might switch over to a Heddon Zara Spook and do some walking action and see if they take it. I love poppers, but sometimes they just aren't working and the spook may clean house in that situation.