I think all of the standard non-human natural variables (clarity, forage, temperature, wind, light levels, etc) apply. I also agree that we're not running controlled trials out there on the water. However, I think there is a major human element to this consisting of two parts.
I'll use the example of fluke-style baits. At my local lake, I can't buy a bite with a Zoom super fluke in any color. I've tried a ton of them. However, the standard caffeine shad in white does exceptionally well. It's a stark difference, I wrote a post about it a while back, and as it was immediately noticable and has been a durable pattern.
I think it's a blend of individual angler and what the fish get habituated to from other anglers. The caffeine shad has a way different action than the zoom, and I think it just works better with how I like to retrieve and probably my personal gear. Additionally I suspect the other anglers are throwing Zoom products (based on trash I find and observation) so the fish learn to avoid something that looks like a Zoom fluke -- no matter the color.