Doesn't sound like color was the determining factor, because you all caught fish. Sounds like the bigger one's went after the bigger profile. This is a guess, there's more info needed like:
* Clarity of water
* Where were the fish caught, every one of them
* Time of day the fish were caught
* Whether conditions
* Structure and/or Cover (I doubt you all fished the same structure at the same time)
What should you use next time? Let a new pattern emerge, it's a new day and you rarely repeat what happened in the past.