I used to believe that too but after I have gotten far more experience and fished many more different locations and lake types my observations are:First off I agree with Rolo.ALSO if you think about it a lot of it can have to do with the particular region you are in or even the particular body of water.(as Sam started to describe) The topography of the lake can be important as well when factoring in wind direction. For quick example. Take a long tapering point on one end of the lake. If you get strong winds,especially over the course of several days,blowing on it or especially across it,then you can bet your bottom dollar that your going to produce better than if it is dead calm. There are certain places I fish when I despise an east wind or places where I'm dying to get a good east wind. Even a place where I absolutely kill them on a strong north wind even if it is freezing cold for the most part simply because the wind generally pushes plankton,insects,bait,and predators toward a desired lake structre or cover type etc.
One thing I am a firm believer of though is that "swirling winds" or wind/breeze that takes on several different directions over the course of the day or even the hour makes for a real s****y day on the water most times. Also makes it hard for me to know where to start or stick with while I'm there