On average, I fish 2 weekends a month....AT MOST. Obviously not as much as I'd like to, but obligations beckon.
That said, there was a time in my life where I fished EVERY weekend, without fail, for a few years. Looking back on that time, it helped me tremendously with my fishing, but hurt me tremendously as well. It took too much time away from my friends, family, and other "stuff". There IS such a thing as "too much of a good thing is too much".
That said, I, too, faced the same issues as you. Don't let them bring you down. Just because they "don't get it", doesn't mean you shouldn't pursue what you enjoy most.
Case in point - Had I listened to "them", this site wouldn't have come into existence.