This summer was a big wake up call for me when launching my boat and the rope snapped. There went the boat. I was a great swimmer as a kid and hadn't swam since I was 20 or so.
Fast forward almost 15 years and my 7 year old and I are the only one's on this secluded lake with no houses and I had no choice but to go after my boat. I stripped down to my underwear so clothes were no issue. But factor in added weight over the years, totally out of shape, like @slonezp a destroyed shoulder from hockey, and the fact I hadn't swam in almost 15 years I'll be the first to admit I wouldn't make it too long with clothes and no PFD. Quite honestly if it weren't for my ability to float on my back (which I wasn't sure I could still do) and rest I might not had made it to my boat and probably would've drowned right there in front of my son.
My point is just because you could and did as a kid doesn't mean you still can. Like @A-Jay said try it out in a controlled environment with somebody else. Swimming isn't like riding a bike!