Funny, I just came in from a bit of shoreline fishing on the lake I now live on (Apartment building though, unfortunately.) using a setup similar to what you're trying to do, including an ultralite rod.
I currently use a 6'6" UL with a 10 size reel, 5# Power Pro/4# P-Line Fluoroclear leader.
I did originally buy it just for panfish, but the first fish I caught on it was a 3 pounder on a live worm, and it handled it great. I thought, why not just use it for everything while on shore? I stocked up on some small baits like 3" stick baits, small craws, little paddle tails and such. I basically only use a 1/16th or 1/8th jigs, shroom heads or ball heads. The jig pictured is a BPS Shroom Head jig, which is available in #1 and 1/0 hooks, I have the #1's. I do have some #1 (I believe) VMC EWG hooks that I doubt I would have any problems with as they are VERY thin, but I have had no need to weedless rig for now.
I say go for it, but 6# won't be a big deal either. If you think you'll be lacking power, try braid main line and just set your drag a little lighter. That hook in the picture had no issue getting buried in a couple keeper's mouths today when I got home from work, and I even forgot to tighten my drag from when I was targeting some 'Gills when the first bass hit today.