In your initial post you wrote "battery" not batteries and did not specify 24 volts. That's a horse of a different color, and a more expensive one. The link you provided states 12v @ 10 amps. You CANNOT get 10 amps from a 50 watt panel (math doesn't lie), well not in my universe anyway. Even for a 12v system with two batteries what you're looking at is Waayyy too small, plus you're looking at needing a 24 volt system unless you want to charge the batteries individually with a 12 volt setup.
Craig's list has many little 12v solar setups for sale from boaters and "off grid" dreamers who realized this too late and blamed the panels for dead (expensive) batteries, not their lack of planning and research.
To answer your question... with a solar panel and "charge controller" you go straight to the battery. Going through a battery charger would be a waste of power, size, money and be overly complicated. So you know, a solar controller IS a battery charger. Your main concern I think will be theft of a solar charger, but only you can answer that.
At any rate, my charger very portable and is 80 watts (panels bought from a cheap yachtsman upgrading after killing his batteries) for a little 35Ah U1 sized battery and it works fine for that.
See the solar panel below. I think it would make a suitable bare bones start and it could be rigged to go portable like mine rather easily. Renogy (@ 12v I use another brand) has lots of larger panels you may want to reach out to them with your location, and battery Ah specs (both very important to know) to see what they can gen up for you.
Panel specs and distributor (note optimum operating current):
https://i.ebayimg.com/images/g/VLcAAOSwPn1eL2XF/s-l500.jpg
Panel price:
www.ebay.com/itm/Renogy-270-Watt-24-Volt-Polycrystalline-Solar-Panel/124062981583