i fish the upper james river alot and like you said just down size a lil for most part. Some baits that work for me are flukes,senkos, t-rig gary yamamoto twin tail hula grub,jigs, tubes,jerk baits, small cranks,spinner baits, top water etc list goes on!
Normally ill look for where creeks flow into river, behind large rocks where current breaks or along or behind islands...my fav is find a rocky section of river thats deeper and drop shot! I also have good amount of success fishing behind sets of rapids, Cast up into the rapids and let current bring your bait back down into the pool below and keep contact with bait and hold on bc smallies will crush it coming down rapids! Also i would start out with alot of weedless type stuff till you learn the river some bc with current you get snagged its hard to turn around and go back up river to get unsnagged you will use alot off trollin motor battery tryin to fight current or by time you start motor it would already be too late. I know my section of river enough to know where i feel safe throwing cranks on certain sections and not in others. You will enjoy smallie fishing in VA im sure just keep it simple! 8-)