I used to think that smallies really just put up more fuss, and weren't necessarily "stronger" fighters than largemouth. Also, considering the fact that most of my biggest smallies were caught on light line/tackle, my impression has been that if you let them run, they tire out. However, after fishing an umbrella rig, with a heavy flipping stick, and 65# braid, my tune has changed. When there is no drag singing, and it's you, a stout rod, and no stretch line, it becomes pretty clear that a smallmouth can simply put the screws to your gear much harder than a comparable largemouth. The shock and power of direction changes and just head shakes is incredible.
For the original question, a smallie over 4 and a largemouth over 6 are what I consider truly "big." My best smallie is 5-12, and my best largie was 7-2. I'm not sure I'd consider either "trophy" fish, but in the northeast, they are big for sure.