Unlike baseball, when I'm fishing I swing at every thing. Soft at first, then hard when I feel the line take hold. I am not the best jig fisherman but this is my process.
Also, I think you have to remember the fish gets in its mouth and has get through the skirt and the weed guard. It's a small window when it comes to the open space you have to set the hook, compared to other baits. To me, this is where the right equipment is crucial and experience is also key.