Half the problem is tracking him down. How do you find him in the first place? He has teleports and you won't know where and when he's going appear and disappear. He rarely does his own dirty work and doesn't give scheduled public appearances either. Heck, he took down the ministry without being seen. How many years did it took to actually find Osama when he can't even teleport around like voldie? Not to mention normal people won't even be able to see his hideout with all those magical crap he places around his headquarters. The three protagonist eluded capture from magical people for quite some time, and they don't even have voldie-level magic. They didn't even finish their magical education. The only reason they got caught was because they muttered the magical swear word. Nuking and/or sniping at voldie requires them to actually find him first, which proved impossible for normal folks.
The other half of the problem is having enough power to actually fight his army. When he regained his full powers, dementors were roaming around Britain. Just having dementors alone would have sucked out your will to fight, and the normal people won't even know its there. Any army you sent against him would be too busy being depressed to do anything useful. Lets not forget the death eaters, giants and werewolves that joined him.
Oh, don't forget how easily the magical ministry placed a wizard in the PM's office. If voldie felt like it, he could have controlled the muggle PM and have him work against the good guys. After all, if he could gain control of the magical ministry, the normal ministry would stand no chance against his infiltration.
The Horcruxes are just icing on cake.
Then again, why didn't the ministry just use Time Turners to go back in time and slaughter child voldie before he went to Hogwarts (or better yet, after he was born)? They have a cabinet full of those things before they got smashed to bits in Phoenix. They could have done that in the first book and be done with it.