I've never really had a problem with any Gym Leaders... Whitney's Miltank was simple to beat 'cuz I, on my SoulSilver playthrough, caught a Heracross in Azelea.
My least favorite, however, would probably be Volkner, Pryce or Koga. They're just so... Ehh. I was never a big fan of Ground-types, so I don't usually have something good to get him with, and Pryce is... Pryce. He just doesn't appeal to me. Koga and his Minimize/Double Team is annoying, and I hate it, but that's only because it's effective.
I never really liked Bugsy either. Mostly because he SUCKED, and I couldn't exactly tell if he was a boy or girl... Let's see... I had some trouble with Clair, but that's 'cuz I was trying to level up my (AWESOME) Skarmory. In Kanto, I always picked Bulbasaur, so I never really had any trouble with Brock, but I hate him because Charmander is the coolest of the three, and him and Misty discouraged anyone from picking him... Speaking of which, Misty's Starmie. It's quite overpowered for that point in the game. I've had a bit of trouble with Chuck, as well, but that's 'cuz I didn't have a good Flying-type. Sinnoh's the only region in which I use the early-game Flying-type. Staraptor is a beast.
Oh, I almost forgot NORMAN. The jerk. And Tate & Liza. Had some major trouble with those 2 battles.
*EDIT* Also, I almost forgot. If we're counting champions, CYNTHIA. Her pokemon are severely overpowered. They are all high-quality sweepers and walls. Worst of all, all of her Pokemon have PERFECT IVs. 31 in each stat. Yeah. It's that bad. Luckily, my Empoleon can barely survive one of her Garchomp's Earthquakes and hit it with an Ice Beam most of the time.