Wallmasters and floormasters scare the living daylights out of me still. Redeads aren't too far behind, especially the Windwaker ones that jump out of coffins. *twitch*
R&B Mammoth in The World Ends With You also took five-BILLION tries to finally defeat, as did Star Wolf in Super Smash Bros: Brawl and pretty much every boss after Amorphes in Avalon Code.
Winner of Most Annoying Baddie for me, though, is the Decus/Alice combination in Tales of Symphonia: Dawn of a New World. I had one day left on my rental, ran up to that boss fight and got killed by Alice in two hits. I ended up reading the spoilers on Wikipedia and returning the game the next day.
I was NOT going waste my life grinding 15 more levels just to see another mediocre cutscene of a game I'd never play again. -_-;;