In single-player games, the most annoying weapon for me to use would have to be any sort of knuckle AND big gun in Fallout 3. For one, the knuckles feel way too insubstantial for me to care enough to get close enough for a hit, and the big guns are so inaccurate without V.A.T.S. and take up so much AP with V.A.T.S that they're basically single-shot affairs at extreme close range (laser rifles in the game also get an honorable mention, since the only-slightly-weaker laser pistol is so much more easier to use, making the rifle dead weight, as well as the ripper and it's variants, which have an incredibly short range and can only do meaningful damage in V.A.T.S., in which they take up ungodly amounts of AP, and which break down easily).
As a weapon which is incredibly annoying to get used on me in a single-player game, my vote would go to...actually, the mini-guns of that same game would most likely take the cake. Whenever an enemy gets their mitts on one, it's as though they have a scope and heat-seeking bullets to boot. But obviously, mines are annoying in every game.
Oh, and multiplayer games. For these, it depends on what type of multiplayer, but I'll go with good ol' fashioned 4 controllers plugged into a box. In this case, if you're new to the game, the most irritating weapon to have used on you is misinformation, for example, when the game's owner tells you block is one button when it's just the duck button. With the situation reversed, with you playing as the experienced gamer, it's all beginner's luck.