Hm, I guess it's more that lyrics distract me from the emotional connection I get from the music. I don't suggest that this is an *exclusive* thing, naturally there are some songs where the lyrics speak to me (usually because that song was *playing* when something important or memorable happened) - of course, that means that it wasn't natively the lyrics that hooked me - it was the connection with the circumstances.
A sort of an example might be the metal music that my partner is fond of - I love the guitar solos, but the lyrics/vocal sections I find awful. They say nothing to me, and get in the way, somehow. It's difficult to put into words (no pun intended), but I just switch off at the lyrics, even if the singer has a nice voice.
I also meant to add in an edit that (for me at least) my own thesis breaks down in the context of songs where the music *is* the lyrics. For example, despite not being religious myself, I adore ecclesiastical music of the chant/polyphony variety.
EDIT: Whoops, clicked reply instead of quote - EightGaugeHippo, this was a reply to your post immediately above!