Firstly, if you're going to do this, DO NOT PUT YOUR ENTIRE LIBRARY ON SHUFFLE. After some relatively sensual tunes, Polysics suddenly yammering in your ears is not sexy.
Secondly, I actually find it offputting, even to appropriate music. A tune will go on for five minutes, he's probably not even put it in by then, then the next tune starts and the mood is altered slightly. By the end of the next tune you're just about getting into it and then the mood is altered again when the next comes on. It makes me feel like I should be taking cues from the music, or that we should be finished by now, it just makes me feel anxious and hurried.
EDIT: And I couldn't do it to something like Barry White, it would just seem ridiculous and I wouldn't be able to take the act seriously. I think last time my boyfriend put on Massive Attack or something along those lines, which worked as well as it could for me.