I keep it on. It doesn't really bother me unless I'm playing with the volume down low and the vibrations are kinda loud. But I don't generally play a lot of action-intensive games, so I'm not likely to permanently lose feelings in my fingertips due to use of the controller.
And I like the rumble in most games. I mean, it's not remotely realistic in any way, but physical sensations in the hands does make things a bit more "immersive" when playing games. And it's so ubiquitous that if I didn't get a mild jolt in the controller when firing a gun of some description, I'd sort of feel kinda weird.
I remember reading an article about how long term use with vibrating controllers was causing health problems for kids, back when they were a new thing. And back when I was living on pocket money, I ended up playing with a third party controller that made me think "Okay, maybe it is possible." When Psycho Mantis told me to put my controller on the floor, he proceeded to throw the ************ across the room.
Sort of related - I remember, back when I read print-magazines about computer games, and they advertised these weird vest type things that promised realistic sensations when you were hit. And I was actually kinda curious about it, even though I knew it wouldn't work as promised because, for fuck's sake, it was compatible with the Playstation and I had official peripherals for that console that weren't supported by more than one game. Like the awesome wheel controller that came with the special edition of Ridge Racer Type 4 (the greatest racing game ever released).
But even though I've met loads of people who remember those advertisements, I don't know anyone who's bought one and I am still really curious about them, but I'd be worried that I'd end up dazed, confused, and constantly shaking about. Like Ozzy.