Typically, my rule of thumb is if it's a hand-held, then absolutely, unless it has awesome music (like Phoenix Wright or something).
If it's a console, then no, unless it's becoming repeatiative and annoying because either the music is just that bad or I've been in the same place listening to the same loop for several hours.
It happens occassionally.