It depends on the moment, really. I can happily talk to someone without looking at them while watching TV, or walking beside them (because turning your head constantly is really awkward). When I'm re-telling a story or explaining something, I'll tend to look away because I'm "seeing" the moment and describing it.
If someone is telling me something, I will tend to look them in the eyes, but often I find I need to focus to keep looking them in the eye for a long time and then I end up not listening, but instead focus on making sure I look 'normal'. I feel keeping a constant gaze is uncomfortable for both parties - especially when you're talking to more than one person, and you have their whole attention. You find yourself turning constantly between each person, and it's just unnecessary. I also find that in moments where you are intensely listening (like if someone is telling you a problem) looking away helps you concentrate on the issue better.
Captcha: Broken heart.
Aw, but Captcha, don't you understand my reasons? I enjoy our brief moments, and anyway, you have no eyes!