I regularly sleep in as little as possible, which is usually boxers or more often, cotton jogging shorts with maybe an 6" inseam. Mainly for the aforementioned "I don't really want to roll around in sheets that the cat pushed half way up my ass while it was cuddling me" (I sleep on my stomach and the cat likes wandering around and sleeping on my back and ass). For me, it's a temperature regulation thing as I tend to run hot and hate waking up with a sweaty shirt on, but if I kick back blankets in the night, my blankets/pillow are never sweaty.
My wife tends to sleep in a shirt, unless post-coital, in which case she usually sleeps topless. I have no idea why she does this, but am now inspired to ask her.

She is extremely well endowed (f cup) and that may or may not be a factor in her choosing to sleep in a shirt. My last long term girlfriend was a b cup and slept topless more often than not. But again, I don't know if size was a factor there, or just personal choice. My six year old daughter loves sleeping topless in the summer. She asked why I sleep shirtless a few months ago and wanted to try it; we just told her she needs to have a shirt on by the time she leaves her bedroom and kinda forgot about it. I don't actually know if she's still doing it now that it's colder because she's buried under blankets at night.
Middle of the night sex is generally the only reason my wife and I ever both sleep totally nude, and that's as much laziness as anything. And even that has dropped off some as our kids are getting old enough to wonder why we are completely naked in bed.
Personally, I also think sheet quality is a big part of the deal. If you have good, comfortable sheets, it may be more comfortable. Long story short, sleep however makes you most comfortable.