A cat fluffing its tail is a good sign to back off and stop what you are doing. Except for one of the rescued cats I live with, it is a bit developmentally slow and fluffs when it is scared, or happy, or sad, or has a treat or is being petted or because OMG the sky is blue. Silly thing. Mostly it should be respected as a fear response.
And nope, cats are only aloof disinterested and don't much care about their owner if they are routinely tossed outside 23 hours a day or otherwise ignored somehow. Oddly enough, your pet learns to not give a fark about you if you do this. My cats had a great deal or care, attention, handling, affection etc and gave plenty back and I feel their loss every day.