Not rude, outside of a few particular circumstances like sitting down for a proper meal at a dinner table. Or, you know, being in some form of church that has particular guidelines for the covering of one's head.
It's just an article of clothing after all.
It's just an article of clothing after all.