I consider myself a closet anime lover, as anime is one of my primary interests, but I never ever let random people or people who don't watch it known that I watch it. Like you mentioned, I also feel like there is a certain stigma that comes with watching anime, so I only even admit to watching anime around people that also watch it, or sometimes, people that used to be really into it, but aren't hard on people that still are.
It's just too much trouble and guess work to let general people know what I like without worrying about how they will judge me for it, so I just avoid spilling it altogether. If my parents weren't around back during my weabboo phase(11-13/14), then even my parents probably wouldn't know that I like it. Even though they know that I'm into it, there are still just so many things in anime that can't really be easily explained or excused, which is why I usually pause and/or turn off whatever anime I am watching if they knock, or tab away if I am reading online or something. This also often applies to jrpgs as well.
Maybe people probably will hardly care as much as I think they would, but it's still just a lot less hassle to leave them in the dark entirely.