Well done, you sir, have extraordinary self control. As to asking if someone were adopted, I don't think that is much to do with racism but assumptions that break the norm or the perceived norm. For example in say, of a couple with blonde hair and their children also have blonde hair except for one, it breaks the expectation of what you you think is normal. It could just be normal genetics or the husband or wife could be their step-parent or they could be adopted or the mother could have had a fling with the postman
Not racist, just confused. It happens with other stuff as well, if you shave your hair when the Worlds Greatest Shave is running but then say you didn't do it for that, people get confused and make assumptions like maybe you have cancer or your workplace requires it or you were just getting hot.