No. It doesn't make you a pervert.
To be honest, the perverts are the ones who immediately imply that there must be something untoward going on.
But, the perception of others is a fickle thing; And these days there's a degree of paranoia around that makes even the most harmless of things a problem.
If you want proof of this, look no further than Micheal Jackson - Now, I'm not saying anything about wether he did anything inappropriate with kids or not;
But look at what the mere accusation of it did to his life;
All the damage to his reputation that it caused - None of which had to do with what he actually did, but merely with what people suspected he did.
Granted, that's an extreme example, but think about that and you see the problem here.
If people are worried about it, you have no way of proving your innocence.
And therein lies the danger to the 'adult' in these situations...
We want to protect children, but it's easy to forget the harm caused by excessive suspicion...