As a Teacher I can tell you that if the parents are ok with your relationship to these children and are comfortable with what goes on, then you shouldnt let it trouble you

. Personally I would not be able or feel comfortable myself with your situation but that is just me

.
Perverted acts are something that is damaging primarily on the child and these children feel happy in your space and your comfortable letting them be there. So no not perverted

, inappriopriate? Well as a practicing proffessional, I would avoid anything like this as though it were the black death. Seriously. We have children visit the house sometimes and I do not allow them to stay with me alone for any real period of time. Why? Because it's wrong for me. That's not my role within my proffession, and anything that could put me in such a light would be reckless for me to entertain.
But you? Your a friend on a personal level to these guys and the parents are fine with it. Don't let it bother you, it's perfectly normal to consider whether or not it's an issue, if anything it shows how much you care about them, and how important thier safety is to you! If this neighbour wouldn't feel fine with it, that's her perspective on her children, and it's wrongful to force her values onto someone else's children, just as it would be unfair to force putting her children into a position that she doesn't feel comfortable with

. She's right, your right and most of all enjoy the great relationship you have with these kid's because it's a special one. As precious to them as it clearly is to you.