It's not really a big deal. He might have just thought you guys did it because you knew the girls were out there. Girls mature fairly early. Plenty of girls have pretty much completed puberty by the time they turn 13-14. Any normal protective father dreads that age and does everything possible to keep his daughter innocent and the little girl he has always known her to be.
I'm not accusing you of anything, but I've found that 9 times out of 10, a shirtless guy is only shirtless because he wants to show off/seem cool. Frat boys, meatheads, wannabe rockstars, and just general douchebags seem to be the vast majority of shirtless man-whores in the world.
I avoid taking my shirt off in public, if only to avoid being associated with the above. Unless you're swimming, at the beach, or in the privacy of your own home there is no reason or need to be shirtless. Depending on your skin tone, wearing a light white shirt may actually keep you cooler than wearing no shirt at all.
Summed up? Not a huge deal, but removing your shirt when you know people you aren't close with can see you is considered immature and a desperate plea for attention. It's like the attention wh**e who wears a very revealing bikini to a neighborhood family friendly barbecue.