Well, if Group B tells you about the stuff they'er doing, does it sound interesting to you? If it doesn't, maybe they know you well enough to think it wouldn't be worth inviting you because you'd feel obliged to go and wouldn't enjoy the trip.
Then again, if it sounds like stuff that does interest you, then it may not be worth sticking around with 'em.
My social skills are minimal to say the least, but I'd at least ask if you can tag along once or twice. You don't have to go apeshit on them about how they never invite you, just drop in the hint that you want to hang out with them sometimes.
At the very least, Group B still comes and talks to you. That's something, right?