Considering fans of the two hate each other, I think it would not be a good idea. You being a fan of both makes you a real rarity. I'm rarely a fan of crossovers in general myself. Unless the people have really good chemistry, it comes off being awkward, and forced. I feel an AVGN/Irate Gamer crossover would be the pinnacle of forced crossovers. Besides, James Rolf(AVGN) is super busy these days. He barely puts out new episodes of Angry Video Game Nerd, I would rather have him focus on doing proper episodes over working with some other internet personality.
Just between you and me(and anyone reading this) I've never cared for AVGN crossover episodes. The Nerd is at his best when he's by himself, or with Mike Matei in some character role, along with his guitarist friend, whose name escapes me at the moment. Otherwise, his crossovers have been pretty meh, and the Toxic Avengers episode is easily the worst AVGN ever made. Way too cringe worthy.