Well, being a programmer/designer by trade, it isn't hard per se, but being able to come up with the ideas yourself and utilize them in a good matter are the challenges of game designing.
Plus, certain people excel at certain things, and for them, game designing may be one of them.