When I was playing I didn't care for him either, for the reasons you stated. Yet when the opportunity came that I could choose to kill him, I found that I just couldn't do it. I had lost a lot of good people on this journey and didn't want any more to die. At that point I figured Ben had finally learned from his mistakes. As he was dangling there, he admitted his wrong-doings and was willing to accept death as the consequence. I thought that once I saved him, he would really start hauling ass and come through in the end, maybe take down a bunch of zombies alone or come up with an unexpected but brilliant plan.
So imagine how frustrated I was when he was brutality killed anyway in the next episode without doing anything. I was like, "What the hell was that?!" It was a really cruel outcome.