I have no problem with digital downloads of games. Heck, it's particularly convenient to be able to download games to handhelds and carry your entire library with you for no extra effort. However, and it's a big however, I expect it to be cheaper than buying the boxed copy, and that doesn't seem to happen often. Presumably part of the cost of a game is the shipping, manufacturing and retailer markup that occurs with physical copies. Digital obviously removes those hurdles, so the game should justifiably be cheaper. Even if you claim that publishers are "footing the bill" for the manufacture and shipping, that still doesn't excuse the retail markup.
The other thing about all-digital is, as many have said, the lack of any tactile representation. I think, on some psychological subconscious level, one feels more validated purchasing a physical object than a digital one. And, a lot of games now come with nifty little incentives to pick up a boxed copy, and I'm a sucker for that sort of collectable stuff.