It all comes down to how to define "fun", but in my head, "fun" is the minute-to-minute laughy-smiley-bouncingoffthewalls type of hilarious time you'd have doing silly things with your friends. On the other side, whatever terminology you decide to use (I use words like "good", "entertaining", and "compelling"), there are the games that aren't necessarily "fun" but engrossing and immersive, with the masterfully crafted gaming experiences that'd make you turn to your friends and say "This game is amazing".
"Fun" is certainly a big part of most video games like casual games, but there are plenty more video games where the entertainment value doesn't solely come from "fun". For example, horror games, or games that have a lot of focus on the story and narrative. I think we can all agree that games like Amnesia, Silent Hill, or KoTOR aren't "fun", but more entertaining and good because they throw you into a rich, immersive world, free for you to explore. It's the compelling gaming experience that matters.
There's a clear distinction between "fun" games like Just Cause 2, Mario Kart, Angry Birds, and Peggle, and then "good" games like Amnesia, Silent Hill, and KoTOR. Some are there purely for the fun-factor, others are there to tell a story and create a compelling gaming experience. Obviously the distinction isn't that black and white, since games that are both fun and compelling like Zelda and Uncharted, but you get the idea.
That same distinction is in films. Most films aren't "fun", but you'd still call them great movies. For example, The Dark Knight, Schindler's List, Citizen Kane, Titanic, and so on. These movies aren't very "fun" to watch, but they're great for other reasons.
I think this entire discussion stems from the implications of the word "game", and the fact that video games are interactive. The word "game" is almost directly linked to the word "fun", and it seems that if a game isn't "fun" then there's no point in playing it. Perhaps this used to be true, back when we only had card games, board games, and imaginary games, like The Floor is Lava, Deuces, and Monopoly. But with video games, it's become an entirely different story.
Video games have become just as relevant of a medium as books and films, both of which partly exist in video games. It's become a new way to tell stories and create compelling experiences, to make you sit and think about yourself and the world around you. So no, video games don't have to be fun, they can be much more. And I'm sure that we can all name a few of our favorite video games that aren't necessarily "fun" but are great for other reasons.