As many have stated above with your logic you would consider anything that is not "productive" as a waste of time. However the term productive is being loosely defined. You could argue that anything beyond keeping yourself alive, reproducing, and protecting your offspring is entirely a waste of time. Thus civilization as we know it is just a big waste of time. If you think about it, we might even be better off in terms of species survival if we never bothered to be "unproductive". Obviously the next line of thought would be to start defining productive in higher modern terms, all of which would revolve around what goal? Prolonging the three main points previously mentioned, but to what end purpose? Why to attempt to further those ... and I bet you can see where this is going. It's a boring cyclical thought process.
Now, we could take an alternate hypothesis and say that the only reason to be productive is to create more spare time to "waste". However, we don't really mean "waste", we really mean "experience" other things. One could argue that regardless of how much it truly accomplishes any act can still teach you something. What it teaches can vary wildly, and while some of those skills/feelings/ideas/etc... may not always serve an immediate personal and practical use, they may serve a purpose down the line or to someone else they are passed onto.
I present you this final closing story to illustrate:
I've had various conversations with my grandfather whom you could say was a down to Earth no-nonsense businessman of the past 80 or so years. He had no particular love or hate of things many of us were interested in, such as Computers, Video Games, and any other media aside from non-fiction or realistic fiction. Most of these conversations were in the realm of politics, general philosophy (without using actual philosophy terms/concepts as he considered it on the same level as fantasy), and business. After one of these conversations with some of my cousins and myself he asked me where I had learned some of the things we were talking about. He was surprised how much we agreed on, and while I know he didn't mean it as an insult, he knew where my general interests lied and they were pretty far from his. And so I explained to him that most of the ideas I had shared or discussed were learned or at least touched upon in the various games I played, books I read, and movies I watched (primarily fantasy and science fiction), and refined through interaction with him and others I've talked to or learned from. He finished with: "I might not get it, but you learned more then your dumb-ass cousin. Guess you're onto something."