Hybrid really.
I have things grouped into multiple areas, more or less.
Favorites, Anime, Star Trek/sci fi (yes, there's enough star trek for it's own shelf. Immediately next to it is Red Dwarf, if you must know)
Recent stuff (a shelf which isn't really sorted, but just contains my most recent films),
Then there's a collection of what's mostly childrens films, and a general 'film' collection for everything else.
The Anime, Star Trek, and favorites areas are sorted are in series order, then alphabetically, and everything else is alphabetical.
I just don't find much need to figure things out by genre much though. And I'd have a hard time deciding what belongs where regardless.
Alphabetical order is just a convenience to prevent total randomness so there's at least some way of figuring out where something I own is.