All kinds of extreme metal, but mostly technical death, progressive, avant-garde (such a pretentious name for a subgenre) and a dab of mathcore.
I know what you're saying, but it sounds like you're just scraping the surface. I found it all pretty intimidating at first too, but if you dig...