I know exactly what you (OP) are talking about. Ecco is one of those games I was barely able to get past the first level, and not because of the difficulty. There's something about that game that built up a lot of dread the first few minutes of playing it that got me to drop the game way before I could literally say I gave the game a chance, and I think it has a lot to do with most of what's been said here.
First off, there's this sense of vulnerability given the fact I'm playing a dolphin. Unlike some machismo spewing gun totting psychopath (the majority of game protagonist) - the fact that I'm an animal that I could develop more sympathy towards compared to the latter case - it felt unnerving playing one. I don't want to get Ecco killed; I like dolphins generally more than I could like people, and that's why it was difficult for me.
Now there's more to it than that.
Another thing is just about every game I've played where surfacing to breath was the gameplay has always been unnerving for me. From games like Sonic the Hedgehog, all the way to Metal Gear Solid 2 and being responsible of EE in the swimming portion; that whole "most surface for air" gameplay has always been some of the hardest bits to play through. Now take Ecco where it was always a factor in the gameplay, and well... there you go!
And finally... and I'm not sure about anyone else over this matter... but... the ocean scares me. This is an entire game that takes place in the one place someone like me could feel the most vulnerable, and given the fact I'm playing an innocent creature like a dolphin, well... it was all just too much for me.