I'm scared to drive, but whenever I do I drive flawlessly as my instructors always tell me. I never take any unnecessary risks, and never go above the speedlimit, because there's no point to. The thing that scares me the most about driving? Experienced drivers. They're always trying to justify "I've done it for years, blah blah blah, nothing ever happens to me!" and they take some seriously panic-inducing risks because they're "experienced." Many experienced drivers I've been with do some seriously stupid driving shit, such as nearly bumping into other cars because they're pissed at the driver, jumping infront of oncoming traffic too slowly and causing the other cars to suddenly break to avoid hitting their dumb asses, etc.
So in essence: What scares me the most about driving are those who claim to be experts, as they take seriously unnecessary risks that get people killed and/or hurt.