My dad was, and is, a punishment for my sins.
I never go to see him nowadays, (I stopped doing that as soon as I could) but when he and Mom were still married, which they were for about twelve years he was something I was stuck with. He didn't use any violence - the thought of using violence probably never even entered his mind - but he was and is the most evil person I've ever known. As you might point out, normally people don't know a plethora of Incredibly Evil People, so that epithet doesn't really tell you a lot. Thus creating a need for details. But to spare you the pain of reading about it (and, more importantly, me the pain of trying to remember it) I'll limit my sad story to this:
1. He made sure to yell at us for every bad thing we did. And I mean every bad thing.
2. He has some sort of Syndrome, like Asberger's or something like it. He has never been diagnosed with it, (he refuses to see a psychologist about it) but everyone agrees that he's got it. So he very often complained about things that no-one else ever saw a problem with, like the mats not being completely tidy in every detail, or the kitchen sink being wet.
3. I regularly went to sleep crying as a kid, the cause being that "I have to live with someone like HIM!"
4. Whenever I had to be with him, it was like I was being forced to do extra geography homework in the sense that I never even considered the idea that it might be something to be happy about.
5. He never ever took responsibility for his mistakes. So whenever he made a mistake, like knocking his glass of water over or driving badly, he explained to Mom how it was her fault and forced her to apologize for the mistakes he had made. Everything was her fault because he had decided that it was.
6. And whenever you tried to argue with him, he always said "don't be ridiculous." To him, that was all the proof that he was right that was needed.
7. His arguing was often very illogical. When I told him "I went there because I thought this was going to happen" he would argue thatI couldn't possibly have thought that was going to happen, honestly believing he knew more about my thinking than I did. As far as he was concerned, he always knew best.
8. This went on for eleven years.
9. It's been difficult remebering all of this, because I've repressed almost all the parts of my childhood that included him. Major parts of my life until I was 11 are completely gone. I remember almost only the parts where books and comics (I loved, and love, books and comics) are involved.
10. Today, I cannot understand the idea of being happy that you have a dad. When I was a kid, I wished for a dad who was nice, but today a nice dad is like 2+2=5. It doesn't make sense, no matter how I see it. A father is by definition something unpleasant to me.
So to answer your question; the worst thing my dad ever did to me was simply being who he was. It wasn't any single event, it was doing all this for eleven years.
My mom? The only really bad thing she ever did was not divorce him until I was eleven because she thought "Children need to grow up in a proper family". That made me really positive towards gays getting kids, because after growing up in a household like that you quickly abandon the idea that man+woman=best raising there is.