Being the pussy that I am in Mass Effect, I'd try to negotiate some kind of peace.
Even if I wasn't just mashing the 'good' dialogue option over and over and actually *gasp* made my own choice, I'd probably try for a peace, because I love them both so much.
But if I has to choose, the Quarians, because they have Tali.
The Geth are slightly more badass however.
EDIT: Reading the OP again, I think I kind of misunderstood it. If I had to choose between wiping out the Quarians or wiping out the Geth, I'd choose to wipe out the Geth. But if there was a war in which both sides would emerge weaker but alive, I'd probably pick the Geth, because they would be less likely to try to kill all the Quarians, and the Quarians would probably try to kill all the Geth and not give them a chance at life.