I think people need to remember:
The reason he skipped the country was the presiding judge at the trial is alleged to be about to overturn his plea/the deal the lawyers made in order to boost his own fame and personal career.
When you have a member of the Judicial system putting their own status before that of justice then I can't blame the man for running, as it was certainly not going to be a "fair trial" - and if I remember correctly, another judge had already agreed when Polanski appealed that the presiding judge in the first trial was corrupt.
He also has done time already. Admittedly only 45 days or something pathetic like that, but it means he has faced what the legal system viewed as "just punishment" for some of the charges (I think it was for providing drugs/alcohol to an under-age girl).
I still think he should be brought to justice, but I highly doubt he will get a fair trial this time around. The whole case is extremely high-profile and I get the feeling that America will "make an example of him". Case-and-point, Polanski has been travelling to Switzerland all throughout the last 30 years (he even oversaw the building of a house there) yet American authorities choose a high-profile event such as an award ceremony to pick him up?
Maybe the victim's feelings will be brought into the equation by the judge, but this poor woman's life is about to be turned upside-down by the media, which is not fair in any sense of the word. She's about to become a victim again, this time to the shoddy practices of modern journalism.
Essentially: Polanski fully deserves to face trial and whatever punishment on the grounds of breaking the law as the State is pushing for a conviction. However I think the whole process of taking on the defendant when the victim doesn't want the charges to be upheld is a difficult situation, and that the trial is unlikely to be fair and there is a real risk of an "example" being made of the man instead of proper justice.