My gut instinct used to be to say "you have to tolerate them, but you don't have to do any more," but then I actually thought about what that would be like when I applied it. What would you say to someone who's racist? "I know you don't like black people, but you have to at least tolerate them." Or someone who's homophobic? "I know you think all gay people are sinners who are going to burn in the eternal fires of hell, but you have to at least tolerate them." And when I think about it that way, "tolerance" seems like an awfully weak criterion for human interaction. A society which "tolerates" people of other ethnicities isn't going to treat them fairly. A society which "tolerates" homosexuality can revoke rights far, far too easily.
I honestly think that from an ethical perspective, you do actually have a bit more of a duty than mere tolerance. You have to examine your prejudices, and ask if you have any rational reason to dislike someone. And if not, you should do your best not only to tolerate them, but to accept them. Obviously, we should probably not accept (or maybe even tolerate) the KKK or WBC. But you can't really ignore the fact that from a historical perspective, "tolerating" people who haven't done anything wrong is absolutely unfair to them from a social standpoint.