Well, in theory, you would be very attracted to the appearance of said clone (according to that one episode of CSI I saw a while back). Personality would be a deciding factor here.
I might be able to stand it. My mind is a roller coaster, going up and down constantly. As long as our mood roller coasters were not in sync, it would work out nicely. In sync coasters would result in one of us killing the other in less than 24 hours though.
(Actually, if I had an opposite sex clone, I'd get a brain transplant done and test if the whole "swapping bodies" effect can actually happen. If it is a clone, then there is a zero chance of the body rejecting the brain.)