There are actually many species in which the male is much smaller than the female, especially amongst insects, fish, arachnids, etc. In these cases, males tend to be solitary, roaming around to find a mate, and often die afterwards. The females, on the other hand, tend to be territorial.
However, given the long gestation cycle of a human infant, I doubt a 'fire and forget' approach to males would be valid or effective. In this case, I think it likely male behaviour would likely be similar o that of bull elephants; whilst larger than the female, they are a good example of solitary males in complex mammals. Bulls leave the matriarchal herd once they reach adulthood, and only come into contact with females when they are seeking a mate. Similar behaviour can also be seen in dolphins, although they tend to form gangs of males, as opposed to leading a solitary existence.
In short, I believe that if females were the larger, stronger sex, human behaviour would be radically different. I think it likely that females would form matriarchal societies, who would collectively form most civilisation (or at least, in a form we recognise). Males, on the other hand, would probably leave upon reaching adulthood, and adopt a hunter-gatherer or bandit lifestyle, only interacting with female societies when the need arises. Of course, this assumes that male behaviour would be the same, and not adopt a more submissive role in line with their smaller stature.