I used to have a permit and carried, but at the time I lived in a really bad neighborhood (you would regularly hear gunshots at night, and pizza joints refused to deliver to us), was working two jobs and going to school, and often had to walk home at midnight, or later.
In that situation, I minimized my exposure (try to keep to the sidewalks and lighted areas), and keep my eyes peeled for potential trouble, sure, but it was very comforting to have the option of defending myself with something besides my fists.
Now that I own a home in the heart of white-bread suburbia, I don't feel the need.
I don't think that's an unreasonable attitude to have.