I have a similar view as OP. I don't do drugs, and have no desire to do drugs (other than maybe an occasional beer), but at the same time I don't condemn those who do drugs. However, I do condemn those who try to convince me to do drugs when I clearly don't want to.
Honestly, I know some really cool people who always smoke pot/cigs, I just don't hang around them when they do so.
As for why I don't want to do drugs. Well, I don't feel like I really need them for any reason. I'd rather spend money on some of my other hobbies (playing music is expensive) that I feel I enjoy a lot more than I would ever enjoy drugs. I also don't necessarily like the method in which you have to take drugs, whether it be needles, through smoke, or whatever. Obviously, there are health risks, but to be honest pot isn't really that bad for you at all.