When their continued existence becomes directly contrary to my survival instinct. When they honestly give me no other choice but to kill them if I, or my loved ones, want to continue breathing. That is the point when they deserve to die from my perspective. Although to be honest, even then they don't really 'deserve' it. That's just the point when I start to value my own life over theirs.
I'm not sure I can express just how much it pisses me off when I hear people say stuff like "X person did Y bad thing. He/she shouldn't be breathing our air" as if air is a precious and painfully finite resource that we must only share out among the deserving.
Revenge killings, legally sanctioned or not, achieve nothing; and what's the point in even having 'Justice' if it's not working towards achieving something greater than the current situation. When you kill someone, you remove all possibility of redemption, rehabilitation and compensation, if ever it existed (but I've never actually seen anyone with the kind of credentials to state that it never did endorse such action, make of that what you will); nothing is gained or learned; the guilty party isn't even really punished (most deaths are quick, and once you're dead, you don't have the consciousness to be sour or sorry about anything); and the people you really end up hurting are the guilty party's loved ones, who most likely didn't do anything to you and are now only more likely to seek revenge of their own, perpetuating the vicious cycle. This is without even touching on the possibility of you getting it wrong, and killing an innocent in your righteous anger at crimes they didn't commit.