It all depends on whether or not their argument is based on personal experience. If it is, the relevance of that experience is certainly open to scrutiny.
For example, take the affirmative action debate. A common refrain (usually from white people) is, "I was poor, but I worked hard, got good grades, and nobody helped me one bit."
That may be perfectly true. However, a black student might be poor, work hard, get good grades, but because he's black, people judge his work to be inferior to his white counterparts.
However, lets say the black guy makes it big and when talking about world wealth inequality says, "I was poor and discriminated against, but I worked hard, got good grades, and made something of myself. The system is fair." He's got privilege too, because despite all this, he had the wonderful privilege of living in the U.S., an advanced first world country, as opposed to being a poor, minority student in, say, Burma.
Privilege means you don't notice those sorts of things. It means that your experience, even with all of its struggles, may still be considerably easier than the experiences of others, solely due to the status conferred by your race, religion, sex, gender, orientation, or even what country you live in.
It's not a discounting of experience. It's saying that your experience isn't the only one out there.