Achievements come in 2 flavors (3 really, but the third is pretty rare)
The first flavor is the casual or completion achievements. The purpose of these achievements is to provide positive feedback, and show proof that something was completed in the game. Some people, largely casual players or players who play a lot of games but don't focus on one to the detriment of others, favor these achievements. They allow you to track your overall progress through the game. Some older games are ENTIRELY these types of achievements, TES4: Oblivion for instance, only has Completion achievements. Other games have very few of these achievements - Blue Dragon for instance does not have very many completion achievements. In multiplayer, these are usually low pressure achievements, requiring you to play, but not necessarily win rated matches, or to win a number of rated matches over any period of time. These achievements are difficult if not impossible to "lock out", or bypass. They are also the most likely to be "Secret", as they usually contain plot spoilers in their names and descriptions.
Examples: "Finished the Level" achievements, "Finished the game on any difficulty" achievements, "Won # of online games" achievements.
The other flavor are actual achievements. I like to call them Trick Achievements. These achievements show off particular dedication to some aspect of the game, and are used mostly as bragging points, or to give the player a sense of completion. They are not always difficult, but they almost always require some dedication to get. Sometimes games will have easy "trick" achievements that are gotten accidentally. Things like "Get 3 headshots in a row". It will probably happen during the course of play without really trying for it. Other games incorrectly award these achievements with almost no work and they're almost automatic, but they still count as "trick" achievements because of what they ask to do - particularly Avatar the Last Airbender, whose only achievements are "Combo" achievements and it's easy to get all 1000 gamerpoints in a single combat. These achievements are also the "Beat on Specific Difficulty Setting" or "Beat using a single weapon" or "Win 10 ranked multiplayer matches in a row" achievements. They are also the "Collect all macguffin" achievements. The achievements can be easy or hard, but they usually cannot be achieved during the course of normal play, and they can often be "locked out", by missing certain things early in the game, causing you to need to replay the game to get them.
The Trick Achievements are the interesting ones, but I think a game that properly utilizes achievements has a good mix of Completion Achievements and Trick Achievements. In a "good game", you should be able to get 300-500 points just playing through the game, and the other 500 points should be dedicated to either extraneous completion or performing feats of awesome.
The third flavor are joke achievements. Sometimes it's as simple as walking into a particular room (which is technically a Trick achievement, but I digress), playing against someone who has a certain achievement or qualification ("Viral" achievements like the one in Brutal Legend that leads back to Tim Shafer, or Grand Theft Auto 4's "Play against a Rockstar Employee" achievement), or failing something so catastrophically that the game has to poke fun at how bad you are by giving you a permanent mark of shame - such as the "Long Road Ahead" achievement for failing a song on Easy in Guitar Hero 2.
In Short, Achievements provide several incentives, from simple positive reinforcement and tracking your progress through a game, to showing off your more difficult accomplishments. On top of the actual achievements themselves, some people compare gamerscore. I don't think that's the way to go, but there it is. I'm more interested in individual achievements because each one has a story (at least the difficult Trick achievements), Gamerscore is a boring quantifier.