It depends on how the difficulty is adjusted, as some games can make significant changes between the different settings.
Sometimes it's just not practical to change the difficulty of a game unless at specific points (between levels/missions) as the types of enemies which are present will change. The Halo series seems like an easy example, and enemies will have different ranks depending on difficulty. On easy, a spot could have a blue elite guarding it; on legendary, it's a red elite. This is in addition to all the numerical changes with a game's difficulty. Sometimes it can mean more or fewer foes as well. If the changes are purely numerical or multipliers, changing the difficulty can be done at any time with ease. Oblivion did this with "you do less damage, they do more damage" as you moved the difficulty slider.
The one time which changing the difficulty shouldn't be allowed is during the middle of a fight, and most games actually do this already. The reasons for this is both practical and to force players to at least try to complete a difficulty higher than they intended. The practical side of things is fairly obvious, it prevents spawning (or despawning) enemies in the middle of the fight and the hassle/glitchy-ness of changing their stats mid-fight (a similar case can be made for many games not allowing for the changing equipment mid-fight); this preserves the game's sense of immersion and it's stability. As for forcing players to commit to an attempt instead of them changing the difficulty mid-fight, it can help them be a better player; it won't prevent reloading of saves and changing it before the fight, but sometimes a little incentive can help. The practical reason probably rules the day here, I imagine.
---
If a game is going to allow for adjustable difficulty settings, it should be allowed as much as possible (within reasonable constraints). Restricting it to only the start of the game would be silly, unless there's some significant changes which result; even them, it should be adjustable at certain points (namely between levels & missions).