They want to make it realistic.
You carry a weapon into combat, because you're a specialist(most of the time).
You're either a rifleman, or a specialist. A specialist can be a sniper, a specialist can be an anti-tank soldier walking around with a rocket lancher on his shoulder, a specialist can be a machine gun crew operating.... a machine gun.
The secondary weapons are usually personal and reflect your tast, just like in the army and poilce here you have a secondary pistol that is usually hidden, to defend yourself when the weapon jams, runs out of ammunition or other bad situations happen.
But this doesn't happen. You always have ammunition in these games, the weapons never jam or break. Therefore you do have a point, but in the other hand the more weapons you have the less time you will use each of the individualy.
Most importantly, in todays multiplayer games you simulate the military - each has its own speciality, whether you are carrying a sniper rifle, a machine gun (do they still do this?), a medical kit, a rocket lancher and so on.
In addition to that, you don't have training in every single f**king gun there is on the planet.
You might know how to handle a rifle and a sub-machine gun, but perhaps your hands are too shakey and you haven't gone through the necessary training to hit an apple 800 meters away.