Probability in Games (XCOM)

[BloatedGuppy]

This was a good article on probability in gaming, using the new XCOM as an example. This was a hot button issue on these forums a week or so ago in which a demographic of individuals was yowling about the cheatin' ways of random, so I thought I'd share it here.

http://sinepost.wordpress.com/2012/10/26/probability-in-games-xcom/

This part in particular amused me...

Your shot either hits or it doesn?t, and then damage is randomly calculated thereafter. But the key thing is whether the shot connects. Waste your chances with 25% shots, and you?ll soon get overpowered. It?s better to maneuver around, destroy the aliens' cover, and then shoot with an 85% chance. And when it misses, you can exclaim stupid things like "My 85% shot missed! What?s the chances of that!" (Spoiler: 15%).

 

[Luca72]

I'm taking a statistics course right now that just started talking about probability. Of course I was immediately thinking "How can I apply this to XCOM?". This article is awesome, in particular the breakdown of rapid fire. 33% to 97% accuracy get a marked accuracy bonus eh? Good stuff!

 

[More Fun To Compute]

Did he actually collect data on hits and misses to see if the reported % was a plausible figure for the actual underlying probability? Firaxis are notorious for thinking about probability displayed on screen in psychological terms. In other words considering lying about % chances to win to confirm faulty expectations of success or other reasons.

 

[BloatedGuppy]

Did he actually collect data on hits and misses to see if the reported % was a plausible figure for the actual underlying probability? Firaxis are notorious for thinking about probability displayed on screen in psychological terms. In other words considering lying about % chances to win to confirm faulty expectations of success or other reasons.

There was a guy on the official forums for XCOM who parsed hundreds of shots. Average predicted accuracy was 67%, his results were 66% (decimal places excluded because I can't remember them). If you're curious about bias, the guy running the experiment posted it in his thread...where he'd originally blasted the game for deceptive accuracy numbers after a bad run of luck on 50 shots.

Where you will run into trouble with XCOM is if you are save scumming, because the game does not regenerate its random seed on a reload. So if you miss a 95% shot on your first attempt, you'll miss it on every subsequent attempt too no matter how often you reload.

 

[More Fun To Compute]

And the same tests were repeated with a range of different % to hit chances? As in 50 trials with 25% chance and 50 with 85% and so on.

 

[BloatedGuppy]

And the same tests were repeated with a range of different % to hit chances? As in 50 trials with 25% chance and 50 with 85% and so on.

As I say, the 67% was the average, so I'm assuming a wide range was used.

I find it far more likely that people are being predictably human and whining about randomness then that Firaxis coded in a special algorithm that only screws you over on particular % shots.

 

[More Fun To Compute]

If we are talking about this in statistical terms then assuming that the % to hit is equally reliable for all values is flawed. If we are talking about this in terms of statistics.

 

[Bostur]

It's a good article about random chances in gaming, although the choice of game used as an example may not have been the best. ;-)

The article of course assumes that what you see is what you get, which can be proven is not the case in XCOM, because sometimes the UI shows different probabilities for the same shot. So it's a given that at least some of those numbers must be lying. If a shot has a 45% chance to hit, it can't have a 25% chance to hit at the same time. 'Features' like that probably twists the psychology of players more than usual.

Even without the UI bugs it still raises some pseudo-philosophical questions about what random numbers are. A topic that Firaxis themselves have wrestled in the past by tweaking the numbers behind the scene. A typical setup of software and hardware used for gaming can not reliably make a random number. In most cases it is possible to generate numbers that are random enough for the purpose, but sometimes games run into limitations. In my experiences it most frequently happens when a game needs to generate a lot of numbers in rapid succession. In those cases there is a risk of 'streaks' of samey numbers. The most notable example in XCOM is getting the upgrade that makes all new reqruits squaddies. When hiring a lot of troops at the same time, chances are there will be a lot of class duplicates, many more than one would expect by rolling a physical dice. The presence of long or frequent streaks doesn't rule out the possibility of a perfect statistical distribution over a long time.

If we make 20 shots and assume 50% hit chance, we would expect aproximately 10 hits and 10 misses. Even if we get exactly 10 hits and 10 misses it's possible to get results that feels very non-random. A few examples where H is a hit and M is a miss:

HHHHHHHHHHMMMMMMMMMM

HMHMHMHMHMHMHMHMHMHM

HHHHHMMMMMHHHHHMMMMM

In all examples there is a perfect 50% distribution between hits and misses, but it would feel very wrong and 'non-random' in a game. The last outcome is probably the closes to what happens when random number generators in games gets criticized for being broken.

It's very hard to tell what an acceptable random outcome must look like, but we can often instinctively tell when something isn't right. Our brains instinctively pick up on those patterns, and often pick up on patterns that isn't even there. Sometimes it's broken and sometimes we just imagine that it is.

 

[More Fun To Compute]

It's very hard to tell what an acceptable random outcome must look like, but we can often instinctively tell when something isn't right. Our brains instinctively pick up on those patterns, and often pick up on patterns that isn't even there. Sometimes it's broken and sometimes we just imagine that it is.

It's not that hard if you have the software and training. But the confidence interval you get from only 10 samples would be too wide to be useful probably.

 

[Rack]

It's very hard to tell what an acceptable random outcome must look like, but we can often instinctively tell when something isn't right. Our brains instinctively pick up on those patterns, and often pick up on patterns that isn't even there. Sometimes it's broken and sometimes we just imagine that it is.

We really can't. Humans are plain awful at guessing how random numbers should work. If it feels appropriately random then it definitely isn't.

 

[Tallim]

Did he actually collect data on hits and misses to see if the reported % was a plausible figure for the actual underlying probability? Firaxis are notorious for thinking about probability displayed on screen in psychological terms. In other words considering lying about % chances to win to confirm faulty expectations of success or other reasons.

There was a guy on the official forums for XCOM who parsed hundreds of shots. Average predicted accuracy was 67%, his results were 66% (decimal places excluded because I can't remember them). If you're curious about bias, the guy running the experiment posted it in his thread...where he'd originally blasted the game for deceptive accuracy numbers after a bad run of luck on 50 shots.

Where you will run into trouble with XCOM is if you are save scumming, because the game does not regenerate its random seed on a reload. So if you miss a 95% shot on your first attempt, you'll miss it on every subsequent attempt too no matter how often you reload.

There was also a guy who used an Unreal Engine script decompiler to actually look at how the RNG is determined.

http://www.schwanenlied.me/yawning/XCOM/XCOMPRNG.html

 

[WoW Killer]

It's a good article about random chances in gaming, although the choice of game used as an example may not have been the best. ;-)

The article of course assumes that what you see is what you get, which can be proven is not the case in XCOM, because sometimes the UI shows different probabilities for the same shot. So it's a given that at least some of those numbers must be lying. If a shot has a 45% chance to hit, it can't have a 25% chance to hit at the same time. 'Features' like that probably twists the psychology of players more than usual.

Even without the UI bugs it still raises some pseudo-philosophical questions about what random numbers are. A topic that Firaxis themselves have wrestled in the past by tweaking the numbers behind the scene. A typical setup of software and hardware used for gaming can not reliably make a random number. In most cases it is possible to generate numbers that are random enough for the purpose, but sometimes games run into limitations. In my experiences it most frequently happens when a game needs to generate a lot of numbers in rapid succession. In those cases there is a risk of 'streaks' of samey numbers. The most notable example in XCOM is getting the upgrade that makes all new reqruits squaddies. When hiring a lot of troops at the same time, chances are there will be a lot of class duplicates, many more than one would expect by rolling a physical dice. The presence of long or frequent streaks doesn't rule out the possibility of a perfect statistical distribution over a long time.

If we make 20 shots and assume 50% hit chance, we would expect aproximately 10 hits and 10 misses. Even if we get exactly 10 hits and 10 misses it's possible to get results that feels very non-random. A few examples where H is a hit and M is a miss:

HHHHHHHHHHMMMMMMMMMM

HMHMHMHMHMHMHMHMHMHM

HHHHHMMMMMHHHHHMMMMM

In all examples there is a perfect 50% distribution between hits and misses, but it would feel very wrong and 'non-random' in a game. The last outcome is probably the closes to what happens when random number generators in games gets criticized for being broken.

It's very hard to tell what an acceptable random outcome must look like, but we can often instinctively tell when something isn't right. Our brains instinctively pick up on those patterns, and often pick up on patterns that isn't even there. Sometimes it's broken and sometimes we just imagine that it is.

It's basically about compression. You can do an easy demonstration with image files:

Load up the GIMP (I don't know of an equivalent filter in Photoshop).

Make a new image; we'll start with 128x128.

Select Filter -> Noise -> Hurl.

Set Randomization to 100%, then put seed and repeat on whatever you like.

You should get something that looks like this:

Now, export as a Bitmap, export once again as a PNG, and compare the images:

(since we've made an image at random, there's a small chance this won't work, but then there's also a small chance we generated the Mona Lisa)

As you can see, the PNG (compressed) is larger than the Bitmap (uncompressed). That's the gist of it. Randomly generated strings tend to be hard to compress. When they're easily compressed (which is of course possible when generated at random), that's when they stand out to us as having some sort of pattern.

 

[gideonkain]

I have missed a shot with a gun that was embedded in the alien's forehead, the gun model literally overlapped with the face of my enemy.

Randomized shots are a very difficult concept to quantify, it's so easy to say to someone "Well if you missed a 95% shot you must have rolled a 1 to 94." but over protracted experience with the game, I personally feel that the base shot percentage should be increased.

On the other side of the coin, I have successfully Critted an alien while firing through two derelict cars and a coffee shop.

Based on my experience, sniper shots are too likely to hit at extreme distances without reason and nearly point blank shots miss more than is acceptable.

It's a minor tweak in the overall mechanics of the game, but the enjoyment of feeling you aren't being victimized by a Random Number Generator is priceless.

As you can see, the PNG (compressed) is larger than the Bitmap (uncompressed). That's the gist of it. Randomly generated strings tend to be hard to compress. When they're easily compressed (which is of course possible when generated at random), that's when they stand out to us as having some sort of pattern.

I applaud you for your in-depth exploration of the issue, but compression algorithms are not equivalent
to a random number generator.

Your comparison is like comparing a novel translated from another language into German or French.

Just because one (let's say German) can be contained in 100kb and the other (French) in 110 kb does not mean that the German version is more succinct.

 

[WoW Killer]

I applaud you for your in-depth exploration of the issue, but compression algorithms are not equivalent
to a random number generator.

I'm not saying they are? Not sure where you got that from. I said that randomly generated strings tend to be hard to compress. Through this we expect, incorrectly, that this is always the case. When an RNG creates a more easily compressible string it stands out to us.

If you generate an image by hand, without using a pseudo-RNG...

...then you'll likely find the PNG to be significantly smaller than the Bitmap:

That tends not to be true of randomly generated images, so long as they are large enough.

What Bostur was saying is that local patterns are what can make an RNG appear as if it were malfunctioning, even when it is working correctly. I was merely adding that this is a definable thing (in other words, it's not always us creating significance where there is none).

 

[Klaflefalumpf]

If this Xcom incarnation has taught me anything, it's that a higher percentage seems to mean a higher chance to miss.

 

[KiloFox]

Did he actually collect data on hits and misses to see if the reported % was a plausible figure for the actual underlying probability? Firaxis are notorious for thinking about probability displayed on screen in psychological terms. In other words considering lying about % chances to win to confirm faulty expectations of success or other reasons.

There was a guy on the official forums for XCOM who parsed hundreds of shots. Average predicted accuracy was 67%, his results were 66% (decimal places excluded because I can't remember them). If you're curious about bias, the guy running the experiment posted it in his thread...where he'd originally blasted the game for deceptive accuracy numbers after a bad run of luck on 50 shots.

Where you will run into trouble with XCOM is if you are save scumming, because the game does not regenerate its random seed on a reload. So if you miss a 95% shot on your first attempt, you'll miss it on every subsequent attempt too no matter how often you reload.

sounds like the way Fire Emblem does it to me... i ran it on an emulator once (i owned the actual game too so it was technically legal. pipe down...) and hit/miss was always the same when i made the same move after loading a savestate. and i rock at Fire Emblem so XCOM should be a cakewalk for me. i havn't got it yet but i want to.
of course, i never made an attack in Fire Emblem that had below an 80% hit rate.

 

[Smooth Operator]

Yes but it's not about the raw numbers, it all hangs in the perception of consistency.
Like someone already mentioned a 50% probability is equally valid when missing 100 consecutive shots or only one intermittently, but one instance will feel like complete nonsense and the other what we expect should happen.

And that is the thing with games, you always need to make sure shit is well presented/consistent or people become disconnected from their actions and gradually refrain from coming back.

 

[DoPo]

I'll leave this here

And just say that any game ever that included any form of RNG, has had loads of people argue about it, since most don't know how probability works. Well, sometimes the RNG is broken but claiming it is without supporting analysis and evidence is just stupid.

 

[ODWX9K]

The key that people need to keep in mind is just because your experience shows that you always miss on X-percent shot doesn't mean that the X-percent stat is necessarily wrong, just you have a different experience.

Take 5 different people together. You could have one that has 0% accuracy, one that has 25% accuracy, one with 50%, one with 75%, and one with 100%, which would combine to be a 25% accuracy average. Take another dozen people, you could get the same results. You could also get different results.

 

[Mycroft Holmes]

http://www.escapistmagazine.com/forums/read/9.391083-X-com-isnt-hard-its-stupid?page=4#15748158

This post best explains my feelings on the matter.

And for those who don't want to read it. I missed about 80 shots in a row all of them above 50% some as high as 95%. Even if we assume all the shots were only 50% coin flips. Given a normal random math distribution the statistical probability should be .0000000000000000000000008272% that such a series can occur. Which is a ridiculously low probability that if 5 million people buy x-com and shoot at 1 enemy every 20 seconds nonstop every day. It should take them 15,220,700,000,000 years(over 15 trillion years of playing this game) before they encounter such a series. If the % to hit listed is the real % to hit and the mathematical distribution is mathematically normal(IE numbers below 50% come up as often as numbers above 50%.)

Take 5 different people together. You could have one that has 0% accuracy, one that has 25% accuracy, one with 50%, one with 75%, and one with 100%, which would combine to be a 25% accuracy average.

The mathematical average for that series is 50% not 25%.

http://www.basic-mathematics.com/finding-the-average.html

0+25+50+75+100=250
250/5=50