Study Says Gamers Can't Tell "Adaptive AI" From a Placebo

[thekingofcuba]

I always love seeing people's reactions to articles like this. Everyone starts crying that the study was poorly designed or some aspect of it ruined the results, regardless of whether or not they know anything about statistics or designing studies. Sure there's plenty of bad studies out there, but this one seems fine.

First off, the point of the study was not to determine whether people could tell the difference between adaptive AI and randomness. They wanted to see if the placebo effect worked in the context of video games the same way it works elsewhere. Maybe that seems obvious to you, but doing good statistics means getting data to back up what you claim.

Secondly, stop complaining about things you assume to be true and actually check the facts. Several people are talking about how the study was ruined because they had the subjects play the game twice, so maybe they just got better, etc. The 21 subjects was just the first round of the study to see if there might be anything to the idea. "A different experimental design, with 40 new subjects, confirmed the effect. This time, half of the players were put in a control group and told that the game was random, while the other half thought the game had built-in AI." There you go, problem solved.

 

[Lightknight]

I mean, maybe the game just generated a more friendly map the second time.

Or, maybe they were now more familiar with how to play the game?

Either way, it seems they fell for it.

Sure, but we also fall for suggestions that an old house is haunted.

 

[CaitSeith]

I mean, maybe the game just generated a more friendly map the second time.

Or, maybe they were now more familiar with how to play the game?

Either way, it seems they fell for it.

Sure, but we also fall for suggestions that an old house is haunted.

But, statistically speaking, it's much more likely to get a more friendly map the second time in Don't Starve, than a house to be actually haunted.

 

[Mistwraithe]

I suspect you could get the exact same results by not even mentioning adaptive AI changes. Just get them to play twice, then after the second time ask them what was better about the second map. There will obviously be differences between two random maps, you're effectively just getting the subjects to say which differences they liked.

Sounds like a terribly designed study. Unless of course there was more scientific rigor to it than that, in which case I guess it is just poor reporting?

 

[Smooth Operator]

Well the effects of placebo have been documented particularly in drugs to have a very wide range, as long as someone truly believes they ate painkillers it is more then likely their pain recedes.
Never the less this study is pretty poor, an incredibly small sample size, only one run for each subject, with random environments and highly subjective feedback. With variables so wildly divergent the tests can be changed by a huge number of factors, it is the testers themselves who came under the placebo effect as they expected one specific result.
Need to get very very specific with this stuff to eliminate any random stuff contaminating results.

And you can see this effect working all around you, people who really soak in the marketing are the perfect example.

 

[Amaror]

Poor study? More like yellow journalism.
The study was never meant to show whether or not gamers could "tell" that their playing with a great AI or a poor one. It was merely done to test whether the placebo effect work for video games in general. The "Adaptive AI" things was just used as something that they could say that would make think people they might be playing something better than they are. But of course the escapist has to phrase it like it's meant as some kind of insult toward gamers to garner clicks. I am getting sick and tired of this sites "journalism".
And i think it's good that they have done this study. The placebo effect is pretty important to consider if your experimenting with focus groups and similar types of experiments.

 

[K12]

The title of this article doesn't reflect the aim of the research. I know you get more hits when you have an article that implies "scientists say gamers are stupid" but can we try

The study was about whether or not the act of saying "this game has cool feature X" made participants report more positively about the game. I'm not suprised that it has this effect, there's a lot of research in the psychology literature that shows this kind on effect on consumers with a range of products. I'm not really sure if "Placebo" is quite the right word for it though.

It's also worth saying that reporting more positive experiences isn't the same as having a more positive experience. For the participants to say "I didn't notice any difference" is basically saying (in their minds) "this new tech you've developed was a waste of your time and money" which is a pretty hard thing to say.

"Adaptive AI" had literally nothing to do with the research objectives. It was just a vague positive sounding feature who's effects wouldn't be readily apparent to the player in a single playthrough.

 

[Lightknight]

I mean, maybe the game just generated a more friendly map the second time.

Or, maybe they were now more familiar with how to play the game?

Either way, it seems they fell for it.

Sure, but we also fall for suggestions that an old house is haunted.

But, statistically speaking, it's much more likely to get a more friendly map the second time in Don't Starve, than a house to be actually haunted.

A randomly generated map is not unlike a haunted house in which some things appear random (creaking of floor boards, a tree in a different place, etc). Suggestion in either occasion can make individuals see things that simply aren't there.

Both examples are a suggestion that there is an intelligence impacting the environment for good or ill.

 

[CaitSeith]

I mean, maybe the game just generated a more friendly map the second time.

Or, maybe they were now more familiar with how to play the game?

Either way, it seems they fell for it.

Sure, but we also fall for suggestions that an old house is haunted.

But, statistically speaking, it's much more likely to get a more friendly map the second time in Don't Starve, than a house to be actually haunted.

A randomly generated map is not unlike a haunted house in which some things appear random (creaking of floor boards, a tree in a different place, etc). Suggestion in either occasion can make individuals see things that simply aren't there.

Both examples are a suggestion that there is an intelligence impacting the environment for good or ill.

Yeah, and? That doesn't make the probability of the game generating an easy map disappear.