Recently, the Steam video game distribution platform changed the way it handles user-submitted reviews. Reactions have been mixed, to say the least.
Users have been able to submit reviews on Steam for several years now. Over time, this feature has evolved to include a number of components. Reviews themselves consist of a thumbs-up or thumbs-down vote, much like YouTube ratings, and then some amount of text. It could be as little as a few words, or many paragraphs. Reviews can then be rated by users, answering the question “Was this review helpful?” A user may answer “Yes”, “No”, or “Funny”, with the latter option being present because Valve, the company that develops Steam, noticed that many reviews were marked as helpful mainly because they were funny (and not genuinely helpful).
Near the top of each game’s page is then an aggregated review score, ranging from “Overwhelmingly Negative” to “Overwhelmingly Positive”. Not long ago, the display of this rating was modified to show two versions of it: one based only on recent reviews, and one based on overall reviews. The “recent” rating was added so that games which improved (or, in some cases, got worse) over time could see that reflected in the aggregate score.
Note that the score isn’t presented numerically, but as a qualitative value, though it’s not hard to figure out what each displayed rating text means.
These scores are, of course, important to selling games. I know I’m less inclined to buy a game if its reviews are “Mixed” or worse. And it turns out that a lot of games–at least many hundreds out of the 10,000 or so games on Steam–were using unscrupulous measures to prop up their scores.
To understand how this works, you must be aware of what developers can do on Steam. If you are the developer of a game on Steam, you have the ability to generate activation keys for your game at will, for free. It’s your game–you can do whatever you want with those keys. This is how developers distribute their keys to bundling sites (like Humble Bundle) and other third-party sellers. Developers also generate keys for reviewers, too. Steam has no way of tracking the origin of a key–it only knows whether the game was bought directly from the Steam store, or was activated with a key.
What some developers have done is give away free keys–sometimes dozens, sometimes many more–with the understanding that in exchange for this free game, a positive review is expected. Even if a good review isn’t explicitly asked for, it’s a well-known phenomenon that people who receive something for free and then expected to comment on it are likely to speak positively, in part because we feel obligated not to look a gift horse in the mouth, but also because we anticipate receiving more free items in the future–and may not get them if we speak ill of the giver. Individuals who are just starting out as (for instance) video game critics tend to fall into this trap a lot when they get free copies of games.
This practice has had a deleterious effect on the ratings of many games on Steam. Titles that could charitably be called “shovelware” have 70% or higher positive ratings. Or rather, they did until Steam’s latest changes.
Instead of going after reviews one-by-one, or even targeting individual users, Steam has opted to unilaterally disregard reviews of games that were activated via a key rather than purchased directly from the Steam store. They no longer factor into the overall qualitative ratings. You can see the effects in this spreadsheet, helpfully shared by Steam Spy.
https://twitter.com/Steam_Spy/status/775979532251791360
I can’t speak for anyone else, but I’ve heard of very, very few of these games. By and large, they look like cheaply-produced garbage. The game whose score plunged the most, something called Simple Ball: Extended Edition (which I won’t even bother to link), saw its score go from 88% positive to 14% positive. Even by Steam standards, 14% is appallingly bad. Very few games are so poorly rated.
Interestingly, the scoring change had the opposite effect on some games–they went from very negative scores to very positive. Elves Adventure–another one I’ve never heard of–went from 28% to 80%. That mainly seems to be a statistical aberration, though, since it originally had 74 reviews and, after the Great Filtering, went down to a measly 5. That’s the story for many of the games in the spreadsheet. A lot of them lost large shares of their reviews–70% to 90%, and even more in some cases.
I certainly don’t have a problem with Valve wanting to fight fake or dishonest reviews. But the approach taken here is too blunt, too inconsiderate of Steam’s users. For one thing, it’s not easy for users to tell which games they bought on the Steam store vs. obtained from a key. You can dig into your purchase history, which has very few options for sorting or searching, but that’s about it. So, a user may not easily be able to tell whether their review is going to “count” or not.
Another is that Valve has basically thrown out the baby with the bathwater, here. In going after a relative minority of bad actors, they are punishing wide swaths of users. Let’s consider all the ways in which someone might obtain a Steam key:
* Buying a physically-boxed game that comes with media and activates via Steam key.
* Obtaining keys from a game funded via a [Kickstarter ](https://www.kickstarter.com/)or other crowdfunding campaign.
* Purchasing a game key from a bundle site or other legitimate game site.
* Receiving a key in a monthly subscription box (like [Loot Crate](https://www.lootcrate.com/)).
* Buying a key on a third-party key sales/exchange site. (Many of these operate on shady legal ground.)
* Free key from a developer for the purpose of reviewing the game.
Because some people have abused developer-issued keys in order to prop up the review scores of bad games, all the other people who obtained their keys legitimately and gave honest reviews are being punished. The punishment, as such, is not severe–it’s obviously the height of First World Problems to be concerned about something like this. But a lot of people put time and effort into their reviews with the expectation that they would count for something–that their review would contribute to the game’s review score, and that it would appear in the list of popular reviews by which many users make their purchasing decisions.
More cynically, it’s possible Valve undertook these measures in part to curb the appeal of buying keys outside of Steam. Users who care about their reviews being taken into account will now be more likely to buy their games directly from Steam, which ensures Valve gets a cut of that money–a cut they don’t get when the game is bought elsewhere.
Nothing going on here is illegal, of course. Valve has every right to curate reviews on their platform however they like. But unilaterally disregarding the reviews of many users who’ve done nothing wrong is blunt and inconsiderate, and I hope they will rethink this policy in the future. Perhaps it would make more sense to purge low-quality reviews in general–it’s not as if all the reviews made on Steam store purchases are invariably fair or of good quality. But analyzing reviews individually is not something Valve is willing to do, since it is difficult if not impossible to automate. This solution is cheap and easy to implement, and so that’s the solution we get.
