→ For Spanish version, click here
Rating systems have been around since the early days of the Internet. Put simply, they are digital instruments to evaluate or rank an experience you’ve had — from a below-par Netflix movie, to those awesome new trousers you bought, to your most recent AirBnB trip (which was probably not that recent, if you’re reading this in 2021).
Most products we build at Umvel include some kind of rating system (we sometimes also call it “feedback system”). We’ve learned quite a bit about the pros and cons of the different systems out there. This article outlines our experience in the matter and talks about some of the pitfalls you might want to keep in mind when choosing a particular feedback system.
We have discovered that a three-step approach works best for understanding which rating system to choose:
- Define feedback goals
- Define rating methods
- Ask the right questions
1. Define the Right Goals
Rating systems are used for different reasons:
To personalize an experience. The user provides feedback on content to get a better, tailor-made experience. This is often the case with content that is “absorbed,” such as video or audio. The incentive to rate is pretty obvious here, as there is a direct benefit for the user: he or she will get a better experience when using the platform.
To aggregate data. This helps the provider to make a better selection of products and content when there is no user data available. To avoid a so-called “cold start”, users select preferred content during an onboarding process, upon which the system creates a somewhat personalized experience.
To inform, organize and prioritize. A rating system is used to inform the user about top content — most sold, watched, clicked, or read content on the platform. This helps the user to make a more informed decision about a future purchase or action. The use of a quantitative rating system, like the star system, can be complemented with a qualitative system, like reviews.
What does this mean for your own service or product? Well, that it is key to be clear about the kind of goal you set for receiving feedback. How will your data be used? For what ultimate purpose? Is it to inform, personalize, aggregate, or all of the above? Be clear, right off the bat, about this question and its answer.
Also, be clear about the incentive users are receiving for giving out data. The user needs to have no doubt about the value he or she is getting out of every action taken on the platform. For example, Netflix sees its “thumbs-up system” as a tool for users to improve their own experience, as Netflix Product VP Todd Yellin explains in this Verge article.
2. Define Rating Methods
Once you’ve determined what your goals are, you have to select the right feedback system. We’ll discuss a couple important ones here, but be mindful of the fact that there are over 10 systems to choose from.
But before we do so, there is one important thing to understand. This is what is known as context of use.
As described by Todd Yellin, there is a big difference between objective quality and “enjoyability” when it comes to media rating and feedback. What does he mean by that?
Zoolander might be the funniest movie we’ve watched — and we’ve probably rewatched it a dozen times. Does that make it an objectively good film? It is hard to tell. The film is not known for its superb cinematography, for example. But is it enjoyable? Definitely.
What about Schindler’s List? It is pretty evident that it was beautifully crafted and that the performances were impressive. Was it good? It was superb! Would we rewatch it again? Probably not , as it takes a big emotional toll on the viewer.
What, then, should be rated? The “pure quality” of the movie, or how likely it is that one would rewatch it? This example shows that you have to carefully think about what you will ask your user, and which metrics you will use.
In the light of the example above, let’s look at some rating systems you could adopt, and when they’re valuable or less valuable.
5–7 star rating . Here, you assign 1 to 5 (or 7) stars to evaluate a service or product. This is helpful when you want to include a granular rating system, that gives users the ability to really grade the service.
The downside of this system is that most ratings are either a 1 or 5 (the so-called “hate/love bias”). This is caused by people being so disappointed or thrilled about a service that they actually take the effort to go online and rate it.
However, for most people (who would vote between 2 and 4 stars), the service might be mediocre. Thus, they won’t make the effort to rate it. This results in a skewed overall rating result.
Note: read here why Netflix switched from a 5-star rating scale, to a ‘thumbs up’ one.
Thumbs up . The user evaluates the service or product by giving it a thumbs up or thumbs down. This system can also be reflected as a “smiley system,” which you can see popping up at airports everywhere.
Pairwise voting. Instead of evaluating a specific service independently, the user is presented with 2 options, out of which he or she selects a preference. This type of voting is especially used in referenda and other elections. However, when applied to content it provides the system with a clear idea of user preference.
Picking favorites — . The user picks a set of favorites from a list of options, so that the system can aggregate and synthesize data, in order to personalize user experience. This is especially helpful to avoid a cold start. By implementing this process at the onboarding stage, the user feels familiar with the app or platform from the very start.
Reviews & testimonials. Perhaps one of the best-known forms of feedback, this method gives users the ability to write a small text about an experience, outlining what they liked or disliked. This qualitative way of gathering feedback is rich in content. It is also time-consuming for both writer and reader.
The table below summarizes the key elements of each rating system, their pros and cons:
3. Ask the Right Questions
Once you have the right goals and are familiar with the pros and cons of a number of different rating systems, you must make sure that you are asking the right questions.
For example, there is a big difference between:
- How would you rate [movie x]?
- How much did you enjoy [movie x]?
The difference is subtle but important:
The first question implies we want to know how the user rates the movie based on its overall quality, or the overall impression it left behind.
The second question refers to the “enjoyment” the movie produced. To what extent was it entertaining, to you?
Another example. When we want to rate a food delivery app, we could ask:
- How likely are you to recommend [restaurant x] to your friends?
- How did you value the service of [restaurant x]?
Again, there is an important difference between the two questions:
The first question implicitly asks the user to rate the service of the restaurant in the light of what the general public would think of their services . It does not focus too much on the individual user itself.
The second question directly asks the user what he or she thought about the restaurant’s service. The user’s preferences might be particular, extreme, or very different from those of the general public, and so the answer will most likely be radically different.
As you can see, it’s of vital importance to think long and hard about the way you phrase your questions, so that you get an answer that fits with the type of information you wish to capture.
🚀 🚀 Liked what you read? Umvel publishes monthly blog articles about technology, development, design and innovation. Check out umvel.com for our full portfolio of services.
