Removal of profile completion update

Remember in the HotelTechAwards when companies across 6 categories made a mad dash for the finish line and ended up creating ties? The reason for this is that like Google's algorithm, the HT Score evolves over time.  The ways it evolves over time are either (a) old variables become bad or less relevant signals of quality or (b) new more relevant variables are introduced to make the rankings better (aka. make it easier for users to find what they are looking for).

In this case, the improvement to the HT Score is coming in the form of removing an old variable that is no longer relevant and is inflating scores (equally across the board for all vendors) and making the HT Score less helpful to users.  This variable is profile completeness, and profile completeness is what caused the multi-way ties across six categories.

In the early days of Hotel Tech Report when the site served as simple directory of categories and companies one of the most important variables to help hoteliers learn about products was providing helpful content on your profile.  As such, if a vendor completed 80% or more of their profile, they received a profile completeness incentive which increased their HT Score by 10.  The problem is that now that almost all vendors have over 80% profile completion, this 10 point boost is actually making the rankings less effective.  

Let's look at an example:

Company 1:
Raw HT Score = 91 pts
Profile completeness = 80%
Actual HT score with completeness boost = 91+10= 101
....but since max HT Score is 100,their score displayed is 100

Company 2:
Raw HT Score = 99 pts
Profile completeness = 80%
Actual HT score with completeness boost = 99+10= 109
...but since max HT Score is 100,their score displayed is 100

So while company 2 technically has a higher score (99) than company 1's (91), both show as 100 after factoring in the profile completeness incentive.

FAQs

Will my company's score change as a result  of this change?

All companies will be impacted equally with the removal of the 10 point profile completeness incentive as a variable.  Only company's with a perfect 100 point HT Score will maintain this.

What can I do to improve my score?

All other variables of the HT Score remain the same.  To read more about the other variables you can do sohere.  Hotel Tech Report users are looking for vendors who have verified customer feedback so its pretty simple, the HT Score is all about customer feedback.  More and better reviews is the key to attracting more hoteliers and improving your HT Score.

Claim profile boost update

Summary:

  • What is changing? While the max HotelTechScore you can see visible on Hotel Tech Report is 100, vendors currently have the ability to earn 110 points because of an extra boost built into the algorithm for claimed profiles
  • The Problem: Vendors currently have the ability to earn 110 points because of an extra boost built into the algorithm for claimed profiles.  Initially this helped new vendors gain a superficial boost while they began their first review campaigns, however, now that 300+ top vendors have claimed their profiles and ran their first review campaigns--it serves no purpose (and actually makes the HT Score less accurate!).  With 5 category ties at the end of the HotelTechAwards, several companies  expressed a need to improve the HotelTechScore.
  • Ranking Change: While the 'claim profile boost' being removed will likely lead to a decreased HT Score, it will do so across the board.  As such, while your score may change--your ranking will not.

Why is this changing?

Like Google's search algorithm that has many signals (or factors) that get improved and refined over time to help improve search and identify high quality content providers, the HotelTechScore is the same.  

With companies surging towards the finish during the 2018 HotelTechAwards, we quickly found many vendors reaching the 100pt ceiling tying for 1st place (imagine if every search result in Google were equally as enticing...Google would be less useful).  As a means for helping hoteliers to identify and understand technology, we continue to add signals and variables to help hoteliers identify and learn about the best tech and what differentiates each product.

New criteria coming soon:

  • Monday February 25th:Preliminary removal of the claim profile boost
  • May 2018:Implementing tie breaker criteria into the HT Score Learn more

Will my companies' HT Score drop after the update?

Unless your company had an actual score of 100, your score will drop to reflect your actual raw score without the claim profile boost.  While scores will drop, rankings will immediately be more accurate.  On Monday February 26th Hotel Tech Report will remove the claim boost for all companies.  If your company's score is lower than expected please review our best practices with tips, tools and templates for running turnkey review campaigns.  If you have any questions as always, please free to reach out anytime via our 24/7 live chat.  


More details and an example of how the claim boost worked

Currently, while the max score you can see visible on Hotel Tech Report is 100, vendors currently have the ability to earn 110 points because of an extra boost built into the algorithm for claimed profiles.

Why can vendors earn higher than 100 currently?
In the early days of Hotel Tech Report, we included a 10 pt boost when a vendor 'claimed' their profile.  We did this in order to give vendors who only had a handful of customer reviews a chance to get reviews and improve their scores.  Since nobody had close to the 100 pt maximum, this helped ensure that vendors across the board (all vendors have this boost) had scores that weren't shockingly low to hoteliers.

What has changed?
Now that our global community of 300+ top hotel tech vendors are committed to our shared mission of transparency and customer feedback to improve our industry, we have had a number of vendors who actually have over 100 pts HotelTechScore.  This is a great thing! 

Congrats to the following vendors: IDeaS, TrustYou, Rainmaker, Mews, Oaky, Guestline, Quore, ALICE, HelloShift, Amadeus, Guestline, HotelIQ and Whistle...your customers  absolutely love your products and services and want to shout it off a mountain top!

So what does this mean?
​Since there is a 10 pts boost for claiming your profile that makes your total points out of 110 instead of 100, but the max HT Score is 100--it means that vendors who have anywhere between 90-100 pts and have claimed their profiles will currently display an HT Score of 100.  

Lets look at an example:

Company A
Raw HT Score: 90 pts
Claimed Profile Boost: 10 pts
Total Score: 100 pts
Displayed Score: 100 pts

Company B
Raw HT Score: 100 pts
Claimed Profile Boost: 10 pts
Total Score: 110 pts
Displayed Score: 100 pts

As you can see, Company B technically has a higher HT Score but because of the claim profile boost both companies appear to have the same HT Score.  It is for this reason we will be removing this boost for claimed profiles.   

Now that almost all companies have claimed their profiles, claiming your profile no longer will add points to your HT Score.

In the example, that means that Company A who has a raw score of 90 will see their score drop when the update is released, and Company B who has a score of 100 will maintain their same score.

Minimum Review Threshold Update

The minimum review threshold is a variable in the HT Score algorithm that serves to level the playing field both for startups (via a ceiling on the impact of review quantity) and large companies (via a floor on the amount of reviews required to be considered a credible/trusted provider within the category):

  • Review quantity ceiling: Limit to the impact of review quantity to level the playing field for smaller companies who have great products that hoteliers love to shine (so bigger isn't always better)
  • Review quantity floor: Ensures that a company can't just get a handful of perfect scores from their friends/early clients and all of a sudden boost to the top of the rankings ahead of established and trusted products in the category with lots of customers who vouch for the product

Update August 2021: Minimum review threshold now includes a decay factor on old reviews per the HT Score Poseidon Algorithm Update (Q3 '21)

TL;DR

  1. The 'Minimum Review Threshold' is just a fancy way of saying 'the average number of reviews' for products in a given category 
  2. The algorithm only looks at products with enough reviews to be considered significant and trusted by buyers (30 reviews+)
  3. It protects against scenarios with limited sample sizes (ex. a company who gets 1x 5-star review won't be able to outrank a company with 200 x 4.5-star reviews)
  4. It creates a level playing field and protects against larger companies always being top rated (bigger isn't always better)
  5. Quantity is a factor in the HotelTechScore but only to the extent that a given product needs to meet the avg. # of reviews in the respective category to be able to be compared side-by-side with other established companies

Related articles: How is the HotelTechScore calculated?




__

Let's dive into the details, rationale and an example of how the Minimum Review Threshold works, shall we...

Reminder of what the minimum review threshold is in the scoring algorithm
If you recall from thearticle on how the HotelTechScore is calculatedone critical element of the score is the minimum review threshold.  The minimum review criteria is quite simply a check and balance that is meant to ensure that there is a sufficient amount of customer feedback to help inform the score.  So for example, if a product has all 5* ratings but only one customer review...they shouldn't have a high score relative to a company that has 4* and 5* ratings but 20 customer reviews.  

This is where the minimum review threshold came in.  It was a way to ensure that products without enough information/insights weren't jumping ahead of companies with lots of high quality reviews to help buyers.  The way we did that initially was to take the average number of reviews for all of the companies that had claimed profiles on Hotel Tech Report and essentially set that as the bar for how many reviews a company needed at a minimum and if a product didn't meet that minimum threshold, their score would get docked to ensure that they didn't move past companies with lots of high quality reviews (for more detail please revisit thearticle).

 - The Problem -
The problem that was brought to our attention by a few top rated HTR vendors (thanks Mackenzie and Alex!) is that in some categories this still leaves certain companies who don't have many reviews looking as good as companies with lots of reviews (and sometimes even passing them).

When we looked into these specific cases we realized that the issue lied in the fact that the minimum review threshold was being calculated site wide and that in several circumstances the problem that we were trying to fix with the minimum review threshold criteria was still happening in certain scenarios.

 - Why it was happening -
The minimum review threshold used to be site wide so in some more developed categories with companies with lots of reviews, the threshold was relatively low which meant getting a few reviews could boost your score a lot very quickly.

 - The Solution -
Change the minimum review threshold to be by category.  After the update, the part of the algorithm criteria that is the minimum review threshold will switch from all HTR profiles, to being calculated within each category.

 - How will this impact you? - 

(a) If your company has a similar amount of reviews relative to others in your category, you will likely see your score go up
(b) For companies who seem to have a high score currently but only a few reviews, your score will probably go down

 - What if I'm in the latter (aka. camp 'b')? -
If you thought you could sneak away with a high score and very few reviews...sorry to rain on your parade...but this is a good thing, we promise! Plus, if you want to prevent your score from decreasing, all you have to do is reach out to customers and get some more reviews! (note: the update goes live Friday 10/27)


Let's look at an example to see how this works...

Angela works at a hypothetical company called Guestly who makes guest happiness software (category also hypothetical...).  Now for Angela to hit the minimum review threshold, she will need to be above the average amount of reviews for profiles in her category with more than 1 review.  So here are the review counts for them:

  1. Happy Guest- 27 reviews
  2. GuestSatis - 13 reviews
  3. Better Toguester - 3 reviews
  4. Guestly - 5 reviews

To meet the criteria for the minimum review threshold we'll take the average of these review counts (27+13+3+5)/4 companies=12.

So in this case, only GuestSatis and Happy Guest meet the minimum review threshold so Guestly will need to get 7 more reviews to check this box and bump their score along this criteria (again, the minimum review threshold is one of several criteria...but it is an important one).  Remember, Hotel Tech Report doesn't rank based on who the best company is (that's a subjective measure) but ranks the companies that provide the most information and transparency and who have the most customers that love their products and services enough to share their experience with others.


Still have questions?

If you have any feedback, ideas or thoughts on how we can continue to improve the HotelTechScore ranking (or anything else!) like Mackenzie and Alex, please reach out any time via the live chat on Hotel Tech Report or by posting on our communityfeature request board.


May 2020 Update: Average for the minimum review threshold calculation now only includes companies with >30 reviews Learn more