Bug Report: 12/3 Deployment Issue
Please be advised that on Tuesday December 3rd HTR had an issue with a production deployment that required us to roll back production to our last backup from the night before.
- Impact: Low
- Scope: Some data that was contributed to the site on Wednesday 12/4
- Companies Affected: All companies who received any reviews, leads, or content on 12/4 impacted the same
- Resolution: We will be restoring all data lost from the couple of hours between the 12/3 backup through the 12/4 restoration
- Targeted Completion: ✅ Resolved (Dec 12, 2024)
Doing so meant that production temporarily lost some data that was updated on the site on that day which is why you may notice the following things missing or in a different state than expected if they were posted, sent or updated on 12/4 after we restored the backup.
Most importantly, we have this data stored in a backup so rest assured it will not be lost and we are working diligently to manually restore it on production and appreciate your patience while we do so. Please keep in mind that this is a very manual process that we have to manage carefully auditing and cross checking data and then carefully restoring the data for each table that could have been impacted so this is a slower process than we would like.
We are working as quickly and diligently as possible and hope to have all data that was updated on 12/4 that got wiped from production when we did the backup restoration and hope to have everything fully recovered by Wednesday Dec 11.
In the meantime, all site functionality and data submitted or updated before or after 12/4 is not impacted and we will keep this bug report post updated with the latest updates as we restore each data type that could have been impacted from the 12/3 backup:
- Review Campaigns (✅ Resolved): If you sent a review campaign on 12/4 then it may look in your review manager like the campaign didn't go out because the activity log events aren't present. Our dev team is working to restore activity log events and campaign activity that once deployed will fully correct statuses and ensure that campaigns sent on 12/4 continue to run smoothly without impact (Target restoration completion Friday Dec 6)
- Reviews (✅ Resolved): Reviews that were submitted, verified, updated or moderated/published on 12/4 may have incorrect statuses or be missing all together. Once review campaign activity is restored we will shift focus to restoring reviews (Target restoration completion: EOD Monday December 9)
- Partner Recommendations (✅ Resolved): Similar to hotelier reviews, partner reccs submitted, updated or published on 12/4 may be showing incorrect status or be missing all together (Target restoration completion: Tuesday Dec 10)
- Case Studies (✅ Resolved): Case studies submitted on 12/4 may be missing from the site and will be restored (Target restoration completion: Weds Dec 11)
- Integrations (✅ Resolved): Any integrations listed or verified on 12/4 may be missing or gotten unverified. We will do our best to try to recover these but in the meantime if you believe an integration was verified and now it is not or its not listed feel free to re-add or re-verify the integration since they're easy to list and verify with the click of a button.
- Posts (✅ Resolved): Any new press releases or guests posts submitted on 12/4 during the data restore period will be recovered.
- Leads (✅ Resolved): Leads accepted on 12/4 may not be showing in the lead history ledger transaction table in the setting tab of your dashboard. Additionally if you accepted a lead on 12/4 it may appear as unpurchased (since we restored db from 12/3) which means you may accept that same lead again. Please note that we will review all transactions from 12/4 once data has been fully restored to ensure that there are no double charges. If you accepted any leads on 12/4 we will contact your team to let you know that leads have been restored from this day and to confirm that there were no double charges so feel free to accept the lead again in the meantime (in the meantime you can check the ledger to confirm there is no double charge as well).
- Miscellaneous: If there are any types of content that you believe your account had issues with please reach out to let us know.
We apologize for the inconvenience but want to reassure you that we are working diligently and will have any data that was lost from the couple hour window between the 12/3 backup and 12/4 restoration recovered in the next couple of days.