[solved] Downtime in the Debricked service 10Mar

  • 10 March 2023
  • 2 replies
  • 52 views

Userlevel 4

The Debricked service is currently down. Our team is aware of the issue and working on it. We hope to bring it up shortly. Definitely not how we want to end the week.

 

Please be aware that our status page has a technical issue, causing it to display inaccurate information. Therefore, it shows the service as "up and running" even though it is experiencing downtime.

 

Sorry for the interruption, everyone. We will keep you updated.


2 replies

We are back up again! We're extremely sorry for this interruption today, it looks like we were still slightly affected by our outage yesterday. Rest assured we are working alongside our third party vendors to fully understand the underlying causes and put in the guardrails in place to prevent it from happening again.

Userlevel 4

⛔️Outage Incident Report

 

We would like to share with you a summary of the issues occurred last week. Please read below:

 

Starting around 03:10 am UTC on March 8th, we got an internal notification that our service was down. At around 7:38 am UTC, we started to investigate the root cause of the issue which was then detected around 20 mins later. Our database provider was experiencing an incident that led to our database being down. From there, at 11.50 am UTC we were able to get our database restored and shortly after a manual deployment to force reconnection to our database, our service functionality was fully restored at 11.55 am UTC, after which all our customers were able to access the service.

On March 9th at around 6.10 am UTC we get notified that our database is unresponsive, related to the issues faced the previous day. Around 8.05 am UTC, we addressed the incident and try to restore our database, which was not successful. At 11.40 am UTC, we initiate contact with our database provider support team to fix the database rebuild issue. From there, we continue our communications with our service provider until around 7.39 pm UTC when a new backup restoration is in progress. Less than an hour later, at 8.20 pm UTC, our database is restored and our service is up and running.

During this outage, our official status page failed to detect that the service was down. This was caused by the status service only pinging the login page, which is not dependent on the database. We have then reached out to our users via email and keep the updates on The Debricked Portal Announcements area.

We understand that the impact of this interruption caused issues with your workflows. We are extremely sorry for this outage. We can guarantee you that we are working with our third-party vendors and internally to put guardrails in place to prevent similar issues to happen again. If you have any questions or concerns, do not hesitate to reach out to support@debricked.com

Reply