Disruption in Integration and SMS service

In connection with an update to the Questback Essentials platform Wednesday 05/07 at 23:00 we experienced three separate issues over the following days. All times given below in CEST.

  1. Instability in the API services caused the service to be unavailable for a total of 4 hours downtime over the week

  2. Email distributions added through the API were scheduled for delivery but not delivered

  3. SMS sendouts were scheduled but not delivered while showing a state of ‘delivered’ in the platform interface

All three issues have now been resolved and we are closely monitoring the system for any discrepancies.

We are truly sorry for any inconvenience caused by these issues and we are investigating how the root causes of these issues occured and how we can work to ensure we do not see similar disruptions again in the future.

A detailed explanation of each issue can be found below.

1. Instability in API service

With a first occurence at 08:00 Tuesday 06/07 we have between 06-08/07 experienced sporadic downtime of the Integration/API service on high load. This has only affected calls to our API and no other part of the service.

In total over the three days, we have experienced 4 hours of downtime where the API has been unresponsive to requests and at time we have seen longer response times than usual.

Several smaller fixes were deployed over the week to remedy these issues while we continued to see interuptions in the service.

At 19:30 on 08/07 a final fix was deployed that solved the root cause of the issue and we are now closely monitoring load and response time of the service.

2. API added emails not being sent

At 12:52 on Friday 08/07 it was discovered that email distributions added through the API had not been sent. Investigation of the issue showed that a change had been implemented with the release 05/07 queued up emails, but did not release them for distribution.

At 15:36 the same day all queued emails were released to their recipients and new added emails were continously released manually until a permanent fix to this issue was deployed at 19:30.

There has been no loss of data or emails not being sent through with this issue, only a delay in time that emails were added to a quest and the actual email sendout.

3. SMS not being delivered

Friday 08/07 at 15:03 it was discovered that SMS sendouts were not correctly triggered. With SMS inside the platform being marked as ‘delivered’, this issue took time to identify. The issue was identified and a fix deployed at 19:30 on 08/07. All queued SMS were released at 19:15 to their recepients.

There has been no loss of data or SMS sendouts due to this issue, only a delay in distribution.

In summary

We sincerely apologise for any inconvenience this disruption might have caused you as a customer. Being reliable and always available are core commitments in our approach to technology. We are currently assessing our setup of warnings and processes to ensure this kind of issue does not happen again in the future.

If you have any questions about this incident, please don’t hesitate to reach out to our support at support@questback.com