Hypercare Status - Canada

Hypercare Status - Canada

Back
Jan 30, 2023 1:02 PM EST

Canadian Region: Monitoring performance issues

Impacted:
Login and Single Sign On - Canadian Region
Viewing Who is On-Call - Canadian Region
Self-serve Scheduling - Canadian Region
Code Teams - Canadian Region
Administration and Scheduling - Canadian Region

Updates

  • Jan 31, 2023 1:15 PM EST

    After an hour of monitoring, Hypercare team believes this slow down incident is now fully resolved. Hypercare engineering team will be modifying our notes functionality to ensure it does not affect performance and bring the feature back into the ecosystem.

    Resolved
  • Jan 31, 2023 12:21 PM EST (54 minutes earlier)

    We believe the issue is now fully resolved. We pinpointed the root cause to the "notes feature" on our platform that unfortunately was pointing towards another database. That database hit a performance limitation. Given that Android did not implement notes yet, Android was not affected and was fully functional.We will continue to monitor the situation for the next hour. If there's continues to be no issues, we will be closing the issue.

    Monitoring
  • Jan 30, 2023 6:14 PM EST (18 hours and 7 minutes earlier)

    Our engineering team has been working tirelessly on this issue. We believe we have highly probable suspicion which is iOS and Web Platforms differ in the amount of information asked of the server in critical actions such as pulling chats, pulling read receipts, and loading your app for the first time. This is most likely causing the inability to perform critical actions during peak periods while Android behaves normally as it could result in the server not being able to respond to the platform in a timely manner resulting in a timeout. The next step is for iOS and Web to minimize the amount of information asked of the server to reduce the time for server to return the result which in return should fix the problem. We are hoping to create a new version of web and android ready for release by tomorrow. We will keep everyone posted on the incident until it is fully resolved.

    Monitoring
  • Jan 30, 2023 3:45 PM EST (2 hours and 29 minutes earlier)

    We are currently suspecting that may not be a performance issue since Android is behaving normally. We are now currently looking at potentially the root cause as slowdowns are still happening intermittently. We will continue to update as we get more information.

    Monitoring
  • Jan 30, 2023 1:29 PM EST (2 hours and 16 minutes earlier)

    The system is currently loading as is. We are monitoring to ensure we notify if the server is slowing down again.

    Monitoring
  • Jan 30, 2023 1:02 PM EST (27 minutes earlier)

    The hypercare team is currently investigating a slowdown in the Canadian region. This currently affects loading chats from your account but may not involve sending messages for example. Our team is looking into temporary measures until the full root issue is resolved. This is currently flagged as the number one priority to resolve.

    Investigating