Elevated 503 responses with Functions service
Incident Report for Gatsby Cloud
Postmortem

Elevated Function 503 Responses

Summary

On July 19th, 2022 at 5:22 AM ET, Gatsby Support was made aware of 503 errors that impacted customer sites. This resulted in an elevated number of 503’s from certain customer functions.

The issues was resolved with a Gateway modification in 2 hours.

Services Impacted

Gateway

Timeline of Events

Identified: 2022-07-19 5:22 AM ET

Resolved: 2022-07-19 7:20 AM ET

Details

Gatsby Engineering was made aware of an increasing number of 503 errors that were manifesting in different forms. When Engineering saw that multiple sites had the issue, they were able to discern there was a Gateway issue. They restarted the Gateway to temporarily fix the issue. Since then, they’ve increased the number of instances of the Gateway service.

Resolved

The root cause of the incident was a Gateway issue, this has been fully resolved by increasing the number of Gateway instances.

Monitoring

Gatsby’s Engineering Team increased the number of Gateway instances and will configure automatic scaling to make sure this doesn’t happen again.

Identified

Gatsby Engineering identified several ways to prevent this issue from occurring again by addressing how the Gateway is currently implemented and how to respond to a sudden increase in traffic.

Posted Aug 17, 2022 - 09:42 PDT

Resolved
This incident has been resolved.
Posted Jul 19, 2022 - 14:11 PDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Jul 19, 2022 - 13:52 PDT
Investigating
We have identified that our Functions service is returning an elevated amount of 503 responses. We are currently investigating.
Posted Jul 19, 2022 - 13:37 PDT
This incident affected: Functions.