On May 11, 2022, at 3:22PM ET, Gatsby’s Engineering team was made aware that Gatsby Functions were returning 404’s because Content Sync, which relies on Functions, returned a 404. After investigation, the root cause was identified as a graphql schema change that was not rolled out properly, which resulted in Functions 404ing.
The issue was resolved with a code fix is less than two hours.
Gatsby Functions
Gatsby CMS Preview
Identified: 2022-05-11 3:22 PM ET
Fix Verified: 2022-05-11 4:27 PM ET
Resolved: 2022-05-11 5:33 PM ET
Gatsby Engineering made a database and a graphql schema change, but the service that owned the data was not rolled out. Newly created functions were not able to be downloaded and run because we could not convert the build for that function to the site it was associated with. It was resolved by deploying code changes.
Functions returned 404’s for some customers. Existing Functions were not impacted.
The root cause was a graphql change that deployed improperly. The issue is now confirmed to be fully resolved.
Gatsby’s Engineering Team has successfully deployed a fix and improved monitoring to ensure no future instances occur.
We identified an issue with our code change deployment process and as a result improved the uptime monitoring of Functions. We installed processes to execute rollbacks more quickly and allow for simultaneous rollouts.