SSL Certificate Errors
Incident Report for Gatsby Cloud
Postmortem

Summary

On May 23 at 1:13 PM ET, Gatsby Support reported that customers with older SSL certificates were having issues with SSL certificate validation. Gatsby was serving expired SSL certificates. Customers with a Gatsby Hosting SSL certificate that was generated more than 3 months ago was affected. Gatsby Engineering mitigated the issues by deleting the old SSL certificates.

The issue was resolved with our third-party hosting provider in 6 hours.

Services Impacted

Gatsby Cloud Certificates

Gatsby Cloud Hosting

Timeline of Events

Identified: 1:13 PM ET

Resolved: 6:43 PM ET

Details

Our third-party hosting provider was improperly serving older SSL certs rather than the newer certificates. Before our provider could resolve the issue, Gatsby Engineering mitigated the issue by deleting the old SSL certificates. Our provider updated their system to ensure new SSL certificates were being properly served going forward.

Customer with a site with SSL certificates generated more than 3 months ago were returning SSL certification errors.

Resolved

The SSL certification issue is now fully resolved due to the hosting provider resolving it on their end.

Monitoring

Gatsby’s Engineering Team successfully mitigated the certificate issue during the incident and installed a process to regenerate certs more easily to safeguard against similar, future incidents.

Identified

Root cause was identified as an operator error at our hosting provider. Our hosting provider has confirmed that the issue is resolved. Gatsby Engineering has installed a process as a redundancy to eliminate the potential risk going forward.

Posted May 27, 2022 - 10:49 PDT

Resolved
This incident has been resolved.
Posted May 23, 2022 - 16:16 PDT
Update
We have validated all domain certs are correct.
Posted May 23, 2022 - 16:16 PDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted May 23, 2022 - 15:11 PDT
Update
Most customer sites have been fixed but we are still verifying all of the sites that were impacted are working correctly. We are working with the Fastly team to roll back changes that were impacting Gatsby Cloud hosting sites.
Posted May 23, 2022 - 14:32 PDT
Update
We are continuing to work on a fix for this issue. Some customer sites are still receiving SSL Certification warnings.
Posted May 23, 2022 - 12:51 PDT
Identified
Some sites on Gatsby Cloud Hosting are receiving invalid SSL certificate errors. We have rolled out a fix for most sites that were impacted, however, there are still a few sites we are investigating.
Posted May 23, 2022 - 11:34 PDT
This incident affected: Hosting.