Shopify experienced a major backend outage on Cyber Monday, leaving merchants unable to login to their admins, edit themes, add products, launch discounts, fulfill orders, send e-mails, or any other of the many things you do from the Shopify admin. Not the greatest timing for an outage on the busiest shopping day of the year!
The issues began surfacing around 9am EST when merchants reported difficulties logging into their Shopify accounts and POS systems, preventing them from processing transactions, and continued until around 3:30pm.
What caused the outage?
Filippos Dematis of dev commerce explained it well in a LinkedIn post, but in a nutshell, the outage was caused by a bug in Shopify's identity authentication system that caused encryption keys to fall out of sync across servers, making valid login sessions fail. The issue had been masked for several months by constant code deployments, but surfaced when Shopify paused updates for BFCM and the broken sync logic was finally exposed.
Here's what pissed me off about this whole situation….
Where was Shopify's communication about the outage all day while it was happening? And during the days that followed?
- No posts on their primary X or LinkedIn accounts.
- Nothing from Tobi or Harley, other than posts pumping up Cyber Monday.
- No email communication to store admins.
Everyone was kept in the dark about the backend outages throughout the entire Cyber Monday and had to hear about it through social media. I feel that this issue deserved open, transparent, and ongoing public communication throughout the day from Shopify and its leadership.
The only X post Shopify made about the outage was via their less followed @ShopifySupport account:
“We're aware of an issue with Admins impacting selected stores, and are working to resolve it. For the most up-to-date information, please refer to our status page at http://shopifystatus.com. Thank you for your patience.”
Tobi barely mentioned it in a reply to a post celebrating BFCM numbers:
“We had some issues that meant that the admin interfaces were unreachable for merchants for a while, but online stores were up and selling throughout the BFCM weekend. You kept going, kept selling, and hit a record this weekend. Thank you and well done.”
Lastly, Shopify posted an incident report that you had to be logged into your Shopify account (how ironic) to read.
Where's the apology?
I've poured through social media posts from Shopify accounts and its leadership, read dozens of articles about the outage in search of a statement from Shopify, and basically scoured the web for the words “apologize” or “sorry” to no avail.
It feels like Shopify has been trying to sweep the outage under the rug so that it can focus on its big BFCM sales numbers, but frankly, the less Shopify has talked about it, the more it's made me want to talk about it! Their silence over the matter has been deafening.
I'm incredibly disappointed in how Shopify handled this outage, and I don't feel that I'm overreacting.
And to clarify, I'm not talking about their engineers. To you folks — great job identifying and resolving the issue as quickly as possible.
I'm specifically talking about Shopify's leadership. Where's the recognition, accountability, and well-deserved apology?
Their response to this outage, not the outage itself, has caused me to lose some respect for Shopify, as well as lose confidence in how they will communicate and handle similar scenarios in the future — of which there will inevitable be.
Merchants, partners, agencies, and everyone else in the Shopify ecosystem deserve a sincere and public apology from leadership regarding the outage, along with clear steps communicated about how Shopify will prevent it from happening in the future. Otherwise, you're no better than the rest of them.