Shopify replaced the frontier model powering the part of Sidekick that answers merchant questions about their store data with a much smaller model it retrains every day on its own production failures. The result is a system Shopify estimates will cost around $1M a year instead of $27M, run 38% faster under load, need 14% fewer GPUs, and score better than the frontier model it replaced.
Quick backstory: Sidekick is Shopify’s AI assistant inside the admin, and one of the things it does is turn questions like “which products are almost out of stock” into queries against Shopify’s Admin GraphQL API, which can run up to 2,000 requests per minute. Until now, every one of those requests went to a general-purpose frontier model from firms like Anthropic or OpenAI. However, sending every query to a third-party frontier model is costly and unnecessary, as most typical queries don’t need that amount of GPU power to answer. Shopify also notes that those models don’t learn from their mistakes and that it has no control over fine-tuning them.
What changed: Shopify built what it calls a flywheel. Every day, a pipeline finds the conversations that scored worst, hands each failure to a panel of frontier reasoning models to critique, merges those critiques into a single fix, replays the conversation with that fix applied, and scores it again. If the repair works, that conversation becomes training data for the smaller model. If it doesn’t, it goes to human annotators who correct it by hand. Shopify also compressed the agent’s system prompt from roughly 6,000 tokens down to about 1,500 learned ones, which is where the speed and GPU savings came from.
To be honest, I don’t fully understand all the mechanics of chain-of-thought distillation or GRPO, but that’s not required to understand the impact of this change. The takeaway is that one of the largest agentic AI deployments in commerce just demonstrated that for a narrow, repetitive task, a small model trained on your own mistakes beats the most expensive model on the market at 4% of the cost. And not just “close enough for the price” — but actually “better.”
This reinforces two trends I’ve been seeing:
- More companies are pulling AI workloads in-house instead of renting frontier intelligence for every request, relying on the frontier models just for training instead of daily processing. In this case, Shopify downgraded the frontier models from answering every merchant question to just grading homework once a day, which is a much smaller bill. The labs are getting pushed from serving inference into training, and losing out on inference volume, which is the profitable part.
- AI costs are collapsing for companies willing to do the engineering. The ridiculous volume of datacenter buildouts happening in the U.S. assumes that businesses and users will rely on frontier-grade compute for every task, but the reality is that many AI tasks can be handled in-house on significantly less powerful machines. I’m even starting to see models small enough to run locally on laptops and phones without sending every request to the cloud. Shopify just proved how big of a savings that can yield, as well as how much more control it gives them over their proprietary tasks.
I’m not saying that OpenAI, Anthropic, and Google are disappearing anytime soon. I’m just pointing out that the business model and reliance on these companies for every AI query is shifting in front of our eyes. Stay tuned.
