Shipping an Agent Is 10× Harder Than the Demo

Building a convincing agent demo with Codex is no longer that difficult. The hard part is putting it into a real business and making the system accountable for every result.

I’ve recently been building a social media agent for small businesses that lack the people and budget to run their accounts professionally.

Large companies can hire a team to research the market, set a strategy, write content, and produce the visuals. At a small company, social media often falls to the owner or an employee who posts whenever there is time. When work gets busy, the account goes quiet.

Agents create another possibility. Work that once required a team can begin with an AI team instead: one that understands the company and its customers, plans the content, then handles topics, copy, visuals, and publishing. The user does not need to execute every step. They only make the decisions that matter: Is the direction right? Is this what I want to say? Can this go out under the company’s name?

Building the demo with Codex was much easier than I expected. Give it some company information, a target audience, and a few references, and it can quickly produce a decent post, pair it with an image, and add it to the publishing queue. Once the flow is running, it already looks a lot like a real product.

Then I tried to put it in users’ hands, and the workload jumped by an order of magnitude.

If I had to put a number on the difference, I would say this: shipping an agent is at least ten times harder than shipping traditional software of the same apparent size.

One good post is only one sample

Demo prompts are usually prepared in advance. The source material is fairly complete and the goal is clear. If the model goes off track, I can rephrase the prompt and try again. Generate a few versions, and one of them will usually be good enough.

Real users do not arrive that neatly. Someone might only say, “I want to talk about agents, but don’t make it sound like an industry report.”

The agent has to work out what to write, which sources to use, what the user genuinely believes, and what “don’t make it sound like an industry report” actually means. Change the topic or shift the context slightly, and the result may change with it.

More importantly, the user does not need one lucky hit. They need a steady run of good content over the next several weeks. The posts cannot repeat one another. They need to sound like the same person, while keeping up with changes in the business.

One good post proves that the system succeeded once. Production has to deal with every post that comes after it.

Agents fail without looking broken

Traditional software is hard to ship too. You still need permissions, error handling, performance, and monitoring. But most of its behavior was specified by people in advance: click this button, enter that state; if the API succeeds, do this next. The rules are relatively explicit.

An agent adds another layer of judgment at runtime.

The final post is shaped by the model, the user’s information, past content, search results, and tool responses. Engineers did not write every branch, and they cannot inspect every possible output before it happens.

That creates an especially difficult kind of failure. The model returns normally. The image is generated. The publishing API reports no error. Yet the post expresses a view the user does not hold, cites information that is already out of date, or repeats a point published the week before.

Technically, everything worked. From the business’s perspective, the post should never have gone out.

Many failures in traditional software announce themselves as errors. An agent’s failures are often hidden in meaning. The system has to check not only whether the action succeeded, but whether it was the right action. No status code can answer the second question.

There is a person hiding inside the demo

Many agent demos look autonomous because the human work stays out of frame.

The person giving the demo chooses a suitable topic, fills in missing context, regenerates a weak first draft, and selects the best result from several options. They still check the facts, the voice, and the brand risk at the end.

The agent generates. The person continues to interpret, select, and approve.

Once the product goes live, those jobs cannot depend on someone improvising behind the scenes. The system needs to know when to ask a follow-up question, which claims must be checked, and which content requires approval. It also needs to remember what the user changed, while distinguishing a lasting preference from a one-off edit.

When I say the user should “only make judgments,” I do not mean removing the person from the process. Their role changes. Instead of carrying out every step, they appear at the few points that genuinely require taste, business judgment, and accountability.

Time and publishing amplify rare mistakes

Problems that stay hidden in a single post emerge quickly when the agent runs an account over time.

Topics begin to repeat. The voice starts to sound templated. Old posts conflict with new positions. The company’s priorities change, source material expires, and both models and platform rules keep moving.

Publishing also changes the outside world. A bad draft can be discarded. Once a post is public, its views, screenshots, and impressions cannot be fully taken back.

So an agent cannot be judged only by the average quality of its writing. We also need to ask what it might say on the user’s behalf when it occasionally fails, and whether the system can stop that content before it is published.

A demo that runs a handful of times makes the capability easy to see. Run the same system long enough in production, and the low-probability failures eventually arrive.

Most of the work of shipping happens outside the model

A demo agent only needs to generate content and call tools to look impressive. A production agent needs several less glamorous capabilities:

• Clear goals: Do not measure only how much content the agent generates. Track how much users are willing to publish, how much they edit, and how often factual errors or repeated ideas appear.
• Checkable quality: Important claims need sources. New content should remain consistent with the user’s past positions, and uncertainty should be made visible.
• Bounded permissions: The agent can have more freedom when exploring topics and drafting, while sensitive language and final publishing require stricter approval.
• Recoverable failures: Platform timeouts, failed schedules, duplicate posts, and incorrect content all need a defined way to be handled.
• A traceable process: We should be able to see which sources a post used, how it changed, who approved it, and which version was ultimately published.

None of these capabilities live inside the model itself. Together, they determine whether we are willing to hand an account over to the agent.

This is when I finally understood that a demo is not a smaller production system. It is a demonstration of capability: proof that something can happen. The product work that follows is about adding boundaries, checks, and recovery around that capability.

Next: how to build one for production

This article explains why the gap is so large. In the next one, I want to look at how a production-grade agent should actually be built.

I often return to Anthropic’s “Building effective agents”. It does not treat more complexity as a sign of a more advanced agent. Its advice is refreshingly practical: start with the simplest approach that works, add complexity only when it measurably improves the result, use fixed workflows for well-defined tasks, and reserve agents for places that genuinely require flexible judgment.

That distinction is especially useful in social media operations. Which steps should be fixed? Which decisions can be left to the model? Where should the user step in? How do we know that a run of posts is getting better, rather than merely sounding more alike?

Those are the questions I want to take apart next.

A demo shows what AI can do. A real product makes people willing to trust it with their account.