OpenAI Performance Factory: Codex Fixes ChatGPT, a Human Still Presses Merge
When ChatGPT gets slower, there is now a good chance an agent noticed before any engineer did. OpenAI published a post on Thursday at 08:00 UTC describing the Performance Factory, an internal system where Codex agents run performance regressions from alert to recovery. In just over three months its first workflow tracked more than 1,700 investigations and root caused over 600 of them, about 17 investigations a day. Read the ratio honestly: roughly a third of investigations ended with a confirmed cause. Post: https://openai.com/index/how-openai-performance-factory-keeps-chatgpt-fast
The workflow is boringly explicit, and that is the point. An alert opens a task. An agent finds when the metric really started to drift (often hours before the alert), which requests were hit, and which deploy or feature flag lines up. A separate Codex session then tries to disprove that diagnosis. Another agent writes the fix and opens a PR. An engineer reviews and deploys. After deployment a different agent checks whether performance returned to baseline, and if recovery is only partial it closes the original task and opens a new one for the remainder. Engineers define all of this in Markdown prompts plus a config listing states, allowed transitions and which steps need a human.
Two design choices answer problems this site has been tracking all month. Task state and findings live on the task, outside any agent's context, so work survives handoffs between agents, engineers, code review and deploys. Agents wake up through subscriptions to events like a review comment or a failed CI check instead of sitting in a loop. And every agent runs under a service identity with scoped permissions, so any action can be traced back to the performance agents. That is the authority layer built as plumbing rather than promised in a system prompt.
The numbers OpenAI gives against itself are the most useful part. In a manual review of 100 root cause suggestions, 70% were graded accurate and 30% needed refinement or did not establish causality. Early agents stopped at the first plausible story, which is why the adversarial reviewer exists; in one case it reopened an image-download diagnosis because the supposed culprit had already recovered while failures continued. Fixes are harder than diagnoses: proposals came out more complicated than needed or introduced new bugs, so fixes get their own adversarial review. Evals backtest against old incidents with memories, writeups and prior runs blocked, and the transcript is inspected for leaks.
My read: this is the clearest production template any lab has published for an agent workflow. Not a smarter model, but states, a second agent whose job is to say you are wrong, a human gate exactly where judgment costs the most, and a daily agent that reads the other agents' logs for broken tools. OpenAI says the bottleneck now is waiting for an engineer to merge, and it is exploring letting agents ship low-risk fixes alone. That is the line to watch: the day the merge button moves is the day this stops being a factory with a foreman.
← Back to all articles
The workflow is boringly explicit, and that is the point. An alert opens a task. An agent finds when the metric really started to drift (often hours before the alert), which requests were hit, and which deploy or feature flag lines up. A separate Codex session then tries to disprove that diagnosis. Another agent writes the fix and opens a PR. An engineer reviews and deploys. After deployment a different agent checks whether performance returned to baseline, and if recovery is only partial it closes the original task and opens a new one for the remainder. Engineers define all of this in Markdown prompts plus a config listing states, allowed transitions and which steps need a human.
Two design choices answer problems this site has been tracking all month. Task state and findings live on the task, outside any agent's context, so work survives handoffs between agents, engineers, code review and deploys. Agents wake up through subscriptions to events like a review comment or a failed CI check instead of sitting in a loop. And every agent runs under a service identity with scoped permissions, so any action can be traced back to the performance agents. That is the authority layer built as plumbing rather than promised in a system prompt.
The numbers OpenAI gives against itself are the most useful part. In a manual review of 100 root cause suggestions, 70% were graded accurate and 30% needed refinement or did not establish causality. Early agents stopped at the first plausible story, which is why the adversarial reviewer exists; in one case it reopened an image-download diagnosis because the supposed culprit had already recovered while failures continued. Fixes are harder than diagnoses: proposals came out more complicated than needed or introduced new bugs, so fixes get their own adversarial review. Evals backtest against old incidents with memories, writeups and prior runs blocked, and the transcript is inspected for leaks.
My read: this is the clearest production template any lab has published for an agent workflow. Not a smarter model, but states, a second agent whose job is to say you are wrong, a human gate exactly where judgment costs the most, and a daily agent that reads the other agents' logs for broken tools. OpenAI says the bottleneck now is waiting for an engineer to merge, and it is exploring letting agents ship low-risk fixes alone. That is the line to watch: the day the merge button moves is the day this stops being a factory with a foreman.
Comments