You naturally assume that a newer, stronger model should be better than the old one in every aspect.
After all, the SOTA model hit new highs on Frontier-Bench, approached Fable 5 on CursorBench, and you’re paying higher API costs — you should rightfully expect better tool calling capabilities.
Then you discover that Opus 4.8, when calling your editing tool, invents fields that don’t exist in your schema.
Not Haiku. Not Sonnet 3.5. It’s the newest, strongest Opus 4.8 and Sonnet 5. The older models are fine.
This is the counterintuitive bug Armin Ronacher ran into while building the Pi editor — an engineering trap serious enough to make every third-party coding harness re-examine their tool design.
0. The Meta-Question: The Model’s Objective Function Is Not Your Objective Function
Armin’s discovery process was remarkably straightforward. Pi has an edit tool with a clearly defined schema. Older models — Claude 3.5 Sonnet, earlier Opus — called this tool by the book: field names, types, and structure all matched the schema.
Then Opus 4.8 and Sonnet 5 arrived, and things went wrong.
The model was inventing fields in the
edits[]array that don’t exist in the schema. The edits themselves were correct, but the parameters didn’t match the schema, so Pi had to reject the request and make the model retry.
Armin’s exact words:
“What surprised me is that this is getting worse with newer Anthropic models.”
To translate: what shocked me isn’t that the model generates malformed tool calls — it’s that this is getting worse on newer models, not better.
This is a highly abnormal regression direction.
1. The Regression Chain: How RL Training Creates “Tool Preferences”
Armin put forward a hypothesis. It hasn’t been officially confirmed by Anthropic, but it holds up perfectly from an engineering logic standpoint.
The new models underwent reinforcement learning training optimized for Claude Code’s built-in tools.
Claude Code’s edit tool uses a search_and_replace pattern: find the old text, replace with new text. This is a specific tool protocol internal to Claude Code.
Pi, on the other hand, uses its own edit tool — similar in function, but with a different schema structure. During RL training, the new models were repeatedly rewarded for “correctly using Claude Code’s edit tool” behavior, causing them to gravitate toward Claude Code’s schema even when faced with other editing tools.
Lesson 1: The more RL training focuses on a specific tool set, the worse the model’s generalization ability becomes on third-party tools. This isn’t a bug — it’s a feature. It’s just that this feature benefits Anthropic, not you.
Simon Willison, sharing Armin’s blog post, added a sharp observation:
“Does this mean third-party coding harnesses like Pi should implement multiple edit tools just so they can use the one with the best performance for the underlying model the user has selected?”
Translation: should third-party tools implement multiple sets of editing tools just to adapt to different models’ preferences for different tool schemas?
It sounds absurd, but it’s becoming engineering reality.
2. Same Story, Different Player
This isn’t unique to Anthropic. OpenAI’s Codex uses an apply_patch mechanism for edits, and OpenAI similarly optimizes models for this tool during training.
So the same regression chain is unfolding: GPT-5 performs flawlessly on Codex’s apply_patch, but give it a custom edit tool and it may produce schema-noncompliant outputs in edge cases.
Every model vendor is optimizing tool calling for their own flagship product, and the direction of optimization is inherently at odds with third-party tool compatibility.
Lesson 2: Tool Calling is not a standard. It’s becoming a vendor lock-in mechanism — use my tool protocol and you get the best experience; design your own protocol and you suffer regression.
What stings even more is that this problem won’t automatically resolve as models “get smarter.” Quite the opposite — the stronger the model, the more focused the RL training, the deeper the preference for specific tool protocols.
3. If You’re Also Building Your Own Coding Harness
Armin and Simon’s discussion surfaced several coping strategies.
Strategy 1: Detect model version, dynamically switch tool schemas.
If your harness supports multiple models, prepare different tool definitions for different model families. Make your tool definitions for the Claude family as close as possible to Claude Code’s built-in tool schema; make definitions for the OpenAI family as close as possible to Codex’s tool schema.
This is essentially shifting the compatibility cost from the model side to the harness side. It works in the short term, but the long-term maintenance burden is significant.
Strategy 2: Add fuzzy matching to the tool call validation layer.
When a model returns a tool call with unknown fields, don’t reject it outright. Instead, try to determine: “is the model applying another tool’s schema?” If so, ignore the extra fields and only take the matching parts.
The tradeoff of this strategy is clear: you’re trading tolerance for success rate, but tolerance itself is a loss of precision.
Strategy 3: Accept the retry cost.
Let the model fail on the first call, get rejected, and then explicitly inform it of the correct schema format in the error message. This strategy seems clunky, but it’s actually the most stable solution right now.
Claude and GPT, upon receiving a clear schema error, can usually correct themselves within 1-2 retries. The cost is one extra API call, but at least you’re not introducing additional complexity.
Lesson 3: In agent engineering, “retry” is not a workaround — it’s an underrated architectural pattern. Models are not deterministic systems. Giving them a chance to self-correct is often more economical than preemptive prevention.
4. The Deeper Implications
Back to the meta-question: 
The model’s objective function is not your objective function.
Anthropic optimizes Claude to perform well inside Claude Code. OpenAI optimizes GPT to perform well inside Codex.
Your coding harness is third-party. From the vendor’s perspective, your tool schema alignment problem is not their priority — unless you have enough user volume to make it a business concern for them.
Simon’s question — “should third-party harnesses implement multiple sets of editing tools” — actually hints at a larger trend:
Tool Calling protocols are regressing from open standards to platform-specific protocols.
Just like iOS’s UIKit vs. Android’s Material Design — it’s not that you can’t go cross-platform, but the “optimal experience” is always tied to a specific platform.
By the same logic, Claude Code’s edit tool = Anthropic’s UIKit. You wouldn’t expect an iOS app to run equally well on Android.
Summary
| Decision | The Fundamental Question It Answers |
|---|---|
| Dynamically switch tool schemas | Who bears the compatibility cost? |
| Fuzzy matching validation | How do you balance tolerance and success rate? |
| Accept retry costs | Which architectural complexity is more worthwhile? |
| Accept platform divergence | Is this a solvable problem or a reality to adapt to? |
Armin’s discovery deserves a pause from everyone building agent toolchains: is the time you spend tuning tool schemas actually solving a technical problem, or are you fighting a business decision?
If you’re one person or a small team building a coding harness, the most pragmatic answer is: don’t fight it. Make yourself compatible, accept retries, and invest your energy in the value of the tool itself.
🎯 Building your own agent system with Claude / GPT?
- 📬 Follow “全栈之巅 - 梦兽编程” for first-hand agent engineering practice
- 🔧 Visit Rex Toolset to discover more AI development tools
References:
- Armin Ronacher, Better Models: Worse Tools , Jul 4 2026
- Simon Willison, Better Models: Worse Tools , Jul 4 2026