Companies Offer 30K for Agent Engineers, Demo Writers Can't Get Hired

On one side, the hiring market: companies offering 30K-40K RMB a month for agent engineers can’t find people who can solve real production problems. On the other side, the job market: developers whose resumes are packed with “independently built XX agents” keep failing interview after interview.

Both are true at once, and they cause each other. The problem is not framework fluency — LangChain or LangGraph takes an afternoon to get running with the docs. What actually blocks both sides is the layer of engineering capability that sits between a demo and a production system.

The essence of this gap is not “can you build an agent” but “can you manage uncertainty” — a demo only needs to succeed once in an ideal environment; production demands that you catch every failure, every time, through network drops, API timeouts, model hallucinations, and traffic spikes.

We break the gap into four skill blocks: business decomposition with human-in-the-loop design, a highly available tool chain with multi-agent architecture, quantitative evaluation with runtime protection, and engineering delivery with continuous iteration. Talking about “what you need” is useless without “what it looks like,” so this article does three things: expand each block, back each one with primary-source production cases (we read the originals from Anthropic, Cognition, and AWS, and verified every number), attach three copy-paste-runnable code examples (Python 3.11, standard library only, outputs are real runs), and draw the boundary conditions that the most-quoted numbers usually leave out.

1. Where the gap is: a demo proves “it runs,” production demands “it never falls over”

Put the two environments side by side:

Our judgment: many developers have polished resumes but lack the engineering skills — concurrency, fault-tolerant degradation, context compression. This is not a regional phenomenon; frontier labs stepped on the exact same rake. In Anthropic’s retrospective on its own multi-agent research system, they admit the early agents were “spawning 50 subagents for simple queries, scouring the web endlessly for nonexistent sources, and distracting each other with excessive updates.” Even a top-tier model team’s first production agent failed on engineering, not on the model.

Conversely, Anthropic’s earlier piece on building effective agents has a counter-intuitive observation: across the teams they worked with, “the most successful implementations weren’t using complex frameworks or specialized libraries. Instead, they were building with simple, composable patterns.” Together the two articles locate the gap precisely: it is not about framework fluency; it is about whether you can catch failure when it arrives. A demo implicitly assumes the outside world cooperates; a production system’s first principle is that the outside world will break, and your value shows in the seconds after it does. The four skill blocks below are all, at heart, answers to “what happens after things break.”

2. Skill one: decompose a fuzzy requirement into an executable task graph

Your boss says “use AI to bring support costs down.” That is not a requirement; it is a wish. The first engineering step is translating it into measurable indicators: what first-contact resolution rate, what escalation-to-human rate, what cost per conversation in cents. Until the indicators exist, every downstream technical choice has no criterion.

The first cut of decomposition is deciding whether an agent should be involved at all. In “Building effective agents,” Anthropic draws a line many people skip: workflows and agents are different things — the former orchestrates LLMs and tools through predefined code paths, the latter lets the LLM dynamically direct its own process. Their advice is blunt to the point of being a buzzkill: find “the simplest solution possible,” so simple that it “might mean not building agentic systems at all.” For well-defined tasks, workflows are predictable, consistent, and cheaper. The first layer of decomposition skill is not “how to split the work for an agent” but “which parts don’t deserve an agent at all” — whatever can be written as a fixed flow goes to a workflow, the LLM only appears at nodes that genuinely need judgment, and cost and failure surface shrink together.

For the parts you do hand to an agent, the mainstream approach is the ReAct loop — the model converges step by step through “think → act → observe → adjust” instead of generating a grand plan in one shot. The pattern comes from Yao et al.’s 2022 paper “ReAct: Synergizing Reasoning and Acting in Language Models,” and virtually every agent framework’s main loop today is a variant of it:

loop:
  thought = llm.think(context)        # think: where are we now
  action  = llm.decide(context)       # act: which tool to call
  result  = tools.run(action)         # observe: what the tool returned
  context.append(thought, action, result)
  if result.indicates_done(): break   # adjust: converge or continue

After decomposition comes the final gate: not every action deserves full automation. At high-risk decision points — large refunds, important contracts — the correct move is to suspend the agent, generate a ticket for human review, and wake it up with the decision. That is human-in-the-loop. Here is a minimal runnable implementation (python demo1_hitl.py):

APPROVAL_THRESHOLD = 500  # 元；超过这个金额必须人工审批

class RefundAgent:
    def __init__(self):
        self.state = "idle"
        self.ticket = None

    def handle_refund(self, order_id: str, amount: float):
        if amount > APPROVAL_THRESHOLD:
            # 高风险动作：挂起，生成工单，交给人
            self.state = "suspended"
            self.ticket = {"type": "refund_approval", "order_id": order_id,
                           "amount": amount, "status": "pending"}
            print(f"[agent] refund {amount} CNY > threshold {APPROVAL_THRESHOLD}, suspending")
            return
        self.state = "running"
        print(f"[agent] refund {amount} CNY <= threshold, executed directly: order {order_id}")
        self.state = "done"

    def on_human_decision(self, approved: bool):
        assert self.state == "suspended", "no suspended task to resume"
        self.ticket["status"] = "approved" if approved else "rejected"
        self.state = "running"  # 带着人工结论唤醒
        if approved:
            print(f"[agent] woke up, ticket approved -> executing refund of {self.ticket['amount']} CNY")
        else:
            print("[agent] woke up, ticket rejected -> refund aborted, notifying user")
        self.state = "done"


agent = RefundAgent()
agent.handle_refund("A-1001", 199)        # 199 元：低于阈值，直接执行
agent2 = RefundAgent()
agent2.handle_refund("A-1002", 5000)      # 5000 元：超过阈值，挂起等审批
agent2.on_human_decision(approved=False)  # 人工驳回 -> 唤醒 -> 终止退款

Real run output (small refund straight through, large one suspended, and after human rejection not a cent moved):

[agent] refund 199 CNY <= threshold, executed directly: order A-1001
[agent] refund 5000 CNY > threshold 500, suspending
[agent] woke up, ticket rejected -> refund aborted, notifying user

In production frameworks, LangGraph’s interrupt and various ticket-system integrations do the same thing: state can be suspended, resumed, and a human decision injected. Whether to add this gate is not about how accurate the model is — it is about “what does it cost if this one goes wrong.” Irreversible actions deserve HITL; adding approval to reversible actions just ruins the experience.

3. Skill two: accept that failure is certain, then design for it

The first layer of tool-calling craft is precision: unambiguous API descriptions, explicit parameter boundaries, and few-shot examples for behavioral alignment. The same query tool described as “queries information” versus described like below produces completely different call accuracy:

{
  "name": "query_order",
  "description": "Query order status and amount by order ID. Paid orders only; for refund progress use query_refund.",
  "parameters": {"order_id": {"type": "string", "pattern": "^[A-Z]-\\d{4}$"}}
}

But no matter how well you write descriptions, calls will fail — networks drop, APIs time out, upstreams die. The standard three-stage fault-tolerant chain in production is intercept, exponential backoff, and local fallback. Here is an origin story many people don’t know: this machinery was not invented in the agent era; it turns eleven this year. In March 2015, AWS’s Marc Brooker published “Exponential Backoff And Jitter” on the AWS Architecture Blog, using simulations to prove that fixed-interval retries make every client hit the upstream at the same moment, and only randomized exponential backoff spreads the retry flood into an approximately constant trickle. A May 2023 update to that post notes that eight years later, “this solution continues to serve as a pillar for how Amazon builds remote client libraries for resilient systems,” and the AWS SDKs’ standard retry modes ship with it built in.

So the following example is not a toy — it is the same retry kernel you use every day in your SDKs (python demo2_retry_breaker.py; the demo scales the delay base from 1 second down to 0.1):

import random, time

random.seed(42)           # 固定随机种子，让演示输出可复现
BASE_DELAY = 0.1          # 演示比例；生产环境用 1.0，即 1s、2s、4s、8s……
MAX_RETRIES = 4
BREAKER_THRESHOLD = 3     # 连续失败 3 次，熔断器断开

class Breaker:
    def __init__(self):
        self.consecutive_failures = 0
        self.state = "closed"

    def record(self, ok: bool):
        if ok:
            self.consecutive_failures = 0
            self.state = "closed"
        else:
            self.consecutive_failures += 1
            if self.consecutive_failures >= BREAKER_THRESHOLD and self.state == "closed":
                self.state = "open"
                print("[breaker] 3 consecutive failures -> open, short-circuiting and alerting")

upstream_calls = 0

def flaky_upstream(fail_times: int):
    """上游模拟：前 fail_times 次调用抛超时，之后恢复正常。"""
    global upstream_calls
    upstream_calls += 1
    if upstream_calls <= fail_times:
        raise TimeoutError("upstream timeout")
    return "real upstream answer"

def call_with_retry_and_fallback(breaker, fail_times: int):
    if breaker.state == "open":
        return "fallback: cached default answer"     # 熔断中：直接走兜底，不调上游
    for attempt in range(MAX_RETRIES + 1):
        try:
            result = flaky_upstream(fail_times)
            breaker.record(ok=True)
            return result
        except TimeoutError:
            breaker.record(ok=False)
            if breaker.state == "open":
                return "fallback: cached default answer"
            if attempt < MAX_RETRIES:
                delay = random.uniform(0, BASE_DELAY * (2 ** attempt))  # 指数退避 + 全抖动
                print(f"[chain] attempt {attempt + 1} failed, retrying in {delay:.2f}s")
                time.sleep(delay)
            else:
                return "fallback: cached default answer"  # 重试耗尽：本地规则兜底

breaker = Breaker()

# 场景一：上游先挂 2 次后恢复 -> 重试生效
r1 = call_with_retry_and_fallback(breaker, fail_times=2)
print(f"[scenario 1] recovered after retries: {r1!r}")

# 场景二：上游持续故障（99 次都失败）-> 重试耗尽 + 熔断器断开
r2 = call_with_retry_and_fallback(breaker, fail_times=99)
print(f"[scenario 2] exhausted retries: {r2!r}")

# 场景三：熔断器仍断开 -> 上游一次都不会被调用
before = upstream_calls
r3 = call_with_retry_and_fallback(breaker, fail_times=99)
print(f"[scenario 3] breaker open, upstream calls unchanged at {upstream_calls}: {r3!r}")

Real run output (all three scenarios in one run):

[chain] attempt 1 failed, retrying in 0.06s
[chain] attempt 2 failed, retrying in 0.01s
[scenario 1] recovered after retries: 'real upstream answer'
[chain] attempt 1 failed, retrying in 0.03s
[chain] attempt 2 failed, retrying in 0.04s
[breaker] 3 consecutive failures -> open, short-circuiting and alerting
[scenario 2] exhausted retries: 'fallback: cached default answer'
[scenario 3] breaker open, upstream calls unchanged at 6: 'fallback: cached default answer'

Three details deserve a pause. First, random.uniform(0, base * 2**attempt) is exactly what the AWS post named “Full Jitter” — the interval shrinking from 0.06s to 0.01s is not a bug; full jitter allows any single interval to shorten while the expected ceiling grows as powers of two. Second, once the breaker opened, the upstream call count froze at 6 — every request you keep sending to a dying upstream burns your own token bill. Third, the “circuit breaker plus alerting” fuse against runaway bills is three lines of code in this chain; without it, a looping agent can burn an unbounded amount of money overnight.

Frontier labs run the same playbook in production, with two agent-specific patches. Anthropic’s retrospective mentions two: restarts from scratch are “expensive and frustrating for users” on long-horizon tasks, so they built checkpoint-and-resume; and “letting the agent know when a tool is failing and letting it adapt works surprisingly well” — error messages are not just logs; they are corrective signals you feed back to the model. Classic retries cover the network, checkpoint-resume covers long tasks, and telling the model about failures covers decisions. Three layers, each with its own job.

The anti-pattern for this section is one line: while True: call_api(). Treating failure as an unlikely exception to pray against, versus treating it as a statistical certainty to design for, is the dividing line between a demo engineer and a production engineer.

4. Skill three: the multi-agent debate, and the context ledger that matters more

The common architectural answer is multi-agent division of labor: a lead agent orchestrates, specialist agents (finance, inventory) each mind their own job. But in June 2025, that answer was publicly challenged — and both sides brought receipts. It was the most worthwhile fight in agent engineering that year, and both sides inform our judgment.

The affirmative, Anthropic, speaks in numbers. Their research system uses an orchestrator-worker pattern (a lead agent coordinating, subagents working in parallel), and the measured result: a multi-agent system with Claude Opus 4 as lead and Claude Sonnet 4 subagents “outperformed single-agent Claude Opus 4 by 90.2% on our internal research eval.” For breadth-first queries like “find all board members of S&P 500 IT companies,” splitting into independent directions pursued simultaneously is a structural advantage. But the same article states the cost just as plainly: “agents typically use about 4× more tokens than chat interactions, and multi-agent systems use about 15× more tokens than chats.” The economic verdict in the original text is cold: multi-agent only works “where the value of the task is high enough to pay for the increased performance.”

The opposition, Cognition (the Devin team), speaks in principles. In “Don’t Build Multi-Agents” (June 12, 2025), Walden Yan gives two context-engineering principles: “Share context, and share full agent traces, not just individual messages”; and “Actions carry implicit decisions, and conflicting decisions carry bad results” — two subagents working separately make assumptions the other can’t see, and the pieces inevitably clash. His example is concrete: have one subagent build the scrolling background of a Flappy Bird clone and another build the bird, and their implicit assumptions about gravity and controls won’t line up, so the combined result is unplayable. His default recommendation is a single-threaded linear agent, with a supporting observation: Claude Code spawns subtasks, but those subtasks only answer questions — they never write code in parallel with the main agent.

Are the two sides contradictory? Our ruling: no — the disagreement lives in task structure. Research tasks are breadth-first with independent sub-directions and few implicit decisions, so multi-agent’s 15× tokens buy a 90% capability gain. Coding tasks are tightly coupled and dense with implicit decisions, so Cognition’s two principles shatter on contact. The criterion is one question: how many implicit decisions must your subtasks share — the more they share, the more they belong in one context.

More mundane than the architecture debate is the context ledger itself. Long conversations have two obvious costs: token spend grows with every turn, and the model’s grip on early content fades with distance. The Anthropic piece has an easily missed quantitative finding: in the BrowseComp evaluation, token usage alone “explains 80% of the variance” in performance — more than tool-call count or model choice. Context is the hardest currency in this business; how you spend it sets the ceiling on system capability. The engineering answer pairs two moves: explicitly extract key facts and pin them in the prompt (or store them in a vector store for retrieval), while a sliding window evicts chit-chat by age. Minimal runnable example (python demo3_context_window.py; token counts use a len(text)//2 estimate for demo purposes only — production must use the model’s real tokenizer):

def est_tokens(text: str) -> int:
    return max(1, len(text) // 2)  # 演示估算；生产用真实 tokenizer

BUDGET = 120  # token 预算，故意调小让效果可见

system_prompt = "你是客服 Agent，只处理订单问题。已知关键事实：订单号 A-1001，金额 5000 元，状态：待人工审批。"
history = [f"第{i}轮闲聊内容：" + "嗯。" * 10 for i in range(1, 13)]  # 12 轮低价值历史
recent = ["用户：我的退款到底什么时候到账？",
          "Agent：查询中，订单 A-1001 的退款金额为 5000 元，需要人工审批。"]

full = [system_prompt] + history + recent
print(f"[before] {len(full)} messages, ~{sum(est_tokens(m) for m in full)} tokens (budget {BUDGET})")

windowed = [system_prompt]                      # pinned：系统提示与关键事实永驻
for msg in reversed(recent + history[-2:]):     # 从最新往最旧装
    if sum(est_tokens(m) for m in windowed) + est_tokens(msg) <= BUDGET:
        windowed.insert(1, msg)
    else:
        break

print(f"[after]  {len(windowed)} messages, ~{sum(est_tokens(m) for m in windowed)} tokens; "
      f"dropped {len(full) - len(windowed)} old turns")

assert sum(est_tokens(m) for m in windowed) <= BUDGET
assert any("A-1001" in m for m in windowed), "pinned 关键事实必须在压缩后存活"
print("[ok] budget met; pinned facts survive; oldest low-value turns evicted")

Real run output:

[before] 15 messages, ~224 tokens (budget 120)
[after]  5 messages, ~84 tokens; dropped 10 old turns
[ok] budget met; pinned facts survive; oldest low-value turns evicted

The point is not how many tokens were saved but “what survived”: order A-1001 is pinned in the system prompt and cannot slide out no matter how the window moves; what got dropped was the ten oldest chit-chat turns. A bare sliding window without fact extraction will eventually slide the user’s order number out of context — the context window is a budget, not a warehouse, and a budget should be spent where it counts.

5. Skill four: without measurement, there is no production

The three metrics that show up most often in job descriptions and conference talks: task completion above 95%, tool-call error rate under 1%, P99 latency under 500ms. All point the right direction, but copying them verbatim will hurt you. The boundary conditions:

• 95% completion: define “done” before chasing the number. Single-turn QA completion and multi-step task completion are different things — the latter is the product of per-step success rates, so a 10-step task at 99% per step lands around 90% end-to-end. Without a defined contract, 95% is just a pretty number.
• 1% tool error rate: the most directly adoptable of the three, because it measures your own engineering chain and doesn’t depend on the model’s mood.
• P99 500ms: on a multi-step LLM chain, this number usually only constrains the non-model parts — the tool calls, retrieval, and auth code you wrote yourself — or the time to first token. LLM generation itself routinely takes seconds; a full-chain 500ms is approached through streaming, concurrent tool calls, and caching, and is flat-out unreachable in many scenarios. Anyone reciting all three numbers as dogma in an interview is telling you they never chased these metrics in production.

More important than any single metric is the evaluation system itself. Hamel Husain’s March 2024 post “Your AI Product Needs Evals” puts it harshly: unsuccessful AI products “almost always share a common root cause: a failure to create robust evaluation systems.” His ladder has three rungs: cheap unit-test-style assertions at level one; human review plus model grading at level two (LLM-as-a-Judge lives here, alongside trace logging); A/B testing at level three. The increasingly common practice of “letting a top model judge every commit” is not a standalone trick — it is the second rung of this ladder. Without hard assertions below it and real-traffic validation above it, a judge model alone gives you a test system that looks like a goalkeeper but is quietly letting goals in: judges drift too, so version your eval sets and spot-check the judge’s verdicts with humans regularly.

Beyond metrics, two runtime safety nets are mandatory. First, loop protection: hard-stop with a step counter and an execution-depth cap — in code it is one line, if steps > MAX_STEPS: break, but it must exist explicitly; its absence is a signed license for infinite loops. Second, the circuit breaker from section 3: consecutive failures cut off calls and fire an alert, protecting both the upstream and your bill.

At the end of the day, an agent system optimizes whatever you measure — talking about “improving quality” before you’ve defined a completion rate is engineering by vibes.

6. Turn skill into process: engineering delivery and continuous iteration

The last block lives not in code but in process. Three practices, each with a real frontier-lab implementation:

1. LLM-as-a-Judge in CI. Hamel’s post mentions that Rechat, the team behind the real-estate AI assistant Lucy, runs evaluation tests on CI infrastructure like GitHub Actions — evaluate on every commit. That is not a vision; it is Tuesday.
2. Canary releases with lossless rollback. Anthropic’s multi-agent system uses “rainbow deployments”: old and new versions run simultaneously while traffic gradually shifts over. Why not cut over directly like a normal service? Because agents run long tasks, and a hard cut throws the user’s half-finished progress in the trash — in their words, “we can’t update every agent to the new version at the same time.” The classic advice of “start at 5% traffic and roll back in milliseconds when errors rise” is the same idea, except the agent version of a canary must also babysit in-flight tasks.
3. Distributed tracing with a Trace ID. Anthropic added full production tracing across the chain, with one detail worth copying: they trace decision patterns and interaction structures, not the conversation contents — so when a user reports the agent “not finding obvious information,” they can tell whether the queries were bad, the sources were poor, or a tool failed, all without touching user privacy. A multi-agent system without Trace ID is flying naked.

None of these three is an invention of the agent era — they are SRE practices transplanted onto a probabilistic system, down to the names. Which confirms the opening thesis: the core of agent engineering is not new magic; it is the old craft of deterministic engineering wrapped around an inherently probabilistic model.

Summary: four blocks, one question

The four rows add up to one sentence: a production agent engineer’s job is to wrap a probabilistic system in a deterministic shell, and to be able to state the thickness of every layer. Writing a demo proves you can make the model move; explaining how each layer of the shell absorbs failure, gets measured, and rolls back proves you can keep it alive. That is what actually sits between the 30K job posting and the resumes that sink without a trace.

FAQ

What is the real gap between writing a demo and running an agent in production?

A demo only needs to succeed once in an ideal environment; production must survive network failures, timeouts, hallucinations, and concurrency every single time. The gap is not framework fluency — it is the engineering that squeezes uncertainty into an SLA: fault tolerance, evaluation, gradual rollout.

Should I adopt a multi-agent architecture?

It depends on task coupling. Anthropic measured multi-agent beating single-agent by 90.2% on research tasks at roughly 15x the token cost; Cognition argues for a single-threaded agent by default unless subtasks need no shared implicit decisions. Both are right — the task structure decides.

Can I directly adopt targets like 95% task completion or 500ms P99?

Not verbatim. 95% depends on how you define “done”; 500ms on a multi-step LLM chain usually only constrains the non-model parts or time to first token. Define the measurement contract first, then optimize toward it.

When is human-in-the-loop mandatory?

When an action is irreversible and the loss exceeds what you can absorb: large refunds, contract signing, destructive operations. The criterion is not model accuracy but “what does it cost if this one goes wrong.”

Sources

• The framework, viewpoints, and the “deterministic shell” reading in this article are original to 「全栈之巅-梦兽编程」; the external sources below are used for cases and figures, and every key quote was checked against the original text.
• Anthropic Engineering, “How we built our multi-agent research system External link ” (Jeremy Hadfield et al., June 2025): the 90.2% multi-agent gain, ~4x tokens for agents and ~15x for multi-agent, token usage explaining 80% of variance, the “50 subagents” early failure, checkpoint-resume, rainbow deployments, and content-free tracing all come from this post.
• Anthropic Engineering, “Building effective agents External link ” (December 2024): the workflow/agent distinction, “the simplest solution possible… might mean not building agentic systems at all,” and the observation that successful implementations favor simple, composable patterns.
• Cognition blog, “Don’t Build Multi-Agents External link ” (Walden Yan, June 12, 2025): the two principles (share full traces; actions carry implicit decisions), the Flappy Bird example, and the Claude Code observation.
• AWS Architecture Blog, “Exponential Backoff And Jitter External link ” (Marc Brooker, March 4, 2015; updated May 2023): the “Full Jitter” name and the “still a pillar after eight years” update note. The backoff in demo2 is Full Jitter.
• Hamel Husain, “Your AI Product Needs Evals External link ” (March 29, 2024): the three-rung evaluation ladder, the “common root cause” claim, and Rechat running evals in CI.
• The ReAct pattern comes from Yao et al.’s 2022 paper “ReAct: Synergizing Reasoning and Acting in Language Models” (searchable on arXiv).
• All three code examples are Python 3.11 standard library only, really executed on our machine before publication, with real outputs shown; delay numbers use the demo scale (0.1s base) — production corresponds to 1s, 2s, 4s, 8s.

Related reading

• Sub-Agents vs Agent Teams: The Architecture Decision That Can Wreck Your System - our earlier deep dive on multi-agent granularity tradeoffs
• One Agent Beats a Thousand: What Ramp Taught Me About Enterprise AI - real practice of human approval for high-risk actions in financial automation
• Locking Down Local AI Agents: Agent Safehouse Review - the other side of runtime protection: permission tightening for local agents

The multi-agent debate: which side are you on

The fight in section four has three camps. Pick yours:

• A. Team Anthropic: breadth-first tasks deserve multi-agent parallelism — 15x tokens for a 90.2% gain is a fair price
• B. Team Cognition: single-threaded linear agents by default; continuous context beats parallelism, which breeds fragility
• C. It depends on the task: loosely coupled subtasks go multi-agent, tightly coupled stay single-threaded — the criterion is implicit-decision density

Tell us your pick and why in the comments. We especially want to hear from people who have actually run multi-agent systems in production — your token bills and incident reports are worth more than ten paragraphs of argument from either side.

If you want to self-test, copy the three code examples, run them, and answer one question: the last time your agent failed in production, did it die in retries, in the breaker, or in the context window? Post your answer in the comments; next week we will pick the most representative failure cases for a follow-up.

If this article described your team’s reality, forward it to the colleague who is preparing for an agent-engineering interview — or the one conducting them. Both sides of this gap deserve to see what the other side looks like.

About 全栈之巅-梦兽编程 (Rexai Programming): deep content on AI programming and full-stack engineering. Search 「梦兽编程」 on WeChat and follow the official account for more hands-on Rust and AI programming tutorials.