If an AI can finish a 2-hour office task in 12 minutes for less than 20 cents, would you hand over your work?
Don’t answer too quickly.
Baidu’s research team just released a fascinating benchmark. Instead of quizzing models on math problems or checking whether an agent clicked the right button, they prepared 100 real office tasks and asked AI to directly deliver finished Word documents, Excel spreadsheets, PowerPoint decks, and PDFs.
The result is full of contradictions: AI is dramatically faster and cheaper — but the output is far from “ready to use.”
What Makes This Test Different
Most office agent benchmarks evaluate the process: did it open the right menu? Did it paste text into the correct cell? Did it save the file?
Real work doesn’t work that way.
Your boss won’t approve a report because an agent clicked 30 buttons correctly. A client won’t accept a spreadsheet with broken formulas because the Excel tool was invoked properly.
People only care about whether the final deliverable is usable.
OmegaUse-OfficeVal focuses on exactly that: the final deliverable. It includes 100 long-horizon tasks derived from real workplace requests — documents, spreadsheets, presentations, and PDFs. Each task takes a human an average of 2.32 hours to complete.
The team also annotated every task with two economic signals: how long a human takes, and what the task is worth on the market. This makes it possible to compare not just “who scored higher,” but “who was faster” and “who was cheaper.”
To reduce subjectivity, they use code-based verifiers instead of asking another LLM to judge — sidestepping the awkwardness of “AI grades AI’s homework.”
The Best Model Scored Only 64% of Human Performance
The team tested GLM-5.2, Qwen3.7-Plus, Kimi K2.6, DeepSeek-V4-Pro, and MiniMax M3, with humans as a baseline.
| Executor | Avg Score | Avg Cost | Avg Time |
|---|---|---|---|
| Human | 27.79 | $6.8560 | 2.324 hours |
| GLM-5.2 | 17.91 | $1.4823 | 0.521 hours |
| Qwen3.7-Plus | 17.51 | $0.2152 | 0.193 hours |
| Kimi K2.6 | 17.00 | $0.7719 | 0.389 hours |
| DeepSeek-V4-Pro | 14.48 | $0.6111 | 0.184 hours |
| MiniMax M3 | 13.82 | $1.7572 | 2.275 hours |
GLM-5.2 scored the highest at 17.91 — roughly 64% of the human baseline. Qwen3.7-Plus was only 0.4 points behind, but averaged just $0.2152 per task (about 1.5 RMB).
On paper, AI looks like a bargain. Qwen3.7-Plus delivers results in 0.193 hours (~11.6 minutes) versus the human average of 2.324 hours — roughly 12x faster, at a fraction of the cost.
But don’t skip the most important column: quality.
These models aren’t cheaper at equal quality. They’re cheaper while delivering noticeably worse quality. Framing “12 minutes per task” as “12 minutes replaces 2 hours of human work” quietly deletes the paper’s most important caveat.
The Real Problem Is the Zero-Score Rate
Averages can hide a lot. Some tasks didn’t just score poorly — they scored nothing at all.
- GLM-5.2 had 14% of tasks score above 50, the best among all models.
- Qwen3.7-Plus had the lowest zero-score rate — but it was still 38%.
- DeepSeek-V4-Pro and MiniMax M3 had zero-score rates of 50% and 51% respectively.
Even with the best-performing model, more than one-third of tasks resulted in complete failure under the current setup.
This matches everyday experience with AI. Extracting fields from a table? Often solid. Adding annotations to existing content? Pretty reliable. But once a task involves complex formatting, aesthetic judgment, multi-file constraints, or requires the agent to decide what “deliverable quality” means — problems start piling up.
The paper also observed domain differences: GLM-5.2 leads on engineering and technology tasks, Qwen3.7-Plus performs better on business operations, DeepSeek-V4-Pro excels on academic papers, while financial data tasks are challenging across the board.
No single model can reliably handle all office work today.
Operating Software Isn’t the Same as Delivering Work
What makes this benchmark significant isn’t the model rankings. It’s the gap between “operation success” and “delivery success.”
Traditional automation is dumb but deterministic: write a rule, and it repeats the rule. AI agents are far more capable — they understand natural language, span multiple tools, and handle unscripted situations.
The tradeoff: they make judgments.
And the hardest part of office tasks is precisely judgment: is this heading prominent enough? Is this table scannable at a glance? Does this slide communicate the key point? When data is missing, should I stop and ask, or fill in with a best guess?
Human colleagues constantly negotiate these questions. An agent tends to keep going until it produces something structurally complete and file-openable — but riddled with problems on closer inspection.
It didn’t crash. It didn’t error out. It just completed the wrong thing competently.
That’s worse than a crash. A crash tells you the task failed. A complete-looking file tempts you to skip review.
The Right Way to Use AI Office Agents Today
I’m not concluding “AI is useless for office work.” Quite the opposite: at a few cents to a few dollars per task, AI can transform the starting point for a huge amount of work. Data extraction, field annotation, format conversion, draft generation — tasks that are time-consuming but judgment-light are ideal first passes for AI.
The problem is the final step.
Today, AI is a fast, cheap executor that still requires a human to sign off. You can let it do the first pass — you can’t let it hit “submit.”
Four steps for productive AI office work:
- Write acceptance criteria upfront. Don’t just say “make a presentation deck.” Specify the audience, page count, data to preserve, and content to avoid.
- Stage the delivery. Confirm the outline first, then the data, then generate the file. Don’t wait until everything is done to check.
- Use a checklist to inspect the output. Spreadsheets: check formulas and units. Slides: check information hierarchy and cropping. PDFs: check fonts, pagination, and links.
- Keep a human accountable for the final result. For anything involving money, contracts, customer data, or public release, this step is non-negotiable.
A practical litmus test: if it takes you longer to review the AI’s output than to do the task yourself, that task isn’t ready for an agent yet.
Cheap Execution, Still Requires Expensive Judgment
Past discussions of AI office work focused on “can it do the task?” This benchmark pushes the question further: is what it produces worth using?
The answer isn’t bleak.
AI can already handle a meaningful share of work at very low cost, and on some tasks it produces genuinely usable output. But the average scores and zero-score rates remind us that stability isn’t there yet. Especially on long workflows, ambiguous requirements, and final deliverable quality — human judgment remains the most expensive and hardest-to-automate link in the chain.
The most effective approach today isn’t handing over your computer. It’s handing over the repetitive work — and keeping the sign-off authority yourself.
Source
- Baidu Agent Frontier Team, OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Suite Tasks with Economic Grounding
- arXiv: 2607.27155
- Full paper: arxiv.org/html/2607.27155
Note: This is a 100-task benchmark. Results are influenced by task composition, model versions, agent scaffolding, and scoring methodology, and should not be directly extrapolated to all office scenarios.