Screenshots or Source Code? StateAct and JarvisHub Reveal Two Futures for GUI Agents

Claude can operate your computer. OpenAI’s agent can browse the web. But what are they actually “seeing”?

The current mainstream approach for Computer-Use agents is: screenshot → vision model understands → output click coordinates. It’s been proven effective. But it has one fundamental problem: using pixels to understand interface state is like using a photograph to understand a building’s wiring diagram. Low information density, high noise.

On July 28, two highly-upvoted papers on HuggingFace Daily Papers — one from Salesforce AI Research, one from a 26-author collaboration — answered the same question from two different angles: what should GUI agents “see”?

The answers point to two distinctly different futures.

0. The Meta-Problem: The “Perception Layer” War for GUI Agents

Imagine teaching a blind person to use Photoshop. You have two options:

Option A: Take a photo of the screen every second and describe it. “Top-left has a blue button, 20 pixels down is a dropdown menu…” This is what most current Computer-Use agents do.

Option B: Tell them the program’s internal state directly. “Layer 3 is selected, brush tool active, color is #FF5500, canvas is 1920×1080…” This is what StateAct aims to do.

Option A’s advantage is universality — any software can be screenshotted. Option B’s advantage is precision — it won’t mistake “button is obscured” for “button doesn’t exist.”

But Option B has a massive prerequisite: you need access to the program’s internal state. For web applications, that’s the DOM and accessibility tree. For desktop apps, that’s OS-level UI automation interfaces. For creative tools (Photoshop, Figma), that’s the canvas object model.

The two papers cover exactly the two ends of this spectrum: StateAct goes the “program state” route, JarvisHub goes the “canvas-native” route.

1. StateAct: Read Code Before Screen

Paper Info

Why This Matters

Current Computer-Use agents have a well-known but rarely-discussed problem: they perform terribly on long-horizon tasks.

A long-horizon task means a complex workflow requiring 10+ steps — like “find all last week’s bugs assigned to me in Jira, change status to resolved, then send a summary email to the PM.”

Screenshot-based approaches work well on 2-3 step simple tasks, but on long-horizon tasks they hit three fatal problems:

1. State drift: Step 5’s screenshot visually looks nearly identical to step 1’s (same page, same buttons). The agent loses track — “did I already complete this?”
2. Occlusion: Popups, dropdowns, loading animations — any obstruction causes pixel-based agents to misread interface state
3. Information bottleneck: A 1080p screenshot has 2 million pixels, but the answer to “what file is currently being edited?” occupies roughly 400 pixels (the filename text area)

StateAct’s core insight: program state is a far higher-information-density perception channel than pixels.

The DOM can directly tell you “there are 3 buttons: Save, Cancel, Delete” without requiring the model to do object detection from pixels. The accessibility tree can directly tell you “focus is on the username input field” without requiring cursor-position reasoning.

Engineering Implications

If StateAct’s approach is validated, Computer-Use agent development will undergo two changes:

Short-term: Agent frameworks need a “state extraction layer” — before taking a screenshot, first obtain structured data from DOM/accessibility tree/UI automation APIs.

Long-term: Software itself may need to provide “machine-readable” state interfaces for agents. Just as today’s websites provide APIs for frontends, future applications may need “Agent APIs” — state descriptions that are more structured than pixels, without requiring understanding of all business logic.

“A program’s internal state, before pixels reach the screen, is already the agent’s best perception input.”

2. JarvisHub: An Open Testbed for Creative Agents

Paper Info

What Is “Canvas-Native”?

Canvas-Native agents don’t just “operate software” — they “create on canvas.”

Current mainstream agent capabilities focus on operational tasks (click, fill, navigate) and analytical tasks (read, summarize, classify). But one category is almost completely neglected: creative tasks — drawing, layout, design, editing multimedia content on a canvas.

These tasks require capabilities fundamentally different from traditional GUI agents:

JarvisHub aims to provide a standardized test platform for this direction — like SWE-bench for coding agents, or WebArena for web agents.

Why “Open”

The “Open Harness” in the title is key. In AI, an open testbed means three things:

1. Standardized task set: Community-recognized tasks for cross-comparing different agents
2. Reproducible evaluation: Same environment, same tasks, same metrics
3. Community-built: Not one company’s internal benchmark, but a public resource usable and improvable by all

JarvisHub is to creative agents what ImageNet was to computer vision: consolidating scattered capability assessments onto a single public platform.

The Multimodal Meaning

“Canvas-Native Multimodal Creative Agents” — the “Multimodal” here has two layers:

1. Input multimodal: Agent receives text instructions + reference images + current canvas state
2. Output multimodal: Agent produces not text, but graphics, designs, video clips

This requires the agent to simultaneously possess language understanding, visual perception, and spatial manipulation capability — precisely the weakest area of current general-purpose models.

3. Two Paths, One Core Assumption

StateAct and JarvisHub appear to discuss different topics — enterprise GUI automation vs. creative tool agents. But they share a foundational assumption:

Pixels are not the agent’s best perception channel.

StateAct’s model: supplement or replace pixels with program state (structured, symbolic).

JarvisHub’s model: use canvas object models (structured, spatial) to understand creative interfaces.

The two paths sit at different positions on the same spectrum:

Pure Pixels ───── Pixels + DOM ──── Canvas Object Model ──── Pure Program State
(current norm)   (near-term opt)     (JarvisHub)              (StateAct ideal)

Which Approach for Which Scenario?

4. Practical Impact for Developers

If You’re Building GUI Agents

What you can do right now:

1. Add a “state extraction layer” to your agent framework. Before screenshots, try getting structured state via DOM/accessibility tree/UI automation. The code change is small; the improvement on long-horizon tasks could be order-of-magnitude.

2. Don’t evaluate agents solely on “task completion rate.” StateAct implies that on long-horizon tasks, the agent’s “state awareness” (knowing where it is in the workflow) matters more than per-step accuracy. Add “state-tracking accuracy” to your evaluation metrics.

If You’re Building SaaS Products

Worth thinking about ahead of time:

JarvisHub’s emergence signals: there will be agents specifically designed to “create content inside your product.” If your product is a design tool, document editor, or multimedia platform, start thinking now:

• Does your canvas/editor have an object model readable by agents?
• If not, are you okay with agents operating your product through screenshots? (The information bottleneck will make the experience terrible)
• Does your product need a “creation API” for agents?

This isn’t sci-fi — Claude and OpenAI can already operate various software through Computer Use. When they start entering creative domains, having or lacking native agent interfaces will determine whether your product is “AI-friendly” or “AI-hostile.”

Summary

Computer-Use agents are undergoing a “perception layer” paradigm shift. From “screenshot + click” to “program state + canvas objects,” this shift will determine whether next-gen agents can only do form-filling, or can truly operate complex software, complete long-horizon tasks, and even perform creative work.

And the core question driving this shift is simple: would you rather have an agent understand your interface through a 2-million-pixel photo, or would you rather tell it directly what’s inside?

🎨 Building GUI agents or creative tools?

• Is your agent using pure pixel-based perception? Try adding a program state extraction layer
• Does your product have an “agent-friendly” object model? If not, this could become a competitive disadvantage within 12 months
• The standardization era for creative agents has begun — JarvisHub is worth watching

📬 Follow 梦兽编程 for more AI agent frontier research analysis.

Sources:

• HuggingFace Daily Papers (July 28, 2026): StateAct External link — Salesforce AI Research
• HuggingFace Daily Papers (July 28, 2026): JarvisHub External link — 26 authors
• AI News: Opus 5 + Agent tools coverage External link