Task-specific benchmarks
Build representative cases with acceptance criteria, expected source use, and known exceptions. Evaluate the workflow rather than only its final prose.
INTELLIGENCE / EVALUATION ENGINE
Evaluate whether work used the correct sources, respected policy, stayed within permissions, and produced a checked outcome. A convincing answer alone is not enough to promote an agent.
02 / THE OPERATING CONTRACT
Build representative cases with acceptance criteria, expected source use, and known exceptions. Evaluate the workflow rather than only its final prose.
Test allowed, blocked, and review-required cases. Include attempts to access restricted data or perform actions outside the worker’s role.
Compare agent, tool, policy, and model versions against the same case set. Regressions need investigation before a version is promoted.
Assess whether the worker paused at the right point and gave the reviewer usable evidence. Unnecessary escalation and missed escalation are different problems.
03 / CONNECTED BY DESIGN
Inspect the sequence.
Each stage has a defined responsibility.
Select a node to inspect its role in the workflow.
Define the source of truth, permitted actions, accountable owner, and acceptance criteria. Scope matters more than an open-ended instruction.
Check the outcome, capture unresolved exceptions, and preserve the relevant policy and evidence. A task is complete only within its defined boundary.
04 / A CONCRETE EXAMPLE
An illustrative update improves summary fluency but fails to identify conflicting reporting periods. The evaluation rejects promotion because the workflow’s evidence check regressed. Task quality is measured by the full operating contract.
Use separate measures for correctness, source accuracy, permission compliance, latency, cost, and escalation. A single headline percentage can conceal a critical failure.
05 / QUESTIONS & BOUNDARIES
Architecture descriptions show intended system behavior. Validate implementation, integration, data handling, and operational controls before relying on a production workflow.
No. Evaluation is specific to a task, case set, configuration, and deployment.
That depends on authorized release policy. High-impact changes may require a human reviewer even when tests pass.
Identity, context, policy, evaluation, verification, execution, and audit form one operating model. Explore the related components below to see their responsibilities.
06 / CONTINUE THE EXPLORATION