Skip to content

Evaluation preparation reference

The coding host normally prepares these inputs and runs the CLI. Start with Prepare an evaluation for a guided workflow. For direct CLI use, run commands in the application checkout with actual IDs and file paths in place of uppercase placeholders.

Revise or reuse an evaluation

A frozen package is immutable. Ask your coding agent to create a new draft from it. Replace EVALUATION_ID and adapt this example to the changes you want:

Select Prepare evaluation, then send:

Create a new evaluation draft from EVALUATION_ID. Add coverage for
tool timeouts and empty search results, using the configured local profile.
Agree the preparation limits with me, validate the revised benchmark,
and obtain an independent review before freezing it.
Preserve the original evaluation and its results.

Prepare evaluation Create a new evaluation draft from EVALUATION_ID. Add coverage
for tool timeouts and empty search results, using the configured local
profile. Agree the preparation limits with me, validate the revised
benchmark, and obtain an independent review before freezing it.
Preserve the original evaluation and its results.

To measure compatible later application source, use a baseline rerun with the same evaluator. Changing cases, behavior definitions, scoring or judge configuration requires a new draft. Preserve the original package and its comparisons.

Create a draft from the CLI 
agentagon _internal eval start --from EVALUATION_ID --profile local --budget-file BUDGET_JSON --author AUTHOR

Supply a real author identity and a preparation budget file. The coding agent normally prepares these inputs for you.

Direct CLI reference

The coding host normally prepares these files and runs these operations. For direct use, replace uppercase placeholders with actual IDs and files:

agentagon _internal eval start --intent INTENT_ID --goal 'Check tool routing' --profile local --budget-file BUDGET_JSON --author AUTHOR
agentagon _internal eval start --audit AUDIT_ID --issue ISSUE_ID --profile local --budget-file BUDGET_JSON --author AUTHOR
agentagon _internal eval lookup EVALUATION_ID --context-file CONTEXT_FILE --goal-file GOAL_FILE --phase initial --limit 5
agentagon _internal eval check EVALUATION_ID --plan-file PLAN_JSON
agentagon _internal eval freeze EVALUATION_ID --review-file REVIEW_JSON
agentagon _internal fix start --evaluation EVALUATION_ID --profile local

A preparation budget file has this shape:

{
  "max_trials": 12,
  "max_elapsed_seconds": 1800,
  "trial_timeout_seconds": 60
}

Use the plan and review contracts and returned templates for the other files. Do not fill a review with claimed results that were not observed. See task evidence helpers for benchmark diagnostics.

A passing benchmark establishes evidence for its declared cases. It does not erase coverage limitations or prove production improvement.