Evals

Evals measure how well one agent does one task, and then search for the configuration that does it best against a goal you write: “fast with good accuracy even at a higher cost”, or “the cheapest setup that still scores 8”. You build and run evals in AI → Evals (/ai/evals).

A run measures the agent as it is today across the models you choose, then (when you let it) the Eval agent proposes batches of candidate configurations, runs each one on a private copy of the agent in a private copy of your Workspace, scores every trial against an explicit objective, and repeats until the objective stops improving or a limit stops it. Every run ends on one checkpoint screen with the evidence: a quality against cost chart, usage per trial, tool and sub-agent charts, workflow before and after pictures, and a recommendation you can Apply or hand back with Run more experiments.

What a test contains

A test is one task for one agent, written as a package in Fontana Files. You edit it in a form or as raw text.

Part	What it does
Agent	The one agent under test. Its model, settings, instructions, skills and reference files are the starting configuration.
Eval agent	The agent that judges answers, turns the goal into an objective, and proposes experiments. Only agents carrying the Eval judge availability appear here.
Models	Default runs the agent’s own model; Selected runs a list of approved models; Any approved runs every approved model. Each model in scope shows its catalog price, and a model without a price cannot be selected.
Workspace and workflow	Optional. The Workspace the task works in, and the workflow it opens. Every trial runs in its own private copy of that Workspace.
Task	The one prompt the agent receives, written as markdown.
Acceptance criteria	A list, one criterion per item, for example Quotes the 30-day return window. Add, edit, remove and reorder items in place. The judge scores each from 0 to 10.
Goal	Free text describing what “better” means for this test. The Eval agent turns it into the objective.
Search space	Six switches that say what the optimiser may change (see below).
Run settings	The mode and the limits of a run (see below).

The form has two columns: Test on the left (agent, Workspace and workflow, task, goal, criteria) and Optimisation on the right (models, search space, run settings).

Quality is judged, everything else is measured

The judge scores the acceptance criteria and nothing else. Cost, latency, turns, tokens and tool calls come from the run itself, including every sub-agent the agent starts, so you do not need criteria for them.

The goal and the objective

At the start of an Auto or Checkpointed run, the Eval agent turns your goal into an objective: a weight for each of quality, cost, latency and turns, plus optional hard limits (a minimum quality score, a maximum cost per trial, a maximum latency). Every trial is scored with the same objective. Quality is the mean criteria score; cost, latency and turns are compared with the baseline measured at the start of the run, so a trial at half the baseline cost scores twice as well on cost.

The checkpoint screen shows the objective in one line. If the derived weights are not usable (for example all zero), the run scores by quality alone and the checkpoint says why. To change the weights or limits, use Edit objective on the Recommendation card while the run is paused at a checkpoint (the person who started the run can do this): every trial so far is re-scored and the recommendation is rebuilt, and the baseline stays as measured. To fix the weights before a run starts, write an objective in the test; the run then uses yours as written. Hard limits keep a trial out of the recommendation, and the chart still shows it, so you can see what the limit cost you.

The search space

Each switch lets the optimiser change one part of the agent’s configuration. A proposal outside the switches is refused, and the refusal is shown on the iteration it came from.

Switch	What the optimiser may change
Models	The model, among the models in scope.
Model settings	Reasoning effort, temperature, maximum output tokens, and the turn budget of a trial (50 to 200 steps), so a cheaper model with more turns is a real candidate.
Instructions	The agent’s system prompt.
Skills	Which granted skill namespaces stay granted, and the SKILL.md text of granted skills outside system.
Abilities	Knowledge Graph search (on or off, automatic search, relevance threshold, and namespaces from the agent’s own list) and workflow search (on or off, automatic search).
Reference files	Add or rewrite reference files in the agent’s package and in its granted skill folders, up to 20 files of 64 KB each per candidate.

Email, Teams, messaging, connected MCP servers and Fontana Browser stay off in every trial, whatever the agent has enabled, so a trial never contacts anyone outside the run.

Run settings

Setting	Default	Range
Mode	Measure	Measure, Auto, Checkpointed
Iterations	3	1 to 10 experiment rounds after the baseline
Candidates per iteration	3	1 to 6
Repeats per candidate	1	1 to 3; quality is the mean across repeats, with the spread shown
Parallel trials	2	1 to 8; use 1 for agents that call slow external services
Maximum spend	10 USD	Any positive amount
Maximum time per trial	20 minutes	1 minute to 2 hours

The form shows the trial budget as “up to N trials”, which is iterations × candidates × repeats. The baseline counts toward it, so a test whose baseline alone (models in scope × repeats) exceeds the budget cannot start until you raise a limit or narrow the models.

Run modes

Mode	What happens
Measure	Runs the baseline: the unchanged agent on every model in scope. It stops at the checkpoint screen, where Run more experiments turns it into an optimisation run.
Auto	Runs the baseline, then iteration after iteration until a stop rule ends the run. You can pause it; it then stops at the next checkpoint.
Checkpointed	Pauses at the checkpoint screen after the baseline and after every iteration, and waits for you.

A run stops when the optimiser reports that more experiments would not move the objective, when the iterations, trials or spend run out, when you cancel it, or when a model’s spend cannot be priced. A run left waiting at a checkpoint for 24 hours finishes on its own, and its results stay available to Apply.

What happens in a run

Every experiment runs on copies. The loop reads your agent, its skills and your Workspace and writes only to private copies, so the originals change only when you Apply.

1. Prepare. Fontana copies the agent’s package (its AGENT.md and reference files) and the folders of its granted skills outside system into a private namespace that only you can open. It also confirms a price for every model in scope.
2. Baseline. The unchanged copy runs the task on every model in scope, as many times as Repeats says.
3. Propose. The Eval agent reads the objective, the search space, the model prices, every trial so far, and the current files of the copy, and proposes the next batch of candidates. Each accepted candidate gets its own private copy with that change applied.
4. Trials. Each trial copies the Workspace, opens the workflow, and runs the task. The agent and its sub-agents work only in that Workspace copy and the candidate’s private copy, at most Parallel trials at a time, each with its share of the remaining spend.
5. Judge and score. The judge scores the answer against the criteria, with evidence from the Workspace copy such as workflow changes and run results. The trial is then scored by the objective, and its Workspace copy is deleted.
6. Checkpoint. The iteration closes on the checkpoint screen. Auto continues by itself; Checkpointed and Measure wait for you.

When the run ends, Fontana deletes the remaining private copies. The trial conversations are kept with the run, because the charts read from them.

The checkpoint screen

The top of the screen shows the run’s status and why it stopped, four totals (trials, iterations, spend, best objective score) and two panels:

• Quality against cost chart. One bubble per candidate and model: quality on the vertical axis, cost on the horizontal axis, and latency as the bubble size. The baseline is hollow and the recommended configuration is ringed. Hover a bubble to see its configuration and the spread of its repeats. The front is the set of configurations that no other configuration matches or beats on quality, cost and latency together; it is drawn from the raw measurements, so you can weigh it differently from the objective.
• Recommendation. The objective in one line (with Edit objective while the run is paused) and the recommended configuration in three, with Apply, Run more experiments and Cancel.

Below them, the iteration timeline lists what each candidate changed, why the Eval agent proposed it, any refusals, and how the objective moved. The trials table lists every trial; expand a row to see its usage charts (prompt, completion and cached tokens, cost and latency per step, in the style of the chat usage panel), its sub-agent tree, a histogram of its tool calls, the judge’s score and reasoning per criterion, and, for workflow tasks, the workflow before and after the trial with the changed nodes marked.

Amounts follow your Show token cost preference: when it is off, the chart plots tokens instead of cost and the spend totals are hidden.

Apply

Apply writes one configuration back to the originals. A confirmation lists every file it will change, with a before and after view of each. Fontana then writes the agent’s fields (model, reasoning effort, call settings, instructions, abilities, skill grants), the changed skill files and the reference files, skill namespaces first and the agent’s namespace last. Each namespace is written in one step, and only if its files are unchanged since the run started. When a namespace has changed, Apply stops there and reports the conflict, so the agent itself is written only after every skill namespace succeeded. The turn budget a trial ran with is shown on the recommendation and is not written by Apply.

Apply needs evals:write and agents:write, and editor access to every namespace it writes. Agents in the system namespace cannot be changed by Apply. Each Apply is recorded in the activity log.

Run more experiments

Run more experiments continues the same run from the checkpoint, optionally with a note to the Eval agent (“try a smaller model with a higher turn budget”). The private copies are rebuilt from the recorded changes. Only the person who started a run can continue it.

Run a test from chat or an external client

An agent that grants the Evals page skill can set up and run a test with tools, in Flow chat or from an external client such as Claude Desktop connected to the Fontana MCP URL.

Tool	What it does
evals_listTests	Lists the tests you can read, with their ids
evals_startRun	Starts a run of one test in the mode its file sets, and returns a run id
evals_getRun	Returns the status, the trials table, the recommendation, and the report
evals_listRuns	Lists the latest runs of one test
evals_continueRun	Runs more experiments on a paused run, with an optional note for the eval agent
evals_cancelRun	Cancels a running or paused run

A test is created or edited by writing its EVAL.md with the Fontana Files tools. The skill’s reference file references/eval-md.md describes every field. A test id is <namespace>/evals/<slug>.

A run works in the background for minutes, so the agent reads it again until it is paused, done, failed, or cancelled. Apply stays on this page, where you see what it will write before you confirm.

Choosing an eval agent

Fontana ships an Evals Agent for all three roles: judge, objective and optimiser. It runs at temperature 0 with no tools and returns structured results rather than prose; the harness, not the Eval agent, applies its proposals to the copies. To use a different agent, open AI → Agent packages, pick the agent, and give it the Eval judge availability. Agents that judge evals are hidden from the normal chat pickers.

Permissions

Evals use their own permission set, so you can let a team read results without letting them spend model credits on runs.

Permission	Allows
evals:read	Open the Evals page, read tests and past runs
evals:write	Create and edit tests; with agents:write, apply a result
evals:run	Start, pause, continue and cancel runs
evals:delete	Delete tests and their runs

Starting a run also needs read access to the agent’s namespace and to the Workspace. Trial detail (answers, files, workflow pictures and usage) is shown to people with read access to both. See Roles and RBAC for how these map to the built-in roles.

Where tests are saved

Each test is a package in a Fontana Files namespace that holds evals: the file evals/<test>/EVAL.md. New tests go to the evals namespace, which tenant editors can write. The folder name is made from the test’s name when you create it and stays the same when you rename the test. The Evals page lists the tests in every evals namespace you can read; in File System, an EVAL.md opens with a rendered preview and its source.

EVAL.md holds the settings as YAML frontmatter and the task, criteria and goal as three sections:

---
name: Refund policy answer
agent: support/agents/support-agent
evalAgent: system/agents/evals-agent
workspace: acme-support
workflow: workflows/returns.wf.json
models: { mode: default }
searchSpace: { models: true, modelSettings: true, instructions: true, skills: true, abilities: true, files: true }
run: { mode: checkpointed, iterations: 3, candidates: 3, repeats: 1, parallel: 2, maxSpendUsd: 10 }
---

## Task
How long do I have to return a product?

## Acceptance criteria
- Quotes the 30-day return window
- Names the form the customer must fill in

## Goal
The cheapest configuration that still scores 8.

Settings you leave out take the defaults above, and every search-space switch defaults to on. The Raw view on AI → Evals edits this file directly and checks it before saving; an unknown key or section is refused with the reason. Saving needs evals:write and editor access to the namespace; deleting a test needs evals:delete and removes its runs. You can also write EVAL.md with the Fontana Files tools. If a saved file cannot be read, the test keeps its last good settings and the error shows on AI → Evals.

What runs are kept

A run keeps the test’s task, criteria and settings as they were when you started it, every trial with its scores, usage and evidence, the objective, the iterations, the chart data and the report, so history stays readable after you edit the test. Eval conversations are saved to your threads namespace on Fontana Files like your other conversations, do not appear in your chat history, and are removed with their run. Because test packages are files, they are included in backup and restore; runs are not, and the configurations you applied live on in the agent and skill files they changed.