Practice · Answer evaluation

Build answer fixtures before testing a model

Use a source-aligned answer and deliberately altered fixtures to check an evaluator’s decisions before measuring model outputs.

Jason Colapietro · Suede AI SEO ·

Give the evaluator a known answer

An answer evaluator needs acceptance criteria you can inspect. The Suede example uses the public credential inventory as its reference and four hand-authored answer fixtures as inputs. One matches the source. Three deliberately change a category total, remove a source URL or rewrite the relationship.

The dated run records eight checks per fixture and four expected decisions matching the actual accept-or-review decisions. That result tests the evaluator on these saved examples. The runner uses zero paid model calls, so its output is a useful starting point before you collect responses from a model.

Write criteria from the source fields

Open evaluateAnswer in checks.mjs. It checks the holder, course total, training achievement total, membership total, Microsoft award types, provider attribution, source URL and relationship. Each result has a name and a Boolean decision. A reviewer can therefore identify the exact failed condition.

The source-aligned fixture is accepted only when all checks pass. The deliberately conflated fixture fails three checks: course total, training achievement total and Microsoft award types. Keeping the check names in the output makes the failure useful for correcting the answer or refining the input contract.

Run the fixtures on their own

Use this focused command from the suede-seo repository. It reads the same fixtures and inventory used by the complete demonstration. Inspect the fixtures before changing them so the intended decision remains separate from the evaluator’s actual output.

node --input-type=module <<'JS'
import { readFileSync } from 'node:fs';
import { evaluateAnswer } from './examples/training-proof/checks.mjs';
const read = p => JSON.parse(readFileSync(p, 'utf8'));
const facts = read('site/assets/credentials/credentials.json');
for (const fixture of read('site/assets/demonstrations/answer-fixtures.json')) {
  const checks = evaluateAnswer(fixture.answer, facts);
  console.log(fixture.name, checks.every(c => c.pass), checks);
}
JS

Add cases that expose an actual gap

This evaluator expects a structured answer. It is a small, readable rubric for this inventory. Its provider check accepts a non-empty subset of known providers, so it does not enforce a complete provider list. That distinction belongs in your acceptance criteria if completeness matters.

Before expanding it, write the new expected behavior and add a fixture that currently fails. For example, a workflow requiring every issuer should compare the required set explicitly. A free-text answer needs an extraction or review step before these structured fields can be checked.

Move from authored fixtures to model responses

Use the published credential-answer-prompt.md as a starting input. Retain the model identifier, settings, complete prompt, date and original response when you collect a model answer. Run the evaluator against the response and inspect failed checks before summarizing the sample.

Building with the Claude API, Craft Precise Prompts for AI Models and Academy Accreditation - Generative AI Fundamentals are the issued training records connected to this exercise. The practical deliverable is a reference set, a readable rubric and a repeatable decision. Claude’s evaluation guidance is a primary reference for defining measurable criteria; your application supplies the actual cases and success threshold.

Issued training connected to this exercise

Inspect the full training inventory · Open the qualifications sheet

Source and recorded result

The demonstration report is dated 2026-10-21. Repeat the command against the revision you intend to inspect and retain the resulting artifact.

Put the workflow to work

Discuss an evaluation workflow. Bring the workflow, source page or answer your team needs to check so the first deliverable has clear acceptance criteria.

Check credential counts and relationships · All three working demonstrations