Skip to content

Evals & observability

kin verifies itself with two eval tracks: a deterministic regression sentinel that runs at commit time (task verify, no model, no network — one leg of task ship, the local promotion gate) and a live golden-set gate that drives the real app against a real endpoint before a promote, run by hand rather than as part of task ship. Alongside them sits an opt-in JSONL span sink that records OTel-GenAI-shaped spans for every turn, model call, and tool invocation. See Testing for how both fit into the wider pytest / live-script layout.

The regression sentinel (task verify-evals)

tests/test_evals.py runs a pinned set of scenarios on the scripted FakeBackend — the same scenario discipline the live driver uses, minus the wire — and compares each scenario's deterministic fingerprint against the checked-in baseline at snapshots/evals_baseline.json:

  • the collapsed event-type sequence (streaming runs collapse to one token, so chunk-size changes don't fire it)
  • the ordered tool-call names, tool errors, and approval requests
  • the terminal shape (done:stop / done:error / done:loop_detected — the doom-loop guard's no-progress stop), error counts, and the retry (redrive) flag
  • parallel-batch stamping (batch_id on concurrent tool calls)
  • the span names the span sink derives for the scenario

Any drift is a named REGRESSION with a field-level diff. Only deterministic code graders gate here — stochastic signals (pass^k, model judgment) are deliberately kept out of the commit path and live in the live gate instead, where they are reported, never blocking.

task verify-evals                       # part of `task verify`
uv run pytest tests/test_evals.py --update-baseline   # re-pin after a REVIEWED contract change
git diff snapshots/evals_baseline.json            # the diff IS the review

Changing a scenario script, an event shape, or a loop contract is a baseline change: re-pin in the same commit and review the diff, exactly like the SVG snapshot discipline.

The live gate (task live-drive-all)

scripts/live_drive_vllm.py drives every scenario through the real KinApp + a real endpoint (the pre-promote check). Two eval features layer on top of the per-scenario assertions:

Pinned baseline. After an --all run the driver reports PASS/REGRESSION against snapshots/live_baseline.json — a checked-in record of which scenarios passed on the last known-good run (plus its model, git sha, and informational wall times). A scenario that was pinned green and now fails is called out as REGRESSED; one that was already failing at pin time reports as known-fail so a pre-existing breakage isn't mistaken for a new one. Re-pin from a known-good run:

task live-drive-all -- --update-baseline

pass^k. Agents are stochastic — a single-shot pass/fail lies (τ-bench's lesson). --passk K re-runs each scenario K times in fresh workdirs and reports pass^k (all K attempts pass) per scenario, both in the summary table and in the JSON aggregate:

task live-drive-all -- --passk 3

pass^k runs are exploratory by design: the baseline compare is skipped and the result is a reported pre-promote signal, never a commit gate (the canonical gate stays K=1).

The span sink

Set KIN_SPANS=1 (or spans_enabled = true in settings.toml) and every session appends OpenTelemetry-GenAI-shaped spans — one JSON object per line — next to its journal:

<session dir>/<project>/<session_id>.spans.jsonl

Three span kinds, derived at the emit seam by harness/spans.py:

Span One per Named attributes
invoke_agent kin user turn (the root of its own trace) gen_ai.operation.name, gen_ai.provider.name, gen_ai.agent.name, gen_ai.conversation.id (the session id — the cross-trace link), gen_ai.request.model, gen_ai.response.finish_reasons, error.type
chat {model} model round inside the turn gen_ai.request.model, gen_ai.response.model (the discovered id), gen_ai.usage.input_tokens, kin.usage.cached_tokens
execute_tool {tool} tool / MCP invocation (subagent tool calls nest under the parent task span) gen_ai.tool.name, gen_ai.tool.call.id, gen_ai.tool.type, error.type

Attribute names follow the OTel GenAI semantic conventions (the semantic-conventions-genai repository, Development status). There is no OTel SDK and no OTLP exporter — the records are hand-emitted JSONL you can jq directly, feed to a collector later, or ignore.

Two properties are contractual:

  • Metadata only. No prompt text, no completion text, no tool arguments or results — timing, names, and token counts. The semconv's content-capture opt-in stays off, so the file is as safe to keep as the journal beside it.
  • Never in the way. The tap wraps the emit callable and swallows every internal error; an unwritable sink disables itself for the session. Spans can go missing — a turn can't break because of them.

Off by default. The toggle is project-safe (a local metadata file, no network / egress / secret) and human-only — the model can't flip it via propose_settings. See Environment variables and settings.toml keys.