Testing¶

The kin-textual test surface is split into two homes:

tests/ — the pytest suite. Headless, no model needed, no real network (most files). Runs under task verify. This is where you add tests for harness internals, headless flows, and the like.
scripts/ — live / smoke / probe scripts. Real model, real network, real subprocess. Runs under task live-drive*, task smoke*, task probe-mcp*. Not pytest; they stay as CLI scripts because their assertion surface doesn't fit a unit-test frame.

The split matches the project character: most of what we test is harness internals (pytest is the right tool); a small but important slice is real-wire behavior (CLI scripts are the right tool).

Pytest layout¶

tests/
├── __init__.py                 # env-before-import gate (sets KIN_HOME etc.)
├── conftest.py                 # shared fixtures + helpers
├── test_settings.py            # headless settings
├── test_banner.py              # headless banner
├── test_integration.py         # end-to-end with FakeBackend + KinApp
├── test_workflow.py            # the 17 closed-namespace / AST-filter tests
├── test_search.py              # FTS5 workspace search
├── test_memory.py              # the memory_20250818 command set
├── test_provider_presets.py    # the first-party preset system
├── test_evals.py               # the regression sentinel (--update-baseline)
├── test_web.py                 # web_fetch SSRF + Brave formatting
├── test_office_security.py     # docx/xlsx read_file: zip-bomb cap + entity-expansion guard
├── test_cli.py                 # `kin --version` + `kin doctor` (install/config/network chunks)
├── test_harness/               # the big split subpackage (~50 files, 780+ tests)
├── test_app/                   # the UI subpackage (~23 files, 330+ tests)
├── test_outpost/               # the outpost dashboard (requires --extra outpost)
├── test_pty.py                 # the out-of-process PTY smoke
├── test_snapshot.py            # the SVG baseline guard
├── test_worktrees.py           # worktree isolation
├── test_perf.py                # synthetic journal perf benchmark
└── test_live_drive.py          # thin wrapper over scripts/_live_drive_scenarios

Most top-level test_<area>.py files correspond 1:1 to a historical verify_<area>.py script they migrated from (Phases 1-5, 2026-07-02) — see the per-task table below. test_office_security.py and test_cli.py are newer, added directly as pytest with no verify_*.py ancestor. The split subpackages (test_harness/, test_app/, test_outpost/) replaced the monolithic verify_harness.py / verify_app.py / verify_outpost.py files (~12K + ~4K + ~9K lines each) with focused submodules.

Adding a tool, a permission kind, or a subagent profile all need matching coverage under tests/test_harness/ — see Extending kin for where each of those lands.

The local promotion gate¶

task ship is the contract before a dev → main merge — see AGENTS.md's Commands table for the day-to-day command surface and Multi-machine dev setup for running it on a second machine (task kloud:verify). No CI server, no GitHub Actions — Blake + AI peers run it as the explicit verification step. The shape:

task ship       # = check + verify + docs-build + docs-audit + memory-check + code-audit + live-drive-dry

Sub-command	What it checks	Wall time
`task check`	ruff + ty + deptry (lint, typecheck, dependency hygiene)	~5s
`task verify`	headless pytest (`uv run pytest tests/ -m "not slow and not live"`)	~50s
`task docs-build`	mkdocs --strict (no broken internal links)	~3s
`task docs-audit`	decision-file convention + SOURCE/orphan drift (fail-green; only convention breaks block)	~1s
`task memory-check`	no stale deleted-root-doc references in the memory tree	~1s
`task code-audit`	no `file:line` cites in code comments/docstrings + no duplicate module-level class defs in a package (DR 0055)	~1s
`task live-drive-dry`	network-free wire smoke (dummy backend at localhost:1)	~6s

If any sub-command fails, the merge is blocked. The contract: "I ran it, here's the output" — not "looks right."

task ship is also wired into the project's CLI surface for ad-hoc use (the same task target, no separate workflow).

Markers¶

Three markers route tests out of the default task verify pass:

Marker	Meaning	Run with
`slow`	machine / perf / Textual-version sensitive	`task test-slow`
`live`	needs real network / real endpoint	`task test-live`
`outpost`	needs the `outpost` extra	the `task _verify-outpost` sub-pass inside `task verify`

The default task verify runs everything that ISN'T marked — i.e. the hermetic headless suite. The slow / live / outpost suites are run on-demand via their dedicated task test-* entry points (or by directly passing -m slow / -m live / -m outpost to pytest).

Conftest surface — fixtures + helpers¶

tests/conftest.py is the shared surface. Every test file imports from it. The headline wins:

Fixtures (function-scoped unless noted)¶

temp_kin_home(tmp_path, monkeypatch) — per-test KIN_HOME override with settings.reset_cache() so cached lookups from prior tests don't bleed through. Use this in any test that mutates KIN_HOME.
temp_session_dir(tmp_path, monkeypatch) — per-test KIN_SESSION_DIR override. Rare; mainly for tests that exercise persistence paths.
fake_backend — a fresh FakeBackend with an empty responder. Pair with the make_responder(...) helper to script turns.
session — a real Session wired with a captured-events list + the FakeBackend. Tests assert on the event stream via session._captured_events.
pilot_app — fresh KinApp per scenario, wired with the FakeBackend via app.run_test(). Yields (app, pilot, sess, captured_events). CRITICAL gotcha: if you do monkeypatch.delenv("KIN_HOME", raising=False) to clear env keys, it UNDOES temp_kin_home's monkeypatch.setenv("KIN_HOME", tmp_path) and the test falls back to ~/.kin (the user's real home). Skip KIN_HOME in any env-clear loop. This bug bit 3 of the original 8 migrations in Phase 2.
env_snapshot(monkeypatch) — capture-and-restore helper for tests that mutate env keys.
_reset_env_between_tests (autouse) — snapshots os.environ at the start of every test, restores it on teardown, AND calls kin.harness.settings.reset_cache() so cached kin_home() lookups from earlier tests don't bleed through. No opt-out by design — the cost of a surprise env leak is much higher than the cost of an opt-out. Tests that legitimately need to mutate env use monkeypatch (auto-restoring) or env_snapshot (manual restore). Cost: ~1ms per test.

Helpers (module-level functions, used inside test bodies)¶

import_tool_module(name) — importlib.import_module(...) alias. Used to bypass kin.harness.tools.__init__'s tool-instance shadow when monkeypatching tool internals. Modules using it: web_fetch, web_context, _media, workflow, etc.
async settle(pilot, predicate, *, tries=80, sleep=0.05) — poll a predicate until truthy. Lifted from verify_integration.py:38 and live_drive_vllm.py:194.
async wait_for_event(events, predicate, *, timeout=2.0) — poll a captured-events list for an event matching the predicate.
clean_terminal(text) / clean_interrupted(text) — strip ANSI / control sequences from a Textual screen dump.
_git_short_sha(cwd=None) — current git short SHA, or "unknown". Duplicated in verify_evals.py:404 and live_drive_vllm.py:1946 historically; consolidated here.
assert_eventually(condition, *, timeout, interval, message) — sync version of settle.
make_responder(*scripted_turns) — build a FakeBackend responder from a list of (text | tool_call | (text, [tool_calls...])) tuples.

Env-before-import architecture¶

tests/__init__.py (NOT conftest.py) sets the KIN_HOME / KIN_SESSION_DIR / KIN_SPANS env vars at MODULE TOP. This is critical because:

Several verify scripts (now test modules) do top-level from kin.harness import presets, settings.
settings.kin_home() is read at first import, not lazily.
Pytest imports tests/__init__.py before any test module — guaranteeing the env is correct for every test module's top-level kin imports.

setdefault (not =) is used so a developer's explicit env wins for local debugging. The conftest session fixture does the teardown (rmtree of the tmpdir).

Do NOT add from kin.harness... at the top of tests/__init__.py itself; that would defeat the ordering guarantee.

The `importlib` shadow-bypass idiom¶

kin.harness.tools.__init__.py re-exports tool INSTANCES, which shadow the submodule names. from kin.harness.tools import web_fetch gives you the tool INSTANCE, not the module. Tests that monkeypatch (e.g. wf._validate_ip, wc.brave_get) need the REAL module object:

# DON'T:
from kin.harness.tools import web_fetch   # returns the Tool INSTANCE
wf = web_fetch
wf._validate_ip = lambda ip: None        # mutates the tool instance, not the module

# DO:
wf = import_tool_module("kin.harness.tools.web_fetch")
wf._validate_ip = lambda ip: None        # mutates the real module

The import_tool_module(name) helper consolidates this idiom in one place. It also covers kin.harness.tools._media (the media submodule that tests/test_web.py needs) and kin.harness.tools.workflow (the workflow executor module that tests/test_workflow.py needs).

Coverage interpretation¶

Lazy SDKs (openai, anthropic, mcp, playwright, tomlkit, etc.) are imported inside the code that needs them. Headless tests using FakeBackend never trigger OpenAIBackend / AnthropicBackend imports, so the SDK modules show up as "no statements" in the coverage report — NOT "missing" or "untested".

Don't interpret low coverage on a lazy-imported module as a regression. Use pytest --cov-context=test for per-test context if a specific module needs diagnosing.

Run task test-cov to see the coverage report:

task test-cov    # = uv run pytest tests/ --cov=src --cov-report=term-missing \
                 #       -m "not slow and not live"

There's NO --cov-fail-under threshold — coverage is visibility, not enforcement. The lazy-SDK interaction makes a meaningful threshold premature (most of kin.harness.backends.* would show 0% under headless tests using FakeBackend).

Adding a new test¶

Pick the right home. Headless? → tests/. Live / smoke / network? → scripts/.
Use the shared fixtures (temp_kin_home, fake_backend, session, pilot_app) instead of hand-rolled env mutation.
Use assert instead of check() — pytest captures failures natively and the test function name becomes the test ID.
Preserve the assertion semantics. Same checks, same expected values — just pytest-shaped.
Add an if __name__ == "__main__": guard at the bottom if you want ad-hoc python tests/test_<area>.py debugging.
If your test is slow / live / outpost, mark it with the appropriate marker so the default task verify skips it.
Don't edit tests/conftest.py if you're working in a parallel branch / migration; add the fixture locally in your test file. The shared conftest is the only one that needs the KIN_HOME / settings reset dance, so adding fixtures there is rarely necessary.

Isolation guarantees¶

The autouse _reset_env_between_tests fixture (see Conftest surface above) closes the 4 known isolation failures from the migration — they now pass in the full suite, not just in isolation. The fixture is the contract: every test starts with a clean os.environ and a fresh kin.harness.settings._cache. No opt-out by design — the cost of a surprise env leak is much higher than the cost of an opt-out.

Tests that need to mutate env use monkeypatch (auto-restoring) or env_snapshot (manual restore). Module-top os.environ[...] = ... mutations are the original sin: pytest collects modules before running anything, so module-top mutations persist for the entire session. The pytest migration removed the ones it found in test_evals.py; the autouse fixture is the backstop for any that slip in later.

Per-task pytest invocations¶

Old `verify_*.py`	New pytest task
`verify_settings.py`	`task verify-settings` (`uv run pytest tests/test_settings.py`)
`verify_banner.py`	`task verify-banner` (`uv run pytest tests/test_banner.py`)
`verify_integration.py`	`task verify-integration` (`uv run pytest tests/test_integration.py`)
`verify_workflow.py`	`task verify-workflow` (`uv run pytest tests/test_workflow.py`)
`verify_search.py`	`task verify-search` (`uv run pytest tests/test_search.py`)
`verify_memory.py`	`task verify-memory` (`uv run pytest tests/test_memory.py`)
`verify_provider_presets.py`	`task verify-provider-presets` (`uv run pytest tests/test_provider_presets.py`)
`verify_evals.py`	`task verify-evals` (`uv run pytest tests/test_evals.py --update-baseline`)
`verify_harness.py`	`task verify-harness` (`uv run pytest tests/test_harness/`)
`verify_app.py`	`task verify-app` (`uv run pytest tests/test_app/`)
`verify_outpost.py`	`uv run --extra outpost pytest tests/test_outpost/` (inside `task verify`'s second pass)
`verify_outpost_live.py`	retired — `tests/test_outpost_live.py` + `task verify-outpost-live` tested the ttyd/tmux browser-terminal surface, deleted wholesale in the middle-way plan's Step 3 ttyd cut (2026-07-06, DR 0033)
`verify_pty.py`	`task verify-pty` (`uv run --group pty pytest tests/test_pty.py -m slow`)
`verify_snapshot.py`	`task verify-snapshot` (`uv run pytest tests/test_snapshot.py -m slow`)
`verify_worktrees.py`	`task verify-worktrees` (`uv run pytest tests/test_worktrees.py -m live`)
`verify_perf.py`	`task verify-perf` (`uv run pytest tests/test_perf.py -m slow`)
`verify_web.py`	`task verify-web` (`uv run pytest tests/test_web.py`)
`live_drive_vllm.py`	stays a CLI script; thin pytest wrapper at `tests/test_live_drive.py`

verify_harness.py was kept as a backward-compat shell through the migration (Phases 1-5) so existing cron / muscle-memory invocations kept working while the pytest files landed; it's now removed (the pytest collection is the canonical entry point).