Scheduled jobs¶

The Outpost dashboard can run kin unattended, on a schedule: each job fire is a fresh headless kin -p subprocess in a workspace, with the run's markdown report kept on disk and every outcome recorded in a durable run history. The scheduler lives inside the Outpost container — the dashboard's own 60-second tick over a WAL sqlite job store — and the dashboard's Scheduled jobs card (the Jobs tab) is the whole management surface. While a fire is in flight, the job row's status badge carries the same pulsing dot as the Activity feed plus a ticking elapsed counter (it reads "running · 42s") — on this dashboard, motion always means live work.

This lane runs enterprise-reliability discipline — deliberately

Unattended execution fails dangerously (a silently-skipped backup, a job double-running while its first run's side effects still land), so the scheduler ships with the full reliability kit: durable single-flight claims, coalescing catch-up, bounded retries with dead-lettering, hard process-group kills, and fail-loud-never-fail-quiet schedule errors. Every invariant below is regression-tested under tests/test_outpost/, with the live end-to-end proof in task live-jobs.

Creating a job¶

On the dashboard's Scheduled jobs card (the Jobs tab): a name, the prompt, a workspace, a schedule, and (optionally) mode / timeout / max retries / token budget (blank = unlimited; the per-run cap from What a fire actually runs) / message me on Matrix (a checkbox — when on, the run report is posted to your Matrix DM on terminal completion; see Messaging). An existing job's edit (pencil) button prefills this same form and Save then overwrites the job via PUT (rather than creating a new one). Three schedule kinds — no natural-language parsing:

Kind	Expression	Example
`every`	a plain interval — `Ns` / `Nm` / `Nh` / `Nd`, floor 60s	`30m`, `2h`, `1d`
`cron`	a 5-field cron expression (via `croniter`, from the `outpost` extra)	`30 6 * * *`
`at`	a one-shot ISO-8601 timestamp (must be in the future)	`2026-07-04T06:30`

Each job carries an IANA timezone (the form defaults to your browser's; the store defaults to UTC). Cron fires are computed in that zone; a naive at timestamp is interpreted in it; and the subprocess runs with TZ=<job tz>, so any "today's date" context kin injects follows the job's clock, not container UTC.

DST semantics (per croniter's arithmetic over the zone): a nonexistent spring-forward local time (e.g. 02:30 on the skip day) is passed over to the next valid fire; a repeated fall-back time fires on its first occurrence. Timestamps stored by the scheduler are always UTC ISO-Z.

What a fire actually runs¶

python -m kin.tui.cli -p --workdir <workspace> --mode <auto|strict> \
    --output <kin_home>/outpost/runs/<job>/<timestamp>.md \
    [--token-budget <n>] -- <prompt>

-p/--print is a boolean flag (headless mode), not the prompt's own switch — the prompt is always the trailing positional, placed after -- so a prompt that happens to start with a dash isn't misparsed as an option.

Mode auto is the default — inside Outpost, the container is the sandbox (KIN_SANDBOX=container), so shell runs unprompted. Mode strict makes the job report-only: every edit/shell call becomes a needs-human error.
The exit-code contract maps to the run status: 0 → ok, 1 → error, 2 → needs_human.
The run journal is saved like any session, so a needs-human run can be resumed in a terminal (kin --resume <id>, printed in the report).
The --output report is browsable from the card's per-job Runs list.
Per-run token budget — set the job's token_budget field and the scheduler passes it as --token-budget <n> on every fire (unset = the CLI's own fallback to KIN_TOKEN_BUDGET / the token_budget settings key, else unlimited); it caps the cumulative tokens a single fire may spend, so a runaway fan-out stops itself before it OOMs the box. The per-run leash is the Tier 2b lever; the cross-run GPU-fairness lever (Tier 2c, keeping Blake's interactive TUI responsive while scheduled jobs share the same vLLM) is the governor — see The governor below.
Auto-compaction defaults ON at 85% of the context window (KIN_COMPACT_THRESHOLD=0.85, stamped by the scheduler's child-env construction) — a scheduled child has no human to answer a context-overflow prompt, so without this a long-running fire used to just die with a backend error at the window ceiling. An operator override (set in the Outpost container's own env) always wins over the stamped default. The interactive TUI is unaffected — its default stays opt-in (compaction.threshold unset / None).

The provider pin — fail closed on model drift¶

At creation the job snapshots the current endpoint (base_url + model) and resolves the serve's concrete active model id (the first non-default entry of GET /v1/models — the default alias's identity at that moment). At every fire the scheduler re-resolves:

Positive mismatch → the fire is skipped (skipped_drift run + a delivery.send alert). A job written and tested against one model must not silently run against whatever got loaded later.
No answer (endpoint down, empty listing) → the job runs anyway; an unreachable endpoint surfaces as an ordinary run failure on the retry path, not a drift skip.
After a deliberate model swap, hit the job's re-pin button to adopt the new id.

The api_key is never snapshotted — the subprocess resolves credentials live from settings.toml (no secrets in jobs.db).

Each job subprocess also carries KIN_JOB_ID / KIN_JOB_NAME in its env. The end-of-run memory reflection pass reads them so repeated fires of the same job update one memory for that job instead of accumulating near-duplicates — the structural answer to memory-rot from scheduled runs. (Disable the reflection pass with memory_reflect = false / KIN_MEMORY_REFLECT=0.)

The governor — cross-run GPU fairness¶

The per-run token leash caps ONE fire. The governor is the cross-run lever: admission control over background runs (the Jobs tab + the v1 machine door) sharing the same vLLM as the live interactive TUI. It keeps a scheduled job from starving a live operator's session under KV-cache contention.

The governor is always present (an in-process module at container/dashboard/governor.py, consulted by the scheduler tick + the v1 submit pre-check). Its levers are live-tunable from the Configure tab's GPU fairness card (card-governor) and persist to <kin_home>/outpost/governor.json. The interactive TUI / Outpost terminal sessions are never governed — they're priority 0 by wire-absence and bypass the governor entirely.

Classes (assigned by code path, not configured)¶

Class	Source	Default vLLM priority
`interactive`	TUI sessions, Outpost terminal	0 (untagged — top tier)
`api`	v1-submitted jobs (and WP7 trigger-fired one-shots)	5
`scheduled`	Jobs-tab `cron` / `every` / `at`	10

Lower number = higher priority in vLLM's --scheduling-policy priority queue (untagged reads as 0/top-tier). Per-job pin via the optional priority field on POST /api/jobs / PUT /api/jobs/{id} overrides the class default (range 1..100; 0 is reserved for the TUI).

Levers (live-tunable from the card)¶

per-class max_concurrent (defaults api 2, scheduled 2): the per-class admission cap. Independent — api jobs filling the api cap don't block scheduled jobs (and vice versa). The global MAX_CONCURRENT_RUNS=4 stays the hard sum cap (the fork-bomb bound).
queue_depth_max (default 16): at jobs + v1 submits queue up to this depth instead of today's reject-at-cap, then 503 capacity-exceeded (the existing v1 taxonomy — additive refinement, NOT a contract change).
per-class tokens_per_hour (default 0 = unlimited): a rolling 60-minute window over completed-run tokens_in + tokens_out. Trailing leash — measured on completion, so a burst that lands inside the window is allowed; the NEXT admission is held until the window slides. Document honestly: this is not a hard cap on a single run.
defer_when_interactive (EXPERIMENTAL, default off): poll vLLM /metrics vllm:num_requests_waiting; hold background admissions while the queue is at/over the threshold (default 2). Fail-open if metrics are unreachable (a metrics outage must not brick the scheduler).

Overload semantics¶

Source	Cap hit	Behavior
recurring (`every` / `cron`)	any lever	Skip + advance — record a `skipped` run with the reason (visible in Activity), advance `next_run_at`.
one-shot (`at` / v1 / trigger)	any lever	Hold + queue — leave `next_run_at` untouched so the next tick re-tries when a slot frees; v1 submit also surfaces 503 once `queue_depth_max` is reached.

No governor-level preemption of running work — GPU-level preemption is vLLM priority scheduling's job via the wire-flip (next section).

The wire-flip — `KIN_GPU_PRIORITY` (the master switch)¶

The vLLM serve must run with --scheduling-policy priority (the per-class priorities have no effect otherwise, and vLLM 400s on a non-zero priority without it). That flag is DONE on kin-one — added 2026-07-05 to all four model dirs under kin-one:~/inference/vllm/models/ and verified (untagged OK, priority-tagged accepted). The remaining switch is the container env.

KIN_GPU_PRIORITY on the Outpost process env is the master switch (set it on the container, not per-job). When truthy (1, on, true, yes, or any positive int), every background-job subprocess is launched with both:

KIN_PROVIDER=openai — the wire flip. The Anthropic /v1/messages protocol has no priority field (verified against vLLM's entrypoints/anthropic/protocol.py), so a priority set on the Anthropic wire is silently dropped server-side. The OpenAI /v1/chat/completions wire carries it for real, so the flip is load-bearing.
KIN_REQUEST_PRIORITY=<per-run value> — the priority tier, now resolved per-run by the governor (job.priority pin or the class default — api 5, scheduled 10), NOT the KIN_GPU_PRIORITY value directly.

0 / off / false / no / unset → dormant (no flip). The dormant-by- default path is what made the governor safe to land ahead of the serve change. Deploy order matters: flip KIN_GPU_PRIORITY AFTER the vLLM serve flag is in place, or every background job 400s on its first request (fail-loud, but a wasted fire).

What you trade away (the wire-flip)¶

Background jobs on the OpenAI wire lose the Anthropic-wire features that drove kin's Anthropic-default decision — parallel-tool-safe content blocks, real count_tokens, thinking-signature replay (see src/kin/harness/backends/factory.py:142-146). Background jobs are typically less interactive than the TUI; the quality hit is bounded and observable. If it proves material, the long-term fix is upstream — a ~20-line PR to vLLM adding priority to AnthropicMessagesRequest.

Fairness within a tier¶

FCFS within a class (a documented lever for later: round-robin by job id). The governor does not solve:

Multiple interactive TUI sessions competing — two live TUIs both run priority-0 and FCFS between them; the interactive-vs-batch split is the scope.
CPU/RAM contention on the outpost box — only GPU KV-cache scheduling. The Docker memswap_limit / pids_limit caps handle the rest.

Verifying it works¶

After enabling, watch a scheduled fire under concurrent TUI load — interactive latency should stay flat (no head-of-line blocking) and the job's usage should show KV-cache reuse / preemption jitter (the job runs slower, doesn't fail). task live-drive with a scheduled job firing concurrently is the live-wire proof; the headless tests/test_outpost/ suite can't see this because run_job is monkeypatched to a fake there. See docs/decisions/0026-gpu-governor.md for the live TTFT numbers (OFF vs ON) that validated the design.

Reliability semantics¶

Catch-up coalesces to one fire (the systemd Persistent=true shape). If the box was down across one or many missed fires, the job fires once at the next tick and the following fire is computed from now — never once-per-missed-interval, never silently skipped. Restart-survival is just the WAL store re-armed at startup; compose's restart: unless-stopped is the process supervisor, which is why there is deliberately no systemd unit to install.
Single-flight, skip-not-queue. A job that comes due while its previous run is still going records a skipped run and waits for its next slot (the K8s CronJob Forbid / Temporal Skip convergence). The claim is a durable running row, so a crash can't double-run: at startup, orphaned claims are resolved — a still-alive child is process-group-killed (killed), a gone one marked lost — and the schedule resumes.
Retries: fixed backoff 60s → 120s → 300s, bounded by the job's max_retries (default 3). Any exit-1 run (and any timeout — it counts as one attempt) retries. Dead-letter: once consecutive failures exceed max_retries the job is disabled with a visible reason and a delivery.send alert — disable-and-alert, never infinite retry. Re-enable from the card after fixing the cause (this also resets the failure count).
needs_human is terminal, never retried — a human gate is not a transient error. You get a high-priority notification; the schedule keeps going; resume the saved session to answer.
Timeouts kill hard. The subprocess leads its own process group; at timeout_s the whole group gets SIGKILL. No soft cancel — a half-cancelled run retrying while its side effects still land is how unattended schedulers double-execute.
Schedule errors fail loud. A cron job whose parser is unimportable (or whose expression stops computing) is disabled with a reason and an alert — never silently treated as complete.

Alerts (drift-skip / needs-human / dead-letter) ride the same delivery.send seam as session alerts — see the messaging guide for the operator setup.

No recursive scheduling — bounded by structure¶

Job CRUD exists only in the dashboard API (/api/jobs). There is deliberately no harness tool for schedule management, so a scheduled agent can't schedule via a tool call. One runaway job can waste a night; a job that can schedule jobs can waste a fleet.

That said, this is not a hard containment wall: the dashboard binds loopback and its CSRF token is forgeable by a process inside the box, so a prompt-injected auto-mode agent can reach the API directly. The real fork-bomb defense is a pair of structural caps that make a create burst terminate regardless of source — a hard ceiling of 100 total jobs (POST /api/jobs returns 409 past it) and 4 concurrent runners (over-cap due jobs record a skipped run and coalesce to the next tick). Tune both in container/dashboard/{jobs,scheduler}.py if your box is bigger.

API surface¶

All mutations are CSRF-gated like the rest of the dashboard.

Route	What
`GET /api/jobs`	jobs, newest first, latest-run status blended in
`POST /api/jobs`	create (`name`, `prompt`, `schedule {kind, expr, tz}`, `workdir`, `mode?`, `timeout_s?`, `max_retries?`, `token_budget?`, `priority?` (1..100, the governor pin), `output_surface?` (`"messenger"` or `""`), `base_url?`, `model?` — the last two default to the current endpoint snapshot)
`PUT /api/jobs/<id>`	partial update (`name`, `prompt`, `schedule`, `workdir`, `mode`, `timeout_s`, `max_retries`, `token_budget` / `priority` — int or null to clear, `output_surface`); `{"enabled": true}` re-enables + clears dead-letter state; `{"repin": true}` re-resolves the model pin
`DELETE /api/jobs/<id>`	delete (run history cascades; reports on disk are kept)
`POST /api/jobs/<id>/trigger`	manual fire — same single-flight guard (409 while a run is in flight)
`POST /api/jobs/<id>/cancel`	kill switch (DR 0071) — SIGKILLs the run's process group if it's live, or disables it before it fires if it's still queued; `409` when there's nothing currently in flight to stop, `200 {"status": "cancelled", "cancelled_at": ...}` otherwise (idempotent — a repeat call echoes the original timestamp)
`GET /api/jobs/<id>/runs`	recent runs with status / exit code / attempt
`GET /api/jobs/<id>/runs/<run_id>/output`	the run's markdown report

The kill switch (DR 0071). A Stop button appears on the run-detail page (/runs/<job_id>/<run_id>) whenever that run is queued or running — confirm ("SIGKILLs the process group. A workspace auto-snapshot, if enabled, was taken at run start.") then it posts to /api/jobs/<id>/cancel and reloads. This is the SAME cancel core the v1 machine door's POST /api/v1/jobs/<job_id>/cancel (PROTOCOL.md §8) and the Matrix stop <job_id> verb (messaging guide) both call — three thin handlers, one shared semantics function (v1_jobs.cancel_job_core), so "stop" behaves identically no matter which door you use. stop only ever kills a run — a recurring job's schedule is untouched (that's what pause is for); calling it when nothing is currently in flight (a healthy cron job idle between fires, a job that already finished) is a no-op 409, not a schedule change.

Recipes — the gallery at the top of the Jobs tab¶

Recipes are templates, not execution. A recipe is a checked-in JSON file under container/dashboard/recipes/ that bundles the prompt template, its binding (schedule or trigger), the config fields the operator fills in at enable time, and the output surface (report, messenger, or github_pr_comment). Enabling a recipe creates a real scheduled_jobs row (and, for trigger-bound recipes, a triggers row) through the same machinery a hand-authored job uses — the scheduler's tick is unchanged; the recipe-tagged job runs identically. A hand-authored job can tag itself messenger too (the message me on Matrix checkbox in the create form) — recipes just pre-set the same column for a curated template.

Six recipes ship in the catalog, all written for Qwen (imperative voice, structured-output-explicit, no Claude-isms):

Recipe	Binding	Output	What it does
`morning-pr-review`	cron 09:00 UTC	messenger	Lists open PRs across one repo, groups by review-needed / stale / unowned, suggests the first action.
`daily-brief-9am`	cron 09:00 UTC	report	One-pager for the morning: open inbox items, overnight activity, suggested focus.
`standup-from-git`	cron 09:00 UTC	report	Yesterday's commits + in-flight branches + dirty tree + today's likely focus.
`weekly-repo-health`	cron Sun 18:00 UTC	report	Weekly health check: open/closed counts, stale PRs, recent CI failures, recommended actions.
`daily-kin-digest`	cron 17:30 UTC	messenger	End-of-day digest across every workspace (terminal sessions + scheduled fires + trigger-driven runs).
`issue-triage`	trigger (github issues.opened)	github_pr_comment	Trigger-bound — a new issue fires it; the agent reads the body, classifies bug/feature/question/docs, applies a label, posts a triage comment.

How the gallery works¶

GET /api/recipes returns the catalog + per-recipe enabled counts + any load_errors (files that failed validation). Bad files are EXCLUDED from the valid catalog and surface on the gallery's load-error badge; the real fail-loud gate is tests/test_outpost/test_recipes.py:: test_bundled_recipes_valid (every bundled file must validate — task verify fails on any invalid bundled file).
POST /api/recipes/{id}/enable (CSRF-gated) validates the operator's config against the recipe's config_fields, renders the prompt through the same restricted {{ }} renderer WP7 uses for triggers (every value sanitized + capped; missing key → RecipeError), and creates a real scheduled_jobs row with recipe_id / recipe_version / output_surface stamped. For trigger-bound recipes the response also carries the one-time trigger secret + per-source setup instructions (mirrors WP7's show-once discipline).
GET /api/recipes/enabled lists every recipe-tagged job.

Version drift — frozen prompts¶

An instantiated job KEEPS its rendered prompt for the lifetime of the row. A future change to the recipe's prompt_template produces a recipe: <id> (updated since v<N>) badge on the job row (warn colour) and the gallery's enabled counter reads as "stale"; the operator can re-enable (creates a new job) but the running job never silently rewrites itself. The recipe author can't break live instances by editing the recipe — the schema is the contract.

Output surfaces¶

report — the run shows up in the Activity feed + the run page. The default; no extra wiring.
messenger — on a terminal completion (ok / needs_human), the scheduler's post-run hook fires delivery.send(kind="run_report") so the Matrix bot (and any future adapter) receives title + ≤1500-char sanitized report head + run deep link. The hook only fires for output.surface == "messenger" jobs; non-recipe jobs skip it.
github_pr_comment — prompt-level. The recipe's prompt embeds the exact gh pr comment / gh issue comment instruction (the agent does the rest). No post-run hook. The bundled issue-triage recipe uses this surface.

Atomicity (trigger-bound recipes)¶

A trigger-bound enable creates BOTH the trigger row AND the scheduled_jobs row in one call. The trigger's workspace is the operator's workspace config field (must be in the service's allowed_workspaces ACL — same fail-closed re-check the WP7 receiver does at fire time). If anything between the two creates fails (bad service_id, cap hit, etc.), the trigger row is soft-disabled so the operator can clean it up; the recipe error surfaces as a 400.

Verifying¶

tests/test_outpost/test_jobs.py (in task verify) drives the schedule math, the coalesce/skip/retry/dead-letter/drift state machine (with fake runners), the API round-trips, and the CSRF gates headlessly. task live-jobs is the live proof: it fires real kin -p subprocesses against a real endpoint and asserts the ok / needs-human / timeout run rows end to end.