Diagnostics: codes, doctor and the live view¶
WebCortex 2.5 answers three questions an operator — or a coding agent — asks when something is wrong:
- Is it set up right?
webcortex doctor, before it runs. - What exactly failed, and what do I do? A stable error code and a hint on every failure, wherever it surfaces.
- What is the running app doing?
GET /_webcortex/status,webcortex status, and thewebcortex_statusMCP tool.
Error codes and hints¶
Every error that reaches a user carries a code from one catalogue and a hint saying what to
do. The existing fields are unchanged; code and hint are added beside them, so a client
written for 2.4 keeps working. The one exception is a request that is not valid HTTP/1, which
hyper answers itself with an empty 400, 414 or 431 (see
Error codes).
{"error": {"status": 423, "message": "actuators are halted (person in the cell); …",
"code": "WC502",
"hint": "The emergency stop is engaged, so nothing that moves hardware runs. An operator releases it with `webcortex release` or POST /_webcortex/release (admin scope)."}}
The code is also sent as the x-webcortex-error response header, so a proxy or a log pipeline
can count failures by kind without parsing bodies.
| Range | Area |
|---|---|
WC0xx |
internal faults |
WC1xx |
configuration, boot and the environment |
WC2xx |
routing, HTTP and request input |
WC3xx |
agents, behaviours, flows and model providers |
WC4xx |
budgets and limits |
WC5xx |
devices, actuators and the emergency stop |
WC6xx |
authentication and security |
The full list, with each hint, is the error code reference. A code keeps its
meaning once released: branch on WC402, not on the wording of a message.
Where codes appear¶
| Surface | Shape |
|---|---|
| HTTP error | {"error": {"status", "message", "code", "hint"}} and header x-webcortex-error |
| Agent run that did not complete | "code" and "hint" on the run result (failed → the failing step's WC3xx, budget_exhausted → WC402, step_limit → WC404, max_tokens → WC312); absent for completed and awaiting_approval |
| A failed step | "code" on the step: a refused tool WC307, a tool that answered with an error carries that error's code (a missing argument is WC206), a provider failure WC301–WC305 |
| MCP tool error | isError: true, the hint appended to the text, and structuredContent: {"error": {"code", "message", "hint"}}; JSON-RPC errors carry error.data.code |
| Watcher insight | "code" beside "error" when the run failed |
| WebSocket error message | {"type": "error", "message", "code", "hint"} on an open device or agent socket (since 2.5.1) |
| CLI | error[WC101]: … / hint: … on stderr, or {"ok": false, "error": {…}} with --json (and always for webcortex check), exit status 1 |
From your own handlers¶
HTTPError takes a code and a hint:
from webcortex import HTTPError
@app.post("/orders/{id}/ship", tool=True, scopes=["write"])
def ship(id: int) -> dict:
if not paid(id):
raise HTTPError(409, f"order {id} is not paid", code="WC214",
hint="Take payment with pay_order first, then ship.")
...
Without code=, the catalogued code for the status is used (409 → WC214, 422 → WC206,
…). The hint is what an agent reads to recover, so write it as an instruction.
Explaining a code¶
$ webcortex explain WC402
WC402 Token budget exhausted · limits
The run hit its token_budget (its own, or the request tree's shared one). Raise token_budget, use a cheaper model alias, or bound context with tool_result_limit / context_window.
$ webcortex explain --list
$ webcortex explain --list --json
On a running app, GET /_webcortex/errors returns the catalogue, and an MCP client can call
webcortex_explain or read webcortex://errors.
webcortex doctor¶
Checks an app before it runs, without starting it. Each check is
{id, status: ok|warn|fail, message, hint, code?}; the command exits 1 if any check failed.
$ webcortex doctor
ok python: Python 3.13.13 (GIL-enabled)
ok core: webcortex 2.5.0, core extension 2.5.0
ok app: app 'bookstore' 0.1.0 loaded
ok manifest: 20 routes (17 native), 18 tools, 3 agents, 1 behaviours, 1 flows
FAIL model:default [WC301]: default → claude-opus-5 (used by agent librarian): model "claude-opus-5" needs ANTHROPIC_API_KEY, which is not set. …
hint: The model's provider needs an API key that is not set. …
FAIL api_key:WEBCORTEX_API_KEY [WC106]: WEBCORTEX_API_KEY is not set: principal 'service' cannot authenticate
hint: export WEBCORTEX_API_KEY=$(webcortex keygen)
ok budgets: every agent and behaviour has a token_budget
ok port: 127.0.0.1:8000 is free
8 ok, 0 warning(s), 2 failure(s)
What it checks:
| Check | Fails / warns when |
|---|---|
python, core |
the interpreter is older than 3.12, or the compiled core is missing or a different version from the Python package (WC103) |
app, manifest |
the module does not import (WC102) or a declaration is invalid (WC101) |
model:<name> |
a model an agent, behaviour, compaction or flow classifier uses does not resolve: a missing key (WC301), an undeclared provider prefix (WC302). This is the runtime's own routing, so the message is exactly what a run would fail with |
online:<url> |
with --online: a provider endpoint does not answer within 3 s (WC303) |
api_key:<ENV>, jwt, upstream:<name>, provider:<name> |
an environment variable the app names is unset (WC106). For a key or JWT secret, also empty or still a starter placeholder starting replace-me (since 2.9). Values never appear in the report |
security.* |
no authentication (WC608), no rate limit, cors("*"), every actuator warning from security_report() |
estop |
actuators that are armed at boot (start_halted=False) |
watcher:<name>, device:<name> |
a watcher whose agent has no token_budget, a plain-http webhook, a camera bearer_env that is unset |
budgets |
agents or behaviours without token_budget |
tracing |
(since 2.9) an OTLP endpoint, declared or in OTEL_EXPORTER_OTLP_*, that has no http:// or https:// scheme or does not parse, so nothing would be exported (warns) |
database, templates |
a SQLite directory or templates directory that does not exist (WC107) |
port / running |
the port is taken by something that is not this app (WC104); when this app is already running, it reports its version and live e-stop state |
webcortex doctor --json prints the report as JSON; app.doctor() and
webcortex.doctor(app) return it.
The live status view¶
GET /_webcortex/status (admin scope once any auth is configured, exactly like every other
control-plane route) describes what the running app is doing:
| Section | Contents |
|---|---|
| top level | app, app_version, webcortex (core version), uptime_secs, started_at_ms, python_workers |
requests |
totals and, per route pattern, request counts by status class, mean / p50 / p95 / p99 / max latency |
errors |
total failures and the last 100: {at_ms, source: http|mcp|agent|watcher, route, status, code, request_id} |
agents |
runs in_flight (agent, caller, depth, steps, tokens so far, running time) and the last 50 recent (outcome, code, steps, tool calls, tokens, tree tokens, cost, duration); per-agent tallies, including cancelled — runs dropped because their request timed out or their client went away; sessions; pending approvals |
usage |
the spend ledger's totals: tokens, estimated cost (null when a model is unpriced), by model |
budgets |
each agent, behaviour and flow: its token_budget and max_steps, tokens spent, largest run, how often it hit a limit |
estop |
halted or armed, the halt state, the actuator tools |
devices |
per device: connected, recent fps, frames in, dropped (over max_fps) and rejected (not an image, empty, too large), last frame and telemetry age, subscribers; per watcher: runs, failures, last outcome and code |
Each recent run also carries trace_id: webcortex trace <id> (or GET /_webcortex/traces/<id>)
shows what that run did span by span. See Traces.
Request ids are the x-request-id the client sent (or the one the server assigned and returned),
so a failure a user reports can be found here. Nothing in the view holds a body, a header, a
query string or a credential; a failure's route is the route pattern (GET /books/{id}), not
the concrete path, wherever a route matched.
Cost of recording¶
Counting is lock-free: each route has a slot of relaxed atomic counters and a 40-bucket latency
histogram, chosen by route id. Only a failure (into a bounded ring) and an agent run (already
measured in seconds) take a lock. Measured with ab, interleaved against 2.4.0: a Rust static
route is within noise (−0.8%), and a flood in which every request is a 404 is about 3% slower;
see
benchmarks/diagnostics/RESULTS.md.
Percentiles are reported as the upper bound of their bucket, capped at the route's exact
max_ms, so they are never optimistic, within 2x of the truth, and never above the maximum.
From the command line¶
Illustrative output:
$ export WEBCORTEX_API_KEY=… # an admin key
$ webcortex status
bookstore 0.1.0 · webcortex 2.5.0 · up 0h12m03s
requests: 1840 2xx 1801 4xx 37 5xx 2 p50 0.256ms p95 1.024ms max 812.4ms
1203 GET /books (4xx 0, 5xx 0, p95 0.512ms)
errors: 39 total; most recent:
WC602 403 POST /books 4c1f…
WC304 agent agent librarian 9e2a…
recent runs:
failed librarian 1 steps 0 tokens WC304
completed librarian 3 steps 2210 tokens $0.0081
$ webcortex status --json
$ webcortex status --url https://api.example.com --key "$ADMIN_KEY"
For a coding agent¶
The app's own MCP endpoint (the one webcortex mcp-config connects) offers, to callers who
may read the control plane:
- the tool
webcortex_status(section:requests | errors | agents | usage | budgets | estop | devices, or omit for everything), - the tool
webcortex_explain(code), - the resources
webcortex://statusandwebcortex://errors.
So "why did that request fail?" is one tool call: find the request id in errors.recent, then
explain its code. An app tool with the same name takes precedence over a built-in.
Testing failure paths offline¶
With WEBCORTEX_FAKE_PROVIDER=1, an agent input of fail:<message> makes the fake provider fail
the call with that message, the way a real provider error reads:
status, out = post("/agents/desk", {"input": "fail:anthropic 529: overloaded"})
assert status == 502 and out["code"] == "WC304"
The diagnostics benchmark (benchmarks/diagnostics/) uses this (and a missing key, and a closed port) to score
eighteen broken scenarios against 2.4.0 and 2.5.0.