Skip to content

Traces

New in 2.8.

A run result lists the steps one agent took. It does not show what the worker it called did, how long the flow around it took, or where the tokens of the whole request tree went. A trace does. Every agent, flow and behaviour run records a tree of spans: the run itself, each model call with its token counts, each tool call, each handoff, and any run a tool call started, nested under the call that started it.

$ webcortex trace latest
  trace 7d1c…e02a  ok  2.41s  4 model call(s)  2 tool call(s)  1830 tokens
  agent front_desk  2.41s  completed
    └ chat default  0.61s  412 in / 38 out
    └ tool billing  1.52s
      └ agent billing  1.52s  completed
        └ chat default  0.58s  390 in / 41 out
        └ tool lookup_invoice  3ms
        └ chat default  0.90s  455 in / 96 out
    └ chat default  0.27s  362 in / 36 out

Tracing is on by default. There is nothing to add to an app to get it. (The tree above shows what the command prints; its timings and token counts are an illustration, not a measurement.)

Finding a trace

  • A run result carries trace_id. So does a flow's envelope.
  • At the edge, the HTTP response carries the header x-webcortex-trace. That is the only way to learn a behaviour's trace, since a behaviour returns whatever its function returned. Since 2.9 a 504 (WC203) for an agent, behaviour or flow run that timed out carries it too, with trace_id in the error body. A flow that times out or loses its client ends its span as cancelled, so it is exported; in 2.8 it stayed open.
  • GET /_webcortex/status lists trace_id on each recent run.
  • "latest" names the newest trace wherever an id is accepted.

Reading traces

GET /_webcortex/traces recent traces, newest first, one summary each
GET /_webcortex/traces/{id} one trace with every span; latest works as an id
webcortex trace [ID] the same, as a table or a tree; --json for the document
MCP tool webcortex_trace the same for a coding agent: no arguments lists, trace_id returns one
app.traces() / app.trace(id) the same from Python

All of it is control-plane data: once authentication is configured it needs the webcortex:admin scope, like /status. A trace that is not held is a 404 with code WC219.

A summary has trace_id, name and kind (of the root span), status, started_unix_ms, duration_ms (null while any span is still open), spans, open_spans, dropped_spans, model_calls, tool_calls, input_tokens, output_tokens, cache_tokens and errors (the codes of failed spans). The token counts are for the whole tree, which is what usage.tree_tokens on the outermost run reports as one number.

A span has:

field
span_id, parent_span_id 16 hex digits; the root's parent is null
kind agent, model, tool, flow, behaviour, handoff or output
name agent billing, chat default, tool lookup_invoice, flow intake, …
start_ms offset from the start of the trace
duration_ms null while the span is open
status ok, error, cancelled (the request timed out or the client left) or open
code the error code of a span that failed
attributes see below

What a span holds, and what it never holds

Attributes use the OpenTelemetry GenAI names where one exists:

attribute on
gen_ai.operation.name invoke_agent, chat, execute_tool
gen_ai.agent.name, gen_ai.tool.name agent and tool spans
gen_ai.request.model, gen_ai.response.model model spans: the name the app used, and what it resolved to
gen_ai.usage.input_tokens, gen_ai.usage.output_tokens model spans
webcortex.usage.cache_read_tokens, webcortex.usage.cache_write_tokens model spans
webcortex.run.id, .status, .steps, .tokens, .depth agent spans
webcortex.flow.name flow spans

A span never holds a prompt, a tool's arguments, a tool's result, a model's answer, a header or a credential. Names, timings, counts and codes only, the same rule the status view follows. The audit trail (/_webcortex/audit) is where arguments are recorded; a trace is safe to send to a third party's collector in a way the audit trail is not.

Limits

Traces are kept in memory: the newest 200, each up to 500 spans, and not across a restart. Past 500, further spans in that trace are counted in dropped_spans rather than stored. Both numbers are settings:

app.tracing(keep=1000, max_spans=2000)
app.tracing(enabled=False)        # no spans, no trace_id on results

Only routes that run a model open spans. A static, query, proxy, page or plain Python route never touches the tracer. For a traced run the cost is a short mutex-guarded write at the start and at the end of each span. It has not been benchmarked.

Exporting to a collector (OTLP)

Finished spans can be sent to any OpenTelemetry collector, or a vendor that accepts OTLP over HTTP, as JSON:

app.tracing(
    otlp_endpoint="http://localhost:4318",     # /v1/traces is appended
    otlp_headers_env="OTLP_HEADERS",           # the NAME of a variable holding "k=v,k2=v2"
    service_name="support-desk",               # defaults to the app's name
)

Or with no code at all, through the standard environment variables, which are read when the app declares no endpoint:

export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318     # /v1/traces is appended
export OTEL_EXPORTER_OTLP_HEADERS="authorization=Bearer …"
webcortex run

OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, when set, is used exactly as given. As everywhere else in WebCortex the credential is named, never embedded: otlp_headers_env is the name of a variable, and the manifest stays safe to log.

The headers variable is read as the OpenTelemetry specification says: keys and values are trimmed and percent-decoded (authorization=Bearer%20abc), and an entry that is not key=value or not a valid header is skipped with a warning naming its position, never its value. An endpoint without http:// or https:// (collector:4318) cannot be used; since 2.9 that is a warning at boot naming the variable it came from, rather than silence, and webcortex doctor reports it too.

Spans are posted in batches (up to 256, at most about a second after the first one in the batch ended), with service.name, service.version (the app's version) and the scope webcortex. Model calls are CLIENT spans and everything else INTERNAL; a failed span has status ERROR with its code as the message and as the attribute webcortex.error.code.

The exporter is deliberately lossy. A batch the collector refuses, or cannot be sent, is dropped and counted, never retried, and the queue between the runtime and the exporter is bounded (4096 spans): a collector that is down must not grow memory inside the app it observes. A run pushed out of the in-memory ring (keep) while it is still running still exports its spans when they end; spans that cannot be exported (over max_spans, or more than 4096 held aside at once) are counted as dropped. GET /_webcortex/traces reports export: {configured, spans_exported, spans_failed, spans_dropped, last_error}, and webcortex trace prints that line. On shutdown the exporter sends one last batch.

Only the protocol's JSON encoding over HTTP is implemented, not protobuf or gRPC. Collectors accept it on port 4318.

Joining a caller's trace

A request that carries a W3C traceparent header joins that trace instead of starting one: the run's root span takes the header's trace id and names the caller's span as its parent, so in your tracing backend the agent run appears under the request that caused it.

curl -s localhost:8000/agents/desk \
  -H 'traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01' \
  -d '{"input": "hi"}'
# "trace_id": "0af7651916cd43dd8448eb211c80319c"

A malformed header is ignored and a fresh trace is started. WebCortex does not yet send traceparent onward to proxied upstreams or to model providers.

Approvals

A run that suspends on an approval gate ends its span with webcortex.run.status: awaiting_approval. When a human decides, the continuation is a new span named agent <name> (resumed) in the same trace, a child of the original, so the wait is visible as the gap between them.

For a coding agent

With the MCP server connected (webcortex mcp-config), an agent debugging the app can ask webcortex_trace for the last run and read where the time and tokens went, then webcortex_explain for any code on a failed span. Together with webcortex_status that is the loop: what is failing, what that run did, what the code means.