Skip to content

The device hub

WebCortex 2.3 sits between the physical world and AI agents. Cameras and sensors stream in. Agents see what they send. What the agents conclude streams out to dashboards, services and other systems. All of this runs in the Rust core: frames, telemetry, fan-out and the watcher loop never touch Python.

 phone / laptop / Pi / IP camera ──frames──▶ ┌──────────────┐ ──tool──▶ agent sees the frame
 sensor / PLC / IMU ─────telemetry─────────▶ │  device hub  │ ◀─insight──┘
                                             │    (Rust)    │
 dashboards, services   ◀── WebSocket ────── │  ring buffer │
 any other system       ◀── webhook POST ─── │  broadcast   │
                                             └──────────────┘
webcortex new site -t hub && cd site
export WEBCORTEX_API_KEY=$(webcortex keygen)
webcortex dev
open http://127.0.0.1:8000/devices/dock/connect    # this laptop's camera becomes "dock"
open http://127.0.0.1:8000/devices/dock/view       # the live stream, and what the agent says

Declaring devices

app.camera("dock", description="the loading dock", max_fps=10)
app.camera("lobby", source="http://192.168.1.40/snapshot.jpg", bearer_env="LOBBY_TOKEN")
app.sensor("dock_env", description="temperature and humidity")
Option Default
description — What the agent is told the device is
source None An HTTP snapshot URL. The camera is pulled whenever a frame is asked for
headers, bearer_env — Credentials for a pulled camera
keep 8 Frames held in the ring buffer
max_fps 30 Ingest ceiling. Faster frames are dropped (WebSocket) or refused with 429 (HTTP). A frame may arrive up to half an interval early, so a camera sending at exactly max_fps with ordinary jitter keeps every frame; the long-run rate never exceeds it
max_frame_bytes 5 MB Frames larger than this are refused (422 over HTTP; over the socket, a message past it closes the connection with 1009 and a warning in the log)
ingest_scopes ["devices:ingest"] Required to send frames and telemetry. The default is shared by every device that keeps it
read_scopes ["devices:read"] Required to read anything back
tool True Whether agents get the device's tools
prefix /devices Where the device's routes are mounted

One key per device

Give each device its own key with only an ingest scope. A stolen camera key can then send frames but cannot read anything back.

That alone does not tie the key to its camera. Ingest is authorised by scope, and every device that keeps the default ingest_scopes accepts the same devices:ingest, so a key stolen from the cheapest camera could also send forged frames and telemetry to every other device, and to the watchers and agents that act on them. To bind each key to one device, give each device a scope of its own:

app.api_key("DOCK_CAMERA_KEY",  id="dock-camera",  scopes=["devices:ingest:dock"])
app.api_key("VAULT_CAMERA_KEY", id="vault-camera", scopes=["devices:ingest:vault"])

app.camera("dock",  ingest_scopes=["devices:ingest:dock"])
app.camera("vault", ingest_scopes=["devices:ingest:vault"])

Now the dock's key gets 403 from /devices/vault/frames. The default is kept for compatibility with existing device keys, so a shared scope is still what you get unless you ask for this.

Each device gets these routes (prefix /devices/<name>):

Route
POST …/frames One encoded image (PNG, JPEG, GIF or WebP; the type is sniffed)
POST …/telemetry One JSON reading, at most 64 KB
GET …/ws WebSocket in. Binary messages are frames, text messages are telemetry
GET …/stream WebSocket out. Every frame, telemetry reading and insight as it happens
GET …/snapshot JSON with the newest frame as an image marker, its age, and the latest telemetry. Agent tool <name>_snapshot
GET …/latest The newest frame's bytes, for an <img>
GET …/telemetry The newest reading. Agent tool <name>_telemetry
GET …/insights Recent insights from watchers. Agent tool <name>_insights
GET …/connect A page that turns a browser's camera into this device
GET …/view A live monitor

GET /_webcortex/devices lists every device with frames in, frames dropped, subscribers and the newest frame.

Connecting a camera

A phone or laptop: open /devices/<name>/connect, paste a key with the ingest scope, and press start. Frames go over the WebSocket as JPEG at the rate and width you choose. Browsers only allow camera access on https:// or localhost, so to use a phone, put the hub behind TLS: a reverse proxy, or a tunnel such as cloudflared or ngrok.

A Raspberry Pi, a Jetson, or any machine with Python uses the standard library client:

import cv2
from webcortex import Image
from webcortex.client import DeviceConnection

cap = cv2.VideoCapture(0)
with DeviceConnection("https://hub.example.com", "dock", key=DEVICE_KEY) as cam:
    while True:
        ok, frame = cap.read()
        if ok:
            cam.send_frame(cv2.imencode(".jpg", frame)[1].tobytes())   # or Image.from_array(frame, bgr=True)
        cam.send_telemetry({"cpu_temp_c": read_cpu_temp()})

The hub answers a refused frame or reading with an error message. Each send first reads any that have arrived, without blocking: cam.last_error is the newest and cam.errors counts them. If the hub closed the socket (a frame over max_frame_bytes is closed with 1009), the next send raises WebSocketError saying why. A device that never reads is never stalled by replies it ignores: the hub drops them rather than wait.

A microcontroller such as an ESP32-CAM sends an HTTP POST:

curl -X POST https://hub.example.com/devices/dock/frames \
     -H "x-api-key: $DEVICE_KEY" -H "content-type: image/jpeg" --data-binary @frame.jpg

An IP camera: set source= to its snapshot URL. Nothing is fetched until a frame is asked for, and everyone who asks while a fetch is in flight shares it (its frame, or its error), so forty agents and viewers looking at once cost the camera one request, not forty. RTSP streams need a sidecar such as ffmpeg or go2rtc, which exposes a snapshot URL or pushes frames.

Agents that look

The <name>_snapshot tool returns the newest frame, and an agent that calls it receives the image itself (see Vision and robotics). Only the last max_images frames stay in context.

app.agent("inspector", tools=["dock_snapshot", "dock_env_telemetry"], scopes=["devices:read"], max_images=2)

Watchers: the hub's middle

app.watch(
    "dock_watch",
    device="dock",
    agent="inspector",
    input="Anything unsafe or unusual at the dock right now?",
    every=15,                          # seconds
    only_on_change=True,               # the default: skip a tick when nothing changed
    scopes=["devices:read"],           # the authority the agent runs with
    max_runs_per_hour=120,             # the spend ceiling
    webhook="https://ops.example.com/hooks/dock",
)

Every every seconds the watcher takes the newest frame (pulling it, for an IP camera) and the newest telemetry, and skips the tick if nothing changed. Changed means different bytes, not a new sequence number. Pass only_on_change=False to run the agent on every tick regardless, for a periodic report on a scene that rarely changes; the hourly ceiling still applies. Otherwise it runs the agent with the frame attached and the telemetry in the input, and publishes the answer as an insight:

{"type": "insight", "watcher": "dock_watch", "device": "dock", "agent": "inspector",
 "seq": 412, "status": "completed", "output": "A pallet is blocking the east exit.",
 "error": null, "run_id": "…", "tokens": 1830, "at_ms": 1790866914641}

The insight goes to every /stream subscriber, to dock_insights, and as a JSON POST to the webhook (with webhook_bearer_env for a bearer token). It is also written to the audit log. A watcher has no caller, so its agent runs with exactly scopes, intersected with the agent's own.

A watcher spends tokens on a timer

The worst case per hour is max_runs_per_hour × the agent's token_budget. Set both. Every run is in GET /_webcortex/usage.

Subscribing

From a browser, the /view page. From Python:

from webcortex.client import subscribe

for event in subscribe("https://hub.example.com", "dock", key=READER_KEY, frames="meta"):
    if event["type"] == "insight":
        notify(event["output"])

The /stream protocol:

Message
{"type": "hello", "device": {…}, "insights": […]} On connect, with recent insights
{"type": "frame", "seq", "media_type", "bytes", "at_ms"}, then a binary message A frame. Add ?frames=meta to receive the header only
{"type": "telemetry", "data": {…}} A reading
{"type": "insight", …} A watcher's conclusion
{"type": "lagged", "skipped": n} A slow subscriber missed n events. Frames are dropped for it rather than buffered

A quiet stream is pinged every 30 seconds, and a subscriber that has not read a message within 30 seconds of it being sent is disconnected, so a viewer that vanished without closing does not hold its subscription forever. Every socket answers a protocol violation with the RFC 6455 close code (1002, 1007 for text that is not UTF-8, 1009 for a message that is too large) and answers a close with a close.

Agents over a WebSocket

Every agent route also accepts a WebSocket. Send a turn and receive each step as it happens, then the result. The socket stays open for the next turn.

from webcortex.client import AgentSocket

with AgentSocket("https://hub.example.com", "/agents/inspector", key=KEY) as agent:
    result = agent.ask("Check the dock", on_step=lambda e: print(e["step"]["kind"], e["step"].get("tool")))
You send You receive
{"input": "…", "images"?: […], "session_id"?: "…", "reset"?: false} {"type": "step", "run_id", "agent", "step": {…}} for each step, then {"type": "result", "result": {…}}

Each turn is held to the app's rate limit, exactly as a separate HTTP request would be (since 2.9): a turn over the limit is not run, and the socket answers with an error carrying WC401 and retry_after, then closes if the client keeps going.

Authentication for sockets

A WebSocket upgrade is authenticated exactly like an HTTP request, and a socket is held to the scopes of the route it is mounted on. Browsers cannot set headers on a WebSocket, so an upgrade request may pass its credential as ?access_token= instead: an API key, or (since 2.9, when app.jwt(...) is configured) a JWT, recognised by its three dot-separated segments. This is accepted only on upgrades and only when no header credential is present. The request log records the path, never the query, but a proxy in front of the hub may log full URLs: give browsers keys with the narrowest scope that works.

A socket opened with a JWT holds that token's authority only until it expires (plus leeway_secs). After that an agent socket refuses the next turn with WC603 and closes with 1008, and a stream or ingest socket is closed. Open a new socket with a fresh token.

A browser opening a socket from another origin is refused (403) unless that origin is listed in app.cors(...), so a page elsewhere cannot open the stream with a key the operator's browser has saved. Devices and webcortex.client send no Origin and are unaffected.

The /connect and /view pages are public. They are static HTML containing no data, and everything they show needs a key.