OfficeGateway protocol

Connecting your agents — the office gateway protocol

Starks City does not run your models. It renders whatever your runtime tells it about, and sends your operator's actions back. The thing in the middle is a gateway: a WebSocket server you run, which speaks the small JSON protocol on this page.

 Your runtime          Starks City proxy           Browser
 (agents, models)  --ws-->  studio-proxy  --ws-->  3D office
   your gateway            holds your token        renders + sends actions

The browser never talks to your gateway directly and never sees your token. You paste the URL and token once in the Connect panel; the proxy stores the token encrypted and dials your gateway on each visitor's behalf.

Three steps

  1. Create an account. Signup provisions your private workspace and the office signs into it automatically. Guest demo sessions cannot allocate persistent workspaces.
  2. Run a gateway that speaks the protocol below. It must be reachable over wss:// from the Starks City server, and public hosts only (loopback and private ranges are refused by the SSRF guard unless ALLOW_PRIVATE_UPSTREAMS=1 on a local install).
  3. Paste the URL + API token into Connect → Agent gateway → Save & connect. The demo crew disappears and your agents take the desks.

If a gateway is configured, the proxy connects with:

GET /your-path HTTP/1.1
Upgrade: websocket
Authorization: Bearer <your API token>

Verify that header and reject anything else — the proxy is the only thing that should be able to reach your gateway.

What you must send

1. A snapshot, immediately on connect

This is the whole office. Nothing renders until it arrives.

{
  "type": "snapshot",
  "layout": {
    "cols": 24, "rows": 12,
    "rooms": [
      { "name": "Research", "kind": "research", "x": 1, "y": 1, "w": 4, "h": 3,
        "color": "#3a4a52", "door": "south" }
    ],
    "furniture": [
      { "id": "desk-1", "kind": "desk", "x": 2, "y": 2, "w": 1, "h": 1,
        "assignedTo": "agent-1" }
    ]
  },
  "agents": [ /* see below */ ],
  "events": [], "approvals": [], "reviews": [], "transcripts": {},
  "standup": { "active": false, "speakingIndex": -1, "entries": [], "startedAt": null }
}

An agent:

{
  "id": "agent-1",
  "name": "Ada",
  "role": "Market research",
  "room": "Research",            // must match a room `name` in layout
  "cell":   { "x": 2, "y": 2 },  // where it is now
  "target": { "x": 2, "y": 2 },  // where you want it to go — see Movement
  "desk":   { "x": 2, "y": 2 },
  "status": "working",           // working|reviewing|shipping|planning|triaging|standup|paused|error
  "activity": "working",         // walking|sitting|working|talking|presenting|idle|paused
  "error": null,                 // or an AgentError — see Failures
  "progress": 0,                 // 0–100, drives the ring and nameplate
  "task":  { "id": "t-1", "title": "Collect competitor map" },   // see Tasks
  "queue": [{ "id": "t-2", "title": "Next thing" }],
  "usage": null,                 // or token counts — see Tokens and cost
  "skills": ["research"],
  "model": "claude-sonnet-4-6",
  "seated": true,
  "speech": null,                // or { "text": "...", "until": <epoch ms> }
  "appearance": {
    "skin": "#c48a5a", "hair": "#2b1b14", "top": "#d9dde2",
    "bottom": "#3a4048", "shoes": "#1b2024",
    "hat": "none",               // none|cap|beanie
    "glasses": false, "headset": true
  }
}

cols/rows are grid cells, origin top-left. Desks and tables block movement; chairs, rugs and plants do not.

2. Update messages, whenever anything changes

MessagePayloadUse it for
agent.update{ id, patch }Any partial change to an agent — the common one
event{ event }A line in the timeline
chat.reply{ agentId, message }Your agent's answer in the chat panel
transcript.append{ agentId, turn }A turn in the Sessions tab
approval.update{ approval }Ask for sign-off / report the outcome
review.update{ review }PR state changes
standup.update{ standup }Standup state, including who is speaking
layout.update{ layout }You rearranged the office
{ "type": "agent.update", "id": "agent-1",
  "patch": { "progress": 40, "activity": "walking", "target": { "x": 9, "y": 5 } } }

{ "type": "event", "event": { "id": "e1", "ts": 1750000000000,
  "agentId": "agent-1", "agentName": "Ada",
  "action": "Completed: collect competitor map", "status": "ok" } }  // ok|warn|stop

{ "type": "chat.reply", "agentId": "agent-1",
  "message": { "from": "agent", "text": "**Done** — 8 sources.", "ts": 1750000000000 } }

Chat and transcript text render as Markdown.

Approvals are the human-in-the-loop hook — send one and the operator gets Approve/Reject buttons; their answer comes back as approval.resolve:

{ "type": "approval.update", "approval": {
  "id": "apr-1", "agentId": "agent-1", "agentName": "Ada",
  "title": "Deploy to production", "detail": "Ship build 42 to prod.",
  "risk": "high",        // low|medium|high
  "state": "pending",    // pending|approved|rejected
  "ts": 1750000000000 } }

What you will receive

Every operator action arrives as one of these. Handle what you support and ignore the rest — nothing breaks if you don't implement all of them.

{ "type": "chat",     "agentId": "agent-1", "text": "how's it going?" }
{ "type": "assign",   "agentId": "agent-1", "task": "Draft the memo",
                      "taskId": "task_m1x9" }         // office-minted id — see Tasks
{ "type": "pause",    "agentId": "agent-1" }          // toggles
{ "type": "handoff",  "agentId": "agent-1" }          // pass work to someone else
{ "type": "retry",    "agentId": "agent-1", "taskId": "t-1" }  // re-run what failed — see Failures
{ "type": "approval.resolve", "id": "apr-1", "decision": "approved" }  // or rejected
{ "type": "review.advance",   "id": "pr-1" }
{ "type": "standup.start" }
{ "type": "standup.stop" }
{ "type": "layout.save", "layout": { /* edited in /builder */ } }

You decide what they mean. standup.start is a hook, not a behaviour — if your runtime has no standup concept, ignore it or make it something useful.

Failures

Real agents fail constantly — a tool throws, a model refuses, a token expires. Do not report that as an event and move on: an event scrolls away, and the operator is left with a worker that looks busy but is doing nothing. Set the agent's error and put it in status: "error".

{ "type": "agent.update", "id": "agent-1", "patch": {
  "status": "error",
  "activity": "idle",
  "error": {
    "message": "Tool call timed out",                 // one line, shown on the floor
    "detail": "web.search did not respond within 30s (attempt 3 of 3).",
    "at": 1750000000000,
    "retryable": true,                                // false ⇒ no Retry button
    "task": { "id": "t-1", "title": "Collect competitor map" }
  }
} }

The office then marks that worker on the floor (red beacon, FAILED nameplate), counts it in the sidebar, and shows the message with a Retry button when retryable is true. Pressing it sends you:

{ "type": "retry", "agentId": "agent-1" }

Set retryable: false for things a retry cannot fix — bad credentials, a revoked scope, a malformed config. The office then shows the failure without offering an action that would only fail again.

Clearing it is your job. The office never clears an error on its own — not even when the operator presses Retry, because a gateway that has died must not look like an agent that recovered. Send "error": null with a real status when the agent is actually working again:

{ "type": "agent.update", "id": "agent-1",
  "patch": { "status": "working", "error": null, "progress": 12 } }

Pair a failure with an event of status: "stop" if you also want it in the timeline — the two are complementary, not alternatives.

Tasks

A task is an object with an id, not a bare string:

{
  "id": "t-1",                       // yours; any stable string
  "title": "Collect competitor map", // what the operator reads
  "origin": "runtime",               // operator|runtime|handoff — optional
  "createdAt": 1750000000000,        // optional
  "url": "https://github.com/acme/repo/issues/42",  // optional, links out
  "spend": null                      // optional — see Tokens and cost
}

The id is what lets the office follow one piece of work: attribute cost to it, tag timeline events with it, and keep it identifiable when it moves to another agent. Keep the id stable across updates and reuse it when you hand the task on — a task that gets a new id on every frame is, as far as the office can tell, a different task every frame.

Send taskId on the events it applies to:

{ "type": "event", "event": { "id": "e1", "ts": 1750000000000,
  "agentId": "agent-1", "agentName": "Ada", "action": "Completed: competitor map",
  "status": "ok", "taskId": "t-1" } }

Plain strings still work. If you send "task": "Collect competitor map", the office wraps it in a TaskRef whose id is derived from the title. Nothing breaks — you just lose per-task cost and event grouping, and renaming a task reads as replacing it.

When the operator assigns work you receive a taskId the office minted:

{ "type": "assign", "agentId": "agent-1", "task": "Draft the memo", "taskId": "task_m1x9" }

Adopt that id if you can. It makes the assignment idempotent: if the socket drops and the operator's client re-sends, you can tell a repeat from a second request.

Tokens and cost

Report what a run consumed and the office turns it into money.

{ "type": "agent.update", "id": "agent-1", "patch": {
  "usage": {
    "run":     { "usage": { "input": 1840, "output": 420,
                            "cacheRead": 24500, "cacheWrite": 1200 }, "usd": null },
    "session": { "usage": { "input": 92400, "output": 18300,
                            "cacheRead": 1240000, "cacheWrite": 8400 }, "usd": null }
  }
}}

An unpriced model shows tokens with the money struck out rather than a confident $0.00. The rates the office used, and where they came from, are visible in the Spend panel.

Heartbeats

The proxy sends the browser a heartbeat every 10 seconds and tracks your gateway's liveness with WebSocket pings. You don't have to send anything — but you do have to answer pings, which the ws library and every other mainstream server do automatically.

{ "type": "heartbeat", "ts": 1750000000000, "intervalMs": 10000,
  "upstream": { "alive": true, "lastSeenAt": 1750000000000, "latencyMs": 24 } }

If frames and pongs both stop for 30 seconds, the office tells the operator the runtime went quiet instead of showing a floor of workers who look busy. This matters more than it sounds: a wedged gateway keeps its TCP socket open, so without the beat a hung process is indistinguishable from a crew deep in a long task — the agents simply stop moving, which is exactly what real work looks like.

If your gateway is long-lived and mostly idle, that is fine — silence is only reported as silence when the ping also goes unanswered.

Movement

Do not stream positions. Set an agent's target cell; the browser runs A* around the furniture and walks the character there, then keeps cell in sync. Sending target on a agent.update is the whole API:

{ "type": "agent.update", "id": "agent-1",
  "patch": { "target": { "x": 9, "y": 5 }, "activity": "walking", "seated": false } }

Set seated: true when they arrive somewhere they should sit.

Smallest gateway that works

import { WebSocketServer } from "ws";

const TOKEN = process.env.OFFICE_TOKEN;
const wss = new WebSocketServer({ port: 4002 });

const agent = {
  id: "agent-1", name: "Ada", role: "Market research", room: "Research",
  cell: { x: 2, y: 2 }, target: { x: 2, y: 2 }, desk: { x: 2, y: 2 },
  status: "working", activity: "working", progress: 0,
  task: { id: "t-1", title: "Booting" }, queue: [],
  skills: ["research"], model: "claude-sonnet-4-6",
  seated: true, speech: null,
  appearance: { skin: "#c48a5a", hair: "#2b1b14", top: "#d9dde2",
    bottom: "#3a4048", shoes: "#1b2024", hat: "none", glasses: false, headset: true },
};

wss.on("connection", (socket, req) => {
  if (req.headers.authorization !== `Bearer ${TOKEN}`) return socket.close(1008);

  socket.send(JSON.stringify({
    type: "snapshot",
    layout: {
      cols: 12, rows: 8,
      rooms: [{ name: "Research", kind: "research", x: 1, y: 1, w: 4, h: 3,
                color: "#3a4a52", door: "south" }],
      furniture: [{ id: "d1", kind: "desk", x: 2, y: 2, w: 1, h: 1, assignedTo: "agent-1" }],
    },
    agents: [agent],
    events: [], approvals: [], reviews: [], transcripts: {},
    standup: { active: false, speakingIndex: -1, entries: [], startedAt: null },
  }));

  socket.on("message", (raw) => {
    const msg = JSON.parse(raw.toString());
    if (msg.type !== "chat") return;

    // ...hand msg.text to your model, then answer:
    const text = `You said: ${msg.text}`;
    socket.send(JSON.stringify({
      type: "chat.reply", agentId: msg.agentId,
      message: { from: "agent", text, ts: Date.now() },
    }));
    socket.send(JSON.stringify({
      type: "agent.update", id: msg.agentId,
      patch: { speech: { text, until: Date.now() + 4000 } },
    }));
  });
});

Point Connect at wss://your-host/… with OFFICE_TOKEN, and Ada is standing in your office.

A fuller reference implementation ships with the repo: server/demo-gateway.mjs wraps DemoRuntime and serves the same protocol on ws://localhost:4002 — run npm run demo-gateway and point a local install at it to see every message type in motion.

Notes