APIs & Devices NotesRohit’s interview study guide
Chapter 07

Real-Time Device State

Getting live status from a device to the web UI, touch controllers and integrations: choosing a transport, designing events with sequence numbers, snapshot plus events, reconnecting across reboots, and serving many clients from a small box.

5 topics
Parts marked Advanced are extra depth. Skip them on a first read or a quick revision.

#Polling, long polling, SSE and WebSockets for device status

Added

A video device's status changes all the time, and the people looking at it care now: a call comes in, someone mutes the room, a USB camera is unplugged, a firmware update reaches 60%, the network link drops and comes back. Plain REST gives you GET /status, and in HTTP the server can't speak first. Something has to carry changes from the device to whoever is watching.

The four ways to do that (short polling, long polling, Server-Sent Events, WebSockets) are compared for web-scale systems in the System Design course's Real-time options topic, with the cost of each worked out for a million clients. On a device the question changes shape. The server is the device itself: a small box whose main job is the call, often serving HTTP/1.1 from an embedded web server with a self-signed certificate. And its listeners are a handful of very different clients: an admin's browser tabs, the room's touch controller, a room control system a partner programmed years ago, a monitoring tool, a cloud management service. Each of those has different abilities and needs, so a device often offers more than one transport over the same events.

A mental model: the device is a receptionist at a front desk. Short polling is visitors walking up every few seconds to ask "anything for me?". Long polling is a visitor waiting at the desk until something arrives, then leaving and queueing again. SSE is the lobby loudspeaker: one announcement reaches everyone listening, but nobody can answer through it. A WebSocket is a walkie-talkie handed to each guest: both sides can talk at any time, but the receptionist has to carry every one of them, and can only carry so many.

Device state store call, audio, camera, network event hub seq numbers, recent-event ring web server, CLI, outbound agent Admin web UI tabs SSE for status, REST for commands Touch controller WebSocket: status and commands Room control system Telnet/TCP notifications, long poll Cloud management device dials out (behind NAT) text/event-stream notify / ?since= outbound wss
One stream of events inside the device, several transports out of it. Who the client is decides the transport.
ClientUsually the best fitWhy
Admin web UI tabSSE for status, REST for commandsStatus flows one way; commands already have REST's validation, ETags and error bodies. EventSource reconnects and resumes by itself.
Admin UI with continuous control (camera joystick, live meters)WebSocketMany small messages both ways with low latency; one connection instead of a request per joystick nudge.
Touch controller or paired accessoryWebSocket (or the vendor's own protocol)Long-lived, two-way, and it owns the room's buttons, so it sends as much as it receives.
Room control system, older partner integrationTelnet/TCP line protocol with notifications, or long pollingControl processors and scripts often speak raw TCP or plain HTTP only; see Asynchronous notifications on a command channel.
Monitoring tool, inventory scriptShort polling at minutesDelay is fine; a stateless GET is the cheapest thing a device can serve.
Cloud management serviceA connection the device opens outwardsThe device sits behind the customer's firewall and NAT, so the cloud can't connect in; the device dials out and keeps that connection up.

What Poly publishes here is narrow, so it's worth saying exactly. The Poly VideoOS REST API Reference Guide (for VideoOS 4.4.0, "Last update: August 2025", HP) documents status as plain request/response endpoints, such as GET /rest/system/status and GET /rest/audio, and a POST /rest/system/reboot, all over HTTPS with an admin session from /rest/session. I found no push channel (WebSocket, SSE or event subscription) in that guide. Whether PolyOS devices offer one, the public documents I found don't say, so everything below teaches the general pattern, not a Poly implementation.

The smallest version of the main pattern is SSE: one long GET that the device keeps writing events to. Here's a simulated device with its real HTTP server, and a client reading the stream the way EventSource does:

device-sse.mjs
// A simulated device streams status changes to a web UI as Server-Sent Events (SSE).
import http from "node:http";

const state = { "audio.muted": false, "audio.volume": 40, "call.state": "idle" };
let seq = 0;
const streams = new Set(); // open SSE responses, one per connected UI tab
let subscriberJoined;
const joined = new Promise((resolve) => (subscriberJoined = resolve));

function set(path, value) { // every state change on the device goes through here
  if (state[path] === value) return; // nothing changed: no event
  state[path] = value;
  const frame = `id: ${++seq}\nevent: status\ndata: ${JSON.stringify({ path, value })}\n\n`;
  for (const res of streams) res.write(frame);
}

const server = http.createServer((req, res) => {
  if (req.url !== "/api/v1/status/events") return res.writeHead(404).end();
  res.writeHead(200, { "Content-Type": "text/event-stream", "Cache-Control": "no-store" });
  res.write("retry: 2000\n: hello from the device\n\n"); // reconnect delay + a comment, no event
  streams.add(res);
  req.on("close", () => streams.delete(res));
  subscriberJoined();
});
await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve));

// The web UI's side: fetch the stream and parse it the way EventSource does.
const res = await fetch(`http://127.0.0.1:${server.address().port}/api/v1/status/events`);
console.log(res.status, res.headers.get("content-type"));
const reader = res.body.pipeThrough(new TextDecoderStream()).getReader();

await joined; // the device changes state only once the UI is listening
set("audio.muted", true);
set("audio.muted", true); // a second press of an already-muted button
set("call.state", "ringing");
set("call.state", "connected");

let buffer = "";
let dispatched = 0;
while (dispatched < 3) {
  const { value } = await reader.read();
  buffer += value;
  let end;
  while ((end = buffer.indexOf("\n\n")) !== -1) { // a blank line ends one event block
    const block = buffer.slice(0, end);
    buffer = buffer.slice(end + 2);
    const msg = { event: "message", data: [] };
    for (const line of block.split("\n")) {
      if (line.startsWith(":")) { console.log("comment  :", line.slice(1).trim()); continue; }
      const [field, ...rest] = line.split(": ");
      if (field === "data") msg.data.push(rest.join(": "));
      else msg[field] = rest.join(": ");
    }
    if (msg.retry) console.log("retry    :", msg.retry, "ms");
    if (msg.data.length === 0) continue; // no data lines: nothing to dispatch
    dispatched++;
    console.log(`event #${msg.id}: ${msg.event}`, msg.data.join("\n"));
  }
}
await reader.cancel();
server.closeAllConnections();
server.close();
console.log("device state:", JSON.stringify(state));
Result
200 text/event-stream
comment  : hello from the device
retry    : 2000 ms
event #1: status {"path":"audio.muted","value":true}
event #2: status {"path":"call.state","value":"ringing"}
event #3: status {"path":"call.state","value":"connected"}
device state: {"audio.muted":true,"audio.volume":40,"call.state":"connected"}
What’s happening
  1. The device keeps its status in one state object and changes it only through set(). That single doorway is what makes events trustworthy: every change produces exactly one event, and a "change" to the same value produces none, which is why the second set("audio.muted", true) (someone pressing mute on an already-muted room) sends nothing.
  2. The UI's fetch opens GET /api/v1/status/events. The device answers 200 with Content-Type: text/event-stream and never ends the response. Its first write is retry: 2000 (reconnect after 2 s if the stream drops) plus a comment line starting with :. Neither is an event: the parser prints the comment and the retry, and dispatches nothing because the block has no data: line.
  3. await joined makes the device change state only once the client is listening, so the output is the same on every run. Each set() writes a block of id:, event: and data: lines ending with a blank line. TCP may deliver several blocks in one chunk or split one across two, so the client appends to buffer and cuts on "\n\n"; it never assumes one read is one event.
  4. The client dispatches three events, #1, #2 and #3. The ids are the device's sequence numbers. A real EventSource remembers the last one and sends it back as Last-Event-ID when it reconnects, which is how the device knows what the client missed (Snapshot plus events builds on this).
  5. The client then cancels its reader and the device closes its connections. The final line confirms the client's view and the device agree: muted: true, call.state: "connected". Commands (mute, volume) would go to the device's REST API, not down this stream: SSE is one-way by design.
AdvancedThe same device over a WebSocket

The WebSocket version suits clients that also send a lot. The client subscribes to topics and sends commands on the same connection, so every message needs a type, and every command needs an id so its reply can be matched:

device-ws.mjs
// The same device over a WebSocket: the client subscribes to topics and sends commands on the same connection.
import { WebSocketServer } from "ws";

const state = { "audio.muted": false, "audio.volume": 40, "call.state": "idle" };
const writable = new Set(["audio.muted", "audio.volume"]);
let seq = 0;
const topicsOf = new Map(); // socket -> Set of topics it subscribed to

function set(path, value) {
  if (state[path] === value) return;
  state[path] = value;
  const frame = JSON.stringify({ type: "event", seq: ++seq, path, value });
  for (const [ws, topics] of topicsOf) if (topics.has(path.split(".")[0])) ws.send(frame);
}

const wss = new WebSocketServer({ port: 0, host: "127.0.0.1" });
wss.on("connection", (ws) => {
  topicsOf.set(ws, new Set());
  ws.send(JSON.stringify({ type: "hello", seq }));
  ws.on("message", (raw) => {
    const msg = JSON.parse(raw);
    if (msg.op === "subscribe") {
      for (const t of msg.topics) topicsOf.get(ws).add(t);
      ws.send(JSON.stringify({ type: "result", id: msg.id, ok: true }));
    } else if (msg.op === "set") {
      if (!writable.has(msg.path)) {
        ws.send(JSON.stringify({ type: "result", id: msg.id, ok: false, error: "read-only" }));
        return;
      }
      set(msg.path, msg.value); // publishes the event first...
      ws.send(JSON.stringify({ type: "result", id: msg.id, ok: true })); // ...then answers the command
      set("call.state", "ringing"); // meanwhile a call comes in (topic "call": this client didn't subscribe)
    }
  });
  ws.on("close", () => topicsOf.delete(ws));
});
await new Promise((resolve) => wss.on("listening", resolve));

// The client uses Node's built-in WebSocket, the same API a browser has.
const client = new WebSocket(`ws://127.0.0.1:${wss.address().port}/api/v1/ws`);
const inbox = [];
let wake = () => {};
client.onmessage = (e) => { inbox.push(JSON.parse(e.data)); wake(); };
const next = async () => { while (!inbox.length) await new Promise((r) => (wake = r)); return inbox.shift(); };

await new Promise((resolve) => (client.onopen = resolve));
console.log("<-", JSON.stringify(await next()));
const commands = [
  { op: "subscribe", id: 1, topics: ["audio"] },
  { op: "set", id: 2, path: "audio.muted", value: true },
  { op: "set", id: 3, path: "call.state", value: "idle" },
];
for (const cmd of commands) {
  console.log("->", JSON.stringify(cmd));
  client.send(JSON.stringify(cmd));
  let msg;
  do console.log("<-", JSON.stringify((msg = await next())));
  while (!(msg.type === "result" && msg.id === cmd.id)); // read until the reply to *this* command
}
client.close();
await new Promise((resolve) => (client.onclose = resolve));
wss.close();
console.log("device seq:", seq, "state:", JSON.stringify(state));
Result
<- {"type":"hello","seq":0}
-> {"op":"subscribe","id":1,"topics":["audio"]}
<- {"type":"result","id":1,"ok":true}
-> {"op":"set","id":2,"path":"audio.muted","value":true}
<- {"type":"event","seq":1,"path":"audio.muted","value":true}
<- {"type":"result","id":2,"ok":true}
-> {"op":"set","id":3,"path":"call.state","value":"idle"}
<- {"type":"result","id":3,"ok":false,"error":"read-only"}
device seq: 2 state: {"audio.muted":true,"audio.volume":40,"call.state":"ringing"}
What’s happening
  1. On connect the device sends hello with its current seq (0: nothing has happened yet). The client side uses Node's built-in WebSocket, the same API browsers have, so this client code would run in an admin page unchanged.
  2. subscribe with id: 1 adds the topic audio to this socket's set; the device answers {"type":"result","id":1,"ok":true}. From now on the device only sends this client events whose path starts with audio..
  3. set with id: 2 mutes the room. Inside the device, set() publishes the event first (seq 1) and then the result. The client reads messages until it sees the result for id: 2, and prints the event it passed on the way. That's the key habit with a shared channel: replies and events interleave, so a client matches replies by id and handles events whenever they come, never "the next message is my answer".
  4. Right after replying, the device takes an incoming call (call.state → ringing, seq 2). This client didn't subscribe to call, so nothing is sent: the final line shows the device at seq: 2 while this client only ever saw seq 1. Hold on to that: with filtered subscriptions, a gap in the global sequence is not a lost event, which matters in Designing device events.
  5. set with id: 3 tries to change call.state, which isn't writable over this API. The device answers ok: false, error: "read-only" instead of closing the connection: a bad command is a normal reply, not a protocol failure.
AdvancedLong polling for clients that only speak HTTP

Long polling is plain HTTP, so it works for clients that can't hold a WebSocket or parse a stream: older control systems, scripts, some partner middleware. The client sends the last sequence number it has; the device answers at once if there's news, or holds the request until there is:

long-poll.mjs
// Long polling on a device: GET /api/v1/status/changes?since=N answers at once if there is news after N,
// otherwise holds the request until the next change or until the wait runs out (then 204, ask again).
import http from "node:http";

const log = [];                 // [{ seq, path, value }], the device's recent changes
const waiting = new Set();      // held requests: { since, res, timer }
const WAIT_MS = 300;            // a real device would hold for ~25 s, under every proxy's idle timeout

function reply(res, since) {
  const changes = log.filter((c) => c.seq > since);
  res.writeHead(200, { "Content-Type": "application/json" }).end(JSON.stringify({ seq: log.at(-1)?.seq ?? 0, changes }));
}
function change(path, value) {
  log.push({ seq: log.length + 1, path, value });
  for (const w of waiting) { clearTimeout(w.timer); waiting.delete(w); reply(w.res, w.since); }
}

let held;
const heldPromise = new Promise((resolve) => (held = resolve));
const server = http.createServer((req, res) => {
  const since = Number(new URL(req.url, "http://device").searchParams.get("since") ?? 0);
  if (log.some((c) => c.seq > since)) return reply(res, since);   // news already: answer now
  const w = { since, res, timer: setTimeout(() => { waiting.delete(w); res.writeHead(204).end(); }, WAIT_MS) };
  waiting.add(w);
  held();
});
await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve));
const base = `http://127.0.0.1:${server.address().port}/api/v1/status/changes`;

async function poll(since) {
  const res = await fetch(`${base}?since=${since}`);
  const body = res.status === 200 ? await res.json() : null;
  console.log(`GET ?since=${since} -> ${res.status}`, body ? JSON.stringify(body) : "(no news, ask again)");
  return body ? body.seq : since;
}

change("audio.muted", true);                  // happened before the client asked
let since = await poll(0);                    // answered at once with seq 1
const pending = poll(since);                  // nothing new: the device holds this one
await heldPromise;
change("call.state", "ringing");              // the held request is answered with this change
since = await pending;
since = await poll(since);                    // nothing happens for WAIT_MS: 204, then the client asks again
server.close();
Result
GET ?since=0 -> 200 {"seq":1,"changes":[{"seq":1,"path":"audio.muted","value":true}]}
GET ?since=1 -> 200 {"seq":2,"changes":[{"seq":2,"path":"call.state","value":"ringing"}]}
GET ?since=2 -> 204 (no news, ask again)
What’s happening
  1. Before the client asks, the device has already recorded change 1 (audio.muted → true). The first poll, ?since=0, finds news immediately and gets 200 with seq: 1 and the change. The client stores seq as its new cursor.
  2. The second poll, ?since=1, finds nothing newer, so the device parks the request in waiting with a timer instead of answering. await heldPromise waits until the device has actually parked it, which keeps the output in order.
  3. The call comes in (change 2). change() answers every parked request straight away with the changes after its cursor: ?since=1 → 200 with seq: 2. Delivery is as fast as SSE's, using nothing but request and response.
  4. The third poll, ?since=2, gets nothing for WAIT_MS and the device answers 204 No Content: "nothing new, ask again". The demo waits 300 ms to stay quick; a real device would hold for about 25 s, below the idle timeout of every proxy or client library in the path.
  5. The cursor is what makes this correct. Changes that happen between one response and the next request aren't lost, because the next request says since=N and gets everything after N from the device's recent log. Without the cursor, long polling silently drops whatever happens in those gaps.
AdvancedPractical notes for each transport on a device
  • SSE heartbeats. Send a comment line (: hb) every 15 s or so when nothing else is happening. The WHATWG HTML standard's own authoring note suggests "every 15 seconds or so" (checked October 2026), so proxies and NAT boxes don't drop the idle connection.
  • SSE and the six-connection limit. Over HTTP/1.1, browsers allow only six connections per browser and domain, and each open EventSource holds one; MDN's SSE guide (checked October 2026) says Chrome and Firefox marked this "Won't fix". Many embedded web servers speak only HTTP/1.1, so an admin with several tabs open on the same device can run out, and the seventh tab's requests hang. Share one stream between tabs (a SharedWorker, or one leader tab rebroadcasting over BroadcastChannel), or serve HTTP/2.
  • Stopping SSE on purpose. Under the HTML standard, EventSource does not reconnect after a non-200 status or a wrong Content-Type; a 204 is the documented way to tell it to stop. So a device that refuses a stream with 503 stops the browser's automatic retries too; the page's code has to schedule the retry itself.
  • Auth. Neither EventSource nor the browser WebSocket constructor lets page code set an Authorization header. Same-origin cookies are sent, so a device UI usually authenticates streams with its session cookie. That makes checking the Origin header on the WebSocket handshake mandatory: otherwise any page the admin visits could open a socket to the device with the admin's cookie (cross-site WebSocket hijacking). See Securing the on-device web UI.
  • Self-signed certificates. A browser shows no warning page for a background fetch, EventSource or WebSocket; if it doesn't already trust the device's certificate for that origin, the connection just fails. Keep streams on the same origin as the admin page, so accepting the certificate once covers both.
A progress bar for a firmware update

The admin UI shows a progress bar while the device installs a firmware update, which ends with a reboot. Do you stream progress over SSE, a WebSocket, or poll?

Show answer

Poll the update's operation resource (for example GET /api/v1/operations/{id} every 2 s), or stream it if a stream already exists, but design for the stream dying, because it will.

What’s happening
  1. The update is a long-running operation: POST returns 202 Accepted and an operation resource whose progress and state the device updates (Long-running operations).
  2. Progress changes slowly (a few percent every few seconds), so a 2 s poll costs the device almost nothing, and polling one URL is the simplest client to write and test.
  3. The update ends with a reboot, which kills every stream and every in-flight request. A stream-based progress bar must reconnect anyway, and after the reboot it's talking to a new boot with new sequence numbers.
  4. So the robust design is the same either way: the UI treats "connection lost" during the reboot phase as expected, keeps probing with the reconnect rules in Reconnection and back-off, and reads the operation's final state (succeeded, failed, rolled-back) from the device once it's back. Streaming is a nicety; the operation resource is the source of truth.

#Designing device events

Added

An event is a promise you'll keep for years. Once a partner's control system reacts to call.state changing to connected, that event's name, its fields and its meaning are an API, exactly like a REST endpoint, and devices in the field will run old firmware long after you've shipped new ones (Rules for compatible change applies to events as much as to resources). So the design questions are the same as for any API, plus three that are specific to streams: what counts as an event, how consumers order them, and how they know they've missed one.

There are two kinds of event, and mixing them up causes most of the trouble. A state event says "this is now the value": audio.muted is true, call.state is connected. Only the latest one per path matters; if a client misses a few, reading the current state fixes it. An occurrence event says "this happened": call.ended with a reason, a button pressed, a peripheral plugged in, an error. Each one matters on its own, and reading the current state afterwards doesn't bring it back.

A mental model: device events are the numbered lines of a ship's log. Each line has a number and the time the officer read off the clock. Clocks drift and get corrected, so two lines can show times out of order, but the line numbers never lie: line 42 was written after line 41. And when the ship gets a new logbook (the device reboots), line numbers start again, which is why each book needs its own label.

State eventOccurrence event
Says"the value is now X""X happened"
Examplesaudio.muted, call.state, network.link, update.progresscall.ended (with reason), button.pressed, peripheral.connected, error.raised
Several in a rowonly the last matterseach one matters
Can be coalesced or throttledyes, keep the latest per pathno
Recovered by a snapshot after a gapyesno: keep them in a log resource (call history, event log) the client can page through

Every event goes out in the same envelope, so every consumer can order, de-duplicate and resume without knowing the event type:

FieldPurpose
bootAn id that changes on every boot. Sequence numbers are only comparable within one boot.
seqIncreases by one for every event the device publishes. The order consumers trust. (boot, seq) identifies an event uniquely.
typeA stable, namespaced name: status.changed, call.ended. Past tense for occurrences.
timeThe device's wall clock (ISO 8601, UTC), for humans and logs. Never used for ordering.
uptimeMsMilliseconds since boot, from a monotonic clock. Right for durations even when the wall clock jumps.
dataThe type-specific payload: for state events, the path and the new value.

If the events also leave the room (to a cloud service or a partner's message bus), the CloudEvents specification is the common envelope to map onto: its required attributes are id, source, specversion and type, with optional time, subject, datacontenttype and dataschema, and producers must make source plus id unique per event (CloudEvents 1.0, checked October 2026). source would be the device (/devices/<serial>) and id its boot:seq. The required and optional core attributes say nothing about ordering. CloudEvents does document an optional sequence extension attribute, a string that consumers compare lexicographically (checked October 2026), so if you map seq onto it, zero-pad it ("00000057") or "10" sorts before "9"; either way, numbering the events stays your job.

Here's that envelope on a device whose clock is wrong at first, as many are just after boot:

event-envelope.mjs
// One envelope for every device event, and why consumers order by seq, not by time.
const device = { boot: "b-7f3a", seq: 0, uptimeMs: 0, wallClock: Date.parse("2026-10-27T09:02:10Z") }; // RTC 2 min fast
const state = { "audio.muted": false, "call.state": "idle" };
const events = [];

function emit(type, data) {
  events.push({
    boot: device.boot,          // changes on every reboot: seq restarts with it
    seq: ++device.seq,          // the only order consumers trust
    type,
    time: new Date(device.wallClock).toISOString(), // wall clock: for humans and logs
    uptimeMs: device.uptimeMs,  // monotonic since boot: for durations
    data,
  });
}
function setState(path, value) { // a *state* event: "this is now the value"
  if (state[path] === value) return;
  state[path] = value;
  emit("status.changed", { path, value });
}
function tick(ms) { device.uptimeMs += ms; device.wallClock += ms; }

setState("call.state", "ringing");
tick(4_000);
setState("call.state", "connected");
const connectedAt = device.uptimeMs;                    // measure the call on the monotonic clock
tick(1_000);
device.wallClock = Date.parse("2026-10-27T09:00:15Z"); // NTP sync: the wall clock jumps back 2 minutes
setState("audio.muted", true);
setState("audio.muted", true);                          // no change: no event
tick(60_000);
setState("call.state", "idle");
emit("call.ended", { reason: "remote-hangup", durationMs: device.uptimeMs - connectedAt }); // an *occurrence*: "this happened"

const show = (e) => `seq ${e.seq}  ${e.time.slice(11, 19)}  up ${String(e.uptimeMs / 1000).padStart(2)}s  ${e.type} ${JSON.stringify(e.data)}`;
console.log("as published (by seq):");
events.forEach((e) => console.log(" ", show(e)));
console.log("sorted by time (wrong):");
[...events].sort((a, b) => a.time.localeCompare(b.time)).forEach((e) => console.log(" ", show(e)));
console.log("one envelope on the wire:", JSON.stringify(events[2]));
Result
as published (by seq):
  seq 1  09:02:10  up  0s  status.changed {"path":"call.state","value":"ringing"}
  seq 2  09:02:14  up  4s  status.changed {"path":"call.state","value":"connected"}
  seq 3  09:00:15  up  5s  status.changed {"path":"audio.muted","value":true}
  seq 4  09:01:15  up 65s  status.changed {"path":"call.state","value":"idle"}
  seq 5  09:01:15  up 65s  call.ended {"reason":"remote-hangup","durationMs":61000}
sorted by time (wrong):
  seq 3  09:00:15  up  5s  status.changed {"path":"audio.muted","value":true}
  seq 4  09:01:15  up 65s  status.changed {"path":"call.state","value":"idle"}
  seq 5  09:01:15  up 65s  call.ended {"reason":"remote-hangup","durationMs":61000}
  seq 1  09:02:10  up  0s  status.changed {"path":"call.state","value":"ringing"}
  seq 2  09:02:14  up  4s  status.changed {"path":"call.state","value":"connected"}
one envelope on the wire: {"boot":"b-7f3a","seq":3,"type":"status.changed","time":"2026-10-27T09:00:15.000Z","uptimeMs":5000,"data":{"path":"audio.muted","value":true}}
What’s happening
  1. The device boots with its real-time clock two minutes fast (09:02:10 instead of 09:00:10), a common state before the network time sync has run. Every event takes boot, the next seq, the wall-clock time and the monotonic uptimeMs.
  2. A call rings (seq 1) and connects 4 s later (seq 2). The code records connectedAt from uptimeMs, the monotonic clock, not from the wall clock.
  3. Then NTP syncs and the wall clock jumps back two minutes to 09:00:15. The mute that follows (seq 3) is stamped earlier than the two events before it, although it happened after them. The second setState("audio.muted", true) publishes nothing: no change, no event.
  4. A minute later the call ends: a state event (call.state → idle, seq 4) and an occurrence (call.ended, seq 5) with a reason and a duration. durationMs is 65,000 − 4,000 = 61,000 from the monotonic clock; computed from wall-clock times it would come out as 09:01:15 − 09:02:14, about −59 s.
  5. Sorting by time puts ringing and connected last, so a consumer replaying in that order would end with call.state = "connected" for a call that has ended. Sorting by seq is always right. That's the rule to state in an interview: sequence numbers for order, monotonic time for durations, wall-clock time for display.
  6. The last line is one envelope as it goes over the wire, with the full ISO time. Every transport in this chapter (SSE data:, a WebSocket message, a long-poll response, a Telnet notification line) carries this same object.
AdvancedValidating events as a tolerant reader

The events are a contract, so write them down as a schema, the same way the REST resources have an OpenAPI document (an AsyncAPI document is the usual equivalent for message channels). Then the consumer validates what it understands and skips what it doesn't, because a newer firmware will add event types and fields the consumer has never seen. Here's the consumer side with zod 4.6:

event-schema.mjs
// The event contract as a schema (zod 4), and a tolerant reader: validate what you know, skip what you don't.
import { z } from "zod";

const envelope = { boot: z.string(), seq: z.number().int().positive(), time: z.iso.datetime() };
const DeviceEvent = z.discriminatedUnion("type", [
  z.object({ ...envelope, type: z.literal("status.changed"), data: z.object({ path: z.string(), value: z.unknown() }) }),
  z.object({ ...envelope, type: z.literal("call.ended"), data: z.object({ reason: z.enum(["local-hangup", "remote-hangup", "network", "error"]), durationMs: z.number() }) }),
  z.object({ ...envelope, type: z.literal("peripheral.connected"), data: z.object({ kind: z.enum(["camera", "microphone", "touch-panel"]), id: z.string() }) }),
]);
const known = new Set(DeviceEvent.options.map((o) => o.shape.type.value));

function receive(raw) {
  const msg = JSON.parse(raw);
  if (!known.has(msg.type)) return `skip   seq ${msg.seq}: unknown type "${msg.type}" (newer firmware?)`;
  const r = DeviceEvent.safeParse(msg);
  if (!r.success) return `reject seq ${msg.seq}: ${r.error.issues.map((i) => `${i.path.join(".")}: ${i.message}`).join("; ")}`;
  return `apply  seq ${r.data.seq}: ${r.data.type} ${JSON.stringify(r.data.data)}`;
}

const t = "2026-10-27T09:00:15Z";
const wire = [
  { boot: "b-7f3a", seq: 7, time: t, type: "status.changed", data: { path: "audio.muted", value: true } },
  { boot: "b-7f3a", seq: 8, time: t, type: "peripheral.connected", data: { kind: "camera", id: "usb-1", vendor: "x" } },
  { boot: "b-7f3a", seq: 9, time: t, type: "ai.framing.changed", data: { mode: "speaker" } },
  { boot: "b-7f3a", seq: 10, time: t, type: "call.ended", data: { reason: "hung-up", durationMs: "61s" } },
];
for (const e of wire) console.log(receive(JSON.stringify(e)));
console.log("known types:", [...known].join(", "));
Result
apply  seq 7: status.changed {"path":"audio.muted","value":true}
apply  seq 8: peripheral.connected {"kind":"camera","id":"usb-1"}
skip   seq 9: unknown type "ai.framing.changed" (newer firmware?)
reject seq 10: data.reason: Invalid option: expected one of "local-hangup"|"remote-hangup"|"network"|"error"; data.durationMs: Invalid input: expected number, received string
known types: status.changed, call.ended, peripheral.connected
What’s happening
  1. DeviceEvent is a discriminated union on type: zod looks at type first and validates the rest against that one branch. The shared envelope fields (boot, seq, time) are spread into each branch. known is built from the schema itself, so the list of understood types can't drift from the validation.
  2. seq 7 is a status.changed and passes: apply.
  3. seq 8 is a peripheral.connected with an extra field vendor. It passes, and the printed data no longer has vendor: zod's z.object strips unknown keys by default. That's the tolerant-reader half of compatibility: a new firmware can add fields without breaking old consumers.
  4. seq 9 has a type this consumer doesn't know, ai.framing.changed, presumably from newer firmware. The consumer skips it rather than failing the stream. Rejecting it would mean an old web UI breaks the day a device is upgraded.
  5. seq 10 is a known type with bad data, and it's rejected with zod's real messages: Invalid option: expected one of "local-hangup"|"remote-hangup"|"network"|"error" and Invalid input: expected number, received string. A rejected event is a gap in what the client knows, so the right reaction is to resync state (next topic), not to carry on as if nothing happened.
  6. If you changed one thing: the strict z.enum on reason is itself a compatibility trap. When firmware adds "transferred", every event with that reason is rejected. For values that will grow, accept any string and map unknown ones to "other" in the UI; keep strict enums for values the consumer truly can't handle.
AdvancedNaming, granularity and payload size
  • Name by resource, then what happened. call.state changes, call.ended, peripheral.connected, update.progress. The names mirror the REST resources, so status.changed with path: "audio.muted" corresponds to GET /api/v1/audio and the web UI uses one model for both.
  • Fat for small state, thin for big state. Put the new value in the event when it's small (a boolean, a volume, a call state): the device is the source of truth, and making 8 clients each GET the resource after every change multiplies the device's work. For big resources (the directory, a log, a full configuration), send a thin event (directory.changed with a new version number) and let clients that care fetch it, conditionally with If-None-Match (see Optimistic concurrency with ETags).
  • Don't stream measurements as events. Audio levels, CPU load and network statistics change continuously. Sample them (2 Hz for a UI meter) on a separate, droppable channel, or serve them on request; never put 30 updates a second into the same sequence as call.ended (the cost is worked out in Constrained devices and many clients).
  • Never repurpose. A field keeps its meaning forever; a new meaning gets a new field or a new type. Old consumers will read the old meaning into it.
Two events, one call

A partner's control system turns the room's lights up when a call ends. It receives status.changed with call.state: "idle" and then call.ended with reason: "remote-hangup". Which one should it react to, and what happens to each if the control system's connection drops for 30 seconds right then?

Show answer

React to the state (call.state becoming idle) for "are we in a call?", and use the occurrence (call.ended) only for things that need the reason or must happen once per call (logging, a "call dropped" alert).

What’s happening
  1. call.state is a state event: only its latest value matters. If the connection drops and the control system misses it, its reconnect snapshot shows call.state: "idle", the lights logic compares "was in a call, now idle", and reacts correctly, just 30 s late.
  2. call.ended is an occurrence: a snapshot can't bring it back, because "the call ended because the remote side hung up" isn't part of the current state. If the replay ring still holds it, the control system gets it on resume; if not, it's gone from the stream.
  3. So anything that must not be missed is also kept as state the client can query: the call ends up in the call history (a log resource with ids), and a consumer that resyncs can page through history since its last-known entry.
  4. The design rule: drive behaviour from state events (they self-heal), use occurrences for detail, and back every occurrence that matters with a queryable log.

#Snapshot plus events

Added

Events alone can't tell a newcomer anything: a web UI that opens at 10:00 doesn't care that the volume changed at 09:58, it needs to know what the volume is. A snapshot alone goes stale the moment it's taken. Every real-time client therefore does both: load a snapshot of the current state, stamped with the sequence number it reflects, then apply only the events after that number. The same mechanism handles every later disconnection: ask for the events you missed, and if the device can't provide them, take a new snapshot.

A mental model: you arrive late to a chess game. You look at the board (the snapshot, "position after move 41") and then follow the moves from 42 on. If you step out for a minute, a friend can tell you the moves you missed, as long as they remember them (the device's replay ring). If you were gone too long, you look at the board again. And if the players have started a new game (the device rebooted), your move numbers belong to the old game: you must look at the new board, whatever number you remember.

First connect device client 41 snapshot@41 42 buffered while waiting: #41, #42 1. subscribe 2. GET /status 3. load @41 skip #41 (in snapshot) apply #42 → live Reconnect ?boot=b-7f3a&since=44 ring holds #43 … #46 replay #45, #46 → live ?boot=b-7f3a&since=46 ring holds #49 … #52: #47, #48 gone new snapshot@52 → live
Subscribe first, so nothing between the snapshot and the subscription is lost; then sequence numbers decide what to skip, apply or resync.

The client's rules fit in five lines:

  1. The snapshot carries boot and the seq it reflects.
  2. Subscribe first, then fetch the snapshot, buffering events that arrive in between (or get both on one channel, below).
  3. For each event: seq ≤ mine → skip (already included); seq = mine + 1 → apply; anything bigger → gap, resync.
  4. On reconnect, send (boot, since). The device replays from its ring if it can, otherwise sends a fresh snapshot.
  5. A different boot means a different sequence: always a fresh snapshot.

Here's the whole cycle in memory, including the wrong order and what it does to the state:

snapshot-events.mjs
// Snapshot plus events: subscribe first, fetch a snapshot stamped with a seq, then apply only events after it.
const RING = 4; // the device keeps only its last 4 events for clients that reconnect
const device = { seq: 40, state: { "audio.muted": false, "audio.volume": 40, "call.state": "idle" }, ring: [], subs: new Set() };

function change(path, value) {
  device.state[path] = value;
  const e = { seq: ++device.seq, path, value };
  device.ring.push(e);
  if (device.ring.length > RING) device.ring.shift();
  for (const deliver of device.subs) deliver(e);
}
const snapshot = () => ({ seq: device.seq, state: { ...device.state } });
function eventsSince(seq) { // null means "can't fill the gap": the client must take a new snapshot
  const oldest = device.ring[0]?.seq ?? device.seq + 1;
  return seq < oldest - 1 ? null : device.ring.filter((e) => e.seq > seq);
}

function makeClient(name) {
  const c = { seq: 0, state: null, buffer: [] };
  const log = (...a) => console.log(`[${name}]`, ...a);
  c.apply = (e) => {
    if (e.seq <= c.seq) return log(`skip #${e.seq} (already in snapshot ${c.seq})`);
    if (e.seq !== c.seq + 1) { log(`gap: have ${c.seq}, got #${e.seq} -> resync`); return c.load(snapshot()); }
    c.state[e.path] = e.value;
    c.seq = e.seq;
    log(`apply #${e.seq} ${e.path}=${e.value}`);
  };
  c.load = (snap) => {
    c.state = snap.state;
    c.seq = snap.seq;
    log(`snapshot at seq ${snap.seq}`);
    for (const e of c.buffer.splice(0)) c.apply(e); // events that arrived while the snapshot was in flight
  };
  c.onEvent = (e) => (c.state ? c.apply(e) : c.buffer.push(e));
  return c;
}
const same = (c) => (JSON.stringify(c.state) === JSON.stringify(device.state) ? "matches device" : "WRONG: " + JSON.stringify(c.state));

// 1. Correct order: subscribe, then ask for the snapshot. Two events happen around the request.
const ui = makeClient("ui");
device.subs.add(ui.onEvent);
change("audio.volume", 45);        // #41: before the device builds the snapshot (so it's included)
const snap = snapshot();           // the device answers GET /status with seq 41
change("audio.muted", true);       // #42: after the snapshot was built, before it reaches the client
ui.load(snap);
console.log("  ui:", same(ui));

// 2. Wrong order: snapshot first, subscribe after. The event in between is lost, and nothing notices.
const naive = makeClient("naive");
naive.apply = (e) => { naive.state[e.path] = e.value; naive.seq = e.seq; }; // ignores seq
naive.load(snapshot());            // seq 42
change("call.state", "ringing");   // #43: before the subscription exists
device.subs.add(naive.onEvent);
change("audio.volume", 50);        // #44
console.log("  naive:", same(naive));

// 3. The ui client drops off the network; events #45..#46 happen. It resumes from its seq.
device.subs.delete(ui.onEvent);
change("call.state", "connected"); // #45
change("audio.muted", false);      // #46
console.log(`[ui] resume since ${ui.seq}: device has #${device.ring[0].seq}..#${device.seq}`);
for (const e of eventsSince(ui.seq) ?? []) ui.apply(e);
device.subs.add(ui.onEvent);
console.log("  ui:", same(ui));

// 4. It drops again, for longer: six events, more than the ring holds.
device.subs.delete(ui.onEvent);
for (const v of [51, 52, 53, 54, 55, 56]) change("audio.volume", v); // #47..#52
const replay = eventsSince(ui.seq);
console.log(`[ui] resume since ${ui.seq}: device has #${device.ring[0].seq}..#${device.seq} ->`, replay ? "replay" : "too old, new snapshot");
if (!replay) ui.load(snapshot());
console.log("  ui:", same(ui));
Result
[ui] snapshot at seq 41
[ui] skip #41 (already in snapshot 41)
[ui] apply #42 audio.muted=true
  ui: matches device
[naive] snapshot at seq 42
[ui] apply #43 call.state=ringing
[ui] apply #44 audio.volume=50
  naive: WRONG: {"audio.muted":true,"audio.volume":50,"call.state":"idle"}
[ui] resume since 44: device has #43..#46
[ui] apply #45 call.state=connected
[ui] apply #46 audio.muted=false
  ui: matches device
[ui] resume since 46: device has #49..#52 -> too old, new snapshot
[ui] snapshot at seq 52
  ui: matches device
What’s happening
  1. Correct order. The ui client subscribes while the device is at seq 40, then asks for a snapshot. Change #41 (volume 45) happens before the device builds the snapshot, so the snapshot is stamped seq 41 and already includes it. Change #42 (mute) happens after the snapshot was built but before it reached the client. Both events wait in buffer because the client has no state yet.
  2. load() installs the snapshot and drains the buffer: #41 is skipped (41 ≤ 41, already in the snapshot) and #42 is applied (42 = 41 + 1). The client matches the device. Without the buffer, #42 would have been lost; without the skip, #41 would have been applied twice, which is harmless for a volume but not for a counter or an occurrence.
  3. Wrong order. The naive client takes its snapshot first (seq 42) and subscribes afterwards. #43, the incoming call, happens in between, before the subscription exists, so this client never receives it. It then gets #44 and, ignoring sequence numbers, applies it. (The [ui] lines in between are the first client, still subscribed, applying both events correctly.) Its final state says call.state: "idle" while the room is ringing: WRONG, and nothing in the client knows. With the rule from step 2, #44 after 42 would have been a gap and triggered a resync, which is why sequence checking is not optional.
  4. Short outage. The ui client drops off the network after #44. Two events happen (#45, #46). On reconnect it asks for events since 44; the ring holds #43..#46, so the device replays #45 and #46, each passing the seq = mine + 1 check. No snapshot was needed, which on a device means less work and less data.
  5. Long outage. It drops again and six volume changes happen (#47..#52), more than the 4-event ring holds. Now the ring starts at #49, and the client needs #47. eventsSince(46) returns null ("can't fill the gap"), so the client loads a new snapshot at seq 52 and matches the device again.
  6. The size of the ring is a trade: bigger means more outages are healed by a cheap replay, smaller means less device memory. Size it to the outages you expect: a 30 s Wi-Fi blip at 2 events a second needs about 60 events.
AdvancedSnapshot and events on one socket

The subscribe-then-fetch dance exists because the snapshot and the events travel on different channels. If the device sends the snapshot as the first message on the stream itself, and picks between replay and snapshot in the same step as adding the subscriber, there's no window for an event to slip through:

ws-snapshot.mjs
// Snapshot plus events over one WebSocket: on connect the device sends a replay or a snapshot, then live events.
// Choosing, sending and subscribing happen in one synchronous step, so no event can slip in between.
import { WebSocketServer } from "ws";

const BOOT = "b-91c2";
const RING = 3;
const dev = { seq: 0, state: { "audio.muted": false, "audio.volume": 40 }, ring: [], subs: new Set() };
function change(path, value) {
  dev.state[path] = value;
  const e = { type: "event", boot: BOOT, seq: ++dev.seq, path, value };
  dev.ring.push(e);
  if (dev.ring.length > RING) dev.ring.shift();
  for (const ws of dev.subs) ws.send(JSON.stringify(e));
}

const wss = new WebSocketServer({ port: 0, host: "127.0.0.1" });
wss.on("connection", (ws, req) => {
  const q = new URL(req.url, "ws://device").searchParams;
  const since = Number(q.get("since"));
  const oldest = dev.ring[0]?.seq ?? dev.seq + 1;
  const canReplay = q.get("boot") === BOOT && since >= oldest - 1 && since <= dev.seq;
  if (canReplay) for (const e of dev.ring) if (e.seq > since) ws.send(JSON.stringify(e));
  if (!canReplay) ws.send(JSON.stringify({ type: "snapshot", boot: BOOT, seq: dev.seq, state: dev.state }));
  dev.subs.add(ws); // same tick as the send above: nothing can happen in between
  ws.on("message", () => ws.send(JSON.stringify({ type: "pong" }))); // lets the test know it has everything
  ws.on("close", () => dev.subs.delete(ws));
});
await new Promise((resolve) => wss.on("listening", resolve));
const url = `ws://127.0.0.1:${wss.address().port}/api/v1/ws`;

async function session(label, query, whileConnected = () => {}) {
  const ws = new WebSocket(`${url}${query}`);
  const got = [];
  let ponged;
  const pong = new Promise((resolve) => (ponged = resolve));
  ws.onmessage = (m) => { const msg = JSON.parse(m.data); msg.type === "pong" ? ponged() : got.push(msg); };
  await new Promise((resolve) => (ws.onopen = resolve));
  whileConnected();
  ws.send("ping"); // the pong comes back after everything the device sent before it
  await pong;
  ws.close();
  await new Promise((resolve) => (ws.onclose = resolve));
  console.log(`${label.padEnd(30)} ${query.padEnd(24)} ->`, got.map((m) => (m.type === "snapshot" ? `snapshot@${m.seq}` : `#${m.seq}`)).join(" "));
  return got.at(-1).seq;
}

change("audio.volume", 45); // #1
change("audio.muted", true); // #2
let seq = await session("first connect", "", () => change("audio.volume", 50)); // snapshot@2, then #3 live
change("audio.volume", 55); // #4 while the client is away
seq = await session("reconnect, gap in the ring", `?boot=${BOOT}&since=${seq}`);
for (const v of [60, 65, 70, 75]) change("audio.volume", v); // #5..#8; the ring keeps #6..#8
await session("reconnect, gap too big", `?boot=${BOOT}&since=${seq}`);
await session("reconnect after a reboot", `?boot=b-7f3a&since=8`);
wss.close();
Result
first connect                                           -> snapshot@2 #3
reconnect, gap in the ring     ?boot=b-91c2&since=3     -> #4
reconnect, gap too big         ?boot=b-91c2&since=4     -> snapshot@8
reconnect after a reboot       ?boot=b-7f3a&since=8     -> snapshot@8
What’s happening
  1. On each connection the device reads boot and since from the URL, decides canReplay (same boot, since not older than the ring allows, and not newer than the device's own seq), sends either the missed events or one snapshot message, and adds the socket to dev.subs, all in one synchronous callback. Node runs one callback at a time, so no change() can run in the middle. On a multi-threaded device the same rule holds with a lock: choosing, sending and subscribing are one atomic step.
  2. First connect (no query): a snapshot at seq 2. While connected, #3 happens and arrives live, right after the snapshot. The client sends ping and waits for the pong so it knows everything before it has arrived; that's how the demo stays deterministic, and it's also a handy "flush" for tests.
  3. Reconnect, gap in the ring: the client had #3, missed #4, and reconnects with since=3. The ring (size 3) holds #2..#4, so the device replays only #4.
  4. Reconnect, gap too big: four more changes (#5..#8) leave the ring at #6..#8. The client says since=4, needs #5, which is gone, so it gets snapshot@8.
  5. Reconnect after a reboot: the client sends an old boot id, b-7f3a. Its since=8 happens to be a valid number for the new boot too, and replaying from it would be silently wrong, since #8 of the old boot and #8 of the new boot are different events. The boot check forces snapshot@8. The since <= dev.seq check catches the other version of this bug: a client from a long-lived old boot asking for since=900 from a device that rebooted and is only at seq 12.
AdvancedCarrying boot and seq in SSE ids

With SSE you don't even write the reconnect logic: EventSource remembers the last id: it saw and sends it in a Last-Event-ID header when it reconnects. So make the id carry both halves, id: b-91c2:57, and have the device parse it on connect. The WHATWG HTML standard (checked October 2026) also lets the server clear the remembered id by sending an id: line with no value, after which the next reconnect sends no Last-Event-ID at all, which is a clean way to force a fresh snapshot.

AdvancedWhat a snapshot costs on a device
  • Keep it one request. If the UI needs audio, call, camera and network status, offer one GET /api/v1/status (or the stream's first message) that returns all of it with one seq. Four separate requests each reflect a different moment, and no single seq describes them together.
  • Make it cheap to build. The device should read its in-memory state, not query each subsystem when a client connects. A dozen tabs reconnecting after a Wi-Fi blip means a dozen snapshots at once.
  • Version it like the resources. A snapshot is a representation; new firmware adds fields to it, and old clients must ignore them.
Which events does the client apply?

A client has a snapshot at boot b-91c2, seq 41. It then receives, in this order: #42, #42 (a duplicate after a proxy retry), #44, #45. What does it do with each, and what state does it end with?

Show answer

Apply #42, skip the duplicate, stop at #44 (gap), resync, and then handle #45 against the new snapshot.

What’s happening
  1. #42: 42 = 41 + 1, so apply it. The client is now at seq 42.
  2. #42 again: 42 ≤ 42, so skip it. Sequence numbers make delivery idempotent: duplicates are harmless.
  3. #44: 44 > 42 + 1, so #43 is missing. The client must not apply #44 (it might depend on #43, and the client's state is already incomplete). It asks the device for events since 42, or a snapshot.
  4. Say the device answers with a snapshot at seq 45, because the device had moved on by the time it built it. Then the #45 that was waiting is skipped (45 ≤ 45) and the client is correct at seq 45. If it had answered with a replay of #43 and #44, the client would apply them and then apply #45 normally.

#Reconnection and back-off

Added

A web app's servers rarely vanish for a minute; a device does it routinely. It reboots after a firmware update or a setting that needs a restart (Apply, restart and rollback flows). It's power-cycled by someone in the room. Its network drops: a cable is moved, Wi-Fi roams, a switch restarts, DHCP hands it a new address. Its web server process restarts while the rest of the device keeps running. Every one of those breaks every stream, and the admin's page must cope without anyone pressing reload.

A good client does six things: it notices quickly that the connection is dead; it keeps showing the last known state, clearly marked as stale; it tells a planned restart from a failure; it retries at a sensible pace, without hammering a device that's still booting; it comes back by itself; and when it's back, it checks what changed (a new boot, new firmware, a lost session) before trusting anything it had. The browser-side mechanics of backoff and jitter for web-scale servers are in the System Design course's Real-time updates in the client; this topic is about the device-shaped parts.

A mental model: a pilot who loses radio contact with the tower. She keeps flying on the last instructions (the last known state, greyed out), retries the frequency at sensible intervals rather than shouting continuously, and when the tower answers, the first question is "what changed while I was gone?", not "carry on as before".

Live Restarting probe every 5 s Reconnecting backoff + jitter Unreachable banner, keep trying 1012 / 1001 1006 / silence after 60 s Connected read hello same boot resume → live new boot snapshot → live new firmware reload the page
How the connection was lost picks the retry pace; what the hello message says decides how much of the old state survives.

Two scenarios, simulated with a fake clock (and a seeded random number generator for the jitter), so the timeline is the same on every run:

reconnect-sim.mjs
// How a web UI reconnects to a device: a planned reboot (firmware update) and an unplanned network drop.
// Fake clock + seeded PRNG: the timeline is the same on every run.
function mulberry32(seed) {
  return () => {
    seed = (seed + 0x6d2b79f5) | 0;
    let t = Math.imul(seed ^ (seed >>> 15), 1 | seed);
    t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
  };
}
const PROBE_MS = 5_000;          // planned restart: probe at a steady pace, the device told us it's coming back
const BASE_MS = 1_000, CAP_MS = 30_000, CONNECT_TIMEOUT_MS = 5_000;
const UNREACHABLE_AFTER_MS = 60_000;

function run(title, device, closeAt, closeCode) {
  console.log(`--- ${title}`);
  const rand = mulberry32(42);
  const ui = { boot: "b-7f3a", firmware: "5.0.1", seq: 57, mode: "live" };
  let now = closeAt, attempt = 0;
  const log = (msg) => console.log(`t=${(now / 1000).toFixed(1).padStart(6)}s  ${`[${ui.mode}]`.padEnd(14)} ${msg}`);
  const planned = closeCode === 1012 || closeCode === 1001; // Service Restart / Going Away
  ui.mode = planned ? "restarting" : "reconnecting";
  log(`closed ${closeCode}${planned ? " (device said it is restarting)" : " (no close frame: connection lost)"}; showing last known state`);
  let retryAfterMs = 0, refused = 0;
  for (;;) {
    const backoff = planned ? PROBE_MS : Math.floor(rand() * Math.min(CAP_MS, BASE_MS * 2 ** attempt)); // full jitter
    const wait = Math.max(backoff, retryAfterMs);
    now += wait;
    attempt++;
    const r = device(now);
    if (r.kind === "refused") { // log the first refusal of a run, then keep probing quietly
      if (!refused++) log(`attempt ${attempt}: connection refused (nothing listening yet), probing every ${wait / 1000} s`);
      continue;
    }
    refused = 0;
    if (r.kind === "timeout") now += CONNECT_TIMEOUT_MS;
    if (!planned && ui.mode !== "unreachable" && now - closeAt >= UNREACHABLE_AFTER_MS) {
      ui.mode = "unreachable";
      log(`${Math.round((now - closeAt) / 1000)} s without the device: show 'Device unreachable', keep retrying`);
    }
    if (r.kind === "timeout") { log(`attempt ${attempt} after ${wait} ms: no answer in 5 s`); continue; }
    if (r.kind === "503") {
      retryAfterMs = r.retryAfter * 1000;
      log(`attempt ${attempt}: 503, Retry-After ${r.retryAfter} s (web server up, services starting)`);
      continue;
    }
    log(`attempt ${attempt}: connected, boot ${r.boot}, firmware ${r.firmware}`);
    if (r.boot === ui.boot) { ui.mode = "live"; log(`same boot: resume since seq ${ui.seq}`); return; }
    ui.mode = "resyncing";
    log("new boot id: seq restarted, drop old state, load a snapshot");
    if (r.firmware !== ui.firmware) log(`firmware ${ui.firmware} -> ${r.firmware}: the page's code is stale, reload the web UI`);
    return;
  }
}

// 1. The admin started a firmware update. The device closes with 1012, reboots, comes back with a new boot id.
run("planned reboot for a firmware update", (t) =>
  t < 70_000 ? { kind: "refused" } : t < 80_000 ? { kind: "503", retryAfter: 8 } : { kind: "ok", boot: "b-91c2", firmware: "5.1.0" },
  12_000, 1012);

// 2. Someone unplugs the network switch for 90 s. The device keeps running; the UI just can't reach it.
run("unplanned network drop", (t) => (t < 110_000 ? { kind: "timeout" } : { kind: "ok", boot: "b-7f3a", firmware: "5.0.1" }), 20_000, 1006);
Result
--- planned reboot for a firmware update
t=  12.0s  [restarting]   closed 1012 (device said it is restarting); showing last known state
t=  17.0s  [restarting]   attempt 1: connection refused (nothing listening yet), probing every 5 s
t=  72.0s  [restarting]   attempt 12: 503, Retry-After 8 s (web server up, services starting)
t=  80.0s  [restarting]   attempt 13: connected, boot b-91c2, firmware 5.1.0
t=  80.0s  [resyncing]    new boot id: seq restarted, drop old state, load a snapshot
t=  80.0s  [resyncing]    firmware 5.0.1 -> 5.1.0: the page's code is stale, reload the web UI
--- unplanned network drop
t=  20.0s  [reconnecting] closed 1006 (no close frame: connection lost); showing last known state
t=  25.6s  [reconnecting] attempt 1 after 601 ms: no answer in 5 s
t=  31.5s  [reconnecting] attempt 2 after 896 ms: no answer in 5 s
t=  39.9s  [reconnecting] attempt 3 after 3409 ms: no answer in 5 s
t=  50.3s  [reconnecting] attempt 4 after 5357 ms: no answer in 5 s
t=  58.1s  [reconnecting] attempt 5 after 2797 ms: no answer in 5 s
t=  78.9s  [reconnecting] attempt 6 after 15797 ms: no answer in 5 s
t=  92.1s  [unreachable]  72 s without the device: show 'Device unreachable', keep retrying
t=  92.1s  [unreachable]  attempt 7 after 8196 ms: no answer in 5 s
t= 110.8s  [unreachable]  attempt 8: connected, boot b-7f3a, firmware 5.0.1
t= 110.8s  [live]         same boot: resume since seq 57
What’s happening
  1. Planned reboot. The admin started a firmware update. At t=12s the device closes the stream with code 1012, "Service Restart" (registered in IANA's WebSocket close code registry, checked October 2026; an SSE stream would get a final event: restarting instead). The UI knows a restart is coming, so it switches to restarting, keeps the last known state on screen, and probes at a steady 5 s: there's no point backing off when the device has told you it will be back, and one admin's tab isn't a thundering herd.
  2. From t=17s every attempt is refused: the device is mid-reboot and nothing is listening. The simulation logs the first refusal and probes quietly until attempt 12 at t=72s, when the web server is up but its services aren't: 503 with Retry-After: 8. The client honours it (wait = max(backoff, retryAfter)), so attempt 13 comes at t=80s, not t=77s.
  3. Attempt 13 connects. The hello says boot b-91c2, not the b-7f3a the UI had, so sequence numbers have restarted: drop the old state and load a snapshot. The firmware also changed, 5.0.1 → 5.1.0, which means the JavaScript in this tab, served by the old firmware, may not match the new API. The right move is to reload the page (after saving anything the admin was typing).
  4. Unplanned drop. At t=20s the network goes away. There's no close frame, so the browser reports 1006 ("Abnormal Closure", which page code sees for every connection that died without a close handshake). The UI goes to reconnecting and uses exponential backoff with full jitter: attempt n waits a random time between 0 and 1 s × 2ⁿ⁻¹ (1 s, 2 s, 4 s…), capped at 30 s. The draws (601, 896, 3409 ms…) come from mulberry32(42).
  5. With the network down, attempts don't fail fast; each one waits for the 5 s connect timeout, which is why the timeline moves faster than the waits alone suggest. At t=92.1s, 72 s after the drop, the UI changes to unreachable and says so plainly, but keeps retrying: devices come back, and nobody should have to reload to notice.
  6. Attempt 8 at t=110.8s connects, and the hello has the same boot id: the device never restarted, only the network did. So the UI resumes from seq 57 (replay from the ring, or a snapshot if the ring can't cover it) and goes live. Two outages, two different pictures for the admin ("Restarting for update" vs "Device unreachable") and two different recovery paths, all decided by the close code and the hello.
AdvancedA real socket across a reboot

The same flow with real sockets: the device sends a reason with its close, stops listening, and comes back on the same port with a new boot id. The client is browser-style code, Node's built-in WebSocket:

ws-reboot.mjs
// A real WebSocket across a simulated reboot: the device closes with 1012, stops listening, and comes back on
// the same port with a new boot id. The client is browser-style code (Node's built-in WebSocket).
import { WebSocketServer } from "ws";

let port = 0;
function bootDevice(boot) {
  const wss = new WebSocketServer({ port, host: "127.0.0.1" });
  wss.on("connection", (ws) => ws.send(JSON.stringify({ type: "hello", boot, seq: 0 })));
  return new Promise((resolve) => wss.on("listening", () => { port = wss.address().port; resolve(wss); }));
}

function connect(attempt) { // resolves with what happened: an open socket and its hello, or a failure
  return new Promise((resolve) => {
    const ws = new WebSocket(`ws://127.0.0.1:${port}/api/v1/ws`);
    let opened = false;
    ws.onopen = () => (opened = true);
    ws.onmessage = (m) => resolve({ ws, hello: JSON.parse(m.data) });
    ws.onclose = (e) => !opened && resolve({ failed: `close ${e.code}${e.reason ? ` "${e.reason}"` : ""}`, attempt });
  });
}

let device = await bootDevice("b-7f3a");
let { ws, hello } = await connect(0);
console.log("connected:", JSON.stringify(hello));
const closed = new Promise((resolve) => (ws.onclose = (e) => resolve(e)));

// Reboot: tell every client why, then stop listening (the OS would refuse connections now).
for (const client of device.clients) client.close(1012, "rebooting for update");
const e = await closed;
console.log(`client saw close ${e.code} "${e.reason}" wasClean=${e.wasClean} -> expect the device back, keep last state`);
await new Promise((resolve) => device.close(resolve));

let attempt = 0;
for (;;) {
  await new Promise((resolve) => setTimeout(resolve, 50 * 2 ** attempt)); // back off (short, for the demo)
  attempt++;
  if (attempt === 3) device = await bootDevice("b-91c2"); // the device finishes booting before the 3rd try
  const r = await connect(attempt);
  if (r.failed) { console.log(`attempt ${attempt}: ${r.failed} (what a browser sees for "connection refused")`); continue; }
  console.log(`attempt ${attempt}: connected:`, JSON.stringify(r.hello));
  console.log(r.hello.boot !== hello.boot ? `boot ${hello.boot} -> ${r.hello.boot}: rebooted, take a fresh snapshot` : "same boot: resume");
  r.ws.close(1000);
  await new Promise((resolve) => (r.ws.onclose = resolve));
  break;
}
device.close();
Result
connected: {"type":"hello","boot":"b-7f3a","seq":0}
client saw close 1012 "rebooting for update" wasClean=true -> expect the device back, keep last state
attempt 1: close 1006 (what a browser sees for "connection refused")
attempt 2: close 1006 (what a browser sees for "connection refused")
attempt 3: connected: {"type":"hello","boot":"b-91c2","seq":0}
boot b-7f3a -> b-91c2: rebooted, take a fresh snapshot
What’s happening
  1. bootDevice("b-7f3a") starts a ws server on a free port and remembers the port, so the "rebooted" device can listen on the same one, as a real device keeps its address and port. Its hello carries the boot id.
  2. To reboot, the device calls close(1012, "rebooting for update") on each client. The client's close event reports code 1012, the reason text, and wasClean=true: a proper close handshake happened. That's the signal for "expected, switch to restarting".
  3. The device stops listening. Attempts 1 and 2 fail, and look at what browser-style code sees: close 1006 with no reason. The WebSocket API deliberately doesn't tell page code why a connection failed (refused, DNS, TLS, timeout all look alike), so the page can't distinguish "still booting" from "wrong address". That's one reason to keep a plain HTTP GET /health the page can fetch while waiting, which does show a 503 and its Retry-After.
  4. Before attempt 3 the device comes back with boot b-91c2. The client connects, compares boot ids, and concludes it must take a fresh snapshot. The short sleeps (50 × 2ⁿ ms) stand in for real backoff; they change how long the run takes, not what it prints.
AdvancedNoticing a dead connection: heartbeats

A connection that dies quietly is worse than one that closes. When Wi-Fi vanishes or a switch drops the link, no close frame arrives and the client's socket looks open. A client that is only receiving has nothing to retransmit, so TCP never notices; and the Linux default for TCP keepalive is 2 hours of idle time before the first probe (tcp_keepalive_time, Linux kernel ip-sysctl documentation, checked October 2026; Android devices may tune these, and the public PolyOS docs don't say). So the protocol needs its own heartbeat, and the client a watchdog:

heartbeat.mjs
// Detecting a dead connection: the device sends an application heartbeat every 10 s; the UI's watchdog
// gives up after 25 s of silence. At t=33 s the network silently stops carrying packets (no close frame).
const HEARTBEAT_MS = 10_000;
const WATCHDOG_MS = 25_000; // 2.5 heartbeats: one late or lost heartbeat is not a failure
const NETWORK_DIES_AT = 33_000;

const events = []; // the fake clock's queue: [time, what]
const schedule = (t, what) => { events.push([t, what]); events.sort((a, b) => a[0] - b[0]); };
let lastHeard = 0;
let watchdogDeadline = WATCHDOG_MS;
const log = (t, msg) => console.log(`t=${String(t / 1000).padStart(4)}s  ${msg}`);

for (let t = HEARTBEAT_MS; t <= 90_000; t += HEARTBEAT_MS) schedule(t, "device sends heartbeat");
schedule(14_000, "device sends status.changed");
schedule(41_000, "device sends status.changed");

while (events.length) {
  const [t, what] = events.shift();
  if (t >= watchdogDeadline) { // the watchdog fires before this event would have been handled
    log(watchdogDeadline, `watchdog: nothing for ${(watchdogDeadline - lastHeard) / 1000} s since t=${lastHeard / 1000}s -> close and reconnect`);
    break;
  }
  if (t >= NETWORK_DIES_AT) { log(t, `${what} -> lost on the way, the UI hears nothing`); continue; }
  lastHeard = t;
  watchdogDeadline = t + WATCHDOG_MS; // any message from the device counts as proof of life
  log(t, `${what} -> received, watchdog reset to t=${watchdogDeadline / 1000}s`);
}
Result
t=  10s  device sends heartbeat -> received, watchdog reset to t=35s
t=  14s  device sends status.changed -> received, watchdog reset to t=39s
t=  20s  device sends heartbeat -> received, watchdog reset to t=45s
t=  30s  device sends heartbeat -> received, watchdog reset to t=55s
t=  40s  device sends heartbeat -> lost on the way, the UI hears nothing
t=  41s  device sends status.changed -> lost on the way, the UI hears nothing
t=  50s  device sends heartbeat -> lost on the way, the UI hears nothing
t=  55s  watchdog: nothing for 25 s since t=30s -> close and reconnect
What’s happening
  1. The device sends a heartbeat every 10 s, and status events whenever they happen. Each message the UI receives, heartbeat or not, resets the watchdog to 25 s later: at t=10s it moves to t=35s, at t=14s (a status event) to t=39s, and so on.
  2. At t=33s the network silently stops carrying packets. The device doesn't know either; it keeps sending, and the heartbeat at t=40s, the status change at t=41s and the heartbeat at t=50s are all lost.
  3. The watchdog's deadline is still t=55s from the heartbeat at t=30s. At t=55s it fires: 25 s of silence. The client closes the socket itself and enters the reconnecting state from the simulation above. Detection took 22 s after the failure, instead of "never".
  4. Why 2.5 heartbeat intervals? One late heartbeat (a busy device, a slow Wi-Fi moment) must not cause a reconnect; two missed in a row is a failure. Shorter intervals detect faster but cost the device a message per client per interval.
  5. And the status change at t=41s that was lost? The reconnect resumes from the last seq the client applied, so the device replays it. Heartbeats find the failure; snapshot plus events repairs the state.

Use application-level heartbeats ({"type":"hb"}, or an SSE comment line) for browser clients: WebSocket ping and pong frames are answered by the browser automatically and are invisible to page code, so they can keep the connection alive but can't drive a watchdog in the page. Server-side code (ws's ping() and the pong event) can use them to reap dead clients on the device.

AdvancedAfter the reconnect: what else changed
  • The session. If the device keeps sessions in memory, the reboot logged everyone out. The reconnect gets 401; send the admin to the login page without losing a half-filled form (keep drafts in sessionStorage), and come back to the same screen.
  • The address. A device on DHCP can come back on a new IP, and the page can't follow it. After the unreachable timeout, say what the page last knew ("Last reached at 10.0.4.17, 2 minutes ago") and how to find the device (its screen, the cloud console). Device unreachable, rebooting and offline states covers the screens.
  • The configuration. A reboot can apply pending settings, roll back a failed update, or restore defaults. The snapshot after a new boot shows the truth; compare it with what the UI expected and say so if they differ ("Update rolled back to 5.0.1").
  • Many devices reconnecting to one cloud. On the device side, a herd is rare: a handful of clients. The reverse is not: after a cloud deploy or an internet outage, thousands of devices reconnect their outbound management connections at once. That's where jitter is essential, on the device's own reconnect loop (the numbers are in the System Design course's Real-time updates in the client).
The update that never comes back

The admin starts a firmware update. The UI shows "Restarting for update…" and probes every 5 s. Ten minutes later the device still refuses connections. What should the UI do?

Show answer

Change what it says, not how it behaves: switch to "The device hasn't come back after 10 minutes" with what to check, and keep probing.

What’s happening
  1. The UI set a deadline when it entered restarting, based on how long the update normally takes (the update's operation resource or release notes can give an expected duration). Ten minutes past it, "restarting" is no longer an honest message.
  2. It must not stop probing: updates can take longer than expected (a slow flash write, a second reboot to finish), and a device that comes back to a page that gave up is a bad experience.
  3. It switches to the unreachable presentation with specific help: check power and the network cable, look at the device's own screen or status light, and check the cloud console if the device is managed there.
  4. When the device does return, the hello decides the rest: a new boot plus new firmware means the update succeeded (reload the page); a new boot with the old firmware likely means it rolled back, and the UI should read the update operation's final state and show it.

#Constrained devices and many clients

Added

The device's job is the call: encoding and decoding video, echo cancellation, the camera. The web server and its streams are guests in the same CPU and memory, and a guest that grows without limit eventually hurts the call or gets the process killed (on an Android-based device the system can stop a process under memory pressure; see Web UIs and APIs on an Android-based device). Meanwhile the number of listeners is not as small as it looks: two admins with three tabs each, the touch controller, a room control system, a monitoring tool, the cloud agent. That's ten streams on a box sized for the call.

So every real-time path on a device has explicit limits, and a defined behaviour at each limit: how many subscribers (slots), how much may queue for each one (a bounded queue), what happens to a subscriber that can't keep up (it gets told to resync, not buffered forever), how big a message may be, and how fast high-rate values are published (throttled at the source). The Scaling WebSockets topic in the System Design course solves fan-out by adding servers; a device can't, so it limits, sheds and degrades.

A mental model: a small café with three tables. When they're full, the fourth party is told "about 30 minutes" at the door rather than being let into the kitchen. Each table gets its orders in sequence; a table that stops eating doesn't get a growing pile of plates, the waiter clears the table and brings the menu again. And the kitchen doesn't announce every stir of the soup, only when a dish is ready.

State store throttle volume 1 per 250 ms Hub seq + serialise once 3 slots queue ≤ 4 each admin tab keeps up touch panel keeps up RESYNC old control queue overflowed no slot left queue, max 4 4th client 503 · 1013
Limits with a defined outcome at each one: no slot, a full queue, a value that changes too fast.

Here's that hub, with a volume slider dragged by an admin while three clients listen and a fourth tries to join:

fanout-limits.mjs
// Fan-out on a constrained device: a fixed number of subscriber slots, a bounded queue per subscriber,
// "resync instead of buffering forever" for slow ones, and coalescing high-rate state at the source.
const MAX_SUBSCRIBERS = 3;
const QUEUE_LIMIT = 4;       // events waiting for one subscriber; beyond this it gets a resync instead
const THROTTLE_MS = 250;     // at most one volume event per 250 ms; the last value always goes out

function simulate(throttle) {
  console.log(`--- volume slider ${throttle ? "throttled at the source" : "published raw"}`);
  let seq = 0;
  const subs = new Map(); // name -> { queue, every, got }
  function subscribe(name, every) { // every: the connection drains one event per `every` 100 ms ticks
    if (subs.size >= MAX_SUBSCRIBERS) return console.log(`${name.padEnd(12)} -> 503, Retry-After: 30 (${subs.size}/${MAX_SUBSCRIBERS} slots in use)`);
    subs.set(name, { queue: [], every, got: [] });
  }
  function publish(path, value) {
    const event = { seq: ++seq, path, value }; // a real hub serialises once and sends the same bytes to everyone
    for (const s of subs.values()) {
      if (s.queue.at(-1)?.resync) continue;   // already told to resync: the snapshot will cover newer events
      if (s.queue.length < QUEUE_LIMIT) s.queue.push(event);
      else s.queue = [{ resync: true, seq }]; // drop the backlog: one small marker instead of unbounded memory
    }
  }
  let lastSent = -Infinity, pending = null;     // throttle with a trailing edge: the final value is never lost
  function onVolume(t, v) {
    if (!throttle || t - lastSent >= THROTTLE_MS) { publish("audio.volume", v); lastSent = t; pending = null; }
    else pending = v;
  }
  subscribe("admin-tab", 1);
  subscribe("touch-panel", 1);
  subscribe("old-control", 4); // a room control system behind a slow link: one event per 400 ms
  subscribe("monitoring", 1);  // a fourth client: no slot left
  for (let tick = 0; tick <= 20; tick++) {
    const t = tick * 100;
    if (t < 1_000) onVolume(t, 41 + tick);                                   // 10 raw changes, 41..50
    if (pending !== null && t - lastSent >= THROTTLE_MS) { publish("audio.volume", pending); lastSent = t; pending = null; }
    if (t === 500) publish("call.state", "connected");
    for (const s of subs.values()) {
      if (tick % s.every) continue;
      const e = s.queue.shift();
      if (e) s.got.push(e.resync ? `RESYNC(snapshot@${seq})` : `#${e.seq}`);
    }
  }
  console.log(`published ${seq} events`);
  for (const [name, s] of subs) console.log(`${name.padEnd(12)} got ${s.got.join(" ")}`);
}
simulate(false);
simulate(true);
Result
--- volume slider published raw
monitoring   -> 503, Retry-After: 30 (3/3 slots in use)
published 11 events
admin-tab    got #1 #2 #3 #4 #5 #6 #7 #8 #9 #10 #11
touch-panel  got #1 #2 #3 #4 #5 #6 #7 #8 #9 #10 #11
old-control  got #1 #2 RESYNC(snapshot@10) #11
--- volume slider throttled at the source
monitoring   -> 503, Retry-After: 30 (3/3 slots in use)
published 5 events
admin-tab    got #1 #2 #3 #4 #5
touch-panel  got #1 #2 #3 #4 #5
old-control  got #1 #2 #3 #4 #5
What’s happening
  1. Three subscriber slots, and the fourth client (monitoring) is refused with 503 and Retry-After: 30 in both runs: a clear "come back later" instead of a connection the device can't afford. Every accepted subscriber has a queue of at most QUEUE_LIMIT = 4 events. Two drain one event per 100 ms tick; old-control, a room control system behind a slow link, drains one per 400 ms.
  2. Raw run. The slider produces 10 changes in one second (volume 41 → 50), each published as an event, plus call.state: connected at t=500ms: 11 events. admin-tab and touch-panel get #1..#11. old-control gets #1 at t=0, then falls behind: by t=400ms its queue holds #2..#5, it drains #2, and #6 and #7 refill it. When #7 arrives at a full queue, the hub throws the backlog away and leaves one marker: resync.
  3. Events #8..#10 aren't queued for old-control at all: the snapshot it will fetch covers them. At t=800ms it drains the marker and fetches a snapshot, which by then is at seq 10, so the output shows RESYNC(snapshot@10); the next event, #11 (volume 50), follows it normally. Its memory use never went above 4 events, and its state ends correct. What it lost is the individual events, including call.state: connected as an event; the snapshot still shows the call as connected, which is exactly the state-versus-occurrence trade from Designing device events.
  4. Throttled run. The same 10 slider changes now pass a throttle at the source: at most one volume event per 250 ms, and a pending last value is always flushed (the trailing edge), so the final volume 50 is never lost. Only 4 volume events go out (41, 44, 47, 50) plus the call event: 5 events instead of 11.
  5. Now even old-control keeps up: its queue never fills, and it receives all five events in order with no resync. Throttling at the source helps every client and keeps the sequence contiguous; shedding at the queue is the safety net for the client that's still too slow.
  6. Two choices this model makes on purpose. The hub numbers each event once and (in a real hub) serialises it once, sending the same bytes to every subscriber, so the cost per extra client is a queue entry, not another JSON.stringify. And "resync" replaces the backlog instead of disconnecting the client: a slow client stays connected and correct, it just gets fewer, coarser updates.
AdvancedFan-out costs from stated assumptions

The numbers that size these limits are easy to estimate. The assumptions below are illustrative, not measurements of any product: change them to the device you have:

fanout-budget.mjs
// What fan-out costs a device, from stated assumptions (illustrative numbers, not a product's real figures).
const subscribers = 8;              // admin tabs, a touch panel, a control system, a monitoring agent...
const eventBytes = 250;             // one JSON status event with its envelope
const statusRate = 5;               // status events per second during a busy call
const meterBytes = 300;             // one audio-level frame (several channels)
const meterHz = 30;                 // if level meters were pushed raw...
const meterHzSampled = 2;           // ...or sampled for a UI that only needs a moving bar
const queueLimit = 64 * 1024;       // bounded per-subscriber send queue
const stallMinutes = 10;            // one subscriber stops reading (laptop lid closed, Wi-Fi stuck)

const kb = (b) => `${(b / 1024).toFixed(1)} KiB`;
const mb = (b) => `${(b / 1024 / 1024).toFixed(1)} MiB`;
const out = (bytesPerSec) => `${kb(bytesPerSec * subscribers)}/s out`;

console.log(`${"status events".padEnd(21)}: ${statusRate}/s x ${eventBytes} B x ${subscribers} = ${out(statusRate * eventBytes)}`);
console.log(`${`meters raw (${meterHz} Hz)`.padEnd(21)}: ${out(meterHz * meterBytes)}`);
console.log(`${`meters sampled (${meterHzSampled} Hz)`.padEnd(21)}: ${out(meterHzSampled * meterBytes)}`);
const perSubPerSec = statusRate * eventBytes + meterHz * meterBytes;
console.log(`one stalled subscriber, unbounded queue, ${stallMinutes} min: ${mb(perSubPerSec * stallMinutes * 60)} held in RAM`);
console.log(`all ${subscribers} subscribers, bounded queues: at most ${kb(subscribers * queueLimit)} (then resync)`);
console.log(`a ${kb(queueLimit)} queue absorbs ${(queueLimit / perSubPerSec).toFixed(1)} s of a stall before that subscriber resyncs`);
Result
status events        : 5/s x 250 B x 8 = 9.8 KiB/s out
meters raw (30 Hz)   : 70.3 KiB/s out
meters sampled (2 Hz): 4.7 KiB/s out
one stalled subscriber, unbounded queue, 10 min: 5.9 MiB held in RAM
all 8 subscribers, bounded queues: at most 512.0 KiB (then resync)
a 64.0 KiB queue absorbs 6.4 s of a stall before that subscriber resyncs
What’s happening
  1. Normal status traffic is tiny: 5 events a second of 250 bytes to 8 subscribers is 9.8 KiB/s. Status events are not what stresses a device.
  2. Pushing raw audio-level meters at 30 Hz to the same 8 subscribers is 70.3 KiB/s, seven times the rest of the traffic combined. Sampled at 2 Hz, which is all a moving bar on a settings page needs, it drops to 4.7 KiB/s. Measurements belong on a sampled, separate channel.
  3. The dangerous case is memory, not bandwidth. One subscriber that stops reading (a laptop lid closed with the tab open, a stuck Wi-Fi client) with an unbounded queue makes the device hold 5.9 MiB after 10 minutes, and growing, from a single client.
  4. With bounded 64 KiB queues, the worst case for all 8 subscribers together is 512 KiB, a fixed number you can budget for. A 64 KiB queue absorbs 6.4 s of stall at this rate before the subscriber is told to resync, which covers short hiccups without holding memory hostage.
AdvancedRefusing clients so each kind understands

How the device says "no" matters, because not every client can read the same answer. Here's a ws endpoint with two slots, a 64 KiB message limit, and both ways of refusing a third client:

ws-limits.mjs
// Connection and message limits on a device's WebSocket endpoint (ws 8), and what each kind of client sees.
import http from "node:http";
import WebSocket, { WebSocketServer } from "ws";

const MAX_CLIENTS = 2;
const server = http.createServer();
let deviceSaw = ""; // what the device's own error handler reported
const wss = new WebSocketServer({ noServer: true, maxPayload: 64 * 1024 }); // ws's default is 100 MiB
server.on("upgrade", (req, socket, head) => {
  const full = wss.clients.size >= MAX_CLIENTS;
  if (full && req.url.endsWith("?policy=http")) { // refuse during the handshake: an HTTP status, no socket
    socket.end("HTTP/1.1 503 Service Unavailable\r\nRetry-After: 30\r\nConnection: close\r\nContent-Length: 0\r\n\r\n");
    return;
  }
  wss.handleUpgrade(req, socket, head, (ws) => {
    ws.on("error", (err) => (deviceSaw = `${err.code}: ${err.message}`)); // without a handler, this would crash the server
    if (full) return ws.close(1013, "too many clients, retry in 30 s"); // accept, then close with a reason
    ws.send(JSON.stringify({ type: "hello", slots: `${wss.clients.size}/${MAX_CLIENTS}` }));
  });
});
await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve));
const url = `ws://127.0.0.1:${server.address().port}/api/v1/ws`;

const opened = (ws) => new Promise((resolve) => ws.addEventListener("message", (m) => resolve(m.data), { once: true }));
const closed = (ws) => new Promise((resolve) => ws.addEventListener("close", (e) => resolve(`close ${e.code} "${e.reason}"`), { once: true }));

const a = new WebSocket(url); // ws library client (a partner integration in Node)
console.log("client 1:", await opened(a));
const b = new globalThis.WebSocket(url); // browser-style client (an admin tab)
console.log("client 2:", await opened(b));

const c = new WebSocket(`${url}?policy=http`);
const status = await new Promise((resolve) => c.on("unexpected-response", (req, res) => resolve(`${res.statusCode}, Retry-After: ${res.headers["retry-after"]}`)));
console.log("client 3 (ws library, HTTP refusal):", status);
const d = new globalThis.WebSocket(`${url}?policy=http`);
console.log("client 4 (browser API, HTTP refusal):", await closed(d), "- the status code is invisible to page code");
const e = new globalThis.WebSocket(`${url}?policy=close`);
console.log("client 5 (browser API, close 1013): ", await closed(e));

const bClosed = closed(b);
b.send("x".repeat(100 * 1024)); // 100 KiB message, over the 64 KiB limit
console.log("client 2 after sending 100 KiB:", await bClosed, "| device:", deviceSaw);

a.close();
await closed(a);
server.close();
Result
client 1: {"type":"hello","slots":"1/2"}
client 2: {"type":"hello","slots":"2/2"}
client 3 (ws library, HTTP refusal): 503, Retry-After: 30
client 4 (browser API, HTTP refusal): close 1006 "" - the status code is invisible to page code
client 5 (browser API, close 1013):  close 1013 "too many clients, retry in 30 s"
client 2 after sending 100 KiB: close 1009 "" | device: WS_ERR_UNSUPPORTED_MESSAGE_LENGTH: Max payload size exceeded
What’s happening
  1. The ws server runs with noServer: true, so the device's own upgrade handler decides before any WebSocket exists. maxPayload is set to 64 KiB; ws's default (in the installed ws 8.22) is 100 MiB, far more than any device should accept from one message. Clients 1 and 2 take both slots (1/2, 2/2).
  2. Refusing in the handshake. For ?policy=http the handler writes a raw 503 Service Unavailable with Retry-After: 30 and no WebSocket is created. The ws library client (client 3, a partner's Node integration) gets an unexpected-response event with the real status and header: it can wait 30 s.
  3. A browser-style client (client 4) gets the same 503, and page code sees only close 1006 with no reason. The browser WebSocket API doesn't expose the handshake response, so the retry hint is lost on the clients most likely to be humans.
  4. Accepting, then closing. For client 5 the device completes the handshake and immediately closes with 1013, "Try Again Later" (IANA close code registry, checked October 2026) and a reason. Now browser code sees close 1013 "too many clients, retry in 30 s", and the admin UI can say so. It costs the device a brief handshake, which is why a device might use 503 for non-browser API clients and 1013 for its own web UI.
  5. Message too big. Client 2 sends 100 KiB. ws stops reading at the frame header, which already announces the length, so it never buffers the 100 KiB; it closes the connection with 1009, "Message Too Big", and the device's error handler records WS_ERR_UNSUPPORTED_MESSAGE_LENGTH: Max payload size exceeded. That handler is not optional: a ws socket that emits error with no listener throws, and an unhandled throw would bring the device's web server down.
AdvancedBackpressure: noticing a slow client in Node

The queues in the model are explicit. In a real Node server the queue is the socket's write buffer, and the signals are already there:

JavaScript
// Excerpt (not run on its own): send to one subscriber without letting its buffer grow.
const MAX_BUFFERED = 64 * 1024;

function sendWs(ws, frame) {            // ws library socket
  if (ws.bufferedAmount > MAX_BUFFERED) return markForResync(ws);
  ws.send(frame);
}

function sendSse(res, frame) {          // Node http.ServerResponse holding an SSE stream
  if (res.writableLength > MAX_BUFFERED) return markForResync(res);
  res.write(frame);                     // returns false once the buffer is past its high-water mark
}
What’s happening
  1. ws.bufferedAmount is the number of bytes the ws socket has queued but not yet handed to the operating system. A subscriber that reads slowly makes it grow; past the limit, the hub stops sending it events and schedules a resync instead, like the marker in the model.
  2. For SSE, res.writableLength is the same measure on a Node response, and res.write() returning false is Node's signal to stop until the 'drain' event. A hub can't pause the device for one client, so instead of waiting for 'drain' it drops that client's backlog and tells it to resync.
  3. markForResync (not shown) is the policy from the model: clear the client's pending events, send one resync message when it has drained, and let the client fetch a snapshot.
  4. Both checks run before every send, which is cheap: reading a number. What they buy is the guarantee from the estimate: no client can make the device hold more than MAX_BUFFERED bytes for it.
AdvancedPer-identity limits, compression and priorities
  • Per-identity limits, not just a global count. One misbehaving script that opens a new stream every minute and never closes the old ones would take every slot. Limit streams per session or API key, close idle ones, and reap dead ones with heartbeats.
  • Compression off. ws leaves permessage-deflate disabled by default on the server, and its README warns it "adds a significant overhead in terms of performance and memory" (ws 8.22, as installed). Status JSON is small; keep it off on a device.
  • Priorities. If the device must shed load, shed the admin UI's meters before the touch controller's call controls, and both before the call itself.
  • One upstream, many viewers. When many people want to watch the same room, don't have each browser connect to the device: let the cloud management service hold one connection and fan out from the cloud, where adding servers is possible. The general HTTP-side limits (payload size, rate limits, caching) are in Designing for constrained devices.
The monitoring script that eats the slots

After a partner deploys a new monitoring script, admins start seeing "too many clients" in the device's web UI. The device allows 8 streams. The script polls nothing and has only one instance. What's probably happening, and what do you change on the device?

Show answer

The script is leaking connections: it opens a new stream on every run (or every error) without closing the old one, and the device counts each one against its 8 slots.

What’s happening
  1. Each run opens a WebSocket or SSE stream, reads what it needs, and exits or loses track of the socket without closing it. The device sees one more open stream per run; after eight, every new client, including real admins, is refused.
  2. Close the dead ones: the device's heartbeat (WebSocket ping/pong from the server side, or an expected reply to an application heartbeat) finds connections whose other end is gone and frees their slots.
  3. Limit per identity: the script authenticates with one API key, so cap that key at, say, 2 streams. Its third connection is refused with 1008 (Policy Violation) or 429, and the admins' slots are untouched.
  4. Report it: log the identity, address and connection count so support can show the partner what their script is doing. The fix belongs in the script, but the device should never let one client deny service to everyone else.

Written fresh for a device-platform full stack role. Runnable examples were run on Node 26; product facts come from public documentation, cited where used.

RohitDownloads · All chapters

Sync progress across devices

Type the same private phrase on your Mac and your phone, and your Learned ticks follow you between them — on every study site.

The phrase never leaves this device: only a fingerprint of it is sent, and the server stores a fingerprint of that. Anyone who knows the phrase could see or change your ticks, so pick something you don’t use elsewhere.

Sync is on in this browser.

Sync ID

This ID must be the same on every device. If another device shows a different one, its phrase is different (capital letters count): tap “Turn off here” on it and type the phrase again exactly.