Review Questions

Three drawers, plus identity and collections: what the admin chose, what the device reports, and what you asked it to do. "/device for identity, read-only; /settings for configuration, split into sections like /settings/audio and /settings/network with GET and PATCH; /status for reported state, read-only, because it changes every second and nobody sets it; collections for things with their own lifecycle, /peripherals created by the device and /calls created by clients dialling; and /operations for reboots and updates, which come back as 202 with an operation to poll. Then the cross-cutting rules: one error format, the major version in the path, units in field names, stable peripheral ids across reboots."

Learn more: Modelling a device as resources.

Because they have different writers and different lifetimes, and mixing them breaks concurrency, auditing and export. "If live state like the temperature or the mute button sits in the settings document, its ETag changes every few seconds, so every conditional write fails; pressing mute shows up in the audit log as a configuration change; and a configuration import wipes runtime state. Live mute is runtime state you can write, so it gets its own resource, PUT /audio/mute with true, while 'mute when a call starts' is a setting, muteOnJoin under /settings/audio."

Learn more: Modelling a device as resources and Optimistic concurrency with ETags.

PUT for absolute state, POST for actions, and an idempotency key on every POST you can't undo. "Volume 40 and mute are PUTs: they say what the state is, so a retry is harmless. Volume up is a relative POST; a lost response plus a retry adds 20 instead of 10, so I keep relative actions for interactive remotes only. Placing a call and rebooting are POSTs that aren't idempotent, so they carry an Idempotency-Key and the device replays the first answer. Hanging up is DELETE /calls/{id}, and a second DELETE that gets 404 counts as done."

Learn more: Methods, status codes and idempotency for device APIs.

It depends on the device's state, which is why partners need these codes documented and stable. "503 with Retry-After while the device is booting: the HTTP server is up but the services behind it aren't. 409 Conflict while it's in a call: the request is valid but conflicts with the current state, and the client can fix that by waiting or scheduling. 202 Accepted with a Location for the operation once it's idle. Not 423 Locked, which is WebDAV's lock model, and not 500, which sends clients into error handling instead of waiting."

Learn more: Methods, status codes and idempotency for device APIs.

PATCH with JSON Merge Patch (RFC 7396, application/merge-patch+json), because an old client's PUT silently deletes settings it doesn't know about. "A management tool built for firmware 4.2 maps the response into its own model, drops the new autoWake field, and its PUT stores the device without it: nobody asked to turn it off, nothing errors. With merge patch it sends only { "display": { "brightness": 60 } } and everything else is left alone. On the server I parse that media type explicitly, return 415 with Accept-Patch for anything else, apply the patch to a copy, validate the result, strict about unknown keys, and swap it in atomically, as RFC 5789 requires."

Learn more: Partial updates: PATCH and JSON Merge Patch.

It can't set a value to null, can't touch one element of an array, and can't say "only if the current value is X". "For most device settings that's fine. Lists that people edit one item at a time, like directory entries or provisioning servers, become collections with their own ids instead of arrays inside the settings, which removes the main reason for anything else. If I genuinely need element-level or conditional edits, I add JSON Patch, RFC 6902, application/json-patch+json, alongside it: a list of operations addressed by JSON Pointer, with a test operation as a precondition, applied all or nothing."

Learn more: Partial updates: PATCH and JSON Merge Patch.

POST /firmware-updates returns 202 Accepted at once, with a Location pointing at an operation resource that survives the reboot the update ends with. "The operation has a small status clients branch on (queued, running, succeeded, failed, cancelled), a finer step the UI shows (downloading, verifying, installing, rebooting), real progress only where something is counted, a cancellable flag, and on failure a problem-details error with a stable type like signature-invalid. Clients poll at the interval in Retry-After or listen to an event stream with polling as the fallback. A second update while one runs gets 409 with a link to the running one, and cancel is a POST to the operation that returns 409 once the flash write has started."

Learn more: Long-running operations.

The process that would send the 200 is the one being restarted, so the answer has nowhere to go. "Its sockets close with it, the new process after boot knows nothing about that connection, and long before a minute-long reboot finishes, proxies, browsers and SDK timeouts give up anyway; a client that sees an error often retries, which is a second reboot. So the device answers 202 with an operation id, makes sure the response is written before it shuts down, keeps the operation record in flash, and the next boot marks it succeeded before the API opens. Clients treat connection refused and 503 during that window as 'in progress', not as failure."

Learn more: Long-running operations and, for Poly's own reboot call, How a web UI, device APIs and back-end services fit together.

Without protection the last write wins and the first admin's change silently disappears; I'd use optimistic concurrency with a strong ETag and a required If-Match. "Every GET returns an ETag; every write must send it back in If-Match. The device compares before doing anything: a match applies the change and returns the new tag; a mismatch returns 412 Precondition Failed with the current tag, and the client re-reads, re-applies its own intent and retries. I'd require the header, answering 428 Precondition Required to writes without it, so no integration can bypass the check, and give each settings section its own tag so an audio change doesn't conflict with a network change."

Learn more: Optimistic concurrency with ETags.

Two different ETag bugs: Express's automatic tags are weak, and a revision counter that restarts on reset repeats old tags. "RFC 9110 says If-Match must use the strong comparison, and under strong comparison a weak tag like W/"1e-…" never matches anything, not even itself, so every write fails and someone eventually turns the check off. The fix is to set your own strong ETag on writable resources. The second bug is ABA: after a factory reset the counter is back at "r2" for a different document, so a script's old If-Match: "r2" wrongly matches. Derive the tag from a hash of the canonical content, or add a reset epoch so a tag never comes back."

Learn more: Optimistic concurrency with ETags.

RFC 9457 problem details everywhere, with the type URI as the contract and everything else as the message. "Every error is application/problem+json with a type that identifies the kind of problem and never changes, a title for that type, the status, a detail for this occurrence that clients must not parse, and an instance. Machine-readable specifics go in extension members, like an errors array of JSON Pointers for validation or activeCall for a busy device, and clients ignore extensions they don't know. Generic cases use about:blank. A 500 never carries a stack trace, only an instance id support can find in the device logs. The types are documented in an error catalogue, and partners switch on type, falling back to the status code."

Learn more: Errors with problem details.

Nothing at all: one request is one transaction. "The device checks every value, then runs cross-setting rules on the complete candidate configuration, such as 'SIP mode needs a SIP server', and only if everything passes does it swap the configuration in one step and bump the revision. Any error returns 422 with every problem and applied: false. I'd offer a dry run that runs the same code minus the final swap and returns a diff with each change marked live or restart-required, require If-Match with the configuration revision so a stale profile can't undo today's edits, and report restart-required settings as pending in /status instead of pretending they took effect."

Learn more: Bulk and transactional configuration.

Confirm or revert: settings that can cut the admin off are applied on probation, and the device itself rolls them back unless someone confirms from the new state. "That's NETCONF's confirmed commit: RFC 6241 says such a commit must be reverted if no confirming commit arrives within the timeout, 600 seconds by default, or if the device reboots first. So POST /config/changes with a confirm timeout arms a timer on the device, the UI polls the new address, and only a successful confirmation from there keeps the change. The timer has to live on the device; a timer in the browser is useless once the browser can't reach it."

Learn more: Bulk and transactional configuration and Apply, restart and rollback flows.

About 36,000 KB of response bodies an hour and 7,200 serialisations, 80% of it settings that change twice a day. "Fix one: a strong ETag and If-None-Match on settings, with freshness checked before the document is built, so nearly every poll is a body-less 304 the device doesn't even render. Fix two: push status as events on one stream per tab, carrying only the fields that changed. Under the assumptions below, that's about 900 KB an hour instead of 36,000."

// An admin dashboard polls /status (2 KB) and /settings (8 KB) every 5 s; five tabs stay open all hour.
// Assumptions (stated, not measured): settings change twice a day; status changes on most polls, and an
// event carrying only the changed fields is about 0.25 KB. Response bodies only; headers left out.
const tabs = 5, intervalS = 5, statusKB = 2, settingsKB = 8, eventKB = 0.25, settingsChangesPerDay = 2;
const pollsPerTab = 3600 / intervalS;                       // polls of each resource, per tab, per hour
const kb = (n) => `${n.toLocaleString("en-US", { maximumFractionDigits: 1 })} KB`;

const plainSettings = tabs * pollsPerTab * settingsKB;
const plainStatus = tabs * pollsPerTab * statusKB;
console.log(`plain polling: settings ${kb(plainSettings)} + status ${kb(plainStatus)} = ${kb(plainSettings + plainStatus)}/hour`);
console.log(`  requests the device serialises: ${(2 * tabs * pollsPerTab).toLocaleString("en-US")}/hour`);

// Fix 1: strong ETag + If-None-Match, freshness checked before rendering: a body only after a change.
const fix1Settings = tabs * (settingsChangesPerDay / 24) * settingsKB;
console.log(`fix 1, settings answered 304 unless changed: ${kb(fix1Settings)}/hour`);
// Fix 2: status pushed as one event stream per tab, only the fields that changed.
const fix2Status = tabs * pollsPerTab * eventKB;
console.log(`fix 2, status as events: ${kb(fix2Status)}/hour`);
const after = fix1Settings + fix2Status;
console.log(`after both: ${kb(after)}/hour, ${(100 * (1 - after / (plainSettings + plainStatus))).toFixed(1)}% less`);

plain polling: settings 28,800 KB + status 7,200 KB = 36,000 KB/hour
  requests the device serialises: 7,200/hour
fix 1, settings answered 304 unless changed: 3.3 KB/hour
fix 2, status as events: 900 KB/hour
after both: 903.3 KB/hour, 97.5% less

Learn more: Designing for constrained devices and Polling, long polling, SSE and WebSockets for device status.

It can't scale out, and it shares the CPU with the device's real job, video and audio, so the API has to be cheap by design and bounded everywhere. "Polling is made cheap with strong ETags and freshness checked before rendering; live status is pushed as events rather than polled; responses are small, with settings in sections, ?fields=, paginated lists with a hard maximum and large files streamed. Every request is bounded: per-route body limits that return 413, per-client rate limits that return 429 with Retry-After, one expensive operation at a time, a connection cap and short timeouts. And the device tells well-behaved clients how to back off, with Retry-After while booting and on running operations. Then measure on the real hardware."

Learn more: Designing for constrained devices.

Browser, device web server, auth and validation, settings service, platform service, and back, with nothing reporting "done" before the platform confirms. "The browser sends a PATCH with only the changed fields; the device's own web server terminates TLS; the API layer checks the session cookie and XSRF header and validates the body against the schema, so bad requests stop before anything changes. The settings service stages the change and calls the platform service that owns the resource, on Android usually a system service over Binder, and waits. On success it commits a new config version, publishes a settings-changed event to every subscriber, and the API returns 200 with the version and a restart-required flag. On failure it rolls back, and the API maps the platform's error to a stable one, like 503 with Retry-After for 'busy'."

Learn more: How a web UI, device APIs and back-end services fit together and Anatomy of a managed device.

In the settings layer's schema, so every writer is refused with the same error. "The value got in through the CLI, but patching the CLI parser leaves REST and provisioning open, and truncating the header in the UI hides the symptom for one viewer. I'd add maxLength to device.local.deviceName in the one schema every front door calls, so CLI, REST and provisioning all get INVALID_VALUE, and the UI reads the same limit before the user presses Save. Then the device-specific part: devices in the field may already hold a long name, so the release needs a migration that meets that old data on first boot."

Learn more: Anatomy of a managed device and Validation shared between UI and device.

A machine-readable description of an HTTP API, and in 3.1 its schemas are real JSON Schema 2020-12. "Paths, operations, parameters, request and response bodies, security schemes, and in 3.1 a root webhooks map for requests the device sends out. Because the file is data, it generates docs and types, validates traffic at runtime and gets diffed for breaking changes. The changes that bite: nullable is gone in favour of type: [string, "null"], exclusiveMinimum is a number instead of a boolean, schema example became examples arrays. And openapi: 3.1.1 is the spec version, info.version is your API's version; mixing them up is a classic slip."

Learn more: OpenAPI 3.1 in an interview.

Shared components for every shape, every operation with its error responses, and CI that checks the file in three ways. "One schema per resource, one Problem schema for every error as application/problem+json, reusable responses and the If-Match header defined once. Every operation gets a unique operationId, a summary, tags and its 4XX responses; partial updates are application/merge-patch+json with a patch schema that has additionalProperties: false; reboots are 202 plus an operation; the server URL is a variable because every device is its own host. In CI, swagger-parser proves it resolves, a linter like Redocly or Spectral proves it makes sense, and a script validates every example against its own schema, because examples are what partners copy."

Learn more: Writing a spec for a device API.

Make the spec executable: compile its schemas at startup and check every request before the handler and every response after it. "Middleware validates the media type, parameters and body against the operation in the spec and turns failures into problem details with JSON Pointers to the bad fields: 415 for a media type the spec doesn't declare, 400 only for a body that can't be parsed, 422 for well-formed JSON that breaks the schema, like volume 150. Handlers only ever see valid data. Responses are validated against the schema for their status code: in tests and staging a violation fails the request, so the first test that hits a contract bug says why; in production I'd log violations, or sample, rather than turn a shipped contract bug into an outage. Compile once, validate many."

Learn more: Validating requests and responses against the spec.

Types, a thin client, docs and mocks, but never values: types check shapes, the device still validates. "openapi-typescript turns the spec into TypeScript types, so a wrong field name, a string where a number belongs or an enum value that doesn't exist is a compile error in the web UI. A thin typed client like openapi-fetch sits on top, or a generated SDK for partners in other languages; Redoc or Swagger UI render the docs and a mock server serves the examples so the UI team can start early. What the types lose: integer becomes number, and minimum, maximum, pattern, minProperties and additionalProperties vanish, so { volume: 150 } compiles and the device rejects it at runtime."

Learn more: Generating types, clients and docs.

Design-first for anything partners and other teams consume, code-first for internal endpoints that ship in lockstep with their only consumer. "The spec pull request is an asynchronous design review: a developer in India proposes an endpoint as a YAML change at the end of their day, I review the shape while they sleep, and the UI team starts against a mock the next morning, with no meeting. Its risk is drift, so the spec is enforced: runtime validation, examples validated in CI, a breaking-change diff on every PR. Code-first with something like Zod is fine for internal endpoints, as long as the generated spec is reviewed and diffed too, because its defaults can quietly turn into partner bugs."

Learn more: Design-first vs code-first.

Only the optional query parameter. "The rename is a removal plus an addition, and removing a field clients read is breaking; the safe way is to add isMuted, keep muted, and deprecate it. Making a response field optional is breaking, because clients that assumed it's always there now need a null check. Narrowing a request range breaks clients that sent 90 yesterday. A new optional query parameter is additive as long as the behaviour without it is unchanged. Direction decides most verdicts: what's safe in a response can break in a request, and the other way round."

Learn more: Contract testing and Rules for compatible change.

Consumer-driven contracts: each consumer records the calls and fields it really uses, and the provider replays every consumer's list against the new build before it ships. "In the chapter's model, renaming muted to isMuted broke the web admin UI's two interactions, while the partner room panel and the fleet manager passed, because they never read that field. That turns a yes/no gate into a decision with names attached: the web UI ships in the same firmware image and can change in the same release; had it broken the partner panel, it would need a deprecation period and a conversation. Pact automates this; for partners who won't run your tooling, ask for the endpoints and fields they use and keep that as a contract file."

Learn more: Contract testing.

It tells the truth about the device's state, makes changes safe, and keeps working while the device is only half alive. "I'd start from the operator's five jobs: is it healthy, change settings, diagnose, update, secure it, which gives a dashboard, settings, diagnostics, maintenance and security. The dashboard is computed from the device's status into a short list of problems with actions, worst first, not a wall of numbers. Wherever they can differ, it shows what's saved, what's active, and when it last heard from the device. And the UI is the API's first client: everything it does goes through the same public REST API scripts and partners use, so a setting only the UI can change is a setting nobody can automate."

Learn more: What device admin UIs must do well.

Describe each setting once, as data, in a JSON Schema served by the device, and generate the form, both validators, the docs and the SDK types from it. "JSON Schema 2020-12 already has type, enum, ranges, patterns, defaults, titles, readOnly, writeOnly and if/then; I add annotation keywords for device facts it lacks, like x-restart and x-group. A generic renderer in React or Angular turns the generated field list into accessible fieldsets: the widget comes from the type, visibility from if/then, constraints become input hints. Because each device serves its own schema, a UI talking to old firmware builds exactly the form that firmware supports."

Learn more: Forms generated from a settings schema.

One schema, compiled in two places: in the browser for instant feedback, on the device as the authority, plus device-only checks the UI can't make. "The browser compiles with allErrors so the form can mark every bad field at once. The device compiles stopping at the first error, with a body-size limit, because its API is reachable by scripts that never saw the form, and Ajv's own security docs say 'Do NOT use allErrors in production'. Rules only the device knows, like the connected display's maximum resolution, come back as 422 problem details with a JSON Pointer, so the UI puts the message on the right field. And the device never trusts that the UI ran."

Learn more: Validation shared between UI and device.

Save the setting, don't restart, and say so. "'Saved. The new provider starts after a restart. This room is in a call: restart now, which ends the call, or later?' The batch goes to the device as one merge patch with If-Match; the device stores it, applies live settings at once and returns which settings are pending a restart. That list is device state, so the dashboard keeps showing 'Restart needed to apply: provider' to anyone who looks until it happens. Silently restarting would drop a live meeting; silently not restarting leaves the admin believing Teams is active."

Learn more: Apply, restart and rollback flows.

Nothing has been installed yet, and the hardest steps haven't started. "After the upload the package still has to pass the checksum and the signature check, then install to the inactive slot, reboot, and pass a post-boot health check before the device marks the new version good. In the chapter's model, five of six scenarios ended badly after a 100% upload: two rejected, two rolled back, one not responding. So the UI shows each phase honestly: real percentages where something is counted, an indeterminate state during the reboot, and 'complete' only when the device reports it's running the new version after its check."

Learn more: Software update flows.

Know which kind of silence it is, keep showing what it last knew, and never queue writes. "One failed poll is noise: keep the data, marked 'as of' the last success. A few in a row is unreachable: a status banner says so, writes are disabled with one disabled fieldset, unsaved edits are kept, and polling backs off to a cap. If the admin started a restart, silence is expected, so it says 'Restarting…', polls steadily and only escalates after a deadline based on the measured boot time. Every status reply carries a boot id: the same id means a network blip, a new one means the device restarted, so reload settings and version and expect to log in again."

Learn more: Device unreachable, rebooting and offline states and Reconnection and back-off.

Export a versioned file that leaves secrets and per-device identity behind; import it through a pipeline that migrates, filters, validates and shows a dry-run diff before one atomic apply. "The export records its format, schema version, source model and firmware, and lists what it omitted: write-only secrets like the admin password, and per-device identity like the room name and IP, which would give fifty rooms the same address. Import migrates old files forward one version at a time, skips what the target model can't take with a reason, validates with the same schema as the forms, and shows only real changes. With a provisioning service, every setting has a source, and the device refuses local writes to provisioned ones with 409."

Learn more: Import, export and provisioning of configuration.

WCAG 2.2 AA done consistently, and Intl plus translator-owned templates for everything locale-dependent. "Real labels on every field; errors as text on the field, linked with aria-describedby, with a suggested fix; nothing conveyed by colour alone; status changes like 'Saved', 'Restarting' and 'Device not responding' in a role="status" live region; visible focus that a sticky bar never covers; targets of at least 24 pixels; a login that works with password managers and paste; session timeouts that warn and can be extended. For localisation: dates, numbers, units, relative times and plurals through Intl, device times shown with the zone labelled, and the device returning error codes and parameters, not English sentences."

Learn more: Accessibility and localisation for admin UIs.

A plain-text protocol over TCP (RFC 854, 1983) with in-band option negotiation, kept because room control systems and integrators' scripts speak short text lines over TCP or RS-232. "Byte 255, IAC, means 'the next bytes are a command': WILL, WONT, DO, DONT negotiate options such as echo or window size, and either side can always refuse, which is why it interoperates with anything. A control panel on the wall that starts a call or mutes the room can drive it with no HTTP stack and no JSON parser, and those integrations are expected to survive hardware generations. Poly's public guides describe exactly this: a command-line API on Telnet port 24 by default or 23, and SSH on port 22 'if you want encrypted access', both off by default. The catch is that Telnet has no encryption, so I'd keep it off by default, local-network only when on, and steer integrators to SSH or REST over TLS."

Learn more: What Telnet is and why devices still expose it and Poly video devices and PolyOS in brief.

FF FC 01 for the first, and nothing for the second. "FF FD 01 is IAC DO ECHO: the client asks the server to echo. A line-mode API doesn't want to, so it refuses with IAC WONT ECHO, FF FC 01: RFC 854 requires an answer to a request for a change of mode. FF FE 01 is IAC DONT ECHO, a request for the mode the server is already in, and RFC 854's loop rule says you don't acknowledge that, or the two sides can bounce replies forever."

Learn more: What Telnet is and why devices still expose it.

The server is a raw TCP line server, and a real Telnet client opens with option negotiation bytes that end up in the command parser. "Many device 'Telnet APIs' are really plain text over TCP, and integrators who use nc or a control system's TCP socket never send an IAC byte, so it works for them. A real Telnet client may start with a burst of IAC DO … and IAC WILL …. The fix is a byte-level Telnet parser in front of the line framer that strips and answers IAC sequences, even if you never intend to support a single option, so the command layer never sees a 255 byte."

Learn more: What Telnet is and why devices still expose it and Building a line-based command server in Node.

The naive server runs four broken commands; a framing server runs exactly volume set 42 and mute get. "TCP is a byte stream, not a message stream: nothing preserves the boundaries of the client's writes, and on localhost you almost always see one event per write, which is why this bug ships and then appears on congested Wi-Fi. So each session keeps a buffer, appends every chunk, runs a command for every newline it finds, and keeps the remainder for the next chunk, with a cap so a client that never sends a newline can't exhaust memory."

// TCP delivers bytes, not messages. The same three chunks, read two ways.
const chunks = ["vol", "ume set 42\r\nmute g", "et\r\n"].map((s) => Buffer.from(s));

// Naive: treat every data event as whole commands.
const naive = [];
for (const c of chunks) naive.push(...c.toString().split("\r\n").filter(Boolean));
console.log("naive :", JSON.stringify(naive));

// Framed: keep a per-session buffer, cut at LF, keep the remainder for the next chunk.
let pending = Buffer.alloc(0);
const framed = [];
for (const c of chunks) {
  pending = Buffer.concat([pending, c]);
  let lf;
  while ((lf = pending.indexOf(0x0a)) !== -1) {
    framed.push(pending.subarray(0, lf).toString("utf8").replace(/\r$/, ""));
    pending = pending.subarray(lf + 1);
  }
  console.log(`after chunk ${JSON.stringify(c.toString())}: commands ${JSON.stringify(framed)}, buffered ${JSON.stringify(pending.toString())}`);
}

naive : ["vol","ume set 42","mute g","et"]
after chunk "vol": commands [], buffered "vol"
after chunk "ume set 42\r\nmute g": commands ["volume set 42"], buffered "mute g"
after chunk "et\r\n": commands ["volume set 42","mute get"], buffered ""

Learn more: Building a line-based command server in Node.

A maximum line length, a maximum number of sessions, an idle timeout, an output cap for slow readers, and an error handler on every socket. "A line over the limit gets one clear error and the rest of it is skipped up to the next newline, so its tail isn't parsed as a new command. server.maxConnections refuses connections beyond the limit before any per-session memory is allocated; on a device I'd rather accept, write error: E_BUSY too many sessions and close, so the integrator sees why. socket.setTimeout only emits timeout; Node doesn't close the socket for you, so the handler writes a reason and ends it. And socket.on('error') means one client resetting its connection can't crash the server."

Learn more: Building a line-based command server in Node and Testing and securing command interfaces.

Predictable for machines first: its main reader is a regular expression written years ago. "Noun then action, volume get, volume set 42, all defined in one table that drives parsing, validation, usage text and help. Every response is one line: success echoes the resulting state in the same form as the get, and every other line starts with a tag, error:, info: or notification:, so a client classifies any line with one prefix check. Errors carry a stable code plus a human message, and only the code is a contract. Arguments are typed and range-checked, one quoting rule works in both directions so replies re-parse with the same tokenizer, and new integrators get a JSON-lines mode. Then I treat the published grammar like a public API: additive changes only."

Learn more: Designing a command grammar.

They echo the request instead of the resulting state. "A script comparing the reply with volume get sees two different strings for the same state, and if the device clamped 70 to 50, the reply lies about what the room is set to. The reply should be the canonical state after the change, formatted exactly as get would print it: volume 42, or an error such as error: E_RANGE level must be 0..50, got 70. Then a script can compare replies directly, and the reply is the truth, not an acknowledgement."

Learn more: Designing a command grammar.

Every unsolicited line identifies itself, replies come back in command order, and the client demultiplexes. "Notification lines start with a prefix such as notification:; every other line resolves the oldest pending command, first in, first out. If commands can complete out of order, a slow diagnostic next to a quick volume get, I add client-chosen tags, #7 diag run answered by #7 …. Subscriptions are per session, idempotent to re-send, and removed when the session closes. A client that assumes 'the next line is my answer' will one day take a notification as its answer and be off by one for the rest of the session."

Learn more: Asynchronous notifications on a command channel.

Not by splitting on colons: the SIP address contains colons, so the fields shift. "Parse from both ends: the fixed fields from the left (notification, callstatus, the direction, the call id), the fixed field from the right (the state), and treat everything in between as the variable field, the address. Values with colons are normal here: SIP URIs, IPv6 addresses, far-site names. If you're designing the format rather than consuming it, quote or escape variable fields, put variable-length fields last, or offer a JSON mode for new integrations."

Learn more: Asynchronous notifications on a command channel.

SSH encrypts the session, authenticates the device by its host key, and supports public-key login so no password crosses the network. "A machine in the middle can't impersonate a device whose host key the client has pinned, and if only public keys are accepted, password guessing disappears from the attack surface. SSH also gives channels: exec runs one command with clean stdout, stderr and an exit status, which is what automation wants, and shell is the interactive session, which can carry exactly the same line protocol as the Telnet port, so moving off Telnet is a transport change, not a rewrite. For the device it means a unique host key generated on first boot with its fingerprint visible, authorised public keys per role, algorithms current clients support, and a plan for key changes after a factory reset."

Learn more: SSH as the secure alternative.

exec. "The device runs exactly one command and closes the channel, so the script reads stdout to the end with no prompt detection and no 'wait until the reply looks complete' timeouts; errors go to stderr with a non-zero exit status, so 'failed' is exit != 0, not a regular expression over text the next firmware might reword. With a shell the script has to wait for the prompt, send the line, find where the reply ends, cope with banners and echo, and send exit, and every one of those is a place for an update to break it."

Learn more: SSH as the secure alternative.

One command layer under all of them, with thin adapters per transport that only translate syntax. "A registry of operations, each with a required role, an input schema and a handler, and one invoke function that checks existence, authorisation and validation, runs the handler and writes the audit log, always in that order. The REST adapter maps resources and JSON to commands and renders errors as problem details; the Telnet and SSH adapters frame lines, tokenize them, convert text to typed values and render codes as lines or exit statuses. Error codes are shared, so E_INVALID is a 422 on REST and error: E_INVALID … on the CLI. That's parity by construction: one range check, one role check, one audit trail, one place to test."

Learn more: One command layer behind REST, Telnet and SSH.

At three levels, plus fuzzing with invariants. "The command layer gets plain unit tests: every command, every argument boundary, every role, and the most valuable assertion in a device suite, 'a rejected command doesn't change state'. The adapters get translation tests: a line reaches the right command with the right typed arguments, and codes render correctly. Then scripted session tests: readable transcripts of sent and expected lines, wildcards for messages whose wording may change, a timeout on every wait, run against a simulator in CI and real hardware in the lab, with fragmented input, oversized lines, concurrent sessions, interleaved notifications and reconnects after reboot. On top, a seeded grammar-aware fuzzer checking properties like 'only clean error codes' and 'replies re-parse to the same tokens'."

Learn more: Testing and securing command interfaces.

About 259,000 a day from one loop (3 × 86,400), more with parallel sessions. "A per-session limit only costs the attacker a reconnect. What actually limits guessing is state that survives the session: failures counted per source address and per account, with growing delays or a temporary lock, and an alert. The better answers remove the problem: Telnet off by default or bound to a management network, and SSH with public keys, where there's no password to guess."

Learn more: Testing and securing command interfaces and Hardening.

Different clients, different transports, all fed by one event hub inside the device. "For the admin web UI, Server-Sent Events for status and REST for commands: status flows one way, commands already have validation, ETags and error bodies, and EventSource reconnects by itself and sends Last-Event-ID. A WebSocket only where the client sends a lot too: a touch controller, a camera joystick, live meters. Long polling with a since cursor for integrations that only speak plain HTTP, the Telnet line protocol with notifications for room control systems, short polling at minutes for monitoring, and a connection the device opens outwards for the cloud, which can't connect in through the customer's firewall."

Learn more: Polling, long polling, SSE and WebSockets for device status.

Connection limits, idle proxies, authentication, and self-signed certificates. "Over HTTP/1.1, browsers allow six connections per domain and each open EventSource holds one, so the seventh tab on the same device hangs: share one stream between tabs or serve HTTP/2. Idle connections get dropped, so send a comment heartbeat about every 15 seconds. Neither EventSource nor the browser WebSocket constructor can set an Authorization header, so streams ride on the session cookie, which makes checking Origin on the WebSocket handshake mandatory, or any page the admin visits can open a socket with their cookie. And a browser shows no warning page for a background stream to an untrusted certificate; it just fails, so keep streams on the admin page's origin."

Learn more: Polling, long polling, SSE and WebSockets for device status and Securing the on-device web UI.

Separate state events from occurrences, and put every event in one envelope with a boot id and a sequence number. "A state event says 'the value is now X', audio.muted is true: only the latest matters, and a snapshot repairs a missed one. An occurrence says 'X happened', call.ended with a reason: each one matters, and a snapshot can't bring it back, so anything important is also kept in a queryable log like call history. The envelope has boot, a seq that increases by one per event within that boot, a stable namespaced type, wall-clock time for display, monotonic uptimeMs for durations, and data. Consumers order by seq, never by timestamp, and read tolerantly: skip unknown types, ignore unknown fields."

Learn more: Designing device events.

Because the device's clock is wrong before NTP and jumps when it syncs, so timestamps can run backwards; sequence numbers never lie. "In the chapter's model the device booted with its clock two minutes fast; after NTP corrected it, a mute that happened later was stamped earlier than the call events before it, and sorting by time ended with call.state = connected for a call that had ended. So the rule is: sequence numbers for order, the monotonic clock for durations, wall-clock time for display. A call duration from uptimeMs was 61 seconds; from the wall clock it came out at about minus 59."

Learn more: Designing device events.

Apply 42, skip the duplicate, stop at 44 because 43 is missing, resync, then handle 45 against the new snapshot. "The rule is three comparisons: at or below my sequence, skip; exactly one more, apply; anything bigger is a gap, so stop applying and ask the device for the events since my sequence, or a fresh snapshot. If the device answers with a snapshot already at 45, the waiting 45 is skipped and the client is correct."

// Snapshot plus events: what a client does with each event after a snapshot at seq 41 (same boot).
let mine = 41;
const arriving = [42, 42, 44, 45];
for (const seq of arriving) {
  if (seq <= mine) console.log(`#${seq}: skip (already have up to ${mine})`);
  else if (seq === mine + 1) { mine = seq; console.log(`#${seq}: apply, now at ${mine}`); }
  else {
    console.log(`#${seq}: gap (expected #${mine + 1}): stop applying, resync since ${mine}`);
    mine = 45; // say the device answers with a fresh snapshot taken at seq 45
    console.log(`     resync: snapshot at seq ${mine}`);
  }
}

#42: apply, now at 42
#42: skip (already have up to 42)
#44: gap (expected #43): stop applying, resync since 42
     resync: snapshot at seq 45
#45: skip (already have up to 45)

Learn more: Snapshot plus events.

The sequence restarted at 1, so the client treats every new event as "already seen" and skips them all. "The client last saw 57; the rebooted device sends 1, 2, 3, which are all at or below 57, so the skip rule throws every one away and the UI shows stale state with no error. The fix is a boot id in every event and in the hello: a different boot means a different sequence, so take a fresh snapshot. The alternative, persisting the sequence so it never goes backwards, costs flash writes, which matters on devices with limited write endurance."

Learn more: Snapshot plus events.

Keep showing the last known state, marked stale; tell a planned restart from a lost connection; retry at a sensible pace; and when it's back, ask what changed. "A close with 1012 'Service Restart' or 1001, or a reboot the UI started, means planned: probe at a steady pace and honour 503 with Retry-After while services start. A 1006, or the heartbeat watchdog firing, means lost: retry with exponential back-off and full jitter, switch to 'Device unreachable' after about a minute, but never stop. On reconnect, the hello decides: the same boot id, resume from my last sequence; a new boot id, take a snapshot; new firmware, reload the page, because the JavaScript in the tab was served by the old firmware."

Learn more: Reconnection and back-off.

Because a client that's only receiving never notices a silently dead connection, and TCP keepalive is far too slow by default. "When Wi-Fi vanishes, no close frame arrives and the socket looks open; a receiver has nothing to retransmit, so TCP never notices, and Linux's default keepalive waits two hours before its first probe. So the device sends an application heartbeat, say every 10 seconds, and the UI's watchdog resets on every message and closes the socket itself after about 25 seconds of silence: two and a half intervals, so one late heartbeat doesn't trigger a reconnect but two missed ones do. Browser code needs application-level heartbeats because WebSocket ping and pong are handled by the browser and invisible to the page."

Learn more: Reconnection and back-off.

Explicit limits, with a defined behaviour at each one. "A fixed number of stream slots, with refusals each client can read: 503 with Retry-After for API clients, and accept-then-close with 1013 for the browser UI, which can't see handshake statuses. A bounded queue per subscriber, checked through bufferedAmount or writableLength; when a slow client overflows it, I drop its backlog and tell it to resync from a snapshot, so memory is capped and the client stays correct. High-rate values are throttled at the source with a trailing edge, so the last value always goes out. And a small maxPayload, compression off, limits per identity, heartbeats to reap dead connections, and each event serialised once for everyone."

Learn more: Constrained devices and many clients.

It's leaking connections: each run opens a stream and never closes the old one, and the device counts each against its slots. "After eight runs every new client, admins included, is refused. On the device I'd reap dead connections with heartbeats, so streams whose other end is gone free their slots, and limit streams per identity: the script's API key gets, say, two, and its third is refused with 1008 Policy Violation or 429 while the admins' slots stay free. Then log the identity, address and connection count so support can show the partner exactly what their script does. The fix belongs in the script, but the device must never let one client deny service to everyone."

Learn more: Constrained devices and many clients.

Two conversations: signalling through the proxy, media directly between the endpoints. "The room sends an INVITE to its SIP proxy with an SDP offer: the codecs it can use and the IP address and ports where it wants to receive media. The proxy answers 100 Trying and forwards it; the callee rings, 180, and answers 200 OK with an SDP answer that picks codecs from the offer and gives its own media address. The caller sends ACK, and from then on RTP, usually SRTP-encrypted, flows over UDP between the two SDP addresses, with RTCP reports carrying loss, jitter and round-trip time. The proxy only sees a handful of messages. That split is how I debug: never rings, look at registration and routing; connects with no video, look at the media path."

Learn more: How a video call works.

NAT breaking the media address in the SDP. "Signalling worked, so registration and routing are fine; the media path isn't. The address in the SDP is whatever the endpoint believes its address is, so a room behind NAT writes its private 10.x address, the far end sends RTP there, and nothing arrives. Firewall 'SIP ALG' features that rewrite SDP on the fly often make it worse. The fixes are a session border controller or H.460 traversal for SIP and H.323, or ICE with STUN and TURN, which WebRTC makes mandatory. I'd confirm by comparing the addresses in the SDP with the addresses packets actually arrive from."

Learn more: How a video call works and WebRTC concepts.

WebRTC gives each side an RTCPeerConnection but leaves signalling to the application. "So first there's a channel between the peers, usually a WebSocket to your own server. The caller creates an SDP offer and sets it as its local description, which starts ICE candidate gathering, and sends it; the callee sets it as its remote description, creates an answer that picks codecs and directions, sets that and sends it back. Meanwhile each side trickles its candidates: host addresses, its public address as seen by a STUN server, and relay addresses on a TURN server. ICE pairs them, checks them in priority order and nominates the best pair that works, falling back to the relay. DTLS then runs over that path, checked against the certificate fingerprint in the SDP, to derive SRTP keys. For more than a few people, an SFU with simulcast rather than a mesh."

Learn more: WebRTC concepts.

Almost certainly a working TURN server. "In development both peers are on friendly networks, so host or server-reflexive pairs work and nobody notices TURN isn't configured. In production some users sit behind symmetric NATs, which pick a new public port per destination so the STUN-learned address is useless, or behind firewalls that block UDP, and they need a relay. I'd deploy TURN on UDP 3478 and on TLS port 443, which gets out of networks that only allow HTTPS-like traffic, give clients short-lived credentials from the server rather than static ones in the page, and watch the share of relayed calls, which is both a reliability signal and a cost line."

Learn more: WebRTC concepts.

No: at 1% loss about 7% of frames are damaged, about 122 a minute, because one lost packet damages a whole frame. "A 2 Mbit/s, 30 fps stream sends about 7 packets per frame, and a frame survives only if all 7 arrive, so the damaged share is 1 − (1 − p)^7. And it's worse than the count, because compressed frames refer to earlier ones, so damage spreads until a keyframe arrives. One NACK retransmission brings it down to about 1.3 damaged frames a minute, as long as the round trip fits in the receiver's buffer."

// Why "1% packet loss" is much worse than 1% bad video: a 2 Mbit/s, 30 fps stream sends about
// 7 packets per frame, and one lost packet damages the whole frame. Losses assumed independent.
const PER_FRAME = 7, FPS = 30;
const damaged = (p) => 1 - (1 - p) ** PER_FRAME;          // share of frames with at least one lost packet
for (const p of [0.001, 0.01, 0.02]) {
  const perMin = damaged(p) * FPS * 60;
  const withNack = damaged(p * p) * FPS * 60;             // lost only if the original and one resend are both lost
  console.log(`loss ${(p * 100).toFixed(1)}%: ${(damaged(p) * 100).toFixed(1)}% of frames, ${perMin.toFixed(1)} damaged/min; with one NACK resend ${withNack.toFixed(2)}/min`);
}

loss 0.1%: 0.7% of frames, 12.6 damaged/min; with one NACK resend 0.01/min
loss 1.0%: 6.8% of frames, 122.3 damaged/min; with one NACK resend 1.26/min
loss 2.0%: 13.2% of frames, 237.4 damaged/min; with one NACK resend 5.03/min

Learn more: Media quality and diagnostics.

Narrow it down, then read the room's per-stream call statistics for a bad call and match the pattern to a cause. "Every call or some, every provider or one, both directions or only what they receive? Then received loss and jitter, the bitrate actually used against what was negotiated, frame rate and resolution, and round-trip time. High loss or jitter with normal bitrate points at the network, usually the room's own link or Wi-Fi; low bitrate with no loss suggests congestion control backing off or a call-speed cap; loss in one direction points at that path; a low frame rate with a clean network suggests the far end or a CPU or camera limit. I'd compare against targets like Microsoft's for Teams: under 1% loss and 30 ms jitter from client to their edge."

Learn more: Media quality and diagnostics.

A latency problem, not a jitter problem: the path is smooth but long or heavily buffered. "Jitter measures the variation in transit time, RFC 3550's smoothed difference between arrival gaps and send gaps, so a constant delay of any size gives jitter near zero. 400 ms round trip is about 200 ms each way before encoding, buffering and decoding, four times Microsoft's 100 ms target. Usual causes are media hairpinning through a far-away VPN concentrator or relay, a VPN that isn't split-tunnelled for media, or a queue that's always full. I'd check which path the media takes and whether delay grows under load; the fix is routing, not a bigger jitter buffer, which only adds delay."

Learn more: Media quality and diagnostics.

Because the server is the device, and a fleet runs many versions of it at once, for years. "A web service deploys once and every client sees the new version; a fleet has hardware that can't take the new release, customers who pin versions for change control or partner certification, devices that are rarely online, staged rollouts and one-way upgrades. So compatibility has to work in two directions: old clients against new firmware, which the firmware owes them, and new clients against old firmware, which the client handles by discovering what each device supports. Poly's public 5.0 notes are a live example: 5.0.0 is one-way, only the second-generation models can run it, and 4.x releases continue alongside it."

Learn more: Why devices make versioning hard.

The firmware owes the old integration; the new dashboard owes the old firmware. "The 2021 integration can't change, nobody will update it, so every newer firmware must keep its requests working and its response fields present with the same meaning: that's backward compatibility, and it's the firmware team's job. The dashboard will meet 7.0 and 7.3 devices that have no microphone zones and never will, so it must find out per device whether zones exist and hide or explain the panel: that's capability discovery, the client's job. Each obligation lands on the only side that can still change."

Learn more: Why devices make versioning hard and Capability discovery.

A major version in the URL, kept for years, with only additive changes inside it. "/api/v1 is visible in logs and docs, caches correctly with no extra headers, works from a browser link, an <img> or any control system's HTTP client, and is trivial to route on a small on-device server. There are no minor versions on the wire: clients learn about new features from a capabilities endpoint. Each firmware advertises the majors it serves, a client picks the highest both sides know, and requests that name no version get the oldest behaviour, because those come from the oldest clients. A header or vendor media type only when one resource needs a new representation, and then always with Vary."

Learn more: Versioning strategies: URL, header and media type.

The browser's cache keys the response on the URL alone, so for up to five seconds whichever version was fetched first is served to both pages. "The new admin UI asks with Api-Version: 2 and the cache stores the v2 body, where volume is an object; the old page asks the same URL without the header within five seconds and gets the cached v2 body, and Math.round of an object is NaN. It only happens when the requests land in that order inside the cache lifetime, which is why it can't be reproduced with curl. The fix is Vary: Api-Version on every response from that URL, or moving the version into the URL, which puts it in the cache key."

Learn more: Versioning strategies: URL, header and media type.

The provider only adds; the reader only reads what it needs; enums are open; types, units, meanings and defaults never change. "New endpoints, new optional request fields and new response fields are fine. Nothing is removed, renamed or given a new meaning, and status and error codes stay the same. Clients are told from day one to ignore unknown fields and to show unknown enum values rather than crash, Martin Fowler's tolerant reader. On writes I'm strict: old firmware that receives a setting it doesn't know rejects the whole write with a list of unknown fields, so an admin never sees 'saved' for something that wasn't. Renames are additive, with both names filled, and removal waits for a new major. Security fixes can justify a break, but then secure by default with an explicit, logged compatibility switch."

Learn more: Rules for compatible change.

(a) breaking, (b) safe for writers but risky for readers, (c) breaking. "Absent and null are different shapes: a client that checks status.sip === null now sees undefined and takes the wrong branch or throws, so keep returning null. Old writers only ever send 0 to 100, still valid, but old readers may now see 110 and draw a volume bar past its end or reject it, so report the range in the capabilities document. And a default is behaviour clients observe: rooms reset to factory settings would start answering calls by themselves, which belongs in a new major version or behind an explicit setting."

Learn more: Rules for compatible change.

Reject the whole write with a list of unknown settings, unless the client asked for lenient handling. "The worst answer is the common default: silently ignore the unknown field and return 200, so the admin sees 'Saved' for a setting that was never applied. Applying volume but not noiseBlock would leave the device half-configured while the request 'failed', so the write is all or nothing: 422 with unknownSettings, because the JSON is fine but its content isn't acceptable to this firmware. If the client sent Prefer: handling=lenient (RFC 7240), it has said it can cope with partial application, so the device applies what it knows, answers 200 with Preference-Applied: handling=lenient and lists what it ignored, and the manager can say 'applied, except noise block: not supported on 7.2'."

Learn more: Rules for compatible change.

Add volumePercent, keep volume, and fill both from one stored value for the life of the major version. "Never redefine volume: a client reading the wrong scale doesn't fail, it sets the room to 7% instead of 70%. The device stores one value and derives both fields from it on every read, so they can't drift; an old client writing { volume: 4 } stores 40, a new one writes volumePercent directly, and a client sending both inconsistently gets a 422 naming the fields. Mark volume deprecated in the spec and remove it only in a new major, once fleet data says almost nobody uses it."

Learn more: Rules for compatible change.

It asks the device, and renders from features, never from version numbers. "Two devices on the same firmware can differ by hardware model, provider mode, licence, connected peripherals and admin policy, so the UI reads a capabilities document with stable feature names, a version per feature, allowed values and limits, and a reason when something is off. Firmware from before the endpoint falls back to a frozen table keyed by version, which is safe because released firmware never changes. And the version check itself is buggy: Number("7.10") is 7.1, which is older than 7.3."

// Two version-comparison mistakes that show up in device clients, and the fix.
console.log(`Number(): is 7.10 newer than 7.3? ${Number("7.10") > Number("7.3")}  (Number("7.10") is ${Number("7.10")})`);
console.log(`strings : is "1.10.0" < "1.9.3"? ${"1.10.0" < "1.9.3"}  (compared character by character)`);
const cmp = (a, b) => {
  const [x, y] = [a, b].map((v) => v.split(".").map(Number));
  for (let i = 0; i < Math.max(x.length, y.length); i++) if ((x[i] ?? 0) !== (y[i] ?? 0)) return (x[i] ?? 0) - (y[i] ?? 0);
  return 0;
};
console.log(`numeric parts: 7.10 newer than 7.3? ${cmp("7.10", "7.3") > 0}; 1.10.0 newer than 1.9.3? ${cmp("1.10.0", "1.9.3") > 0}`);

Number(): is 7.10 newer than 7.3? false  (Number("7.10") is 7.1)
strings : is "1.10.0" < "1.9.3"? true  (compared character by character)
numeric parts: 7.10 newer than 7.3? true; 1.10.0 newer than 1.9.3? true

Learn more: Capability discovery.

Use it as version 2 and build the dropdown from the reported values, including presenter, which is only safe if feature versions are additive. "Hiding camera tracking on the newest devices because they're 'too new' would be the worst outcome a compatibility scheme could produce. If the capability document's own specification says a higher feature version does everything the lower one did, plus more, every request the UI knows for version 2 still works. An incompatible redesign gets a new feature name, such as camera.trackingZones, and the old name keeps being reported for as long as it works."

Learn more: Capability discovery.

Announce in two steps with standard headers, keep behaviour identical, measure who still uses it, and remove only in a new major firmware line. "The firmware sends Deprecation (RFC 9745) on the old resource with a rel="deprecation" link to the migration guide, and Sunset (RFC 8594) with a link to the retirement policy; the resource keeps working exactly as before. On a device nobody can switch anything off on a date: firmware already shipped serves the old API for as long as it runs, so Sunset honestly means 'releases after this date may remove it', and because the headers are baked into firmware, I pick dates I can keep. Brownouts are impossible on customers' devices; instead a later release can turn the old feature off by default for new installs while keeping it on, behind a setting, for upgraded devices that use it."

// Reading Deprecation (RFC 9745: "@" + Unix seconds) and Sunset (RFC 8594: an HTTP-date) from a response.
function read(headers) {
  const deprecated = new Date(Number(headers.deprecation.slice(1)) * 1000);
  const sunset = new Date(headers.sunset);
  const ok = sunset >= deprecated; // RFC 9745 section 4: Sunset must not be earlier than Deprecation
  return `deprecated ${deprecated.toISOString().slice(0, 10)}, sunset ${sunset.toISOString().slice(0, 10)}: ${ok ? "consistent" : "CONTRADICTORY, ask the vendor; treat as possibly gone now"}`;
}
console.log("v1 audio:  ", read({ deprecation: "@1782863999", sunset: "Fri, 31 Dec 2027 23:59:59 GMT" }));
console.log("mystery:   ", read({ deprecation: "@1798761599", sunset: "Wed, 31 Dec 2025 23:59:59 GMT" }));

v1 audio:   deprecated 2026-06-30, sunset 2027-12-31: consistent
mystery:    deprecated 2026-12-31, sunset 2025-12-31: CONTRADICTORY, ask the vendor; treat as possibly gone now

Learn more: Deprecation and sunset.

A matrix: every supported client, including recorded partner integrations, against every firmware in the support window. "On pull requests it runs against simulators built from one factory, with each release's differences written down explicitly, and against recorded fixtures: responses captured from real devices on each released firmware, in every state that matters. Released firmware never changes, so those recordings stay valid forever. Nightly it runs on lab devices; release candidates get hardware-in-the-loop. Before a release I diff its API against the previous one twice, the OpenAPI specs and the recordings, because recordings catch changes nobody wrote down, like a number turning into a string. The matrix runs with fail-fast off, and a red cell blocks the release that caused it until someone decides explicitly."

Learn more: Testing compatibility across versions.

Separate people from programs. "Admins log in through the web UI and get a short session: a random id in an HttpOnly, Secure, SameSite=Strict cookie, with idle and absolute timeouts, and CSRF protection because cookies are sent automatically. Integrations get their own API tokens, minted by an admin, scoped to a job like status:read or calls:control, stored on the device only as a hash, listed in the UI and revocable one at a time. Everything runs over TLS, so Basic auth is only a compatibility option inside HTTPS. I wouldn't put JWTs on a single device: it can look up its own sessions for free and revoke them instantly; signed tokens make sense when someone else, like the fleet service, issues them."

Learn more: Authenticating admins and integrations.

First 403 Forbidden with insufficient_scope; after revocation 401 Unauthorized with invalid_token. "On the first call the token is genuine and current, so authentication succeeded and authorisation failed: 403, with WWW-Authenticate: Bearer realm="device", error="insufficient_scope", scope="system:control", the format RFC 6750 defines, so the integrator knows which scope to ask for and doesn't retry. After revocation the device can't find the token's hash, so the caller is no longer authenticated: 401 with error="invalid_token", which tells the client to get a new token rather than retry."

Learn more: Authenticating admins and integrations.

Encrypt from first boot with a self-signed certificate, and make clients authenticate the device by pinning or by the organisation's own CA, never by turning verification off. "Public CAs won't issue a trusted certificate for a private IP or an internal name, so the device generates its own key at first boot and serves a certificate whose subject alternative names include its addresses and local host name. Clients pin it: trust on first use with the fingerprint compared against the device's screen, or a fingerprint delivered by provisioning. Enterprises have the device create a CSR, their CA signs a certificate for its DNS name, and managed laptops trust it with no prompt. I'd pin the public key rather than the certificate so renewals don't break clients, treat a key change after factory reset as an event an admin confirms, and make sure the device has a correct clock before it validates anyone else's certificate."

Learn more: TLS on devices.

Both break a certificate pin; a public-key (SPKI) pin survives the renewal and still catches the reset. "A certificate fingerprint hashes the whole certificate, so a new serial, new dates and a new signature change it even with the same key. The SPKI hash covers only the public key, so a same-key renewal leaves it identical and the client carries on. A factory reset generates a new key, so the SPKI pin fails and the client stops and asks an admin to re-confirm, which is right: a new key is indistinguishable from an impostor until someone with another source of truth confirms it."

Learn more: TLS on devices.

Assume the attacker can't reach the device but the admin's browser can, and check three things on every state-changing request. "Where did it come from: Sec-Fetch-Site: same-origin or an Origin that's one of the device's own. Which name was used: a Host header that isn't one of the device's own names or IPs gets 421 Misdirected Request, which is what defeats DNS rebinding, because there the browser thinks the request is same-origin. And a per-session CSRF token that pages on other sites can't read. Cookies are __Host-, Secure, HttpOnly, SameSite=Strict, GET never changes state, the UI sends frame-ancestors 'none', and integrations use bearer tokens, which aren't ambient, so they skip CSRF checks."

Learn more: Securing the on-device web UI.

Because the web UI displays strings that arrive through channels the web team never sees, and stored XSS defeats every CSRF defence. "If the UI builds HTML by concatenation, that room name runs script in the device's origin, with the admin's session, the page's CSRF token and every API the session can call. Two layers fix it: validate on input at every interface, with the same schema for the CLI, provisioning and the web form, so a room name has a length and a character set; and escape on output in the UI, because input validation will eventually miss a channel. A strict CSP with no unsafe-inline is the third layer: even if markup gets in, inline handlers like onerror don't run."

Learn more: Securing the on-device web UI and One command layer behind REST, Telnet and SSH.

Credentials and exposure. "No universal default password: each unit gets a unique random secret that isn't derivable from public information like its MAC address, and it does nothing useful until an admin sets a real password, checked by length and a blocklist rather than composition rules, as NIST SP 800-63B-4 says. Login on every interface is rate-limited in a way that slows attackers without locking out the real admin. Then an inventory of every listening port and service, with what the customer hasn't asked for turned off (Telnet, SNMP v1/v2c, debug ports, adb over the network), unauthenticated responses that don't reveal versions, and TLS on with a sane minimum. I'd measure against ETSI EN 303 645 and the OWASP IoT and API Top 10, so it's a checklist, not opinion."

Learn more: Hardening.

Anyone on the network can keep the only admin locked out forever by sending five bad passwords every 30 minutes. "Lockout counts failures for the account, not the person, and the username on a device is documented. In the chapter's simulation, a 15-minute lock cut the attacker to 480 guesses a day and locked the real admin out both times. I'd propose per-source exponential back-off, which slowed one attacker to 105 guesses a day without touching the admin, plus an account-wide ceiling for many-address attacks, a recovery path the network can never lock (the local touch interface or a physical reset), alerts on throttling, and above all a strong unique password, so even 2,100 guesses a day from twenty addresses are harmless."

Learn more: Hardening.

One policy table, consulted by every interface, that maps roles and token scopes to each setting and action. "Reads are filtered per field and secrets are write-only, so nobody, not even an admin, can read a SIP or Wi-Fi password back; they can only replace it. Writes are validated against the schema: an unknown field like isAdmin is a 422, and a forbidden field rejects the whole change with a 403 that names it. Destructive functions like factory reset are admin-only and ask for the password again. In a fleet service I'd also check ownership of every id in the path, so one tenant can't touch another's devices. And hiding a button isn't authorisation: the check lives in the shared command layer, because the API and the CLI answer anyone with a valid session."

Learn more: Least privilege and audit logs.

The log was tampered with: entries 813 to 830 existed and were removed, even though what's left verifies on its own. "A hash chain, where each entry's hash covers its contents and the previous entry's hash, proves only that nobody edited an entry without recomputing everything after it. An attacker can re-hash the whole chain, or simply delete the newest entries, which a chain can't detect alone. The anchor is an outside record: every few minutes the device sends its newest sequence number and hash somewhere its users can't edit, so a truncated or rewritten log no longer matches it. What you can't conclude is what the missing entries said, which is why devices ship the full entries off-device too."

Learn more: Least privilege and audit logs.

Trust comes from a signature the device checks, not from the channel the image arrived on. "The vendor signs a manifest with the product, version, a security version and the image hash, using a key held in an HSM that only the release pipeline can use. The device has the public key in its read-only system image, verifies the signature over the exact bytes before parsing anything, then checks the product, that the security version is at least its stored rollback floor, and the image hash as it streams in. It installs to the inactive A/B slot, reboots, and verified boot checks each stage; if the new version fails its health checks it falls back automatically, and only a healthy boot raises the rollback floor. Every step is audited, and every release ships with an SBOM."

Learn more: Supply chain and update security.

Accidental corruption and tampering between the site and the admin; not a compromised site, not a downgrade, and not a pasted wrong hash. "The hash and the image come from the same place, so whoever can replace one can replace both. An old genuine release has a genuine hash on its own old page, so it passes. The fix moves trust from what the website says to what the vendor's offline key signed: the hash lives inside a signed manifest, the device holds the public key from the factory, a rollback floor refuses old versions, and the admin no longer types anything security-critical."

Learn more: Supply chain and update security and, for rollback and fix-forward, CI/CD for web UI, APIs and firmware.

It lists every component in a build, so "are we affected by this new CVE, and on how many devices?" becomes a query instead of a week of asking teams. "Each firmware build gets a machine-readable bill of materials, SPDX or CycloneDX, with every component's name, version, supplier, package URL, hashes and licences, generated by CI from what was actually built and kept with the release. Joined with the fleet's version counts, a new advisory turns into 'build 5.0.2 is affected, 1,874 devices need 5.1.0'. And regulation is making it mandatory: the EU's Cyber Resilience Act requires an SBOM covering at least top-level dependencies."

Learn more: Supply chain and update security.

A security fix is also an API change, and the shape worth copying is secure by default with an explicit compatibility switch. "Login-then-POST scripts that had worked for years started failing until they learned to send the header, so HP added a setting, on by default and recommended to stay on, for systems that can't send the token. That's the honest pattern for a breaking security change: secure default, a documented switch for clients that can't change, a clear recommendation, and release notes that name it. One thing I'd add that the public notes don't mention: log when the switch is used, so you know who still depends on it."

Learn more: Rules for compatible change and Poly video devices and PolyOS in brief.

In a foreground service inside a privileged system app, or as a native daemon started by init, depending on what it must do and who ships its updates. "Android was designed for phones, where any app can be killed to save memory, so a server that must answer at 3 a.m. has to live somewhere Android treats as 'must stay up'. If it's mostly a front end for framework APIs and we want to ship it as an APK, a foreground service in a system app: preinstalled in priv-app with its privileged permissions allowlisted, signed with the platform key so it holds the platform's signature permissions, and restarted on BOOT_COMPLETED and MY_PACKAGE_REPLACED. If it must be up before the framework, own port 443 directly or survive anything, a native daemon that init restarts, talking to the framework over Binder. Either way the HTTP layer stays thin, and every platform failure maps to a clear status."

Learn more: Web UIs and APIs on an Android-based device.

Installing an update kills the app's process, and START_STICKY doesn't restart a service whose package was replaced. "START_STICKY asks Android to recreate a service killed for memory. After a package replacement nothing in the new version runs until an entry point fires: an activity, a broadcast, a binding. The entry point for this case is the MY_PACKAGE_REPLACED broadcast, sent only to the app that was just updated, and it's one of the documented exemptions from Android 12's ban on starting foreground services from the background, so the receiver may call startForegroundService(). On the chapter's Android 16 emulator, adding it brought the server back within milliseconds of the install."

Learn more: Web UIs and APIs on an Android-based device.

First confirm the real starting point, then split the work along two dials: the OS version on the device and each app's targetSdkVersion. "HP's public 5.0.0 notes say the second-generation systems go from Android 10 to 13 and the TC10 controller from 11 to 12, while the role description says 11 to 13, so I'd ask which base each device and app is on, because if it's 10, the Android 11 changes are part of the jump. Upgrading the OS brings the 'all apps' changes to every app at once, so those are tested with the apps unchanged. Retargeting each app to 33 brings the target-gated ones: explicit exported, PendingIntent mutability, no background foreground-service starts, package visibility, new Bluetooth and Wi-Fi permissions. Build failures first, then runtime behaviour on the new OS: boot, update, reboot, the service being stopped, permission denial."

Learn more: Android 11 to 13: changes that matter and Migrating a legacy device app to a new OS version.

Android 12, for apps targeting 31: no foreground-service starts from the background, and explicit exported. Android 13, for all apps: the Task Manager and the notification permission. "From Android 12 a target-31 app that calls startForegroundService() from the background gets ForegroundServiceStartNotAllowedException, unless the trigger is on the exemption list, such as BOOT_COMPLETED or MY_PACKAGE_REPLACED; being a system app isn't on it by itself. On Android 13, regardless of target, users can stop an app with an ongoing foreground service from the notification drawer, with no callback, so the server can be stopped by a person. And POST_NOTIFICATIONS is a runtime permission, but if it's denied the service keeps running; only its notification disappears from the drawer."

Learn more: Android 11 to 13: changes that matter.

Through Binder: bound services with AIDL interfaces, protected by a signature permission and checked on every call. "Every HTTP request becomes one or more Binder calls into another process. The platform's service declares a signature permission, so only apps signed with the same key, usually the platform key, can bind, and it calls enforceCallingPermission in every method, because the kernel stamps each call with the caller's uid and that can't be forged. On the client I handle all three outcomes of bindService (connected, false, SecurityException), map a dead or disconnected service to 503 with Retry-After, and rebind in onBindingDied, which is what happens when the platform side is updated."

Learn more: Talking to platform services.

The v2 service runs a different method, and nobody gets an error. "AIDL proxies send a transaction number, FIRST_CALL_TRANSACTION plus the method's position in the file, not its name. Inserting a method shifts every later number by one, so v3's setSetting is code 4, which v2 dispatches as registerListener; on the chapter's Android 16 emulator the settings process logged a Parcel warning, changed nothing, and the client saw false for a valid write and null for a valid read. So interfaces are append-only, never reordered or changed; there's an apiVersion() call clients check before using anything newer; and inside the platform build we'd use Stable AIDL, which freezes versions and makes a missing method throw instead of returning silently."

Learn more: Talking to platform services.

Make every seam an owned, versioned contract, and test it from both sides. "The platform team owns the AIDL interface and the settings catalogue; my team owns the OpenAPI document for the device API. Changes start as short written proposals the consuming teams review asynchronously, interfaces are append-only with a version the client checks, and we negotiate at runtime: apiVersion() on the AIDL side, capability discovery on the HTTP side. My team keeps a fake of each platform service so the web UI can be built without a device, and the same contract test suite runs against the fake on every commit and against the real platform build nightly, so behavioural drift shows up as a named failing rule instead of a customer bug."

Learn more: Working across SDK, platform and web workstreams.

The platform build is out of contract today, but the next step is a contract decision, not a quick fix on either side. "The test did its job: a named rule failed against one implementation, which turns an argument about whose code is wrong into a question about the written contract. Until it's resolved, the device API mustn't silently accept the new behaviour, because the HTTP API documents 422 for out-of-range values and partners may rely on it. If clamping really is better, it's a contract change: proposed in writing, reviewed by the consuming teams, versioned, and then the fake and the tests change with it."

Learn more: Working across SDK, platform and web workstreams.

Follow the request down the stack and read each layer's record in time order. "First the admin's side: a HAR or the browser's Network panel gives the status and timing, so I know whether the device answered and with what. Then the device API's access log for that time and request id, then logcat for the API process and for system_server. A few seconds of 'unreachable' with no reboot usually means 503s, not timeouts, and the classic cause is a platform service crashing on the new value: ActivityManager logs the death and 'Scheduling restart of crashed service … in 4000ms', and the back-off grows if it keeps crashing, which matches 'sometimes'. I'd reproduce with curl through adb forward, check state with dumpsys, and toggle a suspected behaviour change with am compat on a debug build."

Learn more: Debugging across the stack.

Ergonomics, a real error model, discoverability and stability, plus being diagnosable. "The common task is one obvious, typed call with sensible defaults, device.updateSettings({ volume: 55 }), with options in an options object so they can grow. Failures are typed classes mapped from the API's problem details, carrying the fields, the request id and whether it's safe to retry, so callers react without parsing strings. The editor teaches the API through types and doc comments, and the examples are tested. And upgrading doesn't break working code: semantic versioning, deprecation before removal, and on a device platform one major that works across the firmware the fleet actually runs. Diagnosable means a logger hook that redacts secrets, because 'what did the SDK send to the room?' is the first question in every support ticket."

Learn more: What makes a good SDK.

SDK: re-login and paging. Application: the dial retry, the toast and the caching policy. "Re-login has exactly one right behaviour: on a 401 because the session vanished, log in once with the stored credentials and repeat the call. Following nextCursor is protocol detail, exposed as an async iterator. Dialling isn't idempotent: if the request reached the device and only the response was lost, a blind retry places a second call, so the SDK reports a timeout that says it may have been applied and lets the application decide. What to show for 'busy', and how stale settings may be, depend on the screen; the SDK makes them easy (typed errors, ETags) without deciding them."

Learn more: What makes a good SDK and Errors, retries and timeouts in an SDK.

Three layers: a typed public surface, one request pipeline every call goes through, and an injectable transport. "The surface is methods like getSettings, updateSettings(patch, options) and listCalls() returning an async iterator, with types generated from the OpenAPI spec and friendlier aliases on top. The pipeline runs in a fixed order: auth, with a lazy single-flight login shared by concurrent calls and one re-login when the device reboots and forgets the session; a per-attempt timeout plus the caller's abort signal; retries only for idempotent calls; problem details mapped to typed errors; debug logging with secrets redacted. The transport is an injected fetch, so the same SDK runs in Node, in the admin UI with cookies, and in tests against a fake device. Every request carries the SDK version."

Learn more: Building a TypeScript client for a device API.

Typed errors mapped from problem details, retries only for transient failures of calls that are safe to repeat, and a timeout and abort signal on every call. "Errors map by the stable problem type first, then by status, onto a base DeviceApiError with the status, the body and the request id, and subclasses like ValidationError with the bad fields or RateLimitError with the delay. Retries happen in one layer, the SDK, for network errors, timeouts, 408, 429 and 5xx, and only for idempotent methods or a POST with an idempotency key; capped exponential back-off with full jitter, honouring Retry-After, about three attempts, and a way to turn them off. Each attempt gets AbortSignal.timeout, combined with the caller's signal by AbortSignal.any: a timeout may be retried, a caller's cancel never is."

Learn more: Errors, retries and timeouts in an SDK.

No blind retry: the call may already be ringing. Make the operation safe to repeat, or make its outcome checkable. "A timeout says nothing about whether the device acted: the request may have been lost on the way in, or the device may have started dialling and the response was lost, in which case a retry puts the room into a second call or gets 'already in a call'. The general fix is an Idempotency-Key on POST /calls: the SDK generates one per dial() and reuses it on retries, and the device returns the original result, so the SDK can then retry safely. Where old firmware can't change, the application reads the active call after a timeout and only dials again if none exists. Meanwhile the error already says 'it may still have been applied'."

Learn more: Errors, retries and timeouts in an SDK and Methods, status codes and idempotency for device APIs.

Semantic versioning about the SDK's own surface, a separate published support matrix for firmware, and capability checks at run time. "There are three version numbers: the SDK's, the device API's and the firmware's, and they move independently. Patches fix, minors add, and majors, batched and rare, cover anything that breaks callers, including subtle cases like a new value in a returned union or changed retry defaults. The support matrix says which device API versions and firmware each SDK major supports; at run time the SDK reads the device's capabilities and throws a clear UnsupportedFeatureError instead of a puzzling 404. Deprecation goes in a minor with @deprecated, a once-only warning and a changelog entry; removal in the next major with a migration guide. Releases come from CI with an API-surface diff, tests against each API version, and npm provenance."

Learn more: Versioning and releasing an SDK.

(a) 1.6.0, (b) 2.0.0, (c) 2.0.0 unless Mode was documented as open, (d) 2.0.0 by most libraries' policy. "A new optional option is a backward-compatible addition: old calls still compile and work. Removing a method breaks existing code however long it was deprecated, so it waits for the next major. A value added to a type the SDK returns breaks every strict caller with an exhaustive switch, so it's major unless the union was published as open from the start. Semver itself doesn't cover runtime support, but users experience losing their Node version as breaking, so most libraries treat it as major."

Learn more: Versioning and releasing an SDK.

The SDK changed under the README: 2.0.0 removed setVolume(), and the example was never run against the release. "Untested examples rot silently whenever the API changes, and the drift is invisible until a user copies the code. Docs are part of the product, so they get tests: pull every example out of the README, run it against a device simulator and the release candidate, compare what it prints with the documented values, and block the release on a mismatch. And the 2.0.0 migration guide should have listed setVolume(level) → updateSettings({ volume: level })."

Learn more: Documentation and examples developers actually use.

Three levels in CI: unit for logic, integration through the real HTTP stack, and contract against the published spec. "Unit tests for what's easy to get wrong: validation, merge rules, state transitions. Integration tests that run the real Express app in process with supertest, sending exactly the headers real clients send, because wiring bugs like a body parser ignoring application/merge-patch+json return 200 for changes that never happened and no unit test sees them. Then contract tests: every response is checked against the OpenAPI spec (status documented, content type documented, body matches), negative tests are generated from the schema's limits, and the spec itself gets a breaking-change diff on every PR. On a device platform the contract layer matters most, because partners and customers' scripts are consumers we don't control."

Learn more: Testing APIs: unit, integration and contract.

(a) unit, (b) integration, (c) contract, (d) end-to-end on a simulator, then hardware in the loop. "(a) is pure validator logic, caught in microseconds with boundary values. (b) is wiring in the error-handling middleware, only visible through the real app, so a supertest request to an unknown path that asserts application/problem+json. (c) is drift the developer's own tests were edited to accept, so only an independent oracle, the published spec saying integer, catches it. (d) is behaviour across UI, API and device: a simulator that models applied versus pending catches the logic in CI, and only real hardware proves the display driver applies it."

Learn more: Testing APIs: unit, integration and contract.

A device simulator built as a first-class product of the team, kept honest by parity checks against real devices. "It speaks the same API, generated from or checked against the spec, and has the device's real state machine for settings, reboots and updates, with a fake clock so a 45-second reboot is instant in tests and fault controls (power loss, slow network, a failed update) on a separate control surface. Every developer runs it locally, CI runs UI and API end-to-end tests on it, and the cloud team can start hundreds. A nightly lab job records real device responses and a parity check fails CI when the simulator drifts. For the Android layer, the Android Emulator or Cuttlefish at the right API levels; for camera, codecs and power loss mid-flash, the lab."

Learn more: Device simulators and emulators.

Only what hardware can show, as matrices, on rigs run as a shared service. "Power cuts at every step of an update, with good and bad builds, so we prove both survival and rollback; peripherals in every plug order; the actual HDMI and audio output; long soak tests for memory and heat; real network impairment; and the update path from every version still in the field. Each rig has switchable power, a serial console, adb and the device API, a switchable USB hub and HDMI capture, driven by a harness like Tradefed or LAVA. Every job starts by flashing and resetting the device, rigs are health-checked and quarantined automatically, jobs are scheduled by capability, failures keep every log and capture, and the lab is remotely bookable, so the India team gets real hardware without shipping devices."

Learn more: Hardware-in-the-loop and lab testing.

Every rig: a regression, block the release. Only rig 3: probably the rig, quarantine and health-check it. Passes on re-run: a flaky test or an intermittent bug, file it and track its flake rate. "A regression fails everywhere, a bad rig fails tests that pass on every other rig, and a flaky test fails and then passes on the same rig. The approach is a limited number of re-runs, then the pattern across rigs. I wouldn't hide the flaky one behind automatic re-runs forever, because it might be a real race, in camera enumeration for example."

Learn more: Hardware-in-the-loop and lab testing.

Build every artefact once, prove it, sign it, and promote the same bytes to the fleet in rings with an automatic health gate. "Pull requests run lint, unit, integration and contract tests, a breaking-change diff of the spec, and end-to-end tests of the built UI on the simulator; merges get a hardware smoke suite; nightly, the full lab matrix. Packaging produces a hashed artefact with a provenance attestation, and release signing happens in an HSM-backed service, never with keys in CI. Rollout uses stable buckets from a hash of release id and serial: 1%, 5%, 20%, 50%, 100%, with soak time, installs only in customers' maintenance windows and never during a call. After each stage a gate compares updated devices with a control group, overall and per model, because a bug on one model is diluted in the aggregate. Halting is cheap and tested; recovery is usually fix-forward."

// Why a fleet-wide dashboard hides a bad release.
// 1) The SRE workbook's example: a 5% canary where 20% of requests fail; the rest of the service is healthy.
const canaryShare = 0.05, canaryFail = 0.2;
console.log(`whole service: ${(canaryShare * canaryFail * 100).toFixed(1)}% failing; canary alone: ${canaryFail * 100}%`);
// 2) A bug on one model (the testing chapter's rollout model): touch panels are about 10% of the updated
//    devices and fail 4%; everything else fails at the background rate of about 0.4%.
const share = 0.1, bad = 0.04, normal = 0.004;
const overall = share * bad + (1 - share) * normal;
console.log(`updated devices overall: ${(overall * 100).toFixed(2)}% against a 0.40% control; touch panels alone: ${bad * 100}%`);
console.log(`overall excess: ${((overall - normal) * 100).toFixed(2)} points; touch-panel excess: ${((bad - normal) * 100).toFixed(2)} points`);

whole service: 1.0% failing; canary alone: 20%
updated devices overall: 0.76% against a 0.40% control; touch panels alone: 4%
overall excess: 0.36 points; touch-panel excess: 3.60 points

Learn more: CI/CD for web UI, APIs and firmware.

They're evaluated locally from a cached, signed config with compiled-in defaults, targeted by hardware and firmware, and they can only switch between code that already shipped. "A device has to work offline and at boot, before the network is up, so every flag has a default in the firmware and the evaluator never throws: it always returns a value and a reason, OpenFeature-style, which is what lets you debug a flag from telemetry. Rules match model, firmware version, tenant and a stable hash of the serial for percentage rollouts, so raising 10% to 25% only ever adds devices. Kill switches for risky features, expiry dates so flags don't pile up, and one rule I'd hold firmly: flags don't change the API's shape, or two devices on the same firmware answer differently and partners can't rely on the spec."

Learn more: Feature flags, telemetry and field diagnostics.

Telemetry buffered on the device and limited to an allowlist; diagnostics on demand, redacted before upload. "Fleet health (crashes, unexpected reboots, update outcomes, call quality) tagged with model and firmware, never with meeting ids or user names, buffered in bounded storage with sequence numbers and a drop counter, so after a weekend offline the collector can tell 'the device dropped data' from 'the pipeline lost it'. For one device, an on-demand diagnostics bundle as a long-running operation, redacted on the device so the support system never holds the secrets, temporary verbose logging that expires by itself, and remote checks behind an admin role with audit logging and correlation ids that tie a click in the web UI to the device's logs."

Learn more: Feature flags, telemetry and field diagnostics.

Building the software that lets a fleet of hardware be configured, controlled, monitored and updated, on a runtime you don't own. "The on-device web UI, the REST and command-line APIs, the connection to cloud management and the integration with the OS. The big difference from a web app: the code runs on thousands of devices on customers' networks, many versions are live at once for years, updates are pulled or pushed through admins and partners on their schedule, and some upgrades are one-way. So I design APIs to stay compatible across versions, make clients discover capabilities instead of assuming them, stage rollouts with health gates, and build diagnostics in, because I can't log in to the device when something goes wrong."

Learn more: What device platform engineering is.

What the public documentation says, and where it stops. "The room systems are the Studio X video bars (X30, X50, X70 and the second-generation X32, X52, X72), the modular G7500 and Studio G62, with TC8 and TC10 touch controllers. They run Poly VideoOS, which HP's community posts and your role description call PolyOS; it's Android-based. VideoOS 5.0.0, released in May 2026, supports only the G62, X32, X52 and X72, needs 4.6.0 first, can't be downgraded, and launched for Microsoft Teams and Device Mode, while 4.7 continues for every provider and the first-generation systems. The same hardware runs Poly Video mode or a partner's app, it's managed through the web interface, Poly Lens, provisioning or the partner's console, and integrators use the REST API and a command-line API over SSH or Telnet. What the public docs don't show is how those interfaces are built inside the device, which is the part I'd like to hear about."

Learn more: Poly video devices and PolyOS in brief.

Don't correct the interviewer from a release note, and don't assert either version: ask. "Something like: 'The public 5.0 notes say the second-generation bars go from Android 10 to 13, and the TC10 controller from 11 to 12. Which baseline is this project's 4.x work on?' It shows I've read the notes and that I care about the exact starting point of a migration, because it changes the work: if the base is Android 10, the Android 11 behaviour changes, like package visibility and the camera and microphone foreground-service types, are part of the jump, not background."

Learn more: Poly video devices and PolyOS in brief and Android 11 to 13: changes that matter.

Inventory, fixtures and a parity check first; settings migrations and contract tests next; then rings with health gates for an upgrade that can't be rolled back. "I'd start with an inventory of the whole 4.x surface: every REST endpoint, CLI command, setting, default and event, including undocumented behaviour integrations rely on, and record real request-response pairs as fixtures. Parity is then measurable: replay them against the 5.x build in CI and classify differences by shape, so added fields pass and removed fields, type changes and status changes become a burn-down list. Settings written by 4.x need numbered, idempotent migrations that keep unknown keys. The contracts with the platform team go under contract tests. And the rollout follows the shape the public notes show: a bridge release, the new OS on one mode first while 4.x keeps getting sustaining releases, every ring gated on health signals with soak time, and a fix-forward pipeline, because there's no rollback."

Learn more: Migrating a legacy device app to a new OS version.

It shrinks the problem from "every 4.x version to 5.0" to "4.6 to 5.0": one source format, one upgrader, one tested path. "Without a bridge, the 5.0 installer and its data migrations would have to handle every older layout of settings, partitions and partner data still in the field, and the test matrix would include every one of those starting points. With one, 4.6.0 can do preparatory work that's easier on the old OS: migrate settings, check space, update the update agent, refuse a mode 5.0 doesn't support. HP's own wording names the reason: skipping 4.6.0 is 'an untested upgrade path that may result in your system not functioning correctly'. The cost is time: devices far behind take two updates."

Learn more: Migrating a legacy device app to a new OS version.

The public notes say a 5.x system can't go back to 4.x, and even a factory restore lands on 5.0.0, so recovery has to be forward. "'Once you update your system to Poly VideoOS 5.0.0, you cannot downgrade', and the release 'also updates the system's factory restore partition'. The notes do describe a manual downgrade in Lens by choosing a previous version, but read together with that line, my reading is that it moves between 5.x builds, never back to 4.x; the notes don't spell that out. So every gate has to be passed before a device upgrades: longer soak in early rings, health checks that block promotion, and a pipeline that can ship a 5.0.x fix quickly, which is what 5.0.1 in August 2026 looks like in public."

Learn more: Poly video devices and PolyOS in brief and CI/CD for web UI, APIs and firmware.

One plausible reason is partner certification dates, but that's an inference, so raise it as a question. "Microsoft's table for Teams devices on Android lists a certification end date of 3 September 2026 for Android 10 and 11, already extended once, and 15 August 2028 for Android 12 and 13. A platform whose Teams rooms run Android 10 has a strong reason to get Teams onto 13 first. HP's notes don't connect 5.0's Teams-first scope to those dates, so I'd say what's public and ask: 'How do you decide which provider modes come next, and what does parity mean for the web UI and API in each?'"

Learn more: Poly video devices and PolyOS in brief and Room systems and provider modes.

No: a month after release only part of the fleet will have it, so their code must check per device and handle "not supported". "In the chapter's adoption model about 53% of the fleet had the new release after four weeks and 18% still didn't after twelve, because Teams and Zoom rooms wait for an admin, devices are switched off, and staged rollouts take weeks. So the answer is: 'You can call it where the device advertises it. Here's the capability flag, here's what older devices return, and here's our rollout schedule so you can watch adoption grow.' If they treat a 404 from old firmware as an outage, they'll page themselves for a version mismatch."

Learn more: What device platform engineering is and Capability discovery.

Start from ownership: for every setting, decide whether the device platform owns it in every mode or only in one, and put that in one registry the UI, the REST API and provisioning all use. "Network, peripherals, security, updates and the provider choice itself belong to the device in every mode; SIP and H.323 call settings belong to the native video mode, because in Teams or Zoom mode the partner app and its cloud own calling. Settings that don't apply are shown read-only with 'managed by Microsoft Teams Rooms', not hidden, and a write that conflicts with the mode gets a 409 with problem details. Changing the provider is a restart-required change with validation up front: is it available for this model and software version, does it support the current network, is there a licence. And the dashboard says which mode the room is in and who manages what."

Learn more: Room systems and provider modes.

Structure: the hard parts of device APIs are API problems under stricter rules; then one real example; then an honest boundary. "Something like: 'An API on a device is used by many firmware versions at once, by partners' scripts I'll never see, for years, so I design for compatibility first: additive changes, tolerant readers, capability flags, deprecations with dates, and recorded fixtures that prove the new build answers like the old one.' Then a real example of keeping an API or client compatible, from your own work, in two sentences. Then the boundary: you build Android apps, so you know the app-level platform; system services, platform signing and OTA pipelines are the platform team's expertise, and your job is a clean contract with them and tests on both sides."

Learn more: Talking about this role in an interview.

A six-line skeleton you fill with your own facts, not an answer to memorise. "One line on who you are: full stack, your years, React, Angular, Node and databases, and you build Android apps. The bridge to devices: a real project where the runtime, the client or the server was something you couldn't fully control, and what made it hard. A decision you made, against which alternative, why, and the result. The Android side: what you've really built, and what you know only at the design level. Leading people: a real example of mentoring, reviewing or leading, or say honestly that it was informal. Why this role: owning the web UI and API layer on a platform where compatibility matters, and the 4.x to 5.x migration is exactly that kind of problem."

Learn more: Talking about this role in an interview and the Tech Lead site's story prompts.

Use STAR with the camera on the action, and make the compatibility decision the centre of the story. "Headline in one sentence, result first if it's strong. Situation and task in about thirty seconds: the API, who depended on it, why it had to change. Action in about a minute: three to five things you did, each with a 'because': how you found out who used what, how you kept old clients working (an added field, a version, a flag), how you told people, how you knew when it was safe to remove the old thing. Result with a number if you have one, and how you know. Then what you'd do differently. Prepare for 'why not just version it?' and 'what if a client never migrated?'"

Learn more: the Tech Lead site's The STAR method and its variants, and Rules for compatible change for the technical half.

Ownership, containment, fix, prevention, in that order, and keep the blame out of it. "Situation and task briefly: what shipped, how you found out it was broken, how many people it affected. Action: what you did first to stop it getting worse (halt, roll back or fix forward, and why that choice), how you found the cause, how you kept the people affected informed, and what you changed so it couldn't happen the same way again: a test, a gate, a check in the pipeline. Result: how quickly it was contained and fixed, and whether the prevention has held. Learning: the one practice you've kept since."

Learn more: the Tech Lead site's Questions for a lead full stack role, and CI/CD for web UI, APIs and firmware for the technical half.

Show that you moved the disagreement onto evidence and a written decision, and kept the relationship. "Situation and task: which teams, what the interface was, what each side wanted and why both views were reasonable. Action: how you made the disagreement concrete (a test that failed, data, a written proposal with options), who decided and how, and what you did to keep working together while it was open. Result: what was decided, whether it held, and what it did for the next disagreement. Prepare for 'what if they'd refused?' and 'what would you do differently?'"

Learn more: Working across SDK, platform and web workstreams and the Tech Lead site's story prompts.

Contracts and mocks so nobody waits, reviews and decisions that work asynchronously, and the overlap hours used for unblocking. "Design-first API changes as spec pull requests a developer can propose at the end of their day and get reviewed overnight; a device simulator and fakes of the platform services so they can build and test without hardware or without waiting on another team; a remotely bookable lab for what only hardware shows; small pull requests with reviews that teach; decisions written down. And the honest part: if you've led or mentored people across time zones, give the real example; if your experience is informal, say so and describe what you'd do in the first weeks."

Learn more: Design-first vs code-first, Device simulators and emulators, and the Tech Lead site's Working across US and India time zones.

Be exact about your experience, then show understanding. "Start with the honest boundary, phrased truthfully for where you are: 'I haven't shipped a Telnet interface in production; I've studied the protocol and worked through runnable models of line-based command servers in Node', only if you have run and changed them. Then the substance: plain text over TCP with option negotiation, kept because room control systems and years of scripts speak it. Show you've read their docs: the command-line API on port 24 by default or 23, SSH on 22 as the encrypted option, both off by default, and Telnet with no port lockout, just a disconnect after three failed logins. Then the trade-off and the design you'd bring: off by default, trusted networks only, steer towards SSH, one command layer behind REST, SSH and Telnet, scripted session tests and fuzzing."

Learn more: Talking about this role in an interview and What Telnet is and why devices still expose it.

Questions that show you've read the public material and think about the problems they actually have. "Which Android baseline is the 4.x work on, given that the public 5.0 notes say 10 to 13 for the Gen2 bars? How do you decide which provider modes come to 5.x next, and what does parity mean for the web UI and API in each? Do the REST API, the command-line API and the web UI share one command layer on the device, or have they grown separately? How do the web and API teams work with the Android platform team day to day? How is the India team organised, how much overlap is there, and what do they own end to end? What do the lab and rollout pipeline look like for a 5.x build, and which health signals block promotion?"

Learn more: Talking about this role in an interview and the Tech Lead site's Questions to ask the interviewer.

Overclaiming Android platform depth, and presenting guesses about their product as facts. "An interviewer who works on the OS will ask one follow-up about Binder, SELinux policy or OTA partitions and find the edge in seconds, so I draw the boundary myself first: what I've built at the app level, what I understand at the design level, and that platform internals are the platform team's expertise. And if I say 'PolyOS uses X internally', I'm inventing; instead: 'the public docs describe the REST API and the command line; I don't know how they're built inside, but here's how I'd structure it', which gets credit for both the research and the design."

Learn more: Talking about this role in an interview.