Portals wire protocol v1

Status: implemented in Portals.Protocol, Portals.Codec.MessagePack, Portals.TermExtensions, and Portals.Handshake. This document must stay in sync with those modules; the modules are authoritative where they disagree.

This is a language-neutral specification: a new SDK can be implemented from this document, protocol/vectors/golden.exs, and protocol/malformed/fixtures.exs alone, without reading Elixir source.

1. Transport framing

Each frame is emitted as a MessagePack-encoded envelope, length-prefixed on the wire as a 4-byte big-endian unsigned length followed by that many bytes of envelope. The length prefix is transport framing, not part of the codec payload measured against max_frame_size below (the codec measures the encoded envelope's own byte size).

2. Envelope shape

Every frame is a MessagePack array whose first element is an integer frame tag:

[tag, field_1, field_2, ...]

Frames have fixed minimum arity and an optional bounded number of trailing fields, given per frame below. Decoders must reject any envelope that is not an array, whose first element is not an integer, whose tag is unknown, or whose arity is out of range, without raising and without malformed input propagating past that connection.

3. Frame tags and layouts

error_map is a string-keyed map with at least kind and message, and may include details, remote (foreign language/exception metadata), and a bounded, normalized stacktrace.

4. Handshake and version matching

The worker connects and sends HELLO first. protocol_version must equal the BEAM's own protocol version (Portals.Protocol.version/0, currently 1) exactly. On a match, the BEAM replies with READY carrying its configured limits (§6) and the connection becomes usable. On any mismatch, the BEAM never sends READY; it closes the connection and worker startup fails deterministically. There is no negotiation of intermediate versions.

HELLO's max_concurrency and max_streams are the worker's own advertised capacities and are independent of each other: max_concurrency bounds concurrently executing unary CALLs, max_streams bounds concurrently open streams (§9.5). A worker that does not implement streaming advertises max_streams 0.

5. Value model

Core MessagePack types:

• nil, true/false
• signed/unsigned integers representable in 64 bits, using the smallest MessagePack integer encoding available
• IEEE 754 double-precision floats
• UTF-8 strings (MessagePack str)
• raw binaries (MessagePack bin) — see §5.2 on how implementations pick str vs bin
• arrays
• string-keyed maps (non-string keys are rejected)

5.1 Erlang-value extensions

Layered on MessagePack ext (fixext1/2/4/8/16 or ext8/16/32), each tagged with a Portals extension type code:

PIDs and references are opaque byte tokens to foreign SDKs: a trusted worker may store and echo them back (e.g. as a MESSAGE target) but must not attempt to construct or interpret their bytes itself.

5.2 str vs bin

The reference Elixir codec treats plain host-language strings as MessagePack str by default. Binary payloads that are not meant to be interpreted as UTF-8 text (large blobs, images, arbitrary bytes) must be wrapped by the sending SDK to force bin encoding (in Elixir: %Portals.Codec.Bin{data: bytes}). Decoders MUST NOT assume bin payloads are valid UTF-8, and MUST return bin payloads to the host language as a distinct byte-buffer type rather than a text string.

5.3 Safe decoding rules

• Atoms decode only to an atom that already exists in the runtime. Decoding text for which no existing atom exists is a decode error ({:unsafe_atom, text}); decoders must never call an unsafe atom-creation primitive. hosts without a persistent atom table (Ruby, Python, Node) MAY decode atoms as native symbols/strings, since the safety concern is specific to the BEAM.
• PIDs and references decode only through a safe, non-code-executing deserialization primitive, and the result is type-checked to confirm it is actually a PID/reference (not an arbitrary term) before it is accepted. Both local and distributed PIDs are accepted.
• Decoders never evaluate, eval, or otherwise execute anything from the wire.

6. Limits

Every connection enforces caller-configured limits (see Portals.Protocol.default_limits/0 for the shipped defaults):

max_frame_size bounds the complete encoded envelope, independent of max_collection_length, which bounds element counts (not raw binary payload sizes — a single large bin/str payload is bounded only by max_frame_size).

Exceeding any limit on encode or decode is a codec error, not an exception; callers must close only the offending connection.

7. Request semantics

• request_id is an unsigned 64-bit integer, monotonically increasing per connection. A connection must be restarted before ID space exhaustion rather than wrapping.
• Exactly one terminal RETURN or ERROR is sent per accepted CALL. Responses may arrive out of order relative to CALLs.
• CANCEL is idempotent: sending it after the terminal response already arrived (or is in flight) has no effect and produces no additional response.
• An unknown request_id on a terminal frame is logged/reported and otherwise ignored, unless it recurs enough to exceed a configurable protocol-error threshold, at which point the connection is closed.
• CALLBACK/CALLBACK_RETURN/CALLBACK_ERROR use their own independent ID space (callback_id) and must never be executed inline on the socket reader — nested calls and out-of-order responses must not deadlock the connection.

8. Reentrant callbacks and PID messaging

• CALLBACK's optional trailing depth is self-reported by the worker (the BEAM does not and cannot independently derive it, since nothing else in the protocol links a CALLBACK to whichever CALL it is nested under). This is safe specifically because Portals workers are trusted (see the PRD's non-goals: hostile workers are out of scope); the BEAM still enforces max_callback_depth and max_in_flight_callbacks against the reported value, it just does not attempt to verify it independently. A CALLBACK omitting depth is treated as depth 1.
• To make depth accounting work across a fully reentrant chain (BEAM calls worker → worker's handler triggers a CALLBACK → the BEAM callback handler makes a nested CALL back into the same worker → that nested call's handler triggers another CALLBACK, etc.), the BEAM echoes the depth it observed on the inbound CALLBACK as the callback_depth field of any CALL frame dispatched from within that callback's handler. A worker's dispatcher reads this field and uses it as the baseline when it next reports a CALLBACK's own depth (baseline + 1). A top-level CALL (not originating inside a callback handler) always carries callback_depth 0.
• The connection's socket reader must never execute the target module.function inline: every CALLBACK is dispatched to an independent supervised task, so a slow, blocked, or reentrant handler can never stall reading further frames or delivering other in-flight responses.
• Exceeding max_callback_depth or max_in_flight_callbacks is reported back as a terminal CALLBACK_ERROR with kind: "overload"; it never silently drops the CALLBACK and never blocks the connection.
• MESSAGE's target_pid is decoded per §5.3 (safe decode, then type-checked as an actual PID) before delivery. Delivery uses ordinary BEAM send/2 semantics: fire-and-forget, no acknowledgement, and no guarantee the target process still exists. Implementations may wrap the delivered value in an envelope (e.g. Elixir's reference implementation defaults to {:portals_message, connection, value}) so a target process can distinguish Portals messages from its other traffic; this is a local delivery-side convention, not a wire-format concern.

9. Bidirectional streaming and backpressure

9.1 Opening a stream

There is no separate "open stream" frame. A stream is opened by the BEAM as an ordinary CALL, and that call's request_id is the stream's stream_id; the two share one ID space per connection, so a stream_id is never ambiguous. The call's RETURN/ERROR is the stream's terminal frame: exactly one arrives, after which both peers release all state for that stream_id.

How a worker decides that a given CALL is a stream is an SDK-local concern, not a wire concern (the reference Python SDK marks streaming targets with @portals.stream_handler and passes the handler a stream object as its first argument). A worker that receives STREAM_DATA, CREDIT or HALF_CLOSE for a stream_id it does not know must ignore the frame rather than fail the connection, so that frames still in flight when a stream terminates are harmless.

Streams are always opened by the BEAM. Once open, both peers may send STREAM_DATA on it; the worker never initiates a stream of its own.

9.2 Credit

Each direction has its own, completely independent encoded-byte credit window. A peer may send STREAM_DATA only while it holds enough credit, and every frame is charged the complete frame size on the wire: the 4-byte length prefix plus the encoded envelope. Credit is never implicit:

• immediately after the CALL that opens a stream, the BEAM sends CREDIT granting the worker its whole inbound window;
• when a worker accepts a stream, it sends CREDIT granting the BEAM its whole inbound window;
• thereafter, a peer sends CREDIT as its application consumes frames, replenishing exactly the bytes consumed.

A window never exceeds the connection's max_stream_byte_credit (§6). A peer that grants more than that, or that sends more bytes than it was granted, has committed a protocol violation; the receiver tears down only that stream (sending CANCEL, or a terminal ERROR with kind: "protocol") and leaves the connection and all other streams running.

CREDIT's optional third field, frame_count, is the number of queued frames the grant acknowledges. It is backward compatible: a CREDIT that omits it is treated as acknowledging exactly one frame, which is what a per-frame-consumption sender produces anyway. Grants that carry no frame acknowledgement (such as the initial window grant) send 0.

9.3 Queued-frame limit

Byte credit alone cannot stop a flood of tiny frames, so each direction additionally enforces a hard cap of max_queued_stream_frames (§6) frames that have been sent but not yet acknowledged by a CREDIT frame_count. A sender must block once it reaches the cap even when byte credit remains; a receiver that observes the cap exceeded treats it as a protocol violation scoped to that stream. The two limits are enforced independently and both apply at all times.

9.4 Half-close and cancellation

HALF_CLOSE closes only the sending direction of the peer that sent it. The other direction stays fully usable — a peer may half-close immediately and keep receiving for the rest of the stream's life. HALF_CLOSE is idempotent, and STREAM_DATA received after the peer's own HALF_CLOSE is a protocol violation scoped to that stream.

CANCEL on a stream's ID cancels the stream in both directions at once. It is idempotent, races safely with the terminal RETURN/ERROR, and requires both peers to release all state for that stream_id: queued inbound frames, outstanding credit, and any blocked producer (which is unblocked with a cancelled error rather than left waiting).

9.5 Capacity

A worker advertises its per-worker stream capacity as max_streams in HELLO (§4), independently of the max_concurrency it advertises for unary calls. max_streams of 0 means the worker does not implement streaming; the BEAM then rejects open_stream locally with an overload error and never sends stream frames to it. Both peers refuse to exceed the advertised count, reporting overload rather than queueing without bound.

The two budgets must stay independent in both directions: long-lived streams must not consume the worker's unary execution capacity (the reference Python SDK runs stream handlers on a separate thread pool), and unary saturation must not prevent a stream from being opened or serviced.

10. Conformance

Portals.Conformance.Runner is a black-box runner: given any module implementing the Portals.Codec behaviour (encode/2, decode/2), it replays every entry in protocol/vectors/golden.exs (must round-trip byte-for-byte semantically) and protocol/malformed/fixtures.exs (must be rejected with the documented error class, never accepted, never raised). A non-Elixir SDK satisfies the same conformance requirement by implementing an equivalent runner against the same two fixture files.