DocsOperationsReconnection & takeover

Reconnection & takeover

Laptops sleep, wifi drops, and VPS processes restart. ngstone is built so the same public URL comes back after all three, usually in well under the time it takes the dead-peer detector to even notice something is wrong.

Instance identity#

Every agent process mints an InstanceID— a stable 16-byte random hex value — once, at startup. It survives reconnects on that process and dies with it; a restarted agent gets a brand new one. Every binding in the edge's registry records the holder's InstanceID, and that identity is what authorizes taking a binding back, not whether the old holder still looks alive.

The takeover rules#

When a Bindrequest names a subdomain — or, for a TCP tunnel, a port — that's already held, the edge resolves it with three rules, in order:

  1. 1

    Same InstanceID — immediate takeover

    The requester is the same agent process reconnecting past a half-open zombie connection. Takeover succeeds instantly, regardless of whether the zombie session still looks alive to the edge.
  2. 2

    Different InstanceID, dead holder — takeover

    The previous holder's session has already been detected as dead. The new session takes the binding.
  3. 3

    Different InstanceID, live holder — rejected

    The edge returns BindErr{subdomain_taken}, or BindErr{port_unavailable}for a TCP port. Two separate agents shouldn't silently fight over a name or a port — liveness is only consulted in this one case.

On any successful takeover the edge sends the old session Evicted{hostname, reason: "takeover"}, force-closes it, and rebinds atomically under the registry lock. The evicted agent updates its status line instead of continuing to advertise a URL it no longer owns. For a TCP tunnel, hostname carries the tcp://host:portURL, and the takeover hands the port's existing listener to the new session instead of closing and reopening it, so no other process can grab the port in between. As for subdomains, the holder and the requester must belong to the same credential owner before any of these rules apply.

A named tunnel can opt into a connector pool with --multi-connector. In that mode, agents with different InstanceID values join the same explicit subdomain when their HTTP/2, access-control, and traffic-policy settings match. A reconnect replaces only its own connector; releasing one connector stops new selection for it and lets its in-flight requests drain while the remaining connectors continue serving traffic.

Network transports#

ngstone defaults to --transport auto. It tries the raw TLS yamux listener on :4443 first, then falls back to an authenticated WebSocket carrying the same yamux session through /_ngstone/control on HTTPS :443. Use --transport raw or --transport websocket to force a path.

--transport quic is opt-in: start the edge with -quic and open 4443/udp. The same yamux session runs over one bidirectional QUIC stream, every connection is address-validated (Retry, and NEW_TOKEN tokens also validate the address), closed if unauthenticated 5 s after admission as on TCP, and the agent never falls back to another transport. QUIC sends a keep-alive every 5 s with a 15 s idle timeout, so a peer that vanishes without closing is detected within about 20 s, compared with immediately for a killed process over TCP (the kernel resets the connection) and up to 45 s for a silent TCP path (partition). A graceful shutdown is noticed at once on every transport.

Reconnect backoff#

On any session error the agent enters a reconnect loop using decorrelated-jitter backoff from 500 ms up to a 30 s cap. Every reconnect attempt sends Takeover: true. Combined with rule (a) above, that makes reconnecting after a network partition effectively instant — the agent doesn't sit around waiting out the 45 s dead-peer detection window before it can have its own name back.

TCP tunnels get the same instant reconnect. When you didn't pass --remote-port, the agent remembers the port the edge last gave it and requests that port again on reconnect, so the advertised tcp://host:port stays the same. If that port has since gone to another tunnel, the edge answers port_unavailable. The agent then drops the remembered port and binds a fresh one in the same handshake, so the tunnel still comes up, at a new address. An explicit --remote-port is never moved: the agent keeps retrying for that port.

HTTP and TLS tunnels without --subdomain do the same with their auto-assigned name. If the name is taken (subdomain_taken) or now belongs to a tunnel of another kind (kind_conflict), the agent binds a fresh random name in the same handshake. A preview credential keeps the one name the edge assigned it; after its first refusal the agent stops asking for remembered names and ports. An explicit --subdomain is never changed.

When the agent reconnects to a different edge in a multi-edge deployment, the new edge takes the port's lease and the old edge evicts its local session at its next poll. Clients can see connection refused for up to about two seconds during the move, and up to 30 seconds if the proxy listener on the old edge cannot bind.

Dead-peer detection#

yamux keepaliveintervaldefault: 10s
Transport-layer ping. Session kill switch is ConnectionWriteTimeout: 45s.
Control-stream Pingintervaldefault: 15s
Application-layer liveness check; three consecutive misses (≈45 s) declares the peer dead.

Why the write timeout isn't 10s

ConnectionWriteTimeoutin yamux is a whole-session kill switch guarding every data send, not a keepalive knob. Set to 10 s, a saturated uplink or flaky hotel wifi mid-download would trip it and kill the entire session — every tunnel, every in-flight request — presenting as "the tunnel randomly drops under load". At 45 s it stays a backstop for genuinely wedged connections while liveness comes from the pings instead.

Both layers converge on roughly 45 seconds worst-case detection — but rule (a) above means the binding's own owner never has to wait on that window to reclaim its name.

Zombie bindings#

When a session dies, its bindings are released via a compare-and-delete, not a blind delete keyed on hostname: each hostname is only removed if the registry still shows the dying session as its owner. This is what keeps a dying session's eventual cleanup from deleting a binding that a newer session already took over — the operation is idempotent, since it can be triggered both from the session-death path and from a graceful drain.

TCP ports work the same way. A takeover moves the port out of the old session's ownership, so when the old session is finally released it closes only the connections it was already relaying and leaves the port, and its listener, with the new owner.