DocsOperationsFailure modes & recovery
Failure modes & recovery
Nothing in ngstone assumes the happy path. Every failure below is detected by something specific and handled by something specific — not by a generic timeout and a shrug.
Agent dies mid-request#
Detection
A stream read/write error on the edge (the session RSTs).
Handling
Requests in flight get an immediate
502error page; the session's Done() fires and ReleaseSession compare-and-deletes its hostnames, so subsequent requests get a cheap 404. There is no replay — the edge can't know whether the request was idempotent. A restarted agent gets a new InstanceID, so it reclaims its name via the dead-holder takeover rule rather than the same-identity one.Edge restarts#
Detection
The agent's control-stream reads or keepalive start erroring.
Handling
All state is rebuilt from the client side: the agent backoff-reconnects, re-authenticates, and re-binds with
Takeover: true, restoring the same public URL — typically under two seconds downtime with Restart=always. Certificates load from the state directory cache, provided the systemd unit is configured correctly.Network partition#
Detection
yamux keepalive failure or three missed application pings — about 45 s either way.
Handling
The edge eventually frees the binding, returning 404s. The agent doesn't wait for that: its unchanged
InstanceID matches the zombie binding on reconnect, so takeover rule (a) rebinds immediately even while the old session still looks alive to the edge. The zombie session is sent Evicted and force-closed.Slow or hung local app#
Detection
ResponseHeaderTimeout at 30 s.Handling
The public client gets a
504; the tunnel session itself is unaffected. Slowness after headers have started is legal — that's just streaming — and is bounded only by client patience and flow control, not by a server-side timer.Oversized request bodies#
Detection
A
Content-Length pre-check for declared-length bodies, MaxBytesReader plus a custom ErrorHandler for chunked ones.Handling
A declared-length body over the 256 MiB cap gets a real
413 before anything crosses the tunnel. A chunked body over the cap is synthesized into a 413 from *http.MaxBytesError — with the caveat that the agent may already have forwarded a partial body to the local app by the time that fires.Certificate issuance failure#
Detection
The startup check on
:4443 plus /healthz reporting each certificate's NotAfter separately.Handling
If the apex order fails, public traffic keeps working fine on the wildcard cert — but no agent can connect, since the control plane depends on the apex cert.
ngstoned refuses to accept on :4443 without a loadable apex cert and exits loudly rather than failing mysteriously later.A healthy wildcard can mask a dead control plane
Public HTTPS traffic looking fine is not proof the control plane is healthy. Watch both certificates'
NotAfter values on /healthz, not just whether tunnels resolve.