Architecture

You don’t need any of this to use Longpipe. It’s here because the design is the reason the SDK is fast, and because “how does enabling three effects not cost three models?” is a fair question.

One encoder, many heads

Longpipe is a hydranet: a single EfficientNet-Lite-derived encoder runs once per frame, and several small decoders read from it.

             video frame
                  │
        ┌─────────▼─────────┐
        │   shared encoder  │   ← runs once
        └─────────┬─────────┘
        ┌─────┬───┴───┬──────┐
        ▼     ▼       ▼      ▼
     matte  face    optical  (future
            keypts   flow     heads)
        │     │        │
        ▼     ▼        ▼
   background  touch-up   temporal
    / blur     reframe    stability

The consequence is the one that matters in practice: enabling a second effect doesn’t cost a second inference. Backgrounds use the matte. Auto-reframe uses the face keypoints. Touch-up uses the keypoints plus a separate landmark model on a small face crop. Temporal stability uses the flow head. Every one of those reads from an encoder pass that was already happening.

The exceptions are honest ones. Touch-up runs a second, small network — a 478-point landmark regressor — on a 256×256 face crop, once per face. And audio denoise isn’t part of this graph at all (see below).

Base network + wrapper

Each preset is a composition of two pieces, not one monolithic network:

• The base network runs at a low internal resolution (128×80 up to 320×192, depending on preset). This is where the encoder and the heads live — it’s the expensive part, so it runs small.
• The U-Net wrapper takes the full-resolution frame, strides it down to feed the base, then upsamples the base’s output back up while re-injecting high-resolution detail. This is what recovers hair and edge detail that a low-res matte alone would lose.

So a preset’s “output resolution” and its “base resolution” are genuinely different numbers, and the gap between them is the wrapper’s job. See Presets & autotune for the table.

Three base networks back the five presets — xs/small share one, medium/large share another, xl has its own — differing in wrapper variant, resolution, and frame cadence.

Temporal stability

Per-frame matting flickers. A matte that is consistently slightly wrong looks far better in video than one that is inconsistently right, so temporal stability is treated as a first-class output rather than a post-hoc smoothing filter.

A small optical-flow head rides the encoder’s cached features and predicts backward flow. Each frame, the previous matte is warped along that flow to line up with the current frame, then blended with the fresh prediction through a gate that backs off where the flow is unreliable (fast motion, occlusion seams). Face keypoints get the identical treatment, which is what keeps the touch-up and reframe boxes from jittering.

On presets that skip frames, the same machinery does double duty: a skipped frame doesn’t reuse a stale matte, it warps the last one forward to where the subject is now.

Frame skipping ≠ dropping frames

Smaller presets run the model on every second or third frame. Compositing still runs on every frame — the background, blur, or touch-up is applied to every single frame you get out. Only the model re-run is gated, and the matte in between is flow-warped rather than frozen. The output is always at full source framerate.

Fully-GPU, zero-copy

Every model in the video path is hand-written WGSL (WebGPU) and GLSL (WebGL2) shaders — no general-purpose inference runtime, no ONNX graph interpreter. Frames enter as GPU textures and never leave the GPU: no readback to JavaScript, no CPU↔GPU round trip per frame, no intermediate ImageData.

Two backends, one interface. WebGPU is used when available; WebGL2 is a full fallback, not a degraded mode. The model code is identical across both — only the shader implementations differ.

The effects themselves compose as a chain of GPU passes ending in exactly one compositor, which is why touch-up and a virtual background can both be on without either one round-tripping through the other.

Worker by default

All model and render work happens in a dedicated Web Worker. The main thread holds a thin shim that owns capability detection, transport setup, audio, and the public API. Your app’s main thread stays free for your app.

Getting frames in and out of that worker is the messiest part of doing this in a browser, because the available APIs differ per browser. Longpipe probes at runtime and picks the best available path on each side — MediaStreamTrackProcessor/MediaStreamTrackGenerator where supported, a transferred-canvas capture path, or an ImageBitmap shuttle as the universal fallback. Chromium, Firefox, and Safari each end up on a different combination. None of it is configurable and none of it is visible to you: you hand in a MediaStream and get a MediaStream back.

Adapting to the device

Longpipe ships five presets rather than one model, and picks between them at runtime.

At init, an autotune step microbenchmarks candidate presets on the actual device — using correctly-shaped zero-filled weights, since GPU cost is driven by tensor shapes rather than values — so it can choose before downloading any weights. It then fetches only the preset it picked.

At runtime, an adaptive controller watches framerate and model time, and swaps presets when the device’s situation changes (another tab starts hammering the GPU, a laptop drops to battery saver). Both are on by default under preset: 'auto' and both are fully overridable. See Presets & autotune.

Audio is a separate pipeline

Audio denoise shares nothing with the above. It runs as WebAssembly in an AudioWorklet on the audio render thread — no GPU, no worker, no interaction with the video model. It has its own three-model lineup and its own init-time probe to pick between them.

The two pipelines meet only at the end, where the cleaned audio track and the processed video track are assembled into the output MediaStream. That means denoise adds no load to the video pipeline, and a heavy video preset doesn’t put your audio at risk. See Audio denoise.

Where this is going

The hydranet framing is a bet: the encoder is the asset, and new effects should arrive as new heads on it rather than as new models. Auto-reframe is the cleanest example so far — it shipped needing no new weights and no new inference at all, because the keypoints it needs were already being produced for something else.

AR effects and lighting correction are the next heads. See the roadmap.