Running the model without the pipeline

EffectsPipeline is the supported path for almost everyone: it owns the worker, weight fetching, autotune, per-browser frame transport, and adaptive preset swaps. If you hand it a MediaStream and want a MediaStream back, use it and skip this page.

This page is for the cases where the pipeline’s shape is wrong for you:

Everything here is a supported public API, exported from a separate entry point:

import { WebGPUBackend, TierModel, TIER_CONFIG, loadWeightsFromBinary } from 'longpipe/model'

Why a separate entry point? longpipe/model includes both backends, and the WebGL2 backend is ~114KB of GLSL. The main longpipe entry carries its backends inside an inlined worker, so exporting these from 'longpipe' would charge every EffectsPipeline user for code they never touch. Importing from 'longpipe' costs exactly what it did before this entry point existed.

The three pieces

The model layer is deliberately small. There are only three things to understand.

A backend wraps WebGPU or WebGL2 and owns a canvas, a dtype, tensor allocation, and the op library. WebGPUBackend and WebGLBackend implement the identical Backend interface — model code never knows which one it’s running on.

TierModel is the matting network: a base network plus a U-Net wrapper. It’s constructed with an input tensor and weights, wires its whole compute graph at construction, and pre-allocates every intermediate buffer. run() takes no arguments and allocates nothing.

TIER_CONFIG says which base network and wrapper each tier uses, and at what resolutions. The .bin file carries only weights — the architecture is code. This is why loading a tier takes a config lookup rather than a model-format parser.

Minimal example: frame in, alpha matte out

import { WebGPUBackend, TierModel, TIER_CONFIG, loadWeightsFromBinary } from 'longpipe/model'

const TIER = 'large'
const cfg  = TIER_CONFIG[TIER]              // { base, wrapper, canvasRes, baseRes }

// 1. Backend. A canvas is required even if you never present to it —
//    both backends are built around one. Any offscreen canvas will do.
const backend = await WebGPUBackend.create({
  canvas: new OffscreenCanvas(cfg.canvasRes.w, cfg.canvasRes.h),
  dtype:  'f32',                            // or 'f16' — see Precision below
})

// 2. Weights. One fetch, parsed into a nested object of typed-array views.
const buf = await fetch(`https://cdn.longpipe.dev/models/v/0.0.5/model_${TIER}.bin`)
  .then(r => r.arrayBuffer())
const w = loadWeightsFromBinary(buf)        // { base, wrapper, flow?, face? }

// 3. Input op — ingests an ImageBitmap or VideoFrame and bilinearly
//    resamples it to the tier's canvas resolution.
const input = backend.ops.Input(cfg.canvasRes.h, cfg.canvasRes.w)

// 4. The model. Graph is wired here; nothing is allocated during run().
const model = new TierModel(
  backend, input.output, w.base, w.wrapper, cfg.wrapper, cfg.base,
)

// 5. Per frame:
input.setSource(frame)                      // ImageBitmap | VideoFrame
input.run()
model.run()

// 6. The matte. model.output is a 4-channel tensor; alpha is channel 0.
const data = await backend.readback(model.output)
const alpha = data.filter((_, i) => i % 4 === 0)   // one float per pixel, 0–1

That’s the whole thing. No worker, no MediaStream, no transport negotiation.

Choosing a backend

import { WebGPUBackend, WebGLBackend } from 'longpipe/model'

const backend = await WebGPUBackend.isAvailable()
  ? await WebGPUBackend.create({ canvas, dtype })
  : await WebGLBackend.create({ canvas, dtype })

isAvailable() does a real adapter request, not just a navigator.gpu check — the two disagree often enough to matter.

Compositing too: RenderOp

If you want the finished picture rather than the matte, RenderOp orchestrates network → alpha upscale → composite, and presents to the backend’s canvas.

import { RenderOp } from 'longpipe/model'

const renderOp = new RenderOp(backend)
renderOp.attachNetwork(model, input, {
  upscaler:   'bilinear',                   // or 'bicubic'
  background: { mode: 'blur', sigma: 8 },   // or { mode: 'solid', color: [r,g,b] }
                                            // or { mode: 'image', image: tensor }
})

// Per frame — RenderOp fans the source out to both the display and network paths:
renderOp.setSource(frame)
renderOp.run()
// backend.canvas now holds the composited frame

Note setSource goes on the RenderOp, not the input op, once a network is attached — it drives the display path (at canvas resolution) and the network path (at the tier’s resolution) from the same frame.

Colors in { mode: 'solid' } are [r, g, b] in the 0–1 range, not 0–255.

Splitting model and display work

RenderOp.run() is runModel() + runDisplay(). Calling them separately is how frame skipping works: run the model every Nth frame, composite every frame against whatever alpha is current.

renderOp.setSource(frame)
if (frameIndex % 2 === 0) renderOp.runModel()   // network + alpha upscale
renderOp.runDisplay()                            // display refresh + composite

The alpha tensor persists between model runs, so the skipped frames still composite correctly — they just use a matte that’s one frame old. This is exactly what the pipeline does for the xs, small and medium presets.

Resolutions

Two numbers per tier, and they are not the same thing:

TIER_CONFIG.large        // { canvasRes: { w: 640, h: 400 }, baseRes: { w: 256, h: 160 }, ... }

The Input op stretches to the target resolution — it does not preserve aspect ratio. If your source aspect doesn’t match the tier’s, crop before ingesting or the matte will be distorted:

// cover-style centre crop to the tier's aspect, at native resolution
const aspect = cfg.canvasRes.w / cfg.canvasRes.h
const srcAspect = img.width / img.height
const [sw, sh] = srcAspect > aspect
  ? [Math.round(img.height * aspect), img.height]
  : [img.width, Math.round(img.width / aspect)]
const cropped = await createImageBitmap(
  img, Math.round((img.width - sw) / 2), Math.round((img.height - sh) / 2), sw, sh,
)

Precision

dtype: 'f16' roughly halves memory traffic and is meaningfully faster on most hardware. On WebGPU it requires the adapter’s shader-f16 feature, and create() throws if you ask for it without support — check first:

const dtype = await WebGPUBackend.hasF16Support() ? 'f16' : 'f32'

Weights come in both forms: model_large.bin and model_large.f16.bin. They’re interchangeable — a backend converts at upload time if the file’s precision doesn’t match its own — so fetching the matching file is a download-size optimization, not a correctness requirement.

What you give up

The pipeline layer is not just a convenience wrapper. Driving the model directly means these are now yours to handle:

If you need any of those, you probably want the pipeline.

Performance notes

Build the graph once. Constructing a TierModel compiles shaders and allocates every buffer. Do it at init and reuse it — never per frame. run() is allocation-free by design.

readback() is a synchronization point. It’s a genuine async buffer map on WebGPU, but on WebGL2 it’s a blocking gl.readPixels. Reading the matte back every frame will stall a WebGL2 render loop. If you’re compositing on the GPU anyway, use RenderOp and never read back at all.

Use sync() for timing. run() only enqueues GPU commands, so performance.now() around it measures CPU dispatch, not GPU work. await backend.sync() is a cheap barrier that doesn’t transfer data.

const t0 = performance.now()
model.run()
await backend.sync()
const ms = performance.now() - t0

Clean up with backend.destroy() when you’re done — it releases GPU resources.

A working reference

sdk/demo/main.ts in the repo is a complete single-frame version of everything above — backend selection, weight fetch with .f16.bin fallback, aspect cropping, TierModel construction, RenderOp compositing, and timing. It’s the file this page was written from.