Scaling past the browser

Where the ceiling is, and why

DuckDB-wasm cannot spill to disk. That is the binding constraint, and it is a hard one: an aggregation that exceeds available memory does not run slowly, it aborts. In practice that puts the sensible in-browser ceiling at roughly 20–50 million events, depending on how wide the log is and what you are computing.

Below that, the browser is genuinely the better place to work — no infrastructure, no data movement, no transfer of custody. Above it, you want an engine.

What an engine is

A compute engine is a container that speaks the same host.sql() data contract the browser does, without a tab’s memory ceiling, thread ban or storage quota. It runs the same plugin bundles: a wasm plugin executes in a sandbox on the engine rather than in the page, and a Python plugin runs natively rather than under Pyodide.

Three deployments, one image:

• Browser (WASM) — always present, no setup.
• Local Docker — one self-contained container on your own machine. Deliberately not a Compose stack: every extra container is one more thing to understand before the feature works.
• On-prem or cloud — the identical image, running next to the data.

What changes, and what does not

What does not change: the artifact tree, the plugins, the actions, the views, and the provenance graph. An artifact on an engine appears in the same tree with a badge, not in a separate list.

What changes: where the bytes live. The governing principle is data locality follows execution — only metadata, previews and paged query results cross the network. The bulk data stays resident on the engine, which is what makes a terabyte log workable over a normal connection, and is also what keeps regulated data inside its own boundary.

Moving artifacts

Artifacts can be copied or moved between engines, including back to the browser. The usual pattern in practice:

1. Import and profile a sample locally, in the tab. Get the analysis right where iteration is fastest.
2. Move to an engine for the full log, running the identical chain of actions.
3. Bring back the results — models, alignment sets, aggregates — which are small, while the log stays put.

For reproducibility

Every execution records its runtime and version, including which engine ran it. A result computed on a cloud engine is as traceable as one computed in the tab, and the record distinguishes them — which matters when a reviewer asks whether a figure came from the sample or the full log.

Next

Packaging your algorithm.