Zipp benchmarks

Headline

Zipp/Node, all 30 rows, geometric mean

0.629×

normal 13 0.543×, hostile 17 0.704×

Zipp/Bun, all 30 rows

0.483×

normal 0.459×, hostile 0.502×

Zipp/Deno, all 30 rows

0.422×

normal 0.467×, hostile 0.390×

Rows where Zipp's median beats Node's

25 / 30

5 rows are slower; they are listed below

Capture: 29 October 2026, the 0.0.31 engine as a clean profile-guided-optimization build recorded as zipp 0.0.31 from source fcfb1934 (the release tag). Native Windows x86-64, four engines, 15 repetitions per row, exact output on every observation, and both artifacts record publishable:true. Every engine runs with its production JIT; Interpreter versus interpreter compares Zipp's interpreter with QuickJS and the JIT-off modes, and Python versus CPython its Python frontend. The all-30 figures weight the 30 rows equally, exp((13 ln G13 + 17 ln G17) / 30) of the two suites' geometric means, whose intervals are given with each suite. Zipp runs in its default mode, which in 0.0.31 sets up Node's globals for the program; on the normal rows that measured within 1% of the plain ECMAScript mode (ZIPP_NO_NODE=1).

Methodology

• Cold process time. Every observation is a fresh process timed with a wall clock around spawn and exit. A paired empty launch of the same engine runs immediately before each full launch; the headline uses the cold time, and the launch-adjusted time is reported as a diagnostic.
• No warm-up and no --jitless games. Every engine runs with its production JIT and an empty environment; any inherited runtime control such as NODE_OPTIONS or ZIPP_NOJIT marks a run diagnostic-only.
• Counterbalanced schedule. A seeded engine order rotates once per repetition so each engine occupies every position; the per-repetition order is recorded in the artifact.
• Intervals. 95% percentile bootstrap with 10,000 replicates, described in the artifact as descriptive, not a hypothesis test. Suite geometric means use a cluster bootstrap by repetition. "Win" below means a Bonferroni-adjusted exact one-sided paired sign test at 0.05 across all comparisons (39 for the normal suite, 51 for hostile).
• Correctness first. Standard output is compared byte for byte against the reference on every repetition. A single mismatch disqualifies the capture.
• Publication gate. Full Node, Bun, Deno, Zipp order with Node as baseline; the complete frozen suite; at least 15 repetitions and 10,000 bootstrap samples; a clean, HEAD-matching, provenance-stamped PGO binary.

Startup

Median empty-process launch, the paired baseline for every row:

Normal suite: 13 rows

Ten headline workloads plus three architecture diagnostics that are deliberately kept out of the headline geometric mean. Cold medians in milliseconds; the interval is the 95% bootstrap on the Zipp/Node ratio; an asterisk marks a Bonferroni-adjusted sign-test win over Node.

Note what parse-large-js measures: a hand-written userland tokenizer over two megabytes of synthetic source. It does not benchmark Zipp's own parser. The diagnostic rows isolate the inline-cache cliff at nine receiver shapes and the sparse-array gap-size axes; they are useful for engineering and would flatter the headline, which is why they are excluded from it.

Hostile suite: 17 rows

The hostile corpus asks how gracefully Zipp performs when JavaScript stops looking like an ideal reducer: six baseline-and-stressor pairs plus five standalone applications, including a React-style reconciler, a warm HTTP router, a bytecode interpreter and the unmodified [email protected] package.

Ordinary geometric mean 0.704× Node [0.700, 0.709], 0.502× Bun, 0.390× Deno; category-balanced 0.738× Node. The suite's aspirational gate (category-balanced at most 1.05×, no category above 1.15×, no row above 1.50×) passes its category-balanced bound, fails the per-category bound (applications 1.217×), and passes the row bound (the worst row, allocation-survival, is 1.344×).

Where Zipp loses

Five of thirty rows have a Node point gap, all of them in the hostile suite: allocation-survival 1.344×, reactish-reconcile 1.217×, warm-router 1.130×, calls-closures 1.122×, shapes-megamorphic 1.038×. The pattern: long-lived allocation, closure-heavy application code and property access across many shapes are where V8's optimizing tiers still win. Against the previous capture (21 October 2026, zipp 0.0.28), all 30 rows went from 0.652× to 0.629× Node; the rows that improved most were async-lived (0.829× to 0.667×) and types-churn (0.693× to 0.605×) and bytecode-vm (1.050× to 0.924×) and async-burst (0.574× to 0.507×), and the one that moved most the other way was property-ic-shapes (0.037× to 0.043×).

Against Bun, Zipp loses json-large (1.045×). Against Deno, Zipp loses no row. On the normal suite Bun is the absolute fastest engine on json-large; Zipp is fastest on every other normal row.

Interpreter versus interpreter

The tables above compare production JITs. This one asks how fast Zipp is as a pure interpreter: the same PGO binary with ZIPP_NOJIT=1 ZIPP_NO_RX_JIT=1, which emits no native code (its JavaScript JIT and its regex JIT both off; the captures before 22 October left the regex JIT on), against Bellard's QuickJS 2026-06-04 and QuickJS-NG 0.17.0 (neither has a JIT), and against Node --jitless (V8's Ignition), Deno --v8-flags=--jitless and Bun with BUN_JSC_useJIT=0 (JavaScriptCore's LLInt). run_native.py staged identical sources for every engine, checked every output against Node's and ran 8 counterbalanced repetitions; cold medians in milliseconds, launch included. It is a diagnostic, not a canonical capture. This interpreter is what the WebAssembly build and the safe-sandbox profile run.

Over all 30 rows Zipp's interpreter, with no native code (ZIPP_NOJIT=1 ZIPP_NO_RX_JIT=1), is faster than Bellard's QuickJS (0.574×, 30 of 30 rows; 0.683× in the previous capture), and faster than QuickJS-NG on all of them (0.326×). QuickJS wins no row. Against the JIT engines with their JITs off it is faster than Node --jitless (0.911×, 21 of 30 rows; 1.105× in the previous capture), faster than Deno --jitless (0.825×, 24 of 30 rows; 0.995× in the previous capture), and faster than Bun without its JIT (0.675×, 28 of 30 rows; 0.804× in the previous capture).

Python versus CPython

Zipp compiles Python 3 to the same register bytecode. tools/python_bench.py runs 36 programs (30 micro kernels, 6 application-shaped) as a fresh process per engine per repetition, interleaved, 15 repetitions, against CPython 3.13.14; every run's output must equal CPython's. The binary is the published 0.0.31 release build (the PGO build has no Python). Each program times its own workload in-process; that work time is the ratio below. Zipp runs with its JIT and again with ZIPP_NOJIT=1, since standard CPython is an interpreter. 8 further programs (the torch ones) have no CPython counterpart and are not included.

With its JIT, Zipp's Python is faster than CPython on 15 of 36 programs; over the micro kernels it takes 0.916× CPython's time, over the application-shaped programs 1.591×, and over all 36 programs 1.004× (1.067× in the previous capture); without the JIT Zipp is 1.422×. Counting the whole process, Zipp's faster start makes it 0.494× CPython's wall time. No intervals are computed for these geometric means; the JIT and no-JIT artifacts hold every observation.

WebAssembly and GPU

None of the numbers above apply to the WebAssembly build, which is an interpreter. The only WebAssembly comparison in the repository, against QuickJS-NG v0.16.2, is marked not publishable: only five hostile rows were comparable, QuickJS-NG led four of them, and Zipp led warm-router at 0.476×. Module size is a settled fact rather than a benchmark: 5,558,860 bytes raw and 1,248,649 Brotli for Zipp's v0.0.12 artifact against 1,528,293 and 417,087 for QuickJS-NG, 3.59× raw and 2.96× on the wire.

Five WebAssembly interpreter cliffs were closed on 5 September 2026 with paired before-and-after measurements on one machine: a 64K non-ASCII charCodeAt loop 4,462 → 3.2 ms, a tokenizer 9,828 → 8.2 ms, a slice loop 449 → 4.3 ms, join with 300K retained strings 8,423 → 138 ms, and a new Function property loop 20.9 → 14.8 ms.

The Python frontend's measurements are in Python versus CPython above. There are no GPU performance numbers at all; the GPU work is validated for correctness only.

The 11 September capture, and why it is slower

An earlier publishable capture exists at commit 14770703 (zipp 0.0.15, 11 September 2026): headline ten 1.003× Node [0.995, 1.009], all 13 rows 0.677×, hostile ordinary 0.875×, with bytecode-vm at 3.59× Node. A paired A/B against the audited baseline e6e0f65d puts that baseline within 1% of the new build on every row, so the regression opened between 8229b3fc and e6e0f65d. The suspect is the v0.0.15 change that made specification-order method calls the default: the fused CallMethod lowering was observably wrong for a getter or a proxy trap and is now used only where argument shapes provably cannot observe the order. No headline is claimed for 14770703; the trade was correctness for speed, and the ledger says so.

1. 28 July 2026
   1.90× Node, headline ten

   First attributable default capture; interpreter-only smoke at 6.42×.

2. 3 August 2026
   PGO adopted at −13.3%

   1.749× to 1.451× in one build change; two more waves reach 1.330× the same day.

3. 19 August 2026
   Nursery on by default: 1.212×
4. 20 August 2026
   First capture under 1.20×

   typedarray-math reaches Node parity.

5. 24 August 2026
   0.9695×: Node parity crossed

   Wave 19 decomposed the three rows that were 60% of the remaining gap.

6. 27 August 2026
   Hostile suite under parity at 0.961×
7. 29 August 2026
   A 2.16× regression, reversed

   A de-fused call lowering; root-caused and reverted the same day.

8. 2 September 2026
   0.878× headline, 0.728× all-30: the public capture
9. 11 September 2026
   1.003× headline after strict call order

   Correctness kept; speed to be recovered.

10. 15 October 2026
    0.795× headline, 0.703× all-30: the previous capture

    All 13 normal rows faster than Node; seven hostile gaps remain.

11. 18 October 2026
    0.769× headline, 0.634× all-30: the 0.0.26 capture

    All 13 normal rows faster than Node; 6 hostile gaps remain, and the bytecode VM is no longer one.

12. 21 October 2026
    0.766× headline, 0.652× all-30: the 0.0.28 capture

    All 13 normal rows and 10 of 17 hostile rows faster than Node; the interpreter at 0.685× Bellard's QuickJS (0.683× rerun with its regex JIT off too).

13. 29 October 2026
    0.744× headline, 0.629× all-30: the 0.0.31 capture

    All 13 normal rows and 12 of 17 hostile rows faster than Node; the interpreter at 0.574× Bellard's QuickJS.

A release measurement for v0.0.31

The published binaries are built by the release recipe: profile-guided since 0.0.31, but with the release's features (Python, GPU and Wasmtime) and their own training run, so each release is also measured as downloaded. These are descriptive release checks, separate from the canonical capture above. For v0.0.31 (the published zipp-0.0.31-windows-x86_64-portable.exe, source fcfb1934, 29 October 2026; Node 24.19.0, Bun 1.3.14 and Deno 2.6.10, 15 counterbalanced repetitions, paired medians with bootstrap intervals, exact output on every row):

The geometric mean is 0.757× Node over the ten headline rows [0.753, 0.759] and 0.549× over all 13 [0.546, 0.553]; Zipp's median is below Node's on every row. A paired A/B of the same 13 rows puts the profile-guided build at 0.996× the release build's time [0.991, 1.003] (the two differ in features and build flags as well as in their profiles): the download is within 1% of the canonical figures. The release and A/B artifacts hold the observations.

Reproduce it

& 'C:\Program Files\Git\bin\bash.exe' tools/pgo.sh      # profile-guided build of target/release/zipp.exe
New-Item -ItemType Directory -Force target\bench-results | Out-Null
python tools\bench.py --zipp target\release\zipp.exe --engines node,bun,deno,zipp `
  --reps 15 --bootstrap-samples 10000 --json target\bench-results\real13.json
python tools\bench_hostile.py --zipp target\release\zipp.exe `
  --reps 15 --bootstrap-samples 10000 --json target\bench-results\hostile17.json
python -m unittest tools.test_bench

• bench/real13_fcfb1934_pgo_2026-10-29.json, the normal-suite capture.
• bench/hostile/head_clean_fcfb1934_pgo_2026-10-29.json, the hostile capture.
• bench/comparison/interp_real13_fcfb1934_2026-10-29.json and its hostile pair, interpreter against interpreter (Zipp with no native code, ZIPP_NOJIT=1 ZIPP_NO_RX_JIT=1, against QuickJS, QuickJS-NG and the JIT-off modes of Node, Deno and Bun).
• bench/python/cpython_fcfb1934_2026-10-29.json and the no-JIT run, Python against CPython 3.13.14 (the second run sets ZIPP_NOJIT=1 for the whole harness).
• bench/comparison/release_real13_fcfb1934_2026-10-29.json and the PGO A/B, the release build as downloaded.
• bench/real13_84012a2d_pgo_2026-10-21.json and its hostile pair, the previous canonical capture (21 October 2026, zipp 0.0.28), with its interpreter pair and Python run.
• bench/real13_f1739644_pgo_2026-10-18.json and its hostile pair, the capture before it (18 October 2026, zipp 0.0.26), with its interpreter pair and Python run.
• bench/real13_fa8e3b86_pgo_2026-10-15.json and its hostile pair, the one before that (15 October 2026, zipp 0.0.24).
• bench/real13_8229b3fc_pgo_2026-09-02.json and the hostile pair, the previous canonical capture (since 17 October 2026 a row of bench/history; these links are to the last commit that held the files).
• bench/real13_14770703_pgo_2026-09-11.json and the hostile pair, the later capture.
• bench/README.md, the suite map and publication gate; bench/hostile/README.md and manifest.json.
• docs/PERF_ROADMAP.md, the checkable roadmap, and the archived performance ledger, 881 KB of experiments including the refuted ones.