Benchmarks

Every number here comes from a file written by ./bench/run.sh (./bench/bucket.sh and ./bench/cluster.sh for the comparisons on a bucket, ./bench/soak.sh for a real bucket), named in its section; sections that name no file use bench/results/2026-11-04-db85998.csv (commit db85998), except the failover row, which needs an S3 API and comes from 2026-11-03-7060271.csv. The short hash in a measurement file’s name (and in the text below) identifies the build that was measured; it is a label, not a commit in this repository. Anyone can rerun it and get their own file; numbers that do not come from such a file do not belong in the docs.

Hardware: a shared cloud container (KVM, 4 vCPU Xeon @ 2.10 GHz, 16 GB), Rust 1.99.1, Wasmtime 50.0.1, Node 22, workerd 2026-10-16. A shared VM is noisy (we saw ±30% between runs on some rows), so read these as orders of magnitude, not decimals. Compared with the previous file (2bfc65f), workerd’s CPU per request moved 2%, so the machine was comparable; WasmKeep’s Rust counter went from 295 to 203 µs of CPU per request and from 9,897 to 15,356 req/s, after the host stopped hopping to the blocking pool for its own bookkeeping. The wake row came out slower (1.07 to 1.39 ms), but three interleaved runs of both commits on the same machine overlapped (1.47 to 1.98 ms after, 1.73 to 1.87 ms before): that row is noise, not a change. Publishable comparisons belong on dedicated hardware: see the benchmark showdown.

WasmKeep against workerd, the same counter

The same actor in three runtimes: one counter object per name, its value in SQLite, GET reads and POST increments.

• WasmKeep, Rust: examples/counter, compiled ahead of time.
• WasmKeep, TypeScript: examples/ts-counter on QuickJS, the default engine, embedded in the component.
• workerd: Cloudflare’s open-source Workers runtime with a SQLite-backed Durable Object, bench/workerd/counter.js.

Each runtime runs as its own server process, and the same client (bench/compare) drives it over loopback HTTP/1.1 keep-alive. Server memory and CPU time come from /proc/<pid>.

From bench/results/2026-11-06-351ea57.csv (commit 351ea57, a clean tree, the first run with transparent huge pages off for the server, see below; the previous run, dec1a82, was within a few percent on every row but memory). Hardware: an AMD Ryzen 9 7950X workstation (16 cores, 32 threads), Rust 1.99.1, Wasmtime 50.0.1, Node 26, workerd 1.20261016.1. Nothing else was running. Both runtimes kept their actor data on tmpfs (/tmp), so no disk is involved on either side.

* workerd runs one thread per process by design: Cloudflare scales it with many processes. Its throughput row measures one process, which is not a fair comparison. CPU time per request is the fair efficiency comparison, and the best proxy for energy we can measure here.

The same comparison on a shared 4-vCPU cloud VM (2026-11-05-ae5416c.csv) gave a different picture:

How to read it

• The machine changes the answer. On the workstation, WasmKeep with Rust uses about 81% of workerd’s CPU per request; on the 4-vCPU VM it used 39%. workerd gains much more from fast cores (555 to 128 µs) than WasmKeep does (215 to 104 µs). A claim like “less than half the CPU” holds on one machine and not on the other, so quote both or neither.
• WasmKeep with Rust is still the fastest per request here. Reads and writes at p50 are about 2.6× faster than workerd’s, and the p99 gap is 2.4×.
• Memory per resident actor is about workerd’s: 250 KB against 277 KB, as on the VM (251 KB). Earlier workstation runs read 571 to 577 KB (files cd48f41 to dec1a82), for every native language alike: this machine’s kernel has transparent huge pages set to always, so each actor’s memory was backed by 2 MiB pages it mostly did not use. wasmkeep serve now turns them off for its own process (--transparent-huge-pages system keeps the kernel’s setting); CPU per request and latency did not move (three runs each way, within 2%).
• TypeScript on WasmKeep costs more than Rust. On QuickJS, the default engine, it uses about 2.2× the CPU of Rust and 3× the memory per resident actor (1,000 resident actors: 936 MB), and about 1.8× workerd’s CPU. Every actor carries its own JavaScript engine, without V8’s JIT: for JavaScript-heavy workloads, workerd is the better home (QuickJS takes 5.8 ms for the js_work load below). TypeScript on WasmKeep is for teams that want WasmKeep’s model in their language. It is not a speed play. StarlingMonkey costs more, see below.
• Writes are not exactly equivalent. A Durable Object’s storage may wait for the disk before answering a write, while WasmKeep’s plain exec commits without an fsync. On tmpfs neither waits for a disk, so the write row is closer to fair here than on a real disk. A durable write on WasmKeep (exec-durable, replicated) costs under 2 ms with an in-memory bucket and its data on tmpfs, see below.
• Not measured yet: idle memory after hibernation (WasmKeep’s idle actors hold none), wake latency per runtime, and workerd with several processes. These are next for the comparison harness.

WasmKeep against celld, every write in a bucket

celld runs Durable Objects on a bucket: each object’s SQLite changes go to S3, much like WasmKeep’s replication. Here both runtimes serve the same counter on the same local bucket:

• WasmKeep, durable: examples/counter built with its durable feature, so POST commits with exec-durable and answers once the write is in the bucket. WasmKeep runs as a one-node cluster, since a node with a bucket is a cluster node.
• WasmKeep, plain exec: the default build of the same counter. Its POST answers before the write reaches the bucket, so it is not equivalent to the other two columns and is shown apart.
• celld: the workerd counter (bench/workerd/counter.js, unchanged), deployed with celld deploy to one celld node. One node waits for the bucket on every write (CELLD_DURABILITY=bucket, set explicitly).

Run by bench/bucket.sh (compare --bucket, compare-bucket/ rows) at commit 488369d, a clean tree, five times with the runtimes’ data on tmpfs and five times on disk (btrfs on an encrypted NVMe SSD), each run on an emptied bucket. Each cell is the median of five runs, with the lowest and highest in brackets; the files are …-data-tmpfs-r1.csv to -r5.csv and …-data-disk-r1.csv to -r5.csv. Hardware: the AMD Ryzen 9 7950X workstation above, Rust 1.99.1, Wasmtime 50.0.1, celld v0.6.3 (the release binary, its build provenance checked by bench/celld/fetch.sh). The bucket is MinIO on loopback with its data on tmpfs, so a write to it costs a local round trip and no disk. Transparent huge pages are off for both servers, as wasmkeep serve sets for itself and the harness sets for celld: a run that left celld on the kernel’s always (e3856a5) read 609 MB at 1000 actors instead of 395 MB.

* Includes the 15 s a WasmKeep cluster node’s first start waits before it listens, checking that no other process runs with its node id (cluster); a restart after a fence or a clean stop waits less. celld does not wait.

How to read it

• Durable against durable, WasmKeep answers a write sooner at p50, on tmpfs and on disk. 0.53 against 0.93 ms on tmpfs and 2.31 against 2.55 ms on disk. At p99 the two are close on tmpfs (1.15 against 1.25 ms) and celld is ahead on disk (3.64 against 6.02 ms). Without a bucket the same durable write takes 102 µs (compare/wasmkeep-rust-durable in 2026-11-10-f163ba0.csv).
• On disk a write takes 1.6 to 1.8 ms longer in both, and throughput is close, celld about 5% ahead. Eight clients completed 1,980 writes per second on WasmKeep against 2,087 on celld at the median; the runs of each overlap (1,974 to 2,032 against 2,003 to 2,198). On tmpfs WasmKeep completed 1.6× celld’s (10,823 against 6,650); celld runs as one multi-threaded process, and this row measures that process.
• A real bucket changes the scale again. MinIO on tmpfs and loopback answers in well under a millisecond. A remote S3 bucket takes tens of milliseconds per write, which would dominate both durable write rows; the read, memory and CPU rows do not touch the bucket.
• WasmKeep uses about 85% of celld’s CPU per request (83% on tmpfs, 85% on disk), 93 to 95% of its memory per resident actor and 84 to 86% of its process memory at 1000 actors, and reads about twice as fast at p50.
• Plain exec on a bucket is barely faster per actor than durable on tmpfs, and much faster on disk. Its response does not wait for the bucket, but an actor uploads an event’s changes before it runs its next event, so the next request to the same actor waits instead. Across many actors the uploads overlap: its throughput is 2.4× the durable build’s on tmpfs and 4.8× on disk.
• One node only. With two or more nodes, celld by default (CELLD_DURABILITY=fleet) acknowledges a write once a follower node has it on disk, or once the bucket has it, whichever comes first. WasmKeep has no such mode: a durable write always waits for the bucket. Three nodes are measured below.
• Start time is a design difference. celld answers 77 to 97 ms after it starts, with its wait for the rest of a fleet turned off for this one-node run (CELLD_READY_FLEET_GATE_MS=0); a WasmKeep cluster node’s first start waits 15 s, by design. A single node without a bucket does not wait (51 ms above).

Three nodes: WasmKeep against celld

Three nodes of each runtime sharing one bucket, run by bench/cluster.sh (compare --cluster): the same counter for the steady-state rows, then each node in turn killed under writes. Rows are cluster3/..., cluster3-wipe/... for the takeovers repeated with the killed node’s directory removed, and cluster3-control/... for a run that must lose writes. Durable here means what each runtime means by it:

• WasmKeep, durable (exec-durable): a write answers once the bucket holds it, on any number of nodes.
• celld, fleet (CELLD_DURABILITY=fleet, celld’s default with two or more nodes): a write answers once a follower node has it on disk, or once the bucket has it, whichever comes first. Up to two followers per object; the bucket upload follows.
• celld, bucket (CELLD_DURABILITY=bucket): a write answers once the bucket holds it, as on WasmKeep.
• WasmKeep, plain exec: answers before its upload, not equivalent, shown apart.

Both runtimes keep their default lease timings (a dead node’s actors move after 15 s on WasmKeep, 10 s on celld) and talk plain HTTP between nodes, since celld has no TLS between peers. Every request goes to a node picked at random, so two in three are forwarded to the actor’s owner. WasmKeep nodes run with --max-instances 2000, --max-connections 512 and --idle-timeout-secs 3600, so no actor hibernates during a run; celld nodes run with transparent huge pages off, as WasmKeep turns them off for itself.

The check for lost writes. After the steady-state rows, each node in turn is killed with SIGKILL while 90 actors are written every 20 ms, one client per actor retrying every 100 ms through a random node. These actors are ledgers, not counters: every attempt writes a token of its own, never sent twice. After each restart, and once more at the end, every ledger’s tokens are read back: an acknowledged token that is missing is a lost write, and a token held twice was applied twice. An actor whose longest wait between two acknowledgements after the kill is over a second was on the dead node, and the end of that wait is when it was writable again. Three more runs go with it:

• The killed node’s disk removed (cluster3-wipe/): the same kills, with the node’s directory deleted before it restarts, so nothing that node alone held survives.
• A control that must lose writes (cluster3-control/): plain exec on nodes that reach the bucket through a relay; half a second before the kill the relay cuts the node off, so each of its actors acknowledges a write the bucket never gets, then the node is killed and its directory removed. If this run loses nothing, the script fails and writes no file.
• Five runs of everything, each on an emptied bucket, once with every node’s data on disk (btrfs on an encrypted NVMe SSD) and once on tmpfs, where an fsync costs nothing.

Hardware: the workstation above, the three nodes of a target as processes on it, one target at a time; the bucket is the same local MinIO, its data on tmpfs. Commit c556110, a clean tree. Each cell is the median of five runs, with the lowest and highest in brackets; the files are …-nodes-disk-r1.csv to -r5.csv and …-nodes-tmpfs-r1.csv to -r5.csv.

Lost writes

The same in both sets of five runs, on disk and on tmpfs:

Timings, every node’s data on disk

* Includes the 15 s node-id check of a first start (cluster). A node restarted after a kill waits less: the surviving nodes have fenced the old process by then.

Timings, every node’s data on tmpfs

The same runs with only the nodes’ data directories moved to tmpfs. An fsync costs nothing there, so this is not a durability setup: it shows what is left of each write once the disk is taken out.

How to read it

• No acknowledged durable write was lost, in ten runs of each target. Sixty kills each for WasmKeep durable, celld fleet and celld bucket (three with the node’s files kept and three with them removed, in each run), every acknowledged token found afterwards. The control lost writes in every run, so the check does find a loss; and plain exec, which may lose writes with its node, lost 0 to 3 of about 86,000 in a run without any help from the relay.
• What the check still does not test. Removing the killed node’s files tests what that node alone held; the followers on the surviving nodes keep their page cache, so an acknowledgement that relies on a follower’s fsync (celld fleet) is not tested against a power loss. All three nodes are processes on one machine. The soak checks WasmKeep’s durable writes on separate machines, over hours.
• Some unacknowledged writes were applied, which is allowed. A write whose client got no answer (the request failed or timed out during a takeover) may still have been applied: 8 to 25 a run on WasmKeep durable, 13 to 26 on celld bucket and 34 to 100 on celld fleet, with data on disk. The client retries with a new token, so none is counted twice.
• Waiting for the bucket in both, on disk, the two are close and celld is ahead on throughput. WasmKeep answers a write sooner at p50 (2.30 against 2.68 ms) and later at p99 (5.89 against 4.17 ms); eight clients, each waiting for its answer, completed 1,710 writes per second against 1,857, about 8% apart where runs of one target differ by 2 to 4%, at the same CPU per write (952 against 958 µs).
• Where the nodes keep their data changes the order. With data on tmpfs WasmKeep completed 9,137 durable writes per second against 5,888 for celld in bucket mode, at 509 against 627 µs of CPU. A durable write takes 1.75 ms longer than a plain one on disk (2.30 against 0.55 ms) and 0.10 ms longer on tmpfs (0.59 against 0.49 ms). These runs change only where the data is; they do not say which step of a write the disk slows.
• celld’s fleet mode is its slowest on a shared disk, and that is not explained. 5.24 ms at p50 and 373 writes per second, against 2.68 ms and 1,857 in bucket mode, the same in five runs. An acknowledgement that takes whichever of the followers and the bucket comes first should not be slower than waiting for the bucket. Here the owner and its followers share one device; whether that is the cause is not measured, and nodes with a disk each would tell. On tmpfs fleet mode answers a write sooner than bucket mode (0.68 against 1.04 ms) but completes fewer (4,238 against 5,888 a second) at twice the CPU.
• WasmKeep reads faster and holds less memory. Reads take 92 against about 220 µs at p50 (both forwarded two times in three), and three nodes hold 1000 actors in 459 against about 520 MB.
• Takeover is close to each one’s lease timing. A dead node’s actors are writable again after about 15 s on WasmKeep and 9 to 10 s on celld, close to each one’s default (15 s and 10 s). Neither side was tuned. A restarted celld node is ready in about 0.15 s; a WasmKeep node in 1.6 s.
• Not measured here: nodes on separate machines, each with its own disk (here the network between nodes is loopback and the disk is shared), a cloud bucket, and mutual TLS between WasmKeep nodes (measured above). An earlier single run with a weaker check for lost writes is in 2026-11-10-d2dea43-cluster-minio-tmpfs.csv.

The same counter in every language

The same actor contract (read, increment, persist) in every guest language WasmKeep supports, each runtime its own server process driven by the same client as above. From bench/results/2026-11-06-351ea57.csv (commit 351ea57, the workstation and run of the workerd table above). TypeScript is on QuickJS, the default engine; StarlingMonkey is the --engine starlingmonkey build of the same source.

The Go column is the counter from before the Go SDK (151 KB; 162 KB with it): the same host calls without the wrapper.

The same table on the shared 4-vCPU cloud VM (2026-11-05-ae5416c.csv, without QuickJS) had the same order at 1.5 to 3.4× this machine’s CPU per request: Rust 215 µs, Zig 214, C 222, Go 364, TypeScript on StarlingMonkey 855, Python 859.

How to read it

• Rust, Zig and C cost the same. Their CPU per request is within 6% of each other, which is run-to-run noise here: most of a request’s cost is the host (HTTP, entering and leaving Wasmtime, SQLite), not the guest. Zig’s smaller component changes deploys and the first compile, not requests.
• Zig needs its stack set. Zig’s default wasm stack is 16 MiB, and a component reserves it in every actor’s memory: the Zig counter took 2.66 MB per resident actor in an earlier run (with huge pages on), against 0.57 MB for C. The examples now build with --stack 1048576, as Rust’s 1 MiB (sdks/zig).
• TinyGo costs about 2.4× the CPU of the native three, with long tails. Its median latency is close to theirs, but its p99 is 10× (reads) to 40× (writes) its median: TinyGo’s garbage collector runs inside the request. Memory per actor is 1.6× theirs.
• The embedded runtimes cost 2.2 to 3.5× the CPU. QuickJS (230 µs) and CPython (259 µs) land close together, StarlingMonkey (368 µs) above them. Memory per resident actor is 3× (QuickJS), 14× (StarlingMonkey) and 31× (CPython) Rust’s: each actor carries its own engine or interpreter. 1,000 resident Python actors took 8.4 GB.
• Python’s CPU per request fell the most from the VM (859 to 259 µs, 3.4×, against 2.1× for Rust): CPython’s interpreter loop gains more from this machine’s caches and clock than the host does.
• Component sizes are from each example’s build.sh (languages).

TypeScript on QuickJS against StarlingMonkey

The same TypeScript counter (examples/ts-counter), one source file built twice: with QuickJS, the default engine since ADR 0018 (npm run build), and with StarlingMonkey (npm run build:starlingmonkey, see languages), next to the Rust counter. From the same file and run as the tables above, 2026-11-06-351ea57.csv. The js_work rows time the counter’s /work path (work.ts): plain JavaScript (building 200 records, a JSON round trip, filtering, sorting, string formatting, a regex, a hash; four rounds), one request at a time, so its latency is the engine’s time.

The comparison that decided the default (2026-11-06-dec1a82.csv, with huge pages on: 1.06 MB against 3.7 MB per actor) had the same CPU and js_work rows within 7%.

How to read it

• QuickJS takes 4.6× less memory per resident actor than StarlingMonkey (741 KB against 3.4 MB), 3× Rust’s. A node holding 1,000 TypeScript actors needs 936 MB instead of 4.17 GB.
• It costs less CPU on the counter: 230 µs per request against 368 µs (37% less), 2.2× Rust instead of 3.5×. Each request’s cost here is mostly entering a fresh instance, the host and SQLite, and a smaller engine is cheaper to instantiate.
• StarlingMonkey is 1.5× faster on plain JavaScript work (3.76 ms against 5.76 ms). Neither engine has a JIT; SpiderMonkey’s interpreter is faster than QuickJS’s. An actor that spends milliseconds computing in JavaScript per event is the case for --engine starlingmonkey (or for Rust).
• Components are 8× smaller (1.7 MB), which shortens deploys and the first compile on a node (202 ms against 704 ms from start to first response with an empty cache).
• What QuickJS does not provide (Web APIs such as fetch, URL, crypto, console, timers) is listed in languages.

Between nodes: plain HTTP against mutual TLS

What a hop between two nodes costs, and what mutual TLS between nodes adds to it (ADR 0011). From bench/results/2026-11-05-d8e9ab7.csv (commit d8e9ab7, the cluster/* rows), with an earlier run of the same suite, 2026-11-05-b07a9d7.csv, for the spread, both on different hardware from the sections above: a workstation (AMD Ryzen 9 7950X, 16 cores / 32 threads, 64 GB), Rust 1.99.1, data in a RAM-backed /tmp (tmpfs). Compare these rows with each other, not with the tables above.

The suite (benches/cluster.rs) runs two pairs of nodes in one process over loopback, each pair sharing an in-memory bucket: one pair plain, one with cluster listeners. In the TLS pair every node publishes a plain address nobody listens on, so a hop in plaintext fails the run instead of being measured. One client, one request at a time, kept alive; nodes pool their connections to each other.

The earlier run predates the change that made forwarding drop hop-by-hop headers, which only adds a header filter to these kept-alive rows. Between the runs, rows moved by up to 7 µs either way: read the TLS cost as a range, not a decimal.

How to read it

• A hop costs about 40 µs on loopback in plain HTTP (106.3 against 63.2 µs), about 50 µs over TLS. On a real network the round trip between the nodes comes on top, the same with or without TLS.
• TLS adds about 10 to 20 µs per hop (9 to 12 µs to a forwarded request, 15 to 21 µs to a call, across the two runs): encryption both ways, and the checks ADR 0011 makes on every request (the peer’s pin checked again, and the answer checked before it is delivered). The call row also carries the signature and the body’s hash, plain or not.
• Not measured: a new connection. Each row reuses a pooled connection, so a full TLS 1.3 handshake (with a client certificate, and no session resumption by design) is not in these numbers. Connections are pooled and kept alive, also for clients that send Connection: close (forwarding drops hop-by-hop headers); they are rebuilt when a peer’s pin changes or its heartbeat lapses.
• Only one request at a time: throughput and CPU per request between nodes, and many clients at once, are not measured yet.

Restore against history length

How long a node takes to restore an actor it has never run (as on failover, or when an actor wakes on another node) as the actor’s history in the bucket grows, and what garbage collection changes. From bench/results/2026-11-05-4c8b21a.csv (commit 4c8b21a, the restore/* and list/* rows; the first run, before collection existed, is 2026-11-05-cd48f41.csv), on the same workstation as the section above (AMD Ryzen 9 7950X, 16 cores / 32 threads), Rust 1.99.1, data in a RAM-backed /tmp (tmpfs), and an in-memory bucket. Compare these rows with each other, not with the tables above.

The suite (benches/restore.rs) runs two nodes sharing the bucket over loopback. A writer gives one counter N replicated writes (one segment each, plus one for on-init); a restorer is then asked for fresh actors whose prefix holds a copy of that history. Each timed request takes the lease, lists the prefix, plans, downloads and applies the chain, and answers a counter GET. Three shapes of the same history:

• As written: the owner uploads a snapshot every 64 segments, so a restore replays fewer than 64 segments, but nothing is ever deleted, so the prefix keeps every segment and snapshot.
• Without periodic snapshots (removed from the copy, the first kept): every segment is replayed. This is what an actor whose periodic snapshots kept failing would leave (a failed one is counted and tried again one segment and 30 s later, then less and less often, down to once every 64 segments).
• After garbage collection (ADR 0012): what the server’s rule (collectable) leaves after the last periodic snapshot once the retention window has passed: that snapshot, the segment at its position and the segments after it.

Medians. The baseline with no history, run again after all the other restore rows, was 0.78 ms both times: restored actors piling up on the node do not inflate the later rows. At 10 writes nothing is collected: the only snapshot is the first one, with no segment at its position.

How to read it

• Periodic snapshots bound the replay, collection bounds the listing. As written, a restore replays at most 63 segments whatever the history’s length, but the prefix grows by one object per write and one per 64 writes: at 10,000 writes a restore takes 6.1 ms against 2.6 ms at 1,000 while replaying fewer segments (17 against 41), because it lists the whole prefix twice and listing 10,158 objects takes 1.7 ms in memory (restores now list the prefix once; not remeasured).
• After collection the prefix holds at most 65 objects, whatever the history’s length (a snapshot and up to 64 segments): at 10,000 writes the restore takes 1.2 ms instead of 6.1 ms, and listing takes 1.7 µs instead of 1.7 ms. The collected rows stay within the range of small histories (1.1 to 2.1 ms here, depending on how many segments follow the last snapshot). On S3 or R2 the difference is larger: one listing page instead of eleven.
• A replayed segment costs about 34 µs here (342 ms for 10,001 segments): one GET, decoding and applying it. Each counter write changes one page, so this is the fixed cost per segment; segments with more pages cost more.
• Not measured: a real bucket. The in-memory bucket answers a listing in one call; S3 and R2 page listings by 1,000 keys, so at 10,158 objects each listing is 11 requests (22 per restore in the version measured above, which listed twice), before the first download. At 20 to 50 ms per request that would add about 0.4 to 1.1 s to a restore (an estimate from the request count, not a measurement), and every replayed segment is one more GET: 82 ms each at the median next to OVHcloud Object Storage, so 1 to 6 s for up to 64 segments (measured). LIST and GET counts against a real bucket, peak RSS during restore, larger databases and pages per segment, and many actors at once are not measured either. The collected rows are histories collected by the rule, copied into the bucket; the server’s own passes (--history-retention-secs, an hour by default) are tested in tests/cluster/, not timed here.

Restart and alarm recovery against actor count

How a node’s restart and the recovery of a dead node’s alarms scale with the number of actors. From bench/results/2026-11-05-4c10a33.csv (commit 4c10a33, the restart/* and alarm_recovery/* rows), on a workstation (AMD Ryzen 9 7950X, 16 cores / 32 threads, 64 GB), Rust 1.99.1, data in a RAM-backed /tmp (tmpfs), an in-memory bucket, and other work running on the machine at the same time. Compare these rows with each other, not with the tables above.

Restarting a node

benches/restart.rs: one standalone node holds N test actors, each created by a request that set an alarm an hour out, so each database is as an actor leaves it (36 KB on disk per actor with its WAL files: 3.7 GB at 100,000). The node is stopped as a crash stops it and started again on the same directory.

Medians. The scan column is Storage::pending_alarms alone (the node runs it at start-up on a blocking thread, then arms what it found). The last column makes every actor’s alarm overdue while the node is down, and counts until each has run on-alarm once (it includes the scan).

• A restart serves at once, whatever the population. The listener does not wait for the scan: 4 ms from start to an actor’s answer at 100,000 actors as at 1,000.
• The scan is linear, about 110 µs per database, on one thread. It opens every database (with the same set-up as an actor’s: WAL mode, pragmas, the host’s tables) to read one row. Until it finishes, no alarm from before the restart is armed: at 100,000 actors an alarm due during the first 11.5 s fires up to 11.5 s late. At this rate a million databases would take about two minutes (an extrapolation; not measured, and 37 GB at this size). A slower disk than tmpfs adds its own cost per open.
• Overdue alarms after a restart run at about 3,000 a second (10,000 in 3.4 s, of which 1.1 s is the scan): each one wakes its actor, runs the event, and the actor stays resident until the pool needs room.

Alarm recovery after a node dies

benches/alarm_recovery.rs: three nodes share the bucket. Node a creates N test actors, each with an alarm at the same instant T, and is killed once the bucket holds them and their alarm index entries (no lease released). Nodes b and c never ran those actors: their sweeps find the alarms in the bucket’s index, and each delivery takes the actor over, restores it and runs on-alarm. The time is from T until every alarm has run and the index is empty. Timings are a tenth of the defaults (lease ttl 1 s, grace 0.5 s, heartbeat 0.3 s, sweep every 1 s instead of 10 s): an alarm can be delivered by another node once it is overdue by a lease period plus grace (1.5 s here, 15 s by default).

Medians. The 10,000 row is from bench/results/2026-11-05-9c495b4.csv (commit 9c495b4, same machine, after the sweep fix; its samples ranged from 23.19 to 23.23 s, and 100 and 1,000 measured the same as before); 38.2 s is the 4c10a33 run (37.2 to 38.2 s). The estimate scales the whole time by ten, since the sweep interval and the lease timings set nearly all of it (the work of a delivery is a few milliseconds and does not scale); it was not measured at the defaults.

• Recovery is capped by the sweep, not by the work. Each node delivers at most 256 alarms per sweep (SWEEP_DELIVER in alarm_delivery.rs), split between the live nodes by a hash of the address. 1,000 alarms take two sweeps after the 1.5 s wait, as two nodes at 256 each predict.
• Fixed: past three overdue periods, the survivors delivered the same alarms. An entry overdue three times longer than a lease period plus grace (4.5 s here, 45 s by default) is anyone’s, so that a node nobody else sees can still be covered. The sweep used to walk first come, first served: both survivors’ cursors sat on the same key and delivered the same 256 a sweep, so 10,000 took 38.2 s (the model: 1.5 s wait, three sharded sweeps at 512, then 33 sweeps of 256 for the remaining 8,464: 37.5 s), one node’s rate however many survive. Now each node delivers its own shard first and keeps an eighth of the cap (32) for anyone’s entries (plan_sweep): 224 own a sweep each, 448 for two, plus whichever reserved picks the other node has not reached. The model: 1.5 s, three sweeps at 512, then 8,464 / 448 = 19 sweeps: 23.5 s, against 23.2 s measured. Two nodes at the full 256 each would take about 21 s; the reserve, which keeps anyone’s entries moving when a node’s own entries never leave the index, costs the difference. At the defaults that is about 45 alarms a second for two survivors, and recovery now grows with the node count. sim/tests/madsim_alarm_sweep.rs checks the rate and that every entry is still delivered when views disagree or a live node never delivers its shard.
• No alarm ran twice in these runs; 14 alarm events were refused by admission and retried, over all samples (25 in the 9c495b4 run).

Not measured

A real bucket at this scale (each delivery restores an actor: listings and GETs with real round trips; the sweep lists the alarm index in pages of 1,000; a handful of alarms at the default timings, on R2, is measured), an SSD or slower disk, peak RSS, a cold page cache after a reboot, a node restarting in a cluster (it also waits for a heartbeat and writes each actor’s ownership record on first use), more than 100,000 actors on a node or 10,000 alarms at once, recovery at the default timings, and alarms whose handler fails or re-arms itself.

Workflow replay against recorded steps

Every event that moves a workflow forward replays its recorded steps from the top (ADR 0019), so its cost grows with them, against the event’s 50 ms CPU budget. crates/wasmkeep-server/benches/workflow.rs times one run of an instance with n recorded steps (each returning an integer), parked on a sleep: the runner loads the steps, replays them, finds the park and sets the alarm. It is reached through an actors.call from a driver actor, so n = 0 is the floor (HTTP, the call, the runner’s own reads). From bench/results/2026-11-06-30e68a8.csv (commit 30e68a8, the AMD Ryzen 9 7950X workstation, a Rust guest, one node, tmpfs).

How to read it

• About 1.5 µs a recorded step, steady from 100 to 5,000 steps. A workflow with thousands of steps replays in a few milliseconds of wall clock, well inside the 50 ms CPU budget (which counts the guest’s share of that time only).
• Recording is not measured here: the first run of each size recorded all its steps in one event, a durable write each.
• A Rust guest. TypeScript replays through JSON and an interpreter (QuickJS is 1.5× slower than StarlingMonkey on plain JavaScript); not measured yet.
• Over a workflow’s life replay is O(n²): 5,000 steps that each wake the workflow once replay about 12.5 million positions in all.

Three nodes against a real bucket

crates/wasmkeep-cli/tests/soak.rs, run by bench/soak.sh: three wasmkeep serve nodes with the default lease timings, eight clients making durable writes (exec-durable) to 60 actors through random nodes for 30 minutes, and every 3 minutes one node killed with kill -9, kept down 75 s, then restarted (9 kills). Before each kill, alarms are set on 8 of its actors, due at the moment it dies. Each of its other actors then gets a client of its own, retrying a write every 100 ms until one is acknowledged. Two more clients write with a plain exec (answered before its upload) to 20 actors of their own: the acknowledged increments lost are reported (plain_lost: values handed out again, as later writes re-cover a lost one, plus what the actors no longer hold at the end), not judged, since a plain write may be lost with its node. For 30 s after each kill, four readers read the killed node’s actors through the live nodes every 100 ms (takeover_reads_*: answers, failures, latency).

A run fails if an acknowledged write is lost (at the end, or in a read during a takeover), a value is handed out twice, an alarm never fires, or the 99th percentile of “writable again after the kill” passes 30 s (WASMKEEP_SOAK_FAILOVER_P99_SECS; the worst is reported, not judged; the runs below were judged by their worst). WASMKEEP_SOAK_ACTORS changes the number of actors. Each run keeps samples.csv, one row per write, probe, alarm, kill and restart, which bench/soak_samples.py turns into a line per kill, and every node’s metrics page at each kill.

Three runs of the same commit (0f2986f: per-actor leases, before ADR 0020; the node-lease runs follow below):

• R2 from a desktop: the three nodes as processes on the workstation above, over a residential connection, to an R2 bucket in Eastern North America.
• OVH from a desktop: the same, to OVHcloud Object Storage in Beauharnois (Standard, One Zone).
• OVH from three VPS: one node per OVH VPS-2 (4 vCPU, 8 GB, Debian 13) in Beauharnois, next to the bucket, installed as in Three nodes (systemd, --app, mutual TLS between nodes); the clients run on the first VPS. The deployment to expect.

From …-0f2986f-soak-r2.csv, …-0f2986f-soak-ovh-bhs.csv and …-0f2986f-soak-ovh-bhs-3vps.csv (the VPS run failed the soak’s 30 s check, see below).

How to read it

• Nothing lost or duplicated in any run, over 27 kills under load. Both stores enforce the conditional writes leases and replication rely on. One difference from moto: R2 returns a version id with a write and none with a read, so the two tokens for one object differ. WasmKeep only hands them back to the bucket, never compares them; the test that did is fixed.

• A durable write costs one conditional PUT, so its latency is the bucket’s: 54 ms next to the bucket, 81 ms from the desktop to the same store, 587 ms from the desktop to R2. The long tails (seconds) are writes that reached an actor while it was being taken over.

• Failover is the lease, then a restore. Takeovers start about 12 s after the kill (10 s ttl + 5 s grace, from the last renewal). The new owner then downloads the actor’s history one object at a time: a snapshot and up to 64 segments, 82 ms each at the median from the VPS (237 ms at p90), so most restores take 1 to 6 s.

• One actor in 267 missed the 30 s target on the VPS (32.0 s). Its restore took 19.8 s, while 21 other restores of similar size on the same node at the same time took 2 to 6 s: one GET hung. The bucket client keeps object_store’s defaults (30 s per request, then a retry), and a restore waits for each object in turn, so one slow request held the whole restore. Since then a restore gives up a read after 5 s of silence and sends it again (ADR 0007, restore step 5). With it, the same soak on the VPS passed: writable again 15.5 s after the kill at the median, 24.6 s at worst, over 269 actors; durable write p50 49 ms, worst 8.1 s; nothing lost or duplicated, 72 of 72 alarms (…-c3d50ba-soak-ovh-bhs-3vps.csv). That run logged no read given up three times, and logged nothing for single retries then, so it does not show whether a stall was cut or none happened; every read given up is logged since.

• Twelve hours on the VPS (the same three nodes, a kill every 20 minutes, commit a72fbac: per-actor leases, before ADR 0020): nothing lost or duplicated over 5,024,032 acknowledged writes, 35 kills and 1,129 actors taken over; 280 alarms fired, none missed. Durable write 47 ms at the median, 375 ms at p99, 18.8 s at worst (a write reaching an actor being taken over). Writable again after its node died: 17.0 s at the median, 31.6 s at worst, so the run failed the soak’s 30 s check once in 1,129. The nodes’ logs show why: after one kill, a surviving node took over 19 actors within 3 s (16 s after the kill, the lease and its grace from the last renewal), and restored them in parallel; one of them, 63 segments, took 17.5 s, while another of 63 on the same node at the same moment took 4.2 s. No read stalled long enough to be cut and sent again (5 s of silence), so it was slow reads, not a hung one; reads are not timed one by one. Alarms after their node died: 22.5 s at the median, 34.1 s at worst; a restarted node was ready within 8.6 s (…-a72fbac-soak-ovh-bhs-3vps-12h.csv).

• Twelve hours on the VPS, on node leases (the same nodes, bucket and cadence, the next night, commit 82dab61: ADR 0020, before the bounded segment uploads): nothing lost or duplicated over 3,659,687 acknowledged writes, 35 kills and 1,072 actors taken over; 279 alarms fired, none missed. Writable again after its node died: 18.2 s at the median, 24.4 s at p99 (the soak’s judged bound is 30 s), 89.6 s at worst, once: that actor’s restore (a listing and 55 reads, one after another) took 56 s, each read slow but never silent for 5 s, so none was cut; the 11 requests whose clients had given up meanwhile then ran, 16 s more. Nothing bounded a restore’s total time then; reads have since been made eight at a time, and a restore’s reads have one deadline (cluster), not measured here. Alarms after their node died: 24.0 s at the median, 35.9 s at worst; a restarted node was ready within 10.4 s. Durable write 57 ms at the median, 674 ms at p99, 29.4 s at worst. The writers wait for each answer, so the 27 % fewer writes than the night before are slower writes (a mean of 93 ms against 69 ms), most of it inside the owner’s durable write; the per-actor runs all ran one night and these the next, so the bucket’s speed and the design are not told apart here. Renewals are gone: PUTs other than segments fell from 3.1 to 7.4 per node and second to 0.80 (node lease writes, 0.33), 1.03 PUTs per durable write against 1.17 (…-82dab617-soak-ovh-bhs-3vps-12h-nodelease.csv).

• Both designs on the same night (the same three VPS and bucket): four 30-minute blocks, a kill every 3 minutes, alternating node leases (82dab61) and per-actor leases (a72fbac, its own bucket prefix and data directories), the harness of 82dab61 for all four. Nothing lost or duplicated, every alarm fired, in all four.

  Block	Leases	Writes	Durable p50 / p99	Writable again p50 / p99 / worst
  1	per node	144,727	53 / 666 ms	17.4 / 23.6 / 34.3 s
  2	per actor	185,012	50 / 419 ms	15.5 / 19.1 / 25.4 s
  3	per node	126,974	62 / 747 ms	16.7 / 26.3 / 37.6 s
  4	per actor	186,427	50 / 367 ms	15.6 / 21.4 / 26.2 s

  On one night, node leases made 27 % fewer writes (the writers wait for each answer, so that is slower writes, mostly in the tail: p99 about 1.7 times), and actors were writable again 1 to 2 s later at the median. So the 12-hour runs’ gap is mostly the design, not the night. Both node-lease blocks ran 82dab61, before the bounded uploads and restore fixes; where in the durable write the time goes is not yet measured (…-ab1-nodelease.csv, …-ab2-peractor.csv, …-ab3-nodelease.csv, …-ab4-peractor.csv; the file names carry the harness’s commit).

• Alarms after their node died: another node may deliver an alarm once it is overdue by the lease plus grace (15 s); then comes the next sweep (every 10 s), a restore and the deliveries in turn: 20 to 30 s at the median.

• Lease renewals were most of the requests (per-actor leases, before ADR 0020, which removed them). Each actor in memory renewed its lease 3.3 s (a third of the ttl) after its previous renewal returned. In the R2 run, Cloudflare counted 47,083 PUT, 4,432 LIST and 47,762 GET for the bucket (…-soak-r2-requests.csv): about 23,000 of the PUTs were renewals, more than the 21,260 writes. On R2 that is about $0.23 of Class A requests beyond the free tier per 30 minutes, and a cluster’s bill grew with how long actors stay in memory, not with what they do. OVHcloud does not bill requests. ADR 0020 replaced them with one lease per node; request counts since are not measured.

The allocator: system against mimalloc

The same suite on the same workstation, twice in a row: main as it is (commit 6063067, glibc’s allocator) and with mimalloc as the server’s global allocator (commit 59f01e8, an experiment that was not adopted). From 2026-11-07-6063067.csv and 2026-11-07-59f01e8.csv.

• No change worth a dependency. CPU per request moved 1 to 3%, and throughput 1 to 2%: within what one run of this suite varies on its own. Memory per resident actor moved under 2%, except TypeScript’s, which fell 4%; the server’s total resident memory grew 4 to 7% for Rust, Zig, C and TypeScript, and fell 2% for Python.
• Latency moves both ways between two runs of the same code. The Rust counter’s median write was 77.5 µs in the first run and 67.4 µs in the second, its median read 59.5 µs and 67.8 µs: a single run’s latency rows differ by up to about 15% without any change.
• What it means for packaging: a static musl build would need a faster allocator than musl’s own to match glibc, and mimalloc brings nothing measurable on glibc; a glibc build linked against an older glibc stays the simpler way to one portable binary. Measured below (Portable builds): it is.

Portable builds

One commit (529fd96) built four ways, each run as the Rust counter’s server by bench/compare, interleaved, three rounds; medians, from 2026-11-07-529fd96-portable-builds.csv. Native is cargo build --release on the workstation (it needs glibc 2.39); glibc 2.28 and musl come from scripts/build-portable.sh.

• The glibc 2.28 build is as fast as the native one, on every row within the rounds’ spread, and runs on any x86_64 Linux with glibc 2.28 or newer (RHEL 8, Debian 10, Ubuntu 20.04 and later). It is the candidate for release binaries.
• A static musl build is slower: 2.3 times the CPU per request, under half the throughput, twice the startup. musl’s own allocator is most of it: with mimalloc the gap narrows to 14% more CPU per request and 13% less throughput, with 8% more resident memory, still behind glibc.
• The musl + mimalloc row is from a second set of rounds against native again (104 µs, 61,200/s there). The workstation had other work running (load average 7 to 14); the builds were interleaved, so it weighs on all of them alike.

WasmKeep on its own

A real bucket adds its PUT round trip to durable writes: 54 ms next to OVHcloud Object Storage, 587 ms from a desktop to R2 (measured). Failover is set by the node lease timings (15 to about 21 s after the node’s last heartbeat) plus a restore; see Running a cluster.

Reproduce

Both entry points require Bash. Full comparisons and bench/run.sh require a soft open-file limit of at least 4096; compare --quick requires 1024 (or unlimited). They stop before building, clearing previous Criterion output or starting a server if the limit is too low. The floor is the smallest power of two at which the server keeps every actor of the workload resident: the server sizes its resident-actor cap from its limit, 256 descriptors reserved, the harness’s --max-connections 64, and three per actor (256 + 64 + 3 * (actors + 2)). Below it the benchmark would measure evictions. Full runs create 1,000 actors; quick runs create 100. Check ulimit -Sn and ulimit -Hn, then explicitly raise only the soft limit in a subshell if the hard limit allows it:

(ulimit -Sn 65536 && ./bench/run.sh)

The benchmark harnesses leave their inherited limits unchanged. On Unix, wasmkeep serve separately attempts to raise its own soft limit within the inherited hard limit, then sizes its connection and resident-actor caps from the effective limit. This does not replace the harness preflight. Choose TMPDIR deliberately: RAM-backed storage and disk have very different creation and write costs. Record the filesystem type with results, using relative paths or placeholders for local directories. bench/run.sh records the kernel’s transparent huge page setting (/sys/kernel/mm/transparent_hugepage/enabled) in a last thp column, unknown where there is none. Earlier files (up to 351ea57) lack it, so check the machine by hand for those. Since commit 351ea57 the server turns them off for itself, so its numbers no longer depend on it, but runs before that on a kernel set to always read about twice the memory per actor (the workstation files cd48f41 to dec1a82). Do not compare memory across that boundary.

./bench/run.sh                     # criterion suites + the comparison (if npm is installed)
WASMKEEP_TEST_S3_BUCKET=... ./bench/run.sh   # also failover, with AWS_* pointing at an S3 API
WASMKEEP_COMPARE_BUCKET=s3://... ./bench/run.sh   # also the comparison on a bucket (compare-bucket/ rows)
WASMKEEP_COMPARE_BUCKET=s3://... BUCKET_LABEL=minio-tmpfs ./bench/bucket.sh     # that comparison alone, in a file of its own
WASMKEEP_COMPARE_BUCKET=s3://... CLUSTER_LABEL=minio-tmpfs ./bench/cluster.sh   # three nodes of WasmKeep and celld, with takeovers
target/release/compare --quick wasmkeep-rust workerd   # just the comparison, smaller

compare --bucket runs each WasmKeep target as a one-node cluster on the bucket WASMKEEP_COMPARE_BUCKET names, each run under a prefix of its own, so every write is also replicated: its rows are compare-bucket/.... Its startup_to_first_response includes the 15 s a cluster node’s first start waits before it listens (cluster). For a local bucket, the MinIO image the Compose example pins serves on loopback; give it a RAM-backed or a disk-backed data directory deliberately, as with TMPDIR, and record which. wasmkeep-rust-durable is the same counter with its POST durable before it returns (exec-durable): fsynced on one node, in the bucket with --bucket. The celld target (bucket runs only) is the workerd counter deployed to one celld node on the same bucket (celld-fleet and celld-bucket are its three-node targets); bench/celld/fetch.sh downloads the pinned release and checks its build provenance with gh attestation verify first.

Before publishing evidence, replace personal directory paths and generated scratch paths with repository-relative paths, [repo] or [scratch]. Stage only the intended artifacts, then run python3 -I scripts/check-benchmark-paths.py. It checks the staged Git blobs under all of bench/ and docs/, so redacting only the working copy cannot conceal an unsafe staged version. Reviewed system paths, product routes and documentation-site links are allowed; test files are scanned too. The check recognizes named-home paths and paths wrapped in Markdown emphasis, validates IPv6 CIDRs, and limits regex/printf escape exceptions to complete quoted patterns or reviewed printf examples. Adjacent private paths still fail. An optional pre-commit hook runs this check locally. CI checks files added or changed by every introduced commit with --range BASE..HEAD, including paths removed before the final commit. Unchanged files inherited from an old base are not attributed to the PR. CI runs after publication, so it cannot prevent the initial push. This checks local paths; review logs for other private information before staging them.

Regression check on pull requests

A pull request that changes the runtime, the server or the benchmarked examples runs scripts/bench-ab.py in CI: the base it merges into and the change are built and measured on the same runner, three rounds of the instantiate, http and actors Criterion suites with the two sides back to back. Each round gives a ratio per benchmark; the check fails if a benchmark’s median ratio is more than 20% slower. Three rounds at that threshold catch large regressions, not small ones, and numbers from shared CI runners are not comparable with each other or with the files in bench/results/: only the ratio between the two sides of one run is, and the job’s summary shows it.

hello/request/raw-hyper (plain hyper, no WasmKeep code) never fails the check; if it moved by more than 15%, the runner was too noisy and the run only reports. A base that cannot be built or measured is reported, not failed. To run it locally against any revision (it keeps its builds and results under target/bench-ab/):

python3 -I scripts/bench-ab.py main