What is WasmKeep?

WasmKeep runs actors: small, long-lived, addressable programs that each own their state. You write an actor once (a class, compiled to a WebAssembly component) and WasmKeep runs as many instances of it as you have names.

/actor/chat/room-42      → the chat-room class, instance "room-42"
/actor/limit/api-key-7   → the rate-limiter class, instance "api-key-7"

The first request to a name creates the actor. There is no “create” call, no registry and no database to provision.

The guarantees

Each actor:

1. Handles one event at a time, in arrival order. HTTP requests, WebSocket messages and alarms queue up in the actor’s mailbox. Inside an actor you never need a lock.
2. Owns a private SQLite database. Nothing else can read or write it. Committed writes survive restarts.
3. Hibernates when idle. Its code leaves memory after a timeout (30 s by default) and is brought back by the next event, in about a millisecond. Data, the pending alarm and open WebSockets all survive this.
4. Is sandboxed and limited. It can only use what the wasmkeep:actor interface grants: its database, alarms, calls to other actors, WebSockets, outbound HTTP, clocks, randomness, and its configuration and secrets (the env interface). What it prints to stdout or stderr reaches the node’s log. Each event gets a CPU budget (50 ms by default) and each actor a memory cap (128 MiB per linear memory, at most two, by default).

Different actors run in parallel, so a server’s throughput comes from having many actors, not from making one actor busy.

What it is good for

Anything with many independent pieces of state, each touched by a modest stream of events (use cases compares this with stateless servers, Postgres and Valkey, and walks through each pattern):

When not to use it

• One big shared dataset (analytics, reporting, search across all users): use a real database. Actors are islands; querying across all of them is slow by design.
• A single very hot object: one actor runs on one core at a time. Split the state (one actor per room, not one per app).
• Stateless functions: plain wasi:http components work in WasmKeep, but Spin or wasmCloud are built for that.
• A global edge network: WasmKeep is self-hosted; it does not route users to the nearest of hundreds of locations.
• Production today: it is pre-alpha. Clustering works and has been soaked against R2 and OVHcloud Object Storage with nodes killed under load (measured), but has not yet carried real users’ traffic.

WasmKeep and Cloudflare Workers compares the two in more detail: where each is the better choice, and why.

Next: Concepts for how actors behave, or your first actor to try it.

How it works, briefly

• Runtime: Wasmtime, with a pooling allocator and ahead-of-time compiled, cached components.
• Interface: a WIT world on top of the standard wasi:http/proxy, so the request handler is ordinary WASI HTTP, plus WasmKeep’s storage, alarms, actor calls and WebSockets.
• Languages: anything that builds a WASI 0.2 component. Rust, TypeScript, Python, C, Zig and Go (TinyGo) have SDKs and pass the same tests; D and MoonBit have experimental SDKs, and standard Go and C# each build a counter (see languages).
• Storage: SQLite in WAL mode, one file per actor, sandboxed (no ATTACH, no extensions, no state-changing pragmas).
• Clustering: each actor’s changes stream to an S3-compatible bucket; several nodes share the bucket and use it to decide, with conditional writes, which node owns each actor. If a node dies, another takes its actors over. No consensus cluster to run: if the bucket is up, WasmKeep is up.

Design details and the reasoning behind each decision are in docs/adr; Concepts and Running a cluster describe how it behaves, and When a node dies, and how it scales says what an application sees of it.

Who it is for, and real-world use cases

Most applications are well served by stateless servers in front of Postgres, with Valkey (or Redis) for caching, queues and pub/sub. WasmKeep does not replace that stack. It is for the parts of an application where that stack gets awkward: state that many requests fight over, connections that must meet in one place, timers that must fire exactly when they should, and code or data that must stay isolated per customer.

The short version:

• A good fit: many small, independent pieces of state (a room, a user, a device, an API key, a document), each receiving a modest stream of events, often with WebSockets or timers attached.
• Not a fit: one big shared dataset you query across (reporting, search, analytics), a single object every request touches, or plain stateless APIs. Details in the overview.

Next to Postgres, Valkey and stateless servers

Use both. Actors are islands: each one’s data is fast to reach through it, and slow to query across. Keep Postgres for what spans everything (reporting, search, billing, analytics) and use actors for the hot, coordinated, real-time parts. A typical split: your existing API and database stay; chat, presence, rate limits, per-user workflows and device state move to actors; actors call your API with outbound HTTP, or push summaries to it, when the rest of the system needs to know.

Use cases

Each pattern below names the actor, what it keeps, and the features it leans on. Where the repository has a working example, it is linked.

Real-time rooms: chat, presence, collaboration, game lobbies

• One actor per room, document or match.
• Keeps: members, recent history (SQLite), whatever the room needs.
• Uses: WebSockets attached to the actor (they survive hibernation), broadcasts to every socket, alarms for timeouts (an idle match closes, a turn times out).
• Why actors: every message for a room arrives at the same place, in order, without a pub/sub layer between servers.
• Example: examples/chat-room, also in TypeScript, Python, C, Zig, Go, D and MoonBit.

Rate limits and quotas

• One actor per API key, IP, user or tenant.
• Keeps: the current window and its count.
• Uses: one event at a time (the count is exact), an alarm to close the window; an idle key deletes itself, so unused keys cost nothing.
• Why actors: no lock, no race between servers, no shared counter that every request contends on.
• Example: examples/rate-limiter.

Scheduled and long-running work: monitors, reminders, workflows

• One actor per monitored URL, reminder, order or workflow run.
• Keeps: its state and history; the next step.
• Uses: alarms (persisted with the actor, retried with backoff when the handler fails, fired by another node if its owner dies), outbound HTTP, exec-durable for steps that must not be lost; for multi-step work, the durable workflows library (Rust and TypeScript, with clients in Go, Python, C and Zig), which runs each workflow on an ordinary actor.
• Why actors: the schedule lives with the state it acts on; there is no cron, job table or lock to keep consistent.
• Example: Watch, an uptime monitor with one actor per URL and a hub that delivers alerts to a webhook; what building it taught us is in dogfood. Workflows: examples/workflows and examples/ts-workflows.

Per-user state: sessions, carts, settings

• One actor per user, cart or session.
• Keeps: the session, the cart, the user’s settings.
• Uses: durable writes, alarms for expiry or reminders, outbound HTTP.
• Why actors: each user’s state is local to its code and touched by one event at a time, so concurrent tabs or retries cannot corrupt it.

AI agents

An agent is state plus a loop: a conversation’s history, what it has learned, the tools it called and what they returned, waiting for a model, a tool or a person, then acting again. That is an actor.

• One actor per conversation, agent or task (a ticket, a document, a customer).
• Keeps: the conversation and its memory in SQLite (messages, facts, summaries, tool results), surviving restarts and node failures.
• Uses:
  • outbound HTTP to call a model and tools: hosted APIs by default (public addresses only); a model server on your network (Ollama, vLLM, an internal gateway) once --allow-outbound opens its range, and --outbound-ca trusts its private CA;
  • WebSockets to stream progress and answers to the user, attached to the agent’s actor and kept while it sleeps;
  • alarms for anything that waits: a reminder, a follow-up, a retry of a failed tool call, a periodic check of an inbox or a feed. Delivered at least once, also when the node dies;
  • actors.call for agents that work together (a planner and workers), with call cycles refused instead of deadlocking;
  • a counter actor per user or key to cap model spend, exact under load.
• Why actors: two messages to the same conversation are handled in order, never interleaved; an agent waiting for a person or a slow tool costs disk, not memory; the history lives next to the code that uses it. Tool code you did not write (a plugin, or code a model produced) runs in the Wasm sandbox with CPU and memory limits, with the pre-alpha caveat given for customer code below.
• Sovereignty: model calls can stay on your network, and the history stays on your servers and in your bucket, for teams that cannot send conversations to a third party’s runtime.
• Example: examples/agent, one actor per conversation calling an OpenAI-compatible endpoint (hosted, or Ollama through --allow-outbound), with the history in SQLite, a WebSocket feed, a reminder tool on an alarm and a cap on model calls per hour.

Limits today, and how to work with them:

• 30 s per event. An event (a request, an alarm, a call) runs at most 30 s of wall-clock time; programs embedding WasmKeep can change it (ActorConfig::wall_per_event), wasmkeep serve cannot yet. A long model call can reach it: stream the answer to the user over the WebSocket as it arrives, and split long work into steps (one tool call or model turn per event, the next one scheduled with an alarm or a call), recording progress in SQLite so a cut step resumes.
• No model runs inside an actor. Actors call models over HTTP; there is no GPU access from Wasm.
• No vector index built in. A conversation’s own memory is small enough to search in SQLite (stored embeddings, compared in code). Search across a large corpus belongs in a dedicated store (pgvector, Qdrant) that agents call.
• Language costs. Python and TypeScript actors carry their runtime: more memory per resident agent than Rust, C or Zig; TypeScript on QuickJS, the default engine, uses about a fifth of StarlingMonkey’s (741 KB against 3.4 MB) (languages, benchmarks).

Device twins and the edge

• One actor per device or site.
• Keeps: the device’s latest state and pending commands.
• Uses: alarms for missed heartbeats, WebSockets for live control.
• Where: on one small server per site, or a cluster with a local MinIO, without outbound network at run time.

Customer code in a SaaS product

• One actor per tenant or plugin instance.
• Keeps: the plugin’s own data, separate from every other tenant.
• Uses: the Wasm sandbox (no filesystem, sockets or environment unless granted; outbound HTTP limited to public addresses), a CPU budget per event, a memory cap per actor, access rules per class.
• Why actors: isolation of code and data per customer, from any language that compiles to a Wasm component.
• Caveat: today a process serves one tenant, and the sandbox protects the host from buggy code, not yet from a determined attacker (security model).

Where it fits operationally

• One VPS: a single binary, data on local disk. A side project can run here for years.
• A small cluster (three nodes or more): any S3-compatible bucket that supports conditional writes (AWS S3, Cloudflare R2, OVHcloud Object Storage, MinIO; R2 and OVHcloud tested), which each node checks at start; a dead node’s actors are restored on another, with no consensus cluster to run (three nodes, or the Compose example to try it locally).
• On-premises and sovereign deployments: everything, the bucket included, can stay on your hardware and in your jurisdiction.
• Hosting providers: the same binary can be offered as a managed service; WasmKeep itself has no hosted cloud.

Status

WasmKeep is pre-alpha. What exists and what is measured is in the ROADMAP and benchmarks. Clustering has been soaked against Cloudflare R2 and OVHcloud Object Storage: three nodes, one killed every 3 minutes under load, 36 kills over four 30-minute runs, nothing acknowledged lost (measured). It has not yet carried real users’ traffic.

WasmKeep and Cloudflare Workers

WasmKeep’s programming model resembles Cloudflare’s Durable Objects: one addressable instance per name, one event at a time, its own storage, alarms and WebSockets. This page says where each is the better choice and why. Numbers come from Benchmarks; everything else describes how the two are built. Two different things carry the Cloudflare name here: workerd, the open-source runtime anyone can run, and Cloudflare Workers, the hosted platform built on it. Much of what makes Workers strong is the platform, not the runtime.

Where WasmKeep is the better choice

• Durable actors on your own servers, with failover. Each actor’s SQLite database is replicated to any S3-compatible bucket, and when a node dies another takes its actors over. With one lease per node (ADR 0020), over twelve hours and 35 kills under load next to the bucket: writable again 18.2 s after the kill at the median, 24.4 s at p99, with nothing acknowledged lost; one actor was writable again only 90 s after the kill, most of it a restore whose total time nothing bounds yet (measured). workerd runs Durable Objects on one machine, without replication or failover between machines; that is the hosted platform’s.
• Any language, as an equal. Actors are standard WebAssembly components run by Wasmtime: Rust, TypeScript, Python, C, Zig and Go (TinyGo) (languages). Workers are built around JavaScript; other languages run as WebAssembly inside a JavaScript isolate, with JavaScript glue.
• Less CPU for compiled languages. On the same counter, WasmKeep with Rust used 104 µs of server CPU per request against workerd’s 128 µs on a 16-core workstation, and 39% of workerd’s on a 4-vCPU VM; reads and writes were about 2.6 times faster at the median. Memory per resident actor is about the same (250 KB against 277 KB) (measured). Compiled WebAssembly runs close to native speed, with no JavaScript engine in between.
• Your data, your bill. The data is SQLite files and a documented replication format (ADR 0007) in a bucket you own. There is no per-request price: a cluster costs its servers and its bucket (on R2, its requests too: measured).
• Durable workflows as a library, on actors and alarms, in Rust and TypeScript with clients in the Go, Python, C and Zig SDKs (workflows).

Where Workers is the better choice

• JavaScript-heavy code. V8 compiles JavaScript as it runs; WasmKeep’s TypeScript runs on QuickJS, an interpreter inside each actor, and used about 1.8 times workerd’s CPU per request on the same counter. Apps that do a lot of work in JavaScript belong on workerd.
• Maturity. Workers has years in production at Cloudflare’s scale, a hardened V8 sandbox, web-standard APIs (fetch, streams, the cache API), npm compatibility and mature tooling. WasmKeep is pre-alpha.
• Concurrency inside a request. A Worker can await several things at once. A WasmKeep handler is synchronous: an outbound http.fetch blocks the actor until it answers, and the actor’s other events wait.
• Hostile tenants side by side. Cloudflare runs untrusted code from many customers on shared machines. WasmKeep’s WebAssembly sandbox is strong, but a tier for tenants who do not trust each other (a process or microVM per tenant) is not built yet (security model).
• The platform around it. A global network that routes users to a nearby location, and managed key-value, queues, SQL and object storage next to the code. WasmKeep is self-hosted, in one region per cluster.

In short

WasmKeep is the better fit for stateful, self-hosted apps, especially in compiled languages, where the data should live in your own bucket and the cost should be your servers. Workers is the better fit for JavaScript-first apps that want a global edge and a mature ecosystem today. The two programming models are close: an app designed around one actor per room, user or document keeps its design on either, though its code has to be ported to the other’s APIs.

Cloudflare Workers, Durable Objects and workerd are Cloudflare’s products, named here only to identify them (licensing).

WasmKeep and celld

celld and WasmKeep solve the same problem the same way: Durable Object-style actors on your own servers, each with its own SQLite database, replicated to an S3-compatible bucket, and taken over by another node when one dies. This page says where each is the better choice. Numbers come from Benchmarks, where both run the same counter on the same machine and the same bucket: one node (measured) and three nodes with a node killed under writes, five runs with the nodes’ data on disk and five on tmpfs (measured). Everything else describes how the two are built, from their documentation. celld v0.6.3 was measured.

Two meanings of durable

The biggest difference is what a write waits for once there are several nodes.

• WasmKeep answers a durable write (exec-durable) once the bucket holds it, on one node or many. A plain exec answers before its upload, and may be lost with its node.
• celld always waits. On one node, it waits for the bucket. With two or more nodes, by default (fleet), it answers once a follower node has the write on disk, or once the bucket has it, whichever comes first; the upload follows. CELLD_DURABILITY=bucket makes it wait for the bucket as WasmKeep does.

A follower’s disk is usually much closer than a bucket, so celld’s default should answer writes sooner when the bucket is in the cloud. That case is not measured yet. In the three-node runs with the nodes’ data on one shared disk, fleet mode was celld’s slowest in every run (and it completed fewer writes than bucket mode on tmpfs too), which is not explained. Against a write that waits for the bucket in both, the numbers below compare like with like.

Neither lost an acknowledged durable write. Every write carried a token of its own, and every token was read back after each restart and at the end: sixty kills for each of WasmKeep durable, celld fleet and celld bucket, half of them with the killed node’s disk removed before it restarted. A control run that must lose writes lost them every time, so the check finds a loss. What it does not test: a power loss on the nodes that survive, and nodes on separate machines (lost writes). The figures below are medians of five runs with the nodes’ data on disk, unless tmpfs is named.

Where WasmKeep is the better choice

• Any language, as an equal. Actors are standard WebAssembly components run by Wasmtime: Rust, TypeScript, Python, C, Zig and Go (TinyGo), with D and MoonBit experimental (languages). celld runs the Workers API on V8: JavaScript first, other languages as WebAssembly inside it.
• Less CPU and memory on one node. On the same counter, each write in the bucket, WasmKeep with Rust used 85% of celld’s CPU per request, held 1000 actors in 334 against 388 MB, answered a write in 2.31 against 2.55 ms at the median (0.53 against 0.93 ms on tmpfs) and read about twice as fast. On disk its throughput was close to celld’s, about 5% behind at the median (measured).
• Faster reads and less memory on three nodes. Reads took 92 against about 220 µs at the median, and three nodes held 1000 actors in 459 against about 520 MB (measured).
• Encrypted traffic between nodes. WasmKeep nodes talk mutual TLS by default (cluster); celld has no TLS between peers, so its peer listener belongs on a private network.
• A write that does not wait. For data that can be rebuilt, a plain exec answers before its upload: 0.55 ms at the median on three nodes, and 13,416 writes per second against 1,710 durable. It is the one mode that lost writes when a node was killed: 0 to 3 of about 86,000 in a run.

Where celld is the better choice

• Existing Workers code. celld runs the Workers API and Durable Objects as they are: the same counter ran on workerd and celld unchanged. WasmKeep has its own API, and Workers compatibility is not planned (roadmap).
• Faster takeover and start. With each one’s defaults, a dead node’s actors were writable again after 9 to 10 s on celld against 15 to 16 s on WasmKeep (medians, data on disk), celld sooner in every run; both are close to their default lease timings (10 s and 15 s). A celld node starts in about 0.1 s; a WasmKeep cluster node’s first start checks for 15 s that no other process has its node id before it listens, and a node restarted after its peers fenced it was ready in about 1.6 s.
• Durable writes on three nodes with data on disk, waiting for the bucket: close, celld ahead. In bucket mode, eight clients completed 1,857 writes per second on celld against 1,710 on WasmKeep, about 8% apart where runs differ by 2 to 4%, at the same CPU per write; WasmKeep answered a single write sooner at the median (2.30 against 2.68 ms) and later at p99 (measured). With the nodes’ data on tmpfs the order reverses (9,137 against 5,888 writes per second); the runs change only where the data is, and do not say which step of a write the disk slows.
• JavaScript-heavy code. V8 compiles JavaScript as it runs; WasmKeep’s TypeScript runs on QuickJS, an interpreter (WasmKeep and Workers).
• More of the platform built in. celld comes with key-value, queues, D1-style SQL and R2-style object storage. WasmKeep offers key-value, queues and workflows as SDK patterns on actors (key-value and queues, workflows).

Not measured yet

• A cloud bucket, next to the nodes. It decides whether celld’s fleet mode answers writes sooner than waiting for the bucket, and by how much.
• Nodes on separate machines. The three nodes ran as processes on one machine and shared its disk.
• Longer runs. WasmKeep has a twelve-hour soak against a real bucket (measured); celld was run for minutes.

In short

Both keep each actor’s data in SQLite, in a bucket you own, and no acknowledged durable write was lost through the kills measured here. WasmKeep fits stateful apps in compiled languages, with less memory per actor, less CPU on one node, faster reads and encrypted traffic between nodes. celld fits Workers code you already have, and starts and takes over faster with its defaults. Which answers a durable write sooner on a cloud bucket is the open question, and the next measurement.

celld is Deno’s project, and Cloudflare Workers and Durable Objects are Cloudflare’s products, named here only to identify them (licensing).

Your first actor in 10 minutes

You will write a small actor in Rust, build it into a WebAssembly component, serve it with WasmKeep, and see its state survive a restart.

Applies to: WasmKeep 0.3.0 (pre-alpha). Commands assume Linux; on macOS, build the server from source (step 1).

1. Get WasmKeep

Install the wasmkeep binary from a release, as in Install (Linux, x86_64 or aarch64), and check it answers:

wasmkeep --version

On macOS, or to run the server from source: git clone https://github.com/wasmkeep/wasmkeep, then cargo build --release -p wasmkeep-cli in it writes target/release/wasmkeep.

To build the actor you need rustup with Rust 1.97 or later, and the WebAssembly target:

rustup target add wasm32-wasip2

2. Create the actor

An actor is a Rust library compiled to a component. Create it anywhere outside a clone of WasmKeep, and add the SDK from crates.io:

cargo new --lib visits && cd visits
cargo add wasmkeep-sdk

cargo add takes the newest version: the SDK carries the version of the server it is released with, so keep the two the same (cargo add wasmkeep-sdk@0.3.0 names one). Add to Cargo.toml:

[lib]
crate-type = ["cdylib"]

# Keeps the component small, as in every example.
[profile.release]
opt-level = "s"
strip = true
lto = true
codegen-units = 1

Replace src/lib.rs with:

#![allow(unused)]
fn main() {
use wasmkeep_sdk::http::{self, IncomingRequest, ResponseOutparam};
use wasmkeep_sdk::{Actor, Value, params, storage};

struct Visits;

impl Actor for Visits {
    // Runs once per actor (and again after you deploy a new version):
    // create your tables here.
    fn on_init() -> Result<(), String> {
        storage::exec(
            "CREATE TABLE IF NOT EXISTS visits (who TEXT NOT NULL)",
            &[],
        )
        .map(drop)
        .map_err(|e| e.to_string())
    }
}

impl http::Guest for Visits {
    fn handle(request: IncomingRequest, out: ResponseOutparam) {
        // `/?who=alice` records a visit by alice.
        let path = request.path_with_query().unwrap_or_default();
        let who = path.split_once("who=").map_or("someone", |(_, w)| w);

        let result = storage::exec("INSERT INTO visits (who) VALUES (?1)", &params![who])
            .and_then(|_| storage::query("SELECT count(*) AS n FROM visits", &[]));
        match result {
            Ok(rows) => {
                let n = rows.get(0, "n").and_then(Value::as_i64).unwrap_or(0);
                let body = format!("hello {who}, you are visitor number {n}\n");
                http::respond(out, 200, "text/plain", body.as_bytes());
            }
            Err(e) => http::respond(out, 500, "text/plain", e.to_string().as_bytes()),
        }
    }
}

wasmkeep_sdk::export!(Visits);
}

Two traits, one macro:

• Actor holds the lifecycle hooks (on_init, on_alarm, and on_websocket_open, on_websocket_message, on_websocket_close). All are optional; by default an actor refuses WebSockets.
• http::Guest is the request handler (standard wasi:http).
• export! wires both into the component.

3. Build it

cargo build --release --target wasm32-wasip2

The component is target/wasm32-wasip2/release/visits.wasm, close to the 81 KB of the Rust counter, which uses the same release profile.

4. Serve it

wasmkeep serve --actor visits=target/wasm32-wasip2/release/visits.wasm

Every actor lives at /actor/<class>/<name>/.... The class is visits; the name is anything you like, and each name is a separate actor. In another terminal:

curl 'localhost:8080/actor/visits/home?who=alice'
# hello alice, you are visitor number 1
curl 'localhost:8080/actor/visits/home?who=bob'
# hello bob, you are visitor number 2
curl 'localhost:8080/actor/visits/about?who=carol'
# hello carol, you are visitor number 1

home and about are two actors, each with its own SQLite database under .wasmkeep/data/actors/default/visits/, in the directory the server started in.

5. Restart it

Stop the server with Ctrl-C, start it again with the same command, and:

curl 'localhost:8080/actor/visits/home?who=dave'
# hello dave, you are visitor number 3

The count survived because it lives in the actor’s database, not in memory.

What just happened

• One event at a time. Requests to home run one after another, in arrival order, so INSERT then SELECT can never interleave with another request to home. You write no locks. Requests to different actors run in parallel.
• Hibernation. After 30 s without events an actor is dropped from memory; the next request brings it back in about a millisecond. Its database stays on disk.
• Sandboxed. The component can reach its own database, alarms, calls to other actors, WebSockets, outbound HTTP, clocks, randomness and its configuration (the env interface; the WASI environment stays empty), and nothing else: no files, no sockets. What it prints (println!) goes to the node’s log.
• Limits. Each event gets 50 ms of CPU (and 30 s of wall time), and each of an actor’s linear memories (at most two) is capped at 128 MiB, by default. A guest that loops forever is stopped and gets a 500; the server and other actors carry on.

Next

• wasmkeep dev . rebuilds and reloads your actor every time you save, keeping its data.
• examples/rate-limiter uses alarms: code that runs later, at least once, even across restarts.
• The release carries more to try without building: key-value and queues and workflows, as classes your actor calls.
• wasmkeep_sdk::actors::call lets one actor call another.
• The design and its guarantees are in Concepts, with an overview of the architecture.

Languages

An actor is a WebAssembly component implementing the wasmkeep:actor world: an ordinary wasi:http request handler, plus on-init, on-alarm and three WebSocket hooks, with access to its SQLite database, alarms, actor calls and WebSockets. Any language whose toolchain can produce a WASI 0.2 component can write actors. WasmKeep runs them directly on Wasmtime, with no JavaScript or other runtime in between unless the language brings its own.

Which language to use

WasmKeep runs WASI 0.2 components on a server. A language works if its toolchain can produce a component that implements the wasmkeep:actor world. Most “WebAssembly support” you read about targets the browser instead (wasm-bindgen, Emscripten, Blazor, Pyodide, Kotlin/Wasm for JS): that is a different target, and does not make a language usable here.

Today

Getting an SDK

Each SDK has the version of the server it is released with (versions); here for 0.3.0.

The release’s files are signed: check a download.

Next, in order

Each is experimental once a counter passes the shared tests; numbers from bench/run.sh come before it is supported.

1. An SDK for standard Go, which now builds actors through componentize-go (below).

Bring your own through the C SDK, the way Zig does: Odin and Nim should work (C-sized components); Swift and Haskell are possible with more effort (no bindings generator, larger runtimes). Recipes, not SDKs.

Not yet

Component sizes

The same counter, as built by each example’s build.sh (sizes on disk; see benchmarks for speed and memory):

TypeScript on QuickJS

QuickJS is the default engine (ADR 0018): wasmkeep-ts build (npm run build in the TypeScript examples) uses componentize-qjs 0.5.0, which the pinned jco 1.41.0 already ships: the QuickJS-ng engine, pre-initialized with your bundle, built without component-model async (WasmKeep’s hooks are synchronous). wasmkeep-ts build --engine starlingmonkey (npm run build:starlingmonkey) builds the same source with ComponentizeJS and StarlingMonkey instead.

The counter, the chat room and the jobs app pass the same tests on both engines (crates/wasmkeep-server/tests/languages.rs, tests/chat.rs, tests/jobs_langs.rs). Measured on one machine (benchmarks), QuickJS against StarlingMonkey: 1.7 MB instead of 13.4 MB, 741 KB of memory per resident actor instead of 3.4 MB, and 230 µs of CPU per request instead of 368 µs on the counter; but 5.8 ms instead of 3.8 ms on plain JavaScript work.

Choose StarlingMonkey when the code (or an npm package it uses) needs the Web APIs below, or spends milliseconds computing in JavaScript per event.

What does not work, or works differently, on QuickJS:

• Fewer Web APIs. QuickJS has no URL, URLSearchParams, crypto (getRandomValues, randomUUID), fetch, Headers, Request, Response, streams, Blob, AbortController, setTimeout or structuredClone; StarlingMonkey has all of these. Neither has Intl. Code or npm packages that use them fail at build time (during pre-initialization) or when called, with an error that names --engine starlingmonkey. They are not stubbed, so libraries that test typeof fetch still pick their fallback. QuickJS does have atob, btoa, queueMicrotask, performance and WeakRef. For outbound HTTP on either engine, the SDK has http.fetch: synchronous, as handlers are (StarlingMonkey’s own fetch returns a promise a handler cannot await).
• console comes from the SDK, writing one line per call to stdout (log, info, debug) or stderr (warn, error), as StarlingMonkey’s does; objects are shown as JSON. WasmKeep logs both (what actors print).
• TextEncoder and TextDecoder come from the SDK, UTF-8 only and without streaming decode. The SDK installs them when the engine lacks them, so the same code runs on both engines.
• A WIT none is null, not undefined. The SDK’s env.get and alarms.get return undefined on both engines; code that imports wasmkeep:* or wasi:* interfaces directly sees null.
• Slower JavaScript. Neither engine has a JIT, and QuickJS’s interpreter is slower: 5.8 ms against 3.8 ms for the same plain JavaScript work (measured). The counter spends little time in JavaScript, so there QuickJS wins on instantiation and the host.
• Not covered by the shared tests: execDurable from TypeScript on either engine (actors.call is, through the jobs app’s key-value and queue clients). Alarms were checked by hand on QuickJS.

Go

TinyGo’s wasip2 target writes a component directly, against a WIT world it is given. The Go SDK is a wasmkeep package over the bindings wit-bindgen-go generates (committed in the module, so an app needs only TinyGo): an Actor of optional hooks, Exec, Query and Rows, alarms, Call, WebSockets, Env, Self and a synchronous Fetch, plus the key-value, queue and workflow clients. TinyGo’s runtime imports wasi:filesystem whether or not the program uses it, so the module’s wit/ adds those imports to the actor world (actor-tinygo); the host provides them inert (no directories).

An app adds the module with go get github.com/wasmkeep/wasmkeep/sdks/go@v0.3.0: the module carries the server’s version number. The wasmkeep package starts at v0.2.0 (v0.1.0 has only the clients); see the SDK’s Install.

The counter, chat room and jobs app pass the shared tests, and fetch_langs checks Fetch against the Rust SDK’s http::fetch. Trade-offs:

• A garbage collector. TinyGo’s runs inside the request: the counter’s p99 is 10× (reads) to 40× (writes) its median, against under 1.5× for Rust (measured). Allocating less per event shortens it.
• TinyGo, not Go. TinyGo compiles most of the standard library, not all of it (its documentation lists which packages build); the SDK keeps to packages it handles well (no fmt or reflection).
• Size. 162 KB for the counter, about twice Rust’s; strconv’s number parsing alone adds about 50 KB, so the SDK reads integers from SQLite without it.
• Not covered by the shared tests: SetAlarm and the Alarm hook, ExecDurable and DeleteSelf from Go (Call is, through the jobs app’s clients).

Standard Go, through componentize-go

Standard Go (not TinyGo) builds actors through componentize-go 0.5.1 with Go 1.27: the counter passes the same counter contract as Rust, and CI builds it. What blocked our own attempt before (Go’s runtime assumes it owns all linear memory past its data, while the WASI 0.1 adapter needs memory before Go starts) does not show: the component instantiates, survives hibernation and restarts.

What works:

• The whole world. componentize-go bindings generates Go packages for every interface of crates/wasmkeep-wit/wit, and the app implements two packages it names: export_wit_world (the five hooks, returning witTypes.Result[witTypes.Unit, string]) and export_wasi_http_incoming_handler (Handle). The component imports exactly the world’s 0.2.12 interfaces.
• Go’s own compiler and runtime, not TinyGo’s subset of them: what builds with go build for wasip1 builds here.

What is missing:

• An SDK. componentize-go’s bindings (go.bytecodealliance.org/pkg) are not the wit-bindgen-go ones the Go SDK wraps for TinyGo, so the SDK does not build on standard Go yet. Its kv, queue and workflow clients import no bindings and would carry over.
• Size. 2.7 MB, most of it Go’s runtime (stripping custom sections saves under 3%), against 162 KB on TinyGo; no numbers from bench/run.sh for CPU or memory per actor yet.
• Rough edges. The module must be named wit_component (the bindings import it by that path), bindings rewrites go.mod from its own template on every run (the example’s build.sh keeps the committed one), the build needs -buildvcs=false inside a git checkout, and the tool downloads its own binary from its GitHub release into the user’s cache directory on first use.

D

LDC compiles D with -betterC (no garbage collector, no D runtime, no exceptions) to objects for wasm32-unknown-wasi; clang links them against wasi-libc as a reactor, and the WASI 0.1 adapter makes the component, as for C. wit-bindgen d generates the bindings, and the D SDK wraps them: a struct of static hooks exported with mixin Actor!S, exec, query and Rows with parameters as plain D arguments, alarms, call, WebSockets, env and self. Without a garbage collector, everything the SDK returns lives until the event ends and is freed then.

The counter and chat room pass the shared tests. D stays experimental until bench/run.sh measures it; the SDK has no outbound HTTP, key-value, queue or workflow clients yet. -betterC also means no D standard library parts that need the runtime (most of Phobos): text is built with reply’s printf formats or arena memory.

MoonBit

wit-bindgen moonbit generates the bindings as a MoonBit module whose world package declares the hooks (declare pub fn on_init, …); the MoonBit SDK’s build.sh adds the SDK package, the app package and a small glue file that implements those declarations by calling the app’s actor (an @wasmkeep.Actor of optional hooks). moon build --target wasm makes a core module with no WASI imports of its own, and wasm-tools makes the component, with UTF-16 strings (MoonBit’s). The SDK wraps storage (exec, query, Rows), alarms, call, WebSockets, env and self_address; failures are MoonBit errors (Failure), raised from the SDK and caught at the hooks.

The counter and chat room pass the shared tests. MoonBit stays experimental: the language is pre-1.0 (its string layout changed in 2026, so the compiler, core library and wit-bindgen are pinned together), bench/run.sh does not measure it yet, and the SDK has no outbound HTTP, key-value, queue or workflow clients. MoonBit’s servers publish versions by their compiler version (0.10.20+65efd5b93, as moon version --all prints it), not by moon’s dated version; CI pins that one, checked by SHA-256.

C# (spike)

C# can produce a WasmKeep actor today. The counter is built with componentize-dotnet 0.8.0-preview00011 and NativeAOT-LLVM 10.0.0-rc.1 on the .NET 10 SDK, and passes the same counter contract as Rust; CI builds it.

What works:

• The whole world. componentize-dotnet runs wit-bindgen c-sharp on crates/wasmkeep-wit/wit at every build; the app implements two generated interfaces (IActorWorldExports for the five hooks, IIncomingHandlerExports for requests) and calls the imports as static methods (IStorageImports.Exec, …).
• One build command. dotnet build restores the compiler and the tools it needs and writes a WASI 0.2 component directly; no adapter.
• Errors as exceptions. A WIT error result is thrown as a WitException; a hook fails its event by throwing WitException<string>.

What is missing:

• Everything is preview. The compiler comes from Microsoft’s dotnet-experimental feed (a release candidate), the packages are previews, and the first restore is about 800 MB.
• No macOS. At the version componentize-dotnet uses, the NativeAOT-LLVM compiler is published for Linux (x64 and arm64) and Windows x64; the macOS packages on the feed are years older, and Counter.csproj names the Linux x64 one.
• An older filesystem import. The component imports wasi:filesystem@0.2.6 where the world has 0.2.12, so wasm-tools component targets rejects it; WasmKeep links the imports semver-compatibly and runs it.
• No SDK. The code names generated types in full (ActorWorld.wit.Imports.wasi.http.v0_2_12.ITypesImports.OutgoingResponse) and does its own HTTP plumbing; an SDK on the bindings, the chat room and numbers from bench/run.sh come before a tier above experimental.
• Size. 2.5 MB as built, 929 KB with its debug sections stripped (wasm-tools strip --all, which build.sh runs): NativeAOT compiles the runtime and the garbage collector in.

Versions

The SDKs carry the server’s version: one number for the wasmkeep binary, the Rust crate and every other SDK manifest, changed together. Use the SDK with the same version as the server you run: other pairings are not tested.

0.2.0 is the first version on the registries (crates.io, npm, PyPI); 0.1.0 was used from a clone of the repository. That the SDK and the WIT package both read 0.2.0 in the 0.2.0 row is a coincidence.

The WIT package version is another number: it is the version of the interface between a component and the host (actor.wit), and changes only when that interface does.

scripts/check-versions.py (run by ./scripts/check.sh and CI) fails when a manifest it lists has another version than the workspace’s in Cargo.toml; an SDK that gains a manifest with a version is added to that list. The C SDK’s is WASMKEEP_SDK_VERSION in wasmkeep.h, the Zig SDK’s version in wasmkeep.zig; a release carries both SDKs as archives of that version (wasmkeep-c-sdk-<version>.tar.gz, wasmkeep-zig-sdk-<version>.tar.gz).

What the tiers require

• Experimental: a counter passes the same black-box test as the Rust one (create, increment, persist across hibernation and restart).
• Supported: an SDK; the counter and the chat room pass the shared tests (--test languages, --test chat); it builds in CI from a clean checkout with pinned tool versions; this page states its trade-offs with numbers from bench/results.
• Recommended: all of that, a guide (how to start an actor and call the storage, alarm, call and WebSocket APIs; today only Rust has one), and benchmark rows close to the best measured.

Languages that embed an interpreter or a garbage-collected runtime (JavaScript, Python, Go, C#) produce larger components and wake more slowly than Rust, C or Zig. The supported ones’ numbers are published from bench/run.sh like the others, so the trade-off is visible before you choose.

Bring your own

If your language can produce a component that imports and exports what the world declares, it works: there is nothing WasmKeep-specific in the binary format. Generate bindings from crates/wasmkeep-wit/wit with your toolchain’s wit-bindgen (or equivalent), and check the result with wasm-tools component wit your.wasm.

Concepts

What an actor is, what happens to it over its life, and what WasmKeep promises about it. Read What is WasmKeep? first for the why; this page is the how. Exact signatures are in the API reference.

Addresses

An actor’s address is tenant/class/name:

• class: one compiled component, given a name when it is served (--actor chat=chat_room.wasm) or deployed.
• name: 1 to 256 bytes of any text without / or control characters, other than . and ... Every distinct name is a separate actor with its own database. There is no “create” step: the first event to a name creates it.
• tenant: the owner of apps (a team, or a client of the operator; not a customer of your app, which is an actor name). A node started with --actor serves the tenant default; --app acme/shop serves app shop of tenant acme. What an app is, and what is planned for it, is in ADR 0021.

Class and tenant are identifiers: lowercase letters, digits and -, starting with a letter, up to 64 bytes ([a-z][a-z0-9-]{0,63}). So is the app name in --app TENANT/APP: the first wasmkeep deploy of an app refuses a tenant or app name outside it. An app deployed before that rule keeps its name. Only the first deploy checks it: wasmkeep secrets set before that accepts any name, so set secrets after the first deploy, or check the name yourself.

Over HTTP, an actor lives at /actor/<class>/<name>, and anything after the name is the path it sees:

GET /actor/chat/lobby/history?limit=20
    → class "chat", actor "lobby", which sees GET /history?limit=20

Names are URL-decoded (room%201 is the actor room 1). The path an actor sees has its address removed; it reads its own address with actors.self-address (actors::self_address() in Rust, actors.self() in TypeScript), for links to itself or anything keyed by its name.

Who may create an actor. By default anyone who can reach the server can create an actor by requesting a new name. For classes whose actors another actor manages, serve them with --access CLASS=existing: outside requests then reach only actors that exist, and a new name answers 404 without creating anything (no database, no lease, nothing in the bucket). Actors create them with actors.call. --access CLASS=existing:main,admin does the same but lets outside requests create those names (a class’s well-known actors). --access CLASS=internal refuses outside requests altogether (403): only actors and alarms reach the class. See ADR 0010. An app deployed with wasmkeep deploy carries its rules in the deploy instead (wasmkeep deploy --access CLASS=MODE), so every node applies the same ones with the code they protect (Running a cluster). Paths outside /actor/ go to the optional stateless component given to serve, except /_wasmkeep/, which the server keeps for itself (/_wasmkeep/health and /_wasmkeep/ready for load balancers, see Monitoring).

Events

Everything that reaches an actor is an event: an HTTP request, a WebSocket open, message or close, an alarm, or a call from another actor.

• An actor handles one event at a time, in arrival order. Its code never runs concurrently with itself, so it needs no locks.
• Different actors run in parallel. Throughput comes from many actors, not one busy actor: split hot state (one actor per room, not one per app).
• Each actor has a queue of up to 1,024 waiting events. A request that finds it full is answered 429 Too Many Requests. A request received whole (no body, or one that arrived in the first read) whose client closes its connection while it waits is not run, nor an actor’s call whose caller timed out meanwhile (nobody would get the answer, and nothing it would write was acknowledged); one that has started runs to its end.
• A node that cannot take an actor for now (the bucket does not answer, or the actor already ran a newer deploy than this node has, which it is fetching) answers 503 Service Unavailable with Retry-After: 1: retry. A fault a retry will not fix (an unreadable ownership record, a history that cannot be restored) is a 500.
• An address without a mailbox needs resident capacity, including a new, hibernated or failed-over actor. At capacity, admission waits up to 250 ms for an idle actor to hibernate and close its resources. If none becomes free, HTTP and local actors.call receive 503 Service Unavailable with Retry-After: 1 before their event runs. A request for an actor another node owns is relayed without reserving a local slot when this node knows the owner (learned in the last second, or read from the actor’s ownership record); if the record cannot be read in time, the request goes through local admission like any other. A due alarm waits for capacity like a request, then retries with backoff and jitter. Each request that waits evicts at most one idle actor, however often it is woken.
• A WebSocket message waking an evicted actor waits for capacity up to the configured event wall limit. Sustained exhaustion closes it with 1013. Close callbacks remain pending through capacity or queue pressure; pending cleanup continues to count against socket limits until delivered or the service shuts down.
• An event may use 50 ms of CPU (--cpu-budget-ms) and 30 s of wall time, including time waiting on storage, calls and outbound HTTP. Time in host calls and time the OS spends on other threads do not count against the CPU budget. An event over either limit is stopped.
• SQL has its own CPU budget: 2 s per event for all of the event’s statements together (--sql-cpu-budget-ms). Past it, the running statement stops with quota-exceeded and later ones are refused until the event ends.
• An event that traps or is stopped fails: a request gets a 500, a transaction it left open is rolled back, and the actor’s instance is discarded (the next event gets a fresh one, with the same database).

Life cycle

first event ─▶ on-init ─▶ resident ──idle 30 s──▶ hibernated
                  ▲          │  ▲                      │
          new version        │  └──── next event ◀─────┘
          deployed           ▼
                        delete-self ─▶ gone (the next event starts afresh)

• on-init runs before the first event an instance handles with a given component version: once for a new actor, and again after each deploy. Use it to create and migrate tables (CREATE TABLE IF NOT EXISTS, then ALTER TABLE guarded by a version you keep in a table). If it fails, the event fails and on-init is tried again with the next event.
• Resident: the instance stays in memory between events, so in-memory variables survive from one event to the next. Treat them as a cache: they are lost on hibernation, a crash, a deploy or a move to another node.
• Hibernation: after --idle-timeout-secs (30 s) without events, the instance is dropped. Its database, its pending alarm and its WebSockets stay. The next event wakes it in about a millisecond.
• delete-self removes the actor once the current event ends: its database and alarm are gone, and the next event to the address starts a new actor.

Storage

Each actor owns one SQLite database. Nothing else can read or write it.

• exec runs a statement and returns the rows changed; query returns rows. One statement per call, with ? parameters.
• Outside a transaction each statement commits on its own. BEGIN … COMMIT across several calls is allowed within one event; a transaction still open when the event ends is rolled back.
• exec returns once the write is committed locally. exec-durable returns only once it is durable: replicated to the bucket in a cluster, fsynced on a single node. Use it when you are about to tell someone “done” (a payment, an order). It costs about a millisecond more locally, and a bucket round trip in a cluster.
• Inside a transaction, exec-durable takes only the COMMIT (or END): run the transaction’s statements with exec, then exec-durable("COMMIT") makes the whole transaction durable. Any other statement there is refused with not-allowed before it runs, since nothing in a transaction is durable before its commit.
• A database is capped at 1 GB by default; past that, writes fail with quota-exceeded and reads still work.
• Values (text and blobs) are capped at 16 MB, and so is everything one query returns; past that, quota-exceeded. Page through big results.
• A statement runs at most as long as the event may (30 s by default); past that it is stopped with quota-exceeded.
• One event may write about 256 MB (counting every rewrite of a page); past that, writes fail with quota-exceeded until the event ends. Split bulk work across events, for example with an alarm.
• Sandboxed: no ATTACH, no extensions, no pragmas that change state or touch files, no TEMP tables, views or triggers (they would escape the size cap). Virtual tables: full-text search (fts3, fts4, fts5) and rtree only.

Alarms

Each actor has one pending alarm: set-alarm(at) replaces it. When it is due, on-alarm runs as an event.

• Alarms survive hibernation, restarts and moves between nodes. In a cluster, if an actor’s node dies, another node fires its alarm a lease period plus grace after it was due, plus up to 10 s, plus restoring the actor (with the defaults, 24 s late at the median and 36 s at worst with nodes next to their bucket, over twelve hours, measured on one lease per node; ADR 0009). An alarm is as durable as the rest of the event that set it.
• They fire at least once. A failed or trapped on-alarm is retried after 1 s, doubling up to 5 minutes, so make handlers idempotent (record what you did in the database, in the same event). While it is being retried, get-alarm returns the next attempt’s time.
• For several timers, keep them in a table and set the alarm to the earliest; reschedule from on-alarm.

Calls between actors

actors.call("class/name", request) sends an HTTP-shaped request to another actor of the same tenant and waits for the answer.

• The caller’s event stays open meanwhile: other events to the caller wait.
• The callee sees x-wasmkeep-caller: tenant/class/name (the name percent-encoded: room 1 arrives as room%201), which outside clients cannot forge, so it can trust who is calling. A request without it comes from an outside client and should get only what any client may do. In a cluster that includes calls from a node still talking plain HTTP to one that refuses it (a rollout done in the wrong order, see cluster): treat them like any client’s.
• A call that would come back to an actor already waiting in the chain (A calls B calls A) is refused with reentrant instead of deadlocking.
• Request bodies are capped at 16 MB (413 when declared larger; a failed read past it otherwise). Calls between actors cap both ways. A handler may answer without reading the body: the server reads and discards what is left (up to the cap, for at most 30 s), so the client can finish sending it and read the answer. The exception is a client that sent Expect: 100-continue straight to the actor’s node, when the handler read nothing of the body: the answer comes without the 100, and the connection closes after it. A client that waited for the 100 has sent nothing and loses nothing. One that sends the body anyway, without waiting, is not read out: its upload is cut off (a broken pipe or a reset), which can cost it the answer, unless the body was small enough to have arrived whole already. Through another node there is no such case: the node that forwards the request sends the 100 itself, and the body is read and discarded like any other.

WebSockets

A client upgrades a GET on the actor’s path. The host owns the socket, not the instance, so connections stay open while the actor hibernates; the next message wakes it.

• on-websocket-open may refuse a connection (it is closed with code 1008 and the guest’s reason). Until it accepts, the socket is invisible to the actor’s other events: no broadcast reaches a socket it may refuse. It runs before the client sees the upgrade’s 101, so once a client is connected, every later event of the actor (a broadcast, say) sees its socket.
• send queues a message; a client that does not keep up gets backpressure (at 256 messages or 4 MiB queued) and that message is dropped for it.
• connections lists the actor’s sockets, so broadcasting is a loop.
• Limits: messages from clients up to 1 MiB (larger closes the socket with 1009); 1,000 sockets per actor and 50,000 per node (more get 503); a client silent for 25 s is pinged, and closed if it does not answer within 30 s.
• In a cluster, a client may connect to any node: the socket is relayed to the actor’s node. If the actor moves, its sockets close with code 1012 and clients reconnect.

Clusters

Several nodes can share one S3-compatible bucket. Each actor then runs on exactly one node at a time, recorded in the bucket and kept by that node’s own lease; requests that reach another node are forwarded, and each actor’s writes are replicated to the bucket after every event. A node that stops cleanly hands its actors over at once; if a node dies, they move to another node in about 15 to 21 s, with no acknowledged durable write lost. See Running a cluster for how it works, When a node dies, and how it scales for what callers see meanwhile and what adding nodes does, and Deploying for a setup.

Configuration and secrets

An API key, a webhook URL or a feature flag reaches actors through the env interface, never through the component: the same build runs in staging and production. The operator writes one file per class, NAME=value per line:

# /etc/wasmkeep/hub.env, readable only by the server's user (chmod 600)
ADMIN_TOKEN=change-me
WEBHOOK_URL="https://hooks.slack.com/services/..."

and serves the class with it: wasmkeep serve --actor hub=hub.wasm --env hub=/etc/wasmkeep/hub.env (also wasmkeep dev). Actors read it with env::get("ADMIN_TOKEN") (Rust), env.get(...) (TypeScript and Python), wk_env(...) (C), wk.env(...) (Zig) or wasmkeep.Env(...) (Go), and get nothing for a name that is not set.

• Per class. A class sees only its own files; every actor of the class sees the same values. --env for a class that is not served is an error (a typo would otherwise leave a class without its token).
• On the node. --env values are read once at start (restart to change them), are never written to the data directory or the bucket, and never appear in logs: a malformed line is reported by file and line number only. A file anyone on the machine can read is accepted with a warning. In a cluster, give every node the same files, or keep secrets in the bucket instead (below).
• Plain configuration can travel with a deploy. For an app deployed with wasmkeep deploy, values that are not secret (URLs, flags) can go with the version instead: wasmkeep deploy --config hub=hub-config.env. They are stored in the bucket in the clear, reach every node with the code, and change without a restart (the next event runs in a new instance; on-init does not rerun) (Running a cluster).
• Secrets can live in the bucket, sealed. wasmkeep secrets set --app TENANT/APP --key FILE --class CLASS NAME (the value from stdin) seals a secret with a key the nodes hold (wasmkeep serve --secrets-key FILE). It reaches every node within a few seconds, with no deploy or restart, and a rollback of code never brings an old secret back. A bucket reader sees neither names nor values (Secrets in the bucket, ADR 0016).
• Which value wins, by name: the deployed configuration, then the app’s secrets, then the node’s --env (logged, never the value).
• Required values. wasmkeep deploy --requires CLASS=NAME,... makes a class refuse its events (503) while one of those names is not set, rather than run without it.
• The format is what systemd’s EnvironmentFile= reads: # comments, an optional export , one pair of quotes around the value removed, nothing inside interpreted (no escapes, no $VAR), a name set twice keeps the later value. Names are letters, digits and _; values cannot hold control characters other than tab.

The WASI environment (std::env::var, process.env) stays empty.

What an actor cannot do

Read files, open sockets, see the node’s environment variables, or reach another tenant’s actors. Many toolchains import wasi:filesystem and wasi:sockets whether or not the code uses them, so components that do still load, but those interfaces reach nothing: no directory is opened to the guest, and every socket and name lookup is refused (tested from inside a guest). The only way out is outbound HTTP.

Outbound HTTP reaches public addresses only. Loopback, private networks (10/8, 172.16/12, 192.168/16), IPv6 outside global unicast, link-local (including cloud metadata at 169.254.169.254), shared and reserved ranges are refused, also when a host name resolves to them: the host resolves the name, checks the address and connects to that address. A refused address fails with destination-ip-prohibited; a name that resolves only to refused addresses fails like a name that does not resolve (so refusals cannot map internal names). An operator who wants actors to reach an internal API allows its range explicitly: wasmkeep serve --allow-outbound 10.1.0.0/16 (repeatable).

The ports browsers and Cloudflare Workers block are refused as well, whatever the address: the Fetch standard’s list of ports that belong to other protocols, such as mail (25, 465, 587), FTP (21), SSH (22) and IRC (6667). A crafted HTTP request can drive some of those servers, as in sending mail through an SMTP relay. Such a request fails with HTTP-request-denied, before any connection. wasmkeep serve --allow-outbound-port 587 opens one (repeatable).

HTTPS certificates are checked against the host name in the URL and the public certificate authorities (Mozilla’s list, built in). An internal API behind a private CA is trusted with --outbound-ca ca.pem (repeatable), for every actor on the node.

Each actor may have 16 outbound requests open at once (256 per node), and each step (connect, first byte, between bytes) waits at most 60 s (30 s unless the guest asks otherwise).

Running a cluster

Pre-alpha. Ownership, forwarding, replication, failover and deploys work and are tested (simulations in sim/ and multi-node tests), and have run against Cloudflare R2 and OVHcloud Object Storage, with nodes killed under load and nothing lost (measured), but not yet under production load or against AWS S3. Do not trust a cluster with data you care about yet (see the roadmap).

Nodes share one S3-compatible bucket (AWS S3, Cloudflare R2, MinIO, …) that supports conditional writes (If-None-Match / If-Match). There is nothing else to run: no consensus cluster, no coordinator.

For the short version of what callers see when a node stops or dies, and of what adding nodes does, read When a node dies, and how it scales.

How it works

• Each node writes a heartbeat to nodes/<id> with its address, every heartbeat period (at most a third of the lease ttl). It is the node’s lease (ADR 0020): written only if the object is still this process’s last write, so a node finds out at its next write when another writer replaced it, and stops. A node runs actors’ events and is ready only while that lease is valid (10 s after its last write, minus a 2 s safety margin). The lease timings tolerate clock skew between nodes (see wasmkeep_cluster::node_lease, proven in simulation).
• Each actor has an ownership record in the bucket at owners/<tenant>/<class>/<name>, naming the node (and its process) that holds it, with a generation of its own for every hold. A node writes it when it takes the actor (its first event after waking) and when it gives it up (hibernation, a drain, a move, a clean stop): there are no renewals while the actor stays in memory. A request that reaches a node not holding the actor is forwarded once to the node the record names, if that node is live. Responses carry x-wasmkeep-node: <id>. If no connection to that node can be opened, or the connection fails before an answer, the client gets a 503 (“the actor’s node did not answer; retry”) and the request is not sent again: whether or not it was written, since a request that was written may have run there, and the 503 does not say which it was. Only a connection kept open that the node had already closed costs nothing: it is replaced before the request is written. (One exception: a WebSocket upgrade that node provably never read, because it refused the TLS handshake or the upgrade came back unsent on a new TLS connection, is sent once more, the client waiting meanwhile; see “A caller its peer briefly stopped trusting” below.) A release that fails is retried in the background; until it lands, the record keeps naming the node.
• A node that stops cleanly (SIGTERM or Ctrl-C) finishes its in-flight events, then releases its actors: the next request for one of them starts it on another node at once. Last, it writes its heartbeat released, which no reader counts as live, and which frees any record whose release did not land.
• If a node dies, a peer that saw its heartbeat unchanged for a lease period plus a write timeout (a third of the ttl plus 2.7 s, 6 s by default), and once 15 s (the ttl plus a 5 s grace period) have passed since its last write, writes it fenced: no reader counts that node live any more, its actors are free, and the next request for one of them starts it on another node (15 to about 21 s after the node’s last write). If its process was only cut off from the bucket, its next write fails and it stops (a service manager restarts it). A node whose own lease runs out (the bucket unreachable for 8 s) drops every actor at once and takes them again, at new generations, once its lease is valid again. After a bucket outage, a node renews its own lease and waits a lease period plus a write timeout again before it fences anyone, so a node that regained the bucket at about the same time has renewed by then. One cut off for longer is fenced, and restarts. Meanwhile a node that gets a request for one of the dead node’s actors answers 503, or keeps the request until the actor is free: see “When a node dies: what a caller sees” below.
• A deleted heartbeat frees nothing: the actors its node held stay held until a process starts with that id (its start check waits, then it takes them back) or the node is fenced: wasmkeep cluster fence <node> writes a fence even with no object, and the actors are free 15 s later. Node leases, ownership records and markers are never collected; wasmkeep cluster forget <node> removes only a retired node’s heartbeat in the format from before node leases, once it is older than 25 s.

When a node dies: what a caller sees

Between a node’s death and the takeover of its actors (15 to about 21 s after its last lease write, with the defaults), no node can run them. A surviving node that gets a request for one of them does one of two things.

It keeps the request and answers once it took the actor over, for a GET or HEAD without a body and for a WebSocket upgrade that arrive from outside. The caller sees one slow answer (the actor’s own, with x-wasmkeep-node naming the new owner, or the 101) instead of a 503 to retry. The takeover is the ordinary one, and the actor is free at the same moment as without the wait: nothing about leases changes, the wait only saves the caller the retries. A kept request asks for the actor up to a second after this node learns it is free (the requests kept for different actors are spread over that second), which is at its own fence of the dead node or, when another node wrote the fence, at its next heartbeat listing (up to 3 s later). A request is kept when the node has a reason to think the owner is gone or going:

• the owner’s heartbeat left this node’s view (10 s after its last write) and has not moved since;
• the owner is still in view, but a connection to it could not be opened from this node, and no lease write of it since then proves it alive;
• the owner is fenced (by a node or by wasmkeep cluster fence), or stopped and released, and its actors are not free yet.

It answers 503 with Retry-After: 1 at once, as before, for everything else: a request with a body, any other method (POST, PUT, DELETE, …), an actors.call (the calling actor gets its error), a request another node forwarded, and any request over the caps below. Such a 503 still means the request was not run here; nothing with a body is ever kept, so a write is never run late for a caller that gave up. An owner that is alive keeps its actors: a node that renews its lease is never taken from, however slow it is to answer, and a request for one of its actors is sent to it as before. The one exception is a node that is alive but to which this node could not open a connection: requests for its actors are kept for a few seconds, until its next lease write shows it alive (see “An owner that is alive but refuses connections” below).

A kept request ends in one of these ways:

• served by this node, which took the actor over;
• sent, once, to another node that took it over first, and answered by it;
• sent to the owner as usual, because the owner turned out to be alive (it wrote its lease again, or restarted);
• 503 “the actor’s node is not reachable; retry” with Retry-After: 1 when the wait runs out, when this node begins to stop, or when its own lease runs out. Never a 504 from the wait itself;
• dropped, and never run, when its client closes the connection first.

--failover-wait-secs sets how long a request is kept at most. The default comes from the lease timing: the ttl plus grace (15 s), the watch before a fence (6 s), one heartbeat period for a node that did not write the fence itself (3 s) and a second over which the kept requests are spread, 25 s in all. --failover-wait-secs 0 turns the wait off: every request gets the answer it got before, at once. A node keeps at most a quarter of its connection limit and 64 requests per actor; a request over either gets the 503 at once. The connection limit is --max-connections (10,000 by default, so 2,500 kept requests) after serve lowered it to fit the open-file limit: a node left at a limit of 1,024 files keeps far fewer, and one whose quarter comes to zero keeps none (counted as over_cap). Nothing identifies the caller before a request is kept, so anyone who can reach the node can fill the places for a dead node’s actors; later callers then get the 503 at once, as without the wait. A kept request holds its connection and its headers, nothing else: no actor is loaded for it, and no bucket request is made for it while the owner’s heartbeat is only stale. While the owner is suspected after a failed connection, or stopped and released, each kept request reads the actor’s ownership record about once a second, as a client retrying every second would; once the actor looks free it asks for it, about once a second until it has it. Each node decides by itself, nothing travels between nodes or into the bucket, so nodes with different settings, or of versions with and without the wait, work together.

What the wait does not cover, with the defaults:

• The requests that find the death out. A request already sent to the node when it died gets the 503 (“did not answer”), or a 504 after 30 s if the host went silent instead of refusing; it may have run. On each surviving node, a request that arrives while the dead node is still in view is sent to it, and gets that 503 when the connection is refused. Only requests after it are kept. If the host is silent rather than gone (a cut cable, a frozen machine), connections to it are not refused but time out after 5 s: every request that arrives before the first of them timed out waits those 5 s for its 503, and one written into a connection that was already open gets a 504 after 30 s.
• A request that arrives in the first seconds can still get the 503. The takeover can come up to about 21 s after the node’s last write, the surviving node may learn of the fence a heartbeat period later, and after a bucket outage the watch before a fence starts over: a request kept from the very moment of the death can reach its 25 s and be answered 503 after waiting that long. One that arrives once the node left the view waits about 15 s at most.
• Another node took the actor first. The kept request is then forwarded to it, and waits for that node’s restore of the actor under the usual 30 s limit: about 55 s in the worst case, ending in the usual 504.
• An owner that is alive but refuses connections. When this node cannot open a connection to a node that is alive (its listener restarts, a firewall or a one-way network fault, the 5 s connection timeout, or this node running out of sockets), the request that finds it out gets the 503 as before, and the GETs and upgrades that follow for that node’s actors are kept instead of sent, even over connections to it that still work. They are released when a heartbeat listing shows a lease write of that node more than 2 s (twice the allowed clock skew) after the failure: about 2 to 8 s later, since a node writes its lease every 3 s and this node lists every 3 s. They are then sent to the owner as usual, and get its answer or the 503 they would have got at once. Nobody takes the actors. The node is then not suspected again for 10 s; while the fault lasts, the next failed connection after that starts the same again.
• The answer after the wait can still be an error: the node is full, the bucket fails during the takeover, or an access rule refuses the request (a class open only to existing actors answers 404 for one that does not exist, after the wait; a class not reachable from outside is refused 403 at once, before any wait).
• A node that drains takes no actor: a request it would hand to a peer is handed over as before, not kept.
• A GET that writes. A kept GET runs up to 25 s after it was sent, and its client may have retried elsewhere meanwhile. An actor whose GET handlers change state should not rely on the wait: turn it off, or make those handlers safe to repeat.
• A load balancer that hides the client’s departure. A kept request is dropped when its connection closes. A proxy that keeps its connection to the node open after its own client gave up hides that, and the request then runs for nobody: see the front door for the settings that matter.

wasmkeep_failover_waits_total counts the kept requests and how each wait ended (monitoring).

Replication

Each actor’s database follows it from node to node (ADR 0007):

• After every event, the owner uploads what the event committed as an LTX file under actors/<tenant>/<class>/<name>/wal/, and a snapshot every 64 uploads. A node that takes an actor over rebuilds its database from the bucket first, unless its local copy is already current.
• An upload the bucket leaves unanswered is given up and sent again: an attempt after 5 s plus a second per megabyte (the next one after twice that), three attempts at most, the whole upload within 20 s plus that second per megabyte. This holds for a write’s file, for an actor’s first snapshot and for the snapshot after a delete. An attempt is one call to the bucket client, which sends the request again within it when the bucket answers with an error (a 5xx, 429 or 408) or the request could not be sent. The client also ends any single request after 30 s by default, and does not send an upload again after that or after a broken connection: the attempt fails. So a request lasts 30 s at most whatever the object’s size, and an attempt as long as the bounds above allow: the second per megabyte gives a first attempt’s request more time up to 25 MB and no further, and an object the bucket cannot take within 30 s in one request is not uploaded.
• A periodic snapshot is an optimisation, and the actor is held while it uploads, so it gets one attempt, which the node does not send again (the client may, within the attempt, after an error answer): given up after 5 s plus a second per megabyte, 10 s at most (6 s under a megabyte), a third of the 30 s a queued caller waits for an actor (the default wall-clock limit, for actors.call, and the wait for a forwarded request’s answer). One that fails costs no write and fails no event (the files already uploaded are the history). It is tried again once the actor has made one more write and 30 s have passed, then, while it keeps failing, 8 writes later, then every 64. With a bucket that leaves every snapshot upload hanging, the worst case is 10 s of hold per try and three tries in any 64 writes (30 s), never two within 30 s, then one try per 64 writes (10 s); an actor that only reads does not try again. The node remembers these waits while an idle actor is out of its memory, so waking it does not try again either; a node that takes the actor over from another, or one that restarted, knows of no wait and tries at the actor’s first event there when 64 writes or more have no snapshot. A database the bucket cannot take within those 10 s gets no periodic snapshot: each try is one bounded attempt, and a restore replays every write since the last snapshot. Uploads not acknowledged are counted by wasmkeep_snapshot_upload_failures_total (monitoring).
• storage.exec commits locally; its upload follows when the event ends. If the node dies in between, that write can be lost. storage.exec-durable returns only once the write is in the bucket, uploaded while the node still held the actor’s lease: an acknowledged durable write survives any node failure. The bucket is its only durable copy: the node does not fsync it locally as well, and trusts its local file again only after checking it against the bucket. So start a node again on the same bucket and prefix: after a crash or a power loss its local files may lack writes the bucket has.
• If an upload fails, the node gives the actor up; the next request restores it from the bucket, possibly on another node.
• A restore’s reads are bounded together. The node lists the actor’s history and reads its last snapshot and the files after it, eight at a time; every request waiting for the actor waits behind that. A read the bucket leaves unanswered for 5 s, or that is slower than 5 s plus a second per megabyte, is sent again, and so is one the bucket refuses with an error a resend can get past (a 5xx or a broken connection, after the bucket client’s own retries), half a second later, then a second: three tries per file. While it pauses, a read gives up its place among the node’s reads, and waits for one again before it is resent. The reads of one restore have 30 s together, plus a second per megabyte of what it reads and a second per eight files: 39 s for 63 small files and a snapshot under a megabyte, about 17 minutes more for a database of 1 GiB. That counts the time the restore waits for the bucket to give it the next file. It does not count the listing (given up only when the bucket sends no key for 5 s, so a long history lists for as long as its pages take), nor work on the local disk (never interrupted), nor the time a read waits for its turn among the node’s other restores (they share 32 reads and 256 MiB at once): while that queue lasts, a caller can wait longer than the deadline. Nor does it count what other restores add to a read’s time. A read’s own limit counts every megabyte being read on the node at that moment, other restores’ included, since they share one link: after a node dies, the node that takes over 32 actors of 8 MiB each gives every read 261 s (5 s plus 256), while each restore, by its own size, has 39 s. So for each file, the seconds that other restores’ megabytes added to the limits of its read’s tries are left out of the count, up to the time the restore waited for that file. A restore is therefore never ended because of what other restores are reading: what is counted for a file is at most what its read is allowed with nothing else read on the node. A restore alone on the node gets nothing more than its 30 s and its own seconds, and slow reads that add up end it. At worst a restore’s waits for the bucket last its own time plus 768 s per file (three tries, each given at most 256 s by the 256 MiB the node reads at once): only while other restores keep the node at that limit and each of its reads is tried three times to the end of its time, which is what each read’s own limit allows. A bucket that never answers ends a restore sooner, whatever else is read: a read is given up after three silences of 5 s. A restore that fails or runs out of time before its database is complete changes nothing on the node: the database is built beside the local file and put in its place only when complete. If putting it in place fails (an error of the local disk), the file that was there has already been moved to <name>.db.replaced, and the node is left with no file for the actor or with the new one unmarked. Either way the next restore starts again from a fresh listing and reads the bucket: a node uses a local file as it is only after checking it against the bucket’s history. The request that waited gets a 503 with Retry-After, as for a bucket that does not answer (a history that cannot be restored without losing data stays an error, 500), the node gives the actor up so another may try, and wasmkeep_restore_failures_total counts it (monitoring).
• storage.delete-self is recorded in the bucket, so a deleted actor stays deleted wherever it runs next. Like a plain exec, it is replicated when the event ends: if replication fails at that moment, the delete can be lost. Once the delete is recorded, the actor’s history in the bucket is erased (see History in the bucket).
• Inside a transaction, exec-durable takes only the COMMIT: it returns once the whole transaction is in the bucket. Any other statement there is refused before it runs, since nothing is durable before COMMIT.
• Pending alarms are listed in the bucket (alarms/, by due time). Every node reads the overdue part of that list every 10 s and fires alarms overdue by more than a lease period plus grace, so an alarm whose node died still fires (ADR 0009).
• When a node rebuilds a database from the bucket, the local file it replaces is kept as <name>.db.replaced rather than deleted. Only one generation is kept: a second replacement of the same actor’s file overwrites the first .replaced copy.

Joining a cluster with existing data: the first node to touch an actor decides its history when the bucket is empty. Start the node that holds the existing databases first, and let it serve each actor once, before adding nodes. A database replaced by mistake can be recovered from .replaced, until a later replacement of that actor’s file overwrites it, so copy it elsewhere before then.

Try it locally

Any S3-compatible store with conditional writes works. Each node proves it at start, before it writes or takes anything: on a throwaway object under probe/, a create over an existing object and a replace with a stale version must both be refused. A store that accepts them (it ignores the conditions) would let two nodes own one actor, so the node logs why, stays not ready and wasmkeep serve exits. A bucket that does not answer yet is tried again, as at any start. Cloudflare R2 and OVHcloud Object Storage pass it (tested); AWS S3 and MinIO document both conditions. For a quick local try, moto’s S3 server is enough (it keeps everything in memory):

pip install "moto[server]" && moto_server -p 5055 &
curl -X PUT -H "Content-Length: 0" http://127.0.0.1:5055/wasmkeep-demo

export AWS_ACCESS_KEY_ID=test AWS_SECRET_ACCESS_KEY=test \
  AWS_REGION=us-east-1 AWS_ENDPOINT=http://127.0.0.1:5055 AWS_ALLOW_HTTP=true
W=examples/counter/target/wasm32-wasip2/release/counter.wasm

wasmkeep serve --actor counter=$W --data-dir .wasmkeep/a --listen 127.0.0.1:8081 \
  --node-id a --advertise http://127.0.0.1:8081 --bucket s3://wasmkeep-demo \
  --cluster-listen 127.0.0.1:7481 --cluster-advertise https://127.0.0.1:7481 &
wasmkeep serve --actor counter=$W --data-dir .wasmkeep/b --listen 127.0.0.1:8082 \
  --node-id b --advertise http://127.0.0.1:8082 --bucket s3://wasmkeep-demo \
  --cluster-listen 127.0.0.1:7482 --cluster-advertise https://127.0.0.1:7482 &

curl -i -X POST localhost:8081/actor/counter/alice   # 1, x-wasmkeep-node: a
curl -i -X POST localhost:8082/actor/counter/alice   # 2, forwarded to a

Flags

Node ids must be unique. A node may take back its own lease at once (that is how a restarted node recovers its actors quickly), so two processes sharing an id would be treated as one node and could both run the same actor. A node therefore checks its id when it starts, before its first heartbeat and before any lease. It reads nodes/<id> every half period for the lease’s ttl plus grace (15 s by default), the time any other node waits before taking a dead node’s actors, even when there is no such object (a process whose object was deleted while it ran is given time to show itself). A process still running rewrites it: the new one then exits with an error naming the other’s address, having taken no actor. A peer’s fence or the old process’s own release of it does not count as running: that process lapsed once the object’s last live write is those 15 s old, and the new one goes on then. A heartbeat in the format from before node leases (ADR 0020) keeps the new one waiting until it is older than its 10 s horizon plus those 15 s, since such a process renews its per-actor leases on a task of its own. Unchanged until the end: the previous process is gone, and the new one writes its first heartbeat, conditioned on the version it read last. So:

• A first start, or a restart while the previous heartbeat is still live, waits those 15 s before wasmkeep serve listens (it logs “checking that no other process runs with this node id”). After a clean stop or a fence of the previous process it waits only until that process’s last live write is 15 s old: about 1.5 s when that has already passed.
• A restart while the old process still runs, or still finishes its graceful shutdown, is refused: let the service manager retry it.
• If the bucket cannot be read, the node keeps retrying and does not start serving.
• Two processes started within a moment of each other can both pass the check, but only one of their first writes can land: each is conditioned on the object the check read last. The other process’s write fails; if it was already running, its next write fails the same way. That one drops its leases at once, takes no new lease, stops heartbeating, and wasmkeep serve shuts down and exits with an error; the other goes on. The listing of the heartbeats still catches a heartbeat with another token, or none (a node of an older version, which writes without a condition). Until then both may run the same actor, so still start a node only once the previous process with its id has stopped.
• Deleting a running node’s heartbeat makes that node exit at its next write, for the same reason.
• wasmkeep serve exits with an error when its node id is refused at start too. A service manager that restarts it at once (Restart=always, a Kubernetes rollout bringing a new pod up before the old one stops, both with one fixed --node-id) retries until the other process has stopped.

When replacing a node, stop the old process before starting the new one with the same id.

WebSockets. A client may open a socket on any node: a node that does not own the actor relays the upgrade to the owner and joins the two connections, so everyone in a room shares one actor wherever they connected. If the actor later moves, its sockets close with code 1012; clients reconnect. A room that hibernates keeps its sockets open on its node but releases its ownership record, so a request that reaches it through another node wakes it there; its sockets on the first node then close with 1012 when one of them next sends a message, or at the node’s owner sweep (once a minute), so a client that only listens is closed too, a minute later at most. Sockets also close with 1012 when their node drains and another node can take its actors (see “Draining a node” below), and when its lease runs out (no write reached the bucket for ttl minus safety, 8 s with the defaults): before any other node can take its actors, and without waiting for the bucket to answer again. A socket closed because its node lost the actor gets no on-websocket-close there; a room keeping a list of who is in it should rebuild it from connections(). Such a close can reach every client of a node at once (a lease that runs out closes them all): clients should reconnect after a random delay on 1012, backing off on each failure, so they do not all reconnect in the same instant. A client that does not answer a close within 5 s is dropped. If the owner does not answer the upgrade, the client gets a 503 and may open the socket again; over mutual TLS between nodes, that 503 can take up to four heartbeat periods (12 s with the defaults) when the owner refuses the relaying node’s certificate (see “A caller its peer briefly stopped trusting” below).

Calls between nodes. actors.call reaches an actor wherever it runs. Nodes prove to each other that a call comes from the cluster with a signature: HMAC-SHA256, keyed with a random secret the first node stores at cluster/secret in the bucket, over the node the call is for, the method, path, body, calling actor, chain of waiting actors, a timestamp and a nonce unique to each send (valid for a minute, and accepted once). Only a signed call’s caller (x-wasmkeep-caller) and chain are believed, so a client cannot pass for an actor or reach an internal class, and a call cycle across nodes is refused instead of deadlocking. The secret itself never travels; a malformed one in the bucket stops calls between nodes rather than weakening them.

Credentials, region and endpoint come from the standard AWS_* environment variables.

Encryption between nodes. Nodes talk to each other over mutual TLS, on a cluster listener every node needs: a node started with --node-id and without one refuses to start, unless given --cluster-plaintext accept. In that mode, nodes without a listener talk over plain HTTP: a call between actors is signed, its body included, but nothing is encrypted and responses are not signed. Keep such nodes on a private network (a node warns at start when --advertise is a public address). A cluster listener on every node:

wasmkeep serve ... --node-id a --advertise http://10.0.0.11:8080 \
  --cluster-listen 0.0.0.0:7443 --cluster-advertise https://10.0.0.11:7443

Forwarded requests, relayed WebSockets and actors.call between two nodes that both have one then travel over mutual TLS 1.3 on that port, in both directions. There is nothing to provision: each node makes a key and a self-signed certificate in memory at start (a restart is a new key) and publishes the certificate’s fingerprint in its heartbeat; a node accepts a peer only if the peer’s certificate matches a live heartbeat in the bucket (ADR 0011). So whoever can write the bucket controls the cluster, as with the call secret.

• Both flags go together, and only with --node-id; --cluster-advertise must be https://. The cluster port serves actor paths and the health and readiness checks, nothing else, and only to other nodes.

• Firewall: only the other nodes need to reach the cluster port; do not expose it to clients or put it behind the load balancer. --listen still serves clients as before.

• A node with a listener reports not ready (/_wasmkeep/ready) for a few seconds after start, until its fingerprint has been published long enough for peers to have seen it.

• Rolling it out on a cluster that ran in accept mode: restart nodes one at a time with the flags, keeping --cluster-plaintext accept. Nodes reach peers that have a listener over TLS and the others over plain HTTP, and still accept plain signed calls on --listen, so a mixed cluster keeps working. A peer that publishes a TLS address is reached over TLS or not at all (a 503 the caller retries), never in plain text after a TLS failure. Until every node refuses plain HTTP (below), keep the public port’s node traffic private too.

• Refusing plain HTTP (--cluster-plaintext refuse, the default): a node’s call then counts only on the cluster port. A signed call arriving on --listen is treated as a client’s (access rules apply, no calling actor, internal classes refused), and peers without a cluster listener are not used: requests and WebSockets for actors they own get a 503 from this node and actors.call to them fails (it logs which peers it skips, once a minute). Clients are served on --listen as before. A refusing node next to plain peers still reports ready (/_wasmkeep/ready): the load balancer will not take it out, so do not get into that state (below).

• Moving a cluster from accept to refuse: first give every node a cluster listener (restart them one at a time with the two flags and --cluster-plaintext accept, and check with wasmkeep cluster status that every live node shows a cluster address and a pin, and no warning about plain nodes: no node may still be plain). Then restart them one at a time again, without --cluster-plaintext accept. Doing it the other way round breaks the cluster while it is mixed: a refusing node cannot reach a plain node’s actors (503), and a plain node’s calls to it count as a client’s (internal classes refuse them, callees see no caller). Going back is the reverse: add accept everywhere before removing a listener.

• The cluster port has its own connection cap (1,024, less when the open-files limit is low; its share is counted before the client connections and resident actors are sized).

• What it costs: on a pooled connection, TLS adds about 9 to 12 µs to a forwarded request and 15 to 21 µs to an actors.call between nodes, on top of a hop of about 40 µs over loopback (measured). Connections between nodes are kept alive, so full handshakes are rare.

• How fast a removed node loses access: revoke a node’s process with wasmkeep cluster fence <node> --bucket ... (ADR 0020): it stops at its next heartbeat write, and its actors are free 15 s after its last write. A service manager that restarts it brings it back as a new process; to remove a machine, stop and disable its service too, and rotate its bucket credentials (security, “Revocation”). A node stops trusting a peer whose heartbeat is fenced, released, deleted or names a new key at its next listing of the heartbeats (one per heartbeat, 3 s by default), and one whose heartbeat stops at its expiry: three heartbeat periods plus the allowed clock skew, 10 s with the defaults, also when the node cannot list the bucket. Relayed WebSockets are cut within 500 ms after that (security, “Revocation”; timed in crates/wasmkeep-server/tests/revocation.rs).

• A caller its peer briefly stopped trusting: if a peer refuses a node’s certificate at the handshake (it saw that node’s heartbeat expire, as a slow bucket write can make it), an actors.call is sent once more after the caller’s next heartbeat reached the bucket and two more periods passed (12 s at most with the defaults; it is not sent again if that heartbeat does not land within two periods); the peer never read the first one. Then it fails with “refused this node’s certificate”. A relayed WebSocket upgrade, which has no body, is sent once more the same way, the client waiting meanwhile; if it is refused again, or the heartbeat does not land, the client gets a 503 asking it to retry. A forwarded request gets that 503 at once: its body is not kept. A call or an upgrade that came back unsent on a new TLS connection to the peer is treated as one refused at the handshake (a refusal can close the connection before the request is written; whatever closed it, the peer read nothing). Nothing else is sent again: not a request whose connection could not be opened, and not one written into a connection that then failed, which the peer may have read.

  The price is the client’s wait when the peer keeps refusing. The upgrade’s 503 then comes after up to four heartbeat periods if the heartbeat lands and the second try is refused too (12 s at most with the default 3 s period, 9 s or less when the next heartbeat lands on time: up to one period for it, then two), and after two periods (6 s) if the heartbeat does not land; a call waits as long for its failure. Answering at once would be faster in that case only: a peer that refuses a live node does so for a moment, and then the wait ends in the open socket or the call’s answer, where a 503 at once would leave the client to open the socket again and fail the call. A client that gives up on a WebSocket handshake sooner than that sees its own timeout instead.

An upgrade of WasmKeep itself, including one that changes the format of signed calls, upgrades every node together (below).

Deploying

Instead of giving every node --actor CLASS=PATH, deploy once to the bucket and let nodes follow:

wasmkeep deploy --app acme/shop --bucket s3://wasmkeep-demo \
  --actor counter=counter.wasm --actor chat=chat_room.wasm
wasmkeep serve --app acme/shop --bucket s3://wasmkeep-demo --node-id a ...
wasmkeep deploy --app acme/shop --bucket s3://wasmkeep-demo --rollback

• One app per tenant per bucket prefix, until each app owns its actors’ state (planned for 0.4.0, ADR 0021). Actors are keyed by tenant, class and name, so two apps of one tenant on the same prefix would share actors. The first deploy of an app refuses when another app of its tenant is deployed on the prefix (it has apps/<tenant>/<app>/current), naming that app: deploy the new one on another prefix (s3://bucket/admin), or under another tenant if another owner runs it (--app globex/admin). Another prefix is a separate cluster with its own cluster secret: the app starts there with no data, its secrets are set again, and it cannot be moved to another prefix later (its secrets are sealed to the prefix). A node also refuses to start when a live node on its prefix serves another app of its tenant. Two apps of one tenant deployed on one prefix before this check are left as they were (in a cluster, still not running at the same time), and deploy and serve warn about them by name; a further one is refused while any of them has a current. The first deploy needs list access to <prefix>apps/<tenant>/ (a listing that fails refuses); two first deploys at the same moment can both pass it. There is no app delete yet. To free the tenant from an app nobody runs, stop its nodes and delete its current, which keeps its versions and secrets:

  aws s3 rm s3://wasmkeep-demo/apps/acme/legacy/current
  # or remove the app entirely (its versions, current and secrets):
  aws s3 rm --recursive s3://wasmkeep-demo/apps/acme/legacy/
  

  Its actors’ data stays under actors/<tenant>/, owners/<tenant>/, leases/<tenant>/ and alarms/<tenant>/, and the next app of the tenant finds the actors of any class with the same name.

• The tenant and app names of a new app match [a-z][a-z0-9-]{0,63}; an app deployed earlier keeps its name.

• deploy compiles and checks every component before uploading anything; a broken one is refused and the current version keeps running.

• Nodes check for a new version every 3 s and switch each actor at its next event. No actor is dropped; on-init runs once per version.

• An actor never runs older code than it already ran: a node that has not caught up yet refuses it (the client retries) and updates at once.

• Rollback makes the previous version current again (as a new deploy), with its access rules and configuration.

• Old versions are removed by wasmkeep deploy itself, after every deploy and rollback (ADR 0012): it keeps the current version and the five before it (as many rollbacks as you can make in a row), and deletes older ones once they stopped being current more than --version-retention-secs ago (default 3600; 0 at once; measured with the bucket’s timestamps against the clock of the machine running the command). Only versions in the current one’s line of previous links are candidates, oldest first, and the pass stops if current changes while it runs. Never collected: a version a concurrent deploy is still writing, one a failed deploy left, the versions a rollback stepped back from, and a version whose manifest this WasmKeep cannot read. Those stay in the bucket (they are small).

Updating the stateless component

A deploy carries actor classes, with their access rules, configuration and required names. It does not carry the stateless component, the positional argument of wasmkeep serve that answers every path outside /actor/ and /_wasmkeep/: that one is a file on each node, read once when the node starts, also with --app:

wasmkeep serve site.wasm --app acme/shop --bucket s3://wasmkeep-demo --node-id a ...

Nothing in the bucket names it, so wasmkeep deploy does not change it, --rollback does not bring an older one back, and cluster status does not show which one a node runs. To update it today:

1. Copy the new file to every node, at the path its serve command names.
2. Restart the nodes one at a time (systemctl restart wasmkeep), waiting for each to be listed live by wasmkeep cluster status and to answer /_wasmkeep/ready before the next.

What happens meanwhile:

• The restarting node makes a clean stop: requests under way get up to 10 s to finish, its actors are given up and start on the other nodes at their next request, and its WebSockets close with 1012. The load balancer’s check of /_wasmkeep/ready takes it out. At its start it checks its node id first, which takes what is left of 15 s since it stopped (Flags).
• The other nodes keep answering, each with the file it was started with. Until the last node has restarted, two versions of the component answer side by side, and a client may get either one from one request to the next. Keep two consecutive versions compatible with each other and with the deployed actors.
• A file that does not compile or instantiate makes serve exit at start. Restarting one node first shows it while the others still run the previous file.
• Going back is the same procedure with the previous file.

The stateless component has no actors.call: it is linked against WASI and wasi:http only. It reaches actors as any client does, over HTTP, under the same outbound rules as actors (public addresses only, unless the node allows a range with --allow-outbound).

Access rules and configuration

Access rules (ADR 0010) and plain configuration are part of the deployed version (ADR 0013): every node applies the same ones, at the same moment as the code they go with, and a node started without flags no longer serves a restricted class open.

wasmkeep deploy --app acme/shop --bucket s3://wasmkeep-demo \
  --access monitor=existing --access hub=existing:main \
  --config hub=hub-config.env

• A deploy changes what it names and keeps the rest. Without --actor, the current classes are kept (copied inside the bucket, not uploaded again). Classes not named keep their rule and configuration, so a CI job deploying new code does not drop rules it never knew about. --access CLASS=open removes a rule, --config CLASS= a class’s configuration. deploy prints what changed (class code, rules, the names of configuration values) and warns when a class removed with a rule leaves an added one open (a rename).

• --config is stored in the clear, as parsed (no comments). Keep secrets in serve --env files on each node: they win over the deployed configuration by name (each overlap is logged by name, never value). deploy warns about names that look like secrets (TOKEN, KEY, PASSWORD…) and URLs with a password, without printing them.

• Rollback brings back the previous version’s rules too. One that would make a class more open (existing to open, say) is refused unless --allow-reopen is given.

• New values drop the class’s resident instances at their next event (the database is kept, on-init does not rerun); rules alone do not. A class made internal closes its open sockets (1008) at their next message.

• A node that reads a version it cannot apply keeps the previous one and reports not ready (/_wasmkeep/ready) until it can.

• --access with --app is refused once rules have been deployed (the start-up error gives the wasmkeep deploy command). Before that it still works, so a cluster started with --access moves in three steps:

  1. upgrade every node, flags unchanged;
  2. deploy the rules once: wasmkeep deploy --app acme/shop --bucket ... --access monitor=existing (nodes switch to them and log that --access is now ignored);
  3. remove --access from the nodes’ command lines.

  A node following an app that never deployed rules, started without --access, logs that every class is open.

Secrets in the bucket

Secrets can live in the bucket, sealed with a key the nodes hold, and change without a deploy or a restart (ADR 0016):

wasmkeep secrets new-key --out secrets.key        # mode 0600; keep it with the bucket credentials
wasmkeep serve ... --app acme/shop --secrets-key secrets.key   # every node
wasmkeep secrets set --bucket s3://acme-wasmkeep/prod --app acme/shop --key secrets.key \
  --class hub ADMIN_TOKEN < token.txt            # one trailing newline removed
wasmkeep secrets list --bucket s3://acme-wasmkeep/prod --app acme/shop --key secrets.key   # names only

• Reaching the nodes. Each node reads the app’s secrets at start and checks for a change every 3 s. A class whose values changed drops its resident instances at their next event (the database is kept, on-init does not rerun); an event under way keeps the values it started with.
• Which value wins, by name: the deployed configuration, then the secrets, then the node’s --env (logged at warn, never the value). --env stays a node-local break-glass.
• Several at once: set --file FILE (entries <class> <name> <length>\n<value>\n, any value, newlines included) and unset --class CLASS NAME... make one change each, so nodes never serve half of it.
• A node that cannot open the current secrets refuses the app’s events with 503 and Retry-After, hands its actors to a node that can, takes none, and reports not ready. It blocks at once on a set it cannot open (no key, another key, an altered object), and after 30 s on one it cannot reach (the object missing or unreadable, current unreadable). Past those windows (one poll, 3 s, or the 30 s) it never runs an actor with an older or empty set. cluster status then shows it draining or drained without a drain marker; undrain does not apply. When no node can open them, every event for the app is 503 until one can. Socket close events on a blocked node do not run. Without --secrets-key (and without --no-secrets), a node refuses to start once the app has secrets.
• Restoring the bucket from a backup can put current below what the running nodes opened; they refuse it (and block after 30 s) until restarted: restart the nodes after a restore (Back up and restore a cluster’s bucket has the whole procedure). Deleting current by hand leaves running nodes with the set they have and restarted ones with none.
• Older nodes. set, unset and reset refuse, without --force, while a live node of the app does not read secrets (an older version, or one run with --no-secrets): it would serve them as unset. Nodes say so on heartbeat line 10; the check counts a heartbeat as live with the default timings, as cluster status does.
• Changing the key. Give every node the new key as well (a second --secrets-key, one node restarted at a time), run wasmkeep secrets rotate-key --key old.key --new-key new.key --app acme/shop --nodes-have-new-key (or --all), then remove the old key from the nodes. The flag is required: a node without the new key cannot open the re-sealed set and answers 503 for the app until it gets the key, and no heartbeat says which keys a node holds. set and unset given two keys seal with the first, and take the same flag when that one is not the key the current set is sealed with. After a key compromise, change the values too: the old key opens the objects it sealed.
• Losing the key. Give every node the new key, then wasmkeep secrets reset --new-key new.key --from FILE --nodes-have-new-key writes a whole set without opening the current one.
• Moving from --env. Upgrade every node, give them the key, set each name, then remove the names from the --env files and restart the nodes one at a time.
• Required values. wasmkeep deploy --requires hub=ADMIN_TOKEN,WEBHOOK_URL makes those names part of the version: while one is not set (by the configuration, the secrets or a node’s --env), the class refuses its requests with 503 (an alarm waits, a socket closes with 1011) instead of running without it, and the node logs the class and the missing names once (again if they go missing again), never a value. A value set to the empty string counts as set. Once the value is set, requests run at once; a waiting alarm runs within the alarm retry ceiling (5 minutes by default); an idle socket closes at its next message. Kept by later deploys; --requires hub= removes them, and a rollback brings back the previous version’s. Required names alone do not end ADR 0013’s migration (only rules or configuration do).

Checking a cluster

wasmkeep cluster status reads the bucket alone, so it works from any machine with read access to the bucket, whether or not any node answers:

wasmkeep cluster status --bucket s3://wasmkeep-demo
wasmkeep cluster status --bucket s3://wasmkeep-demo --json   # for scripts

Cluster s3://wasmkeep-demo (live: heartbeat within 10.0s)

NODE  LIVE  AGE    STATE    VERSION  APP        ADDRESS        CLUSTER ADDRESS  TLS
a     yes   979ms  serving  0.3.0    acme/shop  http://a:8080  https://a:7443   pinned
b     yes   2.2s   serving  0.3.0    acme/shop  http://b:8080  -                plain
c     no    14m    serving  0.3.0    acme/shop  http://c:8080  https://c:7443   pinned

APP        DEPLOYED VERSION  SEQ
acme/shop  9cf4de9152d8dd29  1

2 of 3 nodes live; TLS: 1 with a cluster listener, 1 plain; version: 0.3.0
warning: mixed plain and TLS nodes: b still plain while a use the cluster listener; give every node a cluster listener, then drop --cluster-plaintext accept (ADR 0011)

• Nodes come from their heartbeats (nodes/<id>): the WasmKeep version each runs, the app it serves, its advertised and cluster addresses, and whether it publishes a TLS pin. State is whether it takes actors: serving, or, while it is drained (ADR 0014), draining, drained (no actors or WebSockets left) or stuck (actors or WebSockets left and no node of the same app to hand them to); - from a node that predates draining. A node is live if its heartbeat was written within 10 s (three heartbeats plus the allowed clock skew), the rule the nodes themselves use with the default settings. Ages are measured on the clock of the machine running the command, so keep it in sync too. A heartbeat written fenced (a peer took its lease) or released (a clean stop) is shown as no (fenced) or no (released) whatever its age, and as lease in --json. A stopped or crashed node stays listed, not live, until its heartbeat object is removed from the bucket.
• Apps come from apps/<tenant>/<app>/current: the deployed version and its seq. Which version each node has picked up is not in the bucket (nodes follow within a few seconds).
• Warnings: no live node; live nodes still plain while others use the cluster listener (the state ADR 0011 says not to stay in, and not to refuse plain HTTP from); live nodes on different WasmKeep versions (see Upgrading WasmKeep); a draining node that is stuck; a node lease dated ahead of this clock by more than the allowed skew (ahead_ms in --json). Such a lease never goes stale, so no node fences it and the actors it holds stay held until that date: fix the clock that is wrong (that node’s, or this machine’s if it is behind). wasmkeep cluster fence stops the node, but the fence keeps the date, so its actors free 15 s after it; a fenced or released node whose date is still ahead is warned about too, until then. The nodes log the same, at most once a minute.
• The exit code is 1 when no node is live (an empty prefix included), 0 otherwise, warnings or not; a bucket it cannot read is an error.

Draining a node

To take a node out of service without stopping it first (to replace the machine, resize it, or upgrade it), drain it:

wasmkeep cluster drain node-b --bucket s3://wasmkeep-demo
wasmkeep cluster status --bucket s3://wasmkeep-demo   # wait for STATE drained

While it drains, the node (ADR 0014):

• claims no actor. A request, a call or a WebSocket upgrade for an actor it does not hold goes to another live node serving the same app, which takes it;
• hands over the actors it holds, up to 16 at a time:
  • their WebSockets close with 1012 (clients reconnect and land on the new owner);
  • their queued events finish and are uploaded;
  • the lease is released, and each pending alarm moves to the new owner’s timer, so it fires on time even after this node stops;
• closes the WebSockets of rooms that hibernated with sockets open with 1012. Their clients reconnect through another node; their on-websocket-close events do not run, since the draining node does not take the room back to run them;
• reports drained once it holds nothing, no WebSocket included, and keeps running and forwarding until you stop it. You can then stop it with nothing left to hand over. A WebSocket this node relays to another node is not counted, and is cut when the node stops.

A node with no other node to hand to (alone, or every other node draining or older than this feature) keeps serving its actors and reports stuck. cluster status warns about it. Start another node, or undrain one.

wasmkeep cluster undrain node-b --bucket s3://wasmkeep-demo   # take actors again

Both refuse an id that no live node has (as cluster status counts it): a drain request for a mistyped id would stay in the bucket until an undrain with the same typo. undrain still withdraws a request no node answers to. --force writes or deletes the request anyway (for a node about to start).

During each actor’s hand-over its requests may get a 503 with Retry-After: 1 for a moment. A peer that fails to take an event is skipped for about 10 s. A request that hits a dead peer before that gets its one error.

Moving one actor

wasmkeep actor move acme/room/lobby --to node-c --bucket s3://acme-wasmkeep/prod

The target must be a live node: an id no live node has is refused without --force, and a target that is draining, or too old to take moved actors, gets a warning (the request is written; the owner keeps the actor), as does one that serves another tenant.

Within a few seconds the actor’s owner hands it to node-c (ADR 0017):

1. its WebSockets close with 1012 and their close events run;
2. its queued events finish and are uploaded;
3. the lease is released, and from that moment every event for it on the old owner (requests, calls, alarms, reconnecting sockets) goes to node-c;
4. node-c takes it: it acquires the lease, restores the actor and arms its alarm.

If no node holds the actor, node-c takes it itself. A name that was never an actor is not created.

• The target must be live, serve the same app, not be draining or blocked, and run a version that takes actors (heartbeat line 11, take). Otherwise the request waits, tried on every heartbeat, until it expires.
• Not a pin. Once idle, the actor hibernates on node-c, and its next event claims it wherever that event lands. A draining node does not act on move requests; its drain moves everything.
• Retries. A move that cannot finish (a socket that will not close, a target that does not answer) is tried again on every heartbeat until it works or the request expires (5 minutes).

Upgrading WasmKeep

Upgrading the wasmkeep binary itself (not your application: that is Deploying, which rolls from node to node safely): stop every node, upgrade them all, start them all. WasmKeep does not yet support nodes of different versions in one cluster, even for the length of a rolling restart. The cluster serves nothing between the last stop and the first start, so plan a short window.

1. wasmkeep cluster status --bucket ...: note the nodes it lists as live.
2. Stop every node cleanly (systemctl stop wasmkeep, SIGTERM). Each finishes its requests and releases its actors; a clean stop takes up to about 29 s.
3. Wait until cluster status exits with 1 (no live node), about 10 s after the last stop. A node still listed as live is still running the old binary: stop it first. This check does not depend on version strings.
4. Install the same new binary on every node.
5. Start every node, then run cluster status again: every node live, one version in the summary, no warning.

Going back to the previous binary is the same procedure. Builds made from source report the version of the release they follow (0.3.0 for any build since it), so the version column cannot tell two of them apart: step 3 is what proves no old node is left.

What goes wrong if old and new nodes run together (each is a reason for the rule; there may be others nobody has listed):

• Calls between nodes: the format of signed calls can change between versions. Old and new nodes refuse each other’s calls (logged as “bad, expired or misaddressed signature (… a node of an older version?)”), so calls to internal classes on the other kind fail, and a call cycle across them ends in a timeout instead of reentrant.
• Formats in the bucket: a version can change how heartbeats, ownership records or alarm index entries are written. A node of another version may skip what it cannot read, so it may not see its peers’ ownership or alarms: actors refused, alarms late or missed.
• No other guard: the deploy seq (ADR 0008) keeps an actor from running older application code, not from an older WasmKeep. Nothing compares WasmKeep versions between nodes; the version in each heartbeat is only reported, by cluster status.

Rolling upgrades of WasmKeep across mixed versions are planned (ROADMAP).

History in the bucket

Every write an actor commits becomes an object in the bucket, and every 64 writes its owner adds a snapshot (tried again later when its upload fails, see Replication). After such a snapshot, the owner deletes the history it makes unnecessary: older writes and snapshots, and uploads of earlier owners that no restore would use (ADR 0012). It keeps what a restore needs and anything younger than --history-retention-secs (default 3600, one hour; 0 deletes as soon as the rule allows), so a bug can be noticed before history is gone. The window compares the bucket’s timestamps with the node’s clock: a node whose clock runs ahead shortens it by that much (safety does not depend on it). Deleting an actor erases its history right away, whatever the window: two small objects remain, so it is still known to exist. If the bucket refuses, the erasure is retried a few times while the node holds the actor; after that the history stays until the actor writes again, counted by wasmkeep_history_collect_failures_total. Collection runs in the background, never delays a write, and stops if the node loses the actor’s lease; deleted objects are counted by wasmkeep_history_collected_total (monitoring). An actor that stops writing keeps fewer than 64 writes (and anything that was younger than the window at its last pass) until it writes again. Late uploads of earlier owners that land after a pass stay until the next pass. Old deploy versions are removed by wasmkeep deploy (see Deploying).

When a node dies, and how it scales

What a cluster does when a node stops or dies, what callers see meanwhile, and what adding nodes does and does not do. Running a cluster has the mechanism and the commands; this page is the short answer to “what happens to my application”. It describes the code as it is today; what is planned is on the roadmap.

One node per actor

An actor runs on one node at a time. Its state is one SQLite database with one writer: the node that holds the actor applies its events one at a time, and uploads what each event committed to the bucket. Two nodes running the same actor would be two writers of one history, so a node takes an actor only when the bucket says nobody else holds it (how it works).

A node holds an actor while the actor is in memory. When the actor hibernates (30 s without an event, by default) the node gives it up, and its next event starts it on whichever node that event reaches. A request that reaches another node while the actor is held is forwarded to the holder, once.

A node that stops cleanly

On SIGTERM or Ctrl-C a node:

1. stops accepting connections, so a load balancer’s check of /_wasmkeep/ready fails from then on;
2. lets requests under way finish, for up to 10 s;
3. closes its WebSockets with code 1012;
4. lets each actor finish its queued events, for up to 10 s, and gives each actor up, waiting up to 3 s for those releases to reach the bucket;
5. writes its own lease released, within about 6 s.

Each wait ends as soon as its work is done. Together they bound the whole stop at about 29 s. An actor is free as soon as it is given up: its next request starts it on another node at once, after that node has rebuilt the database from the bucket (unless its local copy is already current). Clients whose socket closed with 1012 open it again and reach the new holder.

To empty a node before stopping it, drain it: it hands its actors over while it still forwards requests, and each actor’s pending alarm moves to the new holder’s timer.

A node that dies

A node that is killed, loses power or is cut off from the bucket gives nothing up. The actors it held in memory stay recorded as its own until another node fences it: a peer that has seen its heartbeat unchanged for 6 s writes the fence once 15 s have passed since the dead node’s last heartbeat (the 10 s lease plus a 5 s grace period). The actors are then free, 15 to about 21 s after the dead node’s last heartbeat with the default timings. The wait is what keeps one writer per actor: a node that is only slow, or cut off for a moment, must have stopped acting on its own before anyone else starts (ADR 0020).

Only actors that were in memory on the dead node wait. An actor that had hibernated there was already given up, and its next request starts it on whichever node it reaches. Two things of a hibernated actor still go with the node: the WebSockets it held for the actor, and the actor’s pending alarm, which sat in that node’s timer and now fires late (below).

What callers get meanwhile

For an actor the dead node held, until the fence:

One case is slower and is not a 503. A request forwarded on a connection the dead node had already accepted, and then never answered, waits 30 s and gets 504 with the body the actor's node did not answer in time and no Retry-After. That is on purpose: the request was written to the node, which may have run it, so the answer does not invite a retry that could run a request that is not idempotent twice. A WebSocket upgrade forwarded the same way gets 503 with Retry-After: 1 instead: it carries no body, and a client opens a socket again anyway.

A 503 does not say whether the request ran: one that was written to the dead node may have run there. Retry what is safe to repeat, and make the rest idempotent. What the actor had committed with exec-durable and acknowledged is in the bucket and survives. A plain exec whose upload had not finished when the node died can be lost (replication).

After the fence, the first request for the actor makes the node it reaches take it: that node rebuilds the database from the bucket, then runs the request. Clients that retry on 503, with a bound, get through without doing anything else.

Requests are refused during the window, not held. Holding them until the takeover is on the roadmap.

Alarms

Pending alarms are listed in the bucket, and every node reads the overdue part of that list every 10 s. An alarm whose node died is fired by another node once it is overdue by a lease period plus grace: with the defaults, 24 s late at the median and 36 s at worst with nodes next to their bucket (measured, ADR 0009). Alarms fire at least once, so a handler that had started on the dead node may run again.

What is not interrupted

• Actors on the other nodes. They keep their holder, their sockets and their alarms. Nothing is moved or restarted because a peer died.
• Requests for hibernated actors, wherever they last ran. Their sockets and alarms on the dead node are another matter (above).
• Stateless components. Every node runs its own copy and needs no other node to answer (below).
• The cluster itself. There is no leader and no quorum: the remaining nodes keep working with the bucket, down to one node.

How it scales

Throughput comes from many actors, not from one busy actor. Each actor handles one event at a time; different actors run in parallel, on every node.

• Where an actor runs. The node that receives the first request for an actor nobody holds takes it. A load balancer that hashes the actor’s path sends every request for one actor to the same node, so actors spread over the nodes as their names hash, and little is forwarded.
• Adding a node. Start it on the same bucket and prefix, with its own node id, and add it to the load balancer. With a consistent hash, a share of the actor paths now reach the new node. An actor still in memory elsewhere stays there (the new node forwards to it); once it hibernates, its next request starts it on the new node.
• Removing a node. Drain it, then stop it.

What does not happen today

• No rebalancing by load. Nothing looks at how busy a node is. An actor in memory stays on its node until it hibernates, until it is moved with wasmkeep actor move, until its node is drained with wasmkeep cluster drain, or until its node stops or dies. A move is not a pin: once the actor hibernates, its next event starts it wherever that event lands.
• No splitting of one actor. An actor that receives more events than it can run one at a time fills its queue (1,024 events, then 429). No setting changes that, and a second node does not help. The application uses more, smaller actors: one per room, user, document or device, not one per application.
• No placement by region or by rule. Any node of the app may take any of its actors.

Load per node in the metrics, and placement, are on the roadmap.

Stateless components

The component given to wasmkeep serve as its positional argument answers every path outside /actor/ and /_wasmkeep/ (the node answers that one itself). It has no database, no lease and nothing in the bucket: each request gets a new instance on the node that received it, so any node answers and the nodes do not coordinate. It scales with the nodes behind the load balancer, within each node’s connection and instance limits and a 30 s wall clock per request.

It is a file each node is started with, not part of a deploy: wasmkeep deploy does not change it. Updating it is a restart of each node in turn, with two versions answering side by side until the last one has restarted (how).

Actors or a database?

Actors are strong where state has an owner:

• State per user, room, document or device. Each actor’s data is next to its code, and nothing else writes it.
• Real time. The sockets of a room meet in one actor, in one order.
• Coordination. One event at a time means a counter, a quota or a lock needs no transaction to be exact.
• Scheduled work per entity. An alarm on the actor, delivered at least once, also when its node dies.

They are weak at questions that span every entity: “all orders over 100 this month”, search, reporting, anything a single SQL query over one big table answers. Actors do not see each other’s data, there is no way to list or query across them, and asking each one in turn is a call per actor.

Keep a database for those questions. A component reaches it over outbound HTTP, the only way out of the sandbox: a database with an HTTP API, or your own service in front of one. There are no raw sockets, so a database’s own wire protocol is not reachable from a component. Outbound HTTP reaches public addresses only unless the operator opens a range with --allow-outbound, refuses the ports browsers block, and allows 16 requests in flight per actor and 256 per node (security model). A common split: actors keep the hot, per-entity state and push a summary to the database when something changes, and the queries that span everything run there (use cases).

Cloudflare Durable Objects, celld and Rivet are built on the same model, stateful actors (how they compare).

Key-value and queues

WasmKeep has no KV store or queue built into the platform. The Rust SDK builds both from actors: a KV shard and a queue are ordinary actors with an SQLite table, reached with actors.call. They get what every actor gets (replication, failover, alarms, access rules) and have the same limits. Why, and what a platform primitive would change, is in ADR 0015. A working app: examples/jobs.

The shard and queue classes are written with the Rust SDK. Callers can be Rust or any other SDK: the TypeScript, Python, Go, C and Zig SDKs have the same clients and talk to the same Rust classes.

Those callers need no Rust toolchain: a release carries the two classes of examples/jobs already built, as kv.wasm (the ready-made shard) and queue.wasm (its queue class: three attempts, a short backoff, pull-only except queue/jobs, which pushes to app/worker), signed like its other files, with the notices of the libraries compiled into them in THIRD_PARTY_NOTICES-components.txt. The example’s README builds them from a clone instead.

# $v: the version of your wasmkeep binary, 0.2.0 or later
curl -fsSLO "https://github.com/WasmKeep/wasmkeep/releases/download/v$v/kv.wasm"
curl -fsSLO "https://github.com/WasmKeep/wasmkeep/releases/download/v$v/queue.wasm"
wasmkeep serve --actor kv=kv.wasm --actor queue=queue.wasm --actor app=your-app.wasm \
  --access kv=internal --access queue=internal

Key-value

Deploy a shard class: a component that exports the SDK’s ready-made shard.

#![allow(unused)]
fn main() {
use wasmkeep_sdk::kv::Shard;
wasmkeep_sdk::export!(Shard);
}

Serve it as class kv (--actor kv=..., plus --access kv=internal), and use it from any other actor:

#![allow(unused)]
fn main() {
use wasmkeep_sdk::kv::{self, PutOptions};

let kv = kv::Client::new("kv", 8);          // 8 shards: kv/shard-0 .. kv/shard-7
let v1 = kv.put("user:42", b"Ada")?;         // the new version, once durable
let entry = kv.get("user:42")?;             // Some(Entry { value, version, expires_at })
kv.compare_and_set("user:42", Some(v1), b"Ada L.")?;   // Err(Conflict) if it moved
kv.compare_and_set("user:43", None, b"Bob")?;          // only if absent
kv.put_with("session:9", b"...", PutOptions { ttl_ms: Some(60_000), ..PutOptions::default() })?;
kv.delete("user:43")?;
let page = kv.list("user:", None, 100)?;    // keys with that prefix, in byte order
}

• Keys are bytes, up to 1 KiB (&str works). Values up to what one call carries (16 MB; keep them small).
• Shards: a key lives in shard fnv1a(key) % n. Every caller must use the same n, and changing it needs a migration, so pick it generously (shards cost nothing while idle).
• Versions change on every write and are never reused within a shard, even after a delete.
• Expiry: an expired key reads as absent; the shard’s alarm deletes it.
• Read cache: Client::with_read_cache(ms) serves get from memory in the caller’s instance for up to ms; the caller’s own writes through that client update it; get_fresh skips it. It is lost with the instance (hibernation, a failed event, a deploy).
• For a shard with code of its own (extra checks, other tables), call the functions in kv::shard from your handler instead of exporting Shard.

Queues

Deploy a queue class. Its settings are code, and may depend on the queue’s name:

#![allow(unused)]
fn main() {
use wasmkeep_sdk::queue::{self, Config};
use wasmkeep_sdk::{actors, http};

struct Queue;

fn config() -> Config {
    match actors::self_address().name.as_str() {
        "emails" => Config::new().consumer("mailer/main"),  // push
        _ => Config::new(),                                 // pull only
    }
}

impl wasmkeep_sdk::Actor for Queue {
    fn on_init() -> Result<(), String> {
        queue::store::init().map_err(|e| e.to_string())
    }
    fn on_alarm() -> Result<(), String> {
        queue::store::on_alarm(&config()).map_err(|e| e.to_string())
    }
}

impl http::Guest for Queue {
    fn handle(request: http::IncomingRequest, out: http::ResponseOutparam) {
        queue::store::handle(&config(), request, out);
    }
}

wasmkeep_sdk::export!(Queue);
}

Producers send to it:

#![allow(unused)]
fn main() {
let emails = wasmkeep_sdk::queue::Client::new("queue/emails");
let id = emails.send(b"welcome ada@example.com")?;     // once durable
emails.send_delayed(b"reminder", 3_600_000)?;
emails.send_batch(&[b"a".as_slice(), b"b"])?;         // one write, all or none
}

Push (a queue with a consumer): its alarm delivers batches to the consumer, which answers one outcome per message:

#![allow(unused)]
fn main() {
use wasmkeep_sdk::queue::{self, Outcome};

// In the consumer's handler:
if queue::is_delivery(&request) {
    queue::handle_delivery(request, out, |_queue, message| {
        match send_email(&message.body) {
            Ok(()) => Outcome::Ack,
            Err(_) => Outcome::Retry(None),   // the queue's backoff
        }
    });
    return;
}
}

Pull (any queue): lease, then ack or retry.

#![allow(unused)]
fn main() {
let inbox = wasmkeep_sdk::queue::Client::new("queue/inbox");
for message in inbox.lease(10, 30_000)? {     // up to 10, hidden for 30 s
    handle(&message.body);
    inbox.ack(&message)?;                     // false if the lease ran out
}
}

Config sets the batch size (10), the visibility timeout (30 s), the attempts before a message is dead-lettered (5) and the retry backoff (1 s doubling to 5 min). Client::stats counts ready, delayed, in-flight and dead messages; Client::dead_letters lists the dead ones with the reason.

TypeScript

The TypeScript SDK (@wasmkeep/sdk) has the client side only: the kv client, the queue client and queue.handleDelivery for push consumers. The shard and queue classes stay Rust: deploy them as the components from examples/jobs, kv (examples/jobs/kv, the SDK’s ready-made shard) and queue (examples/jobs/queue, or your own Rust queue class when you need other settings), built from a clone or downloaded from a release, and call them from a TypeScript actor. The wire protocol, the shard hash and the status codes are the same byte for byte in every SDK, so callers in any language share the same data. examples/ts-jobs is the jobs app in TypeScript.

import { defineActor, kv, queue } from "@wasmkeep/sdk";

const store = new kv.Client("kv", 8);             // the same shard count as every other caller
const emails = new queue.Client("queue/emails");
const inbox = new queue.Client("queue/inbox");

export default defineActor({
  fetch(req) {
    // A push consumer: one outcome per message.
    if (queue.isDelivery(req)) {
      return queue.handleDelivery(req, (_queue, message) => "ack"); // or "retry", or { retryAfterMs: 500 }
    }

    // Calls to shards and queues run in an event, never at module level.
    const v1 = store.put("user:42", "Ada");           // the new version, once durable
    const entry = store.get("user:42");               // { value: Uint8Array, version, expiresAt? } or undefined
    store.compareAndSet("user:42", v1, "Ada L.");     // throws KvError (kind "conflict") if it moved
    store.compareAndSet("user:43", undefined, "Bob"); // only if absent
    store.putWith("session:9", "...", { ttlMs: 60_000, durable: true });
    store.deleteIf("user:43", 7);                     // only at that version
    const page = store.list("user:", undefined, 100); // Uint8Array keys, in byte order

    const id = emails.send("welcome ada@example.com");
    emails.sendDelayed("reminder", 3_600_000);
    emails.sendBatch(["a", "b"]);                     // one write, all or none

    for (const message of inbox.lease(10, 30_000)) {
      inbox.ack(message);                             // false if the lease ran out
    }
    return "ok\n";
  },
});

The names and semantics follow the Rust ones (getFresh, withReadCache(ms), stats(), deadLetters(limit), settle(items)), with these differences:

• Numbers, not 64-bit integers. Versions, ids, times and counts are numbers. The wire still carries 64-bit integers; a version or id past 2^53 - 1 (which a shard or queue does not reach in practice) makes the reply malformed (KvError/QueueError kind refused, status 200) instead of being rounded.
• Errors are exceptions: KvError (kind: conflict, key-too-long, storage, call, refused, with status for refused) and QueueError (storage, call, refused), with the Rust messages. A negative or fractional count, delay or limit throws a RangeError before any call.
• Keys and values are strings (sent as UTF-8, so the same key reaches the same shard from Rust) or Uint8Arrays; values come back as Uint8Array.
• A throwing delivery handler makes fetch answer 500: the queue retries the whole batch with its backoff.

Python

The Python SDK (sdks/python, module wasmkeep) has the same client side: kv, queue and queue.handle_delivery, talking to the same Rust shard and queue classes (deploy them as in TypeScript) with the same protocol, hash and statuses. examples/py-jobs is the jobs app in Python.

from wasmkeep import Actor, kv, queue

store = kv.Client("kv", 8)                       # the same shard count as every other caller
emails = queue.Client("queue/emails")
inbox = queue.Client("queue/inbox")


class App(Actor):
    def fetch(self, req):
        # A push consumer: one outcome per message.
        if queue.is_delivery(req):
            return queue.handle_delivery(req, lambda _queue, m: "ack")  # or "retry", or queue.RetryAfter(500)

        # Calls to shards and queues run in an event, never at module level.
        v1 = store.put("user:42", "Ada")                 # the new version, once durable
        entry = store.get("user:42")                     # Entry(value=b"Ada", version, expires_at) or None
        store.compare_and_set("user:42", v1, "Ada L.")   # raises KvError (kind "conflict") if it moved
        store.compare_and_set("user:43", None, "Bob")    # only if absent
        store.put_with("session:9", "...", ttl_ms=60_000, durable=True)  # expect="any" | "absent" | a version
        store.delete_if("user:43", 7)                    # only at that version
        page = store.list("user:", None, 100)            # bytes keys, in byte order

        message_id = emails.send("welcome ada@example.com")
        emails.send_delayed("reminder", 3_600_000)
        emails.send_batch(["a", "b"])                    # one write, all or none

        for message in inbox.lease(10, 30_000):
            inbox.ack(message)                           # False if the lease ran out
        return "ok\n"

The names and semantics follow the Rust ones (get_fresh, with_read_cache(ms), stats(), dead_letters(limit), settle(items)), with these differences:

• Integers are exact. Versions, ids, times and counts are Python ints, so nothing is rounded. Since an int has no bounds, a count, delay, limit or version outside what Rust’s u32 or u64 would hold raises ValueError before any call.
• Errors are exceptions: KvError (kind: conflict, key-too-long, storage, call, refused, with status for refused) and QueueError (storage, call, refused), with the Rust messages as str(e).
• Options are keyword arguments: put_with(key, value, expect=, ttl_ms=, durable=), where expect is "any", "absent" or a version. list(prefix, after, limit) defaults to every key (b""), from the start, 100 at a time.
• Outcomes are "ack", "retry" or queue.RetryAfter(ms); records (Entry, Message, Settlement, Stats, DeadLetter) are frozen dataclasses with snake_case fields (expires_at, enqueued_at, in_flight, failed_at).
• Keys and values are str (sent as UTF-8, so the same key reaches the same shard from Rust) or bytes; values come back as bytes.
• A raising delivery handler makes fetch answer 500: the queue retries the whole batch with its backoff.

Go

The Go SDK (for TinyGo) has the client side, like TypeScript: the kv client, the queue client and queue.HandleDelivery for push consumers, talking to the Rust shard and queue classes from examples/jobs with the same protocol, hash and status codes. The clients send through a call.Caller: wasmkeep.Caller, the SDK’s own call to another actor (examples/go-jobs is the jobs app in Go).

import (
	"errors"

	"github.com/wasmkeep/wasmkeep/sdks/go/kv"
	"github.com/wasmkeep/wasmkeep/sdks/go/queue"
	"github.com/wasmkeep/wasmkeep/sdks/go/wasmkeep"
)

store := kv.New("kv", 8, wasmkeep.Caller)
v1, err := store.Put("user:42", []byte("Ada"))      // the new version, once durable
entry, err := store.Get("user:42")                  // *kv.Entry, nil if absent
_, err = store.CompareAndSet("user:42", kv.Version(v1), []byte("Ada L."))
if errors.Is(err, kv.ErrConflict) { /* it moved */ }
_, err = store.CompareAndSet("user:43", kv.Absent(), []byte("Bob"))
_, err = store.PutWith("session:9", []byte("..."), kv.PutOptions{TTLMs: 60_000})
keys, err := store.List("user:", 100)               // in byte order; ListAfter pages

emails := queue.New("queue/emails", wasmkeep.Caller)
id, err := emails.Send([]byte("welcome ada@example.com"))
ids, err := emails.SendBatch([][]byte{[]byte("a"), []byte("b")})

inbox := queue.New("queue/inbox", wasmkeep.Caller)
messages, err := inbox.Lease(10, 30_000)
for _, m := range messages {
	acked, err := inbox.Ack(m)                      // false if the lease ran out
}

// A push consumer, in the Fetch hook:
if queue.IsDelivery(req.Method, req.Path) {
	reply := queue.HandleDelivery(queue.CallerFrom(req.Headers), req.Body(),
		func(_ string, m queue.Message) queue.Outcome { return queue.Ack() }) // or queue.Retry(), queue.RetryAfter(500)
	return wasmkeep.Bytes(reply.Status, reply.ContentType, reply.Body)
}

The names and semantics follow the Rust ones (GetFresh, WithReadCache(ms), Retry, RetryAfter, Settle, Stats, DeadLetters(limit)), with these differences:

• Errors are values: *kv.Error (Kind: Conflict, KeyTooLong, Storage, Call, Refused, with Status for Refused) and *queue.Error (Storage, Call, Refused), with the Rust messages. errors.Is(err, kv.ErrConflict) matches by kind; errors.As gives the fields.
• Zero values are the defaults. kv.PutOptions{} is a durable, unconditional write without expiry: Local: true asks for a local write (Rust’s PutOptions::local()), and TTLMs: 0 means no expiry (so a zero TTL cannot be asked for). The zero kv.Expect is kv.Any(), the zero queue.Outcome is queue.Ack().
• Keys are strings (any bytes, so the same bytes reach the same shard from every SDK); values and bodies are []byte. List and ListAfter replace Rust’s Option for after.
• The delivery helper does not touch HTTP: it takes the x-wasmkeep-caller header ("" when absent) and the body, and returns the status, content type and body to answer. A handler that panics traps the event: the queue retries the whole batch with its backoff.
• For TinyGo: the clients avoid reflection-heavy packages (no fmt), so they suit TinyGo’s wasip2 target, and their unit tests run with plain go test.

C

The C SDK (sdks/c, declared in wasmkeep_kv.h, which wasmkeep.h includes) has the same client side as TypeScript: the wk_kv client, the wk_queue client and wk_queue_handle_delivery for push consumers, speaking the same protocol to the same Rust classes. examples/c-jobs is the jobs app in C. Compile wasmkeep_kv.c, wasmkeep_wire.c and wasmkeep_workflow.c with wasmkeep.c (the examples’ build.sh do).

#include "wasmkeep.h"

static const wk_kv store = {.class_name = "kv", .shards = 8};  /* or wk_kv_client("kv", 8) */
static const wk_queue emails = {.address = "queue/emails"};

wk_kv_error err;
uint64_t v1;
wk_kv_put(&store, wk_str("user:42"), wk_str("Ada"), &v1, &err);  /* false on failure: err.kind, err.message */
wk_kv_entry entry;
if (wk_kv_get(&store, wk_str("user:42"), &entry, &err) == 1) {  /* 1 found, 0 absent, -1 failed */
  /* entry.value (wk_bytes), entry.version, entry.expires_at (0: never) */
  wk_kv_entry_free(&entry);
}
wk_kv_compare_and_set(&store, wk_str("user:42"), &v1, wk_str("Ada L."), NULL, &err);  /* WK_KV_CONFLICT if it moved */
wk_kv_compare_and_set(&store, wk_str("user:43"), NULL, wk_str("Bob"), NULL, &err);    /* only if absent */
wk_kv_put_options opts = {.has_ttl = true, .ttl_ms = 60000};  /* zeroed: no condition, durable */
wk_kv_put_with(&store, wk_str("session:9"), wk_str("..."), &opts, NULL, &err);
wk_kv_keys page;
if (wk_kv_list(&store, wk_str("user:"), NULL, 100, &page, &err)) wk_kv_keys_free(&page);

uint64_t id;
wk_queue_error qerr;
wk_queue_send(&emails, wk_str("welcome ada@example.com"), &id, &qerr);

/* A push consumer: one outcome per message; return false to fail the batch. */
static bool work(const char *queue, const wk_queue_message *m, wk_queue_outcome *outcome, void *ctx) {
  *outcome = wk_queue_ack_outcome();  /* or wk_queue_retry_outcome(), wk_queue_retry_after(500) */
  return true;
}

wk_response wk_fetch(wk_request *req) {
  if (wk_queue_is_delivery(req)) return wk_queue_handle_delivery(req, work, NULL);
  return wk_reply(200, "ok\n");
}

The names follow the Rust ones (wk_kv_get_fresh, wk_kv_with_read_cache, wk_kv_delete_if, wk_queue_send_delayed, wk_queue_send_batch, wk_queue_lease, wk_queue_ack, wk_queue_retry, wk_queue_settle, wk_queue_get_stats, wk_queue_get_dead_letters), with these differences:

• Errors are values: each call returns false (or -1) and fills an optional wk_kv_error / wk_queue_error with a kind (WK_KV_CONFLICT, WK_KV_KEY_TOO_LONG, WK_KV_STORAGE, WK_KV_CALL, WK_KV_REFUSED with status; WK_QUEUE_STORAGE, WK_QUEUE_CALL, WK_QUEUE_REFUSED) and the Rust message. A refusal’s message keeps at most 400 bytes of the answer, unchanged (not made valid UTF-8).
• Ownership is explicit: what comes back with memory (an entry, keys, messages, dead letters) is freed with its _free function; returned bytes are NUL-terminated as well. Clients borrow their class name or address. wk_queue_send_batch and wk_queue_settle write into arrays the caller sizes; settle marks items the queue’s reply leaves out as not current.
• Keys and values are wk_bytes (wk_str for a C string), 64-bit integers are uint64_t, and options are optional pointers (NULL: an absent key for compare_and_set, the queue’s backoff for retry).
• A delivery handler returns false to fail the batch (500, retried with the queue’s backoff), where Rust panics and TypeScript throws.

wk_response_header(&r, "etag", v) adds a header to a response (the jobs app’s etag).

Zig

The Zig SDK (sdks/zig) wraps the C clients, with Zig errors and slices: wk.kv.Client, wk.queue.Client and wk.queue.handleDelivery. examples/zig-jobs is the jobs app in Zig.

const wk = @import("wasmkeep");

const store = wk.kv.Client.init("kv", 8);           // a comptime constant is fine
const emails = wk.queue.Client.init("queue/emails");

fn example() !void {
    const v1 = try store.put("user:42", "Ada");      // the new version, once durable
    if (try store.get("user:42")) |found| {          // ?Entry: value, version, expires_at
        var entry = found;
        defer entry.deinit();
    }
    _ = try store.compareAndSet("user:42", v1, "Ada L."); // error.Conflict if it moved
    _ = try store.putWith("session:9", "...", .{ .ttl_ms = 60_000 });
    var page = try store.list("user:", null, 100);   // page.len(), page.at(i)
    defer page.deinit();
    _ = try emails.send("welcome ada@example.com");
}

fn work(queue: []const u8, message: wk.queue.Message) wk.queue.Outcome {
    _ = queue;
    _ = message;
    return .ack; // or .retry, or .{ .retry_after = 500 }
}

pub fn fetch(req: *wk.Request) !wk.Response {
    if (wk.queue.isDelivery(req)) return wk.queue.handleDelivery(req, work);
    return wk.reply(200, "ok\n", .{});
}

The differences from Rust are the C ones, in Zig terms:

• Errors: wk.kv.Error (Conflict, KeyTooLong, Storage, Call, Refused) and wk.queue.Error (Storage, Call, Refused); the message and status of the last failure are wk.kv.lastError() / lastStatus() (and the same in wk.queue).
• Ownership: entries, key pages, leased messages and dead letters have deinit; sendBatch and settle take an allocator for the slice they return.
• A delivery handler returns an Outcome, or an error union of one: an error makes handleDelivery return it, so fetch answers 500 and the queue retries the batch with its backoff.

Consistency and failure

• Per key, strongly consistent. A shard handles one operation at a time, so reads see the latest write and compare-and-set is exact. Only the optional read cache serves older values, up to its age.
• Durable when acknowledged. put, delete and send return once the write is in the bucket (fsynced on a node without one), unless a local write is asked for (PutOptions::local()).
• No transactions across keys, across shards, or between a KV write or a send and the caller’s own database. If the caller’s event fails after calling, the call’s write stays: treat it as a side effect, like outbound HTTP.
• A failed call is ambiguous. Error::Call on a write or a send means it may have applied. Re-read before retrying a compare-and-set; a send retried after a lost reply may queue the message twice.
• At-least-once delivery. A message may be delivered again after a lost ack, a lease that ran out, a failed push call or a failover, with the same id and a higher attempt. Record the ids you handled when a duplicate would matter, in the same event as the work.
• Stale acks are ignored. An ack or retry names its lease (id, attempt); once the message is leased again, the old one does nothing.
• Ordering: oldest visible first; FIFO only with one consumer, batches of one and no retries.
• Failover: a shard or queue is one actor, unavailable while another node takes it over: 15 to about 21 s after its node’s last heartbeat, then a restore (how; on actors, before ADR 0020: 15.5 to 17.0 s at the median with nodes next to their bucket, and 18.2 s on node leases, measured). Pending push deliveries and expiry are alarms, fired after a failover as for any actor (Alarms).

Limits

• Throughput per shard and per queue is about one durable write per bucket round trip. A durable write took 54 ms at the median with nodes next to an OVHcloud bucket (49 ms in a later run), 81 ms from a desktop to the same store and 587 ms from a desktop to R2 (measured): roughly 18 writes a second next to the bucket, under 2 from a desktop to R2. Those figures come from the actor soak; shards and queues were not benchmarked on their own. Spread hot keys over more shards, split a busy queue into several (by key hash), batch sends, or use local writes where losing the last write in a node crash is acceptable.
• Each operation is one call, which holds the caller’s event while it runs. A push delivery holds the queue’s event while the consumer runs: keep consumers short, or pull.
• No re-entry: a consumer must not call the queue that is delivering to it (the call is refused); a shard must not call itself through KV.
• Outside clients get 403 from shards and queues (they answer only actors.call). Serve their classes with --access CLASS=internal as well, so outside requests never create an actor.

If these limits get in the way, say so in an issue: a platform primitive can replace the actors behind the same Client API (ADR 0015).

Workflows

Work that spans minutes to days and must survive restarts and failover: reserve stock, wait for approval, charge, notify, retrying each part on its own. wasmkeep_sdk::workflow runs it on ordinary actors, one actor per workflow instance, with no platform feature behind it (ADR 0019). The Rust and TypeScript SDKs have it (TypeScript below); Go, Python, C and Zig have the client side (Go, Python, C, Zig).

Writing a workflow

A workflow is a function. The runner calls it from the top every time the instance moves forward; inside it, three calls record what happened:

#![allow(unused)]
fn main() {
use wasmkeep_sdk::workflow::{Run, Stop};

fn order(wf: &mut Run, input: &[u8]) -> Result<Vec<u8>, Stop> {
    let reservation: String = wf.step_keyed("reserve", |key| stock::reserve(input, key))?;
    wf.sleep("cool-off", 60_000)?;
    if wf.wait_for_event("approved", 86_400_000)?.is_none() {
        return Err(Stop::fail("not approved in time"));
    }
    let _: String = wf.step_keyed("charge", |key| payments::charge(&reservation, key))?;
    Ok(b"shipped".to_vec())
}
}

• step(name, f) calls f and, once it returns Ok, records its result (String, Vec<u8>, u64, i64, bool or (): convert other types to one of these, or implement Stored); every later run gets the recorded result without calling f. Until then f may run again: an error from f is retried with backoff (1 s doubling, 5 attempts by default), and an attempt cut before its result is recorded runs again (rule 2 below). When the attempts run out the workflow fails, or, with try_step, gets the error to handle.
• sleep(name, ms) and sleep_until(name, at) wait. The instance costs nothing meanwhile: its alarm brings it back.
• wait_for_event(name, timeout_ms) waits for an event delivered with Client::send_event, and returns its payload, or None after the timeout. Each wait takes the oldest event of that name not taken yet.
• The function returns the workflow’s output, or Err(Stop::fail(..)) to fail it. ? passes the runner’s own Stop (waiting) through.

Two rules

1. Between steps, the same decisions every time. The function replays from the top, so it must reach the same steps in the same order given the same input and recorded results. Put anything that varies inside a step: the time, random numbers, reads of data, HTTP calls. Steps are identified by their position, so a loop may repeat a step name; a run that reaches a different step at a recorded position fails the workflow (“the code changed the order of its steps”) rather than mixing two histories. Deploying new code onto running workflows is safe when it only adds steps after the recorded ones.
2. A step may run more than once. Its result is recorded after f returns, so a node failure or a cut in between runs it again. Make steps idempotent, or use step_keyed, which gives the step an idempotency key for the service it calls: unique to the step and to this run of the instance.

The workflow class

The workflow is served as a class of its own, which uses its actors’ alarm, so it does nothing else:

#![allow(unused)]
fn main() {
use wasmkeep_sdk::workflow::{self, Config};

struct Orders;

impl wasmkeep_sdk::Actor for Orders {
    fn on_init() -> Result<(), String> {
        workflow::store::init().map_err(|e| e.to_string())
    }
    fn on_alarm() -> Result<(), String> {
        workflow::store::on_alarm(&Config::new(), order);
        Ok(())
    }
}

impl wasmkeep_sdk::http::Guest for Orders {
    fn handle(r: wasmkeep_sdk::http::IncomingRequest, o: wasmkeep_sdk::http::ResponseOutparam) {
        workflow::store::handle(&Config::new(), r, o)
    }
}

wasmkeep_sdk::export!(Orders);
}

Config sets the attempts, the backoff, how long a finished instance is kept (retention_ms, 7 days by default) and how much of an event a run spends on new steps before continuing in a new event (20 s of the 30 s). Serve the class with --access orders=internal: only other actors start and query workflows (the runner also refuses outside requests, 403).

To try a client in another language without a Rust toolchain, a release carries the example’s workflow already built: workflows-orders.wasm (the orders class of examples/workflows) and workflows-ledger.wasm (the ledger class its steps call, so it needs both), signed like the release’s other files, with the notices of the libraries compiled into them in THIRD_PARTY_NOTICES-components.txt. The example’s README builds them from a clone instead.

# $v: the version of your wasmkeep binary, 0.2.0 or later
for f in workflows-orders.wasm workflows-ledger.wasm; do
  curl -fsSLO "https://github.com/WasmKeep/wasmkeep/releases/download/v$v/$f"
done
wasmkeep serve --actor orders=workflows-orders.wasm --actor ledger=workflows-ledger.wasm \
  --actor app=your-app.wasm --access orders=internal

Starting and following instances

From any other actor:

#![allow(unused)]
fn main() {
use wasmkeep_sdk::workflow::Client;

let orders = Client::new("orders");
orders.start("order-42", b"item=book")?;     // idempotent while kept
let status = orders.status("order-42")?;     // state, wake time, output or error
orders.send_event("order-42", "approved", b"")?;
orders.cancel("order-42")?;
}

• start only records: the instance runs in its own event right after, never inside the caller’s call. A step may therefore call back the actor that started it.
• status reports Running, Waiting (until when, for which events), Completed (with the output), Failed (with the error) or Cancelled, and how many positions are recorded.
• send_event and cancel to an id that was never started answer Error::Unknown and leave nothing behind; an event to a finished instance answers Error::Finished.
• cancel stops the instance at once; a step running at that moment finishes first.
• After its retention, a finished instance deletes itself. Its id is then unknown, and a start with it is a new run (new idempotency keys).

Go

The Go SDK (for TinyGo) has the client only: package workflow starts and follows instances of a workflow class written with the Rust runner, over the same protocol (the wasmkeep-workflow/v1 routes). As with its kv and queue clients, it sends through wasmkeep.Caller.

import (
	"errors"

	"github.com/wasmkeep/wasmkeep/sdks/go/wasmkeep"
	"github.com/wasmkeep/wasmkeep/sdks/go/workflow"
)

orders := workflow.New("orders", wasmkeep.Caller)
status, err := orders.Start("order-42", []byte("item=book")) // *workflow.Status
status, err = orders.Status("order-42")                    // State, WakeAt, WaitingFor, Output, Error, Positions
err = orders.SendEvent("order-42", "approved", nil)
status, err = orders.Cancel("order-42")
if errors.Is(err, workflow.ErrUnknown) { /* never started, or past its retention */ }
if errors.Is(err, workflow.ErrFinished) { /* takes no more events */ }

WakeAt is 0 when there is none, Output nil when there is none (an empty output is non-nil), Error empty when there is none. Errors are *workflow.Error with the Rust SDK’s kinds and messages (ErrUnknown, ErrFinished, ErrCall, ErrRefused). examples/go-jobs drives the orders class of examples/workflows with the same order routes as its Rust app; that class is also in a release, already built.

Python

The Python SDK (sdks/python, module wasmkeep.workflow) has the client only: it starts and follows instances of a workflow class written with the Rust runner, over the same protocol, as its key-value and queue clients do. examples/py-jobs drives the orders class of examples/workflows this way (the class is also in a release, already built).

from wasmkeep import Actor, workflow

orders = workflow.Client("orders")


class App(Actor):
    def fetch(self, req):
        orders.start("order-42", "item=book")            # a Status; idempotent while kept
        orders.send_event("order-42", "approved", b"")
        try:
            status = orders.status("order-42")
        except workflow.WorkflowError as e:
            if e.kind == "unknown":                      # never started, or past its retention
                return "no such order\n"
            raise
        if status.state == "completed":
            return status.output                         # bytes
        return f"{status.state}, waiting for {status.waiting_for}\n"

The names and semantics are the Rust ones, with these differences:

• Status is a frozen dataclass: state is a string ("running", "waiting", "completed", "failed" or "cancelled", also in workflow.STATES), wake_at an int or None, waiting_for a list of names, output bytes or None, error a string or None, positions an int; is_finished is a property.
• Errors are exceptions: WorkflowError with kind unknown (404: no such instance), finished (409: an event to a finished instance), call (the call failed: the instance may or may not have recorded it) or refused (any other answer, with status), worded as Rust’s Display.
• Inputs and payloads are str (sent as UTF-8) or bytes, and default to empty.

C

The C SDK (sdks/c, declared in wasmkeep_workflow.h, which wasmkeep.h includes; compile wasmkeep_workflow.c with the other SDK files) has the client side only: the workflow class itself runs on the Rust runner. It speaks the same protocol as Client, so it starts and follows the same instances.

static const wk_workflow orders = {.class_name = "orders"};  /* or wk_workflow_client("orders") */

wk_workflow_status s;
wk_workflow_error err;
if (!wk_workflow_start(&orders, wk_str("order-42"), wk_str("item=book"), &s, &err))
  return wk_reply(502, "%s\n", err.message);
wk_workflow_status_free(&s);
if (wk_workflow_get_status(&orders, wk_str("order-42"), &s, &err)) {
  /* s.state (WK_WORKFLOW_WAITING, ...), s.wake_at (0: none), s.waiting_for[0 .. s.waiting_count),
   * s.has_output and s.output, s.error (len 0: none), s.positions */
  wk_workflow_status_free(&s);
}
wk_workflow_send_event(&orders, wk_str("order-42"), wk_str("approved"), wk_str(""), &err);
wk_workflow_cancel(&orders, wk_str("order-42"), NULL, &err);  /* NULL: the status is not wanted */

• Each call returns false on failure and fills an optional wk_workflow_error: WK_WORKFLOW_UNKNOWN, WK_WORKFLOW_FINISHED, WK_WORKFLOW_CALL (the call failed; the instance may or may not have recorded it) or WK_WORKFLOW_REFUSED with status (200 for a malformed status), and the Rust message.
• wk_workflow_get_status is Rust’s status (wk_workflow_status is the type). Ids, event names and payloads are wk_bytes; a status is freed with wk_workflow_status_free, and its bytes are NUL-terminated.

The jobs app in C (examples/c-jobs) has the order routes of examples/workflows/app.

Zig

The Zig SDK wraps the C client as wk.workflow (workflow.zig), with Zig errors:

const orders = wk.workflow.Client.init("orders");   // a comptime constant is fine

fn example() !void {
    var status = try orders.start("order-42", "item=book");
    defer status.deinit();
    // status.state (.running, .waiting, ...), status.wake_at (?u64),
    // status.waitingCount() / waitingFor(i), status.output (?[]const u8),
    // status.failure (?[]const u8), status.positions
    try orders.sendEvent("order-42", "approved", "");
    var cancelled = try orders.cancel("order-42");
    cancelled.deinit();
}

wk.workflow.Error is Unknown, Finished, Call or Refused; the message and status of the last failure are wk.workflow.lastError() and lastStatus(). The jobs app in Zig (examples/zig-jobs) has the same order routes.

What is guaranteed

• Every state reported (started, a step’s result, a sleep’s wake time, an event received, completed, failed, cancelled) is written with exec-durable: it survives any node failure.
• While an instance is not finished, its alarm is set, so a trap, a cut or a failover never leaves it stuck: the next attempt comes from the alarm, after the step’s backoff. Runs that keep ending without the function returning (a trap outside any step, a replay over its CPU budget) back off too, and fail the instance after twice its attempts.
• A database error inside the runner is retried, never turned into a failed workflow.
• Every event makes progress: a run spends run_budget_ms on new steps and continues in a new event, but always runs at least one.
• One event at a time per instance: no two runs of the same workflow race.

TypeScript

The same runner, tables and protocol in @wasmkeep/sdk, so a Rust client follows a TypeScript workflow and the other way round. A waiting run stops by throwing a Stop (let it through, or rethrow it from a catch-all: workflow.isStop(e)), and step results are recorded as JSON:

import { env, http, workflow } from "@wasmkeep/sdk";

export default workflow.define(
  (wf, input) => {
    const reservation = wf.step("reserve", (key) => {
      const r = http.fetch(env.get("RESERVE_URL") ?? "", {
        method: "POST",
        headers: { "idempotency-key": key },
      });
      // Any status is a response: throw on failure, so it is retried
      // rather than recorded.
      if (r.status !== 200) throw new Error(`reserve answered ${r.status}`);
      return r.text();
    });
    wf.sleep("cool-off", 60_000);
    if (wf.waitForEvent("approved", 86_400_000) === undefined) {
      throw workflow.fail("not approved in time");
    }
    return `shipped: ${reservation}`;
  },
  // A function, read on each event: module code runs at build time, when
  // the class's configuration is not there yet.
  () => ({ maxAttempts: Number(env.get("ATTEMPTS") ?? 5) }),
);

• wf.step(name, (key) => ...) passes the step’s idempotency key; an exception thrown from it is retried. wf.tryStep returns { ok } or { error, attempts } instead of failing the workflow.
• wf.sleep, wf.sleepUntil and wf.waitForEvent (the payload as bytes, or undefined after the timeout) work as in Rust.
• new workflow.Client("orders") has start, status, sendEvent and cancel; errors are workflow.WorkflowError (kind: unknown, finished, call, refused).
• Steps that call other services use http.fetch, the SDK’s synchronous outbound HTTP, on either engine. A response whose body breaks before its end (the connection closes short of its Content-Length or inside a chunk, a reset, a timeout between bytes) throws a FetchError naming the incomplete body, so a step that checks only the status fails (retried while it has attempts left) rather than recording a cut answer. A body longer than limit is not an error: its first limit bytes are kept. The Rust SDK’s http::fetch does the same, returning an error.

examples/ts-workflows is the order of examples/workflows in TypeScript, started and followed by the Rust app.

Limits

• A step’s work fits in one event: under 30 s of wall clock and the event’s 50 ms of CPU, less what replay used. Longer work goes to something that reports back with send_event.
• Replay costs about 1.5 µs a recorded step in every event (a Rust workflow: 7.7 ms for a run over 5,000 steps, measured). Thousands of steps are fine; tens of thousands approach the event’s CPU budget.
• Large inputs cost CPU in TypeScript on QuickJS, the default engine, which interprets: the input is read in every event, and a step that sends it as a request body copies it again. In the example, a 64 KiB input sent as a body fits in an event; at 256 KiB every run goes over the CPU budget before its first step, and the instance fails after twice its attempts (about a minute with the default backoff), saying so. StarlingMonkey handled 256 KiB. Keep large data out of the input (store it, and pass its key), or raise --cpu-budget-ms.
• No list of instances: an instance is found by its id. Keep a registry (a key-value prefix) if you need one.
• Failover: an instance is unavailable until a peer frees its dead node’s actors (15 to about 21 s after that node’s last write, how) and its actor is restored, and its alarm fires late, as for any actor: 24 s at the median and 36 s at worst with nodes next to their bucket, over twelve hours (measured).

The example in examples/workflows runs an order through every case above.

Where to run it

WasmKeep is a long-lived process that keeps actors, leases, WebSockets and timers in memory, writes many small transactions to local SQLite files, holds three open files per resident actor, and reserves a large virtual address space for its instance pool. That shapes where it runs well.

A container is packaging, not isolation: actors are already sandboxed by WebAssembly. Isolating tenants who may be hostile needs a stronger tier (a process or microVM per tenant), which WasmKeep does not have yet (security model).

Rules that apply everywhere

• A real disk for --data-dir: local NVMe or a fast volume; never a container’s overlay filesystem or a network file system (SQLite relies on local locking and fsync).
• Open files: give the process a high limit (LimitNOFILE=65536 with systemd, --ulimit nofile=65536 with Docker). WasmKeep raises its soft limit to the hard one and caps resident actors to fit (monitoring).
• Virtual memory: no LimitAS or ulimit -v; do not use MemoryDenyWriteExecute (Wasmtime maps compiled code executable). A memory limit on resident memory (cgroups) is fine.
• Transparent huge pages need nothing: WasmKeep turns them off for its own process on Linux, which halves memory per actor where the kernel sets them to always (monitoring). Leave the system setting alone for other software on the machine.
• A stable node identity in a cluster: one --node-id per machine (a StatefulSet on Kubernetes).
• A private network between nodes, in the same region as the bucket; only the load balancer reaches them (front door).
• Local disk durability: on one node, the local disk is the database, so back it up (one server); in a cluster the bucket is the source of truth and a node’s disk can be lost.

What is measured so far

The numbers in Benchmarks come from one workstation and a shared cloud VM: good for comparing commits, languages and runtimes, not for capacity claims. A three-node soak ran against R2 and OVHcloud Object Storage, from a desktop and from three VPS next to the bucket (results): nothing lost over 36 kills in four runs. Put the nodes in the same region as the bucket: a durable write waits for one PUT (54 ms next to it, 587 ms from a desktop to R2), and a failover for a restore of many GETs. The suite has not yet run on dedicated hardware.

One server (VPS)

WasmKeep on one Linux machine, behind a reverse proxy that terminates TLS, run by systemd. This is the right setup for most projects: one node serves thousands of requests per second (see benchmarks), and adding a bucket gives it continuous off-machine backups.

Pre-alpha. Fine for side projects and staging; keep a backup of any data you care about.

1. Install the binary

Each release on GitHub carries, for x86_64 and aarch64 Linux (glibc 2.28 or newer: RHEL 8, Debian 10, Ubuntu 20.04 and later), a tarball with the binary and its third-party notices, a CycloneDX SBOM, SHA256SUMS, and a Sigstore bundle for each file. CI builds and signs them (.github/workflows/release.yml); no key is involved, the signature’s certificate names the workflow and the tag. The example components (counter.wasm and the others) are attached too, with THIRD_PARTY_NOTICES-components.txt, the notices of the libraries compiled into them, and the C and Zig SDKs as archives; each has its own bundle, checked the same way. To check a download with cosign 3 (the version that signs them), here for v0.3.0:

f=wasmkeep-0.3.0-x86_64-unknown-linux-gnu.tar.gz
for file in "$f" "$f.sigstore.json"; do
  curl -fsSLO "https://github.com/WasmKeep/wasmkeep/releases/download/v0.3.0/$file"
done
cosign verify-blob --bundle "$f.sigstore.json" \
  --certificate-identity https://github.com/WasmKeep/wasmkeep/.github/workflows/release.yml@refs/tags/v0.3.0 \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com "$f"
tar xzf "$f" && cd "${f%.tar.gz}" && sudo install -m 755 wasmkeep /usr/local/bin/wasmkeep

SHA256SUMS lists every file of the release, for tools that check checksums; it is signed like the rest, so verify its own bundle the same way before trusting it.

Or build on the server, or on a machine with the same OS and CPU architecture:

git clone https://github.com/wasmkeep/wasmkeep && cd wasmkeep
cargo build --release -p wasmkeep-cli
sudo install -m 755 target/release/wasmkeep /usr/local/bin/wasmkeep

Then a system user and its directories:

sudo useradd --system --home-dir /var/lib/wasmkeep --shell /usr/sbin/nologin wasmkeep
sudo install -d -o wasmkeep -g wasmkeep /var/lib/wasmkeep/{classes,data,cache}

Copy your components (built as in each SDK’s README) into /var/lib/wasmkeep/classes/. The unit below serves two: the counter, which a release carries prebuilt (counter.wasm, downloaded like the binary), and a chat room built from examples/chat-room; leave the chat lines out to serve the counter alone.

Configuration and secrets for a class (API keys, webhook URLs: see Configuration and secrets) go in a file only the service can read:

sudo install -d -m 750 -o root -g wasmkeep /etc/wasmkeep
sudo install -m 640 -o root -g wasmkeep /dev/null /etc/wasmkeep/chat.env
sudoedit /etc/wasmkeep/chat.env       # NAME=value lines

2. Run it with systemd

/etc/systemd/system/wasmkeep.service:

[Unit]
Description=WasmKeep
After=network-online.target
Wants=network-online.target

[Service]
User=wasmkeep
Group=wasmkeep
ExecStart=/usr/local/bin/wasmkeep serve \
  --actor chat=/var/lib/wasmkeep/classes/chat_room.wasm \
  --actor counter=/var/lib/wasmkeep/classes/counter.wasm \
  --data-dir /var/lib/wasmkeep/data \
  --cache-dir /var/lib/wasmkeep/cache \
  --env chat=/etc/wasmkeep/chat.env \
  --listen 127.0.0.1:8080
Environment=RUST_LOG=info
Restart=on-failure
# SIGTERM: in-flight requests get up to 10 s, then actors finish their
# queued events (up to 10 s more) and, in a cluster, hand over. Give it room.
TimeoutStopSec=30
LimitNOFILE=65536
# Hardening. Do not add MemoryDenyWriteExecute (Wasmtime maps compiled code
# executable) or a LimitAS (the instance pool reserves a lot of virtual
# address space up front; it is not resident memory).
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
ReadWritePaths=/var/lib/wasmkeep

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now wasmkeep
journalctl -u wasmkeep -f          # logs; RUST_LOG=debug for more
curl localhost:8080/_wasmkeep/health   # ok
curl localhost:8080/_wasmkeep/ready    # ready (see docs/monitoring.md)
curl -X POST localhost:8080/actor/counter/hello

The first start compiles each component and caches the result in --cache-dir; later starts take a fraction of a second.

3. Put a reverse proxy in front

WasmKeep speaks plain HTTP/1.1 and listens on localhost by default. Let a reverse proxy handle TLS and public traffic. With Caddy (automatic HTTPS, WebSockets included):

actors.example.com {
    reverse_proxy 127.0.0.1:8080
}

With nginx, WebSockets need the upgrade headers:

location / {
    proxy_pass http://127.0.0.1:8080;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host $host;
    proxy_read_timeout 1h;   # idle WebSockets
}

Every actor is reachable by anyone who knows its address, and a request to a new name creates that actor. Put authentication in your actors, or in the proxy. In an actor: give the class a token with --env, and compare the Authorization header with it in fetch without stopping at the first differing byte (Watch’s hub does this, examples/watch/hub). Calls between actors carry x-wasmkeep-caller: <tenant>/<class>/<name>, which the host sets and strips from outside requests, so an actor can accept some changes only from another class.

Classes whose actors only other actors should create (a per-user actor created at sign-up, a worker the scheduler starts) take --access CLASS=existing: a request for an unknown name answers 404 and creates nothing, so no one can fill the disk one name at a time. --access CLASS=internal keeps a class away from outside requests entirely.

Actors’ outbound HTTP reaches public addresses only. If they must call a service on this machine or your private network, allow it explicitly, as narrowly as you can: --allow-outbound 10.0.5.20 (repeatable; ranges like 10.0.5.0/24 work too). Never allow 169.254.169.254: on a cloud VM it hands out the machine’s credentials. If that service’s HTTPS certificate comes from a private CA, add --outbound-ca /etc/wasmkeep/internal-ca.pem. Ports other protocols use (mail on 25, 465 and 587, SSH, FTP and the rest of the list browsers block) are refused too; --allow-outbound-port 587 opens one, for a mail relay your actors call over HTTP.

4. Back up the data

Each actor is one SQLite file: <data-dir>/actors/<tenant>/<class>/<name>.db. Two ways to keep copies:

Continuous, to a bucket (recommended). Run the node as a one-node cluster: every event’s writes are uploaded to an S3-compatible bucket (AWS S3, Cloudflare R2, MinIO, anything with conditional writes), and a lost disk is restored from it on the next request to each actor. Add to ExecStart:

  --node-id vps-1 --advertise http://127.0.0.1:8080 --bucket s3://my-bucket/wasmkeep \
  --cluster-listen 127.0.0.1:7443 --cluster-advertise https://127.0.0.1:7443

With --node-id, a start also checks its node id in the bucket before it listens, for up to 15 s: all of it the first time or after a crash, and after a clean stop only what is left of 15 s since the node stopped (why).

The last two flags give the node a cluster listener on loopback: nodes talk to each other over mutual TLS, and a node in a cluster refuses to start without one (or without --cluster-plaintext accept). Alone, it carries nothing; when a second server joins, move it to an address the other nodes reach.

Then give the unit the bucket credentials (better: from an EnvironmentFile= readable only by root):

Environment=AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... AWS_REGION=auto
Environment=AWS_ENDPOINT=https://<account>.r2.cloudflarestorage.com

The bucket is then the only copy that survives the machine, and it needs a backup of its own: Back up and restore a cluster’s bucket.

Use exec-durable for writes that must survive the machine. Growing to more servers later means starting them on the same bucket, with --advertise set to an address the others can reach (three nodes).

Snapshots. Without a bucket, copy the files with SQLite’s online backup, which is safe while the server runs:

cd /var/lib/wasmkeep/data
find actors -name '*.db' | while read -r db; do
  mkdir -p "/backup/$(dirname "$db")"
  sqlite3 "$db" ".backup '/backup/$db'"
done

A plain cp of a live database can produce a corrupt copy.

5. Updating

• New component version: replace the .wasm and systemctl restart wasmkeep. Each actor runs on-init again on its next event, so migrations happen lazily, per actor.
• No-restart deploys need a bucket: wasmkeep deploy --app acme/shop ... uploads a version, and nodes started with --app acme/shop switch to it within seconds, actor by actor (see Running a cluster).
• New WasmKeep binary: install it and restart. Open WebSockets are closed; clients reconnect.

Sizing

Memory is spent on resident actors only: about 250 KB each for a Rust actor and 740 KB for a TypeScript one on QuickJS, the default engine (see benchmarks); hibernated actors (idle for 30 s) cost nothing but disk. These figures assume transparent huge pages off, which wasmkeep serve arranges for itself (monitoring); with --transparent-huge-pages system on a kernel set to always, expect about twice the memory. At most --max-instances actors (1,000 by default) are resident at once; raise it if more are active within the idle timeout, and budget the memory. CPU scales with request rate.

Three nodes

A WasmKeep cluster: three servers sharing one S3-compatible bucket, behind a load balancer. Any node can lose its machine; its actors continue on the others 15 to about 21 s after its last heartbeat, then a restore (how), with no acknowledged durable write lost (measured: twelve hours and 35 kills on one lease per node, writable again 18.2 s after the kill at the median).

Read Running a cluster for how ownership, forwarding and replication work. This page is the setup. Each node is installed as in One server (binary, user, systemd unit, directories); only the flags differ.

Pre-alpha. The cluster is tested in simulation, with real processes against an S3 API, and against Cloudflare R2 and OVHcloud Object Storage with nodes killed under load (measured), but not yet under production load.

To try a cluster on one machine first, deploy/compose/ runs the three nodes below in containers against MinIO, with the same flags.

                 ┌──────────── load balancer (TLS) ────────────┐
                 ▼                       ▼                     ▼
          node a 10.0.0.11        node b 10.0.0.12      node c 10.0.0.13
                 └─ private network: mutual TLS on port 7443 ──┘
                                         │
                            S3-compatible bucket (shared)

1. The bucket

Any S3-compatible store that supports conditional writes (If-None-Match and If-Match on PUT): AWS S3, Cloudflare R2, OVHcloud Object Storage, MinIO and others. Each node checks this at start and refuses to run on a store that ignores them (Running a cluster). Create one bucket (or a prefix in one) for the cluster, and credentials that can read, write, list and delete in it.

Connection settings by provider

Every setting is an AWS_* variable in the nodes’ environment (the EnvironmentFile= below); the bucket name and an optional prefix go in --bucket s3://BUCKET/PREFIX. Keep the bucket in the same region as the nodes: a durable write waits for one PUT to it, and a failover for many reads (54 ms per durable write next to the bucket, 587 ms from a distant desktop to R2: measured).

“Tested” means the storage, failover and cluster suites and a 30-minute soak passed against it. Every node also checks conditional writes at start, whatever the provider. Requests cost money on R2 and S3 (before ADR 0020, per-actor lease renewals were most of them: see the benchmarks; node leases removed them), none on OVHcloud.

The bucket is the cluster’s source of truth and its root of trust: whoever can write to it can run code on the cluster (deployed components live there) and impersonate nodes (the inter-node secret lives there). Give its credentials to the nodes and to whoever deploys, no one else.

2. The nodes

Every node needs:

• a unique node id ([a-z0-9-]): a, b, c. A node checks its id at start: it watches nodes/<id> in the bucket for up to about 15 s (less after a clean stop or a fence) before taking any actor, and exits with an error if another process still writes it. Two processes started at the same moment can both pass that check; one then finds the other at its next write and exits, but until then both may run the same actor. When replacing a machine, stop the old process before starting the new one under the same id (see Running a cluster).
• an advertise URL the other nodes can reach it at, on the private network.
• a cluster listener (--cluster-listen and --cluster-advertise): a second port, 7443 here, where the nodes reach each other over mutual TLS. Nothing to provision: each node makes its own key at start and the others learn its fingerprint from the bucket (see Running a cluster). Without it a node refuses to start, unless given --cluster-plaintext accept, which makes nodes talk to each other in plain HTTP (a private network only).
• the same --bucket, and either the same --actor flags on every node or, better, --app to follow deploys.

On node a (/etc/wasmkeep/env, mode 600, then EnvironmentFile= in the unit):

AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...
AWS_REGION=us-east-1
# AWS_ENDPOINT=...   # R2, OVHcloud, MinIO: see "Connection settings by provider"
RUST_LOG=info

ExecStart=/usr/local/bin/wasmkeep serve \
  --app acme/shop \
  --bucket s3://acme-wasmkeep/prod \
  --node-id a \
  --advertise http://10.0.0.11:8080 \
  --listen 10.0.0.11:8080 \
  --cluster-listen 10.0.0.11:7443 \
  --cluster-advertise https://10.0.0.11:7443 \
  --data-dir /var/lib/wasmkeep/data \
  --cache-dir /var/lib/wasmkeep/cache

Nodes b and c are the same with their own id and address. Listening on the private address (not 0.0.0.0) keeps the nodes off the public internet: only the load balancer should reach port 8080, and only the other nodes port 7443 (firewall it to 10.0.0.11 to 10.0.0.13; never put it behind the load balancer). Node traffic is encrypted, and 8080 carries no node calls at all: plain HTTP between nodes is refused by default (--cluster-plaintext refuse), so a signed call arriving there is treated as a client’s. A cluster that ran in accept mode (without listeners) moves to this in two rounds (see Operating it below); while any node runs with --cluster-plaintext accept, keep 8080 private too.

Clocks should be synchronised (NTP, as on any cloud VM). The lease timings tolerate a few seconds of skew between nodes.

3. Deploy the application

--app nodes serve whatever was last deployed for the app, so deploy before starting them (from any machine with the bucket credentials). This page uses the counter from examples/counter as the app’s one actor class:

cd examples/counter && cargo build --release --target wasm32-wasip2
wasmkeep deploy --app acme/shop --bucket s3://acme-wasmkeep/prod \
  --actor counter=target/wasm32-wasip2/release/counter.wasm

deploy compiles and checks every component before uploading anything. Later deploys work the same way: nodes pick the new version up within a few seconds and switch each actor at its next event; --rollback returns to the previous version. See Deploying.

4. The load balancer

Send traffic to all three nodes, health-checking GET /_wasmkeep/ready (200 ready; 503 while the node shuts down, cannot reach the bucket or runs out of disk, see Monitoring). A request can go to any node: if another node owns the actor, it is forwarded there (one hop), and responses carry x-wasmkeep-node so you can see where an actor runs. WebSockets work the same way.

No stickiness is needed. Routing by actor path (hashing /actor/<class>/<name>) saves the forwarding hop, and is an optimisation, not a requirement: see the front door for HAProxy and nginx configurations, health checks, WebSocket timeouts, client addresses and WAF placement.

5. Check it

curl -i -X POST https://actors.example.com/actor/counter/alice
#   x-wasmkeep-node: b        (whichever node took the actor)
#   1

Then stop that node (systemctl stop wasmkeep on b) and repeat the request: another node answers at once with 2, the value preserved (a clean stop hands the node’s actors over). Kill it instead (kill -9) to see a crash: the actor comes back 15 to about 21 s after b’s last heartbeat (10 s lease plus 5 s grace, then a peer fences it), plus a restore from the bucket. Restart b: it rejoins and takes on new actors.

From any machine with the bucket credentials, wasmkeep cluster status --bucket s3://acme-wasmkeep/prod lists the nodes from their heartbeats (live or not, version, addresses, TLS) and the deployed version of the app; it exits with 1 if no node is live. See Checking a cluster.

Operating it

• Turning on the cluster listener in a cluster that runs with --cluster-plaintext accept and no listeners: add the two flags (keep accept for now) and restart one node at a time; nodes with and without it keep talking to each other meanwhile. A restarted node reports not ready for a few seconds while its new fingerprint reaches the others.

• Refusing plain HTTP between nodes, once every node has a cluster listener (wasmkeep cluster status shows a cluster address and a pin for every live node, and no warning about plain nodes): remove --cluster-plaintext accept and restart one node at a time; each logs that it refuses plain HTTP. Never do it while any node lacks a listener: a refusing node does not reach that node’s actors (503), and that node’s calls to it are treated as a client’s. To roll back, add accept to every node before removing any listener.

• Upgrading WasmKeep (the binary, not your app): upgrade every node together, not one at a time. systemctl stop wasmkeep on all three, wait until wasmkeep cluster status --bucket s3://acme-wasmkeep/prod exits with 1 (no live node, about 10 s after the last stop), install the new binary on all three, start them, and check cluster status again: three live nodes, one version, no warning. Clients get errors between the last stop and the first start. Old and new nodes refuse each other’s calls and handle alarms differently, so a rolling restart across versions is not supported yet: see Upgrading WasmKeep.

• Restarting a node on the same binary (a configuration change, a host reboot): one node at a time is fine. A stopping node finishes its in-flight requests, then hands its actors over: the other nodes serve them on their next request. WebSockets to that node close and clients reconnect.

• Updating the stateless component (the file given to serve for paths outside /actor/ and /_wasmkeep/): it is not part of a deploy. Copy the new file to every node and restart them one at a time (see Updating the stateless component).

• Adding a node: start it with a new id and add it to the load balancer. An actor in memory on another node stays there, and the new node forwards its requests; once it hibernates, its node gives it up, and its next request starts it on whichever node that request reaches, the new one included (see How it scales).

• Secrets: keep them in the bucket rather than in each node’s --env files. Create a key once (wasmkeep secrets new-key --out secrets.key), give it to every node (--secrets-key /etc/wasmkeep/secrets.key, mode 0600), and set values with wasmkeep secrets set; nodes pick them up within a few seconds (see Secrets in the bucket).

• Removing or replacing a node: drain it first, then stop it once it is drained:

  wasmkeep cluster drain b --bucket s3://acme-wasmkeep/prod
  wasmkeep cluster status --bucket s3://acme-wasmkeep/prod   # STATE drained
  sudo systemctl stop wasmkeep                               # on node b
  

  A drain moves its actors a few at a time while the node keeps forwarding, and moves their pending alarms with them, so they fire on time (see Draining a node). Stopping without draining also works (SIGTERM, as systemctl stop sends): its actors move at once, and alarms of idle actors wait for the sweep. A node that is killed or crashes cannot hand over: its actors move after a peer fences its heartbeat (15 to about 21 s after its last write) and a restore, and their alarms arrive later still (measured).

• Stop timeouts: a clean stop can take up to about 29 s (10 s for in-flight requests, 10 s for actors to finish their queued events, 3 s for the handover to reach the bucket, about 6 s for the node to release its own lease). Allow 30 s: TimeoutStopSec=30 in systemd, --stop-timeout 30 for Docker (its default is 10 s), terminationGracePeriodSeconds: 30 in Kubernetes. A node killed earlier is treated as a crash, which is safe, only slower.

• Losing a node: nothing to do. Its actors move on their next request; writes acknowledged by exec-durable are in the bucket, a plain exec’s last event may be lost (see Running a cluster).

• Logs: each node logs to the journal; RUST_LOG=wasmkeep_server=debug shows leases, forwarding and replication decisions.

• Backups: the bucket is the only durable copy. Copy it somewhere the nodes cannot write, and restore that copy once before you need it (see Back up and restore a cluster’s bucket).

• Bucket growth: each actor’s history older than an hour (--history-retention-secs) is collected after its periodic snapshot, a deleted actor’s at once, and wasmkeep deploy keeps the current version and the five before it (see History in the bucket).

One region

All nodes should be in one region, close to the bucket: every durable write and every ownership change is a round trip to it. Multi-region placement is on the roadmap, not in this release.

The front door: load balancer, WAF and routing

How traffic reaches a cluster on dedicated servers or virtual machines.

clients ── HTTPS, WebSockets
   │
   ▼
WAF / CDN (optional)        Cloudflare, or Coraza/ModSecurity at the proxy
   │
   ▼
load balancer               HAProxy, nginx, Caddy or a cloud LB: terminates TLS
   │  plain HTTP to any node's --listen port, on the private network
   ▼
node a     node b     node c       any node takes any request;
   └── forwards to the actor's owner, at most one hop ──┘
                    │
                 bucket            leases, heartbeats, replication

• TLS ends at the load balancer. Nodes serve plain HTTP on --listen; bind it to the private network.
• Any node will do. If another node owns the actor, the request is forwarded there (once, never twice), and the response says which node served it (x-wasmkeep-node).
• Health-check GET /_wasmkeep/ready: 503 while a node shuts down, cannot reach the bucket or is nearly out of disk (monitoring). /_wasmkeep/health only says the process answers.
• WebSockets need the upgrade passed through and a tunnel or idle timeout above 30 s; the usual default of 60 s works (WasmKeep pings a socket whose client has been silent for 25 s, and closes it 30 s later if there is no answer).
• A server timeout above the failover wait. When a node dies, the others keep a GET, a HEAD or a WebSocket upgrade for one of its actors until they took the actor over, up to 25 s by default (--failover-wait-secs, see what a caller sees when a node dies), and a kept request sent on to the node that took the actor can wait 30 s more there. A load balancer that gives up on the node sooner turns that wait into its own 504: give it 60 s or more for the node’s answer, or lower the wait on the nodes to fit.
• Tell the node when the client is gone. A kept request is dropped, and never run, when its connection closes. If the load balancer keeps its connection to the node open after its own client left, the node cannot know, and runs the request for nobody once the actor is taken over.
• Never expose the metrics listener (--metrics-listen) or the node-to-node traffic to the internet.

Route by actor (recommended)

Hash the actor’s path, /actor/<class>/<name>, so that every request for an actor reaches the same node. The first node to get a request for an actor nobody owns takes its lease, so actors settle where their traffic arrives and forwarding all but disappears. It is only a hint: when nodes come and go, ownership still follows the leases, and a request that lands elsewhere is forwarded.

HAProxy: balance uri depth 3 hashes the first three path segments (/actor, the class, the name):

frontend public
    bind :443 ssl crt /etc/haproxy/site.pem alpn http/1.1
    # Client IP for actors: replace whatever the client sent.
    http-request del-header X-Forwarded-For
    option forwardfor
    default_backend wasmkeep

backend wasmkeep
    balance uri depth 3
    hash-type consistent
    option httpchk GET /_wasmkeep/ready
    http-check expect status 200
    # Above the failover wait (25 s) plus a forward to the new owner.
    timeout server 60s
    timeout tunnel 1h
    # A node that refuses the connection: try another one.
    retries 2
    option redispatch
    # The client left while its request waited: close the node's side too.
    option abortonclose
    default-server inter 2s fall 2 rise 2
    server a 10.0.0.11:8080 check
    server b 10.0.0.12:8080 check
    server c 10.0.0.13:8080 check

timeout server is how long HAProxy waits for a node’s answer: it must be longer than the nodes’ failover wait, or HAProxy answers 504 while the node still holds the request. option redispatch with retries sends a request whose connection a node refused (it just died, and its health check has not failed yet) to another node, where it can be kept, instead of failing it; it applies to connections that could not be opened, so no request is sent twice. option abortonclose asks HAProxy to give up a request whose client closed its connection before an answer, and so to close its connection to the node: that is what lets a node drop a kept request nobody waits for. This example was not run against HAProxy with a kept request: check with your version that a client’s departure closes the node’s side while HAProxy waits for the answer (the node then counts outcome="left" in wasmkeep_failover_waits_total). Another proxy needs its own equivalent; without one, a kept GET runs after the takeover for a client that has left.

nginx: hash a variable holding the actor’s path:

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}
map $uri $wasmkeep_actor {
    ~^(/actor/[^/]+/[^/]+)  $1;
    default                 $uri;
}
upstream wasmkeep {
    hash $wasmkeep_actor consistent;
    server 10.0.0.11:8080;
    server 10.0.0.12:8080;
    server 10.0.0.13:8080;
}
server {
    listen 443 ssl;
    location / {
        proxy_pass http://wasmkeep;
        proxy_http_version 1.1;
        proxy_set_header Host $host;                       # the client's, not the upstream's name
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header X-Forwarded-For $remote_addr;     # replace, never append
        proxy_read_timeout 1h;
    }
}

Open-source nginx has no active health checks: pair it with an external check of /_wasmkeep/ready, or use HAProxy. Caddy is the simplest for one node (see one server); for a cluster its lb_policy uri_hash hashes the whole URI, so requests to different paths of one actor may land on different nodes (correct, with a forwarding hop).

Client addresses

WasmKeep passes request headers through to actors unchanged, except its own x-wasmkeep-* (stripped or set) and, when a node forwards a request to the actor’s owner, hop-by-hop headers (Connection, Keep-Alive, TE, Upgrade and the like, and any header Connection names; RFC 9110 section 7.6.1). An actor that reads X-Forwarded-For trusts whatever the load balancer put there: the load balancer must replace the header, not append to it, and nodes must be reachable only through it.

Static files

Serve static files (HTML, JavaScript, CSS, images, fonts, video, documents) from the front door, not from actors: the reverse proxy itself, or object storage behind a CDN. Route actor paths (/actor/...) to WasmKeep, and everything else to the static origin.

Actors are for state with an identity: one instance per name, one event at a time, a lease and a database. A file has neither. Served by an actor, every request would go through the instance pool and the Wasm sandbox, take the actor’s lease (so one node serves it, not all of them), and the guest would have to implement caching headers (ETag, 304 answers) and range requests (needed for video) itself. A web server or CDN does all of that from disk or cache at almost no CPU cost.

Caddy, files from a directory and actors on the nodes:

example.com {
    @actors path /actor/* /_wasmkeep/*
    handle @actors {
        reverse_proxy 10.0.0.11:8080 10.0.0.12:8080 10.0.0.13:8080 {
            health_uri /_wasmkeep/ready
        }
    }
    handle {
        root * ./public
        file_server
    }
}

With a CDN, the same split works: the static paths go to a bucket or static host, the rest to your load balancer.

A few small files embedded in a component (an admin page, a favicon) are fine; anything larger, changing often, or large media belongs on the static origin. Serving assets directly from the nodes, without an actor, is a possible built-in for later (roadmap).

WAF and rate limits

Filter and rate-limit at the edge: a CDN such as Cloudflare, or a WAF at the proxy (OWASP Core Rule Set with Coraza or ModSecurity), and per-client rate limits in the load balancer (HAProxy stick tables, nginx limit_req). WasmKeep’s own limits are a backstop, not a WAF: a connection cap, 30 s to send headers, 16 MB request bodies, WebSocket message and socket caps (security model). Keep the edge’s body limit at or below 16 MB.

Without a separate load balancer

Every WasmKeep node already routes: whichever node a request reaches, it is served by the actor’s owner. What a load balancer adds is one address that survives a node dying, TLS, and filtering. The three servers can provide those themselves.

DNS round robin with Caddy

Publish one name with every node’s public address (three A or AAAA records, short TTL), and run Caddy beside WasmKeep on each node. WasmKeep listens on the node’s private address (--listen 10.0.0.11:8080, the address the other nodes forward to); Caddy takes the public one:

app.example.com {
    reverse_proxy 10.0.0.11:8080 {
        header_up X-Forwarded-For {remote_host}   # replace, never append
    }
}

Each node gets its own certificate. Requests for an actor owned elsewhere are forwarded by WasmKeep over the private network.

A floating IP with keepalived

Most providers sell an address you can move between servers (Hetzner, OVH, Scaleway, DigitalOcean reserved IPs, or a VRRP address on your own network). keepalived on each node holds it on one healthy node and moves it within seconds when that node or its WasmKeep stops answering. HAProxy on every node (configured as above, with routing by actor and /_wasmkeep/ready checks against all three nodes) terminates TLS on the floating address:

vrrp_script wasmkeep_ready {
    script "/usr/bin/curl -fsS --max-time 2 http://10.0.0.11:8080/_wasmkeep/ready"
    interval 2
    fall 2
    rise 2
}

vrrp_instance front {
    state BACKUP              # all nodes start as backup; priority decides
    interface eth1            # the private network
    virtual_router_id 51
    priority 100              # 100, 90, 80 on the three nodes
    advert_int 1
    unicast_peer { 10.0.0.12 10.0.0.13 }   # the other nodes
    virtual_ipaddress { 203.0.113.10/32 dev eth0 }
    track_script { wasmkeep_ready }
}

On providers whose floating IP is moved through an API (Hetzner Cloud, DigitalOcean), call it from keepalived’s notify_master script instead of announcing the address with virtual_ipaddress. Bind HAProxy to the floating address with net.ipv4.ip_nonlocal_bind = 1, so every node can start it before holding the address.

An outbound tunnel

Some providers reach your servers through a connector: a process on each machine that dials out to the provider and receives the traffic for your name over that connection. No inbound port is opened, the nodes need no public address, and TLS ends at the provider. Run one connector on each node, all for the same name; the provider spreads requests over the connectors that are connected.

The obvious configuration is wrong for a cluster. A connector that sends its traffic to the node on its own machine (http://10.0.0.11:8080) has that node as its only origin. The connector is its own process: it stays connected while WasmKeep on that machine is stopped, restarting or not ready, so it keeps receiving its share of the requests, and answers each with a gateway error for as long as its node is down. The other nodes are healthy the whole time and get none of that share.

Put this guide’s load balancer between each connector and the nodes, bound to loopback. HAProxy on every node, with the wasmkeep backend above (all three nodes, checked on /_wasmkeep/ready, hashed by actor), and this frontend in place of the public one:

frontend tunnel
    # Only the connector on this machine connects here.
    bind 127.0.0.1:8081
    # Whatever the client sent; see "Client addresses" below.
    http-request del-header X-Forwarded-For
    default_backend wasmkeep

Point each connector at http://127.0.0.1:8081. A request now reaches a node that is ready whichever connector received it, and the same node for the same actor from all three, as long as every HAProxy lists the same servers in the same order (the consistent hash depends on it; not exercised here). A machine that is down takes its connector down with it, and the provider stops using that connector.

• TLS is the provider’s: no certificate on the nodes, so no ssl on the bind line.
• The nodes still listen on the private network (--listen 10.0.0.11:8080): the HAProxy of the other machines sends to them there. Nodes forward to each other on the cluster listener (port 7443 in the three-node guide), which must stay reachable between the machines.
• Client addresses. HAProxy sees the connector, at 127.0.0.1, so option forwardfor would write that. The client’s address is in whatever header the provider sets: read that one in your actors. The frontend deletes X-Forwarded-For, which may be the client’s own invention; keep it only if your provider replaces what the client sent. An actor may trust the provider’s header only while nothing but the connectors can reach HAProxy and the nodes (Client addresses).
• WebSockets need the provider to pass the upgrade and to keep an idle connection open for more than 30 s, as for any proxy in front.

Plain DNS round robin, no proxy at all

Not recommended: nodes speak plain HTTP, so there is no TLS without a proxy in front, and WasmKeep has no rate limiting or filtering of its own beyond its protective limits.

Back up and restore a cluster’s bucket

The bucket is the cluster’s only durable copy of everything: a node’s disk is a cache of it (replication). This page says what is in the bucket, how to copy it so that the copy can be restored, how to restore it, and how to check that it works.

Pre-alpha, and a guide without a tool. WasmKeep has no command to back up a bucket, to verify one or a copy of one, to restore one, or to export or restore a single actor. Verifying a bucket and exporting an actor are on the roadmap. Until then a backup is a copy made with an S3 tool, and the only proof that it restores is a restore drill (below).

What is in the bucket

Everything under the prefix given to --bucket s3://BUCKET/PREFIX:

History objects are deleted only by garbage collection, by the node that holds the actor, after a newer snapshot of its own was acknowledged, and only once they are older than --history-retention-secs (one hour by default). A deleted actor’s history is erased at once (History in the bucket, ADR 0012).

Not in the bucket, to keep with the backup:

• the secrets key (--secrets-key). Without it the sealed secrets in a copy cannot be opened;
• each node’s --env files, the stateless component if you serve one, and the serve command lines (the systemd units);
• with --actor instead of --app, the components themselves.

A copy holds every actor’s data and the cluster secret in the clear. Protect it like the bucket (security model).

What a restorable copy needs

To restore an actor, a node lists the actor’s prefix, takes the newest segment of the highest generation as the head, and walks back from it, segment by segment, to the first snapshot on the way (ADR 0007). So a copy restores an actor if it holds that snapshot and every segment after it up to the head. Objects besides those do no harm. A segment missing in the middle is an error, not a rollback: the actor answers 500 until the history is repaired, and the other actors are not affected.

With the cluster stopped: restorable

Stop every node cleanly and wait until wasmkeep cluster status exits with 1 (no live node). A clean stop gives each actor up to 10 s to finish its queued events and their uploads before the node gives it up. A node that logs shutdown: some actors did not finish in time cut some events short: a plain exec among them may not be in the bucket, and is then not in the copy either. Once every process has exited nothing writes or deletes any more, as long as nobody runs wasmkeep deploy or wasmkeep secrets meanwhile, and the copy is the bucket at one moment. This is the one way the code shows to be restorable. Its cost is the cluster being down while the copy runs.

With the cluster running: not established

A copy made object by object reads a listing first and the objects afterwards, while nodes keep writing and collecting. From the code:

• Segments and snapshots are never changed after they are created, so an object that was copied is a good object.
• Garbage collection can delete an object between the listing and its copy. Take an actor whose last snapshot is older than the retention window. It reaches 64 writes during the copy, uploads a new snapshot and deletes the old one with the segments after it that are older than the window. The copy then has the newer segments the listing named, without the snapshot they start from and without the new one, which the listing did not name. That actor does not restore from that copy.
• A second copy into the same destination, started right after the first and without deleting anything there, brings the new snapshot and what follows it. Everything created since the first listing is younger than the retention window, so collection cannot delete it while the second pass runs, if both passes end within the window (an actor deleted meanwhile is erased at once, whatever the window). Extra objects in the destination do no harm to a restore. By that reasoning two passes into an empty destination close the gap for history. That is reasoning, not a test: nothing in the repository restores a cluster from such a copy.
• The other objects are copied at other moments than the history. An ownership record copied before a takeover, next to segments copied after it, carries a lower generation than those segments, and the next hold would use a generation that is already in the history. An alarm index copied before or after its actor’s database can list an alarm the database does not have, or miss one it has. What a cluster started on such a copy does in each of these cases is not established.

So: a copy of a running cluster is better than none, and most actors in it restore. It is not a backup the code shows to be restorable. If the cluster cannot stop, bucket versioning is the other way, with unknowns of its own.

Bucket versioning

With versioning turned on in the bucket (AWS S3, and the S3-compatible stores that implement it), a delete or a replace keeps the earlier version of the object. Collection then removes nothing for good, and the state of every key at one past moment can be rebuilt from the versions, without stopping anything. The state at one moment is what a cluster whose nodes all died at that moment would leave: nodes that die give nothing up, and the next ones take their actors from the bucket, which is what failover does.

What it costs and what is not checked:

• Rebuilding “every key as it was at time T” into another prefix is a job for the provider’s tools or a script over the version listing. WasmKeep has no command for it, and nothing in the repository runs a cluster on a versioned bucket or restores one from versions: the paragraph above is what versioning promises, not something measured here.
• Nothing is deleted any more, so the bucket grows until a lifecycle rule expires old versions. That rule’s age is how far back you can restore.
• A deleted actor’s data stays in the old versions until that rule removes it. delete-self erases nothing for good before then.

An example copy

An example with rclone, one S3 tool among several. It is an example, not a tested procedure: nothing in the repository runs it, against any provider. Check how your tool lists, retries and verifies on your store, then prove the copy with a restore drill.

# The cluster is stopped: this must print no live node and exit with 1.
wasmkeep cluster status --bucket s3://acme-wasmkeep/prod

# Copy the prefix to another bucket, into a new folder per day.
rclone copy src:acme-wasmkeep/prod dst:acme-wasmkeep-backup/2026-11-12/prod

# Compare sizes and checksums of both sides.
rclone check src:acme-wasmkeep/prod dst:acme-wasmkeep-backup/2026-11-12/prod

Keep the copy in another account or at another provider than the bucket, with credentials the nodes do not have: whoever can write the bucket can also delete it. Start the nodes again when the check has passed.

Restoring

1. Stop every node and wait until wasmkeep cluster status exits with 1. Never restore under running nodes: they hold a newer deploy and newer secrets in memory than the copy names. They keep running the newer deploy, and they refuse the older secrets: 503 for the app after 30 s, until they are restarted (Secrets in the bucket).

2. Copy the backup into an empty prefix, not over the damaged one. Objects written after the backup carry higher positions and generations than the copy’s, and a restore takes the highest as the head: left in place, they win over what you copied back. Keeping the damaged prefix also keeps a way back.

   rclone copy dst:acme-wasmkeep-backup/2026-11-12/prod src:acme-wasmkeep-restored/prod
   

   Keep the prefix’s name (prod here) and change the bucket if the app has secrets in the bucket: a sealed secret opens only under the prefix, tenant and app it was sealed for. Under another prefix, write the secrets again with wasmkeep secrets reset.

3. Move each node’s data directory aside (--data-dir), or decide to keep it knowingly. A node replaces a local database that is not at the bucket’s head, keeping the old file as <name>.db.replaced (one generation only: a second replacement overwrites it). But for an actor the bucket knows nothing about, the local database becomes its first snapshot: with the old directories in place, actors created after the backup come back from disk, on whichever node has their file.

4. Start the nodes with --bucket naming the restored prefix, and with the same node ids as before.

5. Check: wasmkeep cluster status lists every node live, and the deployed version and seq of the backup. Request a few actors you know.

What a restored cluster is:

• The state of the backup. Every write since is gone, durable or not.
• The deploy and the secrets of the backup. apps/.../current names the version that was current then. Deploy again for a newer one, and set again any secret changed since.
• Free to take. In a copy made with the cluster stopped, the node leases say released, and a record that still names one of those nodes is free: the first request for an actor starts it. In a copy made while nodes ran, the leases say live and records name the processes of that time. A node started under the same id replaces its lease after the start check (about 15 s when the copy’s lease says live), which frees the records naming its old process. An id nobody starts again is fenced by the other nodes, or with wasmkeep cluster fence <node>.
• Alarms that came due since the backup fire soon after the start, and anything the actors did outside between the backup and the restore (a webhook sent, an email, a call to your API) is unknown to the restored state and may happen again. Alarms are at least once; a restore makes that visible.
• Drain requests of the backup apply again: a node that was draining when the copy was made drains again. wasmkeep cluster undrain <node> withdraws it. A move request in the copy may apply again too, for up to 5 minutes after the restore copy: a node counts a move request’s age from the object’s date in the bucket, which a copy normally sets anew (not tested). It expires by itself.

One actor cannot be restored alone today: there is no command for it, and copying one actor’s old objects under a running cluster changes nothing, since its current history has the higher positions.

A restore drill

A backup that has never been restored is a hope. Do this before you need it, and again after every WasmKeep upgrade:

1. Copy last night’s backup into an empty prefix of a scratch bucket, with the same prefix name.
2. Start one node on it, on a machine that is not part of the cluster, with an empty data directory, the same app and the secrets key: wasmkeep serve --app acme/shop --bucket s3://scratch/prod --node-id drill .... Restored actors fire their overdue alarms and call out, and outbound HTTP reaches public addresses by default: if the actors send webhooks or call an API, first block that machine’s outbound traffic except to the scratch bucket’s endpoint (a firewall rule that allows only the bucket), and give it no --allow-outbound. The node itself must still reach the bucket: without it, it never starts serving.
3. Request a sample of actors, including the largest databases and some that had not been written for a long time. A 500 is an actor that did not restore: look at the node’s log for the missing object.
4. Compare what they answer with what you expect at the time of the backup.
5. Write down how long the copy and the first requests took. That is your restore time.

The drill reads every object only for the actors it requests. Until a verify command exists, the sample is the measure of the copy.

Monitoring

What a node tells you about itself: two health endpoints on the main port, and Prometheus metrics on a port of their own.

Health and readiness

A node is not ready once it starts shutting down; in a cluster, until it has checked that the bucket enforces conditional writes and that no other process runs with its node id (for good if either fails: serve exits), while it has not loaded the cluster secret or its node lease (the heartbeat, ADR 0020) is not valid, which takes the bucket refusing or not answering its writes for the lease’s ttl minus its safety margin (8 s by default; calls and durable writes would fail); and when the disk holding --data-dir has less than 1 GiB free (or 5% of a smaller disk), sampled every 5 s off the request path. The public answer gives no reason (the port faces clients); wasmkeep_not_ready{reason} on the metrics page does.

A node that stops for good says why in the same words in that reason and at the start of serve’s exit line, followed there by the details:

Point load balancers at /_wasmkeep/ready, and use /_wasmkeep/health for “restart it if it stops answering”.

Open files

Every resident actor holds three open files (its database, write-ahead log and shared-memory index), every connection one. At start, wasmkeep serve raises its soft open-files limit to the hard limit, then caps connections and resident actors to what fits, logging a warning when that lowers them. With a cluster listener (--cluster-listen), its connections (up to 1,024) are counted first. Give the process a generous limit (LimitNOFILE=65536 in the systemd unit, or ulimit -n) to serve many actors at once.

The resident cap includes actors still closing their databases and blocking database work left by a cancelled event. An address without a mailbox (including a hibernated or failed-over actor) waits up to 250 ms for an idle actor’s resources to close. If capacity remains full, its request receives 503 Service Unavailable with Retry-After: 1. Registered mailboxes can still receive events, subject to their own queue limit (429). Relaying to another owner does not take a local actor slot when the owner is known (learned in the last second, or read from the actor’s ownership record within the admission wait); otherwise the request goes through local admission. Clients should honor the retry delay and bound their retries.

Alarms refused by admission use exponential backoff with jitter: a delay between half and all of a ceiling that starts at 100 ms and reaches 5 s. The retry counter below makes sustained admission pressure observable. WebSocket messages retain their frame while waiting up to the event wall limit; close callbacks remain pending through capacity or queue pressure. Closing sockets keep counting against socket limits until cleanup ends.

This admission rule uses the startup budget; it is not a live count of all process descriptors. Unrelated files and outbound connections still need the budget’s reserved headroom. The resident metric below counts registered mailboxes, so it can briefly fall before retiring resources release capacity.

Memory

Each resident actor costs about 250 KB for Rust and 740 KB for TypeScript on QuickJS (benchmarks); wasmkeep_actors_resident times that, plus the compiled classes, is roughly the process’s resident memory.

On Linux, wasmkeep serve and wasmkeep dev turn transparent huge pages off for their own process. Where the kernel sets them to always (some desktop and tuned distributions do; most servers use madvise), each actor’s memory would otherwise sit on 2 MiB pages it mostly does not use: on one such machine, a Rust actor measured 577 KB instead of 250 KB, with no gain in CPU or latency. To check a running node:

cat /sys/kernel/mm/transparent_hugepage/enabled   # the kernel's setting
pid=$(pidof wasmkeep)
grep THP_enabled /proc/$pid/status                 # 0: off for WasmKeep

--transparent-huge-pages system keeps the kernel’s setting instead. If the process cannot turn them off, it logs a warning and carries on.

Metrics

wasmkeep serve ... --metrics-listen 127.0.0.1:9100
curl 127.0.0.1:9100/metrics

Off by default. The page names classes and counts events, so bind it to loopback or a private address that only your Prometheus reaches. Labels are class names, event kinds and fixed reasons, never actor names.

Not there yet: per-tenant labels (one tenant per process until multi-tenancy), traps by reason (wasmkeep_events_total{outcome="failed"} and wasmkeep_cpu_budget_exceeded_total cover them for now) and parked WebSockets (sockets are not parked yet; wasmkeep_websockets_open counts them).

Bucket requests

Every bucket request a node makes is counted: leases, heartbeats, the alarm index, replication, restores and following a deployed app. operation is get, head, put (conditional writes included), delete, list, copy, rename or multipart. status is:

Watch status="error"; the other two are part of normal operation. A listing counts once per call, however many pages it took: S3 and R2 return 1,000 keys a page, so a restore of an actor with 10,000 objects in the bucket counts 2 list requests here but bills about 22.

Failovers

wasmkeep_failovers_total counts, on the node that took them, actors taken from another node whose node lease was fenced (or released with the actor still recorded as its own, ttl plus grace earlier): that node died, was cut off from the bucket, or stopped without releasing the actor (ADR 0020). An actor that moves because its owner hibernated it, shut down, or gave it up is a hand-off (the owner released it) and is not counted, nor is a node taking back its own actors after a restart. Sum it over nodes for the cluster; outside rolling restarts of crashed nodes it should stay near zero.

Grafana dashboard

deploy/grafana/wasmkeep.json shows every metric above except wasmkeep_classes, the guest log counters (wasmkeep_guest_log_lines_total, wasmkeep_guest_log_dropped_total), the history collection counters (wasmkeep_history_collected_total, wasmkeep_history_collect_failures_total), wasmkeep_snapshot_upload_failures_total, wasmkeep_signatures_refused_full_total, wasmkeep_websocket_owner_closes_total, wasmkeep_websocket_sweep_reads_total, wasmkeep_restore_failures_total, the listing and restore-read histograms (wasmkeep_listing_seconds, wasmkeep_restore_read_seconds), the node lease write and fence counters (wasmkeep_node_lease_writes_total, wasmkeep_node_fences_total) and wasmkeep_requests_abandoned_total. To see it on a running cluster, the Compose example starts three nodes with Prometheus and Grafana, the dashboard loaded. Otherwise import it in Grafana (Dashboards, New, Import) and pick your Prometheus data source; it needs Grafana 10 or later. It assumes Prometheus adds an instance label to each node’s series, as it does by default, and filters by node and class.

A test (the_grafana_dashboard_uses_only_metrics_that_exist) fails if the dashboard queries a metric the page does not have.

Alerts worth having

These are in deploy/prometheus/alerts.yml as Prometheus rules, with one more for a node Prometheus cannot scrape (up == 0 for a minute): a node that is down exports nothing, so none of the rules below would say so. Load the file with rule_files: in prometheus.yml and check it with promtool check rules after editing. It assumes a scrape job named wasmkeep, and the rules on durable write latency and on failed events wait 5 minutes before they fire. Two of the rules below hold “when no node was lost”: a rule cannot know that, so they fire on every fence and failover, a deliberate kill included, and their text says so. The Compose example loads the file. A test (the_alert_rules_use_only_metrics_that_exist) fails if a rule names a metric the page does not have.

• wasmkeep_node_lease_failures_total increasing on any node: two failures in a row and the node drops every actor it holds.
• increase(wasmkeep_node_fences_total[15m]) above 0 when no node was lost.
• histogram_quantile(0.99, rate(wasmkeep_durable_write_seconds_bucket[5m])) above 1 s.
• rate(wasmkeep_events_total{outcome="failed"}[5m]) above 1% of all events for a class.
• /_wasmkeep/ready failing on any node for 2 minutes.
• rate(wasmkeep_bucket_requests_total{status="error"}[5m]) above 0 for 5 minutes.
• increase(wasmkeep_failovers_total[15m]) above 0 when no node was lost.

What actors print

What an actor writes to stdout or stderr (Rust’s println! and eprintln!, console.log in TypeScript, print in Python, printf in C) reaches the node’s log, one entry per line, as target wasmkeep::guest with the actor’s address and the stream; a stateless component’s says actor=stateless:

INFO wasmkeep::guest: order 42 shipped actor=acme/orders/42 stream="stdout"

• Limits: --guest-log-rate lines a second per actor instance (50 by default, with bursts of twice that) and 1,000 a second for the whole node. Past them, lines are dropped and counted (wasmkeep_guest_log_dropped_total), and the actor’s next logged line is preceded by a warning saying how many were. --guest-log-rate 0 discards everything.
• Lines: a line is logged when its newline arrives; one left without it is logged when the instance ends (hibernation, a trap). Lines are cut at 4 KB, and control characters are escaped (\u{1b}), so a guest cannot forge entries or colour a terminal. Writes never block or fail the guest.
• Filtering: the usual RUST_LOG, e.g. RUST_LOG=info,wasmkeep::guest=warn to keep only the drop warnings, or wasmkeep::guest=off.
• Actors log what they choose to: whatever they print (a secret included) lands in the node’s log, and wherever that is shipped.

API reference

wasmkeep:actor@0.2.0, the interface between WasmKeep and every actor, in any language. Each entry gives the WIT definition, then its name in each SDK (Rust, TypeScript, Python, C, Zig). The SDKs add conveniences (typed rows, JSON, broadcast); see each SDK’s README.

The wasmkeep:actor package: everything an actor can call on the host, and everything the host calls on an actor.

An actor is addressed as tenant/class/name. One component implements one class; each distinct name is an independent instance with its own SQLite database. The host delivers events (requests, alarms) to an instance one at a time, in arrival order: an actor never runs concurrently with itself.

Versioning: 0.x minor versions may break guests; every change comes with a migration note (docs/adr). See docs/adr/0002-wasmkeep-actor-wit-v0.1.md and docs/adr/0003-websockets-wit-v0.2.md.

What an actor implements

HTTP requests arrive through the standard wasi:http/incoming-handler export (handle), which the TypeScript, Python, C and Zig SDKs expose as their fetch handler and the Rust SDK as http::Guest::handle. The world adds:

on-init

on-init: func() -> result<_, string>;

Runs before the first event an instance handles with this component version: once for a new actor, and again after each deploy. Use it to create or migrate tables. An error fails the event; it is retried with the next event.

on-alarm

on-alarm: func() -> result<_, string>;

Runs when the pending alarm is due. An error or trap schedules a retry.

on-websocket-open

on-websocket-open: func(conn: u64, path: string) -> result<_, string>;

A client upgraded a GET to path (the part after the actor’s name, with its query). Return an error to refuse it: the connection is then closed with code 1008 and the error as reason. It runs before the host answers the upgrade (101), so the client is connected only once the actor has accepted it. What the actor sends meanwhile waits in the socket’s queue, with its usual cap (backpressure past 4 MiB). A client that leaves before the 101 (or a node relaying for it, which gives up after 30 s) still gets this event if it was already queued, followed by on-websocket-close with code 1006.

on-websocket-message

on-websocket-message: func(conn: u64, msg: message);

A message arrived on one of this actor’s connections.

on-websocket-close

on-websocket-close: func(conn: u64, code: u16, reason: string);

A connection closed (by the client, by close, or by the host).

Interface storage

The actor’s own SQLite database. Nothing else can read or write it.

Each call runs one statement. Outside an explicit transaction every statement commits on its own. BEGIN … COMMIT across several calls within one event is allowed; a transaction still open when the event ends (or traps) is rolled back. A statement that fails in a way that rolls the open transaction back (it ran past its deadline or the event’s SQL CPU budget, say) ends that transaction: nothing of it is kept, and every later storage statement of the event is refused with not-allowed (but a ROLLBACK, which is what already happened, succeeds).

exec

exec: func(sql: string, params: list<value>) -> result<u64, sql-error>;

Runs a statement that returns no rows; returns the number of rows changed. Returns once the write is committed to the local database.

exec-durable

exec-durable: func(sql: string, params: list<value>) -> result<u64, sql-error>;

Like exec, but returns only once the write is durable: replicated to the bucket when the node has one, fsynced locally when it does not. Inside a transaction it takes only COMMIT (or END), and makes the whole transaction durable; any other statement there is refused with not-allowed before it runs. Outside one it does not take a statement that opens or controls a transaction (BEGIN, SAVEPOINT): run those with exec.

query

query: func(sql: string, params: list<value>) -> result<rows, sql-error>;

Runs a statement and returns its rows.

delete-self

delete-self: func();

Deletes this actor: its database is removed once the current event ends, and its pending alarm is cleared. The next event to this address starts a fresh actor.

Interface alarms

One pending alarm per actor. Alarms survive restarts and fire at least once: an on-alarm that fails or traps is retried, so handlers must be idempotent.

set-alarm

set-alarm: func(at: u64);

Schedules on-alarm at at (milliseconds since the Unix epoch), replacing any pending alarm. A time in the past fires as soon as possible.

get-alarm

get-alarm: func() -> option<u64>;

The pending alarm time, if any.

clear-alarm

clear-alarm: func();

Cancels the pending alarm, if any.

Interface actors

Calls between actors of the same tenant.

request (record)

A fully buffered request, delivered to the callee’s wasi:http/incoming-handler like any other request.

response (record)

call-error (variant)

actor-address (record)

An actor’s address. Within a tenant, class/name is what call takes; the name is decoded (any text without / or control characters).

call

call: func(address: string, request: request) -> result<response, call-error>;

Sends request to the actor at address (class/name, within the caller’s tenant) and waits for its response. The caller’s own event stays in progress meanwhile: other events to the caller wait.

The callee sees x-wasmkeep-caller: <tenant>/<class>/<name>, the name percent-encoded (every byte but ASCII letters, digits and -._~), so room 1 arrives as room%201. The host removes that header from requests coming from outside, so the callee can trust it.

self-address

self-address: func() -> actor-address;

This actor’s own address: an actor does not otherwise know its name or class (requests arrive with /actor/<class>/<name> removed, and alarms carry nothing). For links to itself, for telling other actors who it is, and for anything keyed by its name.

Interface websockets

WebSocket connections to this actor.

The host accepts upgrades on the actor’s path and owns the sockets, so a connection stays open while the actor hibernates; the next message wakes it. A connection id is only valid in the actor it was opened on.

message (variant)

A WebSocket message.

send-error (variant)

send

send: func(conn: u64, msg: message) -> result<_, send-error>;

Queues msg on the connection.

close

close: func(conn: u64, code: u16, reason: string);

Closes the connection with a close code (1000 = normal) and reason.

connections

connections: func() -> list<u64>;

This actor’s open connections, oldest first.

Interface env

Configuration and secrets the operator gives an actor class: an API key, a webhook URL, a feature flag. Values come, by name, last wins, from the deployed version’s plain configuration (wasmkeep deploy --config CLASS=FILE), the app’s secrets sealed in the bucket (wasmkeep secrets set), and files on the node (wasmkeep serve --env CLASS=FILE), never from the component. They are the same for every actor of the class and read-only, and change only between events, never during one: when a deploy or a secrets change lands (the next event runs in a new instance) or the node restarts.

get

get: func(name: string) -> option<string>;

The value of name, or none if it is not set.

Interface types

Types shared by several interfaces.

value (variant)

A SQL value, mapping one to one onto SQLite’s storage classes.

rows (record)

The result set of a query: column names, then one list of values per row, in column order.

sql-error-kind (enum)

Why a statement failed.

sql-error (record)

Also available

The actor world includes the standard wasi:http/proxy world: outbound HTTP (wasi:http/outgoing-handler), clocks, randomness and stdio (what an actor writes to stdout and stderr goes to the node’s log, line by line, within --guest-log-rate). The wasi:cli interfaces most standard libraries import are present but empty: no arguments, environment or terminal. wasi:filesystem and wasi:sockets, which many toolchains import whether or not the code uses them, are linked but reach nothing: no directory is opened to the guest, and every socket and name lookup is refused. A component importing an interface the host does not provide is refused at deploy time.

Benchmarks

Every number here comes from a file written by ./bench/run.sh (./bench/bucket.sh and ./bench/cluster.sh for the comparisons on a bucket, ./bench/soak.sh for a real bucket), named in its section; sections that name no file use bench/results/2026-11-04-db85998.csv (commit db85998), except the failover row, which needs an S3 API and comes from 2026-11-03-7060271.csv. The short hash in a measurement file’s name (and in the text below) identifies the build that was measured; it is a label, not a commit in this repository. Anyone can rerun it and get their own file; numbers that do not come from such a file do not belong in the docs.

Hardware: a shared cloud container (KVM, 4 vCPU Xeon @ 2.10 GHz, 16 GB), Rust 1.99.1, Wasmtime 50.0.1, Node 22, workerd 2026-10-16. A shared VM is noisy (we saw ±30% between runs on some rows), so read these as orders of magnitude, not decimals. Compared with the previous file (2bfc65f), workerd’s CPU per request moved 2%, so the machine was comparable; WasmKeep’s Rust counter went from 295 to 203 µs of CPU per request and from 9,897 to 15,356 req/s, after the host stopped hopping to the blocking pool for its own bookkeeping. The wake row came out slower (1.07 to 1.39 ms), but three interleaved runs of both commits on the same machine overlapped (1.47 to 1.98 ms after, 1.73 to 1.87 ms before): that row is noise, not a change. Publishable comparisons belong on dedicated hardware: see the benchmark showdown.

WasmKeep against workerd, the same counter

The same actor in three runtimes: one counter object per name, its value in SQLite, GET reads and POST increments.

• WasmKeep, Rust: examples/counter, compiled ahead of time.
• WasmKeep, TypeScript: examples/ts-counter on QuickJS, the default engine, embedded in the component.
• workerd: Cloudflare’s open-source Workers runtime with a SQLite-backed Durable Object, bench/workerd/counter.js.

Each runtime runs as its own server process, and the same client (bench/compare) drives it over loopback HTTP/1.1 keep-alive. Server memory and CPU time come from /proc/<pid>.

From bench/results/2026-11-06-351ea57.csv (commit 351ea57, a clean tree, the first run with transparent huge pages off for the server, see below; the previous run, dec1a82, was within a few percent on every row but memory). Hardware: an AMD Ryzen 9 7950X workstation (16 cores, 32 threads), Rust 1.99.1, Wasmtime 50.0.1, Node 26, workerd 1.20261016.1. Nothing else was running. Both runtimes kept their actor data on tmpfs (/tmp), so no disk is involved on either side.

* workerd runs one thread per process by design: Cloudflare scales it with many processes. Its throughput row measures one process, which is not a fair comparison. CPU time per request is the fair efficiency comparison, and the best proxy for energy we can measure here.

The same comparison on a shared 4-vCPU cloud VM (2026-11-05-ae5416c.csv) gave a different picture:

How to read it

• The machine changes the answer. On the workstation, WasmKeep with Rust uses about 81% of workerd’s CPU per request; on the 4-vCPU VM it used 39%. workerd gains much more from fast cores (555 to 128 µs) than WasmKeep does (215 to 104 µs). A claim like “less than half the CPU” holds on one machine and not on the other, so quote both or neither.
• WasmKeep with Rust is still the fastest per request here. Reads and writes at p50 are about 2.6× faster than workerd’s, and the p99 gap is 2.4×.
• Memory per resident actor is about workerd’s: 250 KB against 277 KB, as on the VM (251 KB). Earlier workstation runs read 571 to 577 KB (files cd48f41 to dec1a82), for every native language alike: this machine’s kernel has transparent huge pages set to always, so each actor’s memory was backed by 2 MiB pages it mostly did not use. wasmkeep serve now turns them off for its own process (--transparent-huge-pages system keeps the kernel’s setting); CPU per request and latency did not move (three runs each way, within 2%).
• TypeScript on WasmKeep costs more than Rust. On QuickJS, the default engine, it uses about 2.2× the CPU of Rust and 3× the memory per resident actor (1,000 resident actors: 936 MB), and about 1.8× workerd’s CPU. Every actor carries its own JavaScript engine, without V8’s JIT: for JavaScript-heavy workloads, workerd is the better home (QuickJS takes 5.8 ms for the js_work load below). TypeScript on WasmKeep is for teams that want WasmKeep’s model in their language. It is not a speed play. StarlingMonkey costs more, see below.
• Writes are not exactly equivalent. A Durable Object’s storage may wait for the disk before answering a write, while WasmKeep’s plain exec commits without an fsync. On tmpfs neither waits for a disk, so the write row is closer to fair here than on a real disk. A durable write on WasmKeep (exec-durable, replicated) costs under 2 ms with an in-memory bucket and its data on tmpfs, see below.
• Not measured yet: idle memory after hibernation (WasmKeep’s idle actors hold none), wake latency per runtime, and workerd with several processes. These are next for the comparison harness.

WasmKeep against celld, every write in a bucket

celld runs Durable Objects on a bucket: each object’s SQLite changes go to S3, much like WasmKeep’s replication. Here both runtimes serve the same counter on the same local bucket:

• WasmKeep, durable: examples/counter built with its durable feature, so POST commits with exec-durable and answers once the write is in the bucket. WasmKeep runs as a one-node cluster, since a node with a bucket is a cluster node.
• WasmKeep, plain exec: the default build of the same counter. Its POST answers before the write reaches the bucket, so it is not equivalent to the other two columns and is shown apart.
• celld: the workerd counter (bench/workerd/counter.js, unchanged), deployed with celld deploy to one celld node. One node waits for the bucket on every write (CELLD_DURABILITY=bucket, set explicitly).

Run by bench/bucket.sh (compare --bucket, compare-bucket/ rows) at commit 488369d, a clean tree, five times with the runtimes’ data on tmpfs and five times on disk (btrfs on an encrypted NVMe SSD), each run on an emptied bucket. Each cell is the median of five runs, with the lowest and highest in brackets; the files are …-data-tmpfs-r1.csv to -r5.csv and …-data-disk-r1.csv to -r5.csv. Hardware: the AMD Ryzen 9 7950X workstation above, Rust 1.99.1, Wasmtime 50.0.1, celld v0.6.3 (the release binary, its build provenance checked by bench/celld/fetch.sh). The bucket is MinIO on loopback with its data on tmpfs, so a write to it costs a local round trip and no disk. Transparent huge pages are off for both servers, as wasmkeep serve sets for itself and the harness sets for celld: a run that left celld on the kernel’s always (e3856a5) read 609 MB at 1000 actors instead of 395 MB.

* Includes the 15 s a WasmKeep cluster node’s first start waits before it listens, checking that no other process runs with its node id (cluster); a restart after a fence or a clean stop waits less. celld does not wait.

How to read it

• Durable against durable, WasmKeep answers a write sooner at p50, on tmpfs and on disk. 0.53 against 0.93 ms on tmpfs and 2.31 against 2.55 ms on disk. At p99 the two are close on tmpfs (1.15 against 1.25 ms) and celld is ahead on disk (3.64 against 6.02 ms). Without a bucket the same durable write takes 102 µs (compare/wasmkeep-rust-durable in 2026-11-10-f163ba0.csv).
• On disk a write takes 1.6 to 1.8 ms longer in both, and throughput is close, celld about 5% ahead. Eight clients completed 1,980 writes per second on WasmKeep against 2,087 on celld at the median; the runs of each overlap (1,974 to 2,032 against 2,003 to 2,198). On tmpfs WasmKeep completed 1.6× celld’s (10,823 against 6,650); celld runs as one multi-threaded process, and this row measures that process.
• A real bucket changes the scale again. MinIO on tmpfs and loopback answers in well under a millisecond. A remote S3 bucket takes tens of milliseconds per write, which would dominate both durable write rows; the read, memory and CPU rows do not touch the bucket.
• WasmKeep uses about 85% of celld’s CPU per request (83% on tmpfs, 85% on disk), 93 to 95% of its memory per resident actor and 84 to 86% of its process memory at 1000 actors, and reads about twice as fast at p50.
• Plain exec on a bucket is barely faster per actor than durable on tmpfs, and much faster on disk. Its response does not wait for the bucket, but an actor uploads an event’s changes before it runs its next event, so the next request to the same actor waits instead. Across many actors the uploads overlap: its throughput is 2.4× the durable build’s on tmpfs and 4.8× on disk.
• One node only. With two or more nodes, celld by default (CELLD_DURABILITY=fleet) acknowledges a write once a follower node has it on disk, or once the bucket has it, whichever comes first. WasmKeep has no such mode: a durable write always waits for the bucket. Three nodes are measured below.
• Start time is a design difference. celld answers 77 to 97 ms after it starts, with its wait for the rest of a fleet turned off for this one-node run (CELLD_READY_FLEET_GATE_MS=0); a WasmKeep cluster node’s first start waits 15 s, by design. A single node without a bucket does not wait (51 ms above).

Three nodes: WasmKeep against celld

Three nodes of each runtime sharing one bucket, run by bench/cluster.sh (compare --cluster): the same counter for the steady-state rows, then each node in turn killed under writes. Rows are cluster3/..., cluster3-wipe/... for the takeovers repeated with the killed node’s directory removed, and cluster3-control/... for a run that must lose writes. Durable here means what each runtime means by it:

• WasmKeep, durable (exec-durable): a write answers once the bucket holds it, on any number of nodes.
• celld, fleet (CELLD_DURABILITY=fleet, celld’s default with two or more nodes): a write answers once a follower node has it on disk, or once the bucket has it, whichever comes first. Up to two followers per object; the bucket upload follows.
• celld, bucket (CELLD_DURABILITY=bucket): a write answers once the bucket holds it, as on WasmKeep.
• WasmKeep, plain exec: answers before its upload, not equivalent, shown apart.

Both runtimes keep their default lease timings (a dead node’s actors move after 15 s on WasmKeep, 10 s on celld) and talk plain HTTP between nodes, since celld has no TLS between peers. Every request goes to a node picked at random, so two in three are forwarded to the actor’s owner. WasmKeep nodes run with --max-instances 2000, --max-connections 512 and --idle-timeout-secs 3600, so no actor hibernates during a run; celld nodes run with transparent huge pages off, as WasmKeep turns them off for itself.

The check for lost writes. After the steady-state rows, each node in turn is killed with SIGKILL while 90 actors are written every 20 ms, one client per actor retrying every 100 ms through a random node. These actors are ledgers, not counters: every attempt writes a token of its own, never sent twice. After each restart, and once more at the end, every ledger’s tokens are read back: an acknowledged token that is missing is a lost write, and a token held twice was applied twice. An actor whose longest wait between two acknowledgements after the kill is over a second was on the dead node, and the end of that wait is when it was writable again. Three more runs go with it:

• The killed node’s disk removed (cluster3-wipe/): the same kills, with the node’s directory deleted before it restarts, so nothing that node alone held survives.
• A control that must lose writes (cluster3-control/): plain exec on nodes that reach the bucket through a relay; half a second before the kill the relay cuts the node off, so each of its actors acknowledges a write the bucket never gets, then the node is killed and its directory removed. If this run loses nothing, the script fails and writes no file.
• Five runs of everything, each on an emptied bucket, once with every node’s data on disk (btrfs on an encrypted NVMe SSD) and once on tmpfs, where an fsync costs nothing.

Hardware: the workstation above, the three nodes of a target as processes on it, one target at a time; the bucket is the same local MinIO, its data on tmpfs. Commit c556110, a clean tree. Each cell is the median of five runs, with the lowest and highest in brackets; the files are …-nodes-disk-r1.csv to -r5.csv and …-nodes-tmpfs-r1.csv to -r5.csv.

Lost writes

The same in both sets of five runs, on disk and on tmpfs:

Timings, every node’s data on disk

* Includes the 15 s node-id check of a first start (cluster). A node restarted after a kill waits less: the surviving nodes have fenced the old process by then.

Timings, every node’s data on tmpfs

The same runs with only the nodes’ data directories moved to tmpfs. An fsync costs nothing there, so this is not a durability setup: it shows what is left of each write once the disk is taken out.

How to read it

• No acknowledged durable write was lost, in ten runs of each target. Sixty kills each for WasmKeep durable, celld fleet and celld bucket (three with the node’s files kept and three with them removed, in each run), every acknowledged token found afterwards. The control lost writes in every run, so the check does find a loss; and plain exec, which may lose writes with its node, lost 0 to 3 of about 86,000 in a run without any help from the relay.
• What the check still does not test. Removing the killed node’s files tests what that node alone held; the followers on the surviving nodes keep their page cache, so an acknowledgement that relies on a follower’s fsync (celld fleet) is not tested against a power loss. All three nodes are processes on one machine. The soak checks WasmKeep’s durable writes on separate machines, over hours.
• Some unacknowledged writes were applied, which is allowed. A write whose client got no answer (the request failed or timed out during a takeover) may still have been applied: 8 to 25 a run on WasmKeep durable, 13 to 26 on celld bucket and 34 to 100 on celld fleet, with data on disk. The client retries with a new token, so none is counted twice.
• Waiting for the bucket in both, on disk, the two are close and celld is ahead on throughput. WasmKeep answers a write sooner at p50 (2.30 against 2.68 ms) and later at p99 (5.89 against 4.17 ms); eight clients, each waiting for its answer, completed 1,710 writes per second against 1,857, about 8% apart where runs of one target differ by 2 to 4%, at the same CPU per write (952 against 958 µs).
• Where the nodes keep their data changes the order. With data on tmpfs WasmKeep completed 9,137 durable writes per second against 5,888 for celld in bucket mode, at 509 against 627 µs of CPU. A durable write takes 1.75 ms longer than a plain one on disk (2.30 against 0.55 ms) and 0.10 ms longer on tmpfs (0.59 against 0.49 ms). These runs change only where the data is; they do not say which step of a write the disk slows.
• celld’s fleet mode is its slowest on a shared disk, and that is not explained. 5.24 ms at p50 and 373 writes per second, against 2.68 ms and 1,857 in bucket mode, the same in five runs. An acknowledgement that takes whichever of the followers and the bucket comes first should not be slower than waiting for the bucket. Here the owner and its followers share one device; whether that is the cause is not measured, and nodes with a disk each would tell. On tmpfs fleet mode answers a write sooner than bucket mode (0.68 against 1.04 ms) but completes fewer (4,238 against 5,888 a second) at twice the CPU.
• WasmKeep reads faster and holds less memory. Reads take 92 against about 220 µs at p50 (both forwarded two times in three), and three nodes hold 1000 actors in 459 against about 520 MB.
• Takeover is close to each one’s lease timing. A dead node’s actors are writable again after about 15 s on WasmKeep and 9 to 10 s on celld, close to each one’s default (15 s and 10 s). Neither side was tuned. A restarted celld node is ready in about 0.15 s; a WasmKeep node in 1.6 s.
• Not measured here: nodes on separate machines, each with its own disk (here the network between nodes is loopback and the disk is shared), a cloud bucket, and mutual TLS between WasmKeep nodes (measured above). An earlier single run with a weaker check for lost writes is in 2026-11-10-d2dea43-cluster-minio-tmpfs.csv.

The same counter in every language

The same actor contract (read, increment, persist) in every guest language WasmKeep supports, each runtime its own server process driven by the same client as above. From bench/results/2026-11-06-351ea57.csv (commit 351ea57, the workstation and run of the workerd table above). TypeScript is on QuickJS, the default engine; StarlingMonkey is the --engine starlingmonkey build of the same source.

The Go column is the counter from before the Go SDK (151 KB; 162 KB with it): the same host calls without the wrapper.

The same table on the shared 4-vCPU cloud VM (2026-11-05-ae5416c.csv, without QuickJS) had the same order at 1.5 to 3.4× this machine’s CPU per request: Rust 215 µs, Zig 214, C 222, Go 364, TypeScript on StarlingMonkey 855, Python 859.

How to read it

• Rust, Zig and C cost the same. Their CPU per request is within 6% of each other, which is run-to-run noise here: most of a request’s cost is the host (HTTP, entering and leaving Wasmtime, SQLite), not the guest. Zig’s smaller component changes deploys and the first compile, not requests.
• Zig needs its stack set. Zig’s default wasm stack is 16 MiB, and a component reserves it in every actor’s memory: the Zig counter took 2.66 MB per resident actor in an earlier run (with huge pages on), against 0.57 MB for C. The examples now build with --stack 1048576, as Rust’s 1 MiB (sdks/zig).
• TinyGo costs about 2.4× the CPU of the native three, with long tails. Its median latency is close to theirs, but its p99 is 10× (reads) to 40× (writes) its median: TinyGo’s garbage collector runs inside the request. Memory per actor is 1.6× theirs.
• The embedded runtimes cost 2.2 to 3.5× the CPU. QuickJS (230 µs) and CPython (259 µs) land close together, StarlingMonkey (368 µs) above them. Memory per resident actor is 3× (QuickJS), 14× (StarlingMonkey) and 31× (CPython) Rust’s: each actor carries its own engine or interpreter. 1,000 resident Python actors took 8.4 GB.
• Python’s CPU per request fell the most from the VM (859 to 259 µs, 3.4×, against 2.1× for Rust): CPython’s interpreter loop gains more from this machine’s caches and clock than the host does.
• Component sizes are from each example’s build.sh (languages).

TypeScript on QuickJS against StarlingMonkey

The same TypeScript counter (examples/ts-counter), one source file built twice: with QuickJS, the default engine since ADR 0018 (npm run build), and with StarlingMonkey (npm run build:starlingmonkey, see languages), next to the Rust counter. From the same file and run as the tables above, 2026-11-06-351ea57.csv. The js_work rows time the counter’s /work path (work.ts): plain JavaScript (building 200 records, a JSON round trip, filtering, sorting, string formatting, a regex, a hash; four rounds), one request at a time, so its latency is the engine’s time.

The comparison that decided the default (2026-11-06-dec1a82.csv, with huge pages on: 1.06 MB against 3.7 MB per actor) had the same CPU and js_work rows within 7%.

How to read it

• QuickJS takes 4.6× less memory per resident actor than StarlingMonkey (741 KB against 3.4 MB), 3× Rust’s. A node holding 1,000 TypeScript actors needs 936 MB instead of 4.17 GB.
• It costs less CPU on the counter: 230 µs per request against 368 µs (37% less), 2.2× Rust instead of 3.5×. Each request’s cost here is mostly entering a fresh instance, the host and SQLite, and a smaller engine is cheaper to instantiate.
• StarlingMonkey is 1.5× faster on plain JavaScript work (3.76 ms against 5.76 ms). Neither engine has a JIT; SpiderMonkey’s interpreter is faster than QuickJS’s. An actor that spends milliseconds computing in JavaScript per event is the case for --engine starlingmonkey (or for Rust).
• Components are 8× smaller (1.7 MB), which shortens deploys and the first compile on a node (202 ms against 704 ms from start to first response with an empty cache).
• What QuickJS does not provide (Web APIs such as fetch, URL, crypto, console, timers) is listed in languages.

Between nodes: plain HTTP against mutual TLS

What a hop between two nodes costs, and what mutual TLS between nodes adds to it (ADR 0011). From bench/results/2026-11-05-d8e9ab7.csv (commit d8e9ab7, the cluster/* rows), with an earlier run of the same suite, 2026-11-05-b07a9d7.csv, for the spread, both on different hardware from the sections above: a workstation (AMD Ryzen 9 7950X, 16 cores / 32 threads, 64 GB), Rust 1.99.1, data in a RAM-backed /tmp (tmpfs). Compare these rows with each other, not with the tables above.

The suite (benches/cluster.rs) runs two pairs of nodes in one process over loopback, each pair sharing an in-memory bucket: one pair plain, one with cluster listeners. In the TLS pair every node publishes a plain address nobody listens on, so a hop in plaintext fails the run instead of being measured. One client, one request at a time, kept alive; nodes pool their connections to each other.

The earlier run predates the change that made forwarding drop hop-by-hop headers, which only adds a header filter to these kept-alive rows. Between the runs, rows moved by up to 7 µs either way: read the TLS cost as a range, not a decimal.

How to read it

• A hop costs about 40 µs on loopback in plain HTTP (106.3 against 63.2 µs), about 50 µs over TLS. On a real network the round trip between the nodes comes on top, the same with or without TLS.
• TLS adds about 10 to 20 µs per hop (9 to 12 µs to a forwarded request, 15 to 21 µs to a call, across the two runs): encryption both ways, and the checks ADR 0011 makes on every request (the peer’s pin checked again, and the answer checked before it is delivered). The call row also carries the signature and the body’s hash, plain or not.
• Not measured: a new connection. Each row reuses a pooled connection, so a full TLS 1.3 handshake (with a client certificate, and no session resumption by design) is not in these numbers. Connections are pooled and kept alive, also for clients that send Connection: close (forwarding drops hop-by-hop headers); they are rebuilt when a peer’s pin changes or its heartbeat lapses.
• Only one request at a time: throughput and CPU per request between nodes, and many clients at once, are not measured yet.

Restore against history length

How long a node takes to restore an actor it has never run (as on failover, or when an actor wakes on another node) as the actor’s history in the bucket grows, and what garbage collection changes. From bench/results/2026-11-05-4c8b21a.csv (commit 4c8b21a, the restore/* and list/* rows; the first run, before collection existed, is 2026-11-05-cd48f41.csv), on the same workstation as the section above (AMD Ryzen 9 7950X, 16 cores / 32 threads), Rust 1.99.1, data in a RAM-backed /tmp (tmpfs), and an in-memory bucket. Compare these rows with each other, not with the tables above.

The suite (benches/restore.rs) runs two nodes sharing the bucket over loopback. A writer gives one counter N replicated writes (one segment each, plus one for on-init); a restorer is then asked for fresh actors whose prefix holds a copy of that history. Each timed request takes the lease, lists the prefix, plans, downloads and applies the chain, and answers a counter GET. Three shapes of the same history:

• As written: the owner uploads a snapshot every 64 segments, so a restore replays fewer than 64 segments, but nothing is ever deleted, so the prefix keeps every segment and snapshot.
• Without periodic snapshots (removed from the copy, the first kept): every segment is replayed. This is what an actor whose periodic snapshots kept failing would leave (a failed one is counted and tried again one segment and 30 s later, then less and less often, down to once every 64 segments).
• After garbage collection (ADR 0012): what the server’s rule (collectable) leaves after the last periodic snapshot once the retention window has passed: that snapshot, the segment at its position and the segments after it.

Medians. The baseline with no history, run again after all the other restore rows, was 0.78 ms both times: restored actors piling up on the node do not inflate the later rows. At 10 writes nothing is collected: the only snapshot is the first one, with no segment at its position.

How to read it

• Periodic snapshots bound the replay, collection bounds the listing. As written, a restore replays at most 63 segments whatever the history’s length, but the prefix grows by one object per write and one per 64 writes: at 10,000 writes a restore takes 6.1 ms against 2.6 ms at 1,000 while replaying fewer segments (17 against 41), because it lists the whole prefix twice and listing 10,158 objects takes 1.7 ms in memory (restores now list the prefix once; not remeasured).
• After collection the prefix holds at most 65 objects, whatever the history’s length (a snapshot and up to 64 segments): at 10,000 writes the restore takes 1.2 ms instead of 6.1 ms, and listing takes 1.7 µs instead of 1.7 ms. The collected rows stay within the range of small histories (1.1 to 2.1 ms here, depending on how many segments follow the last snapshot). On S3 or R2 the difference is larger: one listing page instead of eleven.
• A replayed segment costs about 34 µs here (342 ms for 10,001 segments): one GET, decoding and applying it. Each counter write changes one page, so this is the fixed cost per segment; segments with more pages cost more.
• Not measured: a real bucket. The in-memory bucket answers a listing in one call; S3 and R2 page listings by 1,000 keys, so at 10,158 objects each listing is 11 requests (22 per restore in the version measured above, which listed twice), before the first download. At 20 to 50 ms per request that would add about 0.4 to 1.1 s to a restore (an estimate from the request count, not a measurement), and every replayed segment is one more GET: 82 ms each at the median next to OVHcloud Object Storage, so 1 to 6 s for up to 64 segments (measured). LIST and GET counts against a real bucket, peak RSS during restore, larger databases and pages per segment, and many actors at once are not measured either. The collected rows are histories collected by the rule, copied into the bucket; the server’s own passes (--history-retention-secs, an hour by default) are tested in tests/cluster/, not timed here.

Restart and alarm recovery against actor count

How a node’s restart and the recovery of a dead node’s alarms scale with the number of actors. From bench/results/2026-11-05-4c10a33.csv (commit 4c10a33, the restart/* and alarm_recovery/* rows), on a workstation (AMD Ryzen 9 7950X, 16 cores / 32 threads, 64 GB), Rust 1.99.1, data in a RAM-backed /tmp (tmpfs), an in-memory bucket, and other work running on the machine at the same time. Compare these rows with each other, not with the tables above.

Restarting a node

benches/restart.rs: one standalone node holds N test actors, each created by a request that set an alarm an hour out, so each database is as an actor leaves it (36 KB on disk per actor with its WAL files: 3.7 GB at 100,000). The node is stopped as a crash stops it and started again on the same directory.

Medians. The scan column is Storage::pending_alarms alone (the node runs it at start-up on a blocking thread, then arms what it found). The last column makes every actor’s alarm overdue while the node is down, and counts until each has run on-alarm once (it includes the scan).

• A restart serves at once, whatever the population. The listener does not wait for the scan: 4 ms from start to an actor’s answer at 100,000 actors as at 1,000.
• The scan is linear, about 110 µs per database, on one thread. It opens every database (with the same set-up as an actor’s: WAL mode, pragmas, the host’s tables) to read one row. Until it finishes, no alarm from before the restart is armed: at 100,000 actors an alarm due during the first 11.5 s fires up to 11.5 s late. At this rate a million databases would take about two minutes (an extrapolation; not measured, and 37 GB at this size). A slower disk than tmpfs adds its own cost per open.
• Overdue alarms after a restart run at about 3,000 a second (10,000 in 3.4 s, of which 1.1 s is the scan): each one wakes its actor, runs the event, and the actor stays resident until the pool needs room.

Alarm recovery after a node dies

benches/alarm_recovery.rs: three nodes share the bucket. Node a creates N test actors, each with an alarm at the same instant T, and is killed once the bucket holds them and their alarm index entries (no lease released). Nodes b and c never ran those actors: their sweeps find the alarms in the bucket’s index, and each delivery takes the actor over, restores it and runs on-alarm. The time is from T until every alarm has run and the index is empty. Timings are a tenth of the defaults (lease ttl 1 s, grace 0.5 s, heartbeat 0.3 s, sweep every 1 s instead of 10 s): an alarm can be delivered by another node once it is overdue by a lease period plus grace (1.5 s here, 15 s by default).

Medians. The 10,000 row is from bench/results/2026-11-05-9c495b4.csv (commit 9c495b4, same machine, after the sweep fix; its samples ranged from 23.19 to 23.23 s, and 100 and 1,000 measured the same as before); 38.2 s is the 4c10a33 run (37.2 to 38.2 s). The estimate scales the whole time by ten, since the sweep interval and the lease timings set nearly all of it (the work of a delivery is a few milliseconds and does not scale); it was not measured at the defaults.

• Recovery is capped by the sweep, not by the work. Each node delivers at most 256 alarms per sweep (SWEEP_DELIVER in alarm_delivery.rs), split between the live nodes by a hash of the address. 1,000 alarms take two sweeps after the 1.5 s wait, as two nodes at 256 each predict.
• Fixed: past three overdue periods, the survivors delivered the same alarms. An entry overdue three times longer than a lease period plus grace (4.5 s here, 45 s by default) is anyone’s, so that a node nobody else sees can still be covered. The sweep used to walk first come, first served: both survivors’ cursors sat on the same key and delivered the same 256 a sweep, so 10,000 took 38.2 s (the model: 1.5 s wait, three sharded sweeps at 512, then 33 sweeps of 256 for the remaining 8,464: 37.5 s), one node’s rate however many survive. Now each node delivers its own shard first and keeps an eighth of the cap (32) for anyone’s entries (plan_sweep): 224 own a sweep each, 448 for two, plus whichever reserved picks the other node has not reached. The model: 1.5 s, three sweeps at 512, then 8,464 / 448 = 19 sweeps: 23.5 s, against 23.2 s measured. Two nodes at the full 256 each would take about 21 s; the reserve, which keeps anyone’s entries moving when a node’s own entries never leave the index, costs the difference. At the defaults that is about 45 alarms a second for two survivors, and recovery now grows with the node count. sim/tests/madsim_alarm_sweep.rs checks the rate and that every entry is still delivered when views disagree or a live node never delivers its shard.
• No alarm ran twice in these runs; 14 alarm events were refused by admission and retried, over all samples (25 in the 9c495b4 run).

Not measured

A real bucket at this scale (each delivery restores an actor: listings and GETs with real round trips; the sweep lists the alarm index in pages of 1,000; a handful of alarms at the default timings, on R2, is measured), an SSD or slower disk, peak RSS, a cold page cache after a reboot, a node restarting in a cluster (it also waits for a heartbeat and writes each actor’s ownership record on first use), more than 100,000 actors on a node or 10,000 alarms at once, recovery at the default timings, and alarms whose handler fails or re-arms itself.

Workflow replay against recorded steps

Every event that moves a workflow forward replays its recorded steps from the top (ADR 0019), so its cost grows with them, against the event’s 50 ms CPU budget. crates/wasmkeep-server/benches/workflow.rs times one run of an instance with n recorded steps (each returning an integer), parked on a sleep: the runner loads the steps, replays them, finds the park and sets the alarm. It is reached through an actors.call from a driver actor, so n = 0 is the floor (HTTP, the call, the runner’s own reads). From bench/results/2026-11-06-30e68a8.csv (commit 30e68a8, the AMD Ryzen 9 7950X workstation, a Rust guest, one node, tmpfs).

How to read it

• About 1.5 µs a recorded step, steady from 100 to 5,000 steps. A workflow with thousands of steps replays in a few milliseconds of wall clock, well inside the 50 ms CPU budget (which counts the guest’s share of that time only).
• Recording is not measured here: the first run of each size recorded all its steps in one event, a durable write each.
• A Rust guest. TypeScript replays through JSON and an interpreter (QuickJS is 1.5× slower than StarlingMonkey on plain JavaScript); not measured yet.
• Over a workflow’s life replay is O(n²): 5,000 steps that each wake the workflow once replay about 12.5 million positions in all.

Three nodes against a real bucket

crates/wasmkeep-cli/tests/soak.rs, run by bench/soak.sh: three wasmkeep serve nodes with the default lease timings, eight clients making durable writes (exec-durable) to 60 actors through random nodes for 30 minutes, and every 3 minutes one node killed with kill -9, kept down 75 s, then restarted (9 kills). Before each kill, alarms are set on 8 of its actors, due at the moment it dies. Each of its other actors then gets a client of its own, retrying a write every 100 ms until one is acknowledged. Two more clients write with a plain exec (answered before its upload) to 20 actors of their own: the acknowledged increments lost are reported (plain_lost: values handed out again, as later writes re-cover a lost one, plus what the actors no longer hold at the end), not judged, since a plain write may be lost with its node. For 30 s after each kill, four readers read the killed node’s actors through the live nodes every 100 ms (takeover_reads_*: answers, failures, latency).

A run fails if an acknowledged write is lost (at the end, or in a read during a takeover), a value is handed out twice, an alarm never fires, or the 99th percentile of “writable again after the kill” passes 30 s (WASMKEEP_SOAK_FAILOVER_P99_SECS; the worst is reported, not judged; the runs below were judged by their worst). WASMKEEP_SOAK_ACTORS changes the number of actors. Each run keeps samples.csv, one row per write, probe, alarm, kill and restart, which bench/soak_samples.py turns into a line per kill, and every node’s metrics page at each kill.

Three runs of the same commit (0f2986f: per-actor leases, before ADR 0020; the node-lease runs follow below):

• R2 from a desktop: the three nodes as processes on the workstation above, over a residential connection, to an R2 bucket in Eastern North America.
• OVH from a desktop: the same, to OVHcloud Object Storage in Beauharnois (Standard, One Zone).
• OVH from three VPS: one node per OVH VPS-2 (4 vCPU, 8 GB, Debian 13) in Beauharnois, next to the bucket, installed as in Three nodes (systemd, --app, mutual TLS between nodes); the clients run on the first VPS. The deployment to expect.

From …-0f2986f-soak-r2.csv, …-0f2986f-soak-ovh-bhs.csv and …-0f2986f-soak-ovh-bhs-3vps.csv (the VPS run failed the soak’s 30 s check, see below).

How to read it

• Nothing lost or duplicated in any run, over 27 kills under load. Both stores enforce the conditional writes leases and replication rely on. One difference from moto: R2 returns a version id with a write and none with a read, so the two tokens for one object differ. WasmKeep only hands them back to the bucket, never compares them; the test that did is fixed.

• A durable write costs one conditional PUT, so its latency is the bucket’s: 54 ms next to the bucket, 81 ms from the desktop to the same store, 587 ms from the desktop to R2. The long tails (seconds) are writes that reached an actor while it was being taken over.

• Failover is the lease, then a restore. Takeovers start about 12 s after the kill (10 s ttl + 5 s grace, from the last renewal). The new owner then downloads the actor’s history one object at a time: a snapshot and up to 64 segments, 82 ms each at the median from the VPS (237 ms at p90), so most restores take 1 to 6 s.

• One actor in 267 missed the 30 s target on the VPS (32.0 s). Its restore took 19.8 s, while 21 other restores of similar size on the same node at the same time took 2 to 6 s: one GET hung. The bucket client keeps object_store’s defaults (30 s per request, then a retry), and a restore waits for each object in turn, so one slow request held the whole restore. Since then a restore gives up a read after 5 s of silence and sends it again (ADR 0007, restore step 5). With it, the same soak on the VPS passed: writable again 15.5 s after the kill at the median, 24.6 s at worst, over 269 actors; durable write p50 49 ms, worst 8.1 s; nothing lost or duplicated, 72 of 72 alarms (…-c3d50ba-soak-ovh-bhs-3vps.csv). That run logged no read given up three times, and logged nothing for single retries then, so it does not show whether a stall was cut or none happened; every read given up is logged since.

• Twelve hours on the VPS (the same three nodes, a kill every 20 minutes, commit a72fbac: per-actor leases, before ADR 0020): nothing lost or duplicated over 5,024,032 acknowledged writes, 35 kills and 1,129 actors taken over; 280 alarms fired, none missed. Durable write 47 ms at the median, 375 ms at p99, 18.8 s at worst (a write reaching an actor being taken over). Writable again after its node died: 17.0 s at the median, 31.6 s at worst, so the run failed the soak’s 30 s check once in 1,129. The nodes’ logs show why: after one kill, a surviving node took over 19 actors within 3 s (16 s after the kill, the lease and its grace from the last renewal), and restored them in parallel; one of them, 63 segments, took 17.5 s, while another of 63 on the same node at the same moment took 4.2 s. No read stalled long enough to be cut and sent again (5 s of silence), so it was slow reads, not a hung one; reads are not timed one by one. Alarms after their node died: 22.5 s at the median, 34.1 s at worst; a restarted node was ready within 8.6 s (…-a72fbac-soak-ovh-bhs-3vps-12h.csv).

• Twelve hours on the VPS, on node leases (the same nodes, bucket and cadence, the next night, commit 82dab61: ADR 0020, before the bounded segment uploads): nothing lost or duplicated over 3,659,687 acknowledged writes, 35 kills and 1,072 actors taken over; 279 alarms fired, none missed. Writable again after its node died: 18.2 s at the median, 24.4 s at p99 (the soak’s judged bound is 30 s), 89.6 s at worst, once: that actor’s restore (a listing and 55 reads, one after another) took 56 s, each read slow but never silent for 5 s, so none was cut; the 11 requests whose clients had given up meanwhile then ran, 16 s more. Nothing bounded a restore’s total time then; reads have since been made eight at a time, and a restore’s reads have one deadline (cluster), not measured here. Alarms after their node died: 24.0 s at the median, 35.9 s at worst; a restarted node was ready within 10.4 s. Durable write 57 ms at the median, 674 ms at p99, 29.4 s at worst. The writers wait for each answer, so the 27 % fewer writes than the night before are slower writes (a mean of 93 ms against 69 ms), most of it inside the owner’s durable write; the per-actor runs all ran one night and these the next, so the bucket’s speed and the design are not told apart here. Renewals are gone: PUTs other than segments fell from 3.1 to 7.4 per node and second to 0.80 (node lease writes, 0.33), 1.03 PUTs per durable write against 1.17 (…-82dab617-soak-ovh-bhs-3vps-12h-nodelease.csv).

• Both designs on the same night (the same three VPS and bucket): four 30-minute blocks, a kill every 3 minutes, alternating node leases (82dab61) and per-actor leases (a72fbac, its own bucket prefix and data directories), the harness of 82dab61 for all four. Nothing lost or duplicated, every alarm fired, in all four.

  Block	Leases	Writes	Durable p50 / p99	Writable again p50 / p99 / worst
  1	per node	144,727	53 / 666 ms	17.4 / 23.6 / 34.3 s
  2	per actor	185,012	50 / 419 ms	15.5 / 19.1 / 25.4 s
  3	per node	126,974	62 / 747 ms	16.7 / 26.3 / 37.6 s
  4	per actor	186,427	50 / 367 ms	15.6 / 21.4 / 26.2 s

  On one night, node leases made 27 % fewer writes (the writers wait for each answer, so that is slower writes, mostly in the tail: p99 about 1.7 times), and actors were writable again 1 to 2 s later at the median. So the 12-hour runs’ gap is mostly the design, not the night. Both node-lease blocks ran 82dab61, before the bounded uploads and restore fixes; where in the durable write the time goes is not yet measured (…-ab1-nodelease.csv, …-ab2-peractor.csv, …-ab3-nodelease.csv, …-ab4-peractor.csv; the file names carry the harness’s commit).

• Alarms after their node died: another node may deliver an alarm once it is overdue by the lease plus grace (15 s); then comes the next sweep (every 10 s), a restore and the deliveries in turn: 20 to 30 s at the median.

• Lease renewals were most of the requests (per-actor leases, before ADR 0020, which removed them). Each actor in memory renewed its lease 3.3 s (a third of the ttl) after its previous renewal returned. In the R2 run, Cloudflare counted 47,083 PUT, 4,432 LIST and 47,762 GET for the bucket (…-soak-r2-requests.csv): about 23,000 of the PUTs were renewals, more than the 21,260 writes. On R2 that is about $0.23 of Class A requests beyond the free tier per 30 minutes, and a cluster’s bill grew with how long actors stay in memory, not with what they do. OVHcloud does not bill requests. ADR 0020 replaced them with one lease per node; request counts since are not measured.

The allocator: system against mimalloc

The same suite on the same workstation, twice in a row: main as it is (commit 6063067, glibc’s allocator) and with mimalloc as the server’s global allocator (commit 59f01e8, an experiment that was not adopted). From 2026-11-07-6063067.csv and 2026-11-07-59f01e8.csv.

• No change worth a dependency. CPU per request moved 1 to 3%, and throughput 1 to 2%: within what one run of this suite varies on its own. Memory per resident actor moved under 2%, except TypeScript’s, which fell 4%; the server’s total resident memory grew 4 to 7% for Rust, Zig, C and TypeScript, and fell 2% for Python.
• Latency moves both ways between two runs of the same code. The Rust counter’s median write was 77.5 µs in the first run and 67.4 µs in the second, its median read 59.5 µs and 67.8 µs: a single run’s latency rows differ by up to about 15% without any change.
• What it means for packaging: a static musl build would need a faster allocator than musl’s own to match glibc, and mimalloc brings nothing measurable on glibc; a glibc build linked against an older glibc stays the simpler way to one portable binary. Measured below (Portable builds): it is.

Portable builds

One commit (529fd96) built four ways, each run as the Rust counter’s server by bench/compare, interleaved, three rounds; medians, from 2026-11-07-529fd96-portable-builds.csv. Native is cargo build --release on the workstation (it needs glibc 2.39); glibc 2.28 and musl come from scripts/build-portable.sh.

• The glibc 2.28 build is as fast as the native one, on every row within the rounds’ spread, and runs on any x86_64 Linux with glibc 2.28 or newer (RHEL 8, Debian 10, Ubuntu 20.04 and later). It is the candidate for release binaries.
• A static musl build is slower: 2.3 times the CPU per request, under half the throughput, twice the startup. musl’s own allocator is most of it: with mimalloc the gap narrows to 14% more CPU per request and 13% less throughput, with 8% more resident memory, still behind glibc.
• The musl + mimalloc row is from a second set of rounds against native again (104 µs, 61,200/s there). The workstation had other work running (load average 7 to 14); the builds were interleaved, so it weighs on all of them alike.

WasmKeep on its own

A real bucket adds its PUT round trip to durable writes: 54 ms next to OVHcloud Object Storage, 587 ms from a desktop to R2 (measured). Failover is set by the node lease timings (15 to about 21 s after the node’s last heartbeat) plus a restore; see Running a cluster.

Reproduce

Both entry points require Bash. Full comparisons and bench/run.sh require a soft open-file limit of at least 4096; compare --quick requires 1024 (or unlimited). They stop before building, clearing previous Criterion output or starting a server if the limit is too low. The floor is the smallest power of two at which the server keeps every actor of the workload resident: the server sizes its resident-actor cap from its limit, 256 descriptors reserved, the harness’s --max-connections 64, and three per actor (256 + 64 + 3 * (actors + 2)). Below it the benchmark would measure evictions. Full runs create 1,000 actors; quick runs create 100. Check ulimit -Sn and ulimit -Hn, then explicitly raise only the soft limit in a subshell if the hard limit allows it:

(ulimit -Sn 65536 && ./bench/run.sh)

The benchmark harnesses leave their inherited limits unchanged. On Unix, wasmkeep serve separately attempts to raise its own soft limit within the inherited hard limit, then sizes its connection and resident-actor caps from the effective limit. This does not replace the harness preflight. Choose TMPDIR deliberately: RAM-backed storage and disk have very different creation and write costs. Record the filesystem type with results, using relative paths or placeholders for local directories. bench/run.sh records the kernel’s transparent huge page setting (/sys/kernel/mm/transparent_hugepage/enabled) in a last thp column, unknown where there is none. Earlier files (up to 351ea57) lack it, so check the machine by hand for those. Since commit 351ea57 the server turns them off for itself, so its numbers no longer depend on it, but runs before that on a kernel set to always read about twice the memory per actor (the workstation files cd48f41 to dec1a82). Do not compare memory across that boundary.

./bench/run.sh                     # criterion suites + the comparison (if npm is installed)
WASMKEEP_TEST_S3_BUCKET=... ./bench/run.sh   # also failover, with AWS_* pointing at an S3 API
WASMKEEP_COMPARE_BUCKET=s3://... ./bench/run.sh   # also the comparison on a bucket (compare-bucket/ rows)
WASMKEEP_COMPARE_BUCKET=s3://... BUCKET_LABEL=minio-tmpfs ./bench/bucket.sh     # that comparison alone, in a file of its own
WASMKEEP_COMPARE_BUCKET=s3://... CLUSTER_LABEL=minio-tmpfs ./bench/cluster.sh   # three nodes of WasmKeep and celld, with takeovers
target/release/compare --quick wasmkeep-rust workerd   # just the comparison, smaller

compare --bucket runs each WasmKeep target as a one-node cluster on the bucket WASMKEEP_COMPARE_BUCKET names, each run under a prefix of its own, so every write is also replicated: its rows are compare-bucket/.... Its startup_to_first_response includes the 15 s a cluster node’s first start waits before it listens (cluster). For a local bucket, the MinIO image the Compose example pins serves on loopback; give it a RAM-backed or a disk-backed data directory deliberately, as with TMPDIR, and record which. wasmkeep-rust-durable is the same counter with its POST durable before it returns (exec-durable): fsynced on one node, in the bucket with --bucket. The celld target (bucket runs only) is the workerd counter deployed to one celld node on the same bucket (celld-fleet and celld-bucket are its three-node targets); bench/celld/fetch.sh downloads the pinned release and checks its build provenance with gh attestation verify first.

Before publishing evidence, replace personal directory paths and generated scratch paths with repository-relative paths, [repo] or [scratch]. Stage only the intended artifacts, then run python3 -I scripts/check-benchmark-paths.py. It checks the staged Git blobs under all of bench/ and docs/, so redacting only the working copy cannot conceal an unsafe staged version. Reviewed system paths, product routes and documentation-site links are allowed; test files are scanned too. The check recognizes named-home paths and paths wrapped in Markdown emphasis, validates IPv6 CIDRs, and limits regex/printf escape exceptions to complete quoted patterns or reviewed printf examples. Adjacent private paths still fail. An optional pre-commit hook runs this check locally. CI checks files added or changed by every introduced commit with --range BASE..HEAD, including paths removed before the final commit. Unchanged files inherited from an old base are not attributed to the PR. CI runs after publication, so it cannot prevent the initial push. This checks local paths; review logs for other private information before staging them.

Regression check on pull requests

A pull request that changes the runtime, the server or the benchmarked examples runs scripts/bench-ab.py in CI: the base it merges into and the change are built and measured on the same runner, three rounds of the instantiate, http and actors Criterion suites with the two sides back to back. Each round gives a ratio per benchmark; the check fails if a benchmark’s median ratio is more than 20% slower. Three rounds at that threshold catch large regressions, not small ones, and numbers from shared CI runners are not comparable with each other or with the files in bench/results/: only the ratio between the two sides of one run is, and the job’s summary shows it.

hello/request/raw-hyper (plain hyper, no WasmKeep code) never fails the check; if it moved by more than 15%, the runner was too noisy and the run only reports. A base that cannot be built or measured is reported, not failed. To run it locally against any revision (it keeps its builds and results under target/bench-ab/):

python3 -I scripts/bench-ab.py main

Security model

What WasmKeep protects, against whom, and how; what it does not do yet. The limits below are enforced by the code (each has a test), not aspirations.

Pre-alpha. Run code you trust, or code from people you trust. The Wasm sandbox protects the host from buggy actors; it is not yet a promise against a determined, hostile tenant (side channels such as Spectre are out of scope until a process- or VM-per-tenant tier exists).

Who can do what

Layers

Front door (crates/wasmkeep-server/src/http.rs)

• 30 s to send request headers, and idle limit between kept-alive requests.
• 10,000 connections per node (--max-connections); more wait in the kernel backlog. Accept errors back off instead of spinning.
• Request bodies: 16 MB (413 when declared larger; a failed read past it). What a handler leaves unread is read and discarded, within the same 16 MB and for at most 30 s after the handler let go of the body (not the body of a request sent with Expect: 100-continue whose handler read none of it: it is never asked for, see concepts).
• WebSockets: 1 MiB per client message, 256 messages or 4 MiB queued per client, 1,000 sockets per actor and 50,000 per node, idle clients pinged then closed. A socket is invisible to the actor’s other events until on-websocket-open accepts it. Close reasons never carry host errors.
• x-wasmkeep-* headers from clients are stripped or ignored: only the host sets who is calling.

Who may create an actor (ADR 0010): per class, open (default), existing (outside requests reach only existing actors; nothing is created for unknown names), existing:NAME,..., or internal (actors only).

The guest sandbox

• Capabilities: only what the wasmkeep:actor world imports. Filesystem and sockets interfaces that toolchains import anyway are inert (no directories, every socket and lookup refused).
• CPU: 50 ms of guest time per event, and 2 s of SQL CPU per event (SQLite runs in the host, so it has its own budget, measured with the thread’s CPU clock and enforced inside a statement); wall clock 30 s per event (also for a stateless component’s request, response included).
• Memory: 128 MiB per linear memory, 2 memories and 32 core instances per component; resident actors capped at 90% of the instance pool, the least recently active idle one hibernating to make room.
• Outbound HTTP: public addresses only (the host resolves, checks and connects to the checked address; TLS verified against the name); --allow-outbound and --outbound-ca open specific internal services. The ports browsers block are refused too (the Fetch standard’s list: SMTP 25, 465 and 587, FTP, SSH, IRC and other non-HTTP services), since a crafted HTTP request can drive some of those servers; --allow-outbound-port opens one. 16 requests in flight per actor, 256 per node.
• What a guest prints reaches the node’s log within a line rate per instance and per node (--guest-log-rate; past it, lines are dropped and counted), cut at 4 KB a line, with control characters escaped: a guest cannot flood the log, forge entries of its own or send terminal escapes. What it prints is its choice, secrets included, so the node’s log is as sensitive as the actors make it.

SQLite sandbox (crates/wasmkeep-storage/src/db.rs)

• No ATTACH, VACUUM, extensions, fts3_tokenizer, or temporary tables, views, triggers or indexes (ordinary ones are allowed, but not on host tables); virtual tables limited to fts3/fts4/fts5/rtree; pragmas from an allow-list; defensive mode; trusted_schema=OFF.
• Host tables (_wasmkeep_*) invisible to guests.
• 1 GB per database; 16 MB per value and per query result; a statement stopped at the event’s wall clock.
• 256 MB of write-ahead log between two captures: past it, statements that write fail until the event ends (reads, commit and rollback still run). Capture reads the log frame by frame and holds only the newest version of each changed page, so rewriting the same rows costs no memory (a transaction touching many distinct pages still holds them all).

Between nodes (crates/wasmkeep-server/src/cluster.rs)

• Calls carry an HMAC-SHA256 signature (secret from the bucket, never sent) over the target node, method, path, the body’s SHA-256, caller, chain, time and a nonce unique to each send; valid for a minute, accepted once (the replay record expires in order and holds up to 500,000 signatures, several thousand calls a second into one node). The receiver checks the body against it before the actor runs (only a validly signed call is buffered). Heartbeats count only under their own key.
• Without a cluster listener, only with --cluster-plaintext accept (by default serve refuses to start a cluster node without one: nodes talk over mutual TLS by default): plain HTTP, not encrypted, and responses are not signed: keep node-to-node traffic on a private network. A node without one warns when it advertises a public address.
• With one (--cluster-listen, --cluster-advertise; ADR 0011): mutual TLS 1.3 to and from peers that also have one, each side’s certificate pinned in its live heartbeat; no session resumption; the client’s pin is checked again on every request. The port serves actor paths and the health and readiness checks only, and never forwards a request again.
• Revocation: wasmkeep cluster fence <node> (ADR 0020) writes a fence over the node’s heartbeat (its node lease): the running process stops at its next heartbeat write, no peer trusts it from its next listing, and the actors it held are free 15 s after its last write. A fence revokes that process, not the machine: a service manager that restarts it starts a new process, which passes its start check once the fence is 15 s old and is trusted again. To revoke a machine, also stop its service and disable the restart, and rotate what it holds: the bucket credentials, and the cluster secret (cluster/secret) if it may have been read. Deleting the heartbeat does not revoke cleanly: the node stops too, but its actors stay held until a fence or a new process with its id. With no heartbeat object, the fence cluster fence creates is dated by the bucket’s own time for that write, never earlier than the operator’s clock (a clock running behind would free the actors too early). A peer stops being trusted once its heartbeat is fenced or released, deleted, names a new key, or expires (its ttl, three heartbeat periods, plus the allowed clock skew); a node that cannot list the heartbeats trusts each pin only until its heartbeat’s expiry. From then on its handshakes are refused, its kept-alive requests get a 503, nothing is sent to it (pooled connections included), relayed WebSockets are cut on either side within 500 ms, and answers from it or to it (forwarded responses, actors.call) are not delivered, even if already streaming. A request it sent before that still runs to the end: only its answer is withheld.
• --cluster-plaintext refuse (the default; phase B of ADR 0011) closes the plaintext path: a node’s call counts only on the cluster port. A signed call on the public port is treated as an outside client’s (access rules apply, no caller, no chain, internal classes refused), and a valid signature there is spent, so a plaintext copy can never be replayed onto the cluster port; peers without a cluster listener are not used (their actors get a 503 from this node), and TLS peers are never reached in plaintext. The forwarded header gives no privilege on any port: it only stops a request being forwarded again.
• One app per tenant per bucket prefix, until each app owns its state (planned for 0.4.0, ADR 0021). Actors are keyed by tenant, class and name, so a second app of the tenant on the same prefix would share the first one’s actors. The first wasmkeep deploy of an app refuses when another app of its tenant is deployed on the prefix, and a node refuses to start while a node of another app of its tenant is live there. Not caught: two apps of one tenant deployed on one prefix before this check (they keep sharing actors; deploy and serve --app warn), two first deploys at the same moment, and a deploy by a CLI older than 0.3.0. Deploy the second app on another prefix, or under another tenant if another owner runs it (deploying).
• One bucket prefix is one trust domain: every node on it signs calls with the same cluster secret, whatever app or tenant it serves.

The bucket (crates/wasmkeep-server/src/replica.rs)

• Restore streams one object and one page at a time; chains that would build a database over twice the node’s cap, objects larger than such a database, and files without checksums are refused; every file’s chain checksums are verified before the result is put in place.
• Every read of a restore is given up after 5 s of silence and sent again, up to three times; then the restore fails and the next event restores from a fresh listing.
• A node proves at start, before it writes a heartbeat or takes a lease, that the bucket enforces conditional writes: on a throwaway object under probe/, a create over it and a replace with a stale version must be refused. A bucket that ignores them would let two nodes own one actor, so it is refused: the node stays not ready and serve exits (wasmkeep_cluster::bucket_check).

Configuration and secrets

--env CLASS=FILE (NAME=value files, readable only by the server’s user). Values never reach the data directory, the bucket or the logs; parse errors name the file and line, never the value.

wasmkeep deploy --config CLASS=FILE stores plain configuration in the bucket, in the clear, as part of the version (ADR 0013): only for what anyone who can read the bucket may see (whoever can read it can already read every actor’s data). Comments are not uploaded; names that look like secrets are warned about without the value; the manifest holds digests, not values; Debug output of configuration shows names only. Node --env values win by name.

wasmkeep secrets stores an app’s secrets in the bucket, sealed (ADR 0016): ChaCha20-Poly1305 under a key derived per app from a 32-byte secrets key, which the operator gives to the nodes (--secrets-key, refused if its group or others can read it) and to the machine running wasmkeep secrets. Each object opens only at its own bucket prefix, tenant, app and sequence number. Values come from stdin or a file, never from the command line, and are never printed or logged.

• Protects against anyone who can read the bucket, or its backups: they see neither names nor values, and cannot forge or move a sealed object.
• Does not protect against whoever can write the bucket: they can deploy code that reads env.get and sends the values out. Signed deploys (planned, not yet designed) would narrow that.
• A node that cannot open the current secrets (no key, another key, an altered object at once; 30 s without reaching the current set) refuses the app’s events with 503, hands its actors to a node that can, takes none, and reports not ready. Past those windows it never runs an actor with an older or empty set; when no node can open them, the app’s events are all 503.
• cluster/secret (node-call signing) is still stored in the clear; a bucket reader can sign node calls on a node started with --cluster-plaintext accept. With refuse (the default) node calls are honoured only on the mutual-TLS cluster port, whose peers are pinned in their heartbeats.

Access rules travel with the deployed version too: a node following an app applies the version’s rules, decided with the class each event takes, so no event runs new code under an old rule or the reverse. --access is refused with --app once rules are deployed. A rollback that reopens a class needs --allow-reopen.

Known gaps

• Hostile multi-tenancy: every actor of a node shares one process (see the note above).
• Node-to-node traffic goes over mutual TLS by default (ADR 0011): a node in a cluster needs --cluster-listen and --cluster-advertise, and refuses plain HTTP between nodes. A node started with --cluster-plaintext accept opens the plaintext path again: without a listener it reaches peers at their plain address, and with one it stays reachable, signed calls included, on its public port; there a response to a call is not signed, and someone on the path can read traffic and change responses. Refusing nodes still serve outside clients on the public port as usual, and the mode is not published: nothing stops a node from being started in accept mode later, so each node’s flags are what count.
• Host work other than SQL (encoding sends and HTTP, WebSocket frames) is not charged to any CPU budget; it is small per call, and the wall clock bounds it.
• In a cluster, a deleted actor of an existing class can be woken from outside (as an empty actor).
• Limits are per node. A process serves one tenant (several tenants per process is planned but not started), so today that is also per tenant; within a tenant, one class or actor can use the node’s whole share (instance pool, connections, sockets), bounded per actor only where noted above.

Audits

Before the first public release the code had an internal security review in ten parts (front door, guest boundary, cluster); each fix came with a test that fails without it. Later hardening: plain HTTP between nodes refused by default, outbound HTTP refusing the ports browsers block, and limits on what actors print reaching the node’s log (all described above). No independent third-party audit has been done yet.

Reporting a vulnerability

Report privately through the repository’s security advisories, not in a public issue.

Licensing and compliance

What WasmKeep’s license obligations are, what it depends on, and what users inherit when they build actors. The dependency list is kept current by scripts/licenses.sh.

WasmKeep itself

• License: Apache-2.0 for all code written for this repository (LICENSE), with its explicit patent grant from every contributor. See NOTICE.
• Name: the WasmKeep name (and any logo, once there is one) is reserved (TRADEMARK.md).
• Contributions: under the Contributor License Agreement (CLA.md). The CLA also lets the maintainer offer commercial licenses (ADR 0004); contributions stay available under Apache-2.0.
• Authorship: written by Julien Turbide with AI models (mostly Claude in Claude Code, with Codex and Kimi for parts and reviews) under his direction and review. The implementation is original; what comes from other projects is used as published, under its own licence: the WASI interfaces vendored in crates/wasmkeep-wit/wit/deps/ (and copied to sdks/typescript/wit/deps/, crates/wasmkeep-sdk/wit/deps/, sdks/go/wit/deps/ and sdks/python/wasmkeep/wit/deps/), published by the WebAssembly Community Group under the W3C Community Contributor License Agreement, bindings generated from WIT files, and the dependencies listed by scripts/licenses.sh (see the README’s Authors and History and use of AI). Where WasmKeep interoperates with another project’s format or protocol, it is an independent implementation of the published format:
  • LTX (the replication file format) matches github.com/superfly/ltx (Apache-2.0). The reference library is used only by the conformance tool in conformance/ltx (a development tool, never shipped), whose output files are the test fixtures in crates/wasmkeep-storage/tests/ltx.
  • WIT, WASI, the component model: open standards of the W3C WebAssembly Community Group and the Bytecode Alliance.

Dependencies

All Rust dependencies (Cargo.lock, about 400 crates) are under permissive licenses: MIT, MIT-0, Apache-2.0 (some WITH LLVM-exception), BSD-2/3-Clause, 0BSD, ISC, Zlib, Unicode-3.0, Unlicense, BSL-1.0, CC0-1.0 and CDLA-Permissive-2.0 (the Mozilla CA list in webpki-roots). None is copyleft. SQLite (bundled by rusqlite) is in the public domain.

scripts/licenses.sh lists every crate with its license and fails on one outside that set. cargo deny check (deny.toml) checks the same set, refuses crates from other registries or git, and fails on known vulnerabilities (RustSec); its one accepted advisory is bincode being unmaintained, reached only by the simulation tool. Run both before a release. The list is an inventory: it does not contain the notices themselves. A binary distribution must also carry each dependency’s copyright lines, NOTICE contents and license texts (Apache-2.0 and MIT require the notices to travel with the binary; CDLA-Permissive-2.0 requires its text). scripts/notices.sh gathers them with cargo about (about.toml): every crate in the binary for the release targets, grouped by license with each license’s full text, the CDLA-Permissive-2.0 text included, and the NOTICE files dependencies ship. The release workflow puts that file in every tarball, next to the binary. The example components a release carries (counter.wasm and the others) link the Rust SDK’s dependencies: scripts/notices.sh --components writes their notices the same way, for wasm32-wasip2, and the release attaches that file, THIRD_PARTY_NOTICES-components.txt, next to them. scripts/sbom.sh writes a CycloneDX SBOM of the wasmkeep binary for one target.

Development-only tools, never part of WasmKeep: workerd (Apache-2.0, in bench/workerd), the Go LTX conformance tool, wasm-tools and wit-bindgen (Apache-2.0 WITH LLVM-exception), Zig (MIT), TinyGo (BSD-3-Clause), componentize-py (Apache-2.0), mdBook (MPL-2.0, builds the docs site only).

What users inherit when they build actors

A component bundles its language’s runtime. Distributing a component (not running it on your own servers) carries that runtime’s license:

Running components on your own servers (the normal use of WasmKeep) is not distribution.

Patents

Apache-2.0 gives users a patent license from every contributor for their contributions. It says nothing about third parties’ patents. WasmKeep’s design draws on long-published, widely implemented ideas (the actor model, 1973; virtual actors as in Microsoft Orleans, 2014; single-writer leases on conditional writes; WAL shipping as in Litestream and LiteFS).

Trademarks of others

WasmKeep’s docs compare it with, or name, other products (Cloudflare Workers and Durable Objects, workerd, celld (Deno), wasmCloud, Spin, Rivet, Microsoft Orleans, Litestream, LiteFS, Fly.io, AWS S3, Cloudflare R2, OVHcloud Object Storage, MinIO, Ceph, Hetzner, DigitalOcean, Scaleway, Kubernetes, Firecracker). The names are used only to identify those products; they belong to their owners, and WasmKeep is not affiliated with or endorsed by any of them.

Privacy

WasmKeep collects nothing by default. The usage ping is opt-in (--usage-ping), sends the seven fields documented in telemetry.md, and has no built-in destination. Actor data stays on the operator’s machines and bucket.

Cryptography

TLS (rustls with ring) for actors’ outbound HTTPS and for mutual TLS 1.3 between nodes (certificates made in memory with rcgen), HMAC-SHA256 signatures on node calls, and ChaCha20-Poly1305 with HKDF-SHA256 (ring) for secrets sealed in the bucket. The bucket client (object_store) and the usage ping use rustls with AWS-LC (aws-lc-rs) for their HTTPS, and object_store also uses AWS-LC for S3 request signing. All are standard algorithms from published open-source libraries. Check the export rules of your jurisdiction before distributing binaries.

Usage ping

WasmKeep can send an anonymous usage report, so the project learns how many installations exist and what they look like. It is off by default: a node sends nothing unless it is started with --usage-ping.

wasmkeep serve --usage-ping --actor counter=counter.wasm

What is sent

One small JSON document, a minute after the node starts and then once a day, as an HTTP POST:

{"schema":1,"version":"0.3.0","os":"linux","arch":"x86_64","cluster":true,"classes":"1-9","resident_actors":"10-99"}

That is all. No identifier for the node or the installation, no host names, addresses, tenant, class or actor names, no request data, no configuration. The receiving end sees the connection’s IP address, as any HTTP server does; how the collector treats it will be stated here when one exists.

Each report is written to the node’s log (sending the usage ping) before it goes out, so you can see exactly what was sent. Failing to send is logged at debug level and never affects serving.

Where it goes

Nowhere yet: the project does not run a collector, so in current builds --usage-ping logs that it has no endpoint and sends nothing. When a collector exists, its address will be built into the release and listed here.

WASMKEEP_USAGE_PING_URL sends the report to another URL instead, for example your own collector.

The code is short: crates/wasmkeep-cli/src/usage.rs.

Dogfood log

What building real applications on WasmKeep taught us: every friction point, with what we did about it. Newest first within each app.

Watch (uptime monitor), examples/watch

One monitor actor per URL, checking on its alarm; one hub actor that creates monitors, records alerts durably and delivers them to a webhook with retries, and serves a live dashboard. About 600 lines of Rust in two components.

What worked well

• The actor model fits. Each monitor is a few dozen lines of state and one alarm. Scheduling, persistence and isolation came for free; nothing needed a lock, a queue or a cron daemon.
• Alarms as the schedule. “Check, record, set the next alarm” is the whole loop, and it survives hibernation, restarts and (since ADR 0009) the death of the node.
• exec-durable where it matters. The hub records alerts durably, so an acknowledged state change survives a crash, while ordinary bookkeeping stays fast.
• Calls with a trusted caller header. The hub accepts alerts only from monitors (x-wasmkeep-caller), with no shared secret to manage.
• WebSockets from the actor. The dashboard streams each monitor’s checks live with five lines of guest code.

Friction, and what we did

Alert delivery semantics

The hub durably records each monitor’s state change before sending a webhook. Repeated reports of its last recorded state are skipped, even if an alarm retry uses a new timestamp. A later up/down transition is a new alert, and each monitor is deduplicated separately. Existing alert history supplies the last state, so existing databases need no migration.

Webhook delivery remains at least once: a crash after the receiver accepts a webhook but before the hub durably records delivery can send it again. A lost HTTP response has the same ambiguity. This is not an exactly-once guarantee; a receiver requiring one needs its own idempotency handling. Failed deliveries remain pending and retry normally.

Releasing

For maintainers. A release is a signed tag on a commit of main that CI passed on; two workflows do the rest, and a person approves each step that cannot be undone.

What a release is

One version for all of it: the workspace version in Cargo.toml, which every SDK manifest carries (scripts/check-versions.py).

Steps

The SDKs are published with every release from 0.2.0 on; 0.1.0 is on no registry.

1. Prepare. In one pull request: the version in Cargo.toml and every manifest, publish = true in crates/wasmkeep-sdk/Cargo.toml (it stays so from then on), the release’s section in CHANGELOG.md (it becomes the release text), and install commands in the docs at the new version. Merge it, and wait for CI to pass on that commit of main.
2. Tag. Two signed tags on that commit: v<version>, and sdks/go/v<version> for the Go module. release.yml waits for CI on the commit, builds and signs everything, pushes the image and opens a draft release. A tag on a commit CI did not pass on fails here: run CI again on the commit, then the workflow again. Never move or delete a tag: the module proxy and the registries keep what they saw.
3. Publish the draft. Read it, then publish it. That starts publish.yml.
4. Approve the registries. publish.yml first checks the release (the tag is the version of every manifest, CI passed on its commit, the registries environment is protected), builds every package from that commit and asks each registry what it already has. Only then does each registry that lacks the version wait for a reviewer. Before approving, look at the run: it is publish.yml, started by the release, on the tag. Approving publishes: a version on a registry stays there for good.
5. Check from outside. In an empty directory: cargo add wasmkeep-sdk, npm install @wasmkeep/sdk, pip install wasmkeep and go get github.com/wasmkeep/wasmkeep/sdks/go@v<version> each give the new version, and each registry’s page names this repository as the source.

If a run stops half-way

• A registry refused, or an approval was rejected or ran out: use “Re-run failed jobs” on the run, or run the workflow by hand on the tag (Run workflow, with the tag as the ref). It publishes the same files.
• publish.yml itself is wrong at the tag: it cannot be fixed for that tag, since a run always uses the tag’s own copy. Publish what is missing by hand from a checkout of the tag (cargo publish, npm publish, twine upload), and fix the workflow for the next release.

Either way a registry that has the version starts no job and asks for no approval, and PyPI is asked file by file, so a release that got its wheel but not its source archive gets the missing one. Nothing is retried in a loop: a registry that refused keeps its error in the job’s log.

Set up once

The registries trust this repository’s publish.yml by its file name and its environment, with no stored token (“trusted publishing”). They do not check the branch or the tag: a copy of the file on any branch has the same identity. The registries environment is therefore the boundary, and its settings matter.

• GitHub, the registries environment: required reviewers; deployment limited to the tags v*; administrators not allowed to bypass it; “prevent self-review” once there is more than one maintainer. publish.yml refuses to run if the environment is missing, has no reviewers or accepts any ref.
• GitHub, the tags: the ruleset that keeps v* and sdks/go/v* from being moved or deleted. With several maintainers, also restrict who may create them.
• The names. A registry can trust a workflow only for a package that exists, except PyPI. Reserve the crate and the npm package once, by hand, with a placeholder version 0.0.0 that says so in its description: cargo publish of a crate named wasmkeep-sdk, and npm publish of @wasmkeep/sdk. Reserve the unscoped npm name wasmkeep-ts the same way, as a placeholder that points to @wasmkeep/sdk: it is the name of the SDK’s build command, and npx looks a command up as a package when it is not installed. Every real version then goes out through publish.yml, with a provenance statement.
• crates.io: in the crate’s settings, a trusted publisher: owner WasmKeep, repository wasmkeep, workflow publish.yml, environment registries.
• npm: in the package’s settings, a trusted publisher with the same values, with npm publish allowed (a new one allows only staged publishes otherwise). npm removes a trusted publisher that has not published within two days of being created: create it during step 1 of the first release, not earlier. The fields are case-sensitive, and repository.url in package.json must be this repository.
• PyPI: a pending publisher for the project wasmkeep with the same four values. The first run creates the project. A pending publisher does not reserve the name: make the first release soon after.

publish.yml keeps its file name and its jobs stay in it: a renamed or reusable workflow would be refused by the registries.

0001. Target WASI 0.2 and build Rust guests with plain cargo

• Status: accepted
• Date: 2026-11-02
• Milestone: M0

Context

Which WASI version to target was an open question at the start: wasi:http 0.2 is stable, 0.3 brings native async. Wasmtime 50 (pinned for M0) ships host implementations of both (wasmtime-wasi-http features p2 and p3).

The initial design named cargo-component as the Rust guest toolchain. Since Rust gained the tier-2 wasm32-wasip2 target, a plain cargo build --target wasm32-wasip2 of a cdylib already emits a component, and the official wasip2 crate provides the wasi:http/proxy bindings and export! macro.

Decision

• Guests target WASI 0.2 (wasi:http/proxy@0.2). The host links wasmtime_wasi::p2 and wasmtime_wasi_http::p2 only; the p3 features are off.
• Rust guests are built with plain cargo for wasm32-wasip2, using the wasip2 crate. cargo-component is not required. The target is listed in rust-toolchain.toml, so rustup installs it automatically.
• WASI 0.3 is revisited when the guest toolchains (wit-bindgen async, TinyGo, componentize-py, ComponentizeJS) treat it as stable, and at the latest when M1’s wasmkeep:actor world needs async exports. Because tenants program against WasmKeep’s own WIT world, the migration is confined to the host bindings and the SDK.

Consequences

• Every guest toolchain WasmKeep plans to support (Rust, TinyGo, componentize-py, ComponentizeJS) can produce 0.2 components today; 0.3 would exclude some of them.
• Request handling uses the synchronous-export, async-host model: one event at a time per instance, which is also the actor model’s contract.
• One fewer tool to install for contributors and in CI. The wasmkeep-sdk crate (M1) will use wit-bindgen directly for the wasmkeep:actor world.
• A later move to 0.3 is a breaking change for guests and needs its own ADR and migration note.

0002. The wasmkeep:actor WIT world, v0.1.0

• Status: accepted; extended to v0.2 by ADR 0003 (0.x may still break). Deferred items since: env added 2026-11-04 (deploys and secrets in ADRs 0013 and 0016); guest stdout and stderr reach the node’s log since 2026-11-06, not in M1 as planned (until then the host discarded them).
• Date: 2026-11-02
• Milestone: M1
• Source: crates/wasmkeep-wit/wit/actor.wit

Context

The initial design sketched the world (storage, alarms, actors.call, exports on-init, on-alarm, on-websocket-message, and the wasi:http handler) but left several types undefined (value, row, request, response, the error types) and imported wasi:logging, which Wasmtime does not provide. This ADR records how v0.1.0 fills those gaps. The interface is the product, so every choice below favours “small now, additive later”.

Decisions

Shape: two worlds

• actor-core: WasmKeep’s own imports (storage, alarms, actors) and exports (on-init, on-alarm). Host bindings (wasmkeep-wit, wasmtime bindgen!) and guest bindings (wasmkeep-sdk, wit-bindgen) are generated from this world only.
• actor: actor-core + wasi:http/proxy@0.2.12 + the wasi:cli imports that Rust’s standard library always pulls in (environment, exit, terminal-*), served empty by the host. This is the world components are validated against (wasm-tools component targets, in CI and later in wasmkeep deploy), so it doubles as the capability list: filesystem and sockets are not in it.

Why: generating bindings for actor would duplicate the WASI types already provided by wasmtime-wasi-http (host) and the wasip2 crate (guest). With the split, each side reuses the standard WASI bindings and adds only WasmKeep’s.

Storage

• value is a variant over SQLite’s storage classes: null, integer(s64), real(f64), text(string), blob(list<u8>).
• query returns rows { columns, values }: column names once, then rows as lists of values. Smaller on the wire than a list of records with names.
• One statement per call (multiple-statements error otherwise): no accidental batches, no stacked-query injection.
• Autocommit per statement; explicit BEGIN…COMMIT across calls inside one event is allowed, and a transaction left open when the event ends is rolled back. (No implicit per-event transaction: it would hold the write lock across outbound calls and surprise anyone who catches an error.)
• sql-error is a record of an enum kind plus a message, so guests can match on the kind (constraint, quota-exceeded, …) without parsing.
• exec-durable is in v0.1 even though replication arrives in M2, so guest code written now does not change then. On a node without a bucket it means “committed and fsynced locally”. This settles the open question (per-call opt-in vs always durable) as per-call opt-in.

Alarms

set-alarm(at), clear-alarm() as first sketched, plus get-alarm() (Durable Objects users expect it; a rate limiter needs it to avoid re-scheduling). At-least-once delivery is stated in the interface docs.

Actor-to-actor calls

call(address, request) -> result<response, call-error> with fully buffered request / response records (method, path, headers, body). The callee receives it through its normal wasi:http/incoming-handler, so there is no second export to implement and every language can handle it. address is class/name inside the caller’s tenant; cross-tenant calls are impossible by construction. call-error includes reentrant: with strict one-event-at-a-time execution, calling yourself or any actor already waiting in the call chain would deadlock, so the host refuses it up front.

Lifecycle exports return results

on-init and on-alarm return result<_, string>. A failing migration must fail the event instead of trapping; a failing alarm is retried.

Deferred, with reasons

• WebSockets (on-websocket-message, accept/send/close): without a send API an exported message handler is dead weight, and adding an export later is a breaking change anyway. They arrive together in a later minor version (ADR 0003).
• wasi:logging: Wasmtime has no host implementation. Guests write to stderr; the host will capture and tag it per actor (M1). Revisit if a standard logging interface lands in WASI.
• env (config and secrets): no example needs it yet.
• Actor identity (own address): the request path carries the name; add an import when an example needs it.

Consequences

• Guests implement two traits in Rust: wasmkeep_sdk::Actor (hooks, default no-ops) and wasmkeep_sdk::http::Guest, and call wasmkeep_sdk::export!.
• Every example is checked in CI against the actor world, so the WIT, the SDK and the examples cannot drift apart.
• The host must link only what the actor world lists. The server links all of WASI 0.2; restricting the linker to the world was planned for M1 and is still open.
• Any change to actor.wit needs a superseding ADR once 0.1.0 is published, and an RFC issue once the project takes outside proposals (see CONTRIBUTING.md).

0003. WebSockets in the actor world (wasmkeep:actor@0.2.0)

• Status: accepted
• Date: 2026-11-02
• Milestone: M1
• Supersedes: nothing; extends ADR 0002, which deferred WebSockets

Context

M1 calls for WebSockets in and out, with connections parked across hibernation, and the chat-room example needs them. ADR 0002 deferred them because an exported message handler without a send API is dead weight.

Decision

Package version 0.2.0 adds, to actor-core:

• An imported websockets interface: send(conn, message), close(conn, code, reason), connections(), with message = text(string) | binary(list<u8>) and send-error = unknown-connection | backpressure.
• Three exports: on-websocket-open(conn, path) -> result<_, string>, on-websocket-message(conn, message), on-websocket-close(conn, code, reason).

Semantics:

• The host accepts a GET with Upgrade: websocket on the actor’s path, answers 101, and asks the guest with on-websocket-open; an error closes the socket with 1008 and the error as reason. The guest never sees a socket, only a connection id.
• Every message and the close are mailbox events, serialized with the actor’s requests and alarms, in arrival order per connection. The reader waits for each event before reading the next frame: that is the backpressure.
• Sockets are owned by the host, so they stay open while the actor hibernates; the next message (or any other event) wakes it. This is the “park on the host” option from the initial design’s open questions; the alternative (close and ask clients to reconnect) is kept for actor migration in M2.
• A connection id is only valid in the actor it was opened on: send, close and connections are scoped to the calling actor.
• send never blocks the guest: 256 queued messages per connection, then backpressure (the message is dropped; the guest decides what to do).

Why not wasi:http upgrades

wasi:http 0.2 has no upgrade or bidirectional streaming story, and handling sockets inside the instance would pin it in memory, which defeats hibernation.

Consequences

• Breaking for guests at the binary level: components built for 0.1.0 lack the new exports and must be rebuilt. Source compatibility is kept by wasmkeep-sdk: the new Actor hooks have defaults (refuse the upgrade, ignore messages and closes), and every example rebuilt unchanged.
• Messages are text or binary; ping/pong is handled by the host. Close codes and reasons follow RFC 6455 (reasons truncated to 123 bytes).
• Migration note: rebuild your component against wasmkeep-sdk (or regenerate bindings against wasmkeep:actor@0.2.0) and implement the three hooks to accept WebSockets.

0004. A CLA, so the project can also be licensed commercially

• Status: accepted
• Date: 2026-11-02

Context

The project had to choose between a CLA and a DCO. A DCO is friendlier: a contributor certifies they have the right to submit their work, under the project’s license. A CLA also grants the maintainer direct, broader rights from each contributor, including licensing the whole codebase under other terms. I decided the project should keep the option to offer commercial licenses, so it asks for a CLA.

Decision

• Contributors sign an individual CLA (CLA.md) granting copyright and patent licenses that allow distribution under any terms, including commercial ones. Contributors keep their copyright.
• The core stays Apache-2.0; the CLA promises that accepted contributions stay available under Apache-2.0 (no relicensing away from open source).
• Enforcement by a CLA check on pull requests (the CLA Assistant Lite GitHub Action): a contributor signs once by posting a comment, and the signature is recorded in the repository. A corporate CLA is added when the first company contributes.

Consequences

• Some contributors avoid CLAs; the friction is one comment on the first pull request, and the reason is stated plainly in CONTRIBUTING.
• The name is protected separately (TRADEMARK.md).

0005. Replicate as LTX v3 files, with our own codec

• Status: accepted (2026-11-02: the codec round-trips against superfly/ltx v0.5.3 in CI)
• Date: 2026-11-02
• Milestone: M2

Context

M2 ships each actor’s committed changes to the bucket and restores them on another node. The open question was whether to reuse the LTX format (Litestream and LiteFS) for tooling compatibility, or a simpler format of our own.

Findings (2026-11-02):

• The reference implementation (github.com/superfly/ltx, Go) is at format version 3: LTX1 magic, 100-byte header, 6-byte page header with optional per-page LZ4, a 16-byte trailer with a CRC-ISO-64 file checksum, and rolling pre-apply / post-apply database checksums.
• The only Rust crate, litetx 0.1.0 (Fly.io), has one release from 2023-09 and implements the older 4-byte page header: it cannot read current files.
• Whatever we ship needs the same ingredients anyway: page-level transaction files, a monotonic transaction id, integrity checksums (on every segment, and wasmkeep bucket verify), and snapshots.

Decision

• Segments are LTX v3 files, written and read by our own codec in wasmkeep-storage (no dependency on litetx).
• Conformance is tested, not assumed: CI generates files with the Go ltx tool and checks we decode them, and checks the Go tool verifies files we write. Until that passes we do not claim compatibility.
• Fencing does not live in the file format: the lease generation is part of every object key (wal/<generation>-<txid>.ltx as first planned; the final key format is in ADR 0007), and conditional writes reject stale generations.
• Page payloads use LZ4 (lz4_flex, pure Rust) as LTX v3 does; snapshots are LTX snapshot files (all pages), not zstd copies of the database.

Alternatives

• Own format: about the same amount of code once checksums and snapshots are included, and no tooling. Rejected.
• litetx crate: unmaintained and incompatible with current LTX.
• Ship raw SQLite WAL files: WAL frames carry salts and checksums tied to one WAL file; restoring across checkpoints is fragile. LTX exists to fix this.

Consequences

• Operators can inspect and verify our bucket with Litestream’s ltx tool.
• We own a codec of roughly 600 lines plus tests; the spec is short and stable.
• New dependency when implemented: lz4_flex. CRC-ISO-64 is small enough to implement directly with a test vector.

Conformance notes (2026-11-02)

• conformance/ltx (Go, pinned to superfly/ltx v0.5.3) writes the fixtures in crates/wasmkeep-storage/tests/ltx and verifies the files our encoder writes. CI runs both directions and fails if the reference starts writing different fixtures.
• Files are not byte-identical across implementations: LZ4 output differs, and the file checksum covers compressed sizes and index offsets. Content, headers and both database checksums are identical.
• We do not read LZ4 frame-format pages (older writers); they are rejected as unsupported rather than misread.

0006. Deterministic simulation tool

• Status: accepted: madsim, after the bake-off below (it reversed the initial proposal)
• Date: 2026-11-02
• Milestone: M2

Context

The project requires a deterministic simulation suite for leases, failover and replication, built before the replication code, with seeded runs and fault injection, 200 seeds per push and 20,000 nightly. There are two candidates, madsim and turmoil, and the plan is to try both on a toy lease protocol first.

What we know before trying:

Either way, the cluster logic must not touch the network, clock or bucket directly: it is written against small traits (Bucket with conditional put, a clock, a node-to-node transport), so the simulation can drive it.

Initial proposal (superseded by the results)

turmoil, for being less intrusive than madsim’s global runtime swap.

The bake-off

1. A toy lease protocol on a Bucket trait (conditional create/update with versions, as S3 and MinIO provide), three nodes, a TTL and renewals.
2. The same protocol run under both tools with partitions, crashes, delays and a slow bucket, checking the invariant “never two live owners”.
3. A deliberately planted bug (renewal without the version check) must be found by both within the per-push seed budget, and a failing seed must reproduce exactly.
4. Record: time per 1,000 seeds, lines of harness code, whether failures reproduce, and anything that broke determinism. The ADR is then accepted or amended with those numbers.

Results (2026-11-02)

The same protocol code (sim/src/lease.rs then, now crates/wasmkeep-cluster/src/lease.rs, written against Bucket and Env traits), the same bucket semantics, the same seeded fault schedule (links held then released for up to 2.5 s, which is the late-write case, partitions, node crash and restart), 3 nodes, 20 s simulated per seed, debug build, 4 vCPU container:

Both find the bug and both are deterministic. madsim is about 100 times faster because it jumps virtual time to the next event, where turmoil steps a fixed 1 ms tick. At the nightly budget (20,000 seeds) that is about two minutes against several hours, and it leaves room for far more seeds.

The intrusiveness concern did not materialise: it only matters for code that runs under the simulation, and the simulated code is protocol logic behind traits, with no Wasmtime, SQLite or network dependencies.

Building the turmoil harness also surfaced a harness bug worth remembering: a simulated bucket that served one connection at a time let a single held client block everyone (turmoil reported the backlog overflow). A simulated dependency must be as concurrent as the real one, or it hides races.

Decision

• madsim for deterministic simulation; the turmoil harness was removed so only one is maintained.
• Cluster protocols (leases, replication ordering, fencing) are written as logic generic over Bucket, Env and transport traits, in modules that compile without tokio, hyper or object_store. Production adapters (object_store, tokio) implement the traits outside those modules. This is what lets --cfg madsim build them.
• CI runs the simulations with 200 seeds on every push; the nightly run uses 20,000 (release build).

0007. Replication protocol: chained segments and the acknowledgement rule

• Status: accepted (2026-11-02: the madsim replication case passes, and removing the acknowledgement rule is caught)
• Amended by ADR 0020 (2026-11-07): ownership is a node lease plus an ownership record per actor; where this record says lease, read the actor’s hold (LeaseView: the hold exists and the node lease is valid).
• Amended by ADR 0022 (planned, 0.4.0): history keys move under the app’s root, state/<tenant>/<app>/.
• Date: 2026-11-02
• Milestone: M2

Context

Leases (wasmkeep_cluster::lease) give each actor at most one owner at a time, under the skew bound. Replication must turn that into the guarantee WasmKeep promises: a write acknowledged as durable is never lost, and a stale owner can never corrupt what the next owner restores.

Two facts make this harder than “upload files”:

1. Ownership is time-based. An upload started while the lease was valid can complete after it expired, after another node took over and listed the bucket (a slow PUT that still succeeds).
2. A conditional create of a new key succeeds for a stale owner too. Key names alone do not fence.

Decision

Capture

Actor databases run in WAL mode with automatic checkpoints off. After each event commits, the replicator reads the WAL frames appended since its last position, keeps the last version of each page up to the last commit frame, and encodes one LTX file (ADR 0005) for the next transaction id. Pre- and post-apply checksums come from per-page checksums kept in memory. Once a segment is uploaded and the WAL holds only shipped frames, the replicator checkpoints (TRUNCATE).

Keys: every segment names its parent

actors/<tenant>/<class>/<name>/
  wal/<txid:016x>-<generation:016x>-<parent generation:016x>.ltx
  snapshot/<txid:016x>-<generation:016x>.ltx

A segment is transaction txid written by lease generation generation, applied on top of the segment at txid - 1 written by parent generation (0 for transaction 1, on the empty database). The objects therefore form a tree of histories. A snapshot is the database at one position of that tree. All writes are conditional creates; fixed-width hex makes listing order transaction order.

Acknowledgement rule

A segment counts as durable only if its upload completed while this node’s grant was still valid (grant.is_valid(now) checked after the PUT returns). exec-durable waits for that. An upload that completes too late is reported as a failure, and the node stops acting as owner; the write may or may not survive, which is what an unacknowledged write is allowed to do.

Restore

1. The head is the newest segment of the highest generation in the listing.
2. Follow parent links back to the empty database, stopping at the first snapshot on the way.
3. Apply that snapshot, then the segments after it. Each segment’s pre-apply checksum must equal the running database checksum.
4. A missing link or a checksum mismatch stops the restore: the actor is marked degraded and refuses events rather than silently rolling back.
5. (Added 2026-11-07.) The restore lists the prefix once, after acquiring, and fetches the objects that listing names. Every read is given up when the bucket stays silent for 5 s (no answer, no next chunk of a body, no next page of a listing) and sent again, up to three times; then the restore fails and the next event restores from a fresh listing. A request the bucket leaves hanging (one held a takeover for 19.8 s in the real-bucket soak) costs seconds, not the client’s 30 s per attempt. A read given up is dropped whole before anything of it is applied.
6. (Added 2026-11-08.) The restore reads up to 8 objects ahead of the one it applies, within 32 MiB of them by the listing’s sizes (always at least one), and a read is also given up, and sent again, once it has taken 5 s plus a second per started megabyte of everything read alongside it (its own size included) from its request, however it keeps moving. An object whose size is not the listing’s is refused. One node’s restores together read at most 32 objects at once, and at most 256 MiB of them at once by the listing’s sizes, one read taking at most 32 MiB of that however large its object (one kill starts many restores at once; a larger share would let one read hold every other restore behind it). Objects read wait to be applied in each restore’s own read-ahead (memory stays per restore). A read’s time starts once it has its turn, and counts every byte the node is reading alongside it, asked again while it lasts. Objects are still applied one at a time, in the plan’s order. In the 12-hour soak one restore of 55 objects took 56 s, each read about a second, none silent for 5 s.
7. (Added 2026-11-11.) A read the bucket refuses with an error a resend may get past is sent again after a pause (0.5 s, then 1 s), within the same three tries: a GET changes nothing, so a resend is always safe. (Corrected 2026-11-12.) During the pause the read gives up its place under the node’s limits of step 6 (its slot, its megabytes, its bytes counted in flight), which it used to hold while it only waited, and waits for one again before the resend; that second wait is not in the deadline below, as the first is not. An answer no resend changes (the object is gone, access is denied) still fails the restore at once. A restore’s reads together end within one deadline: 30 s, plus a second per started megabyte of the objects it reads (the rate below which one read is given up) and a second per eight objects. Its clock runs while the restore waits for the bucket to give it the object it applies next, and only then: not during the listing (its length is not known before it ends; it keeps the rule of step 5), not while an object is applied or the file is put in place (local work, never cut), and not while a read waits for the node’s shared limit on reads before it is asked for (many restores start at once after a kill, and a deadline counting their queue would end them all half-way). (Corrected 2026-11-12.) The deadline is sized from the restore’s own bytes, but a read’s limit (step 6) counts every megabyte in flight on the node, so restores reading side by side were each ended by their own deadline with every read inside its limit (32 actors of 8 MiB taken over at once: 261 s per read, 39 s per restore), and tried again together. For each object, the seconds that other restores’ bytes in flight added to the limits of its read’s tries (the limit as it is, less the limit with this restore’s bytes alone, most seen during the try) are therefore not counted, up to the time the restore waited for that object. What is counted for an object is then at most what its tries are allowed with no other restore on the node, so other restores’ reads never end a restore; one alone on the node has exactly the deadline above. Worst case: the deadline above plus 768 s per object (three tries, at most 256 s each from the node’s 256 MiB in flight), which the reads’ own limits already allowed; a bucket that never answers still ends a read, and the restore, after three silences of 5 s. A restore that fails before its database is complete leaves the local file and its sidecar untouched (the database is built beside it). One that fails while putting the complete database in place (a local disk error: the old file is set aside and its sidecar removed, the new file renamed in, its sidecar written) has set the old file aside, and leaves no file in place or the new one without a sidecar. Neither is used as it is: a local file spares the reads only when a sidecar names the chain’s head and the file’s checksum is the one the sidecar records, so the next restore reads the bucket again (and, if the new file was in place, sets it aside over the file set aside before: one generation is kept). In every case a failed restore writes nothing to the actor’s history and gives the lease up as a failed restore did before: the next event restores from a fresh listing, so it adds no ordering the protocol did not already have.

Why every acknowledged write is on that chain: generation h acquires only after every earlier grant ran out or was released, and lists after acquiring, so it sees every acknowledged upload. An acknowledgement needs the upload to have completed while the grant was valid; a node releases only after the actor’s state is dropped, so after its last upload completed and with nothing left to acknowledge. Either way, every acknowledged upload is in the bucket before h can acquire (S3 listings are strongly consistent). The highest generation m in its listing restored a chain containing everything acknowledged before m began (by induction), and only m could acknowledge anything after that; generations between m and h wrote nothing. So h restores, and builds on, a chain containing every acknowledged write. A stale upload that lands later is on no chain anyone built on. If nobody wrote after it, it extends the chain it was built on and counts as a late commit, which is allowed for an unacknowledged write. Releases make that window shorter: an upload cut short by the event’s wall-clock limit can still land after the next owner listed, if the node gave the actor up and released right after.

The simulation (sim/tests/madsim_replication.rs) checks this with owners that release (as on hibernation, shutdown and failed uploads), and catches a node that keeps working and acknowledges after releasing.

Snapshots and GC

Every 64 segments or 64 MB, the owner uploads a snapshot of its head. Snapshots off the current chain are never used. GC deletes segments and snapshots that are not needed to restore the current head, after a retention window, never on the write path.

Alternatives

• Start markers with fencing (the first version of this ADR): each new generation writes start/<g> with the txid it restored to, and restore drops lower-generation segments above it. The simulation broke it. A marker can itself land late: generation B’s marker upload is slow, B’s grant runs out and B never runs, generation C restores without seeing it and builds on a segment, and then B’s marker lands and fences off that segment. The result is a hole under C’s acknowledged writes (seed 10). Chained keys need no markers at all.
• Highest generation per transaction id, no links: this cannot tell a stale snapshot from a current one at the same transaction.
• A per-actor head object updated with compare-and-swap after every segment: this fences exactly, but doubles the bucket round trips of every durable write.

Consequences

• sim/tests/madsim_replication.rs runs three nodes on the real lease and planner code, with crashes, held traffic, skewed clocks and slow requests that succeed late. Over 200 seeds of 30 s: about 5,700 acknowledged writes, 2,300 ownership changes and 97 uploads landing after their grant, with every acknowledged write restored. Acknowledging late uploads is caught (seed 20; seed 24 since snapshots record their contents and are also taken right after a restore, 2026-11-05). The test also fails if late uploads stop occurring, so it cannot pass vacuously. Garbage collection (ADR 0012) runs in the same simulation. Since 2026-11-07 it runs on node leases (ADR 0020, wasmkeep_sim::node, whose holds act by the server’s own rule), with the lease on a task of its own and its writes bounded as the server bounds them: over 200 seeds, 7,061 acknowledged writes, 1,073 generations, 31 uploads landing after their hold ended and 796 fences, every write restored (2,000 seeds: 69,952 writes, 7,892 fences); acknowledging late uploads is caught at seed 0, after a release at 2, after a lapse ended the hold at 0.
• Restore planning is a pure function of key names, in wasmkeep-cluster.
• Clock skew beyond the configured bound can break the acknowledgement rule, exactly as it can break leases. The bound is a documented operating requirement (NTP).
• The simulation found a harness bug on the way: restarted nodes reused reply tags, so a dead process’s late reply could answer a new request. Tags now start at a random offset per client.

0008. wasmkeep deploy and rolling upgrades through the bucket

• Status: accepted (implemented 2026-11-02; the mixed-version test refuses old code, and removing the check fails it)
• Date: 2026-11-02
• Milestone: M2

Context

The last M2 item on the roadmap: wasmkeep deploy uploads artifacts and manifests, and nodes roll to the new version without dropping actors. Today each node is started with --actor CLASS=PATH and runs whatever files it was given. A cluster needs one source of truth for which code runs, with no control plane besides the bucket (a design rule, see the overview: if the bucket is up, WasmKeep is up).

What already exists:

• ActorService::replace_class swaps a class at the next event boundary. Running events finish on the old code, databases and sockets are kept, and on-init runs once per component version.
• The component cache compiles each artifact once per node.
• Leases make an actor run on one node at a time.

The new hazard is mixed versions during a rollout. Nodes pick up a new version at slightly different moments. Suppose an actor runs on-init for version 7 on node A (migrating its schema), then moves to node B, which still runs version 6. B would run old code on a migrated database.

Decision

Bucket layout

apps/<tenant>/<app>/<version>/<class>.wasm   as uploaded, immutable
apps/<tenant>/<app>/<version>/manifest       "wasmkeep-manifest 1", previous <version>, class <name> <sha256>
apps/<tenant>/<app>/current                  {version, seq}; replaced with If-Match only

The manifest is a few lines of text (no JSON dependency); seq lives in current. version is the first 16 hex digits of the SHA-256 of the class table, so a redeploy of the same code is a no-op. seq is a counter that increases with every change of current, including rollbacks.

wasmkeep deploy

1. Compile every component locally and link it against the actor world, with the same check serve runs. A broken component is refused before anything is uploaded.
2. Upload the artifacts and the manifest with conditional create. They are immutable; an existing object with the same key is the same content.
3. Read current, then replace it with {version, seq: old + 1}, using If-Match on the version read. A concurrent deploy fails loudly instead of silently overwriting.

wasmkeep deploy --rollback re-points current to the manifest’s previous, with a new, higher seq. Code never goes back in seq, even when it goes back in content.

Nodes

wasmkeep serve --bucket ... --app <tenant>/<app> replaces --actor. The node reads current at start and with every heartbeat (3 s). When it changes, the node:

1. downloads and compiles all the new classes;
2. swaps them in, and the seq with them, at once (apply_deploy).

A node that cannot build a version keeps the previous one, whole, and reports the error in its heartbeat.

No old code on newer data

Each actor’s database records, in _wasmkeep_meta, the highest deploy seq that has run an event on it. A node never runs an event on an actor whose recorded seq is above its own. Instead it re-reads current at once and upgrades first. If it still cannot (the new version fails to build on it), it refuses with 503 and Retry-After, so the actor is never downgraded. The check costs one meta read per activation, not per event.

Rollback stays safe: rolling back produces a higher seq, so it is allowed everywhere. Schema compatibility on rollback is the application’s job, as it is for any schema migration.

Alternatives

• Push to nodes (deploy calls every node). It needs node discovery and reachability from the deploy machine, and it fails partially. Polling the bucket fits “the bucket is the only shared state”.
• A version per actor in the lease record. That couples deploys to the lease protocol, which is proven and should stay small. The meta row lives with the data it protects and is replicated with it.

Consequences

• A rollout takes about one heartbeat interval (3 s) to reach every node. Actors switch at their next event, and none is dropped.
• New CLI command, new serve mode, a manifest format and a test: two nodes, deploy v1, write, deploy v2 with an on-init migration, move the actor to the node that hasn’t polled yet, and assert it upgrades first or refuses rather than running v1.
• Artifact garbage collection (old versions) joins the bucket GC work.

Implementation notes (2026-11-02)

• Classes dropped by a new version (or a rollback) stop being served: new requests get 404, and their actors’ databases are kept.
• A node that refuses an actor answers with an error and wakes its follower at once; in the test the node caught up and served within a second, instead of waiting for its next poll.

Revision (2026-11-04): the sequence travels with the code

The node’s sequence was read when an actor recorded it, separately from the class the event had taken. Classes were replaced one by one, then the sequence raised. That left three ways to break “no old code on newer data”, found from a 1-in-80 test flake under load and an adversarial review:

• Old code under the new sequence. An event took the v1 class, then waited (database open, instantiation); deploy 2 landed meanwhile; the event read sequence 2 and recorded it, running v1.
• New code under the old sequence. An event between “class replaced” and “sequence raised” ran v2 and recorded 1. A resident instance never re-recorded, so a node still on deploy 1 could later take the actor over and run v1 on data v2 had written.
• Half a deploy. A class that failed to build left the ones before it replaced and the sequence not raised, indefinitely (retried, and torn down, every poll).

Now each class carries the sequence it was swapped in under. An event takes both together, and records the class’s sequence, not the node’s. A deploy is compiled whole, then swapped in with its sequence under one lock. An instance records again whenever its class’s sequence moves past what it recorded (the same bytes redeployed keep their instances). A class no longer served refuses events, including for actors already resident. Tests: crates/wasmkeep-server/tests/deploys.rs, mutation-checked; a storage fault-injection delay holds an actor’s start-up while a deploy lands.

A second review round added: classes are stamped with their sequence when the node starts (ActorConfig::deploy_seq), so nothing runs unchecked before the follower’s first update; a class removed by a deploy closes its resident actors’ sockets (1012) and drops their alarms from the timer (they stay in the database and the bucket’s index); the follower backs off after a failed update (up to 5 minutes, wake-ups included) instead of recompiling every poll. A node serving --actor classes (sequence 0) does not check sequences at all: do not point one at a bucket that deployed apps use.

Revision (2026-11-05): rules, configuration and lineage

ADR 0013 adds access rules and plain configuration to the version (manifest format 2), puts previous into the version id (the class table alone made a directory reused under a stale previous), and adds a sticky third field to current that tools and nodes from before refuse.

0009. Alarms in a cluster: an index in the bucket and a sweep

• Status: accepted (implemented 2026-11-03, revised after three rounds of adversarial review the same day; index keys tagged with the owner’s generation and a nonce on 2026-11-05, closing a race with deletes that land late; R4, an alarm from before joining listed before the first upload, the same day). Mutation-checked tests cover the sweep delivering an orphaned alarm, sweeps getting past stuck entries, the cursor under an escaped prefix, committed retries, backoff for handlers that reschedule and fail, a fresh backoff per alarm, recording the listed entry, stale-entry cleanup, late deletes after a takeover and after an owner’s own reschedule (the alarm still delivered by a node that never ran the actor), the owner’s generation in new keys, and legacy entries next to tagged ones. Not covered by tests: listing from exec-durable, giving the actor up after a failed listing, the timer’s backoff horizon, resuming after the delivery cap, and sharding when nodes disagree; R1 to R4 and the fence are also checked under deterministic simulation (sim/tests/madsim_alarm_index.rs): three owners in turns on node leases (wasmkeep_sim::node, ADR 0020: holds, fences, releases), state uploads following replication’s late-commit rule, requests applied late, deletes timing out and landing later, crashes, with the invariant that a committed alarm’s named entry exists checked every 50 ms. Some nodes join with an alarm in their own data. 200 seeds per push; untagged keys, no generation check, no fence, a first upload before its entry, and clearing while the bucket has nothing for the actor are each caught by some seed (the last two with a budget of 2,000 seeds; the empty-bucket case is reached by a fault aimed at a joining node’s first upload, and a separate test shows the rules keep every entry under that fault). Cluster tests cover R4 (an_alarm_from_single_node_data_is_indexed_before_its_first_upload) and the entry left while the bucket has nothing (an_entry_for_an_actor_the_bucket_never_had_stays_without_creating_it). Not modelled there: exec-durable’s mid-event index update, deleting an actor, the PUT retry, legacy migration and timer delivery (cluster tests cover those). One known exception to the invariant as the simulation states it: if recording the new entry in the database fails after its PUT, the event is still replicated naming the previous entry (at another time), while the new, unnamed entry lists the alarm; nothing is lost (that entry is delivered, then removed as unnamed).
• Amended by ADR 0020 (2026-11-07): ownership is a node lease plus an ownership record per actor; where this record says lease, read the actor’s hold (LeaseView: the hold exists and the node lease is valid). sim/tests/madsim_alarm_index.rs runs on node leases since 2026-11-07 (wasmkeep_sim::node, holds acting by the server’s own rule): over 200 seeds, 1,511 takeovers, 3,993 alarms and the invariant held throughout (2,000 seeds: 15,086 takeovers); every unsafe variant is caught, within 20 seeds. On node leases a takeover during a joining node’s first upload needs that node cut off and fenced (its lease keeps renewing while the upload is only slow), so “clearing while the bucket has nothing” is reached by a fault aimed at that upload (delayed at the bucket, the node cut off), under which the rules also hold.
• Amended by ADR 0022 (planned, 0.4.0): the alarm index moves under the app’s root, state/<tenant>/<app>/.
• Date: 2026-11-03
• Milestone: M3

Context