Monitoring

What a node tells you about itself: two health endpoints on the main port, and Prometheus metrics on a port of their own.

Health and readiness

A node is not ready once it starts shutting down; in a cluster, until it has checked that the bucket enforces conditional writes and that no other process runs with its node id (for good if either fails: serve exits), while it has not loaded the cluster secret or its node lease (the heartbeat, ADR 0020) is not valid, which takes the bucket refusing or not answering its writes for the lease’s ttl minus its safety margin (8 s by default; calls and durable writes would fail); and when the disk holding --data-dir has less than 1 GiB free (or 5% of a smaller disk), sampled every 5 s off the request path. The public answer gives no reason (the port faces clients); wasmkeep_not_ready{reason} on the metrics page does.

A node that stops for good says why in the same words in that reason and at the start of serve’s exit line, followed there by the details:

Point load balancers at /_wasmkeep/ready, and use /_wasmkeep/health for “restart it if it stops answering”.

Open files

Every resident actor holds three open files (its database, write-ahead log and shared-memory index), every connection one. At start, wasmkeep serve raises its soft open-files limit to the hard limit, then caps connections and resident actors to what fits, logging a warning when that lowers them. With a cluster listener (--cluster-listen), its connections (up to 1,024) are counted first. Give the process a generous limit (LimitNOFILE=65536 in the systemd unit, or ulimit -n) to serve many actors at once.

The resident cap includes actors still closing their databases and blocking database work left by a cancelled event. An address without a mailbox (including a hibernated or failed-over actor) waits up to 250 ms for an idle actor’s resources to close. If capacity remains full, its request receives 503 Service Unavailable with Retry-After: 1. Registered mailboxes can still receive events, subject to their own queue limit (429). Relaying to another owner does not take a local actor slot when the owner is known (learned in the last second, or read from the actor’s ownership record within the admission wait); otherwise the request goes through local admission. Clients should honor the retry delay and bound their retries.

Alarms refused by admission use exponential backoff with jitter: a delay between half and all of a ceiling that starts at 100 ms and reaches 5 s. The retry counter below makes sustained admission pressure observable. WebSocket messages retain their frame while waiting up to the event wall limit; close callbacks remain pending through capacity or queue pressure. Closing sockets keep counting against socket limits until cleanup ends.

This admission rule uses the startup budget; it is not a live count of all process descriptors. Unrelated files and outbound connections still need the budget’s reserved headroom. The resident metric below counts registered mailboxes, so it can briefly fall before retiring resources release capacity.

Memory

Each resident actor costs about 250 KB for Rust and 740 KB for TypeScript on QuickJS (benchmarks); wasmkeep_actors_resident times that, plus the compiled classes, is roughly the process’s resident memory.

On Linux, wasmkeep serve and wasmkeep dev turn transparent huge pages off for their own process. Where the kernel sets them to always (some desktop and tuned distributions do; most servers use madvise), each actor’s memory would otherwise sit on 2 MiB pages it mostly does not use: on one such machine, a Rust actor measured 577 KB instead of 250 KB, with no gain in CPU or latency. To check a running node:

cat /sys/kernel/mm/transparent_hugepage/enabled   # the kernel's setting
pid=$(pidof wasmkeep)
grep THP_enabled /proc/$pid/status                 # 0: off for WasmKeep

--transparent-huge-pages system keeps the kernel’s setting instead. If the process cannot turn them off, it logs a warning and carries on.

Metrics

wasmkeep serve ... --metrics-listen 127.0.0.1:9100
curl 127.0.0.1:9100/metrics

Off by default. The page names classes and counts events, so bind it to loopback or a private address that only your Prometheus reaches. Labels are class names, event kinds and fixed reasons, never actor names.

Not there yet: per-tenant labels (one tenant per process until multi-tenancy), traps by reason (wasmkeep_events_total{outcome="failed"} and wasmkeep_cpu_budget_exceeded_total cover them for now) and parked WebSockets (sockets are not parked yet; wasmkeep_websockets_open counts them).

Bucket requests

Every bucket request a node makes is counted: leases, heartbeats, the alarm index, replication, restores and following a deployed app. operation is get, head, put (conditional writes included), delete, list, copy, rename or multipart. status is:

Watch status="error"; the other two are part of normal operation. A listing counts once per call, however many pages it took: S3 and R2 return 1,000 keys a page, so a restore of an actor with 10,000 objects in the bucket counts 2 list requests here but bills about 22.

Failovers

wasmkeep_failovers_total counts, on the node that took them, actors taken from another node whose node lease was fenced (or released with the actor still recorded as its own, ttl plus grace earlier): that node died, was cut off from the bucket, or stopped without releasing the actor (ADR 0020). An actor that moves because its owner hibernated it, shut down, or gave it up is a hand-off (the owner released it) and is not counted, nor is a node taking back its own actors after a restart. Sum it over nodes for the cluster; outside rolling restarts of crashed nodes it should stay near zero.

Grafana dashboard

deploy/grafana/wasmkeep.json shows every metric above except wasmkeep_classes, the guest log counters (wasmkeep_guest_log_lines_total, wasmkeep_guest_log_dropped_total), the history collection counters (wasmkeep_history_collected_total, wasmkeep_history_collect_failures_total), wasmkeep_snapshot_upload_failures_total, wasmkeep_signatures_refused_full_total, wasmkeep_websocket_owner_closes_total, wasmkeep_websocket_sweep_reads_total, wasmkeep_restore_failures_total, the listing and restore-read histograms (wasmkeep_listing_seconds, wasmkeep_restore_read_seconds), the node lease write and fence counters (wasmkeep_node_lease_writes_total, wasmkeep_node_fences_total) and wasmkeep_requests_abandoned_total. To see it on a running cluster, the Compose example starts three nodes with Prometheus and Grafana, the dashboard loaded. Otherwise import it in Grafana (Dashboards, New, Import) and pick your Prometheus data source; it needs Grafana 10 or later. It assumes Prometheus adds an instance label to each node’s series, as it does by default, and filters by node and class.

A test (the_grafana_dashboard_uses_only_metrics_that_exist) fails if the dashboard queries a metric the page does not have.

Alerts worth having

These are in deploy/prometheus/alerts.yml as Prometheus rules, with one more for a node Prometheus cannot scrape (up == 0 for a minute): a node that is down exports nothing, so none of the rules below would say so. Load the file with rule_files: in prometheus.yml and check it with promtool check rules after editing. It assumes a scrape job named wasmkeep, and the rules on durable write latency and on failed events wait 5 minutes before they fire. Two of the rules below hold “when no node was lost”: a rule cannot know that, so they fire on every fence and failover, a deliberate kill included, and their text says so. The Compose example loads the file. A test (the_alert_rules_use_only_metrics_that_exist) fails if a rule names a metric the page does not have.

• wasmkeep_node_lease_failures_total increasing on any node: two failures in a row and the node drops every actor it holds.
• increase(wasmkeep_node_fences_total[15m]) above 0 when no node was lost.
• histogram_quantile(0.99, rate(wasmkeep_durable_write_seconds_bucket[5m])) above 1 s.
• rate(wasmkeep_events_total{outcome="failed"}[5m]) above 1% of all events for a class.
• /_wasmkeep/ready failing on any node for 2 minutes.
• rate(wasmkeep_bucket_requests_total{status="error"}[5m]) above 0 for 5 minutes.
• increase(wasmkeep_failovers_total[15m]) above 0 when no node was lost.

What actors print

What an actor writes to stdout or stderr (Rust’s println! and eprintln!, console.log in TypeScript, print in Python, printf in C) reaches the node’s log, one entry per line, as target wasmkeep::guest with the actor’s address and the stream; a stateless component’s says actor=stateless:

INFO wasmkeep::guest: order 42 shipped actor=acme/orders/42 stream="stdout"

• Limits: --guest-log-rate lines a second per actor instance (50 by default, with bursts of twice that) and 1,000 a second for the whole node. Past them, lines are dropped and counted (wasmkeep_guest_log_dropped_total), and the actor’s next logged line is preceded by a warning saying how many were. --guest-log-rate 0 discards everything.
• Lines: a line is logged when its newline arrives; one left without it is logged when the instance ends (hibernation, a trap). Lines are cut at 4 KB, and control characters are escaped (\u{1b}), so a guest cannot forge entries or colour a terminal. Writes never block or fail the guest.
• Filtering: the usual RUST_LOG, e.g. RUST_LOG=info,wasmkeep::guest=warn to keep only the drop warnings, or wasmkeep::guest=off.
• Actors log what they choose to: whatever they print (a secret included) lands in the node’s log, and wherever that is shipped.