[RFC][Architecture] Control Plane Persistence Layer: Scalability, Durability, and Latency Evaluation

[MushuEE]

1. Objective & Context

Agent Substrate decouples the high-frequency lifecycle of Actors (sandboxed agent workloads) from Workers (physical Kubernetes Pods). To meet our North Star target of sub-100ms workload activation (wakeup) latency at a scale of 1 billion registered actors and 1,000+ concurrent wakeups/second, Substrate bypasses the standard Kubernetes API server for real-time scheduling and state routing, offloading dynamic states to a dedicated high-performance database.

This RFC aims to evaluate the trade-offs of our current database choice (Redis/Valkey) against alternatives like Aerospike or traditional consensus-backed relational databases (e.g. Cloud Spanner, CockroachDB), summarizing local benchmark results, production operations concerns, and data structure redesign requirements.

---

2. Architectural Dimensions & Trade-offs

To choose the right storage engine for Substrate's dynamic instance state, we evaluate candidates across five key dimensions:

Comparative Database Matrix

Dimension	Redis (Valkey)	Aerospike (Community)	Cloud Bigtable (NoSQL)	Cloud Spanner / CockroachDB	TBD (e.g. DynamoDB / ScyllaDB)
Primary Storage Model	100% In-Memory (RAM)	Hybrid (Index in RAM, Data on SSD)	Wide-column key-value (Colossus SSD)	Distributed transactional disk/file	Your proposal here
Write Latency	Ultra-low (sub-100µs)	Very low (sub-300µs)	Low (2ms – 5ms)	High (5ms – 15ms due to Paxos sync)	TBD
Consensus & Replication	Async or Sync via WAIT	Paxos-like mesh replication	High-speed master-replica replication	Paxos/Raft consensus (Strict serializability)	TBD
Node Consistency	Eventually consistent (lag risk)	Strong consistency per record	Strongly consistent per row (local)	External consistency (immediate)	TBD
License Type	BSD-3 (Valkey) / SSPL (Redis)	AGPL-3.0 / Commercial	Proprietary (Google Cloud Managed)	Proprietary / Business Source License (BSL)	TBD (prefer Open Source)
Managed Service Availability	High (Memorystore, ElastiCache)	Limited (Aerospike Cloud)	High (Fully managed by GCP)	High (Fully managed serverless)	TBD

---

Deep Dive: Production Concerns by Database

A. Redis / Valkey

• Data Durability Concerns: By default, Redis uses asynchronous replication. If a master node fails, writes in the replication buffer are lost. While appendfsync always provides complete durability, it historically degrades throughput on standard cloud storage.
• Operational Mitigation: Implementing the WAIT command allows synchronous replication (blocking writes until committed to $N$ replicas) without sacrificing in-memory speed.
• Memory Cost at Scale: At 1KB per actor record, storing 1 billion total registered actors requires 1 Terabyte of RAM, costing thousands of dollars per month in idle memory.
• License Considerations: Redis recently migrated to a restrictive SSPL/RSALv2 license. Valkey (a Linux Foundation-backed BSD-3 fork) is the approved, open-source drop-in replacement.

B. Aerospike

• Cost Efficiency at Scale: Unlike Redis, Aerospike stores the raw data records on raw block SSDs/NVMe, keeping only the keys/indexes in memory. This reduces the RAM footprint by up to 90%, making 1 billion actors highly cost-efficient.
• Consensus Model: Uses Paxos-like mesh clustering. It supports highly reliable, durable single-record writes with built-in generation checks.
• Community Restrictions: Community Edition (AGPL-3.0) lacks advanced features like multi-site cross-datacenter replication (XDR) and dynamic partition rebalancing under severe cluster splits, which are gated behind expensive enterprise commercial licenses.

C. Cloud Spanner / CockroachDB

• Write Latency Penalty: To guarantee absolute transaction durability, these databases require network roundtrips for consensus (Raft/Paxos) and fsync to disk. This increases write latencies to 5ms – 15ms, making them completely incompatible with a 100ms wakeup budget if placed in the critical path.
• The Hybrid Cache Model: In this model, Spanner acts as the durable, cold source of truth for 1 billion actors, while Redis/Valkey is used strictly as a transient, ephemeral session routing cache for currently active actors.

D. Cloud Bigtable (GCP Managed NoSQL)

• Write Latency & Throughput: Bigtable provides excellent write throughput and low latency (2ms – 5ms for writes under steady-state), which is significantly faster than Spanner because it bypasses multi-zone Paxos synchronous roundtrips on local writes.
• The Lock Challenge: Bigtable does not support multi-row ACID transactions or native atomic TTL/lease deletions (Bigtable garbage collection and TTL cells are eventually consistent and can take hours to delete). Implementing Substrate's lease-lock contract (AcquireLock/ReleaseLock) directly on Bigtable is very difficult and would require an external coordinator like etcd or Redis.
• Scheduling / Range Scans: Bigtable does not support $O(1)$ random member pops natively. To select an idle worker, we would need a row key indexing strategy (e.g., worker:<pool>#STATUS_IDLE#<pod_name>) to perform fast prefix range scans, avoiding full-table scans.
• Operational & License Concerns: Fully managed by Google Cloud, eliminating operational overhead. However, it is proprietary and has a minimum cost profile (minimum 1-node cluster cost) which can be high for small local developer setups.

E. TBD / Other Backends (e.g., AWS DynamoDB, ScyllaDB, Apache Cassandra)

• Encouraging Contributions: We strongly welcome community proposals and pull requests to implement additional backend stores that satisfy Substrate's store.Interface contract!
• Design Considerations: If you are interested in implementing a client for AWS DynamoDB, ScyllaDB, or Cassandra, please use this RFC issue thread to discuss key schema patterns, distributed locking strategies, and how you plan to avoid $O(N)$ worker range scan overhead.

---

3. Local Benchmarks Performed

We implemented a comparative, unified benchmark suite (store_benchmark_test.go) to test Valkey/Redis against Aerospike on a high-performance local workstation.

Benchmark Configuration:

• Valkey/Redis: Standalone on localhost:6379, tested with everysec (default) and always (sync fsync) AOF policies.
• Aerospike: Community Edition 8.1.2 on localhost:3000, using optimized PutBins APIs and generation concurrency.
• Hardware: Intel(R) Xeon(R) CPU @ 2.20GHz, backed by high-speed local SSD storage.

Note

Important Networking Caveat: These benchmarks were performed locally using virtual interfaces (vNIC) on Docker containers on a single host workstation. Network hop latency was negligible (<10µs). In a distributed production GKE cluster where client pods and database pods reside on different physical nodes, each query roundtrip will incur real cross-node network hop latency, which typically adds ~300µs – 500µs to every database request. This overhead must be factored into production latency projections.

Performance Comparison (Actor Creation / Writes):

BenchmarkCreateActor_Redis-24                  14149   79,814 ns/op   1,518 B/op   25 allocs/op
BenchmarkCreateActor_Redis_AlwaysFsync-24      13494   87,054 ns/op   1,388 B/op   24 allocs/op
BenchmarkCreateActor_Aerospike-24               4785  252,157 ns/op   1,867 B/op   27 allocs/op

gantt
    title Write Operation Latency (ns)
    dateFormat  X
    axisFormat %s
    section Latencies
    Redis (everysec)  :active, 0, 79814
    Redis (always fsync) :crit, 0, 87054
    Aerospike (durable NVMe) : 0, 252157

Loading

Crucial Benchmark Insights:

1. Local SSD/NVMe Performance: Enabling appendfsync always on Redis only increased latency by ~9% (shifting from 79.8µs to 87.0µs). On ultra-fast storage backings (like GCP Local SSDs), Redis can achieve RPO = 0 durability with virtually zero latency penalty.
2. Aerospike Performance: At 252 microseconds per write, Aerospike is ~3.1x slower than Redis but remains exceptionally fast, consuming only 27 CPU allocations and operating well within our sub-millisecond budget.

---

4. Core Data Structure & Scheduling Requirements

Regardless of the chosen storage engine, Substrate's control plane must refactor two legacy database patterns to support scaling:

A. Eliminating the $O(N)$ Scheduling Bottleneck

Currently, when an actor is resumed, the AssignWorkerStep calls ListWorkers (running a full-table keyspace scan over all master nodes), filters for idle ones, and shuffles them in memory. This scale-blocking bottleneck must be replaced by an $O(1)$ Idle Worker Queue/Set:

graph LR
    subgraph Idle Worker Pool
        Set["pool:{default:pool-alpha}:idle_workers"]
        Set -->|1. SPOP| Worker["pod-abc"]
    end
    ControlPlane -->|Claim Worker| Set
    ControlPlane -->|2. Update Worker State| DB_Worker["worker:default:pool-alpha:pod-abc"]

Loading

B. Slot-Aware Sharding (Multi-Key Transactions)

Redis Cluster restricts transactions (WATCH/MULTI) to keys that hash to the exact same cluster slot.
To atomically assign a worker to an actor without key collision errors:

• We must co-locate keys on the same shard using Redis Hash Tags (e.g. {actor:<actor-id>}).
• Example keys:
  • Actor: actor:{actor-123}
  • Worker lease: {actor:actor-123}:worker

---

5. Discussion Points for the Community

We would love feedback from the maintainers and community on these directions:

1. Should we officially support a Hybrid Cache architecture?
   Using Cloud Spanner/Postgres as the persistent registry for 1 billion actors, while keeping Valkey/Redis solely as a transient, fast in-memory session routing table for currently active actors.
2. Does the community favor open-source Valkey (BSD-3) or the commercial/AGPL Aerospike Community Edition?
   Considering the operational overhead and licensing constraints of both.
3. Should we proceed with implementing the $O(1)$ Idle Worker Set scheduling pull request?
   We have prototyped this locally and can submit it to clean up the $O(N)$ scan bottleneck in workflow_resume.go.

[brandonrjacobs]

Will there be a community meeting / gathering upcoming? we have some experiences with option 1.

[thockin]

Short answer: sure, why not?

Longer answer: Let's give it a week or maybe 2? :)

[MushuEE]

https://about.gitlab.com/blog/how-we-diagnosed-and-resolved-redis-latency-spikes/: They found that when the database size is large (tens of gigabytes), allocating the page tables for the child process during a fork() takes tens to hundreds of milliseconds, during which the primary event loop is completely frozen.

Dragonfly: Redis Scaling & Multi-Threaded Architectural Pitfalls: Breaks down the exact OS-level limits of fork() copy-on-write memory duplication spikes (which can double RAM consumption and trigger OOM crashes) and replication buffer overflows under high QPS. (I am already dubious of RAM limitations with 1B actors NFR.

Redis/Valkey Deep Dive & Control Plane Persistence Layer Evaluation

Team Summary & TL;DR

• My Understanding Reached: Redis/Valkey cannot be utilized as a unified, strongly consistent (RPO = 0) persistence engine for Substrate's 1 Billion Actors. Running Valkey in a highly durable posture breaks the single-threaded execution model, creates devastating tail-latency walls, and risks silent data loss during failovers.
• The Architectural Split: Substrate must adopt a decoupled hybrid architecture:
  1. Durability Layer (Persistent Registry): A distributed, consensus-backed relational or wide-column store (e.g., Cloud Spanner, CockroachDB, or a highly indexed Cloud Bigtable schema) to handle long-term actor states, metadata, and checkpoints safely.
  2. Speed Layer (Ephemeral Cache & Scheduler): Valkey/Redis running 100% in-memory (appendfsync no, async replication) to handle scheduling indexes ($O(1)$ idle_workers queue) and dynamic session routing at sub-100µs latencies and 50k+ QPS.

---

Section 1: Redis / Valkey Under Extreme Durability Constraints

1. Fsync Always Bottleneck: Event Loop Serialization

To guarantee a zero-data-loss SLA (RPO = 0), Redis/Valkey must write to disk on every transaction by configuring appendfsync always. In a single-threaded event loop, this converts a ultra-fast memory engine into a highly bottlenecked I/O blocker.

sequenceDiagram
    autonumber
    Client->>Valkey Master: 1. Write Command (e.g., UPDATE Actor State)
    activate Valkey Master
    Note over Valkey Master: aeProcessEvents reads from socket
    Note over Valkey Master: Execute modification in-memory
    Note over Valkey Master: write() user buffer to OS Page Cache
    Note over Valkey Master: Invoke fsync() system call
    activate Kernel I/O
    Valkey Master-->>Kernel I/O: Suspend event loop thread
    Note over Kernel I/O: Package dirty pages into bio structures
    Note over Kernel I/O: Dispatch blocks to NVMe physical write queue
    Note over Kernel I/O: Flash Translation Layer (FTL) performs NAND erase/write
    Kernel I/O-->>Valkey Master: Hardware Interrupt / Unblock thread
    deactivate Kernel I/O
    Note over Valkey Master: Resume event loop
    Valkey Master->>Client: 2. Return ACK / Success Response
    deactivate Valkey Master

Loading

Failure Mechanics:

• System Call Blocking: By default, standard system writes are asynchronous (returning once copied to kernel buffers). Under appendfsync always, the event loop must synchronously wait for fsync() to complete. This blocks the single execution thread entirely.
• Physical NVMe Controller Saturation: The NVMe FTL controller encounters severe Write Amplification (WAF) due to physical flash erase-and-write cycles on persistent blocks.
• The Latency Wall: This stalls the event loop cycle time from $&lt;100\mu\text{s}$ to $1\text{ms} - 3\text{ms}$, dropping throughput ceiling to $\sim 300 - 1,000$ QPS per shard and triggering an exponential tail-latency queueing wall.

---

2. Replication & The WAIT Command: Network AZ Saturated Blocking

To avoid local disk I/O bottlenecks, clients can issue the WAIT <num_replicas> <timeout> command to achieve synchronous replication across Availability Zones (AZs).

graph TD
    subgraph "gRPC App Server"
        Pool[Application Thread Pool]
        C1[Client Thread 1]
        C2[Client Thread 2]
        C3[Client Thread 3]
    end

    subgraph "Valkey Cluster (Cross-AZ)"
        Master[Valkey Master Node]
        Replica[Valkey Replica Node]
    end

    C1 -->|Issue Write & WAIT| Master
    C2 -->|Saturate Connection| Master
    C3 -->|Queue up...| Master
    
    Master -->|1. Stream replication buffer| Replica
    Master -.->|2. Event Loop BLOCKED on WAIT| Master
    
    Replica -->|Cross-AZ Network Jitter RTT: 10ms - 50ms| Replica
    Replica -->|3. REPLCONF ACK| Master
    
    style Master fill:#f96,stroke:#333,stroke-width:2px
    style Pool fill:#f99,stroke:#333,stroke-width:2px

Loading

Failure Mechanics:

• Baseline Latency Inflation: Cross-AZ network round-trips introduce a physical latency floor of $1.5\text{ms} - 3\text{ms}$, immediately breaking the sub-10ms NFR requirement.
• Network Jitter Cascades: Micro-congestions in cloud networking (hypervisor CPU steals, TCP retransmissions) stall the master event loop during the WAIT sync block.
• Thread Pool Starvation: If a replica slows down, client connections quickly build up at the application layer. The gRPC thread pool is rapidly exhausted, leading to cascade service outages.
• Consistency Ambiguity: If WAIT hits its timeout, it returns a count of replicas synchronized fewer than requested. The state is now in an ambiguous split state, failing the zero-data-loss SLA.

---

3. Consensus Deficit: Split-Brain & Last-Write-Wins Override

Redis Cluster relies on asynchronous gossip and master-majority votes for cluster health and failovers. It lacks formal Paxos/Raft state-machine consensus.

sequenceDiagram
    autonumber
    Note over Partition A: Isolated (Split-Brain)
    Client A->>Old Master (M1): Write State "A" (Accepted locally)
    
    Note over Partition B: Active Majority
    Replica 1 (R1)->>Replica 1 (R1): Detect M1 offline
    Note over Replica 1 (R1): Majority elects R1 as New Master (M2)
    Client B->>New Master (M2): Write State "B" (Accepted)
    
    Note over Partition A & B: Network Partition Heals
    Old Master (M1)->>New Master (M2): Join cluster config
    Note over Old Master (M1): Realizes demotion to Replica
    Note over Old Master (M1): DISCARDS ENTIRE DB (Wipes local state "A")
    New Master (M2)->>Old Master (M1): RDB Snapshot Sync (Overwrites M1 with state "B")
    Note over Old Master (M1): State "A" is lost silently forever

Loading

Failure Mechanics:

• Split-Brain Dual-Writes: If a network partition isolates the primary master (M1), a replica (R1) in the majority partition will be promoted to be the new master (M2). Clients in both partitions will write concurrently.
• Eventual Overwrite Wiping: Once the partition heals, the old master M1 is demoted. Because replication is eventually consistent, M1 must throw away its entire local database and reload the RDB snapshot of M2, resulting in silent, catastrophic write loss.
• Valkey 8 Core Improvements:
  • Dictionary-per-Slot Model: Wipes out slot-mapping metadata, saving 16 bytes per key-value pair (saving 16 GB RAM for 1 billion actors).
  • Key Embedding: Eliminates SDS external pointers, reducing key memory footprints by 8-10%.
  • Dual-Channel Replication: Decouples RDB snapshot streams from dynamic write commands, eliminating replication buffer overflows under high QPS.
  • Fork-less Snapshot Optimization: Minimizes parent thread copy-on-write overhead to prevent Out-of-Memory (OOM) failures during database dumps.

Memory Footprint at Scale

A. Raw Actor Payload Sizing (from ateapi.proto)

Based on the Actor message schema:

1. actor_id (UUID string, e.g., "actor-35400dc4-d56b-46d1-916f-ce79596e2add") $\to \mathbf{42 \text{ B}}$
2. version (int64) $\to \mathbf{8 \text{ B}}$
3. actor_template_namespace / name (e.g., "default" / "standard-agent-template") $\to \mathbf{30 \text{ B}}$
4. status (Enum) $\to \mathbf{4 \text{ B}}$
5. ateom_pod_namespace / name / ip (e.g., "workers" / "worker-pod-abc-123" / "10.120.15.24") $\to \mathbf{50 \text{ B}}$
6. last_snapshot / in_progress (GCS URIs) $\to \mathbf{160 \text{ B}}$

• Raw Binary Footprint: $\approx 294$ Bytes.
• JSON-Serialized Footprint (Protojson): Because ateredis.go serializes the Actor message as a JSON string, field name keys and quotes inflate the raw payload size to ~400 to 450 Bytes of text.
• Key Name size: "actor:actor-35400dc4-d56b-46d1-916f-ce79596e2add" $\to \mathbf{48 \text{ B}}$

B. Physical Redis/Valkey Allocator & Metadata Overhead

Valkey/Redis allocates memory using jemalloc, which rounds allocations to fixed bin sizes. Each string key-value pair carries structural overhead:

1. Dict Entry: $32$ Bytes (pointers for hashing slots on 64-bit).
2. Key Robj: $16$ Bytes (internal object metadata).
3. Key SDS String: $8$ B header + 48 B string + 1 null B = 57 B $\to$ Rounded up by jemalloc to 64 Bytes.
4. Value Robj: $16$ Bytes.
5. Value SDS String: $8$ B header + 400 B JSON + 1 null B = 409 B $\to$ Rounded up by jemalloc to 512 Bytes.

• Total Physical Memory per Actor:
  $$\text{RAM/Actor} = 32 (\text{Dict}) + 16 (\text{Key Robj}) + 64 (\text{Key SDS}) + 16 (\text{Value Robj}) + 512 (\text{Value SDS}) \approx \mathbf{640 \text{ Bytes}}$$

(Valkey 8's slot-mapping dictionary removal and key embedding cuts this down to $\approx \mathbf{550\text{ Bytes}}$).

C. Scale Requirements for 1 Billion Registered Actors

If the entire registry is stored in Valkey memory:

• Total RAM Required:
  $$1,000,000,000 \text{ actors} \times 640 \text{ Bytes/actor} \approx \mathbf{640 \text{ GB of RAM}}$$
• Cluster Cost (GCP/AWS): Storing 640 GB in memory requires 16 database nodes (8 active shards + 8 replicas for High Availability with 128GB RAM each). Operational TCO is ~$16,000 / month strictly in memory instance costs.

D. The Ephemeral Cache Optimization (Cache-Aside Model)

Since agent workloads are idle most of the time, we should only keep currently active actor states in memory:

• At any peak second, suppose we have 100,000 active actors residing on workers.
• Active Cache RAM:
  $$100,000 \text{ active actors} \times 640 \text{ Bytes/actor} \approx \mathbf{64 \text{ MB of RAM}}$$
• TCO Advantage: Valkey active cache runs on a tiny $5/\text{month}$ micro-instance, while 1 Billion cold records are stored on SSDs in Spanner/Bigtable for ~$25/month, dropping database costs by 99.8%.

[MushuEE]

RFC: Aerospike as a Hybrid Flash Persistence Engine

deep dive on Aerospike Community Edition to see if it can act as a strongly consistent (RPO = 0) data store for Substrate's persistent registry.

Because Aerospike bypasses the OS filesystem (VFS) and writes directly to raw NVMe block storage, I wanted to see if we could use it to handle our strict sub-10ms latency budget and 1 Billion registered actors.

Here is the detailed breakdown of where it fits, where it falls on its face, and why it's unfortunately a no-go for our consistency goals.

---

TL;DR / Executive Summary

• The Consensus: Aerospike Open Source (Community Edition) is a no-go for Substrate's persistent registry.
• The Blocker: While Aerospike's hybrid memory engine is incredibly cost-effective, Strong Consistency (SC) is strictly an Enterprise-only commercial feature. The open-source Community Edition (AGPL-3.0) only supports AP mode (eventually consistent).
• The Risk: During network partitions, Aerospike CE will suffer from split-brain writes, dirty reads, and silent write loss, failing our strict zero-data-loss (RPO = 0) SLA.

---

Deep Dive: Aerospike's HMA & Block-Storage Engine

Aerospike uses a Hybrid Memory Architecture (HMA): it stores the primary indexes in RAM and offloads record payloads directly to NVMe SSDs.

1. Bypassing the VFS: Raw Block sequential Buffering

Instead of writing records through a traditional filesystem layer (which introduces OS buffer overhead and sync locks), Aerospike interacts directly with raw NVMe block storage.

graph TD
    subgraph "Aerospike Server Process (RAM Buffer)"
        Record[Incoming Write Request]
        Buffer["Large Write Block Buffer (wblocks: 8MB)"]
        Index["In-Memory Index (Key Digest -> Physical NVMe Offset)"]
    end

    subgraph "Operating System Layer"
        VFS["Virtual File System (VFS) - BYPASSED"]
        PageCache["Kernel Page Cache - BYPASSED"]
    end

    subgraph "Physical Storage Layer (NVMe SSD)"
        RawBlock["Raw Block Device (/dev/nvme0n1)"]
        NAND["Physical NAND Flash Cells"]
    end

    Record -->|Buffer command| Buffer
    Buffer -->|Buffer Full / Flush Timer| RawBlock
    RawBlock -->|Sequential Raw block writes| NAND
    
    Record -->|Update digest metadata| Index

    style VFS fill:#ddd,stroke:#999,stroke-dasharray: 5 5
    style PageCache fill:#ddd,stroke:#999,stroke-dasharray: 5 5
    style RawBlock fill:#f96,stroke:#333,stroke-width:2px

Loading

How it works:

• Large Write Blocks (wblocks): Writes are accumulated in memory buffers and flushed to NVMe block partitions in large 8MB write blocks (wblocks). This aligns perfectly with SSD page mapping, keeping the Write Amplification Factor (WAF) near 1.0 and preventing write-wear latency spikes.
• Predictable 1-I/O Reads: The index is in memory, mapping the key digest directly to its SSD physical offset. Finding a record is a local $O(1)$ CPU lookup, meaning any read query takes exactly 1 physical SSD block read I/O, guaranteeing highly consistent sub-millisecond $p99$ tail latencies.

---

2. Memory & Cost Footprint at Scale (Empirical Calculations)

Let's run the math for 1 Billion registered actors based on the fields in ateapi.proto (UUID, templates, pod IPs, GCS URIs):

• Raw JSON payload: $\approx 400\text{ Bytes}$.
• Key name: "actor:<actor-id>" $\to$ $\approx 48\text{ Bytes}$.

Here is how Aerospike's RAM footprint compares to Valkey:

• Valkey/Redis (All in RAM):
  Valkey's extensive metadata wrapping (dict entry pointers, robj headers, jemalloc page rounding) inflates our 400B payload to a physical footprint of ~640 Bytes per actor.
  • Valkey RAM required: ~640 GB (requires a 16-node HA cluster costing ~$16,000 / month).
• Aerospike HMA (Index in RAM, Data on SSD):
  Aerospike stores only a fixed-size primary index entry of exactly 64 Bytes per record in RAM, offloading the 400B JSON payload entirely to SSD.
  • Aerospike RAM required: ~64 GB of RAM + ~400 GB of SSD (requires only 2 database nodes costing ~$300 / month). This is a 90% memory footprint reduction!

---

3. The Gotchas: Consistency Limits & Defrag Storms

Despite the massive cost savings, Aerospike CE has two major operational challenges that make it a no-go for us:

A. The AP Mode Split-Brain Hazard

Paxos is only used for cluster membership maps; it is never used on the transaction hot path.
Because Strong Consistency (SC) mode is an Enterprise-only commercial feature, the open-source Community Edition (AGPL-3.0) operates strictly in AP mode (eventually consistent).

sequenceDiagram
    autonumber
    Note over Partition A: Isolated Old Master
    Client A->>Node 1 (Old Master): Write "v2"
    Note over Node 1 (Old Master): Accepts write (AP Mode)
    
    Note over Partition B: Majority Partition
    Note over Node 2 & 3: Paxos Gossip detects Node 1 missing
    Note over Node 2 & 3: Re-map Partition Maps (Node 2 becomes New Master)
    Client B->>Node 2 (New Master): Write "v3"
    Note over Node 2 (New Master): Accepts write
    
    Note over Partition A & B: Network Partition Heals
    Node 1 (Old Master)->>Node 2 (New Master): Re-sync partition map
    Note over Node 2 (New Master): Conflict Resolution: Generation checks
    Note over Node 2 (New Master): Sees "v3" is newer / higher generation
    Note over Node 1 (Old Master): Discards "v2" (Dirty Write Loss)

Loading

During network splits, the isolated master will continue accepting writes. Once healed, Aerospike CE resolves conflicts using generation metadata, silently discarding the isolated writes and resulting in permanent data loss.

B. The Defragmentation "Defrag Storm"

As records are updated/deleted, background defragmentation scans raw blocks, consolidates live records, and frees space, creating background Write Amplification (minimum 2X).
Under continuous high-QPS writes (50k+ QPS), if the defragmentation queue (defrag_q) backs up, the defrag thread saturates NVMe I/O queues, causing unpredictable tail-latency spikes ($p99 &gt; 50\text{ms}$).

C. Real-World Case Studies

• Adobe: Had to meticulously tune defrag-lwm-pct and defrag-sleep to prevent defragmentation threads from starving local NVMe controllers under write-heavy workloads.
• Abnormal Security: Bypassed in-RAM indexing completely for large datasets by pivoting to RocksDB on raw SSDs, as keeping billions of keys in memory is still operationally expensive even under HMA.

[MushuEE]

RFC: Substrate Control Plane Latency Critical Path Analysis

Title: [RFC][Performance] Wakeup Latency Budget: Critical Path & Time-Loss Analysis

Hi team,

If we are going to meet our North Star target of a sub-100ms actor wakeup/activation latency, we need to audit the entire critical path of a ResumeActor request. We need to know exactly where every microsecond is spent, where the real "time losses" occur, and how our choice of database directly impacts this budget.

Here is my empirical audit of the wakeup critical path, allocating latency from the client's initial gRPC call down to gVisor sandbox restoration on physical workers.

---

Wakeup Latency Budget Timeline

Under a best-case scenario (warm cache, local SSD, and a pre-compiled gVisor golden snapshot), here is how our 100ms latency budget is partitioned:

gantt
    title Wakeup Latency Cascade (Scale: 1ms = 10 Days)
    dateFormat  YYYY-MM-DD
    section Control Plane
    1. Client Request RTT (+1.5ms)          :a1, 2026-01-01, 15d
    2. Load Actor Metadata (+0.5ms)         :a2, after a1, 5d
    3. Claim Idle Worker (SPOP) (+0.2ms)    :a3, after a2, 2d
    4. State Sync Write (RPO=0) (+3.0ms)    :crit, a4, after a3, 30d
    section Workload Restore
    5. gRPC Dial to Worker Pod (+1.0ms)     :a5, after a4, 10d
    6. Pull Snapshot Chunks (GCS) (+80.0ms) :active, a6, after a5, 800d
    7. gVisor VM Memory Restore (+8.0ms)   :active, a7, after a6, 80d
    section Finalization
    8. Write STATUS_RUNNING (+4.0ms)        :a8, after a7, 40d
    9. Client Response RTT (+1.8ms)         :a9, after a8, 18d

Loading

---

Latency Allocation & Time-Loss Breakdown

Phase 1: The Database & Control Plane Layer (Target: < 10ms)

This is the path handled by workflow_resume.go in the ateapi server.

Component	Operational Mechanism	Valkey (In-Memory)	Aerospike HMA	Cloud Bigtable	Critical Time-Loss / Risk
1. Request RTT	Client gRPC transport over TCP	$0.5 - 2\text{ms}$	$0.5 - 2\text{ms}$	$0.5 - 2\text{ms}$	TCP handshake/TLS termination overhead.
2. Load Metadata	Key-value fetch of Actor state	$<0.2\text{ms}$	$0.5 - 1.5\text{ms}$	$2.0 - 5.0\text{ms}$	Bigtable SSTable block index binary search and disk read cache misses.
3. Claim Worker	Selecting a random idle worker pod	$<0.1\text{ms}$ (SPOP)	$0.5 - 1.5\text{ms}$	$0.5 - 1.5\text{ms}$	LEGACY FAILURE: Listing all workers ($O(N)$ keyspace scan) takes $50 - 250\text{ms}$, immediately blowing the entire budget. Bypassed via $O(1)$ SPOP or prefix range scans.
4. Sync Write	Persisting worker assignment (RPO=0)	$<0.1\text{ms}$ (Async) $1 - 3\text{ms}$ (Fsync Always) $2 - 5\text{ms}$ (Sync replication)	$0.5 - 1.5\text{ms}$ (Block SSD write)	$2.0 - 5.0\text{ms}$ (Colossus Commit Log)	THE ACCELERATION WALL: Synchronous transactions (appendfsync always or multi-AZ Paxos consensus) inject physical block flash erase delays and network roundtrips into the thread loop, stalling the entire pipeline.

---

Critical Path Optimization Summary

To reliably hit our sub-100ms wake-up target, our optimization plan must focus on these three pillars:

1. Speed-up Scheduling (O(1)): Ensure worker claims use Valkey's atomic SPOP ($&lt;0.2\text{ms}$) or optimized Bigtable prefix range scans ($1\text{ms}$). Never list all workers.
2. Decouple Durability (Avoid synchronous locks): Write worker leases to Valkey asynchronously (async replication), removing network/disk-sync overhead from the wakeup path.
3. Localize Snapshots (Atelet Cache): Implement local snapshot caching directly in the memory/NVMe disk of worker pods. If the target golden snapshot is already cached locally on the worker node, snapshot loading latency drops from $80\text{ms}$ (GCS fetch) to $&lt;5\text{ms}$ (local NVMe load), guaranteeing we achieve our sub-100ms wakeup target!

[thockin]

I am spending some time learning this area, too. In all my time in Kubernetes-land I have had precious few opportunities to explore the options in this space. Brandon Jacobs (@brandonrjacobs) if you have expertise (or even just battle scars), let's hear them!

I think we need to set some ground rules and agree on principles and priorities.

Licensing:

1. Whatever we use must be Apache, MIT, or similarly licensed.
2. Anything AGPL is off the table. IME, no corporate lawyer will touch it.
3. Anything which is commercially licensed with "source available" or "free below X usage" is off the table.

Goals:

1. Upper-bound cardinality of O(1B) actors. Any solution must scale up to that.
2. API-operation latency in the low single-digit milliseconds. A lot of the user-experience is going to be dominated by network transfer time, so this is not the most critical aspect, but we can't be sloppy.
3. Throughput in the thousands of write-ops per second.

Principles:

1. Few users will be in the highest end of that scale, so we need a solution which scales down as well as up.
2. Next to meeting the performance goals, our [CRITICAL BUG] We are completely logoless! #1 priority is operability, including scaling up and failure recovery.
3. While a pluggable storage layer would be great, that is not a primary goal.

[thockin]

From my own exploration, ValKey still seems like a plausible answer, but there are a LOT of things we need to prove and configuration decisions we need to make. TiKV seems like another possible option, but I know far less about it.