Skip to content

Read from vaxine stream, send transactions to slot server#4

Merged
v0idpwn merged 2 commits into
mainfrom
v0idpwn/vax-101-apply-operations-from-antidote-log-into
Jul 14, 2022
Merged

Read from vaxine stream, send transactions to slot server#4
v0idpwn merged 2 commits into
mainfrom
v0idpwn/vax-101-apply-operations-from-antidote-log-into

Conversation

@v0idpwn
Copy link
Copy Markdown
Contributor

@v0idpwn v0idpwn commented Jul 13, 2022

No description provided.

@linear
Copy link
Copy Markdown

linear Bot commented Jul 13, 2022

VAX-101 Apply operations from Antidote log into Electric

Antidote log tailer exposes an interface for retrieving operations from log. Feed those operations into Electric Pg server and ensure they are pushed to Logical Repl. subscribers.

(Please document here for reference if interface with Antidote is push or pull)

Comment thread mix.lock Outdated
%{
"antidote_pb_codec": {:git, "git@github.com:vaxine-io/vaxine.git", "8b0b285113e30f556b9a18511946e6d5d1d582dc", [sparse: "apps/antidote_pb_codec"]},
"antidotec_pb": {:git, "git@github.com:vaxine-io/vaxine.git", "8b0b285113e30f556b9a18511946e6d5d1d582dc", [sparse: "apps/antidotec_pb"]},
"antidote_pb_codec": {:git, "git@github.com:vaxine-io/vaxine.git", "f9935eb57cf603cd4937654cc2e015b673577961", [branch: "defnull/wal-traversal", sparse: "apps/antidote_pb_codec"]},
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You may use main, or concrete commit here. wal-traversal branch has been merged

@v0idpwn v0idpwn marked this pull request as ready for review July 14, 2022 17:17
@v0idpwn v0idpwn changed the title wip: vx_client read stream -> postgres_tcp_server Read from vaxine stream, send transactions to slot server Jul 14, 2022
@v0idpwn v0idpwn force-pushed the v0idpwn/vax-101-apply-operations-from-antidote-log-into branch 2 times, most recently from c999779 to b2d87be Compare July 14, 2022 17:22
@v0idpwn v0idpwn force-pushed the v0idpwn/vax-101-apply-operations-from-antidote-log-into branch from b2d87be to 0e0feae Compare July 14, 2022 17:27
@v0idpwn v0idpwn merged commit 773dfcc into main Jul 14, 2022
@v0idpwn v0idpwn deleted the v0idpwn/vax-101-apply-operations-from-antidote-log-into branch July 14, 2022 17:32
icehaunter pushed a commit that referenced this pull request Apr 17, 2023
Moved in proto file and Satellite client
KyleAMathews pushed a commit that referenced this pull request Nov 1, 2024
KyleAMathews added a commit that referenced this pull request Oct 13, 2025
Based on thorough validation against PostgreSQL documentation:

**Issue #1 - Troubleshooting "must be owner" error:**
- REMOVED incorrect suggestion to use GRANT ALL PRIVILEGES
- PostgreSQL ownership rights cannot be granted via privileges
- ALTER TABLE and adding tables to publications require actual ownership

**Issue #2 - Quick Start ownership transfer:**
- ADDED GRANT CREATE ON SCHEMA public before ownership transfer
- PostgreSQL requires new owner to have CREATE privilege on schema
- Without this, ALTER TABLE ... OWNER TO will fail

**Issue #3 - Publication ownership requirements:**
- CLARIFIED that you must own BOTH publication AND each table
- Updated Core Permission Requirements table
- Per PostgreSQL docs: Adding a table additionally requires owning that table

**Issue #4 - AWS wal_level defaults:**
- CORRECTED incorrect claim about wal_level defaults
- PostgreSQL default is replica (not minimal for RDS)
- RDS/Aurora use standard PostgreSQL defaults

All fixes validated by research agents against official PostgreSQL docs.

Thanks to external reviewer for catching these critical errors.
KyleAMathews added a commit that referenced this pull request Apr 4, 2026
…ky SSE test

Add new static analysis rule detecting #publish/#onMessages calls inside
catch blocks or HTTP error status handlers — catches the Bug #4 pattern
(publishing stale 409 data to subscribers). RED/GREEN verified.

Also: update SPEC.md loop-back site line numbers, fix 409 handler comment,
DRY improvements to model-based tests, fix flaky SSE fallback test
(guard controller.close() against already-closed stream, widen SSE
request count tolerance).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
KyleAMathews added a commit that referenced this pull request Apr 10, 2026
Adds test/pbt-micro.test.ts — a dedicated PBT suite that exercises
twelve narrow invariants in the ShapeStream client. Each TARGET's
opening comment documents the invariant under test, and the PBTs
shook loose eight real bugs that are fixed in this commit:

  #1 canonicalShapeKey used URLSearchParams.set() for custom params,
     collapsing duplicate keys (e.g. ?tag=a&tag=b → ?tag=b) so two
     genuinely distinct shapes shared a cache key. Switched to append().

  #2 Shape#process clobbered shouldNotify by assignment in three
     places, so any sequence where a change message followed a
     status-change message would silently drop the change's
     notification. OR-accumulate instead, and track hadData before
     must-refetch clears state so subscribers still see the reset.

  #3 SubsetParams GET serialization dropped limit=0 and offset=0
     via falsy checks. Switched to explicit !== undefined guards.

  #4 Shape#requestedSubSnapshots dedup keyed on bigintSafeStringify,
     which preserves insertion order, so permutation-equivalent
     params produced different keys and re-execution fired the
     same snapshot N times. Added canonicalBigintSafeStringify to
     helpers.ts that recursively sorts object keys.

  #5 snakeToCamel collapsed runs of underscores into a single
     camelCase boundary, so user_id and user__id (distinct db
     columns) decoded to the same app key, corrupting rows with
     mapped values. snakeToCamel now preserves (n-1) literal
     underscores for a run of n, and camelToSnake's boundary
     regex was widened to ([a-z_])([A-Z]) so the round-trip is
     injective.

  #6 Shape#reexecuteSnapshots caught and discarded errors from
     stream.requestSnapshot, silently dropping failed sub-snapshot
     re-execution on shape rotation. Errors are now collected and
     the first one is surfaced via #error + #notify.

  #7 SnapshotTracker populated xmaxSnapshots and
     snapshotsByDatabaseLsn in addSnapshot but never cleaned them
     up — neither on removeSnapshot nor on addSnapshot with a
     repeated mark. A later shouldRejectMessage eviction loop
     would walk the stale reverse index and wrongly delete the
     current snapshot, allowing duplicate change messages to slip
     through. Stored databaseLsn on each entry and added
     #detachFromReverseIndexes that runs on both add (before
     overwriting) and remove.

  #8 Shape#awaitUpToDate never observed the stream's error state,
     so calling requestSnapshot on a terminally-errored stream
     would hang forever on the setInterval polling loop. The
     helper now checks #error / stream.error up front, subscribes
     to the stream's onError, and settles the internal promise
     via reject on any terminal error path.

Also:
  - vitest.pbt.config.ts — dedicated config that skips the
    real-Electric globalSetup and includes both model-based
    and pbt-micro test files.
  - bin/pbt-soak.sh — soak runner that loops PBT iterations
    with fresh seeds, captures counterexamples on failure.
  - test/pbt-micro.test.ts TARGET 4 (UpToDateTracker) restores
    real timers in afterEach so TARGET 12's setInterval doesn't
    hang when the suite runs end-to-end.
  - SPEC.md cross-references the L6 fetchSnapshotWithRetry PBT
    from the unconditional-409-cache-buster invariant.
  - model-based.test.ts gains response builders for update,
    delete, and mixed-batch 200s with lsn/op_position/txids
    headers for more realistic change sequences.

Verified with 319 unit tests, 44 PBT tests at 500 runs each
(and 2000-run soak), 62 column-mapper/snapshot-tracker tests,
typecheck clean, eslint clean, static-analysis tests clean.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
balegas added a commit that referenced this pull request May 3, 2026
…mpotent close

Phase 2 Task 5 from R1 #4.

execWithStdinViaPost's writeStdin and closeStdin had two defects:
- writeStdin appended to stdinBuf with no guard. After closeStdin
  fired the actual POST, any further write was silently buffered
  and lost (the data was never sent — the request had already
  flushed).
- closeStdin re-fired `void start()` on every call. start() itself
  was idempotent via its `started` flag, but the contract is
  cleaner if closeStdin is itself a no-op the second time.

Fix: a `closed` flag. writeStdin throws if called after close;
closeStdin is a no-op the second time.

Defence-in-depth: StdioBridge calls writeStdin then closeStdin
sequentially per turn, so the new throw can't fire under normal
usage. The guard catches future caller misuse loudly instead of
silently swallowing data.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
balegas added a commit that referenced this pull request May 8, 2026
#1 — `mcp.json` is now explicit opt-in. New `loadProjectMcpConfig`
option (boolean | string) on `BuiltinAgentsServer`; defaults to off
because stdio servers spawn local commands. The Electron desktop and
the `electric-ax` CLI opt in; library embedders get the safe default.

#2 — `stop()` tears down MCP resources. Added `Registry.close()`
(closes every transport, forgets auth state, emits a final empty
snapshot). `BuiltinAgentsServer.stop()` now also disposes the
`mcp.json` chokidar watcher and unregisters the `mcp` tool provider.

#3 — `RuntimeRegistry.register()` accumulates types per runtime
instead of last-write-wins, fixing `/api/runtimes` losing earlier
types when entity-type registration POSTs land in parallel.

#4 — `applyMerged` is async/await so `mcpRegistry.applyConfig`
rejections actually reach the catch (previously voided inside a
`.then`, escaping as unhandled rejections). Optional
`onConfigError` callback exposed for embedders.

#5 — `composeToolsWithProviders` warns when a named MCP server in
`mcp.tools(['x'])` is unavailable (unknown or not yet ready). Wildcard
sentinels stay silent; missing names dedupe within a single call.

#6a — `hashConfig()` includes `timeoutMs` so timeout-only edits no
longer skip the reconfigure path and leave the entry's stale.

#6b — `mcp.tools()` (no arg) is the canonical "every registered
server" form. `mcp.tools('*')` kept for back-compat. Built-ins
`horton`/`worker` and the docs use the no-arg form.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants