feat(appview): PDS-Firehose konsumieren
Gegenstück zum subscribeRepos-Endpoint: WebSocket-Consumer mit persistiertem seq-Cursor, Reconnect-Backoff und Behandlung von #info/OutdatedCursor. Eigene Cursor-Tabelle statt einer Zeile in jetstream_cursor: dort steht ein time_us in der Größenordnung 1.7e15, die seq ist ein kleiner Zähler ab 1. Geteilt hätte GREATEST den PDS-Cursor sofort in eine Zukunft geschoben, die die PDS nie erreicht. Kein neuer Indexer-Pfad — jede Op wird in die Single-Op-Form übersetzt, die apply_commit schon vom Jetstream kennt. Push und Firehose liefern denselben Commit doppelt; das ist unkritisch, weil die Schreibpfade Upserts sind und der Dedupe-Index der Notifications den Rest abfängt. Mit einem Test festgehalten statt vorausgesetzt. Der CAR-Reader ist neu (es gab nur einen Writer, und der liegt in einem Binary-Crate ohne lib-Target). Der CBOR-Reader arbeitet mit explizitem Offset, weil ein Frame zwei hintereinander geschriebene Werte sind, und akzeptiert CID-Links in beiden Schreibweisen — die Blöcke tragen Strings. /healthz meldet beide Ströme getrennt; sie fallen unabhängig voneinander aus. Verifiziert mit totem Push-Ziel: der Post kam trotzdem an. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013HC9HLrUU1LNwkzp8nkDLX
This commit is contained in:
co-authored by
Claude Opus 5
parent
d6947c2576
commit
124a90dc07
@@ -0,0 +1,44 @@
|
||||
-- AppView database schema 0010: cursor for the local PDS firehose.
|
||||
--
|
||||
-- Why a second cursor table
|
||||
--
|
||||
-- The AppView now consumes two event streams, and they are numbered in
|
||||
-- completely different spaces:
|
||||
--
|
||||
-- * `jetstream_cursor.cursor` is a Jetstream `time_us` — microseconds
|
||||
-- since the epoch, produced by a public relay we do not control.
|
||||
-- * this table's `cursor` is the `seq` of our own PDS's
|
||||
-- `com.atproto.sync.subscribeRepos` — a small monotonic counter
|
||||
-- that starts at 1 in a fresh PDS database.
|
||||
--
|
||||
-- Sharing one row between them would mean the larger of the two values
|
||||
-- (always the Jetstream timestamp) permanently swallowing the other:
|
||||
-- `cursor_advance` uses GREATEST, so the very first Jetstream event
|
||||
-- would push the PDS cursor to ~1.7e15 and every subsequent
|
||||
-- subscribeRepos connect would ask for a sequence the PDS will never
|
||||
-- reach. Hence a table of its own, deliberately in the same shape as
|
||||
-- `jetstream_cursor` so both read/advance the same way.
|
||||
--
|
||||
-- Shape
|
||||
-- id pinned to 1 by a CHECK — a single-row table, the same
|
||||
-- pattern `jetstream_cursor` uses. It makes "advance the
|
||||
-- cursor" a plain UPDATE with no upsert dance and makes a
|
||||
-- second row impossible to create by accident.
|
||||
-- cursor the last `seq` we durably applied. 0 means "nothing
|
||||
-- yet": the consumer then subscribes without a `cursor`
|
||||
-- query parameter, which the PDS reads as "start from the
|
||||
-- current head" rather than replaying the entire repo
|
||||
-- history into a fresh index.
|
||||
-- updated_at observability only — how stale the stream is can be
|
||||
-- read straight off the row.
|
||||
--
|
||||
-- The row is inserted here so `cursor_advance`'s UPDATE always has a
|
||||
-- target; `pds_firehose::cursor_get` still tolerates a missing row and
|
||||
-- returns 0.
|
||||
|
||||
CREATE TABLE pds_firehose_cursor (
|
||||
id INTEGER PRIMARY KEY DEFAULT 1 CHECK (id = 1),
|
||||
cursor BIGINT NOT NULL DEFAULT 0,
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
INSERT INTO pds_firehose_cursor (id, cursor) VALUES (1, 0);
|
||||
Reference in New Issue
Block a user