feat(appview): PDS-Firehose konsumieren

Gegenstück zum subscribeRepos-Endpoint: WebSocket-Consumer mit
persistiertem seq-Cursor, Reconnect-Backoff und Behandlung von
#info/OutdatedCursor.

Eigene Cursor-Tabelle statt einer Zeile in jetstream_cursor: dort steht ein
time_us in der Größenordnung 1.7e15, die seq ist ein kleiner Zähler ab 1.
Geteilt hätte GREATEST den PDS-Cursor sofort in eine Zukunft geschoben, die
die PDS nie erreicht.

Kein neuer Indexer-Pfad — jede Op wird in die Single-Op-Form übersetzt, die
apply_commit schon vom Jetstream kennt. Push und Firehose liefern denselben
Commit doppelt; das ist unkritisch, weil die Schreibpfade Upserts sind und
der Dedupe-Index der Notifications den Rest abfängt. Mit einem Test
festgehalten statt vorausgesetzt.

Der CAR-Reader ist neu (es gab nur einen Writer, und der liegt in einem
Binary-Crate ohne lib-Target). Der CBOR-Reader arbeitet mit explizitem
Offset, weil ein Frame zwei hintereinander geschriebene Werte sind, und
akzeptiert CID-Links in beiden Schreibweisen — die Blöcke tragen Strings.

/healthz meldet beide Ströme getrennt; sie fallen unabhängig voneinander
aus.

Verifiziert mit totem Push-Ziel: der Post kam trotzdem an.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013HC9HLrUU1LNwkzp8nkDLX
This commit is contained in:
tomdebone
2026-09-10 07:08:23 +02:00
co-authored by Claude Opus 5
parent d6947c2576
commit 124a90dc07
11 changed files with 2849 additions and 1 deletions
+36 -1
View File
@@ -16,7 +16,7 @@ use std::sync::atomic::{AtomicBool, AtomicI64, AtomicU64, Ordering};
use std::sync::Arc;
use std::time::Duration;
use tokio::sync::mpsc;
use tracing::{debug, info, trace, warn};
use tracing::{debug, info, warn};
use crate::indexer;
@@ -31,6 +31,18 @@ pub struct Stats {
/// writes this; `/healthz` reads it. Wrapped in `Arc` so the consumer
/// can hold its own clone without borrowing from us.
pub jetstream_connected: Arc<AtomicBool>,
/// Whether the **local PDS** firehose WebSocket is currently up
/// ([`crate::pds_firehose`]). Separate from `jetstream_connected`
/// because the two streams fail independently and for different
/// reasons: a dead Jetstream means no view of the wider network, a
/// dead PDS firehose means the AppView has lost the guaranteed
/// delivery path for its *own* users' records and is running on the
/// best-effort push alone. `/healthz` has to be able to say which.
pub pds_connected: AtomicBool,
/// Number of `#commit` frames applied from the PDS firehose.
pub pds_frames_processed: AtomicU64,
/// Highest `seq` applied from the PDS firehose in this process.
pub pds_last_seq: AtomicI64,
}
impl Default for Stats {
@@ -40,6 +52,9 @@ impl Default for Stats {
last_event_time_us: AtomicI64::new(0),
last_cursor_persisted_us: AtomicI64::new(0),
jetstream_connected: Arc::new(AtomicBool::new(false)),
pds_connected: AtomicBool::new(false),
pds_frames_processed: AtomicU64::new(0),
pds_last_seq: AtomicI64::new(0),
}
}
}
@@ -50,6 +65,9 @@ impl std::fmt::Debug for Stats {
.field("events_processed", &self.events_processed())
.field("last_event_time_us", &self.last_event_time_us.load(Ordering::Relaxed))
.field("jetstream_connected", &self.jetstream_connected())
.field("pds_connected", &self.pds_connected())
.field("pds_frames_processed", &self.pds_frames_processed())
.field("pds_last_seq", &self.pds_last_seq())
.finish()
}
}
@@ -85,6 +103,23 @@ impl Stats {
pub fn jetstream_connected(&self) -> bool {
self.jetstream_connected.load(Ordering::Relaxed)
}
/// Is the local PDS firehose connected right now?
pub fn pds_connected(&self) -> bool {
self.pds_connected.load(Ordering::Relaxed)
}
pub fn pds_frames_processed(&self) -> u64 {
self.pds_frames_processed.load(Ordering::Relaxed)
}
/// Highest PDS-firehose `seq` this process has applied. 0 before the
/// first frame — note this is the *in-process* high-water mark, not
/// the persisted cursor, which lives in `pds_firehose_cursor` and
/// survives restarts.
pub fn pds_last_seq(&self) -> i64 {
self.pds_last_seq.load(Ordering::Relaxed)
}
}
/// The thing the Jetstream consumer calls once per event.