feat(appview): PDS-Firehose konsumieren
Gegenstück zum subscribeRepos-Endpoint: WebSocket-Consumer mit persistiertem seq-Cursor, Reconnect-Backoff und Behandlung von #info/OutdatedCursor. Eigene Cursor-Tabelle statt einer Zeile in jetstream_cursor: dort steht ein time_us in der Größenordnung 1.7e15, die seq ist ein kleiner Zähler ab 1. Geteilt hätte GREATEST den PDS-Cursor sofort in eine Zukunft geschoben, die die PDS nie erreicht. Kein neuer Indexer-Pfad — jede Op wird in die Single-Op-Form übersetzt, die apply_commit schon vom Jetstream kennt. Push und Firehose liefern denselben Commit doppelt; das ist unkritisch, weil die Schreibpfade Upserts sind und der Dedupe-Index der Notifications den Rest abfängt. Mit einem Test festgehalten statt vorausgesetzt. Der CAR-Reader ist neu (es gab nur einen Writer, und der liegt in einem Binary-Crate ohne lib-Target). Der CBOR-Reader arbeitet mit explizitem Offset, weil ein Frame zwei hintereinander geschriebene Werte sind, und akzeptiert CID-Links in beiden Schreibweisen — die Blöcke tragen Strings. /healthz meldet beide Ströme getrennt; sie fallen unabhängig voneinander aus. Verifiziert mit totem Push-Ziel: der Post kam trotzdem an. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013HC9HLrUU1LNwkzp8nkDLX
This commit is contained in:
co-authored by
Claude Opus 5
parent
d6947c2576
commit
124a90dc07
@@ -16,7 +16,7 @@ use std::sync::atomic::{AtomicBool, AtomicI64, AtomicU64, Ordering};
|
||||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
use tokio::sync::mpsc;
|
||||
use tracing::{debug, info, trace, warn};
|
||||
use tracing::{debug, info, warn};
|
||||
|
||||
use crate::indexer;
|
||||
|
||||
@@ -31,6 +31,18 @@ pub struct Stats {
|
||||
/// writes this; `/healthz` reads it. Wrapped in `Arc` so the consumer
|
||||
/// can hold its own clone without borrowing from us.
|
||||
pub jetstream_connected: Arc<AtomicBool>,
|
||||
/// Whether the **local PDS** firehose WebSocket is currently up
|
||||
/// ([`crate::pds_firehose`]). Separate from `jetstream_connected`
|
||||
/// because the two streams fail independently and for different
|
||||
/// reasons: a dead Jetstream means no view of the wider network, a
|
||||
/// dead PDS firehose means the AppView has lost the guaranteed
|
||||
/// delivery path for its *own* users' records and is running on the
|
||||
/// best-effort push alone. `/healthz` has to be able to say which.
|
||||
pub pds_connected: AtomicBool,
|
||||
/// Number of `#commit` frames applied from the PDS firehose.
|
||||
pub pds_frames_processed: AtomicU64,
|
||||
/// Highest `seq` applied from the PDS firehose in this process.
|
||||
pub pds_last_seq: AtomicI64,
|
||||
}
|
||||
|
||||
impl Default for Stats {
|
||||
@@ -40,6 +52,9 @@ impl Default for Stats {
|
||||
last_event_time_us: AtomicI64::new(0),
|
||||
last_cursor_persisted_us: AtomicI64::new(0),
|
||||
jetstream_connected: Arc::new(AtomicBool::new(false)),
|
||||
pds_connected: AtomicBool::new(false),
|
||||
pds_frames_processed: AtomicU64::new(0),
|
||||
pds_last_seq: AtomicI64::new(0),
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -50,6 +65,9 @@ impl std::fmt::Debug for Stats {
|
||||
.field("events_processed", &self.events_processed())
|
||||
.field("last_event_time_us", &self.last_event_time_us.load(Ordering::Relaxed))
|
||||
.field("jetstream_connected", &self.jetstream_connected())
|
||||
.field("pds_connected", &self.pds_connected())
|
||||
.field("pds_frames_processed", &self.pds_frames_processed())
|
||||
.field("pds_last_seq", &self.pds_last_seq())
|
||||
.finish()
|
||||
}
|
||||
}
|
||||
@@ -85,6 +103,23 @@ impl Stats {
|
||||
pub fn jetstream_connected(&self) -> bool {
|
||||
self.jetstream_connected.load(Ordering::Relaxed)
|
||||
}
|
||||
|
||||
/// Is the local PDS firehose connected right now?
|
||||
pub fn pds_connected(&self) -> bool {
|
||||
self.pds_connected.load(Ordering::Relaxed)
|
||||
}
|
||||
|
||||
pub fn pds_frames_processed(&self) -> u64 {
|
||||
self.pds_frames_processed.load(Ordering::Relaxed)
|
||||
}
|
||||
|
||||
/// Highest PDS-firehose `seq` this process has applied. 0 before the
|
||||
/// first frame — note this is the *in-process* high-water mark, not
|
||||
/// the persisted cursor, which lives in `pds_firehose_cursor` and
|
||||
/// survives restarts.
|
||||
pub fn pds_last_seq(&self) -> i64 {
|
||||
self.pds_last_seq.load(Ordering::Relaxed)
|
||||
}
|
||||
}
|
||||
|
||||
/// The thing the Jetstream consumer calls once per event.
|
||||
|
||||
Reference in New Issue
Block a user