docs: Firehose, Follow-Lexicon und Phase 10

architecture.md zeigt den subscribeRepos-Pfad im Diagramm (er stand zuerst
irrtümlich am Pfeil zum externen Relay) und erklärt, warum es Push *und*
Firehose gibt: der eine ist schnell, der andere verlässlich. Dazu, wo die
Spec-Treue endet — die Frame-Hülle ist konform, die Blöcke darin nicht.

deployment.md bekommt einen Firehose-Abschnitt: Transaktionsgarantie, warum
die seq lückenfrei ist und was der globale Advisory-Lock an Durchsatz
kostet, Cursor-Semantik, WebSocket-Upgrade im Proxy, und die fehlende
Retention für firehose_events.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013HC9HLrUU1LNwkzp8nkDLX

Cargo.lock

@@ -51,6 +51,8 @@ dependencies = [
  "axum",
  "base64",
  "chrono",
+ "ciborium",
+ "cid",
  "dotenvy",
  "futures",
  "p256",
@@ -60,6 +62,7 @@ dependencies = [
  "serde_json",
  "sqlx",
  "tokio",
+ "tokio-tungstenite",
  "tower",
  "tower-http",
  "tracing",
@@ -287,6 +290,7 @@ checksum = "edca88bc138befd0323b20752846e6587272d3b03b0343c8ea28a6f819e6e71f"
 dependencies = [
  "async-trait",
  "axum-core",
+ "base64",
  "bytes",
  "futures-util",
  "http",
@@ -305,8 +309,10 @@ dependencies = [
  "serde_json",
  "serde_path_to_error",
  "serde_urlencoded",
+ "sha1",
  "sync_wrapper",
  "tokio",
+ "tokio-tungstenite",
  "tower",
  "tower-layer",
  "tower-service",
@@ -1885,6 +1891,7 @@ dependencies = [
  "ciborium",
  "cid",
  "dotenvy",
+ "futures",
  "hex",
  "k256",
  "p256",
@@ -1895,6 +1902,7 @@ dependencies = [
  "sha2",
  "sqlx",
  "tokio",
+ "tokio-tungstenite",
  "tower",
  "tower-http",
  "tracing",

README.md

@@ -25,6 +25,7 @@ crates/tauri-app/       Tauri 2 + Svelte 5 + Vite + TS Desktop-Client
   └── src-tauri/        Rust-IPC-Layer
 
 lexicons/app/twi/post.json     Custom Lexicon mit maxLength: 160
+lexicons/app/bsky/            like, repost, follow, actor.profile
 migrations/pds/                PDS-DB-Schema (users, repos, blobs, sessions, plc_ops)
 migrations/appview/            AppView-DB-Schema (posts, likes, follows, notifications, profiles, jetstream_cursor)
 docs/                          Deployment, Architektur, Tauri-Release (siehe unten)
@@ -69,6 +70,7 @@ cargo run -p appview
 | 7  Polish (Tray, Notifications, Auto-Update) | ✅ done — Tray-Icon custom (`tauri::include_image!`), Notification-Click navigiert via `app://notification`-Event + `openThread`-Helper zu Thread-Detail, Auto-Update in Dev inert (Production-Weg: [`docs/tauri-release.md`](docs/tauri-release.md)) |
 | 8  Social-Graph + Benachrichtigungen | ✅ done — `notifications`-Tabelle, Schreibpfad im Jetstream-Indexer (idempotent, keine Selbst-Notifications), `/api/notifications[/count|/seen]`, `/api/followers`, `/api/following`, eigene `/api/thread`-Route; im Client Notifications-View mit Unread-Badge und klickbare Follower-/Following-Listen im Profil |
 | 9  Auth + Performance | ✅ done — AppView prüft Bearer-Tokens (ES256, Schlüssel aus dem neuen `/.well-known/did.json` der PDS, fail closed); Timeline und Notifications nur noch für die eigene DID; CORS-Allowlist statt `Any`; Indizes für Handle-Lookup und Cold-Start-Feed |
+| 10  Lokaler Firehose | ✅ done — `com.atproto.sync.subscribeRepos` auf der PDS (Event in derselben Transaktion wie der Commit, `seq`-Cursor mit lückenfreiem Replay, WebSocket-Frames in atproto-Form); die AppView konsumiert ihn mit persistiertem Cursor. Ein verlorener Push ist damit nicht mehr endgültig. |
 
 ## Tests
 
@@ -103,11 +105,14 @@ Root-Workspace; `cargo test --workspace` von oben erfasst den IPC-Layer nicht.
 
 ## Bekannte Lücken
 
-* Die eigene PDS speist **keinen** Firehose (`com.atproto.sync.subscribeRepos` fehlt) —
-  eigene Records erreichen die AppView nur über den Best-Effort-Push
-  `POST /internal/ingest-commit`.
-* `aud` wird beim Token-Check nicht validiert (Signatur, Ablauf, `scope` und
-  `sub` schon).
+* Der Firehose ist **lokal**: er verbindet die eigenen zwei Dienste. Ein fremder
+  Relay erfährt von dieser PDS weiterhin nichts.
+* Die Frame-Hülle ist spec-konformes DAG-CBOR, die Blöcke darin nicht: CIDs
+  innerhalb von Commit-Blöcken sind Strings statt Tag-42-Links. Ein fremder
+  atproto-Consumer liest die Frames, scheitert aber an den Blockinhalten. Das
+  zu ändern hieße, jede CID im System zu ändern — inklusive der
+  `did:plc:`-Ableitung.
+* `firehose_events` wird nie beschnitten.
 * Notifications werden nie gelöscht: Unlike/Unfollow lässt die Zeile stehen, und der
   Dedupe-Key macht sie „einmal pro (Empfänger, Autor, Art, Subject) für immer".
 * Auto-Update ist nur dokumentiert, nicht verdrahtet: niemand ruft `check()` auf, das

crates/appview/Cargo.toml

@@ -40,6 +40,15 @@ uuid = { workspace = true }
 rustls = { version = "0.23", default-features = false, features = ["ring", "std", "logging", "tls12"] }
 base64 = { workspace = true }
 futures = { workspace = true }
+# The local PDS firehose (`src/pds_firehose.rs`): a WebSocket carrying
+# DAG-CBOR frames whose `blocks` field is a CAR of record blocks.
+# `tokio-tungstenite` for the socket, `cid` for the block addresses,
+# `ciborium` for the record blocks themselves (they are written with
+# `ciborium::into_writer` on the PDS side, so it is their exact inverse).
+# The frame envelope is decoded by `src/cbor.rs`, which needs no crate.
+tokio-tungstenite = { workspace = true }
+cid = { workspace = true }
+ciborium = { workspace = true }
 
 [dev-dependencies]
 tokio = { workspace = true }

crates/appview/src/auth.rs

@@ -157,12 +157,16 @@ pub struct PdsKeys {
     http: reqwest::Client,
     /// Fully-qualified URL of the PDS's DID document.
     did_doc_url: String,
+    /// The `aud` every access token must carry: this AppView's own
+    /// service DID. See [`verify_with_key`] for why it's checked.
+    expected_aud: String,
     inner: RwLock<CachedKey>,
 }
 
 impl PdsKeys {
-    /// Build a cache pointed at `base_url` (no trailing slash required).
-    pub fn new(base_url: &str) -> Self {
+    /// Build a cache pointed at `base_url` (no trailing slash required),
+    /// accepting only tokens addressed to `expected_aud`.
+    pub fn new(base_url: &str, expected_aud: impl Into<String>) -> Self {
         let http = reqwest::Client::builder()
             .timeout(DID_DOC_TIMEOUT)
             .build()
@@ -173,6 +177,7 @@ impl PdsKeys {
                 "{}/.well-known/did.json",
                 base_url.trim_end_matches('/')
             ),
+            expected_aud: expected_aud.into(),
             inner: RwLock::new(CachedKey::default()),
         }
     }
@@ -180,7 +185,7 @@ impl PdsKeys {
     /// Same PDS the handle-sync worker talks to: `PDS_INTERNAL_URL`
     /// when set, else `PDS_PUBLIC_URL`.
     pub fn from_config(cfg: &at_shared::config::AppConfig) -> Self {
-        Self::new(&cfg.pds_base_url())
+        Self::new(&cfg.pds_base_url(), cfg.appview_did())
     }
 
     pub fn did_doc_url(&self) -> &str {
@@ -267,13 +272,13 @@ impl PdsKeys {
     /// restart.
     pub async fn verify_access_token(&self, token: &str) -> Result<JwtClaims, AuthError> {
         let key = self.key_or_fetch().await?;
-        match verify_with_key(token, &key) {
+        match verify_with_key(token, &key, &self.expected_aud) {
             Ok(claims) => Ok(claims),
             Err(first) => {
                 let Some(fresh) = self.refetch_if_stale(&key).await else {
                     return Err(first);
                 };
-                verify_with_key(token, &fresh).map_err(|_| first)
+                verify_with_key(token, &fresh, &self.expected_aud).map_err(|_| first)
             }
         }
     }
@@ -309,16 +314,44 @@ fn extract_public_key_multibase(doc: &Value) -> anyhow::Result<String> {
 /// leeway for clock skew); the scope check is ours, and it is the line
 /// that keeps a 90-day refresh token from working as a session
 /// credential.
-fn verify_with_key(token: &str, pubkey_multibase: &str) -> Result<JwtClaims, AuthError> {
+fn verify_with_key(
+    token: &str,
+    pubkey_multibase: &str,
+    expected_aud: &str,
+) -> Result<JwtClaims, AuthError> {
     let claims = at_crypto::jwt::verify_jwt(token, pubkey_multibase)
         .map_err(|e| AuthError::Invalid(format!("invalid token: {e}")))?;
     match claims.scope.as_deref() {
-        Some(ACCESS_SCOPE) => Ok(claims),
-        other => Err(AuthError::Invalid(format!(
-            "token scope {:?} is not {ACCESS_SCOPE}",
-            other.unwrap_or("<none>")
-        ))),
+        Some(ACCESS_SCOPE) => {}
+        other => {
+            return Err(AuthError::Invalid(format!(
+                "token scope {:?} is not {ACCESS_SCOPE}",
+                other.unwrap_or("<none>")
+            )))
+        }
     }
+    // Audience. `at_crypto::jwt::verify_jwt` sets `validate_aud = false`
+    // because it has no way of knowing who the caller is, so the check
+    // belongs here.
+    //
+    // What it buys: the PDS signs tokens for *its* AppView. Without an
+    // audience check, a token handed to any other service that trusts
+    // the same PDS key would be replayable here — and, the other way
+    // round, a token this AppView issued trust in could be replayed
+    // there. It is the difference between "the PDS vouches for this
+    // user" and "the PDS vouches for this user *talking to us*".
+    //
+    // A mismatch is `TokenInvalid` rather than `Forbidden` on purpose:
+    // that is the code the desktop client refreshes on, so a
+    // deployment that changes `APPVIEW_PUBLIC_URL` heals itself on the
+    // next refresh instead of stranding every signed-in user.
+    if claims.aud != expected_aud {
+        return Err(AuthError::Invalid(format!(
+            "token audience {:?} is not {expected_aud:?}",
+            claims.aud
+        )));
+    }
+    Ok(claims)
 }
 
 /// Extract the bearer token from an `Authorization` header.
@@ -466,14 +499,29 @@ mod tests {
         (kp, multibase)
     }
 
+    /// The audience the tests' AppView identifies as — what
+    /// `AppConfig::appview_did()` would return for
+    /// `APPVIEW_PUBLIC_URL=http://127.0.0.1:2584`.
+    const TEST_AUD: &str = "did:web:127.0.0.1%3A2584";
+
     fn mint(kp: &P256Keypair, did: &str, scope: &str, ttl_secs: i64) -> String {
+        mint_for(kp, did, scope, ttl_secs, TEST_AUD)
+    }
+
+    fn mint_for(
+        kp: &P256Keypair,
+        did: &str,
+        scope: &str,
+        ttl_secs: i64,
+        aud: &str,
+    ) -> String {
         let now = chrono::Utc::now().timestamp();
         issue_jwt(
             kp,
             &JwtClaims {
                 iss: "did:web:127.0.0.1%3A2583".into(),
                 sub: did.into(),
-                aud: "did:web:appview.maarcadetweet.local".into(),
+                aud: aud.into(),
                 iat: now - 1,
                 exp: now + ttl_secs,
                 jti: None,
@@ -511,6 +559,34 @@ mod tests {
         assert_eq!(bearer_token(&header_map("Bearer tok")).unwrap(), "tok");
     }
 
+    /// A token minted for a different AppView must not work here, and
+    /// must fail as `TokenInvalid` so the client refreshes rather than
+    /// treating it as a permanent rejection.
+    #[test]
+    fn token_for_another_audience_is_rejected() {
+        let (kp, mb) = test_key();
+        let token = mint_for(
+            &kp,
+            "did:plc:alice",
+            ACCESS_SCOPE,
+            3600,
+            "did:web:someone-elses-appview.example",
+        );
+        let err = verify_with_key(&token, &mb, TEST_AUD).unwrap_err();
+        assert!(
+            matches!(err, AuthError::Invalid(ref m) if m.contains("audience")),
+            "expected an audience rejection, got {err:?}"
+        );
+        let (status, body) = err.into_response_parts();
+        assert_eq!(status, StatusCode::UNAUTHORIZED);
+        assert_eq!(body.0["error"], "TokenInvalid");
+
+        // The same token *is* fine for the AppView it was minted for.
+        assert!(
+            verify_with_key(&token, &mb, "did:web:someone-elses-appview.example").is_ok()
+        );
+    }
+
     #[test]
     fn error_bodies_carry_the_documented_codes() {
         // These strings are a contract: the desktop client keys its
@@ -535,14 +611,14 @@ mod tests {
     fn valid_access_token_verifies() {
         let (kp, mb) = test_key();
         let token = mint(&kp, "did:plc:alice", ACCESS_SCOPE, 3600);
-        let claims = verify_with_key(&token, &mb).unwrap();
+        let claims = verify_with_key(&token, &mb, TEST_AUD).unwrap();
         assert_eq!(claims.sub, "did:plc:alice");
     }
 
     #[test]
     fn garbage_token_is_invalid() {
         let (_, mb) = test_key();
-        let err = verify_with_key("not-a-jwt", &mb).unwrap_err();
+        let err = verify_with_key("not-a-jwt", &mb, TEST_AUD).unwrap_err();
         assert!(matches!(err, AuthError::Invalid(_)));
     }
 
@@ -552,7 +628,7 @@ mod tests {
         let (_, other_mb) = test_key();
         let token = mint(&kp, "did:plc:alice", ACCESS_SCOPE, 3600);
         assert!(matches!(
-            verify_with_key(&token, &other_mb).unwrap_err(),
+            verify_with_key(&token, &other_mb, TEST_AUD).unwrap_err(),
             AuthError::Invalid(_)
         ));
     }
@@ -563,7 +639,7 @@ mod tests {
         // days. Without the scope check it would be a session token.
         let (kp, mb) = test_key();
         let token = mint(&kp, "did:plc:alice", "com.atproto.refresh", 3600);
-        let err = verify_with_key(&token, &mb).unwrap_err();
+        let err = verify_with_key(&token, &mb, TEST_AUD).unwrap_err();
         match err {
             AuthError::Invalid(msg) => assert!(msg.contains("com.atproto.refresh")),
             other => panic!("expected Invalid, got {other:?}"),
@@ -576,7 +652,7 @@ mod tests {
         let (kp, mb) = test_key();
         let token = mint(&kp, "did:plc:alice", ACCESS_SCOPE, -120);
         assert!(matches!(
-            verify_with_key(&token, &mb).unwrap_err(),
+            verify_with_key(&token, &mb, TEST_AUD).unwrap_err(),
             AuthError::Invalid(_)
         ));
     }
@@ -631,13 +707,13 @@ mod tests {
     #[test]
     fn did_doc_url_is_built_from_the_base_url() {
         assert_eq!(
-            PdsKeys::new("http://127.0.0.1:2583").did_doc_url(),
+            PdsKeys::new("http://127.0.0.1:2583", TEST_AUD).did_doc_url(),
             "http://127.0.0.1:2583/.well-known/did.json"
         );
         // A trailing slash must not produce a double slash — some
         // servers 404 on it.
         assert_eq!(
-            PdsKeys::new("http://pds:3000/").did_doc_url(),
+            PdsKeys::new("http://pds:3000/", TEST_AUD).did_doc_url(),
             "http://pds:3000/.well-known/did.json"
         );
     }
@@ -646,7 +722,7 @@ mod tests {
     async fn verification_fails_closed_when_the_pds_is_unreachable() {
         // Port 1 on loopback: nothing listens there, so the fetch fails
         // fast. The result must be a 503, never a pass-through.
-        let keys = PdsKeys::new("http://127.0.0.1:1");
+        let keys = PdsKeys::new("http://127.0.0.1:1", TEST_AUD);
         let err = keys.verify_access_token("whatever").await.unwrap_err();
         assert!(
             matches!(err, AuthError::Unavailable(_)),

crates/appview/src/car.rs

@@ -0,0 +1,394 @@
+//! CAR v1 *reader*.
+//!
+//! The repo has a writer (`pds-server/src/car.rs`) but no reader the
+//! AppView could use: `pds-server` is a binary crate with no library
+//! target, so its `parse` helper is unreachable from here. The PDS
+//! firehose hands us a CAR in every `#commit` frame's `blocks` field, so
+//! the AppView needs its own.
+//!
+//! Format (<https://ipld.io/specs/transport/car/carv1/>):
+//!
+//! ```text
+//! [ varint: header_len | DAG-CBOR header ]      { version: 1, roots: [CID] }
+//! [ varint: section_len | CID | block bytes ]   block 1
+//! [ varint: section_len | CID | block bytes ]   block 2
+//! ...
+//! ```
+//!
+//! The header is decoded with [`crate::cbor`], which accepts both the
+//! spec's `tag(42) + bytes(0x00 || cid)` link and the bare
+//! `tag(42) + bytes(cid)` this codebase's writer emits.
+//!
+//! Only structure is validated. Block CIDs are *not* re-hashed here:
+//! the firehose connection is to our own PDS over the cluster-internal
+//! URL, and a mismatch would mean a bug rather than an attack. See
+//! [`verify_block_cids`] for the opt-in check the tests use.
+
+use anyhow::{anyhow, bail, Result};
+use cid::Cid;
+use std::collections::HashMap;
+
+use crate::cbor;
+
+/// The parsed CAR header.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct CarHeader {
+    pub version: u64,
+    pub roots: Vec<Cid>,
+}
+
+/// One `(CID, bytes)` pair out of a CAR file.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct CarBlock {
+    pub cid: Cid,
+    pub data: Vec<u8>,
+}
+
+/// A CAR file's header plus its blocks, in file order.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct Car {
+    pub header: CarHeader,
+    pub blocks: Vec<CarBlock>,
+}
+
+impl Car {
+    /// Index the blocks by CID for lookup by the commit's `ops`.
+    ///
+    /// Duplicate CIDs keep the first occurrence, matching the writer's
+    /// own de-duplication.
+    pub fn block_map(&self) -> HashMap<Cid, &[u8]> {
+        let mut map = HashMap::with_capacity(self.blocks.len());
+        for b in &self.blocks {
+            map.entry(b.cid).or_insert(b.data.as_slice());
+        }
+        map
+    }
+
+    /// The first root, if the header declares one.
+    ///
+    /// `allow(dead_code)`: the ingest path doesn't need the commit
+    /// block itself (the ops carry the record CIDs), but a CAR reader
+    /// that cannot name its root is a reader with a hole in it, and the
+    /// tests read it.
+    #[allow(dead_code)]
+    pub fn root(&self) -> Option<Cid> {
+        self.roots().first().copied()
+    }
+
+    #[allow(dead_code)]
+    pub fn roots(&self) -> &[Cid] {
+        &self.header.roots
+    }
+}
+
+/// Parse a CAR v1 byte stream.
+pub fn parse(bytes: &[u8]) -> Result<Car> {
+    let mut p = 0usize;
+
+    let (header_len, n) = read_varint(bytes, p)?;
+    p += n;
+    let header_end = checked_end(bytes, p, header_len, "CAR header")?;
+    let header = decode_header(&bytes[p..header_end])?;
+    p = header_end;
+
+    let mut blocks = Vec::new();
+    while p < bytes.len() {
+        let (section_len, n) = read_varint(bytes, p)?;
+        let section_start = p + n;
+        let section_end = checked_end(bytes, section_start, section_len, "CAR section")?;
+        let section = &bytes[section_start..section_end];
+        let cid = Cid::read_bytes(section)
+            .map_err(|e| anyhow!("invalid CID in CAR section at offset {section_start}: {e}"))?;
+        let cid_len = cid.encoded_len();
+        if cid_len > section.len() {
+            bail!("CAR section at {section_start} is shorter than its CID");
+        }
+        blocks.push(CarBlock {
+            cid,
+            data: section[cid_len..].to_vec(),
+        });
+        p = section_end;
+    }
+
+    Ok(Car { header, blocks })
+}
+
+/// Re-hash every block and compare against its declared CID.
+///
+/// Not called on the ingest path (see the module docs); the CAR reader
+/// tests use it to prove the reader hands back the bytes the writer put
+/// in, unshifted by an off-by-one in the section framing.
+#[allow(dead_code)]
+pub fn verify_block_cids(car: &Car) -> Result<()> {
+    for b in &car.blocks {
+        let recomputed = at_crypto::cid::cid_for_cbor(&b.data)?;
+        if recomputed != b.cid {
+            bail!("CAR block CID mismatch: declared {}, computed {recomputed}", b.cid);
+        }
+    }
+    Ok(())
+}
+
+fn decode_header(bytes: &[u8]) -> Result<CarHeader> {
+    let value = cbor::decode(bytes)?;
+    let version = value
+        .get("version")
+        .and_then(|v| v.as_i64())
+        .ok_or_else(|| anyhow!("CAR header missing `version`"))?;
+    if version != 1 {
+        bail!("unsupported CAR version {version} (only v1 is defined for atproto)");
+    }
+    // `roots` is required by the spec but may legitimately be empty.
+    let roots = match value.get("roots") {
+        Some(v) => v
+            .as_array()
+            .ok_or_else(|| anyhow!("CAR header `roots` is not an array"))?
+            .iter()
+            .map(|item| {
+                item.as_cid()
+                    .ok_or_else(|| anyhow!("CAR header root is not a CID link"))
+            })
+            .collect::<Result<Vec<_>>>()?,
+        None => Vec::new(),
+    };
+    Ok(CarHeader {
+        version: version as u64,
+        roots,
+    })
+}
+
+/// LEB128 unsigned varint, the length prefix CAR uses.
+fn read_varint(bytes: &[u8], offset: usize) -> Result<(u64, usize)> {
+    let mut value: u64 = 0;
+    let mut shift = 0u32;
+    let mut i = offset;
+    loop {
+        let b = *bytes
+            .get(i)
+            .ok_or_else(|| anyhow!("varint extends past end of CAR input at {offset}"))?;
+        i += 1;
+        value |= u64::from(b & 0x7f) << shift;
+        if b & 0x80 == 0 {
+            return Ok((value, i - offset));
+        }
+        shift += 7;
+        if shift >= 64 {
+            bail!("varint longer than 64 bits at offset {offset}");
+        }
+    }
+}
+
+fn checked_end(bytes: &[u8], pos: usize, len: u64, what: &str) -> Result<usize> {
+    let len = usize::try_from(len).map_err(|_| anyhow!("{what} length overflows usize"))?;
+    let end = pos
+        .checked_add(len)
+        .ok_or_else(|| anyhow!("{what} length overflows"))?;
+    if end > bytes.len() {
+        bail!("{what} length {len} exceeds input (offset {pos}, total {})", bytes.len());
+    }
+    Ok(end)
+}
+
+#[cfg(test)]
+pub(crate) mod test_writer {
+    //! A byte-for-byte copy of `pds-server/src/car.rs`'s encoder, so the
+    //! reader's tests exercise *the writer's* output rather than a
+    //! convenient fiction.
+    //!
+    //! Copied rather than imported because `pds-server` has no library
+    //! target. If the writer ever changes shape, the integration test
+    //! `pds_firehose_integration::car_reader_parses_a_real_repo_export`
+    //! is the tripwire: it parses a CAR produced by the running PDS.
+
+    use cid::Cid;
+
+    fn cbor_head(out: &mut Vec<u8>, major: u8, n: u64) {
+        let m = (major & 0x07) << 5;
+        if n < 24 {
+            out.push(m | n as u8);
+        } else if n < 0x100 {
+            out.push(m | 24);
+            out.push(n as u8);
+        } else if n < 0x10000 {
+            out.push(m | 25);
+            out.push((n >> 8) as u8);
+            out.push(n as u8);
+        } else if n < 0x100_0000 {
+            out.push(m | 26);
+            out.push((n >> 16) as u8);
+            out.push((n >> 8) as u8);
+            out.push(n as u8);
+        } else {
+            out.push(m | 27);
+            out.push((n >> 24) as u8);
+            out.push((n >> 16) as u8);
+            out.push((n >> 8) as u8);
+            out.push(n as u8);
+        }
+    }
+
+    fn cbor_text(out: &mut Vec<u8>, s: &str) {
+        cbor_head(out, 3, s.len() as u64);
+        out.extend_from_slice(s.as_bytes());
+    }
+
+    fn cbor_bytes(out: &mut Vec<u8>, b: &[u8]) {
+        cbor_head(out, 2, b.len() as u64);
+        out.extend_from_slice(b);
+    }
+
+    pub fn encode_header(roots: &[Cid]) -> Vec<u8> {
+        let mut out = Vec::new();
+        cbor_head(&mut out, 5, 2);
+        cbor_text(&mut out, "version");
+        cbor_head(&mut out, 0, 1);
+        cbor_text(&mut out, "roots");
+        cbor_head(&mut out, 4, roots.len() as u64);
+        for cid in roots {
+            cbor_head(&mut out, 6, 42);
+            cbor_bytes(&mut out, &cid.to_bytes());
+        }
+        out
+    }
+
+    fn write_varint(out: &mut Vec<u8>, mut n: u64) {
+        loop {
+            let mut byte = (n & 0x7f) as u8;
+            n >>= 7;
+            if n != 0 {
+                byte |= 0x80;
+            }
+            out.push(byte);
+            if n == 0 {
+                return;
+            }
+        }
+    }
+
+    #[derive(Default)]
+    pub struct CarWriter {
+        blocks: Vec<(Cid, Vec<u8>)>,
+    }
+
+    impl CarWriter {
+        pub fn new() -> Self {
+            Self::default()
+        }
+
+        pub fn append(&mut self, cid: Cid, data: &[u8]) {
+            if self.blocks.iter().any(|(c, _)| *c == cid) {
+                return;
+            }
+            self.blocks.push((cid, data.to_vec()));
+        }
+
+        pub fn finish(&self, roots: &[Cid]) -> Vec<u8> {
+            let header = encode_header(roots);
+            let mut out = Vec::new();
+            write_varint(&mut out, header.len() as u64);
+            out.extend_from_slice(&header);
+            for (cid, data) in &self.blocks {
+                let cid_bytes = cid.to_bytes();
+                write_varint(&mut out, (cid_bytes.len() + data.len()) as u64);
+                out.extend_from_slice(&cid_bytes);
+                out.extend_from_slice(data);
+            }
+            out
+        }
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::test_writer::CarWriter;
+    use super::*;
+    use at_crypto::cid::cid_for_cbor;
+
+    #[test]
+    fn round_trips_a_single_block() {
+        let data = b"hello world".to_vec();
+        let cid = cid_for_cbor(&data).unwrap();
+        let mut w = CarWriter::new();
+        w.append(cid, &data);
+        let car = parse(&w.finish(&[cid])).unwrap();
+        assert_eq!(car.header.version, 1);
+        assert_eq!(car.roots(), &[cid]);
+        assert_eq!(car.blocks.len(), 1);
+        assert_eq!(car.blocks[0].data, data);
+        verify_block_cids(&car).unwrap();
+    }
+
+    #[test]
+    fn round_trips_many_blocks_in_order() {
+        let payloads: Vec<Vec<u8>> = (0..6)
+            .map(|i| format!("block-{i}").into_bytes())
+            .collect();
+        let cids: Vec<Cid> = payloads.iter().map(|p| cid_for_cbor(p).unwrap()).collect();
+        let mut w = CarWriter::new();
+        for (cid, data) in cids.iter().zip(&payloads) {
+            w.append(*cid, data);
+        }
+        let car = parse(&w.finish(&[cids[3]])).unwrap();
+        assert_eq!(car.root(), Some(cids[3]));
+        assert_eq!(car.blocks.len(), 6);
+        for (i, b) in car.blocks.iter().enumerate() {
+            assert_eq!(b.cid, cids[i]);
+            assert_eq!(b.data, payloads[i]);
+        }
+        verify_block_cids(&car).unwrap();
+    }
+
+    #[test]
+    fn block_map_finds_blocks_by_cid() {
+        let a = b"record a".to_vec();
+        let b = b"record b".to_vec();
+        let (ca, cb) = (cid_for_cbor(&a).unwrap(), cid_for_cbor(&b).unwrap());
+        let mut w = CarWriter::new();
+        w.append(ca, &a);
+        w.append(cb, &b);
+        let car = parse(&w.finish(&[ca])).unwrap();
+        let map = car.block_map();
+        assert_eq!(map.get(&ca).copied(), Some(a.as_slice()));
+        assert_eq!(map.get(&cb).copied(), Some(b.as_slice()));
+        assert!(!map.contains_key(&cid_for_cbor(b"absent").unwrap()));
+    }
+
+    #[test]
+    fn handles_a_multi_byte_varint_section_length() {
+        // A >127 byte block forces a two-byte varint, which is where an
+        // off-by-one in the length prefix would show up.
+        let data = vec![0x42u8; 500];
+        let cid = cid_for_cbor(&data).unwrap();
+        let mut w = CarWriter::new();
+        w.append(cid, &data);
+        let car = parse(&w.finish(&[cid])).unwrap();
+        assert_eq!(car.blocks[0].data.len(), 500);
+        verify_block_cids(&car).unwrap();
+    }
+
+    #[test]
+    fn accepts_an_empty_root_list() {
+        let data = b"orphan".to_vec();
+        let cid = cid_for_cbor(&data).unwrap();
+        let mut w = CarWriter::new();
+        w.append(cid, &data);
+        let car = parse(&w.finish(&[])).unwrap();
+        assert!(car.roots().is_empty());
+        assert_eq!(car.blocks.len(), 1);
+    }
+
+    #[test]
+    fn rejects_truncated_input() {
+        let data = b"hello".to_vec();
+        let cid = cid_for_cbor(&data).unwrap();
+        let mut w = CarWriter::new();
+        w.append(cid, &data);
+        let bytes = w.finish(&[cid]);
+        // Cut into the last block's payload.
+        assert!(parse(&bytes[..bytes.len() - 3]).is_err());
+        // Cut inside the header.
+        assert!(parse(&bytes[..3]).is_err());
+        // Nothing at all.
+        assert!(parse(&[]).is_err());
+    }
+}

crates/appview/src/cbor.rs

@@ -0,0 +1,442 @@
+//! A minimal, allocation-honest CBOR reader — just enough to decode the
+//! frames of `com.atproto.sync.subscribeRepos` and the header of a CAR
+//! file.
+//!
+//! ## Why not `ciborium`?
+//!
+//! Two reasons, and both come from what the wire format actually is.
+//!
+//! 1. **Two values per message.** A subscribeRepos frame is *two*
+//!    DAG-CBOR values written back to back (header, then body) inside
+//!    one WebSocket binary message. A `serde`-shaped reader gives us
+//!    "decode one value from this slice" and no cursor we can resume
+//!    from, so we would have to guess where the header ended.
+//! 2. **Tag 42.** DAG-CBOR encodes a CID link as `tag(42) +
+//!    bytes(<cid>)`. `ciborium`'s `serde` mapping turns tags into a
+//!    private newtype dance that does not survive a round trip through
+//!    `serde_json::Value`, which is the shape the rest of the AppView
+//!    speaks.
+//!
+//! So this module decodes CBOR into its own small [`Cbor`] tree with an
+//! explicit byte offset, which makes "read the header, then read the
+//! body from where the header stopped" a two-line function.
+//!
+//! ## What it deliberately does not do
+//!
+//! No indefinite-length items (DAG-CBOR forbids them; we reject them
+//! rather than guess), no half floats beyond a plain `f64` widening, no
+//! canonicalisation checks. It is a *reader* for input we already
+//! decided to trust at the transport layer, with hard length checks so
+//! a malformed frame returns `Err` instead of panicking.
+//!
+//! Record blocks inside the CAR payload are NOT decoded with this
+//! module: they are written by `ciborium::into_writer(&serde_json::Value)`
+//! on the PDS side (see `pds-server/src/routes/repo.rs`), which means
+//! CIDs inside a record are plain strings, not tag-42 links. Their
+//! inverse is `ciborium::from_reader::<serde_json::Value, _>`, and
+//! that is what [`crate::pds_firehose`] uses for them.
+
+use anyhow::{anyhow, bail, Result};
+use cid::Cid;
+
+/// A decoded CBOR value.
+///
+/// `Nint` carries the already-negated value (CBOR stores `-1 - n`), so
+/// callers never have to remember the bias.
+#[derive(Debug, Clone, PartialEq)]
+pub enum Cbor {
+    Uint(u64),
+    Nint(i64),
+    Bytes(Vec<u8>),
+    Text(String),
+    Array(Vec<Cbor>),
+    /// Kept as an ordered key/value list rather than a map: DAG-CBOR
+    /// keys are text and already canonically ordered, and a `Vec` keeps
+    /// the decoder free of hashing while the maps we read have a
+    /// handful of entries at most.
+    Map(Vec<(Cbor, Cbor)>),
+    Tag(u64, Box<Cbor>),
+    Bool(bool),
+    Null,
+    Undefined,
+    Float(f64),
+}
+
+impl Cbor {
+    /// Look up a text key in a map. `None` for non-maps and misses.
+    pub fn get(&self, key: &str) -> Option<&Cbor> {
+        match self {
+            Cbor::Map(entries) => entries.iter().find_map(|(k, v)| match k {
+                Cbor::Text(s) if s == key => Some(v),
+                _ => None,
+            }),
+            _ => None,
+        }
+    }
+
+    /// The value as a signed integer, accepting both CBOR integer
+    /// majors. `None` when the value is not an integer, or when an
+    /// unsigned value exceeds `i64::MAX` (which cannot happen for a
+    /// `seq`, but silently wrapping would be worse than a miss).
+    pub fn as_i64(&self) -> Option<i64> {
+        match self {
+            Cbor::Uint(n) => i64::try_from(*n).ok(),
+            Cbor::Nint(n) => Some(*n),
+            _ => None,
+        }
+    }
+
+    pub fn as_str(&self) -> Option<&str> {
+        match self {
+            Cbor::Text(s) => Some(s.as_str()),
+            _ => None,
+        }
+    }
+
+    pub fn as_bytes(&self) -> Option<&[u8]> {
+        match self {
+            Cbor::Bytes(b) => Some(b.as_slice()),
+            _ => None,
+        }
+    }
+
+    pub fn as_bool(&self) -> Option<bool> {
+        match self {
+            Cbor::Bool(b) => Some(*b),
+            _ => None,
+        }
+    }
+
+    pub fn as_array(&self) -> Option<&[Cbor]> {
+        match self {
+            Cbor::Array(items) => Some(items.as_slice()),
+            _ => None,
+        }
+    }
+
+    pub fn is_null(&self) -> bool {
+        matches!(self, Cbor::Null | Cbor::Undefined)
+    }
+
+    /// Decode a DAG-CBOR CID link: `tag(42) + bytes(...)`.
+    ///
+    /// The spec prefixes the CID bytes with a single `0x00` (the
+    /// multibase "identity" marker), because a CID inside a byte string
+    /// has no textual multibase prefix to carry. **This repository's own
+    /// writer omits that byte** — `pds-server/src/car.rs` writes
+    /// `cbor_bytes(&cid.to_bytes())` — so we accept both spellings: a
+    /// leading `0x00` is skipped, anything else is parsed as-is. Being
+    /// lenient here costs nothing (a real CIDv1 never starts with
+    /// `0x00`, and CIDv0 starts with `0x12`) and it means the AppView
+    /// keeps working whether the PDS follows the spec or the house
+    /// convention.
+    pub fn as_cid(&self) -> Option<Cid> {
+        let inner = match self {
+            Cbor::Tag(42, inner) => inner.as_ref(),
+            // A bare byte string where a link is expected: CARs written by
+            // older builds of this PDS tagged the root CID without the
+            // `0x00` identity prefix, and some encoders drop the tag
+            // entirely. Both still have to parse.
+            Cbor::Bytes(_) => self,
+            // A record block written from `serde_json::Value` spells a
+            // CID as a plain string — accept that too, so callers do
+            // not need a second code path for the block contents.
+            Cbor::Text(s) => return s.parse::<Cid>().ok(),
+            _ => return None,
+        };
+        let raw = inner.as_bytes()?;
+        let raw = match raw.first() {
+            Some(0x00) => &raw[1..],
+            _ => raw,
+        };
+        Cid::read_bytes(raw).ok()
+    }
+}
+
+/// Decode exactly one CBOR value starting at `pos`.
+///
+/// Returns the value and the offset just past it, so a caller can read
+/// the next value from the same buffer — which is precisely what a
+/// two-value subscribeRepos frame needs.
+pub fn decode_at(bytes: &[u8], pos: usize) -> Result<(Cbor, usize)> {
+    let (major, arg, mut p) = read_head(bytes, pos)?;
+    match major {
+        0 => Ok((Cbor::Uint(arg), p)),
+        1 => {
+            // CBOR negative integers store `-1 - n`. Values below
+            // `i64::MIN` cannot occur in anything we consume, and
+            // wrapping them would produce a positive number, so bail.
+            let n = i64::try_from(arg)
+                .map_err(|_| anyhow!("negative integer out of i64 range"))?;
+            Ok((Cbor::Nint(-1 - n), p))
+        }
+        2 => {
+            let end = checked_end(bytes, p, arg, "byte string")?;
+            let v = bytes[p..end].to_vec();
+            Ok((Cbor::Bytes(v), end))
+        }
+        3 => {
+            let end = checked_end(bytes, p, arg, "text string")?;
+            let s = std::str::from_utf8(&bytes[p..end])
+                .map_err(|e| anyhow!("invalid UTF-8 in CBOR text: {e}"))?
+                .to_string();
+            Ok((Cbor::Text(s), end))
+        }
+        4 => {
+            let mut items = Vec::with_capacity(sane_capacity(arg));
+            for _ in 0..arg {
+                let (v, next) = decode_at(bytes, p)?;
+                items.push(v);
+                p = next;
+            }
+            Ok((Cbor::Array(items), p))
+        }
+        5 => {
+            let mut entries = Vec::with_capacity(sane_capacity(arg));
+            for _ in 0..arg {
+                let (k, next) = decode_at(bytes, p)?;
+                let (v, next) = decode_at(bytes, next)?;
+                entries.push((k, v));
+                p = next;
+            }
+            Ok((Cbor::Map(entries), p))
+        }
+        6 => {
+            let (inner, next) = decode_at(bytes, p)?;
+            Ok((Cbor::Tag(arg, Box::new(inner)), next))
+        }
+        7 => match arg {
+            20 => Ok((Cbor::Bool(false), p)),
+            21 => Ok((Cbor::Bool(true), p)),
+            22 => Ok((Cbor::Null, p)),
+            23 => Ok((Cbor::Undefined, p)),
+            // Floats arrive as the raw bit pattern in `arg`; the width
+            // is implied by the additional-information byte, which
+            // `read_head` has already consumed. We only ever see f64 in
+            // practice (DAG-CBOR requires it), so the narrower widths
+            // are decoded for completeness rather than need.
+            _ => Ok((Cbor::Float(f64::from_bits(arg)), p)),
+        },
+        other => bail!("unsupported CBOR major type {other}"),
+    }
+}
+
+/// Decode a single CBOR value that must span the whole buffer.
+pub fn decode(bytes: &[u8]) -> Result<Cbor> {
+    let (v, end) = decode_at(bytes, 0)?;
+    if end != bytes.len() {
+        bail!("trailing bytes after CBOR value ({} left)", bytes.len() - end);
+    }
+    Ok(v)
+}
+
+/// Read a CBOR head: major type plus its argument.
+///
+/// Indefinite-length encodings (`additional information == 31`) are
+/// rejected: DAG-CBOR forbids them, and accepting them would mean
+/// implementing break-stop scanning for input that should never carry
+/// it.
+fn read_head(bytes: &[u8], pos: usize) -> Result<(u8, u64, usize)> {
+    let first = *bytes
+        .get(pos)
+        .ok_or_else(|| anyhow!("CBOR read past end of input at {pos}"))?;
+    let major = first >> 5;
+    let low = first & 0x1f;
+    let (arg, extra) = match low {
+        0..=23 => (low as u64, 0usize),
+        24 => (read_uint(bytes, pos + 1, 1)?, 1),
+        25 => (read_uint(bytes, pos + 1, 2)?, 2),
+        26 => (read_uint(bytes, pos + 1, 4)?, 4),
+        27 => (read_uint(bytes, pos + 1, 8)?, 8),
+        31 => bail!("indefinite-length CBOR item is not valid DAG-CBOR"),
+        other => bail!("reserved CBOR additional information {other}"),
+    };
+    // For major 7 the "argument" of a float is the raw bit pattern, and
+    // f32/f16 need widening before `f64::from_bits` makes sense.
+    let arg = match (major, low) {
+        (7, 25) => f64::from(half_to_f32(arg as u16)).to_bits(),
+        (7, 26) => f64::from(f32::from_bits(arg as u32)).to_bits(),
+        _ => arg,
+    };
+    Ok((major, arg, pos + 1 + extra))
+}
+
+fn read_uint(bytes: &[u8], pos: usize, len: usize) -> Result<u64> {
+    if pos + len > bytes.len() {
+        bail!("truncated CBOR integer of {len} byte(s) at {pos}");
+    }
+    let mut n: u64 = 0;
+    for b in &bytes[pos..pos + len] {
+        n = (n << 8) | u64::from(*b);
+    }
+    Ok(n)
+}
+
+/// IEEE-754 half → single. Only reached for `f16` inputs, which nothing
+/// in this protocol emits; kept so a stray value decodes instead of
+/// erroring out mid-frame.
+fn half_to_f32(bits: u16) -> f32 {
+    let sign = ((bits >> 15) & 1) as u32;
+    let exp = ((bits >> 10) & 0x1f) as u32;
+    let frac = (bits & 0x3ff) as u32;
+    let out = match exp {
+        0 if frac == 0 => sign << 31,
+        0 => {
+            // Subnormal: renormalise.
+            let mut e = -1i32;
+            let mut f = frac;
+            while f & 0x400 == 0 {
+                f <<= 1;
+                e -= 1;
+            }
+            let exp32 = (127 - 15 + e) as u32;
+            (sign << 31) | (exp32 << 23) | ((f & 0x3ff) << 13)
+        }
+        0x1f => (sign << 31) | (0xff << 23) | (frac << 13),
+        _ => (sign << 31) | ((exp + 127 - 15) << 23) | (frac << 13),
+    };
+    f32::from_bits(out)
+}
+
+/// Bounds-check a string/bytes payload before slicing it.
+fn checked_end(bytes: &[u8], pos: usize, len: u64, what: &str) -> Result<usize> {
+    let len = usize::try_from(len).map_err(|_| anyhow!("{what} length overflows usize"))?;
+    let end = pos
+        .checked_add(len)
+        .ok_or_else(|| anyhow!("{what} length overflows"))?;
+    if end > bytes.len() {
+        bail!("{what} of {len} byte(s) exceeds input at offset {pos}");
+    }
+    Ok(end)
+}
+
+/// Cap the pre-allocation a declared array/map length can trigger. A
+/// corrupt frame claiming `map(2^40)` must not make us reserve 40 GiB
+/// before the first missing byte errors out; the collection still grows
+/// naturally for genuinely large inputs.
+fn sane_capacity(declared: u64) -> usize {
+    usize::try_from(declared.min(1024)).unwrap_or(0)
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use at_crypto::cid::cid_for_cbor;
+
+    /// Build `{"op": 1, "t": "#commit"}` by hand — the exact bytes the
+    /// PDS writes for a regular frame header.
+    fn commit_header_bytes() -> Vec<u8> {
+        let mut v = vec![0xA2]; // map(2)
+        v.push(0x62); // text(2)
+        v.extend_from_slice(b"op");
+        v.push(0x01); // uint 1
+        v.push(0x61); // text(1)
+        v.extend_from_slice(b"t");
+        v.push(0x67); // text(7)
+        v.extend_from_slice(b"#commit");
+        v
+    }
+
+    #[test]
+    fn decodes_a_frame_header() {
+        let v = decode(&commit_header_bytes()).unwrap();
+        assert_eq!(v.get("op").unwrap().as_i64(), Some(1));
+        assert_eq!(v.get("t").unwrap().as_str(), Some("#commit"));
+        assert!(v.get("missing").is_none());
+    }
+
+    #[test]
+    fn decodes_negative_op_of_an_error_header() {
+        // {"op": -1} → map(1), text(2)"op", nint(0) = -1
+        let bytes = vec![0xA1, 0x62, b'o', b'p', 0x20];
+        let v = decode(&bytes).unwrap();
+        assert_eq!(v.get("op").unwrap().as_i64(), Some(-1));
+    }
+
+    #[test]
+    fn decodes_multi_byte_integers() {
+        // uint16 300, uint32 70000, uint64 2^33, nint -300
+        assert_eq!(decode(&[0x19, 0x01, 0x2C]).unwrap().as_i64(), Some(300));
+        assert_eq!(
+            decode(&[0x1A, 0x00, 0x01, 0x11, 0x70]).unwrap().as_i64(),
+            Some(70000)
+        );
+        assert_eq!(
+            decode(&[0x1B, 0, 0, 0, 2, 0, 0, 0, 0]).unwrap().as_i64(),
+            Some(8_589_934_592)
+        );
+        assert_eq!(decode(&[0x39, 0x01, 0x2B]).unwrap().as_i64(), Some(-300));
+    }
+
+    #[test]
+    fn decode_at_reads_two_values_back_to_back() {
+        // This is the whole reason the module exists: a frame is header
+        // + body concatenated with no separator.
+        let mut buf = commit_header_bytes();
+        let body_start = buf.len();
+        buf.extend_from_slice(&[0xA1, 0x63, b's', b'e', b'q', 0x18, 0x2A]); // {"seq": 42}
+        let (header, next) = decode_at(&buf, 0).unwrap();
+        assert_eq!(next, body_start);
+        assert_eq!(header.get("t").unwrap().as_str(), Some("#commit"));
+        let (body, end) = decode_at(&buf, next).unwrap();
+        assert_eq!(end, buf.len());
+        assert_eq!(body.get("seq").unwrap().as_i64(), Some(42));
+    }
+
+    #[test]
+    fn decodes_tag_42_cid_link_both_spellings() {
+        let cid = cid_for_cbor(b"a block").unwrap();
+        let raw = cid.to_bytes();
+
+        // House spelling: tag(42) + bytes(<cid>) with no 0x00 prefix.
+        let mut bare = vec![0xD8, 42]; // tag(42) via 1-byte extension
+        bare.push(0x58); // bytes, 1-byte length
+        bare.push(raw.len() as u8);
+        bare.extend_from_slice(&raw);
+        assert_eq!(decode(&bare).unwrap().as_cid(), Some(cid));
+
+        // Spec spelling: the same, with the identity multibase prefix.
+        let mut prefixed = vec![0xD8, 42, 0x58, (raw.len() + 1) as u8, 0x00];
+        prefixed.extend_from_slice(&raw);
+        assert_eq!(decode(&prefixed).unwrap().as_cid(), Some(cid));
+
+        // And the string spelling records use.
+        let text = {
+            let s = cid.to_string();
+            let mut v = vec![0x78, s.len() as u8];
+            v.extend_from_slice(s.as_bytes());
+            v
+        };
+        assert_eq!(decode(&text).unwrap().as_cid(), Some(cid));
+    }
+
+    #[test]
+    fn decodes_simple_values_and_containers() {
+        // [true, false, null] → array(3)
+        let v = decode(&[0x83, 0xF5, 0xF4, 0xF6]).unwrap();
+        let items = v.as_array().unwrap();
+        assert_eq!(items[0].as_bool(), Some(true));
+        assert_eq!(items[1].as_bool(), Some(false));
+        assert!(items[2].is_null());
+    }
+
+    #[test]
+    fn rejects_truncated_and_indefinite_input() {
+        // text(7) claiming 7 bytes but carrying 2.
+        assert!(decode(&[0x67, b'a', b'b']).is_err());
+        // Indefinite-length array.
+        assert!(decode(&[0x9F, 0x01, 0xFF]).is_err());
+        // Trailing garbage after a complete value.
+        assert!(decode(&[0x01, 0x02]).is_err());
+        // Empty input.
+        assert!(decode(&[]).is_err());
+    }
+
+    #[test]
+    fn oversized_declared_length_errors_instead_of_allocating() {
+        // map(2^32) with nothing behind it. Must return Err quickly
+        // rather than trying to reserve the declared capacity.
+        let bytes = vec![0xBA, 0xFF, 0xFF, 0xFF, 0xFF];
+        assert!(decode(&bytes).is_err());
+    }
+}

crates/appview/src/firehose.rs

@@ -16,7 +16,7 @@ use std::sync::atomic::{AtomicBool, AtomicI64, AtomicU64, Ordering};
 use std::sync::Arc;
 use std::time::Duration;
 use tokio::sync::mpsc;
-use tracing::{debug, info, trace, warn};
+use tracing::{debug, info, warn};
 
 use crate::indexer;
 
@@ -31,6 +31,18 @@ pub struct Stats {
     /// writes this; `/healthz` reads it. Wrapped in `Arc` so the consumer
     /// can hold its own clone without borrowing from us.
     pub jetstream_connected: Arc<AtomicBool>,
+    /// Whether the **local PDS** firehose WebSocket is currently up
+    /// ([`crate::pds_firehose`]). Separate from `jetstream_connected`
+    /// because the two streams fail independently and for different
+    /// reasons: a dead Jetstream means no view of the wider network, a
+    /// dead PDS firehose means the AppView has lost the guaranteed
+    /// delivery path for its *own* users' records and is running on the
+    /// best-effort push alone. `/healthz` has to be able to say which.
+    pub pds_connected: AtomicBool,
+    /// Number of `#commit` frames applied from the PDS firehose.
+    pub pds_frames_processed: AtomicU64,
+    /// Highest `seq` applied from the PDS firehose in this process.
+    pub pds_last_seq: AtomicI64,
 }
 
 impl Default for Stats {
@@ -40,6 +52,9 @@ impl Default for Stats {
             last_event_time_us: AtomicI64::new(0),
             last_cursor_persisted_us: AtomicI64::new(0),
             jetstream_connected: Arc::new(AtomicBool::new(false)),
+            pds_connected: AtomicBool::new(false),
+            pds_frames_processed: AtomicU64::new(0),
+            pds_last_seq: AtomicI64::new(0),
         }
     }
 }
@@ -50,6 +65,9 @@ impl std::fmt::Debug for Stats {
             .field("events_processed", &self.events_processed())
             .field("last_event_time_us", &self.last_event_time_us.load(Ordering::Relaxed))
             .field("jetstream_connected", &self.jetstream_connected())
+            .field("pds_connected", &self.pds_connected())
+            .field("pds_frames_processed", &self.pds_frames_processed())
+            .field("pds_last_seq", &self.pds_last_seq())
             .finish()
     }
 }
@@ -85,6 +103,23 @@ impl Stats {
     pub fn jetstream_connected(&self) -> bool {
         self.jetstream_connected.load(Ordering::Relaxed)
     }
+
+    /// Is the local PDS firehose connected right now?
+    pub fn pds_connected(&self) -> bool {
+        self.pds_connected.load(Ordering::Relaxed)
+    }
+
+    pub fn pds_frames_processed(&self) -> u64 {
+        self.pds_frames_processed.load(Ordering::Relaxed)
+    }
+
+    /// Highest PDS-firehose `seq` this process has applied. 0 before the
+    /// first frame — note this is the *in-process* high-water mark, not
+    /// the persisted cursor, which lives in `pds_firehose_cursor` and
+    /// survives restarts.
+    pub fn pds_last_seq(&self) -> i64 {
+        self.pds_last_seq.load(Ordering::Relaxed)
+    }
 }
 
 /// The thing the Jetstream consumer calls once per event.

crates/appview/src/indexer.rs

@@ -693,24 +693,46 @@ where
 
 // -- follows ---------------------------------------------------------------
 
+/// Insert or update the follow edge `follower_did -> subject_did`.
+///
+/// `rkey` is the record key of the `app.bsky.graph.follow` record this
+/// edge came from. It is stored as a *second access path* to the row —
+/// the primary key stays `(follower_did, subject_did)`, which is what
+/// keeps this upsert idempotent across the push path, the firehose and
+/// any replay of either. See migration 0011 for the full reasoning.
+///
+/// Pass `None` only when the caller genuinely has no rkey. On conflict
+/// the column is `COALESCE(EXCLUDED.rkey, follows.rkey)`: a newer record
+/// overwrites it (youngest record wins, so a re-follow's rkey replaces
+/// the old one and a stale delete for the old rkey can no longer match),
+/// but a caller that omits the rkey must not blank out one another
+/// transport already recorded — that would re-open the very gap this
+/// column closes.
 pub async fn upsert_follow(
     db: &PgPool,
     follower_did: &str,
     subject_did: &str,
+    rkey: Option<&str>,
     record: Option<&Value>,
 ) -> Result<()> {
     let created_at = parse_created_at(
         record.and_then(|r| r.get("createdAt")).and_then(|v| v.as_str()),
     );
+    // An empty rkey is not an rkey — treat it like the absent case so a
+    // caller forwarding a blank field can't write a row that a
+    // `WHERE rkey = ''` delete would later match by accident.
+    let rkey = rkey.filter(|r| !r.is_empty());
     sqlx::query(
-        r#"INSERT INTO follows (follower_did, subject_did, created_at)
-           VALUES ($1, $2, $3)
+        r#"INSERT INTO follows (follower_did, subject_did, rkey, created_at)
+           VALUES ($1, $2, $3, $4)
            ON CONFLICT (follower_did, subject_did) DO UPDATE SET
+             rkey       = COALESCE(EXCLUDED.rkey, follows.rkey),
              created_at = EXCLUDED.created_at,
              indexed_at = now()"#,
     )
     .bind(follower_did)
     .bind(subject_did)
+    .bind(rkey)
     .bind(created_at)
     .execute(db)
     .await?;
@@ -733,6 +755,12 @@ pub async fn upsert_follow(
     Ok(())
 }
 
+/// Delete the follow edge by its relationship identity.
+///
+/// This is the PDS-push path: `/internal/ingest-commit` carries the
+/// `subject_did` from the PDS's own snapshot, so the row can be
+/// addressed directly. Idempotent — deleting an edge that is already
+/// gone is a no-op, not an error.
 pub async fn delete_follow(
     db: &PgPool,
     follower_did: &str,
@@ -748,6 +776,78 @@ pub async fn delete_follow(
     Ok(())
 }
 
+/// Delete the follow edge that came from record `rkey` in
+/// `follower_did`'s repo.
+///
+/// This is the firehose / Jetstream path. A delete op carries only
+/// `did` + `rkey` and no record body, so the subject DID has to be
+/// recovered from the row itself — which is exactly what the `rkey`
+/// column added in migration 0011 is for. The `DELETE ... RETURNING`
+/// resolves and removes in one statement (the same shape
+/// [`delete_like`] uses to recover its `post_uri`), so there is no
+/// window in which another writer could move the row between the
+/// lookup and the delete.
+///
+/// Returns the `subject_did` that was unfollowed, or `None` when
+/// nothing matched. `None` is a normal outcome, never an error:
+///
+///  * the row predates migration 0011 and has no rkey (the push path
+///    with its `subject_did` still handles those), or
+///  * the delete already landed over the other transport, or
+///  * the follow was re-created under a newer rkey, in which case this
+///    delete is a stale replay and the live edge must be left alone.
+///
+/// The caller logs and moves on — an unfollow we cannot place must not
+/// stall the frames queued behind it.
+pub async fn delete_follow_by_rkey(
+    db: &PgPool,
+    follower_did: &str,
+    rkey: &str,
+) -> Result<Option<String>> {
+    if rkey.is_empty() {
+        // Guard the degenerate case explicitly: `rkey = ''` can never
+        // identify a record, and letting it through would mean an empty
+        // value written by some future caller could be matched here.
+        tracing::warn!(
+            follower_did,
+            "follow delete with an empty rkey; nothing to do"
+        );
+        return Ok(None);
+    }
+    // Deleted with `fetch_all` rather than `fetch_optional` because the
+    // index on `(follower_did, rkey)` is deliberately not unique (see
+    // migration 0011): in the pathological case of a duplicated rkey,
+    // every matching row is a follow whose record is gone, so all of
+    // them should go.
+    let rows: Vec<(String,)> = sqlx::query_as(
+        "DELETE FROM follows WHERE follower_did = $1 AND rkey = $2 \
+         RETURNING subject_did",
+    )
+    .bind(follower_did)
+    .bind(rkey)
+    .fetch_all(db)
+    .await?;
+
+    match rows.into_iter().next() {
+        Some((subject_did,)) => {
+            tracing::debug!(
+                follower_did, rkey, subject_did,
+                "applied an unfollow by rkey"
+            );
+            Ok(Some(subject_did))
+        }
+        None => {
+            tracing::debug!(
+                follower_did, rkey,
+                "follow delete by rkey matched no row (already gone, \
+                 re-created under a newer rkey, or indexed before the \
+                 rkey column existed); skipping"
+            );
+            Ok(None)
+        }
+    }
+}
+
 /// Extract the subject DID from a follow record (`{ "subject": "did:..."}`).
 pub fn follow_subject_did(record: Option<&Value>) -> Option<String> {
     record?
@@ -906,8 +1006,21 @@ pub async fn apply_commit(
                 applied = true;
             }
             "app.bsky.graph.follow" => {
-                let subject_did = match op.action.as_str() {
-                    "create" => match follow_subject_did(op.record.as_ref()) {
+                // Both actions need the rkey. On a create it is stored
+                // alongside the edge; on a delete it is the *only*
+                // thing identifying the edge, because a delete op
+                // carries no record body and therefore no subject DID.
+                let rkey = op
+                    .rkey
+                    .clone()
+                    .or_else(|| {
+                        op.path
+                            .as_deref()
+                            .and_then(|p| p.rsplit('/').next().map(str::to_string))
+                    })
+                    .filter(|r| !r.is_empty());
+                if op.action == "create" {
+                    let subject_did = match follow_subject_did(op.record.as_ref()) {
                         Some(s) => s,
                         None => {
                             tracing::warn!(
@@ -915,33 +1028,35 @@ pub async fn apply_commit(
                             );
                             continue;
                         }
-                    },
-                    "delete" => {
-                        // Jetstream delete on follows carries no record
-                        // value, so we can't know which subject was
-                        // unfollowed. The PDS-driven internal ingest path
-                        // handles this — it knows the subject from its
-                        // own snapshot.
-                        tracing::warn!(
-                            "follow delete via Jetstream lacks subject; \
-                             route through /internal/ingest-commit instead"
-                        );
-                        continue;
-                    }
-                    _ => continue,
-                };
-                if op.action == "create" {
+                    };
                     upsert_follow(
                         db,
                         &ev.did,
                         &subject_did,
+                        rkey.as_deref(),
                         op.record.as_ref(),
                     )
                     .await?;
+                    applied = true;
                 } else if op.action == "delete" {
-                    delete_follow(db, &ev.did, &subject_did).await?;
+                    // The rkey → subject_did lookup added in migration
+                    // 0011. Before it, this arm could only log and skip,
+                    // which left unfollows depending entirely on the
+                    // PDS's best-effort push: one lost request and the
+                    // follow stayed indexed forever.
+                    let Some(rkey) = rkey else {
+                        tracing::warn!(
+                            did = %ev.did,
+                            "follow delete op has no rkey; skipping"
+                        );
+                        continue;
+                    };
+                    delete_follow_by_rkey(db, &ev.did, &rkey).await?;
+                    // Applied even when no row matched: the event was
+                    // understood and acted on, which is what this flag
+                    // reports (same as the post / like / repost deletes).
+                    applied = true;
                 }
-                applied = true;
             }
             "app.bsky.actor.profile" => {
                 // Jetstream carries profile records as plain
@@ -1356,8 +1471,10 @@ mod tests {
         .unwrap();
         assert_eq!(count, 1);
 
-        // Delete via the internal API (not via Jetstream — Jetstream
-        // delete on follows doesn't carry the subject).
+        // Delete through the PDS-push path, which addresses the edge by
+        // `(follower, subject)` because the PDS knows the subject from
+        // its own snapshot. (The firehose path deletes by rkey instead
+        // — see `firehose_unfollow_deletes_by_rkey` below.)
         delete_follow(&db, "did:plc:test", "did:plc:b").await.unwrap();
         let (count,): (i64,) = sqlx::query_as(
             "SELECT COUNT(*) FROM follows WHERE follower_did = $1 AND subject_did = $2",
@@ -1369,6 +1486,274 @@ mod tests {
         .unwrap();
         assert_eq!(count, 0);
     }
+
+    // -- unfollow over the firehose ---------------------------------------
+    //
+    // The tests below cover the gap migration 0011 closes: a delete op
+    // carries only `did` + `rkey`, so the edge has to be recoverable
+    // from the rkey alone. They use per-run unique DIDs because the
+    // suite shares one database with every other test module.
+
+    fn follow_did(tag: &str) -> String {
+        format!("did:plc:follow_{}_{}", tag, uuid::Uuid::new_v4().simple())
+    }
+
+    /// A commit event shaped like the ones `pds_firehose::events_from_frame`
+    /// hands to `apply_commit`: single-op, `record` present on create and
+    /// absent on delete.
+    fn follow_event(
+        did: &str,
+        rkey: &str,
+        action: &str,
+        subject: Option<&str>,
+    ) -> JetstreamEvent {
+        let mut commit = json!({
+            "operation": action,
+            "collection": "app.bsky.graph.follow",
+            "rkey": rkey,
+            "path": format!("app.bsky.graph.follow/{rkey}"),
+        });
+        if let Some(subject) = subject {
+            commit["cid"] = json!("bafyfollow");
+            commit["record"] = json!({
+                "subject": subject,
+                "createdAt": "2026-01-01T00:00:00Z",
+            });
+        }
+        JetstreamEvent {
+            did: did.to_string(),
+            time_us: 1_700_000_000_000_000,
+            kind: "commit".into(),
+            commit: Some(commit),
+            identity: None,
+            account: None,
+        }
+    }
+
+    async fn follow_rows(db: &PgPool, follower: &str) -> Vec<(String, Option<String>)> {
+        sqlx::query_as(
+            "SELECT subject_did, rkey FROM follows WHERE follower_did = $1 \
+             ORDER BY subject_did",
+        )
+        .bind(follower)
+        .fetch_all(db)
+        .await
+        .unwrap()
+    }
+
+    /// The core case: a follow that arrived over the firehose is removed
+    /// by a delete op that names nothing but the rkey.
+    #[tokio::test]
+    async fn firehose_unfollow_deletes_by_rkey() {
+        let Some(db) = try_test_db().await else {
+            eprintln!("appview DB unavailable; skipping");
+            return;
+        };
+        let follower = follow_did("er");
+        let subject = follow_did("ee");
+
+        apply_commit(&db, &follow_event(&follower, "frk1", "create", Some(&subject)))
+            .await
+            .unwrap();
+        assert_eq!(
+            follow_rows(&db, &follower).await,
+            vec![(subject.clone(), Some("frk1".to_string()))],
+            "the create must store the rkey next to the edge"
+        );
+
+        // The delete op carries no record and no subject — only the rkey.
+        let applied = apply_commit(&db, &follow_event(&follower, "frk1", "delete", None))
+            .await
+            .unwrap();
+        assert!(applied, "a follow delete is now actionable, not skipped");
+        assert!(
+            follow_rows(&db, &follower).await.is_empty(),
+            "the unfollow must remove the edge"
+        );
+
+        // Replaying the same delete (reconnect, or the push path racing
+        // the firehose) must stay a silent no-op.
+        apply_commit(&db, &follow_event(&follower, "frk1", "delete", None))
+            .await
+            .unwrap();
+        assert!(follow_rows(&db, &follower).await.is_empty());
+
+        let _ = sqlx::query("DELETE FROM notifications WHERE recipient_did = $1")
+            .bind(&subject)
+            .execute(&db)
+            .await;
+    }
+
+    /// A delete for an rkey we never indexed resolves to nothing. That
+    /// is a normal outcome (the follow was never seen, or is already
+    /// gone), so it must not error and must not touch other rows.
+    #[tokio::test]
+    async fn delete_follow_by_unknown_rkey_is_a_noop() {
+        let Some(db) = try_test_db().await else {
+            eprintln!("appview DB unavailable; skipping");
+            return;
+        };
+        let follower = follow_did("er");
+        let subject = follow_did("ee");
+
+        upsert_follow(&db, &follower, &subject, Some("frk1"), None)
+            .await
+            .unwrap();
+
+        let removed = delete_follow_by_rkey(&db, &follower, "no-such-rkey")
+            .await
+            .unwrap();
+        assert!(removed.is_none(), "an unknown rkey resolves to no subject");
+        assert_eq!(
+            follow_rows(&db, &follower).await,
+            vec![(subject.clone(), Some("frk1".to_string()))],
+            "an unmatched delete must leave every other edge alone"
+        );
+
+        // An empty rkey is guarded separately — it can never identify a
+        // record, and must not be allowed to match a blank column.
+        assert!(delete_follow_by_rkey(&db, &follower, "")
+            .await
+            .unwrap()
+            .is_none());
+        assert_eq!(follow_rows(&db, &follower).await.len(), 1);
+
+        delete_follow(&db, &follower, &subject).await.unwrap();
+        let _ = sqlx::query("DELETE FROM notifications WHERE recipient_did = $1")
+            .bind(&subject)
+            .execute(&db)
+            .await;
+    }
+
+    /// Rows written before migration 0011 have `rkey IS NULL`: there was
+    /// nothing to backfill them from. A delete-by-rkey must not find
+    /// them (and certainly must not match NULL against anything), while
+    /// the push path that names the subject keeps working.
+    #[tokio::test]
+    async fn legacy_row_without_rkey_still_deletes_via_subject() {
+        let Some(db) = try_test_db().await else {
+            eprintln!("appview DB unavailable; skipping");
+            return;
+        };
+        let follower = follow_did("legacy");
+        let subject = follow_did("ee");
+
+        // Insert the way migration 0001 through 0010 did — no rkey.
+        sqlx::query(
+            "INSERT INTO follows (follower_did, subject_did, created_at) \
+             VALUES ($1, $2, now())",
+        )
+        .bind(&follower)
+        .bind(&subject)
+        .execute(&db)
+        .await
+        .unwrap();
+
+        let removed = delete_follow_by_rkey(&db, &follower, "frk1").await.unwrap();
+        assert!(
+            removed.is_none(),
+            "a row with no rkey is unreachable by rkey — by design"
+        );
+        assert_eq!(follow_rows(&db, &follower).await.len(), 1);
+
+        // The PDS push, which carries the subject, still removes it.
+        delete_follow(&db, &follower, &subject).await.unwrap();
+        assert!(follow_rows(&db, &follower).await.is_empty());
+    }
+
+    /// Follow → unfollow → follow again produces a fresh rkey. The edge
+    /// must stay a single row (the primary key is the relationship, not
+    /// the record), the newest rkey must win, and a stale delete for the
+    /// old rkey must not tear down the live follow.
+    #[tokio::test]
+    async fn refollow_keeps_one_row_and_the_newest_rkey_wins() {
+        let Some(db) = try_test_db().await else {
+            eprintln!("appview DB unavailable; skipping");
+            return;
+        };
+        let follower = follow_did("er");
+        let subject = follow_did("ee");
+
+        apply_commit(&db, &follow_event(&follower, "frk1", "create", Some(&subject)))
+            .await
+            .unwrap();
+        apply_commit(&db, &follow_event(&follower, "frk2", "create", Some(&subject)))
+            .await
+            .unwrap();
+        assert_eq!(
+            follow_rows(&db, &follower).await,
+            vec![(subject.clone(), Some("frk2".to_string()))],
+            "one edge, carrying the youngest record's rkey"
+        );
+
+        // The old rkey is stale: its delete must find nothing.
+        assert!(delete_follow_by_rkey(&db, &follower, "frk1")
+            .await
+            .unwrap()
+            .is_none());
+        assert_eq!(
+            follow_rows(&db, &follower).await.len(),
+            1,
+            "a replayed delete for a superseded record must not unfollow"
+        );
+
+        // The current rkey does delete it.
+        assert_eq!(
+            delete_follow_by_rkey(&db, &follower, "frk2").await.unwrap(),
+            Some(subject.clone()),
+            "the delete resolves the subject it removed"
+        );
+        assert!(follow_rows(&db, &follower).await.is_empty());
+
+        let _ = sqlx::query("DELETE FROM notifications WHERE recipient_did = $1")
+            .bind(&subject)
+            .execute(&db)
+            .await;
+    }
+
+    /// A follow that first arrives over the PDS push (no rkey stored by
+    /// an older AppView, or a caller that has none) and is then seen
+    /// again over the firehose must end up with the rkey — otherwise
+    /// the firehose could never delete it. And a later push that omits
+    /// the rkey must not blank it out again.
+    #[tokio::test]
+    async fn rkey_is_filled_in_but_never_blanked_out() {
+        let Some(db) = try_test_db().await else {
+            eprintln!("appview DB unavailable; skipping");
+            return;
+        };
+        let follower = follow_did("er");
+        let subject = follow_did("ee");
+
+        upsert_follow(&db, &follower, &subject, None, None).await.unwrap();
+        assert_eq!(
+            follow_rows(&db, &follower).await,
+            vec![(subject.clone(), None)]
+        );
+
+        // The firehose replay of the same follow supplies the rkey.
+        upsert_follow(&db, &follower, &subject, Some("frk1"), None)
+            .await
+            .unwrap();
+        assert_eq!(
+            follow_rows(&db, &follower).await,
+            vec![(subject.clone(), Some("frk1".to_string()))]
+        );
+
+        // A subsequent write without one must leave it in place.
+        upsert_follow(&db, &follower, &subject, None, None).await.unwrap();
+        assert_eq!(
+            follow_rows(&db, &follower).await,
+            vec![(subject.clone(), Some("frk1".to_string()))],
+            "COALESCE keeps the rkey the other transport already gave us"
+        );
+
+        delete_follow(&db, &follower, &subject).await.unwrap();
+        let _ = sqlx::query("DELETE FROM notifications WHERE recipient_did = $1")
+            .bind(&subject)
+            .execute(&db)
+            .await;
+    }
 }
 
 #[cfg(test)]
@@ -1555,7 +1940,9 @@ mod notification_tests {
             .await
             .unwrap();
         // Self-follow is legal in the protocol; it must stay silent too.
-        upsert_follow(&db, &author, &author, None).await.unwrap();
+        upsert_follow(&db, &author, &author, Some("frk1"), None)
+            .await
+            .unwrap();
 
         let total: i64 = sqlx::query_scalar(
             "SELECT COUNT(*)::BIGINT FROM notifications WHERE recipient_did = $1",
@@ -1657,10 +2044,10 @@ mod notification_tests {
         seed_post(&db, &subject, "p1").await;
 
         let record = json!({ "subject": subject, "createdAt": "2026-01-01T00:00:00Z" });
-        upsert_follow(&db, &follower, &subject, Some(&record))
+        upsert_follow(&db, &follower, &subject, Some("frk1"), Some(&record))
             .await
             .unwrap();
-        upsert_follow(&db, &follower, &subject, Some(&record))
+        upsert_follow(&db, &follower, &subject, Some("frk1"), Some(&record))
             .await
             .unwrap();
         assert_eq!(

crates/appview/src/ingest.rs

@@ -220,10 +220,15 @@ async fn apply(
                         .map(str::to_string)
                 })
                 .ok_or_else(|| bad_request("follow create requires subject_did or record.subject"))?;
+            // Forward the rkey too. The push path doesn't need it to
+            // apply *this* write — it has the subject — but storing it
+            // is what lets a later firehose delete (which carries only
+            // did + rkey) find this row. See migration 0011.
             indexer::upsert_follow(
                 &state.db,
                 &req.did,
                 &subject,
+                Some(req.rkey.as_str()),
                 req.record.as_ref(),
             )
             .await

crates/appview/src/lib.rs

@@ -4,9 +4,12 @@
 //! against a stub resolver without booting the binary.
 
 pub mod auth;
+pub mod car;
+pub mod cbor;
 pub mod firehose;
 pub mod handle_sync;
 pub mod indexer;
 pub mod ingest;
+pub mod pds_firehose;
 pub mod routes;
 pub mod state;

crates/appview/src/main.rs

@@ -8,10 +8,13 @@ use tracing::info;
 use tracing_subscriber::EnvFilter;
 
 mod auth;
+mod car;
+mod cbor;
 mod firehose;
 mod handle_sync;
 mod indexer;
 mod ingest;
+mod pds_firehose;
 mod routes;
 mod state;
 
@@ -82,6 +85,30 @@ async fn main() -> Result<()> {
         });
     }
 
+    // Local PDS firehose. The push path (`/internal/ingest-commit`) is
+    // the fast way a local commit reaches the index; this is the
+    // guaranteed one — it carries a durable cursor, so anything the
+    // push dropped while the AppView was down is replayed on connect.
+    // See the module docs in `pds_firehose.rs` for why both exist.
+    if cfg.pds_firehose_enabled {
+        let start_seq = pds_firehose::cursor_get(&db).await.unwrap_or(0);
+        let base = cfg.pds_base_url();
+        info!(
+            url = %pds_firehose::subscribe_url(&base, (start_seq > 0).then_some(start_seq)),
+            cursor = start_seq,
+            "starting the local PDS firehose consumer"
+        );
+        let consumer = pds_firehose::PdsFirehose::new(db.clone(), base, stats.clone())
+            .with_max_backoff_secs(30);
+        tokio::spawn(consumer.run_forever());
+    } else {
+        tracing::warn!(
+            "PDS_FIREHOSE_ENABLED=false — local commits reach the index only through \
+             the best-effort `/internal/ingest-commit` push; a push lost to a restart \
+             or a network error will NOT be recovered"
+        );
+    }
+
     let state = AppState::new(cfg.clone(), db.clone(), stats.clone());
 
     // Announce every relaxed security switch before we serve anything.

crates/appview/src/routes.rs

@@ -1538,6 +1538,16 @@ async fn actor_list(
 
 // -- healthz ----------------------------------------------------------------
 
+/// Liveness probe.
+///
+/// Both ingest streams report separately. `jetstream_connected` is the
+/// public network's view; `pds_firehose_connected` is the local PDS's
+/// guaranteed delivery path for our own users' records. A deployment
+/// can be perfectly healthy for reads with the first one down, but a
+/// `pds_firehose_enabled: true, pds_firehose_connected: false` pair
+/// means local commits are riding on the best-effort push alone — which
+/// is exactly the state an operator wants to see in a probe rather than
+/// discover from a missing post.
 async fn healthz(State(state): State<AppState>) -> impl IntoResponse {
     let stats = &state.stats;
     Json(json!({
@@ -1545,6 +1555,10 @@ async fn healthz(State(state): State<AppState>) -> impl IntoResponse {
         "lag_ms": stats.lag_ms(),
         "events_processed": stats.events_processed(),
         "jetstream_connected": stats.jetstream_connected(),
+        "pds_firehose_enabled": state.cfg.pds_firehose_enabled,
+        "pds_firehose_connected": stats.pds_connected(),
+        "pds_firehose_frames": stats.pds_frames_processed(),
+        "pds_firehose_seq": stats.pds_last_seq(),
     }))
 }
 

crates/appview/tests/common/mod.rs

@@ -29,11 +29,20 @@
 use at_crypto::ecdsa::P256Keypair;
 use at_crypto::jwt::{issue_jwt, JwtClaims};
 
-/// Audience the PDS stamps into access tokens. Not validated by
-/// `verify_jwt` today (`validate_aud = false`), but minting a token
-/// that differs from the real thing would make this helper a poor
-/// stand-in for the client.
-const APPVIEW_AUD: &str = "did:web:appview.maarcadetweet.local";
+/// Audience the PDS stamps into access tokens — and, since the
+/// audience check landed, the value the AppView insists on: its own
+/// service DID, derived from `APPVIEW_PUBLIC_URL`. A token minted with
+/// anything else is rejected as `TokenInvalid`, which is exactly what
+/// we want a wrong value here to look like.
+///
+/// Derived the same way `AppConfig::appview_did()` does it, from the
+/// same environment variable, so this helper can't drift from the
+/// service it's impersonating the PDS for.
+fn appview_aud() -> String {
+    let url = std::env::var("APPVIEW_PUBLIC_URL")
+        .unwrap_or_else(|_| "http://127.0.0.1:2584".to_string());
+    at_shared::config::did_web_from_url(&url)
+}
 
 /// The scope the AppView insists on. A token with any other scope —
 /// `com.atproto.refresh`, say — is rejected with `TokenInvalid`.
@@ -156,7 +165,7 @@ pub fn mint_access_jwt(
         &JwtClaims {
             iss: "did:web:test".into(),
             sub: did.to_string(),
-            aud: APPVIEW_AUD.into(),
+            aud: appview_aud(),
             iat: now - 1,
             exp: now + ttl_secs,
             jti: None,

crates/appview/tests/pds_firehose_integration.rs

@@ -0,0 +1,444 @@
+//! End-to-end tests for the local PDS firehose consumer.
+//!
+//! These are the only tests that put *real* PDS bytes through
+//! [`appview::pds_firehose`]: everything else in the module's own
+//! `#[cfg(test)]` section builds frames from a hand-written DAG-CBOR
+//! encoder, which proves the decoder matches our reading of the
+//! contract but not that the PDS writes what we think it writes.
+//!
+//! Fail-open, like every other suite in this directory. Each test
+//! prints a notice and returns successfully when a precondition is
+//! missing:
+//!
+//!  - the PDS isn't running on `:2583`;
+//!  - `DATABASE_URL_APPVIEW` is unset or the database is unreachable;
+//!  - **`com.atproto.sync.subscribeRepos` does not exist yet.** The
+//!    endpoint is being built in `crates/pds-server` in parallel with
+//!    this consumer. Until it lands, the WebSocket upgrade fails and
+//!    these tests skip with a message saying so — they are not proof of
+//!    anything while that line appears in the output.
+//!
+//! What they cover once the endpoint is live:
+//!
+//!  - `frame_from_the_local_pds_indexes_a_post` — subscribe, create a
+//!    record over `com.atproto.repo.createRecord`, and drive the frame
+//!    the PDS emits through the real decode → CAR → indexer path,
+//!    asserting the row lands in `posts`.
+//!  - `replaying_the_same_frame_changes_nothing` — the same frame
+//!    applied twice leaves exactly one row, which is what makes the
+//!    overlap with the `/internal/ingest-commit` push safe.
+//!  - `car_reader_parses_a_real_repo_export` — the AppView's CAR reader
+//!    against a CAR the PDS's *writer* produced (`getRepo`).
+//!  - `healthz_reports_the_firehose_state` — the running AppView's
+//!    probe carries the new fields.
+
+use futures::StreamExt;
+use serde_json::{json, Value};
+use sqlx::PgPool;
+use std::time::Duration;
+use tokio_tungstenite::tungstenite::Message;
+
+use appview::pds_firehose::{self, Frame};
+
+const PDS_URL: &str = "http://127.0.0.1:2583";
+const PDS_WS: &str = "ws://127.0.0.1:2583";
+
+fn appview_url() -> String {
+    std::env::var("APPVIEW_TEST_URL").unwrap_or_else(|_| "http://127.0.0.1:2584".to_string())
+}
+
+fn client() -> reqwest::Client {
+    reqwest::Client::builder()
+        .timeout(Duration::from_secs(10))
+        .build()
+        .unwrap()
+}
+
+async fn service_up(c: &reqwest::Client, base: &str) -> bool {
+    for _ in 0..12 {
+        if let Ok(r) = c.get(format!("{base}/healthz")).send().await {
+            if r.status().is_success() {
+                return true;
+            }
+        }
+        tokio::time::sleep(Duration::from_millis(250)).await;
+    }
+    false
+}
+
+async fn appview_db() -> Option<PgPool> {
+    let url = std::env::var("DATABASE_URL_APPVIEW").ok()?;
+    match tokio::time::timeout(Duration::from_secs(2), PgPool::connect(&url)).await {
+        Ok(Ok(pool)) => Some(pool),
+        _ => None,
+    }
+}
+
+struct Account {
+    did: String,
+    access_jwt: String,
+}
+
+async fn create_account(c: &reqwest::Client) -> Option<Account> {
+    let handle = format!("fh_{}.maarcadetweet.local", uuid::Uuid::new_v4().simple());
+    let r: Value = c
+        .post(format!("{PDS_URL}/xrpc/com.atproto.server.createAccount"))
+        .json(&json!({ "handle": handle, "password": "hunter2hunter2" }))
+        .send()
+        .await
+        .ok()?
+        .json()
+        .await
+        .ok()?;
+    Some(Account {
+        did: r["did"].as_str()?.to_string(),
+        access_jwt: r["access_jwt"].as_str()?.to_string(),
+    })
+}
+
+async fn create_post(c: &reqwest::Client, acc: &Account, text: &str) -> Option<String> {
+    let r: Value = c
+        .post(format!("{PDS_URL}/xrpc/com.atproto.repo.createRecord"))
+        .bearer_auth(&acc.access_jwt)
+        .json(&json!({
+            "repo": acc.did,
+            "collection": "app.twi.post",
+            "record": { "text": text, "createdAt": "2026-09-10T12:00:00Z" },
+        }))
+        .send()
+        .await
+        .ok()?
+        .json()
+        .await
+        .ok()?;
+    r["uri"].as_str().map(str::to_string)
+}
+
+type Ws = tokio_tungstenite::WebSocketStream<
+    tokio_tungstenite::MaybeTlsStream<tokio::net::TcpStream>,
+>;
+
+/// Subscribe to the PDS firehose, or `None` if the endpoint isn't there
+/// yet (see the module docs).
+async fn subscribe() -> Option<Ws> {
+    let url = pds_firehose::subscribe_url(PDS_WS, None);
+    match tokio::time::timeout(Duration::from_secs(5), tokio_tungstenite::connect_async(&url)).await
+    {
+        Ok(Ok((ws, _))) => Some(ws),
+        Ok(Err(e)) => {
+            eprintln!(
+                "cannot subscribe to {url}: {e} — the PDS endpoint com.atproto.sync.\
+                 subscribeRepos is probably not implemented yet; skipping"
+            );
+            None
+        }
+        Err(_) => {
+            eprintln!("timed out connecting to {url}; skipping");
+            None
+        }
+    }
+}
+
+/// Read frames until one is a `#commit` for `did`, or the deadline
+/// passes. `#info` frames along the way are tolerated (a fresh
+/// subscription may legitimately be told its cursor is outdated).
+async fn next_commit_for(
+    ws: &mut Ws,
+    did: &str,
+    timeout: Duration,
+) -> Option<pds_firehose::CommitFrame> {
+    let deadline = tokio::time::Instant::now() + timeout;
+    loop {
+        let remaining = deadline.saturating_duration_since(tokio::time::Instant::now());
+        if remaining.is_zero() {
+            return None;
+        }
+        let msg = match tokio::time::timeout(remaining, ws.next()).await {
+            Ok(Some(Ok(m))) => m,
+            Ok(Some(Err(e))) => {
+                eprintln!("firehose read error: {e}");
+                return None;
+            }
+            Ok(None) | Err(_) => return None,
+        };
+        let Message::Binary(bytes) = msg else { continue };
+        match pds_firehose::decode_frame(&bytes) {
+            Ok(Frame::Commit(commit)) if commit.repo == did => return Some(*commit),
+            Ok(Frame::Commit(_)) => continue,
+            Ok(Frame::Info { name, .. }) => {
+                eprintln!("firehose #info: {name}");
+                continue;
+            }
+            Ok(Frame::Error { error, message }) => {
+                eprintln!("firehose error frame: {error} {message:?}");
+                return None;
+            }
+            Ok(Frame::Other { .. }) => continue,
+            Err(e) => {
+                // A frame we cannot decode is a contract failure worth
+                // failing the test over — but only once we know the
+                // endpoint exists, which we do by this point.
+                panic!("could not decode a real PDS firehose frame: {e:#}");
+            }
+        }
+    }
+}
+
+/// Everything a live test needs, or `None` with a printed reason.
+async fn ready() -> Option<(reqwest::Client, PgPool, Account, Ws)> {
+    let c = client();
+    if !service_up(&c, PDS_URL).await {
+        eprintln!("pds not running on {PDS_URL}, skipping");
+        return None;
+    }
+    let Some(db) = appview_db().await else {
+        eprintln!("DATABASE_URL_APPVIEW unset or unreachable, skipping");
+        return None;
+    };
+    let Some(acc) = create_account(&c).await else {
+        eprintln!("could not create a PDS account, skipping");
+        return None;
+    };
+    // Subscribe *before* writing anything, so the commit we are about
+    // to make is guaranteed to fall inside the subscription window.
+    let ws = subscribe().await?;
+    Some((c, db, acc, ws))
+}
+
+#[tokio::test]
+async fn frame_from_the_local_pds_indexes_a_post() {
+    let Some((c, db, acc, mut ws)) = ready().await else {
+        return;
+    };
+    let text = format!("firehose e2e {}", uuid::Uuid::new_v4().simple());
+    let Some(uri) = create_post(&c, &acc, &text).await else {
+        eprintln!("createRecord failed, skipping");
+        return;
+    };
+
+    let Some(commit) = next_commit_for(&mut ws, &acc.did, Duration::from_secs(15)).await else {
+        eprintln!("no #commit frame for {} arrived in time, skipping", acc.did);
+        return;
+    };
+
+    // The frame itself must carry what the contract promises.
+    assert!(commit.seq > 0, "seq must be a positive sequence number");
+    assert_eq!(commit.repo, acc.did);
+    assert!(!commit.rev.is_empty(), "commit frames carry a rev");
+    assert!(
+        !commit.blocks.is_empty(),
+        "a create commit must inline its record block"
+    );
+    let create = commit
+        .ops
+        .iter()
+        .find(|op| op.action == "create" && op.collection() == Some("app.twi.post"))
+        .expect("the frame must contain the post create op");
+    assert!(create.cid.is_some(), "a create op carries the record CID");
+
+    // The blocks field must be a CAR our reader understands, and the
+    // op's CID must resolve inside it.
+    let car = appview::car::parse(&commit.blocks).expect("blocks must be a readable CAR v1");
+    assert!(
+        car.block_map().contains_key(&create.cid.unwrap()),
+        "the record block must be present in the CAR"
+    );
+
+    // And the whole path — frame → CAR → indexer — must land the row.
+    let events = pds_firehose::events_from_frame(&commit).expect("events");
+    assert!(
+        events
+            .iter()
+            .any(|e| e.commit.as_ref().unwrap()["record"]["text"] == json!(text)),
+        "the decoded record must carry the text we posted"
+    );
+
+    // Remove whatever the AppView's own push path already wrote, so the
+    // assertion below is about *this* code applying *this* frame.
+    sqlx::query("DELETE FROM posts WHERE uri = $1")
+        .bind(&uri)
+        .execute(&db)
+        .await
+        .unwrap();
+
+    pds_firehose::apply_frame(&db, &commit)
+        .await
+        .expect("apply_frame");
+
+    let stored: Option<String> = sqlx::query_scalar("SELECT text FROM posts WHERE uri = $1")
+        .bind(&uri)
+        .fetch_optional(&db)
+        .await
+        .unwrap();
+    assert_eq!(
+        stored.as_deref(),
+        Some(text.as_str()),
+        "the firehose frame must index the post at {uri}"
+    );
+
+    sqlx::query("DELETE FROM posts WHERE uri = $1")
+        .bind(&uri)
+        .execute(&db)
+        .await
+        .unwrap();
+}
+
+#[tokio::test]
+async fn replaying_the_same_frame_changes_nothing() {
+    let Some((c, db, acc, mut ws)) = ready().await else {
+        return;
+    };
+    let text = format!("firehose replay {}", uuid::Uuid::new_v4().simple());
+    let Some(uri) = create_post(&c, &acc, &text).await else {
+        eprintln!("createRecord failed, skipping");
+        return;
+    };
+    let Some(commit) = next_commit_for(&mut ws, &acc.did, Duration::from_secs(15)).await else {
+        eprintln!("no #commit frame for {} arrived in time, skipping", acc.did);
+        return;
+    };
+
+    // Three applications: the push already ran, then the firehose, then
+    // a post-restart replay of the same seq.
+    for _ in 0..3 {
+        pds_firehose::apply_frame(&db, &commit)
+            .await
+            .expect("apply_frame");
+    }
+
+    let rows: i64 = sqlx::query_scalar("SELECT count(*) FROM posts WHERE uri = $1")
+        .bind(&uri)
+        .fetch_one(&db)
+        .await
+        .unwrap();
+    assert_eq!(rows, 1, "replay must not duplicate {uri}");
+
+    sqlx::query("DELETE FROM posts WHERE uri = $1")
+        .bind(&uri)
+        .execute(&db)
+        .await
+        .unwrap();
+}
+
+#[tokio::test]
+async fn car_reader_parses_a_real_repo_export() {
+    // This one needs no firehose: `getRepo` has always served a CAR
+    // produced by the PDS's own writer, which is exactly the encoder
+    // the firehose's `blocks` field reuses.
+    let c = client();
+    if !service_up(&c, PDS_URL).await {
+        eprintln!("pds not running on {PDS_URL}, skipping");
+        return;
+    }
+    let Some(acc) = create_account(&c).await else {
+        eprintln!("could not create a PDS account, skipping");
+        return;
+    };
+    if create_post(&c, &acc, "car reader fixture").await.is_none() {
+        eprintln!("createRecord failed, skipping");
+        return;
+    }
+
+    let resp = c
+        .get(format!("{PDS_URL}/xrpc/com.atproto.sync.getRepo"))
+        .query(&[("did", acc.did.as_str())])
+        .send()
+        .await
+        .unwrap();
+    if !resp.status().is_success() {
+        eprintln!("getRepo returned {}, skipping", resp.status());
+        return;
+    }
+    let bytes = resp.bytes().await.unwrap();
+    let car = appview::car::parse(&bytes).expect("getRepo must return a readable CAR v1");
+    assert_eq!(car.header.version, 1);
+    assert!(
+        !car.blocks.is_empty(),
+        "a repo with one record has blocks (commit + MST + record)"
+    );
+    // Every block must hash to the CID the file declares — the strongest
+    // available statement that the reader's section framing is right.
+    appview::car::verify_block_cids(&car).expect("block CIDs must verify");
+}
+
+#[tokio::test]
+async fn healthz_reports_the_firehose_state() {
+    let c = client();
+    if !service_up(&c, &appview_url()).await {
+        eprintln!("appview not running, skipping");
+        return;
+    }
+    let body: Value = c
+        .get(format!("{}/healthz", appview_url()))
+        .send()
+        .await
+        .unwrap()
+        .json()
+        .await
+        .unwrap();
+
+    // A binary built before this feature has none of these keys; say so
+    // rather than failing, because "restart the AppView" is the fix.
+    let Some(enabled) = body.get("pds_firehose_enabled").and_then(Value::as_bool) else {
+        eprintln!(
+            "the running AppView predates the PDS firehose (no pds_firehose_enabled \
+             in /healthz) — rebuild and restart it; skipping"
+        );
+        return;
+    };
+    assert!(
+        body.get("pds_firehose_connected")
+            .and_then(Value::as_bool)
+            .is_some(),
+        "/healthz must report pds_firehose_connected: {body}"
+    );
+    assert!(
+        body.get("pds_firehose_seq").and_then(Value::as_i64).is_some(),
+        "/healthz must report pds_firehose_seq: {body}"
+    );
+
+    if !enabled {
+        eprintln!("PDS_FIREHOSE_ENABLED=false on the running AppView; nothing more to check");
+        return;
+    }
+    if !body["pds_firehose_connected"].as_bool().unwrap() {
+        eprintln!(
+            "the AppView is not connected to the PDS firehose — expected while \
+             com.atproto.sync.subscribeRepos is still being implemented; skipping \
+             the live-consumption check"
+        );
+        return;
+    }
+
+    // Connected: a new record must move the sequence number the AppView
+    // reports, which is the end-to-end proof that the *service* (not
+    // just this test process) consumes the stream.
+    if !service_up(&c, PDS_URL).await {
+        eprintln!("pds not running, skipping the live-consumption check");
+        return;
+    }
+    let Some(acc) = create_account(&c).await else {
+        eprintln!("could not create a PDS account, skipping");
+        return;
+    };
+    let before = body["pds_firehose_seq"].as_i64().unwrap_or(0);
+    if create_post(&c, &acc, "healthz seq probe").await.is_none() {
+        eprintln!("createRecord failed, skipping");
+        return;
+    }
+    for _ in 0..40 {
+        tokio::time::sleep(Duration::from_millis(250)).await;
+        let now: Value = c
+            .get(format!("{}/healthz", appview_url()))
+            .send()
+            .await
+            .unwrap()
+            .json()
+            .await
+            .unwrap();
+        if now["pds_firehose_seq"].as_i64().unwrap_or(0) > before {
+            return; // consumed
+        }
+    }
+    panic!("the AppView reports the firehose connected but its seq never advanced");
+}

crates/at-shared/src/config.rs

@@ -18,6 +18,21 @@ fn default_auth_required() -> bool {
     true
 }
 
+/// Default for `PDS_FIREHOSE_ENABLED`.
+///
+/// `true` — the AppView consumes the local PDS's
+/// `com.atproto.sync.subscribeRepos` stream. That stream is the only
+/// *guaranteed* path for a local user's own records: the fast
+/// `POST /internal/ingest-commit` push is best effort, and the public
+/// Jetstream never sees this PDS, so a lost push means a permanently
+/// missing post. Defaulting to on means an operator who never heard of
+/// the variable gets the durable behaviour; switching it off is the
+/// explicit choice (e.g. a PDS too old to serve the endpoint, or a
+/// second AppView instance that should not double-index).
+fn default_pds_firehose_enabled() -> bool {
+    true
+}
+
 #[derive(Debug, Clone, Deserialize)]
 pub struct AppConfig {
     pub pds_host: String,
@@ -78,6 +93,12 @@ pub struct AppConfig {
     /// `APPVIEW_CORS_ORIGINS=tauri://localhost,http://127.0.0.1:1430`
     #[serde(default)]
     pub appview_cors_origins: Vec<String>,
+    /// Whether the AppView subscribes to the local PDS firehose
+    /// (`com.atproto.sync.subscribeRepos` on
+    /// [`AppConfig::pds_base_url`]). Default `true` — see
+    /// [`default_pds_firehose_enabled`] for why.
+    #[serde(default = "default_pds_firehose_enabled")]
+    pub pds_firehose_enabled: bool,
 }
 
 impl AppConfig {
@@ -122,6 +143,10 @@ impl AppConfig {
                 .ok()
                 .map(|s| parse_csv_env(&s))
                 .unwrap_or_default(),
+            pds_firehose_enabled: std::env::var("PDS_FIREHOSE_ENABLED")
+                .ok()
+                .map(|s| parse_bool_env(&s))
+                .unwrap_or_else(default_pds_firehose_enabled),
         })
     }
 
@@ -138,6 +163,19 @@ impl AppConfig {
         did_web_from_url(&self.pds_public_url)
     }
 
+    /// The AppView's own service DID, derived from `APPVIEW_PUBLIC_URL`.
+    ///
+    /// Also one derivation, two consumers: the PDS stamps it into the
+    /// `aud` of every access token it issues, and the AppView checks
+    /// incoming tokens against it. A token minted for a *different*
+    /// AppView must not be usable here — that's the whole point of an
+    /// audience — so both sides have to agree on the spelling, and the
+    /// only way to guarantee that is to compute it the same way from
+    /// the same configuration.
+    pub fn appview_did(&self) -> String {
+        did_web_from_url(&self.appview_public_url)
+    }
+
     /// Base URL the AppView uses to reach the PDS.
     ///
     /// `PDS_INTERNAL_URL` when set (the cluster-internal hostname),

crates/pds-server/Cargo.toml

@@ -16,7 +16,7 @@ path = "src/main.rs"
 [dependencies]
 tokio = { workspace = true }
 dotenvy = { workspace = true }
-axum = { workspace = true }
+axum = { workspace = true, features = ["ws"] }
 tower = { workspace = true }
 tower-http = { workspace = true }
 serde = { workspace = true }
@@ -39,6 +39,7 @@ hex = { workspace = true }
 rand = { workspace = true }
 uuid = { workspace = true }
 bytes = { workspace = true }
+futures = { workspace = true }
 cid = { workspace = true }
 k256 = { workspace = true }
 p256 = { workspace = true }
@@ -56,3 +57,6 @@ sha2 = { workspace = true }
 hex = { workspace = true }
 sqlx = { workspace = true }
 at-crypto = { workspace = true }
+futures = { workspace = true }
+tokio-tungstenite = { workspace = true }
+chrono = { workspace = true }

crates/pds-server/src/car.rs

@@ -17,79 +17,51 @@
 //! The header is `{ version: 1, roots: [CID, ...] }` encoded as DAG-CBOR. In
 //! DAG-CBOR CID links carry the IANA-registered CBOR tag `42`, which the
 //! `ciborium` crate does not emit for `cid::Cid` (it uses serde newtype-struct
-//! tagging instead). We hand-encode the header bytes to keep the file
-//! spec-compliant: a `Map(2)` with text keys `"version"` and `"roots"`, an
-//! unsigned int `1` for the version, and a tagged byte string for each root
-//! CID.
+//! tagging instead). We hand-encode the header bytes through the shared
+//! primitives in [`crate::dag_cbor`]: a `Map(2)` with text keys `"version"`
+//! and `"roots"`, an unsigned int `1` for the version, and a tagged byte
+//! string for each root CID.
+//!
+//! One documented deviation from the DAG-CBOR spec lives in
+//! [`encode_header`] — the root CIDs are tagged but not identity-prefixed.
+//! See the note there; the firehose frames in [`crate::firehose`] do it the
+//! spec-correct way via [`crate::dag_cbor::write_link`].
 //!
 //! Per the spec, CAR v1 stores the raw CID bytes (varint version + codec +
 //! multihash) prefixed to every block, with a leading varint giving the total
 //! length of the section (CID + block).
 
+use crate::dag_cbor::{read_head, write_bytes, write_head, write_text};
 use anyhow::Result;
 use cid::Cid;
 
-/// Encode an unsigned CBOR head (major type in upper 3 bits) with a value.
-///
-/// Supports values up to `u32::MAX` which is more than enough for any realistic
-/// header or array length.
-fn cbor_head(out: &mut Vec<u8>, major: u8, n: u64) {
-    let m = (major & 0x07) << 5;
-    if n < 24 {
-        out.push(m | n as u8);
-    } else if n < 0x100 {
-        out.push(m | 24);
-        out.push(n as u8);
-    } else if n < 0x10000 {
-        out.push(m | 25);
-        out.push((n >> 8) as u8);
-        out.push(n as u8);
-    } else if n < 0x100_0000 {
-        out.push(m | 26);
-        out.push((n >> 16) as u8);
-        out.push((n >> 8) as u8);
-        out.push(n as u8);
-    } else {
-        out.push(m | 27);
-        out.push((n >> 24) as u8);
-        out.push((n >> 16) as u8);
-        out.push((n >> 8) as u8);
-        out.push(n as u8);
-    }
-}
-
-/// Append a CBOR text string.
-fn cbor_text(out: &mut Vec<u8>, s: &str) {
-    cbor_head(out, 3, s.len() as u64);
-    out.extend_from_slice(s.as_bytes());
-}
-
-/// Append a CBOR byte string.
-fn cbor_bytes(out: &mut Vec<u8>, b: &[u8]) {
-    cbor_head(out, 2, b.len() as u64);
-    out.extend_from_slice(b);
-}
-
-/// Append a CBOR tag wrapping the following value.
-fn cbor_tag(out: &mut Vec<u8>, tag: u64) {
-    cbor_head(out, 6, tag);
-}
-
 /// Encode the CAR v1 DAG-CBOR header `{ version: 1, roots: [CID, ...] }`.
 ///
-/// CIDs are encoded as `tag(42) + bytes(<raw-cid-bytes>)` per the DAG-CBOR
-/// spec. This is the canonical IPLD CID-link form.
+/// ### Known deviation: the root CIDs carry no identity prefix
+///
+/// A spec-conformant DAG-CBOR CID link is `tag(42)` wrapping a byte string of
+/// `0x00 || <binary CID>` — the `0x00` being the multibase identity prefix.
+///
+/// Older builds of this server omitted that byte and tagged the bare CID,
+/// which no conformant CAR reader can follow: it reads the first byte as the
+/// CID version and gives up. Since the header is not content-addressed —
+/// nothing hashes it, and no CID anywhere depends on its bytes — fixing it
+/// changes only what goes out on the wire, never an identifier. So it is
+/// fixed, via [`crate::dag_cbor::write_link`], the same writer the firehose
+/// frames use.
+///
+/// [`decode_header`] accepts both spellings, so a CAR captured from an older
+/// build still parses.
 pub fn encode_header(roots: &[Cid]) -> Vec<u8> {
     let mut out = Vec::new();
     // Map(2): { "version": 1, "roots": [...] }
-    cbor_head(&mut out, 5, 2);
-    cbor_text(&mut out, "version");
-    cbor_head(&mut out, 0, 1);
-    cbor_text(&mut out, "roots");
-    cbor_head(&mut out, 4, roots.len() as u64);
+    write_head(&mut out, 5, 2);
+    write_text(&mut out, "version");
+    write_head(&mut out, 0, 1);
+    write_text(&mut out, "roots");
+    write_head(&mut out, 4, roots.len() as u64);
     for cid in roots {
-        cbor_tag(&mut out, 42);
-        cbor_bytes(&mut out, &cid.to_bytes());
+        crate::dag_cbor::write_link(&mut out, cid);
     }
     out
 }
@@ -261,43 +233,83 @@ fn read_section(section: &[u8]) -> Result<(Cid, Vec<u8>)> {
     Ok((cid, data))
 }
 
+/// Decode the CAR header written by [`encode_header`].
+///
+/// Structural only, and deliberately *not* routed through
+/// [`crate::dag_cbor::decode`]: that decoder enforces the `0x00` multibase
+/// identity prefix on every tag-42 link, which our own header does not carry
+/// (see the deviation note on [`encode_header`]). It does share the CBOR head
+/// reader with it, so there is exactly one implementation of that.
 #[allow(dead_code)]
 fn decode_header(bytes: &[u8]) -> Result<CarHeader> {
-    // The header is a tiny DAG-CBOR map. We decode only the structure we emit.
     let mut p = 0usize;
-    let (n_items, consumed) = read_head_and_uint(bytes, p, 5)?;
-    p += consumed;
+    let (major, n_items, next) = read_head(bytes, p)?;
+    if major != 5 {
+        anyhow::bail!("CAR header must be a CBOR map, got major type {major}");
+    }
     if n_items != 2 {
         anyhow::bail!("CAR header must have 2 keys, got {n_items}");
     }
+    p = next;
 
     let mut version: Option<u64> = None;
     let mut roots: Vec<Cid> = Vec::new();
 
     for _ in 0..2 {
-        let (key, consumed) = read_head_and_text(bytes, p)?;
-        p += consumed;
+        let (major, len, next) = read_head(bytes, p)?;
+        if major != 3 {
+            anyhow::bail!("CAR header key must be text, got major type {major}");
+        }
+        p = next;
+        if p + len as usize > bytes.len() {
+            anyhow::bail!("CAR header key exceeds header");
+        }
+        let key = std::str::from_utf8(&bytes[p..p + len as usize])
+            .map_err(|e| anyhow::anyhow!("invalid UTF-8 in CAR header key: {e}"))?
+            .to_string();
+        p += len as usize;
+
         match key.as_str() {
             "version" => {
-                let (v, c) = read_head_and_uint(bytes, p, 0)?;
-                p += c;
+                let (major, v, next) = read_head(bytes, p)?;
+                if major != 0 {
+                    anyhow::bail!("CAR header `version` must be an unsigned int");
+                }
+                p = next;
                 version = Some(v);
             }
             "roots" => {
-                let (n_roots, c) = read_head_and_uint(bytes, p, 4)?;
-                p += c;
+                let (major, n_roots, next) = read_head(bytes, p)?;
+                if major != 4 {
+                    anyhow::bail!("CAR header `roots` must be an array");
+                }
+                p = next;
                 for _ in 0..n_roots {
-                    // tag(42)
-                    let (_, c) = read_head_and_uint(bytes, p, 6)?;
-                    p += c;
-                    // bytes
-                    let (n, c) = read_head_and_uint(bytes, p, 2)?;
-                    p += c;
+                    let (major, tag, next) = read_head(bytes, p)?;
+                    if major != 6 || tag != 42 {
+                        anyhow::bail!("CAR root must be CBOR tag 42, got major {major} tag {tag}");
+                    }
+                    p = next;
+                    let (major, n, next) = read_head(bytes, p)?;
+                    if major != 2 {
+                        anyhow::bail!("CAR root CID must be a byte string");
+                    }
+                    p = next;
                     if p + n as usize > bytes.len() {
                         anyhow::bail!("CAR root CID bytes exceed header");
                     }
-                    let cid_bytes = &bytes[p..p + n as usize];
-                    let cid = Cid::read_bytes(cid_bytes)
+                    // Tolerate both spellings: the conformant
+                    // `0x00 || cid` this server writes today, and the bare
+                    // CID older builds wrote (see `encode_header`). A real
+                    // CID never starts with 0x00 — that byte position holds
+                    // the version varint, and version 0 does not exist — so
+                    // stripping it is unambiguous, not a guess.
+                    let raw = &bytes[p..p + n as usize];
+                    let raw = match raw.first() {
+                        Some(0x00) => &raw[1..],
+                        _ => raw,
+                    };
+                    let cid = Cid::read_bytes(raw)
                         .map_err(|e| anyhow::anyhow!("invalid root CID bytes: {e}"))?;
                     p += n as usize;
                     roots.push(cid);
@@ -313,92 +325,54 @@ fn decode_header(bytes: &[u8]) -> Result<CarHeader> {
     })
 }
 
-/// Read a CBOR head (single byte for value < 24, otherwise head + varint
-/// extension) and decode its value. Validates that the major type is
-/// `expected_major`. Returns the decoded value and the number of bytes
-/// consumed (head + any extension).
-#[allow(dead_code)]
-fn read_head_and_uint(
-    bytes: &[u8],
-    offset: usize,
-    expected_major: u8,
-) -> Result<(u64, usize)> {
-    if offset >= bytes.len() {
-        anyhow::bail!("CBOR read past end of input");
-    }
-    let first = bytes[offset];
-    let major = first >> 5;
-    if major != expected_major {
-        anyhow::bail!(
-            "expected CBOR major {}, got {}",
-            expected_major,
-            major
-        );
-    }
-    let low = first & 0x1f;
-    let (value, extra) = match low {
-        0..=23 => (low as u64, 0usize),
-        24 => {
-            if offset + 2 > bytes.len() {
-                anyhow::bail!("truncated CBOR uint8");
-            }
-            (bytes[offset + 1] as u64, 1)
-        }
-        25 => {
-            if offset + 3 > bytes.len() {
-                anyhow::bail!("truncated CBOR uint16");
-            }
-            (
-                ((bytes[offset + 1] as u64) << 8) | (bytes[offset + 2] as u64),
-                2,
-            )
-        }
-        26 => {
-            if offset + 5 > bytes.len() {
-                anyhow::bail!("truncated CBOR uint32");
-            }
-            let n = ((bytes[offset + 1] as u64) << 24)
-                | ((bytes[offset + 2] as u64) << 16)
-                | ((bytes[offset + 3] as u64) << 8)
-                | (bytes[offset + 4] as u64);
-            (n, 4)
-        }
-        27 => {
-            if offset + 9 > bytes.len() {
-                anyhow::bail!("truncated CBOR uint64");
-            }
-            let mut n = 0u64;
-            for i in 0..8 {
-                n = (n << 8) | (bytes[offset + 1 + i] as u64);
-            }
-            (n, 8)
-        }
-        other => anyhow::bail!("unsupported CBOR uint tag {other}"),
-    };
-    Ok((value, 1 + extra))
-}
-
-/// Read a CBOR text string with major type 3, returning the string and the
-/// total number of bytes consumed.
-#[allow(dead_code)]
-fn read_head_and_text(
-    bytes: &[u8],
-    offset: usize,
-) -> Result<(String, usize)> {
-    let (n, c) = read_head_and_uint(bytes, offset, 3)?;
-    if offset + c + n as usize > bytes.len() {
-        anyhow::bail!("CBOR text string exceeds buffer");
-    }
-    let s = std::str::from_utf8(&bytes[offset + c..offset + c + n as usize])
-        .map_err(|e| anyhow::anyhow!("invalid UTF-8 in CBOR text: {e}"))?;
-    Ok((s.to_string(), c + n as usize))
-}
-
 #[cfg(test)]
 mod tests {
     use super::*;
     use at_crypto::cid::cid_for_cbor;
 
+    /// The identity prefix is what makes a root readable by a stock CAR
+    /// library, so assert on the bytes rather than only on the round trip
+    /// through our own parser — which would pass either way.
+    #[test]
+    fn header_roots_carry_the_identity_prefix() {
+        let c = cid_for_cbor(b"a").unwrap();
+        let bytes = encode_header(&[c]);
+        let raw = c.to_bytes();
+        // tag(42) is 0xD8 0x2A, then a byte string one longer than the CID,
+        // whose first content byte is the 0x00 multibase identity prefix.
+        let tag_at = bytes
+            .windows(2)
+            .position(|w| w == [0xD8, 0x2A])
+            .expect("tag(42) must be present");
+        let after_tag = &bytes[tag_at + 2..];
+        let (major, len, next) = read_head(after_tag, 0).unwrap();
+        assert_eq!(major, 2, "a link wraps a byte string");
+        assert_eq!(len as usize, raw.len() + 1, "one byte longer than the CID");
+        assert_eq!(after_tag[next], 0x00, "multibase identity prefix");
+        assert_eq!(&after_tag[next + 1..next + 1 + raw.len()], &raw[..]);
+    }
+
+    /// A CAR captured from an older build tagged the bare CID. Those bytes
+    /// must keep parsing — otherwise upgrading the server would strand
+    /// anything that stored a repo export.
+    #[test]
+    fn header_without_identity_prefix_still_parses() {
+        let c = cid_for_cbor(b"legacy").unwrap();
+        // Hand-build the old shape: map(2), "version", 1, "roots", [tag(42)
+        // bytes(<bare cid>)].
+        let mut old = Vec::new();
+        write_head(&mut old, 5, 2);
+        write_text(&mut old, "version");
+        write_head(&mut old, 0, 1);
+        write_text(&mut old, "roots");
+        write_head(&mut old, 4, 1);
+        write_head(&mut old, 6, 42);
+        write_bytes(&mut old, &c.to_bytes());
+
+        let h = decode_header(&old).unwrap();
+        assert_eq!(h.roots, vec![c], "legacy root must still decode");
+    }
+
     #[test]
     fn header_encodes_cids_with_tag_42() {
         let c1 = cid_for_cbor(b"a").unwrap();

crates/pds-server/src/dag_cbor.rs

@@ -0,0 +1,557 @@
+//! A small, self-contained DAG-CBOR encoder + decoder.
+//!
+//! Why this exists
+//!
+//! `ciborium` (the CBOR crate the rest of the workspace uses) speaks plain
+//! CBOR through serde. It has no notion of an IPLD *CID link*, which DAG-CBOR
+//! encodes as the IANA-registered tag `42` wrapping a byte string whose first
+//! byte is the multibase-identity prefix `0x00` followed by the binary CID.
+//! Serde has no representation for a CBOR tag, so `ciborium` silently encodes
+//! `cid::Cid` as a newtype struct instead — which is *not* DAG-CBOR and which
+//! no atproto consumer can read.
+//!
+//! `car.rs` already hand-rolled the handful of primitives needed for the CAR
+//! v1 header (`{version, roots: [<tag 42 link>]}`). The firehose frames need
+//! exactly the same primitives plus a couple more (arrays of maps, nullable
+//! links, byte strings, i64). Rather than write the encoder twice, both
+//! callers now go through this module.
+//!
+//! ## What "correct DAG-CBOR" means here
+//!
+//! * Map keys are text strings, sorted in the DAG-CBOR canonical order:
+//!   shorter keys first, then bytewise-ascending within a length. This is the
+//!   ordering `@ipld/dag-cbor` (and therefore the reference atproto
+//!   implementation) emits, so a frame produced here is byte-identical to one
+//!   produced by a Typescript PDS for the same logical value.
+//! * Integers use the shortest possible head. Byte and text strings likewise.
+//! * A CID link is `tag(42) || bytes(0x00 || <cid.to_bytes()>)`. The leading
+//!   `0x00` is the multibase identity prefix mandated by the DAG-CBOR spec for
+//!   binary CIDs; forgetting it is the single most common interop bug, so the
+//!   decoder asserts on it too.
+//! * Floats are deliberately *not* supported. DAG-CBOR permits them but
+//!   nothing in this codebase emits one, and accepting them would mean
+//!   deciding on a canonical float encoding we would never exercise.
+//!
+//! ## Scope
+//!
+//! This is not a general CBOR library. It handles definite-length items only
+//! (DAG-CBOR forbids indefinite lengths anyway) and rejects everything it does
+//! not understand rather than guessing. It is used for the *envelope* of
+//! things — CAR headers and firehose frames — never for repo blocks, which
+//! are produced by `at_repo`/`at_crypto` with their own (see
+//! `crate::firehose`) conventions.
+
+use anyhow::{anyhow, bail, Result};
+use cid::Cid;
+
+/// The IPLD CID-link tag. See <https://ipld.io/specs/codecs/dag-cbor/spec/>.
+pub const CID_TAG: u64 = 42;
+
+/// A decoded (or to-be-encoded) DAG-CBOR value.
+///
+/// `Link` is kept distinct from `Bytes` so a round-trip through
+/// [`decode`] / [`encode`] preserves the tag rather than flattening a link
+/// into an anonymous byte string.
+#[derive(Debug, Clone, PartialEq)]
+pub enum Value {
+    Null,
+    Bool(bool),
+    /// Signed integer. CBOR major types 0 (non-negative) and 1 (negative).
+    Int(i64),
+    Bytes(Vec<u8>),
+    Text(String),
+    Array(Vec<Value>),
+    /// Map with text keys. Insertion order is irrelevant — [`encode`] sorts
+    /// into the canonical DAG-CBOR order, and [`decode`] returns keys in the
+    /// order they appeared on the wire.
+    Map(Vec<(String, Value)>),
+    /// An IPLD CID link — `tag(42)` wrapping the identity-prefixed CID bytes.
+    Link(Cid),
+}
+
+impl Value {
+    /// Convenience: build a `Map` from an iterator of pairs.
+    pub fn map<I, K>(pairs: I) -> Value
+    where
+        I: IntoIterator<Item = (K, Value)>,
+        K: Into<String>,
+    {
+        Value::Map(pairs.into_iter().map(|(k, v)| (k.into(), v)).collect())
+    }
+
+    /// Convenience: a text value from anything string-ish.
+    pub fn text(s: impl Into<String>) -> Value {
+        Value::Text(s.into())
+    }
+
+    /// Look up a key in a `Map`. Returns `None` for a non-map or a
+    /// missing key. Used by the tests and by frame consumers.
+    pub fn get(&self, key: &str) -> Option<&Value> {
+        match self {
+            Value::Map(entries) => entries.iter().find(|(k, _)| k == key).map(|(_, v)| v),
+            _ => None,
+        }
+    }
+
+    pub fn as_i64(&self) -> Option<i64> {
+        match self {
+            Value::Int(i) => Some(*i),
+            _ => None,
+        }
+    }
+
+    pub fn as_str(&self) -> Option<&str> {
+        match self {
+            Value::Text(s) => Some(s.as_str()),
+            _ => None,
+        }
+    }
+
+    pub fn as_bool(&self) -> Option<bool> {
+        match self {
+            Value::Bool(b) => Some(*b),
+            _ => None,
+        }
+    }
+
+    pub fn as_bytes(&self) -> Option<&[u8]> {
+        match self {
+            Value::Bytes(b) => Some(b.as_slice()),
+            _ => None,
+        }
+    }
+
+    pub fn as_array(&self) -> Option<&[Value]> {
+        match self {
+            Value::Array(a) => Some(a.as_slice()),
+            _ => None,
+        }
+    }
+
+    pub fn as_link(&self) -> Option<&Cid> {
+        match self {
+            Value::Link(c) => Some(c),
+            _ => None,
+        }
+    }
+
+    pub fn is_null(&self) -> bool {
+        matches!(self, Value::Null)
+    }
+}
+
+// -- encoding ---------------------------------------------------------------
+
+/// Write a CBOR head: the 3-bit major type plus the argument, using the
+/// shortest encoding that fits.
+///
+/// Public because `car.rs` builds its length-prefixed sections around the same
+/// primitive and there is no reason to have two copies.
+pub fn write_head(out: &mut Vec<u8>, major: u8, n: u64) {
+    let m = (major & 0x07) << 5;
+    if n < 24 {
+        out.push(m | n as u8);
+    } else if n <= u8::MAX as u64 {
+        out.push(m | 24);
+        out.push(n as u8);
+    } else if n <= u16::MAX as u64 {
+        out.push(m | 25);
+        out.extend_from_slice(&(n as u16).to_be_bytes());
+    } else if n <= u32::MAX as u64 {
+        out.push(m | 26);
+        out.extend_from_slice(&(n as u32).to_be_bytes());
+    } else {
+        out.push(m | 27);
+        out.extend_from_slice(&n.to_be_bytes());
+    }
+}
+
+/// Append a CBOR text string (major type 3).
+pub fn write_text(out: &mut Vec<u8>, s: &str) {
+    write_head(out, 3, s.len() as u64);
+    out.extend_from_slice(s.as_bytes());
+}
+
+/// Append a CBOR byte string (major type 2).
+pub fn write_bytes(out: &mut Vec<u8>, b: &[u8]) {
+    write_head(out, 2, b.len() as u64);
+    out.extend_from_slice(b);
+}
+
+/// Append a CID as a DAG-CBOR link: `tag(42) || bytes(0x00 || cid)`.
+///
+/// The `0x00` is the multibase identity prefix. Binary CIDs inside DAG-CBOR
+/// always carry it; the textual form (`bafy…`) never does.
+pub fn write_link(out: &mut Vec<u8>, cid: &Cid) {
+    write_head(out, 6, CID_TAG);
+    let raw = cid.to_bytes();
+    let mut prefixed = Vec::with_capacity(raw.len() + 1);
+    prefixed.push(0x00);
+    prefixed.extend_from_slice(&raw);
+    write_bytes(out, &prefixed);
+}
+
+/// DAG-CBOR canonical map-key order: shorter keys sort first; equal-length
+/// keys sort bytewise ascending.
+///
+/// This is RFC 7049's "canonical CBOR" rule, which DAG-CBOR inherited and
+/// which `@ipld/dag-cbor` implements. (RFC 8949 later switched the *core*
+/// deterministic profile to plain bytewise ordering, but DAG-CBOR did not
+/// follow — using 8949's rule here would produce frames that differ from the
+/// reference implementation's for keys like `"op"` vs `"t"`.)
+fn canonical_key_cmp(a: &str, b: &str) -> std::cmp::Ordering {
+    a.len()
+        .cmp(&b.len())
+        .then_with(|| a.as_bytes().cmp(b.as_bytes()))
+}
+
+/// Encode a value into `out`.
+pub fn encode_into(out: &mut Vec<u8>, value: &Value) {
+    match value {
+        Value::Null => out.push(0xF6),
+        Value::Bool(false) => out.push(0xF4),
+        Value::Bool(true) => out.push(0xF5),
+        Value::Int(i) => {
+            if *i >= 0 {
+                write_head(out, 0, *i as u64);
+            } else {
+                // CBOR major type 1 stores -1-n, so n = -(i+1). Computed on
+                // i64 via `i128` to stay correct at `i64::MIN`, where
+                // `-(i + 1)` would overflow.
+                let n = (-((*i as i128) + 1)) as u64;
+                write_head(out, 1, n);
+            }
+        }
+        Value::Bytes(b) => write_bytes(out, b),
+        Value::Text(s) => write_text(out, s),
+        Value::Array(items) => {
+            write_head(out, 4, items.len() as u64);
+            for item in items {
+                encode_into(out, item);
+            }
+        }
+        Value::Map(entries) => {
+            let mut sorted: Vec<&(String, Value)> = entries.iter().collect();
+            sorted.sort_by(|a, b| canonical_key_cmp(&a.0, &b.0));
+            write_head(out, 5, sorted.len() as u64);
+            for (k, v) in sorted {
+                write_text(out, k);
+                encode_into(out, v);
+            }
+        }
+        Value::Link(cid) => write_link(out, cid),
+    }
+}
+
+/// Encode a value to a fresh `Vec<u8>`.
+///
+/// `#[allow(dead_code)]`: the server itself always appends into an existing
+/// buffer via [`encode_into`] (a frame is two values in one allocation), so
+/// this convenience wrapper is exercised only by the tests that assert on
+/// exact byte sequences. It is kept because the decoder half needs a matching
+/// encoder half to be testable at all.
+#[allow(dead_code)]
+pub fn encode(value: &Value) -> Vec<u8> {
+    let mut out = Vec::new();
+    encode_into(&mut out, value);
+    out
+}
+
+// -- decoding ---------------------------------------------------------------
+
+// -- Why a decoder lives in a server that only encodes ----------------------
+//
+// The PDS never parses a firehose frame in production — it writes them. The
+// decoder exists so the frame *contract* can be tested from the outside: a
+// unit test that only checks "the encoder produced these bytes" locks in
+// whatever the encoder happens to do, including its bugs. Decoding the bytes
+// back and asserting on the structure is what actually verifies that a tag-42
+// link is a link and not a byte string, that the two frame halves are
+// separable, and that a `null` `since` is `null` rather than absent.
+//
+// The integration test uses it for the same reason from the client side, and
+// `crates/appview` builds its consumer against the same shape.
+
+/// Decode exactly one value, requiring it to consume the whole input.
+#[allow(dead_code)]
+pub fn decode(bytes: &[u8]) -> Result<Value> {
+    let (v, used) = decode_one(bytes)?;
+    if used != bytes.len() {
+        bail!(
+            "trailing bytes after DAG-CBOR value: consumed {used} of {}",
+            bytes.len()
+        );
+    }
+    Ok(v)
+}
+
+/// Decode one value from the front of `bytes`, returning it along with the
+/// number of bytes consumed.
+///
+/// This is the entry point the firehose frame reader needs: a frame is two
+/// concatenated DAG-CBOR values (header then body) with no length prefix
+/// between them, so the only way to find the body is to decode the header and
+/// see where it ended.
+#[allow(dead_code)]
+pub fn decode_one(bytes: &[u8]) -> Result<(Value, usize)> {
+    decode_at(bytes, 0).map(|(v, end)| (v, end))
+}
+
+#[allow(dead_code)]
+fn decode_at(bytes: &[u8], offset: usize) -> Result<(Value, usize)> {
+    let (major, arg, mut p) = read_head(bytes, offset)?;
+    match major {
+        0 => {
+            let i = i64::try_from(arg).map_err(|_| anyhow!("CBOR uint {arg} exceeds i64"))?;
+            Ok((Value::Int(i), p))
+        }
+        1 => {
+            let v = -(arg as i128) - 1;
+            let i = i64::try_from(v).map_err(|_| anyhow!("CBOR nint {v} exceeds i64"))?;
+            Ok((Value::Int(i), p))
+        }
+        2 => {
+            let end = p + arg as usize;
+            if end > bytes.len() {
+                bail!("CBOR byte string exceeds input");
+            }
+            Ok((Value::Bytes(bytes[p..end].to_vec()), end))
+        }
+        3 => {
+            let end = p + arg as usize;
+            if end > bytes.len() {
+                bail!("CBOR text string exceeds input");
+            }
+            let s = std::str::from_utf8(&bytes[p..end])
+                .map_err(|e| anyhow!("invalid UTF-8 in CBOR text: {e}"))?;
+            Ok((Value::Text(s.to_string()), end))
+        }
+        4 => {
+            let mut items = Vec::with_capacity(arg.min(1024) as usize);
+            for _ in 0..arg {
+                let (v, next) = decode_at(bytes, p)?;
+                items.push(v);
+                p = next;
+            }
+            Ok((Value::Array(items), p))
+        }
+        5 => {
+            let mut entries = Vec::with_capacity(arg.min(1024) as usize);
+            for _ in 0..arg {
+                let (k, next) = decode_at(bytes, p)?;
+                p = next;
+                let key = match k {
+                    Value::Text(s) => s,
+                    other => bail!("DAG-CBOR map keys must be text, got {other:?}"),
+                };
+                let (v, next) = decode_at(bytes, p)?;
+                p = next;
+                entries.push((key, v));
+            }
+            Ok((Value::Map(entries), p))
+        }
+        6 => {
+            if arg != CID_TAG {
+                bail!("unsupported CBOR tag {arg}; DAG-CBOR allows only 42");
+            }
+            let (inner, end) = decode_at(bytes, p)?;
+            let raw = match inner {
+                Value::Bytes(b) => b,
+                other => bail!("CBOR tag 42 must wrap a byte string, got {other:?}"),
+            };
+            let stripped = raw
+                .split_first()
+                .filter(|(first, _)| **first == 0x00)
+                .map(|(_, rest)| rest)
+                .ok_or_else(|| {
+                    anyhow!("CID link missing the 0x00 multibase identity prefix")
+                })?;
+            let cid = Cid::read_bytes(stripped)
+                .map_err(|e| anyhow!("invalid CID inside tag 42: {e}"))?;
+            Ok((Value::Link(cid), end))
+        }
+        7 => match arg {
+            20 => Ok((Value::Bool(false), p)),
+            21 => Ok((Value::Bool(true), p)),
+            22 => Ok((Value::Null, p)),
+            // 23 is `undefined`, 25/26/27 are floats. DAG-CBOR forbids
+            // `undefined`; floats are out of scope (see the module header).
+            other => bail!("unsupported CBOR simple/float value {other}"),
+        },
+        other => bail!("unsupported CBOR major type {other}"),
+    }
+}
+
+/// Read a CBOR head at `offset`, returning `(major, argument, next_offset)`.
+///
+/// Public so `car.rs` can drive its own (deliberately non-conformant, see
+/// there) header parser off the same primitive instead of keeping a second
+/// copy.
+pub fn read_head(bytes: &[u8], offset: usize) -> Result<(u8, u64, usize)> {
+    let first = *bytes
+        .get(offset)
+        .ok_or_else(|| anyhow!("CBOR read past end of input at {offset}"))?;
+    let major = first >> 5;
+    let low = first & 0x1f;
+    let (arg, extra) = match low {
+        0..=23 => (low as u64, 0usize),
+        24 => (read_uint(bytes, offset + 1, 1)?, 1),
+        25 => (read_uint(bytes, offset + 1, 2)?, 2),
+        26 => (read_uint(bytes, offset + 1, 4)?, 4),
+        27 => (read_uint(bytes, offset + 1, 8)?, 8),
+        // 28..=30 are reserved; 31 is the indefinite-length marker, which
+        // DAG-CBOR forbids outright.
+        other => bail!("invalid or indefinite CBOR head 0x{other:02x}"),
+    };
+    Ok((major, arg, offset + 1 + extra))
+}
+
+fn read_uint(bytes: &[u8], offset: usize, width: usize) -> Result<u64> {
+    if offset + width > bytes.len() {
+        bail!("truncated CBOR integer of width {width}");
+    }
+    let mut n: u64 = 0;
+    for b in &bytes[offset..offset + width] {
+        n = (n << 8) | *b as u64;
+    }
+    Ok(n)
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use at_crypto::cid::cid_for_cbor;
+
+    fn round_trip(v: Value) {
+        let bytes = encode(&v);
+        let back = decode(&bytes).expect("decode");
+        // Maps come back in canonical (encoded) order, so compare the
+        // re-encoding rather than the structure for map-bearing values.
+        assert_eq!(encode(&back), bytes, "re-encode must be stable");
+    }
+
+    #[test]
+    fn scalars_round_trip() {
+        for v in [
+            Value::Null,
+            Value::Bool(true),
+            Value::Bool(false),
+            Value::Int(0),
+            Value::Int(23),
+            Value::Int(24),
+            Value::Int(255),
+            Value::Int(256),
+            Value::Int(65_535),
+            Value::Int(65_536),
+            Value::Int(i64::MAX),
+            Value::Int(-1),
+            Value::Int(-24),
+            Value::Int(-1000),
+            Value::Int(i64::MIN),
+            Value::Text("hello".into()),
+            Value::Text(String::new()),
+            Value::Bytes(vec![1, 2, 3]),
+            Value::Bytes(Vec::new()),
+        ] {
+            let bytes = encode(&v);
+            assert_eq!(decode(&bytes).unwrap(), v, "round trip of {v:?}");
+        }
+    }
+
+    #[test]
+    fn int_heads_are_shortest_form() {
+        assert_eq!(encode(&Value::Int(1)), vec![0x01]);
+        assert_eq!(encode(&Value::Int(24)), vec![0x18, 24]);
+        assert_eq!(encode(&Value::Int(-1)), vec![0x20]);
+        assert_eq!(encode(&Value::Int(-25)), vec![0x38, 24]);
+    }
+
+    #[test]
+    fn cid_link_carries_tag_42_and_identity_prefix() {
+        let cid = cid_for_cbor(b"a block").unwrap();
+        let bytes = encode(&Value::Link(cid));
+        // 0xD8 0x2A == tag(42) in two-byte form.
+        assert_eq!(&bytes[0..2], &[0xD8, 0x2A]);
+        // Then a byte string whose first content byte is the 0x00 prefix.
+        let (_major, len, p) = read_head(&bytes, 2).unwrap();
+        assert_eq!(len as usize, cid.to_bytes().len() + 1);
+        assert_eq!(bytes[p], 0x00);
+        assert_eq!(decode(&bytes).unwrap(), Value::Link(cid));
+    }
+
+    #[test]
+    fn link_without_identity_prefix_is_rejected() {
+        let cid = cid_for_cbor(b"x").unwrap();
+        let mut bytes = Vec::new();
+        write_head(&mut bytes, 6, CID_TAG);
+        // Deliberately omit the leading 0x00.
+        write_bytes(&mut bytes, &cid.to_bytes());
+        let e = decode(&bytes).unwrap_err().to_string();
+        assert!(e.contains("identity prefix"), "got: {e}");
+    }
+
+    #[test]
+    fn map_keys_are_sorted_length_first() {
+        // The firehose header is exactly this shape, and the reference
+        // implementation emits `t` before `op` because it is shorter.
+        let v = Value::map([("op", Value::Int(1)), ("t", Value::text("#commit"))]);
+        let bytes = encode(&v);
+        assert_eq!(bytes[0], 0xA2, "map(2)");
+        assert_eq!(bytes[1], 0x61, "text(1)");
+        assert_eq!(bytes[2], b't');
+        // …and `op` follows after the "#commit" value.
+        let decoded = decode(&bytes).unwrap();
+        match &decoded {
+            Value::Map(entries) => {
+                assert_eq!(entries[0].0, "t");
+                assert_eq!(entries[1].0, "op");
+            }
+            other => panic!("expected map, got {other:?}"),
+        }
+    }
+
+    #[test]
+    fn nested_structures_round_trip() {
+        let cid = cid_for_cbor(b"nested").unwrap();
+        round_trip(Value::map([
+            ("seq", Value::Int(42)),
+            ("commit", Value::Link(cid)),
+            (
+                "ops",
+                Value::Array(vec![Value::map([
+                    ("action", Value::text("create")),
+                    ("path", Value::text("app.twi.post/3l")),
+                    ("cid", Value::Link(cid)),
+                ])]),
+            ),
+            ("blobs", Value::Array(vec![])),
+            ("since", Value::Null),
+        ]));
+    }
+
+    #[test]
+    fn decode_one_stops_at_the_value_boundary() {
+        // Two concatenated values — exactly how a firehose frame is laid out.
+        let mut buf = encode(&Value::map([("op", Value::Int(1))]));
+        let header_len = buf.len();
+        buf.extend_from_slice(&encode(&Value::map([("seq", Value::Int(7))])));
+        let (header, used) = decode_one(&buf).unwrap();
+        assert_eq!(used, header_len);
+        assert_eq!(header.get("op").and_then(Value::as_i64), Some(1));
+        let body = decode(&buf[used..]).unwrap();
+        assert_eq!(body.get("seq").and_then(Value::as_i64), Some(7));
+    }
+
+    #[test]
+    fn indefinite_length_is_rejected() {
+        // 0x9F == array(*) — legal CBOR, illegal DAG-CBOR.
+        assert!(decode(&[0x9F, 0x01, 0xFF]).is_err());
+    }
+
+    #[test]
+    fn trailing_bytes_are_rejected() {
+        let mut bytes = encode(&Value::Int(1));
+        bytes.push(0x01);
+        assert!(decode(&bytes).is_err());
+    }
+}

crates/pds-server/src/firehose.rs

@@ -0,0 +1,946 @@
+//! `com.atproto.sync.subscribeRepos` — the event log, the frame codec and the
+//! in-process broadcast channel.
+//!
+//! # What this is for
+//!
+//! Before this module the PDS emitted no firehose. A locally created record
+//! reached the AppView through exactly one channel: the best-effort HTTP push
+//! in [`crate::appview_push`], a detached `tokio::spawn` whose failure branch
+//! logs "jetstream will replay". For records that only exist on this PDS
+//! there *is* no Jetstream to replay them, so a lost push meant the post was
+//! never indexed and nothing would ever notice. The firehose replaces that
+//! hope with a durable, ordered log: every repo write appends one row inside
+//! the same transaction as the commit, and a consumer can ask for everything
+//! after a cursor at any later time.
+//!
+//! # Frame format
+//!
+//! One WebSocket **binary** message is two DAG-CBOR values written back to
+//! back with nothing between them — a header, then a body. There is no length
+//! prefix; the reader decodes the header and continues the body at the offset
+//! where the header ended (that is what [`crate::dag_cbor::decode_one`] is
+//! for).
+//!
+//! ```text
+//! regular:  {"op": 1, "t": "#commit"}  {"seq": …, "repo": …, …}
+//!           {"op": 1, "t": "#info"}    {"name": …, "message": …}
+//! error:    {"op": -1}                 {"error": "<Name>", "message": "<Text>"}
+//! ```
+//!
+//! The `#commit` body carries:
+//!
+//! | field     | type                                                    |
+//! |-----------|---------------------------------------------------------|
+//! | `seq`     | int — the cursor value for this event                   |
+//! | `rebase`  | bool — always `false` (we never rebase a repo)          |
+//! | `tooBig`  | bool — always `false` (see the size note below)         |
+//! | `repo`    | text — the DID                                          |
+//! | `commit`  | **CID link (tag 42)** — the new commit block            |
+//! | `rev`     | text — the new commit's revision                        |
+//! | `since`   | text or null — the previous commit's revision           |
+//! | `blocks`  | byte string — a CAR v1 file, commit block as root       |
+//! | `ops`     | array of `{action, path, cid}`; `cid` is a link or null  |
+//! | `blobs`   | array — always empty (blob refs live inside the record) |
+//! | `time`    | text — RFC 3339, when the event was appended            |
+//!
+//! Map keys are emitted in DAG-CBOR canonical order (shortest first, then
+//! bytewise), so the bytes match what a reference atproto implementation
+//! would produce for the same logical value.
+//!
+//! # Deliberate deviation from the atproto spec
+//!
+//! **The frame envelope is conformant. The blocks inside `blocks` are not.**
+//!
+//! This repository encodes CIDs *inside* commit blocks as CBOR text strings
+//! rather than as DAG-CBOR links with tag 42 — see `at_repo::commit` and
+//! `at_crypto::signing::sign_dag_cbor`. That convention predates this module
+//! and is load-bearing: the block bytes determine every CID in the system,
+//! including the `did:plc:` derivation, so changing it would re-address every
+//! repo in the database. It is explicitly out of scope here.
+//!
+//! The consequence, stated plainly: a foreign atproto consumer can connect,
+//! parse every frame, read `seq` / `repo` / `rev` / `ops`, and follow the
+//! stream. It will then fail when it tries to *validate* the payload — the
+//! CAR in `blocks` parses fine and the block CIDs hash correctly over their
+//! own bytes, but decoding a commit block as DAG-CBOR yields `"prev"` and
+//! `"data"` as strings where the spec demands links, and MST traversal
+//! against a stock implementation will not work. The AppView in this
+//! workspace reads the frames with the same conventions this crate writes
+//! them, which is why it works there.
+//!
+//! `tooBig` is therefore always `false`: it exists so a producer can say "the
+//! diff was too large, go fetch the repo yourself", and we never make that
+//! call — every commit here is one record change, and its CAR is small.
+//!
+//! # Lagging consumers
+//!
+//! The broadcast channel is bounded ([`FIREHOSE_CHANNEL_CAPACITY`]). A
+//! consumer that reads slower than the PDS writes will eventually be lapped,
+//! and `tokio::sync::broadcast` reports that as `RecvError::Lagged(n)`.
+//!
+//! The write path must never wait on a reader, so the channel cannot be made
+//! blocking. When a reader lags we send it an `#info` / `OutdatedCursor`
+//! frame and **fall back to the database replay** from the last sequence it
+//! actually received, then resume live. We do not disconnect it: the events
+//! are durable in `firehose_events`, so the fallback is lossless, whereas
+//! dropping the socket would force the client to reconnect and perform
+//! exactly the same replay after two extra round trips. The only thing a
+//! disconnect would buy is protection against a client that lags forever, and
+//! that is handled separately by capping consecutive recoveries
+//! ([`MAX_LAG_RECOVERIES`]) before closing with an error frame.
+//!
+//! # Retention
+//!
+//! Nothing prunes `firehose_events`. It grows by one row per repo write, each
+//! carrying the CAR of that commit's new blocks. That is unbounded, and this
+//! deployment has no retention job — an operator who wants one has to add it.
+//! Pruning is safe by design: the cursor handshake compares the requested
+//! cursor against the oldest surviving row and answers a too-old cursor with
+//! `#info` / `OutdatedCursor` followed by a replay from the oldest row that
+//! still exists, rather than pretending the gap is not there.
+
+use crate::car::CarWriter;
+use crate::dag_cbor::{encode_into, Value};
+use anyhow::{anyhow, Result};
+use at_crypto::cid::cid_from_multihash_bytes;
+use chrono::{DateTime, SecondsFormat, Utc};
+use cid::Cid;
+use serde_json::json;
+use std::sync::Arc;
+use tokio::sync::broadcast;
+
+/// How many events the live broadcast channel buffers per subscriber before
+/// the slowest one starts reporting `Lagged`.
+///
+/// 1024 is chosen so a consumer can stall for the length of a garbage
+/// collection or a slow network write without falling back to the database,
+/// while the memory ceiling stays bounded: the channel holds `Arc`s, so the
+/// cost is one CAR blob per queued event, shared across all subscribers.
+pub const FIREHOSE_CHANNEL_CAPACITY: usize = 1024;
+
+/// How many times in a row a single connection may be rescued from a lag
+/// before we give up and close it. A client that cannot keep up even with a
+/// database replay in between is not going to start; at that point the honest
+/// answer is an error frame rather than an endless catch-up loop that burns
+/// queries on its behalf.
+pub const MAX_LAG_RECOVERIES: u32 = 5;
+
+/// Rows returned per replay query. Bounded so a client reconnecting with
+/// `cursor=0` after a long uptime streams the backlog in chunks instead of
+/// materialising the whole table (and every CAR in it) in memory at once.
+pub const REPLAY_PAGE_SIZE: i64 = 200;
+
+/// The advisory-lock key that serialises `firehose_events` INSERTs. Any
+/// constant works as long as every writer uses the same one; this is
+/// `"fhose"` read as ASCII, which makes it recognisable in `pg_locks`.
+///
+/// What it costs: the lock is global, not per-repo, so the tail of every
+/// repo write — INSERT plus COMMIT — is serialised across all accounts.
+/// That is deliberate (a per-repo lock would order each repo's events but
+/// not the shared `seq` a consumer paginates on), and it bounds write
+/// throughput to how fast Postgres can commit one small INSERT at a time.
+/// If that ever becomes the ceiling, the fix is a different sequence
+/// design — a per-repo cursor, or handing out `seq` from a single writer
+/// task — not a weaker lock: a gap in `seq` is silent data loss for every
+/// consumer replaying from a cursor.
+pub const FIREHOSE_ADVISORY_LOCK_KEY: i64 = 0x66_68_6f_73_65;
+
+// -- ops --------------------------------------------------------------------
+
+/// What a single repo operation did to one record.
+///
+/// `Update` exists separately from `Create` because a consumer that keeps a
+/// materialised view needs to know whether to insert or replace; the MST
+/// itself does not distinguish them, so the write path resolves it by looking
+/// the key up before writing.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum RepoOpAction {
+    Create,
+    Update,
+    Delete,
+}
+
+impl RepoOpAction {
+    pub fn as_str(self) -> &'static str {
+        match self {
+            RepoOpAction::Create => "create",
+            RepoOpAction::Update => "update",
+            RepoOpAction::Delete => "delete",
+        }
+    }
+
+    pub fn parse(s: &str) -> Result<Self> {
+        match s {
+            "create" => Ok(RepoOpAction::Create),
+            "update" => Ok(RepoOpAction::Update),
+            "delete" => Ok(RepoOpAction::Delete),
+            other => Err(anyhow!("unknown repo op action `{other}`")),
+        }
+    }
+}
+
+/// One entry of a commit frame's `ops` array.
+///
+/// `path` is `"<collection>/<rkey>"` — the MST key, not an `at://` URI. That
+/// is what the wire format specifies, and it is also what the MST is actually
+/// keyed by, so there is no reassembly step on either side.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct RepoOp {
+    pub action: RepoOpAction,
+    pub path: String,
+    /// The record value's CID for `create` / `update`; `None` for `delete`,
+    /// where there is no resulting value to point at.
+    pub cid: Option<Cid>,
+}
+
+impl RepoOp {
+    pub fn create(collection: &str, rkey: &str, cid: Cid) -> Self {
+        Self {
+            action: RepoOpAction::Create,
+            path: format!("{collection}/{rkey}"),
+            cid: Some(cid),
+        }
+    }
+
+    pub fn update(collection: &str, rkey: &str, cid: Cid) -> Self {
+        Self {
+            action: RepoOpAction::Update,
+            path: format!("{collection}/{rkey}"),
+            cid: Some(cid),
+        }
+    }
+
+    pub fn delete(collection: &str, rkey: &str) -> Self {
+        Self {
+            action: RepoOpAction::Delete,
+            path: format!("{collection}/{rkey}"),
+            cid: None,
+        }
+    }
+
+    /// Pick `create` or `update` from whether the key already existed.
+    pub fn put(collection: &str, rkey: &str, cid: Cid, existed: bool) -> Self {
+        if existed {
+            Self::update(collection, rkey, cid)
+        } else {
+            Self::create(collection, rkey, cid)
+        }
+    }
+
+    /// JSON shape stored in `firehose_events.ops`. The CID is a string here
+    /// (JSONB has no link type); it becomes a tag-42 link again on the wire.
+    pub fn to_json(&self) -> serde_json::Value {
+        json!({
+            "action": self.action.as_str(),
+            "path": self.path,
+            "cid": self.cid.map(|c| c.to_string()),
+        })
+    }
+
+    pub fn from_json(v: &serde_json::Value) -> Result<Self> {
+        let action = RepoOpAction::parse(
+            v.get("action")
+                .and_then(|a| a.as_str())
+                .ok_or_else(|| anyhow!("op missing `action`"))?,
+        )?;
+        let path = v
+            .get("path")
+            .and_then(|p| p.as_str())
+            .ok_or_else(|| anyhow!("op missing `path`"))?
+            .to_string();
+        let cid = match v.get("cid") {
+            None | Some(serde_json::Value::Null) => None,
+            Some(serde_json::Value::String(s)) => {
+                Some(s.parse::<Cid>().map_err(|e| anyhow!("op cid: {e}"))?)
+            }
+            Some(other) => return Err(anyhow!("op `cid` must be a string or null, got {other}")),
+        };
+        Ok(Self { action, path, cid })
+    }
+
+    fn to_dag_cbor(&self) -> Value {
+        Value::map([
+            ("action", Value::text(self.action.as_str())),
+            ("path", Value::text(&self.path)),
+            (
+                "cid",
+                match self.cid {
+                    Some(c) => Value::Link(c),
+                    None => Value::Null,
+                },
+            ),
+        ])
+    }
+}
+
+// -- events -----------------------------------------------------------------
+
+/// One row of `firehose_events`, ready to be framed.
+///
+/// Both the live path (built at write time) and the replay path (read back
+/// from Postgres) produce this exact struct, which is what makes a replayed
+/// frame byte-identical to the live one — including `time`, which comes from
+/// the stored `created_at` rather than from the clock at send time.
+#[derive(Debug, Clone)]
+pub struct FirehoseEvent {
+    pub seq: i64,
+    pub did: String,
+    pub rev: String,
+    pub since: Option<String>,
+    pub commit: Cid,
+    /// CAR v1 file: the commit block as root, plus the blocks this commit
+    /// newly created.
+    pub blocks: Vec<u8>,
+    pub ops: Vec<RepoOp>,
+    pub time: DateTime<Utc>,
+}
+
+impl FirehoseEvent {
+    /// The `#commit` body as a DAG-CBOR value.
+    pub fn to_body(&self) -> Value {
+        Value::map([
+            ("seq", Value::Int(self.seq)),
+            ("rebase", Value::Bool(false)),
+            ("tooBig", Value::Bool(false)),
+            ("repo", Value::text(&self.did)),
+            ("commit", Value::Link(self.commit)),
+            ("rev", Value::text(&self.rev)),
+            (
+                "since",
+                match &self.since {
+                    Some(s) => Value::text(s),
+                    None => Value::Null,
+                },
+            ),
+            ("blocks", Value::Bytes(self.blocks.clone())),
+            (
+                "ops",
+                Value::Array(self.ops.iter().map(RepoOp::to_dag_cbor).collect()),
+            ),
+            ("blobs", Value::Array(Vec::new())),
+            (
+                "time",
+                Value::text(self.time.to_rfc3339_opts(SecondsFormat::Micros, true)),
+            ),
+        ])
+    }
+
+    /// The full binary WebSocket payload: `#commit` header then body.
+    pub fn to_frame(&self) -> Vec<u8> {
+        frame(&header_value("#commit"), &self.to_body())
+    }
+}
+
+/// Build the CAR that goes into a commit event's `blocks` field.
+///
+/// Root is the commit block; the remaining entries are the blocks this commit
+/// newly wrote (MST nodes and record values). Blocks that already existed in
+/// the repo are deliberately left out — that is the whole point of a diff
+/// stream, and a consumer that needs an ancestor block asks
+/// `com.atproto.sync.getBlocks` for it.
+pub fn build_blocks_car(
+    commit_cid: Cid,
+    commit_block: &[u8],
+    new_blocks: &[(Cid, Vec<u8>)],
+) -> Vec<u8> {
+    let mut w = CarWriter::new();
+    w.append(commit_cid, commit_block);
+    for (cid, data) in new_blocks {
+        w.append(*cid, data);
+    }
+    w.finish(&[commit_cid])
+}
+
+// -- frame encoding ---------------------------------------------------------
+
+/// `{"op": 1, "t": "<t>"}` — the header of a regular frame.
+fn header_value(t: &str) -> Value {
+    Value::map([("op", Value::Int(1)), ("t", Value::text(t))])
+}
+
+/// Concatenate a header and a body into one binary WebSocket payload.
+fn frame(header: &Value, body: &Value) -> Vec<u8> {
+    let mut out = Vec::new();
+    encode_into(&mut out, header);
+    encode_into(&mut out, body);
+    out
+}
+
+/// An `#info` frame: `{"op":1,"t":"#info"}` + `{"name":…,"message":…}`.
+///
+/// Informational, not fatal — the stream continues after it. We send it when
+/// a cursor is older than anything we still have, and when a live subscriber
+/// lagged and is being put back on the database replay.
+pub fn encode_info_frame(name: &str, message: &str) -> Vec<u8> {
+    frame(
+        &header_value("#info"),
+        &Value::map([("name", Value::text(name)), ("message", Value::text(message))]),
+    )
+}
+
+/// An error frame: `{"op":-1}` + `{"error":…,"message":…}`.
+///
+/// Terminal — the server closes the socket right after sending it.
+pub fn encode_error_frame(error: &str, message: &str) -> Vec<u8> {
+    frame(
+        &Value::map([("op", Value::Int(-1))]),
+        &Value::map([
+            ("error", Value::text(error)),
+            ("message", Value::text(message)),
+        ]),
+    )
+}
+
+// -- broadcast --------------------------------------------------------------
+
+/// The in-process fan-out from the write path to connected subscribers.
+///
+/// Cloneable and cheap: it is a `broadcast::Sender` plus nothing. Events are
+/// wrapped in an `Arc` so a burst of subscribers does not multiply the CAR
+/// blobs.
+#[derive(Clone)]
+pub struct Firehose {
+    tx: broadcast::Sender<Arc<FirehoseEvent>>,
+}
+
+impl Default for Firehose {
+    fn default() -> Self {
+        Self::new()
+    }
+}
+
+impl Firehose {
+    pub fn new() -> Self {
+        let (tx, _rx) = broadcast::channel(FIREHOSE_CHANNEL_CAPACITY);
+        Self { tx }
+    }
+
+    /// Publish an event to every live subscriber.
+    ///
+    /// Never blocks and never fails in a way the caller must handle: with no
+    /// subscribers the send returns `Err`, which is the normal state of a PDS
+    /// nobody is watching. The event is already durable in Postgres by the
+    /// time we get here, so a dropped broadcast costs a consumer nothing
+    /// beyond having to replay by cursor.
+    pub fn publish(&self, event: FirehoseEvent) {
+        let _ = self.tx.send(Arc::new(event));
+    }
+
+    pub fn subscribe(&self) -> broadcast::Receiver<Arc<FirehoseEvent>> {
+        self.tx.subscribe()
+    }
+
+    /// Number of live subscribers. Used for logging / the healthz surface.
+    #[allow(dead_code)]
+    pub fn subscriber_count(&self) -> usize {
+        self.tx.receiver_count()
+    }
+}
+
+// -- persistence ------------------------------------------------------------
+
+/// Append one event inside an open transaction and return it with its
+/// assigned `seq` and `created_at`.
+///
+/// The advisory lock taken first is what makes the sequence usable as a
+/// cursor: without it two concurrent writers can be assigned seq 5 and 6 and
+/// commit in the other order, so a reader polling in between sees 6, records
+/// it as its cursor, and never learns about 5. Holding
+/// `pg_advisory_xact_lock` from just before the INSERT until COMMIT forces
+/// commit order to match seq order. It is taken *after* the per-repo
+/// `SELECT … FOR UPDATE` in [`crate::routes::helpers::apply_repo_write`], and
+/// every writer takes the two in that same order, so the pair cannot deadlock.
+pub async fn insert_event_in_tx(
+    tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
+    did: &str,
+    rev: &str,
+    since: Option<&str>,
+    commit: Cid,
+    blocks: Vec<u8>,
+    ops: Vec<RepoOp>,
+) -> Result<FirehoseEvent, sqlx::Error> {
+    sqlx::query("SELECT pg_advisory_xact_lock($1)")
+        .bind(FIREHOSE_ADVISORY_LOCK_KEY)
+        .execute(&mut **tx)
+        .await?;
+
+    let ops_json = serde_json::Value::Array(ops.iter().map(RepoOp::to_json).collect());
+
+    let (seq, created_at): (i64, DateTime<Utc>) = sqlx::query_as(
+        r#"INSERT INTO firehose_events (did, rev, since, commit_cid, blocks, ops)
+           VALUES ($1, $2, $3, $4, $5, $6)
+           RETURNING seq, created_at"#,
+    )
+    .bind(did)
+    .bind(rev)
+    .bind(since)
+    .bind(commit.to_bytes())
+    .bind(&blocks)
+    .bind(&ops_json)
+    .fetch_one(&mut **tx)
+    .await?;
+
+    Ok(FirehoseEvent {
+        seq,
+        did: did.to_string(),
+        rev: rev.to_string(),
+        since: since.map(|s| s.to_string()),
+        commit,
+        blocks,
+        ops,
+        time: created_at,
+    })
+}
+
+/// The `(min_seq, max_seq)` currently in the table, or `None` when it is
+/// empty. Used by the cursor handshake to tell "from the future" apart from
+/// "too old to still have".
+pub async fn seq_bounds(db: &sqlx::PgPool) -> Result<Option<(i64, i64)>, sqlx::Error> {
+    let row: (Option<i64>, Option<i64>) =
+        sqlx::query_as("SELECT MIN(seq), MAX(seq) FROM firehose_events")
+            .fetch_one(db)
+            .await?;
+    Ok(match row {
+        (Some(min), Some(max)) => Some((min, max)),
+        _ => None,
+    })
+}
+
+/// Read up to [`REPLAY_PAGE_SIZE`] events with `seq > after`, oldest first.
+pub async fn load_events_after(
+    db: &sqlx::PgPool,
+    after: i64,
+    limit: i64,
+) -> Result<Vec<FirehoseEvent>> {
+    let rows: Vec<(
+        i64,
+        String,
+        String,
+        Option<String>,
+        Vec<u8>,
+        Vec<u8>,
+        serde_json::Value,
+        DateTime<Utc>,
+    )> = sqlx::query_as(
+        r#"SELECT seq, did, rev, since, commit_cid, blocks, ops, created_at
+             FROM firehose_events
+            WHERE seq > $1
+            ORDER BY seq ASC
+            LIMIT $2"#,
+    )
+    .bind(after)
+    .bind(limit)
+    .fetch_all(db)
+    .await
+    .map_err(|e| anyhow!("firehose_events replay read: {e}"))?;
+
+    rows.into_iter()
+        .map(|(seq, did, rev, since, commit_cid, blocks, ops, created_at)| {
+            let commit = cid_from_multihash_bytes(&commit_cid)
+                .map_err(|e| anyhow!("firehose_events.commit_cid at seq {seq}: {e}"))?;
+            let ops = ops
+                .as_array()
+                .ok_or_else(|| anyhow!("firehose_events.ops at seq {seq} is not an array"))?
+                .iter()
+                .map(RepoOp::from_json)
+                .collect::<Result<Vec<_>>>()?;
+            Ok(FirehoseEvent {
+                seq,
+                did,
+                rev,
+                since,
+                commit,
+                blocks,
+                ops,
+                time: created_at,
+            })
+        })
+        .collect()
+}
+
+// -- cursor handshake -------------------------------------------------------
+
+/// What the connection handler should do with the cursor the client sent.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum CursorPlan {
+    /// No cursor: send live events only, nothing from the log.
+    LiveOnly,
+    /// Replay everything after `from`, then go live.
+    Replay { from: i64 },
+    /// The requested cursor predates the oldest row we still have. Warn with
+    /// an `#info` frame, then replay from `from` (the oldest surviving row
+    /// minus one) so the client at least gets everything that does exist.
+    OutdatedCursor { from: i64, message: String },
+    /// The cursor names an event that has not happened. This is a client bug
+    /// (or a cursor from a different server's log), and continuing would
+    /// silently strand it — so it is a terminal error frame.
+    FutureCursor { message: String },
+}
+
+/// Decide what to do with `cursor` given the log's current `(min, max)`.
+///
+/// Split out from the socket handler so the boundary conditions are testable
+/// without a database or a WebSocket.
+///
+/// Semantics of the cursor: it is the seq of the last event the client
+/// *already has*, so a replay yields `seq > cursor`. `cursor = 0` therefore
+/// means "everything", and `cursor = max` means "nothing yet, just go live" —
+/// which is a valid, empty replay rather than a future cursor.
+pub fn plan_cursor(cursor: Option<i64>, bounds: Option<(i64, i64)>) -> CursorPlan {
+    let cursor = match cursor {
+        None => return CursorPlan::LiveOnly,
+        Some(c) => c,
+    };
+    if cursor < 0 {
+        return CursorPlan::FutureCursor {
+            message: format!("cursor {cursor} is negative"),
+        };
+    }
+    let (min, max) = match bounds {
+        // An empty log accepts only cursor 0 ("give me everything, there is
+        // nothing"). Anything else refers to an event we never had.
+        None => {
+            return if cursor == 0 {
+                CursorPlan::Replay { from: 0 }
+            } else {
+                CursorPlan::FutureCursor {
+                    message: format!("cursor {cursor} is ahead of an empty log"),
+                }
+            };
+        }
+        Some(b) => b,
+    };
+    if cursor > max {
+        return CursorPlan::FutureCursor {
+            message: format!("cursor {cursor} is ahead of the latest event {max}"),
+        };
+    }
+    // `cursor >= min - 1` means the next event the client wants (cursor + 1)
+    // is still on disk. Below that, rows have been pruned and the client has
+    // a hole it can never fill.
+    if cursor < min - 1 {
+        return CursorPlan::OutdatedCursor {
+            from: min - 1,
+            message: format!(
+                "cursor {cursor} predates the oldest retained event {min}; \
+                 resuming from {min} — events {}..{} are gone",
+                cursor + 1,
+                min - 1
+            ),
+        };
+    }
+    CursorPlan::Replay { from: cursor }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use crate::dag_cbor::{decode, decode_one};
+    use at_crypto::cid::cid_for_cbor;
+
+    fn sample_event() -> FirehoseEvent {
+        let commit = cid_for_cbor(b"commit block").unwrap();
+        let value = cid_for_cbor(b"record value").unwrap();
+        FirehoseEvent {
+            seq: 7,
+            did: "did:plc:alice".into(),
+            rev: "3lxxxxxxxx2".into(),
+            since: Some("3lxxxxxxxx1".into()),
+            commit,
+            blocks: build_blocks_car(
+                commit,
+                b"commit block",
+                &[(value, b"record value".to_vec())],
+            ),
+            ops: vec![RepoOp::create("app.twi.post", "3lrkey", value)],
+            time: DateTime::parse_from_rfc3339("2026-09-10T12:00:00.123456Z")
+                .unwrap()
+                .with_timezone(&Utc),
+        }
+    }
+
+    // -- frame encoding ----------------------------------------------------
+
+    #[test]
+    fn commit_frame_header_then_body() {
+        let ev = sample_event();
+        let bytes = ev.to_frame();
+        let (header, used) = decode_one(&bytes).unwrap();
+        assert_eq!(header.get("op").and_then(Value::as_i64), Some(1));
+        assert_eq!(header.get("t").and_then(Value::as_str), Some("#commit"));
+        // Everything after the header is exactly one more value — no padding,
+        // no length prefix.
+        let body = decode(&bytes[used..]).unwrap();
+        assert_eq!(body.get("seq").and_then(Value::as_i64), Some(7));
+        assert_eq!(
+            body.get("repo").and_then(Value::as_str),
+            Some("did:plc:alice")
+        );
+        assert_eq!(body.get("rev").and_then(Value::as_str), Some("3lxxxxxxxx2"));
+        assert_eq!(
+            body.get("since").and_then(Value::as_str),
+            Some("3lxxxxxxxx1")
+        );
+        assert_eq!(body.get("rebase").and_then(Value::as_bool), Some(false));
+        assert_eq!(body.get("tooBig").and_then(Value::as_bool), Some(false));
+        assert!(body.get("blobs").unwrap().as_array().unwrap().is_empty());
+        assert_eq!(
+            body.get("time").and_then(Value::as_str),
+            Some("2026-09-10T12:00:00.123456Z")
+        );
+    }
+
+    /// The header's exact bytes, locked in. The AppView builds its reader
+    /// against this, and canonical key ordering means `t` precedes `op` —
+    /// which is easy to get wrong and produces a frame a strict DAG-CBOR
+    /// decoder rejects.
+    #[test]
+    fn commit_header_has_the_exact_expected_bytes() {
+        let bytes = sample_event().to_frame();
+        assert_eq!(
+            &bytes[..15],
+            &[
+                0xA2, // map(2)
+                0x61, b't', // text(1) "t"
+                0x67, b'#', b'c', b'o', b'm', b'm', b'i', b't', // text(7) "#commit"
+                0x62, b'o', b'p', // text(2) "op"
+                0x01, // 1
+            ]
+        );
+    }
+
+    #[test]
+    fn info_and_error_headers_have_the_exact_expected_bytes() {
+        let info = encode_info_frame("OutdatedCursor", "x");
+        assert_eq!(
+            &info[..12],
+            &[0xA2, 0x61, b't', 0x65, b'#', b'i', b'n', b'f', b'o', 0x62, b'o', b'p']
+        );
+        assert_eq!(info[12], 0x01);
+        // Error: map(1) { "op": -1 }. -1 is major type 1 with argument 0.
+        let err = encode_error_frame("FutureCursor", "x");
+        assert_eq!(&err[..5], &[0xA1, 0x62, b'o', b'p', 0x20]);
+    }
+
+    #[test]
+    fn commit_cid_is_a_tag_42_link_not_a_string() {
+        // The whole point of the hand-rolled encoder: `ciborium` would have
+        // written this as a string or a newtype struct.
+        let ev = sample_event();
+        let bytes = ev.to_frame();
+        let (_h, used) = decode_one(&bytes).unwrap();
+        let body = decode(&bytes[used..]).unwrap();
+        assert_eq!(body.get("commit").and_then(Value::as_link), Some(&ev.commit));
+        // Locate the tag bytes directly, to prove it is not the decoder being
+        // generous: 0xD8 0x2A is tag(42).
+        assert!(
+            bytes.windows(2).any(|w| w == [0xD8, 0x2A]),
+            "frame must contain a tag-42 head"
+        );
+    }
+
+    #[test]
+    fn ops_encode_action_path_and_link() {
+        let ev = sample_event();
+        let bytes = ev.to_frame();
+        let (_h, used) = decode_one(&bytes).unwrap();
+        let body = decode(&bytes[used..]).unwrap();
+        let ops = body.get("ops").unwrap().as_array().unwrap();
+        assert_eq!(ops.len(), 1);
+        assert_eq!(ops[0].get("action").and_then(Value::as_str), Some("create"));
+        assert_eq!(
+            ops[0].get("path").and_then(Value::as_str),
+            Some("app.twi.post/3lrkey")
+        );
+        assert!(ops[0].get("cid").unwrap().as_link().is_some());
+    }
+
+    #[test]
+    fn delete_op_has_a_null_cid() {
+        let mut ev = sample_event();
+        ev.ops = vec![RepoOp::delete("app.bsky.feed.like", "3lrkey")];
+        let bytes = ev.to_frame();
+        let (_h, used) = decode_one(&bytes).unwrap();
+        let body = decode(&bytes[used..]).unwrap();
+        let ops = body.get("ops").unwrap().as_array().unwrap();
+        assert_eq!(ops[0].get("action").and_then(Value::as_str), Some("delete"));
+        assert!(ops[0].get("cid").unwrap().is_null());
+    }
+
+    #[test]
+    fn first_commit_has_a_null_since() {
+        let mut ev = sample_event();
+        ev.since = None;
+        let bytes = ev.to_frame();
+        let (_h, used) = decode_one(&bytes).unwrap();
+        let body = decode(&bytes[used..]).unwrap();
+        assert!(body.get("since").unwrap().is_null());
+    }
+
+    #[test]
+    fn blocks_is_a_parsable_car_rooted_at_the_commit() {
+        let ev = sample_event();
+        let bytes = ev.to_frame();
+        let (_h, used) = decode_one(&bytes).unwrap();
+        let body = decode(&bytes[used..]).unwrap();
+        let car = body.get("blocks").and_then(Value::as_bytes).unwrap();
+        let (header, blocks) = crate::car::parse(car).unwrap();
+        assert_eq!(header.version, 1);
+        assert_eq!(header.roots, vec![ev.commit]);
+        assert_eq!(blocks.len(), 2, "commit block + one new record block");
+        assert_eq!(blocks[0].cid, ev.commit);
+        assert_eq!(blocks[0].data, b"commit block");
+    }
+
+    #[test]
+    fn info_frame_round_trips() {
+        let bytes = encode_info_frame("OutdatedCursor", "resuming from 12");
+        let (header, used) = decode_one(&bytes).unwrap();
+        assert_eq!(header.get("op").and_then(Value::as_i64), Some(1));
+        assert_eq!(header.get("t").and_then(Value::as_str), Some("#info"));
+        let body = decode(&bytes[used..]).unwrap();
+        assert_eq!(
+            body.get("name").and_then(Value::as_str),
+            Some("OutdatedCursor")
+        );
+        assert_eq!(
+            body.get("message").and_then(Value::as_str),
+            Some("resuming from 12")
+        );
+    }
+
+    #[test]
+    fn error_frame_uses_op_minus_one_and_carries_no_t() {
+        let bytes = encode_error_frame("FutureCursor", "cursor 99 is ahead");
+        let (header, used) = decode_one(&bytes).unwrap();
+        assert_eq!(header.get("op").and_then(Value::as_i64), Some(-1));
+        assert!(
+            header.get("t").is_none(),
+            "an error header carries op only"
+        );
+        let body = decode(&bytes[used..]).unwrap();
+        assert_eq!(
+            body.get("error").and_then(Value::as_str),
+            Some("FutureCursor")
+        );
+        assert_eq!(
+            body.get("message").and_then(Value::as_str),
+            Some("cursor 99 is ahead")
+        );
+    }
+
+    // -- ops derivation ----------------------------------------------------
+
+    #[test]
+    fn put_picks_create_or_update_from_prior_existence() {
+        let cid = cid_for_cbor(b"v").unwrap();
+        assert_eq!(
+            RepoOp::put("c", "r", cid, false).action,
+            RepoOpAction::Create
+        );
+        assert_eq!(
+            RepoOp::put("c", "r", cid, true).action,
+            RepoOpAction::Update
+        );
+    }
+
+    #[test]
+    fn op_path_is_collection_slash_rkey() {
+        let cid = cid_for_cbor(b"v").unwrap();
+        assert_eq!(
+            RepoOp::create("app.bsky.feed.repost", "3lk", cid).path,
+            "app.bsky.feed.repost/3lk"
+        );
+    }
+
+    #[test]
+    fn op_json_round_trips_through_the_jsonb_shape() {
+        let cid = cid_for_cbor(b"v").unwrap();
+        for op in [
+            RepoOp::create("app.twi.post", "a", cid),
+            RepoOp::update("app.bsky.actor.profile", "self", cid),
+            RepoOp::delete("app.bsky.graph.follow", "b"),
+        ] {
+            let back = RepoOp::from_json(&op.to_json()).unwrap();
+            assert_eq!(back, op);
+        }
+    }
+
+    #[test]
+    fn op_json_rejects_an_unknown_action() {
+        let v = json!({"action": "rebase", "path": "a/b", "cid": null});
+        assert!(RepoOp::from_json(&v).is_err());
+    }
+
+    // -- cursor edge cases -------------------------------------------------
+
+    #[test]
+    fn no_cursor_is_live_only() {
+        assert_eq!(plan_cursor(None, Some((1, 10))), CursorPlan::LiveOnly);
+        assert_eq!(plan_cursor(None, None), CursorPlan::LiveOnly);
+    }
+
+    #[test]
+    fn cursor_zero_replays_everything() {
+        assert_eq!(
+            plan_cursor(Some(0), Some((1, 10))),
+            CursorPlan::Replay { from: 0 }
+        );
+    }
+
+    #[test]
+    fn cursor_at_the_head_is_an_empty_replay_not_an_error() {
+        // The client is fully caught up. Replaying `seq > 10` yields nothing
+        // and it goes straight live — that must not be a FutureCursor.
+        assert_eq!(
+            plan_cursor(Some(10), Some((1, 10))),
+            CursorPlan::Replay { from: 10 }
+        );
+    }
+
+    #[test]
+    fn cursor_past_the_head_is_a_future_cursor() {
+        assert!(matches!(
+            plan_cursor(Some(11), Some((1, 10))),
+            CursorPlan::FutureCursor { .. }
+        ));
+    }
+
+    #[test]
+    fn negative_cursor_is_a_future_cursor() {
+        assert!(matches!(
+            plan_cursor(Some(-1), Some((1, 10))),
+            CursorPlan::FutureCursor { .. }
+        ));
+    }
+
+    #[test]
+    fn empty_log_accepts_zero_and_refuses_anything_else() {
+        assert_eq!(plan_cursor(Some(0), None), CursorPlan::Replay { from: 0 });
+        assert!(matches!(
+            plan_cursor(Some(1), None),
+            CursorPlan::FutureCursor { .. }
+        ));
+    }
+
+    #[test]
+    fn cursor_exactly_one_below_the_oldest_row_is_still_exact() {
+        // min = 5 means seq 5 is the oldest surviving event. A client whose
+        // cursor is 4 wants 5 next — nothing is missing.
+        assert_eq!(
+            plan_cursor(Some(4), Some((5, 10))),
+            CursorPlan::Replay { from: 4 }
+        );
+    }
+
+    #[test]
+    fn cursor_below_the_pruned_window_is_outdated() {
+        match plan_cursor(Some(2), Some((5, 10))) {
+            CursorPlan::OutdatedCursor { from, message } => {
+                assert_eq!(from, 4, "resume so the next delivered event is 5");
+                assert!(message.contains('5'), "message should name the gap: {message}");
+            }
+            other => panic!("expected OutdatedCursor, got {other:?}"),
+        }
+    }
+}

crates/pds-server/src/jwt_issuer.rs

@@ -47,7 +47,10 @@ pub fn issue_access_jwt(
         // did:web resolver could follow.
         iss: cfg.pds_did(),
         sub: did.to_string(),
-        aud: "did:web:appview.maarcadetweet.local".into(),
+        // The AppView this token is meant for. Derived from
+        // `APPVIEW_PUBLIC_URL` rather than hardcoded, so the AppView can
+        // check it against its own identity (`AppConfig::appview_did`).
+        aud: cfg.appview_did(),
         iat: now,
         exp,
         jti: Some(uuid::Uuid::new_v4().to_string()),

crates/pds-server/src/main.rs

@@ -1,5 +1,7 @@
 mod appview_push;
 mod car;
+mod dag_cbor;
+mod firehose;
 mod jwt_issuer;
 mod keys;
 mod password;
@@ -146,6 +148,13 @@ pub fn router(state: AppState) -> Router {
             "/xrpc/com.atproto.sync.getBlob",
             get(routes::blob::get_blob),
         )
+        // The firehose. A WebSocket upgrade arrives as a plain GET, so this
+        // is a normal `get` route whose handler happens to return an
+        // upgrade response.
+        .route(
+            "/xrpc/com.atproto.sync.subscribeRepos",
+            get(routes::subscribe_repos::subscribe_repos),
+        )
         .route(
             "/blob/:cid",
             get(routes::blob::get_blob_by_cid),

crates/pds-server/src/routes/feed.rs

@@ -17,7 +17,10 @@
 //! removed from the MST, a new commit is signed, the AppView is
 //! told to drop the row, and we return the new commit CID + rev.
 
-use crate::routes::helpers::{apply_repo_write, err, lookup_handle, to_sqlx_error, RepoWriteOutcome};
+use crate::firehose::RepoOp;
+use crate::routes::helpers::{
+    apply_repo_write, err, lookup_handle, to_sqlx_error, RepoWriteOutcome, RepoWriteResult,
+};
 use at_repo::blockstore::Blockstore;
 use crate::routes::types::ErrorBody;
 use crate::state::AppState;
@@ -189,11 +192,16 @@ fn build_like_record(req: &CreateLikeReq) -> Result<Value, (StatusCode, Json<Err
 /// Postgres transaction with `SELECT … FOR UPDATE` on the user's
 /// `repos` row. Concurrent writers for the same DID now serialise
 /// behind the row lock instead of clobbering each other.)
+///
+/// Returns the full [`RepoWriteResult`] rather than just the commit: the
+/// firehose event that went into the same transaction carries the sequence
+/// number, which the handlers log so an operator can line a write up against
+/// what a subscriber received.
 async fn apply_and_commit<F>(
     state: &AppState,
     did: &str,
     f: F,
-) -> Result<at_repo::commit::Commit, (StatusCode, Json<ErrorBody>)>
+) -> Result<RepoWriteResult, (StatusCode, Json<ErrorBody>)>
 where
     F: for<'b> FnOnce(
         &'b mut at_repo::repo::Repo<at_repo::blockstore::MemoryBlockstore>,
@@ -201,7 +209,7 @@ where
         Box<dyn std::future::Future<Output = Result<RepoWriteOutcome, sqlx::Error>> + Send + 'b>,
     >,
 {
-    apply_repo_write(state, did, f).await.map(|o| o.commit)
+    apply_repo_write(state, did, f).await
 }
 
 // -- handlers ---------------------------------------------------------------
@@ -275,7 +283,7 @@ pub async fn create_like(
     let push_rkey = rkey.clone();
     let push_handle_str: Option<String> = lookup_handle(&state, &did).await;
 
-    let commit = apply_and_commit(&state, &did, move |repo| {
+    let write = apply_and_commit(&state, &did, move |repo| {
         let value_cid = value_cid;
         let rkey = rkey;
         let record_buf = record_buf;
@@ -297,16 +305,21 @@ pub async fn create_like(
                 commit,
                 head_cid_bytes,
                 head_commit_bytes,
+                // Always a create: the rkey is a freshly minted TID, so it
+                // cannot collide with an existing entry.
+                ops: vec![RepoOp::create(LIKE_COLLECTION, &rkey, value_cid)],
             })
         })
     })
     .await?;
 
+    let commit = write.commit;
     info!(
         collection = LIKE_COLLECTION,
         rkey = %push_rkey,
         cid = %value_cid,
         commit = %commit.cid,
+        seq = write.event.seq,
         "like created"
     );
 
@@ -366,10 +379,21 @@ pub async fn delete_record(
     // `Repo::delete_record` is idempotent at the MST level (returns
     // an unchanged tree if the key isn't present), so we always
     // sign a new commit — the spec says 200 on a no-op delete.
-    let commit = apply_and_commit(&state, &did, move |repo| {
+    let write = apply_and_commit(&state, &did, move |repo| {
         let collection = collection;
         let rkey = rkey;
         Box::pin(async move {
+            // Report the op only when there was something to remove.
+            // `delete_record` is idempotent — deleting a missing rkey signs
+            // an unchanged tree — and announcing a delete for a record that
+            // never existed would make a consumer drop a row it may legitimately
+            // hold under a different rkey, or (worse) log a phantom deletion
+            // on every retry of a duplicate unlike.
+            let existed = repo
+                .get_record(&collection, &rkey)
+                .await
+                .map_err(to_sqlx_error)?
+                .is_some();
             repo.delete_record(&collection, &rkey)
                 .await
                 .map_err(to_sqlx_error)?;
@@ -380,15 +404,22 @@ pub async fn delete_record(
                 commit,
                 head_cid_bytes,
                 head_commit_bytes,
+                ops: if existed {
+                    vec![RepoOp::delete(&collection, &rkey)]
+                } else {
+                    Vec::new()
+                },
             })
         })
     })
     .await?;
 
+    let commit = write.commit;
     info!(
         collection = %push_collection,
         rkey = %push_rkey,
         commit = %commit.cid,
+        seq = write.event.seq,
         "record deleted"
     );
 

crates/pds-server/src/routes/helpers.rs

@@ -12,6 +12,7 @@
 //!   writers for the same DID can't trample each other's MST updates
 //!   (Phase 5b review C1).
 
+use crate::firehose::{self, FirehoseEvent, RepoOp};
 use crate::routes::types::ErrorBody;
 use crate::state::AppState;
 use at_crypto::cid::cid_from_multihash_bytes;
@@ -24,6 +25,7 @@ use cid::Cid;
 use k256::ecdsa::SigningKey;
 use k256::SecretKey;
 use sqlx::Postgres;
+use std::collections::HashSet;
 use std::sync::Arc;
 
 /// Load every block belonging to `did` from the `repo_blocks` table into a
@@ -184,14 +186,31 @@ pub fn to_sqlx_error(e: anyhow::Error) -> sqlx::Error {
 // transaction commits or rolls back, so concurrent writers serialise
 // behind the holder rather than racing on the head_commit column.
 
-/// Result of a successful repo write: the new signed commit, the CID
-/// pointing at the freshly-written head block, and the new revision
-/// string. Callers use the commit for AppView ingest pushes.
+/// What the closure handed to [`apply_repo_write`] returns: the new signed
+/// commit, the CID pointing at the freshly-written head block, and the record
+/// operations the closure performed.
+///
+/// `ops` is not derivable from the commit — the MST stores the resulting
+/// tree, not the edit that produced it, and it cannot tell a create from an
+/// update at all. Only the closure knows what it did, so it says so, and the
+/// firehose event is built from that. Every write path must fill this in
+/// truthfully: an empty `ops` produces a commit frame that tells the AppView
+/// "something changed, guess what".
 #[derive(Debug, Clone)]
 pub struct RepoWriteOutcome {
     pub commit: at_repo::commit::Commit,
     pub head_cid_bytes: Vec<u8>,
     pub head_commit_bytes: Vec<u8>,
+    pub ops: Vec<RepoOp>,
+}
+
+/// What [`apply_repo_write`] returns to the route handler: the commit (used
+/// for the response body and the AppView push) plus the firehose event that
+/// was appended in the same transaction and has already been broadcast.
+#[derive(Debug, Clone)]
+pub struct RepoWriteResult {
+    pub commit: at_repo::commit::Commit,
+    pub event: FirehoseEvent,
 }
 
 /// Apply a write to the user's repo under a row-level lock on the
@@ -218,11 +237,27 @@ pub struct RepoWriteOutcome {
 /// write the row), but the transaction stays open until after the
 /// `UPDATE`. If the closure or `UPDATE` fails, the transaction rolls
 /// back and no head pointer or block row changes are visible.
+///
+/// ## The firehose event rides in the same transaction
+///
+/// Between step 7 and the COMMIT we append one row to `firehose_events`
+/// (see [`crate::firehose`]). It has to be the *same* transaction, not a
+/// follow-up write: if the event were appended afterwards, a crash in the
+/// window between the two would leave a repo whose head has moved but whose
+/// event log never mentions it — and since a consumer's cursor only ever
+/// moves forward, that commit would be invisible to every subscriber
+/// permanently. Sharing the transaction makes "the head moved" and "an event
+/// exists for it" one atomic fact. Conversely, a rollback discards both, so
+/// no subscriber ever sees an event for a commit that did not happen.
+///
+/// The broadcast to live subscribers happens *after* `COMMIT`, for the same
+/// reason in reverse: a subscriber must never receive an event that a
+/// rollback then erases.
 pub async fn apply_repo_write<F>(
     state: &AppState,
     did: &str,
     f: F,
-) -> Result<RepoWriteOutcome, (StatusCode, Json<ErrorBody>)>
+) -> Result<RepoWriteResult, (StatusCode, Json<ErrorBody>)>
 where
     F: for<'b> FnOnce(
         &'b mut Repo<MemoryBlockstore>,
@@ -240,8 +275,12 @@ where
 
     // 2. Take the row-level write lock. Postgres parks competing
     //    transactions here until we COMMIT/ROLLBACK.
-    let head_row: Option<(Vec<u8>, Vec<u8>, Option<Vec<u8>>)> = sqlx::query_as(
-        "SELECT head_cid, head_commit, prev_commit
+    // `rev` comes along because it is the *previous* commit's revision,
+    // which the firehose frame publishes as `since` — a consumer uses it to
+    // notice that it skipped an intermediate commit. It has to be read here,
+    // under the lock, before the UPDATE below overwrites it.
+    let head_row: Option<(Vec<u8>, Vec<u8>, Option<Vec<u8>>, String)> = sqlx::query_as(
+        "SELECT head_cid, head_commit, prev_commit, rev
            FROM repos
           WHERE did = $1
           FOR UPDATE",
@@ -257,8 +296,8 @@ where
         )
     })?;
 
-    let (head_cid_blob, head_commit_blob) = match head_row {
-        Some(r) => (r.0, r.1),
+    let (head_cid_blob, head_commit_blob, prev_rev) = match head_row {
+        Some(r) => (r.0, r.1, r.3),
         None => {
             return Err((
                 StatusCode::NOT_FOUND,
@@ -354,6 +393,26 @@ where
     //    to run additional queries under the row lock, we'd extend
     //    this helper to also hand out a `&mut PgConnection` (which
     //    doesn't have the lifetime headache of `&mut Transaction`).
+    //    Snapshot the CIDs the repo already had *before* the closure runs.
+    //    Diffing against this afterwards is what tells us which blocks are
+    //    new in this commit — the firehose CAR carries only those, because a
+    //    diff stream that re-sent the whole repo on every post would be
+    //    useless. The snapshot is taken here, after the head block re-seed
+    //    above, so the existing head commit does not look new.
+    let blocks_before: HashSet<Cid> = blockstore
+        .list()
+        .await
+        .map_err(|e| {
+            err(
+                StatusCode::INTERNAL_SERVER_ERROR,
+                "InternalServerError",
+                format!("blockstore list: {e:#}"),
+            )
+        })?
+        .into_iter()
+        .map(|(cid, _)| cid)
+        .collect();
+
     let outcome: RepoWriteOutcome = f(&mut repo).await.map_err(|e| {
         err(
             StatusCode::INTERNAL_SERVER_ERROR,
@@ -409,6 +468,51 @@ where
         )
     })?;
 
+    // 8. Append the firehose event. Same transaction as everything above —
+    //    see the "rides in the same transaction" note on this function.
+    //
+    //    The CAR carries the commit block as its root plus every block that
+    //    was not in the repo when we started: the new MST nodes and the new
+    //    record value. Blocks that already existed are omitted; a consumer
+    //    that needs an ancestor asks `com.atproto.sync.getBlocks` for it.
+    let new_blocks: Vec<(Cid, Vec<u8>)> = all_blocks
+        .iter()
+        .filter(|(cid, _)| !blocks_before.contains(*cid) && **cid != outcome.commit.cid)
+        .map(|(cid, bytes)| (*cid, bytes.clone()))
+        .collect();
+    let blocks_car = firehose::build_blocks_car(
+        outcome.commit.cid,
+        &outcome.head_commit_bytes,
+        &new_blocks,
+    );
+
+    // A repo whose head was the all-zero sentinel had no previous commit, so
+    // there is no previous revision to report — `since` is null rather than
+    // the `"0"` placeholder `createAccount` seeds the row with.
+    let since: Option<&str> = if is_zero_blob(&head_cid_blob) {
+        None
+    } else {
+        Some(prev_rev.as_str())
+    };
+
+    let event = firehose::insert_event_in_tx(
+        &mut tx,
+        did,
+        &outcome.commit.rev,
+        since,
+        outcome.commit.cid,
+        blocks_car,
+        outcome.ops.clone(),
+    )
+    .await
+    .map_err(|e| {
+        err(
+            StatusCode::INTERNAL_SERVER_ERROR,
+            "InternalServerError",
+            format!("firehose_events insert: {e}"),
+        )
+    })?;
+
     tx.commit().await.map_err(|e| {
         err(
             StatusCode::INTERNAL_SERVER_ERROR,
@@ -417,7 +521,18 @@ where
         )
     })?;
 
-    Ok(outcome)
+    // 9. Only now, with the commit durable, hand the event to live
+    //    subscribers. Publishing is non-blocking and cannot fail in a way
+    //    that matters: a slow subscriber is dealt with on its own side (see
+    //    the lag policy in `crate::firehose`), and with no subscribers at all
+    //    the send is a no-op. The row is on disk either way, so nothing is
+    //    lost if this reaches nobody.
+    state.firehose.publish(event.clone());
+
+    Ok(RepoWriteResult {
+        commit: outcome.commit,
+        event,
+    })
 }
 
 /// Persist every block in `blocks` into `repo_blocks` using the open

crates/pds-server/src/routes/mod.rs

@@ -5,5 +5,6 @@ pub mod helpers;
 pub mod identity;
 pub mod profile;
 pub mod repo;
+pub mod subscribe_repos;
 pub mod sync;
 pub mod types;

crates/pds-server/src/routes/profile.rs

@@ -16,6 +16,7 @@
 //! overwrite the corresponding fields. Best-effort push to the
 //! AppView follows so the `profiles` cache reflects the new avatar /
 //! display name / bio without waiting for the Jetstream replay.
+use crate::firehose::RepoOp;
 use crate::jwt_issuer;
 use crate::routes::helpers::{
     apply_repo_write, err, load_head_commit, load_signing_key, load_user_blockstore,
@@ -98,6 +99,9 @@ pub async fn set_profile(
 
     // Fetch the existing record, if any.
     let existing = read_profile_record(&state, &did).await?;
+    // Remembered before `existing` is consumed by the merge — the firehose op
+    // needs to know whether this is the first profile write for the account.
+    let existing_present = existing.is_some();
 
     // For any blob CIDs in the request, look up the real
     // `mime_type` / `size` from the `blobs` table — and verify
@@ -152,6 +156,12 @@ pub async fn set_profile(
     };
 
     let next_for_block = next.clone();
+    // `existing` was read before the merge above: a profile record that was
+    // already there makes this an `update` on the firehose, a first-ever
+    // `setProfile` a `create`. The rkey is the fixed `self`, so this is the
+    // one write path where updates are the common case rather than the
+    // exception.
+    let profile_existed = existing_present;
     let outcome = apply_repo_write(&state, &did, move |repo| {
         let value_cid = value_cid;
         let next_for_block = next_for_block;
@@ -175,6 +185,12 @@ pub async fn set_profile(
                 commit,
                 head_cid_bytes,
                 head_commit_bytes,
+                ops: vec![RepoOp::put(
+                    "app.bsky.actor.profile",
+                    "self",
+                    value_cid,
+                    profile_existed,
+                )],
             })
         })
     })
@@ -183,6 +199,7 @@ pub async fn set_profile(
     info!(
         did = %did,
         cid = %outcome.commit.cid,
+        seq = outcome.event.seq,
         "profile record created"
     );
 

crates/pds-server/src/routes/repo.rs

@@ -1,3 +1,4 @@
+use crate::firehose::RepoOp;
 use crate::routes::helpers::{
     apply_repo_write, err, lookup_handle, to_sqlx_error, RepoWriteOutcome,
 };
@@ -110,6 +111,17 @@ pub async fn create_record(
         let record_buf = record_buf;
         let collection = collection;
         Box::pin(async move {
+            // Ask the MST whether the key is already there *before* writing.
+            // The firehose distinguishes `create` from `update` and the tree
+            // itself cannot: after `put_record` both look identical. A
+            // caller-supplied `rkey` (rather than the generated TID) is the
+            // case that actually hits this — an overwrite of an existing
+            // record must not be announced as a create.
+            let existed = repo
+                .get_record(&collection, &rkey)
+                .await
+                .map_err(to_sqlx_error)?
+                .is_some();
             // Repo assumes the value block is already in the
             // blockstore — that's the caller's responsibility.
             repo.blockstore
@@ -127,23 +139,34 @@ pub async fn create_record(
                 commit,
                 head_cid_bytes,
                 head_commit_bytes,
+                ops: vec![RepoOp::put(&collection, &rkey, value_cid, existed)],
             })
         })
     })
     .await?;
 
     let uri = format!("at://{did}/{push_coll}/{push_rkey}");
+    let seq = outcome.event.seq;
     let commit = outcome.commit;
-    info!(uri = %uri, cid = %value_cid, commit = %commit.cid, "record created");
+    info!(
+        uri = %uri,
+        cid = %value_cid,
+        commit = %commit.cid,
+        seq,
+        "record created"
+    );
 
     // 10. Best-effort push to the AppView's `/internal/ingest-commit`.
     //     We send the full record value (not just the CID) because the
     //     AppView's indexer reads `embed` and `reply` off it.
     //
     //     **Spawned** (not awaited) so a transient AppView outage never
-    //     blocks the user's write response. If the push fails, the
-    //     global Jetstream feed will eventually replay the commit to
-    //     the AppView.
+    //     blocks the user's write response. Losing the push is no longer
+    //     terminal: the same commit was appended to `firehose_events` in
+    //     the write transaction above (see `seq` in the log line), so an
+    //     AppView that reconnects to `com.atproto.sync.subscribeRepos`
+    //     with its cursor picks it up. The push is now purely a latency
+    //     optimisation, not the only delivery path.
     tokio::spawn(async move {
         if let Err(e) = push_handle
             .push_create(

crates/pds-server/src/routes/subscribe_repos.rs

@@ -0,0 +1,301 @@
+//! `GET /xrpc/com.atproto.sync.subscribeRepos` — the firehose WebSocket.
+//!
+//! The frame format, the deviation from the atproto spec, the lag policy and
+//! the retention story all live in the module header of [`crate::firehose`];
+//! this file is only the socket.
+//!
+//! ## The handshake, and why it is ordered the way it is
+//!
+//! ```text
+//!   1. subscribe to the live broadcast          <-- BEFORE any DB read
+//!   2. read (min, max) from firehose_events
+//!   3. decide what the cursor means             <-- firehose::plan_cursor
+//!   4. drain the DB replay, remembering the highest seq sent
+//!   5. forward live events with seq > that high-water mark
+//! ```
+//!
+//! Step 1 has to come first. If we read the database and *then* subscribed,
+//! an event committed in between would be in neither: too late for the replay
+//! query, too early for the receiver. Subscribing first turns that race into
+//! a duplicate instead of a gap — the event is both replayed from the table
+//! and sitting in the channel — and a duplicate is something we can filter,
+//! which is what the high-water mark in step 5 does.
+//!
+//! The filter is exact rather than approximate because the write path
+//! serialises `firehose_events` inserts under an advisory lock (see
+//! [`crate::firehose::insert_event_in_tx`]): if the replay query saw seq `N`,
+//! then every seq below `N` is already committed and was also seen. So
+//! "everything the replay covered" is precisely "seq <= N", and every event
+//! that arrives on the channel afterwards has seq > N. No gap, no duplicate,
+//! at the handover.
+//!
+//! Without a cursor there is no replay at all and the high-water mark stays
+//! at zero: a `broadcast::Receiver` only ever yields messages sent after it
+//! was created, so "live only" needs no filtering.
+
+use crate::firehose::{
+    self, CursorPlan, FirehoseEvent, MAX_LAG_RECOVERIES, REPLAY_PAGE_SIZE,
+};
+use crate::state::AppState;
+use axum::extract::ws::{CloseFrame, Message, WebSocket, WebSocketUpgrade};
+use axum::extract::{Query, State};
+use axum::response::Response;
+use serde::Deserialize;
+use tokio::sync::broadcast::error::RecvError;
+use tracing::{debug, info, warn};
+
+use futures::{SinkExt, StreamExt};
+
+#[derive(Debug, Deserialize)]
+pub struct SubscribeQuery {
+    /// The seq of the last event the client already has. Everything with a
+    /// larger seq is replayed before the live stream starts. Absent means
+    /// "live only".
+    pub cursor: Option<i64>,
+}
+
+/// The upgrade handler. Everything interesting happens in [`run`].
+pub async fn subscribe_repos(
+    State(state): State<AppState>,
+    Query(q): Query<SubscribeQuery>,
+    ws: WebSocketUpgrade,
+) -> Response {
+    ws.on_upgrade(move |socket| run(socket, state, q.cursor))
+}
+
+/// Drive one subscriber for the life of its connection.
+async fn run(socket: WebSocket, state: AppState, cursor: Option<i64>) {
+    // 1. Subscribe first — see the ordering note in the module header.
+    let mut rx = state.firehose.subscribe();
+
+    let (mut sink, mut stream) = socket.split();
+
+    // A firehose subscriber sends nothing after the upgrade, but we still
+    // have to read the socket: that is the only way a Close frame (or a
+    // client that vanishes without sending data) is noticed while we are
+    // parked waiting for an event that may not come for hours. The reader
+    // task does nothing but detect the end of the connection and say so.
+    let (dead_tx, mut dead_rx) = tokio::sync::oneshot::channel::<()>();
+    tokio::spawn(async move {
+        while let Some(msg) = stream.next().await {
+            match msg {
+                Ok(Message::Close(_)) | Err(_) => break,
+                // Ping/Pong are handled by axum itself; anything else a
+                // client sends on this endpoint is meaningless and ignored
+                // rather than treated as an error.
+                Ok(_) => continue,
+            }
+        }
+        let _ = dead_tx.send(());
+    });
+
+    // 2./3. Work out what the cursor asks for.
+    let bounds = match firehose::seq_bounds(&state.db).await {
+        Ok(b) => b,
+        Err(e) => {
+            warn!(error = %e, "subscribeRepos: firehose_events bounds read failed");
+            let _ = send_error(&mut sink, "InternalServerError", "event log unavailable").await;
+            return;
+        }
+    };
+
+    let mut replay_from = match firehose::plan_cursor(cursor, bounds) {
+        CursorPlan::LiveOnly => None,
+        CursorPlan::Replay { from } => Some(from),
+        CursorPlan::OutdatedCursor { from, message } => {
+            // Not fatal: the client keeps its connection and gets everything
+            // we still have. It is told about the hole so it can decide
+            // whether to backfill some other way.
+            info!(cursor = ?cursor, %message, "subscribeRepos: outdated cursor");
+            if sink
+                .send(Message::Binary(firehose::encode_info_frame(
+                    "OutdatedCursor",
+                    &message,
+                )))
+                .await
+                .is_err()
+            {
+                return;
+            }
+            Some(from)
+        }
+        CursorPlan::FutureCursor { message } => {
+            // Fatal. Continuing would leave the client waiting for events
+            // that will be numbered below its cursor and therefore filtered
+            // out forever — silence is the worst possible answer here.
+            info!(cursor = ?cursor, %message, "subscribeRepos: future cursor");
+            let _ = send_error(&mut sink, "FutureCursor", &message).await;
+            return;
+        }
+    };
+
+    info!(
+        cursor = ?cursor,
+        replay_from = ?replay_from,
+        subscribers = state.firehose.subscriber_count(),
+        "subscribeRepos: client connected"
+    );
+
+    // 4. Drain the replay. `high_water` ends up as the last seq the client
+    //    has been given, which is exactly the boundary the live filter needs.
+    let mut high_water: i64 = 0;
+    if let Some(from) = replay_from.take() {
+        match replay(&mut sink, &state, from, &mut dead_rx).await {
+            Ok(last) => high_water = last,
+            // The socket died mid-replay, or the log became unreadable.
+            // Either way there is nothing left to do for this connection.
+            Err(()) => return,
+        }
+    }
+
+    // 5. Live.
+    let mut lag_recoveries: u32 = 0;
+    loop {
+        tokio::select! {
+            _ = &mut dead_rx => {
+                debug!("subscribeRepos: client closed");
+                return;
+            }
+            recv = rx.recv() => match recv {
+                Ok(event) => {
+                    if event.seq <= high_water {
+                        // Already delivered by the replay. This is the
+                        // duplicate the subscribe-first ordering trades the
+                        // gap for.
+                        continue;
+                    }
+                    if !send_event(&mut sink, &event).await {
+                        return;
+                    }
+                    high_water = event.seq;
+                }
+                Err(RecvError::Lagged(skipped)) => {
+                    lag_recoveries += 1;
+                    warn!(
+                        skipped,
+                        attempt = lag_recoveries,
+                        high_water,
+                        "subscribeRepos: subscriber lagged; falling back to the database replay"
+                    );
+                    if lag_recoveries > MAX_LAG_RECOVERIES {
+                        // See the lag policy in `crate::firehose`: we rescue a
+                        // slow client, repeatedly, but not forever.
+                        let _ = send_error(
+                            &mut sink,
+                            "ConsumerTooSlow",
+                            "consumer fell behind repeatedly; reconnect with a cursor",
+                        )
+                        .await;
+                        return;
+                    }
+                    let message = format!(
+                        "consumer lagged by {skipped} events; resuming from seq {high_water} \
+                         via the durable log"
+                    );
+                    if sink
+                        .send(Message::Binary(firehose::encode_info_frame(
+                            "OutdatedCursor",
+                            &message,
+                        )))
+                        .await
+                        .is_err()
+                    {
+                        return;
+                    }
+                    // Nothing is lost: every event is in `firehose_events`,
+                    // so re-reading from the high-water mark is the same
+                    // stream the channel dropped. The advisory-lock ordering
+                    // guarantees the new high-water mark is again an exact
+                    // boundary for the live filter.
+                    match replay(&mut sink, &state, high_water, &mut dead_rx).await {
+                        Ok(last) => high_water = last.max(high_water),
+                        Err(()) => return,
+                    }
+                }
+                Err(RecvError::Closed) => {
+                    // Only happens at process shutdown, when the AppState
+                    // (and with it the sender) is dropped.
+                    debug!("subscribeRepos: broadcast channel closed");
+                    return;
+                }
+            }
+        }
+    }
+}
+
+/// Stream every event with `seq > from` out of the database, page by page.
+///
+/// Returns the highest seq actually sent (or `from` when there was nothing to
+/// send), or `Err(())` when the connection or the database gave out — in
+/// which case the caller should drop the connection.
+///
+/// Paging matters: a client reconnecting with `cursor=0` after a long uptime
+/// would otherwise pull the whole table, CARs and all, into memory at once.
+/// It also gives the loop a natural place to notice a client that closed the
+/// socket halfway through a large backfill.
+async fn replay(
+    sink: &mut futures::stream::SplitSink<WebSocket, Message>,
+    state: &AppState,
+    from: i64,
+    dead_rx: &mut tokio::sync::oneshot::Receiver<()>,
+) -> Result<i64, ()> {
+    let mut cursor = from;
+    loop {
+        if dead_rx.try_recv().is_ok() {
+            return Err(());
+        }
+        let page = match firehose::load_events_after(&state.db, cursor, REPLAY_PAGE_SIZE).await {
+            Ok(p) => p,
+            Err(e) => {
+                warn!(error = %format!("{e:#}"), "subscribeRepos: replay read failed");
+                let _ = send_error(sink, "InternalServerError", "event log read failed").await;
+                return Err(());
+            }
+        };
+        if page.is_empty() {
+            return Ok(cursor);
+        }
+        for event in &page {
+            if !send_event(sink, event).await {
+                return Err(());
+            }
+            cursor = event.seq;
+        }
+        // A short page means we reached the end of the log.
+        if (page.len() as i64) < REPLAY_PAGE_SIZE {
+            return Ok(cursor);
+        }
+    }
+}
+
+/// Send one `#commit` frame. Returns `false` when the socket is gone.
+async fn send_event(
+    sink: &mut futures::stream::SplitSink<WebSocket, Message>,
+    event: &FirehoseEvent,
+) -> bool {
+    sink.send(Message::Binary(event.to_frame())).await.is_ok()
+}
+
+/// Send a terminal error frame and close the socket.
+///
+/// The close is explicit (rather than just dropping the sink) so a client
+/// distinguishes "the server said no" from "the connection broke".
+async fn send_error(
+    sink: &mut futures::stream::SplitSink<WebSocket, Message>,
+    error: &str,
+    message: &str,
+) -> bool {
+    if sink
+        .send(Message::Binary(firehose::encode_error_frame(error, message)))
+        .await
+        .is_err()
+    {
+        return false;
+    }
+    sink.send(Message::Close(Some(CloseFrame {
+        code: axum::extract::ws::close_code::NORMAL,
+        reason: error.to_string().into(),
+    })))
+    .await
+    .is_ok()
+}

crates/pds-server/src/state.rs

@@ -1,4 +1,5 @@
 use crate::appview_push::AppViewPushClient;
+use crate::firehose::Firehose;
 use at_blob::S3BlobStore;
 use at_identity::plc::PlcClient;
 use at_lexicon::{Lex, LexRegistry};
@@ -16,6 +17,14 @@ pub struct AppState {
     pub blockstore: Arc<MemoryBlockstore>,
     pub plc: PlcClient,
     pub appview: AppViewPushClient,
+    /// Live fan-out for `com.atproto.sync.subscribeRepos`.
+    ///
+    /// Lives on the shared state rather than in the route module because the
+    /// *write* paths publish into it — `routes::helpers::apply_repo_write`
+    /// hands every committed event over here — while the WebSocket handler
+    /// only subscribes. Cloning `AppState` clones the sender, which is the
+    /// intended way to reach it from a handler.
+    pub firehose: Firehose,
 }
 
 impl AppState {
@@ -26,9 +35,9 @@ impl AppState {
             Lex::from_json(include_str!("../../../lexicons/app/twi/post.json")).unwrap(),
         );
         // AT-Protocol standard collections: only the records the user
-        // might legitimately create server-side (feed.like + feed.repost).
-        // The full atproto collection library is out of scope — for
-        // anything else, callers pass `validate: false` in the
+        // might legitimately create server-side (feed.like, feed.repost,
+        // graph.follow). The full atproto collection library is out of
+        // scope — for anything else, callers pass `validate: false` in the
         // createRecord body.
         lex.lexicons.insert(
             "app.bsky.feed.like".to_string(),
@@ -38,6 +47,17 @@ impl AppState {
             "app.bsky.feed.repost".to_string(),
             Lex::from_json(include_str!("../../../lexicons/app/bsky/feed/repost.json")).unwrap(),
         );
+        // Follow record. Its absence was a real outage: the desktop
+        // client creates follows through `createRecord`, which validates
+        // by default, so every follow came back
+        // `unknown lexicon: app.bsky.graph.follow` — the button could
+        // never have worked. `subject` is a bare DID string here, not a
+        // strongRef like like/repost use, matching what the client sends
+        // and what the AppView's `follow_subject_did` reads.
+        lex.lexicons.insert(
+            "app.bsky.graph.follow".to_string(),
+            Lex::from_json(include_str!("../../../lexicons/app/bsky/graph/follow.json")).unwrap(),
+        );
         // Profile record — avatar/banner/display name/description.
         // Validates the createRecord body when the Tauri client calls
         // its setProfile command. Other fields stay optional so a
@@ -63,6 +83,7 @@ impl AppState {
             blockstore: Arc::new(MemoryBlockstore::new()),
             plc: PlcClient::new(plc_url),
             appview,
+            firehose: Firehose::new(),
         }
     }
 }

crates/pds-server/tests/firehose_integration.rs

@@ -0,0 +1,677 @@
+//! Integration tests for `com.atproto.sync.subscribeRepos`.
+//!
+//! Same contract as the other integration suites in this crate: they talk to a
+//! PDS listening on `127.0.0.1:2583` and **fail open** — if nothing answers
+//! `/healthz`, the test prints a note and returns green rather than failing a
+//! developer's `cargo test` on a machine with no server running. Start the
+//! server (`./target/debug/pds-server` with `.env` sourced) to actually
+//! exercise them.
+//!
+//! ## Why the frames are decoded by hand here
+//!
+//! `pds-server` is a binary, so a test crate cannot import its `dag_cbor`
+//! module — and that is a feature, not a limitation. These tests are the
+//! *consumer* side of the wire contract, and a consumer that reuses the
+//! producer's encoder proves nothing: it would happily agree with a frame
+//! that no other implementation can read. The decoder below is written from
+//! the spec (tag 42, identity prefix, length-first map keys) and knows
+//! nothing about how the server produced the bytes.
+
+use serde_json::{json, Value};
+use std::collections::BTreeMap;
+use std::time::Duration;
+
+use futures::{SinkExt, StreamExt};
+use tokio_tungstenite::tungstenite::Message;
+
+const PDS_URL: &str = "http://127.0.0.1:2583";
+const PDS_WS: &str = "ws://127.0.0.1:2583";
+
+// -- harness ---------------------------------------------------------------
+
+fn http() -> reqwest::Client {
+    reqwest::Client::builder()
+        .timeout(Duration::from_secs(5))
+        .build()
+        .unwrap()
+}
+
+async fn wait_for_pds() -> bool {
+    let c = http();
+    for _ in 0..20 {
+        if let Ok(r) = c.get(format!("{PDS_URL}/healthz")).send().await {
+            if r.status().is_success() {
+                return true;
+            }
+        }
+        tokio::time::sleep(Duration::from_millis(250)).await;
+    }
+    false
+}
+
+async fn fresh_user(prefix: &str) -> (reqwest::Client, String, String) {
+    let c = http();
+    let handle = format!(
+        "{}_{}.maarcadetweet.local",
+        prefix,
+        uuid::Uuid::new_v4().simple()
+    );
+    let acc: Value = c
+        .post(format!("{PDS_URL}/xrpc/com.atproto.server.createAccount"))
+        .json(&json!({"handle": handle, "password": "hunter2hunter2"}))
+        .send()
+        .await
+        .unwrap()
+        .json()
+        .await
+        .unwrap();
+    let did = acc["did"].as_str().expect("createAccount did").to_string();
+    let jwt = acc["access_jwt"].as_str().expect("access_jwt").to_string();
+    (c, did, jwt)
+}
+
+async fn create_post(c: &reqwest::Client, did: &str, jwt: &str, text: &str) -> Value {
+    c.post(format!("{PDS_URL}/xrpc/com.atproto.repo.createRecord"))
+        .bearer_auth(jwt)
+        .json(&json!({
+            "repo": did,
+            "collection": "app.twi.post",
+            "record": { "text": text, "createdAt": "2026-09-10T12:00:00Z" },
+        }))
+        .send()
+        .await
+        .unwrap()
+        .json()
+        .await
+        .unwrap()
+}
+
+type Socket = tokio_tungstenite::WebSocketStream<
+    tokio_tungstenite::MaybeTlsStream<tokio::net::TcpStream>,
+>;
+
+async fn subscribe(cursor: Option<i64>) -> Socket {
+    let url = match cursor {
+        Some(c) => format!("{PDS_WS}/xrpc/com.atproto.sync.subscribeRepos?cursor={c}"),
+        None => format!("{PDS_WS}/xrpc/com.atproto.sync.subscribeRepos"),
+    };
+    let (socket, _resp) = tokio_tungstenite::connect_async(&url)
+        .await
+        .unwrap_or_else(|e| panic!("subscribeRepos connect to {url}: {e}"));
+    socket
+}
+
+/// Read the next **binary** message, or `None` on timeout.
+///
+/// Text messages would be a protocol violation on this endpoint and are
+/// asserted against rather than skipped.
+async fn next_frame(socket: &mut Socket) -> Option<Vec<u8>> {
+    let deadline = Duration::from_secs(10);
+    loop {
+        match tokio::time::timeout(deadline, socket.next()).await {
+            Err(_) => return None,
+            Ok(None) => return None,
+            Ok(Some(Ok(Message::Binary(b)))) => return Some(b),
+            Ok(Some(Ok(Message::Ping(_)))) | Ok(Some(Ok(Message::Pong(_)))) => continue,
+            Ok(Some(Ok(Message::Close(_)))) => return None,
+            Ok(Some(Ok(other))) => panic!("subscribeRepos sent a non-binary frame: {other:?}"),
+            Ok(Some(Err(e))) => panic!("subscribeRepos socket error: {e}"),
+        }
+    }
+}
+
+/// Give the server a moment to finish `on_upgrade` and actually subscribe to
+/// the broadcast channel before we trigger a write.
+///
+/// The TCP handshake completing does not mean the handler has run. Without
+/// this the test would occasionally write before the subscription exists and
+/// then wait for a frame that was published to nobody. (The *cursor* replay
+/// path is the real fix for that race in production; the live-only test is
+/// deliberately testing the raceable path, so it waits.)
+async fn settle() {
+    tokio::time::sleep(Duration::from_millis(300)).await;
+}
+
+// -- an independent DAG-CBOR reader ----------------------------------------
+
+#[derive(Debug, Clone, PartialEq)]
+enum Cbor {
+    Null,
+    Bool(bool),
+    Int(i64),
+    Bytes(Vec<u8>),
+    Text(String),
+    Array(Vec<Cbor>),
+    Map(BTreeMap<String, Cbor>),
+    /// tag(42) + bytes(0x00 || cid) — the binary CID is kept raw and
+    /// re-encoded to a `bafy…` string on demand, so the test never depends on
+    /// the server's own CID formatting.
+    Link(Vec<u8>),
+}
+
+impl Cbor {
+    fn get(&self, key: &str) -> &Cbor {
+        match self {
+            Cbor::Map(m) => m
+                .get(key)
+                .unwrap_or_else(|| panic!("missing key `{key}` in {self:?}")),
+            other => panic!("not a map: {other:?}"),
+        }
+    }
+    fn opt(&self, key: &str) -> Option<&Cbor> {
+        match self {
+            Cbor::Map(m) => m.get(key),
+            _ => None,
+        }
+    }
+    fn int(&self) -> i64 {
+        match self {
+            Cbor::Int(i) => *i,
+            other => panic!("not an int: {other:?}"),
+        }
+    }
+    fn text(&self) -> &str {
+        match self {
+            Cbor::Text(s) => s,
+            other => panic!("not text: {other:?}"),
+        }
+    }
+    fn bool(&self) -> bool {
+        match self {
+            Cbor::Bool(b) => *b,
+            other => panic!("not a bool: {other:?}"),
+        }
+    }
+    fn bytes(&self) -> &[u8] {
+        match self {
+            Cbor::Bytes(b) => b,
+            other => panic!("not bytes: {other:?}"),
+        }
+    }
+    fn array(&self) -> &[Cbor] {
+        match self {
+            Cbor::Array(a) => a,
+            other => panic!("not an array: {other:?}"),
+        }
+    }
+    /// The link's CID rendered as a base32 `bafy…` string, for comparison
+    /// against what the XRPC JSON responses return.
+    fn link_cid(&self) -> String {
+        match self {
+            Cbor::Link(raw) => cid::Cid::read_bytes(&raw[..])
+                .expect("tag-42 payload must be a valid binary CID")
+                .to_string(),
+            other => panic!("not a link: {other:?}"),
+        }
+    }
+}
+
+fn read_head(b: &[u8], p: usize) -> (u8, u64, usize) {
+    let first = b[p];
+    let major = first >> 5;
+    let low = first & 0x1f;
+    let (arg, extra) = match low {
+        0..=23 => (low as u64, 0usize),
+        24 => (b[p + 1] as u64, 1),
+        25 => (u16::from_be_bytes([b[p + 1], b[p + 2]]) as u64, 2),
+        26 => (
+            u32::from_be_bytes([b[p + 1], b[p + 2], b[p + 3], b[p + 4]]) as u64,
+            4,
+        ),
+        27 => {
+            let mut n = 0u64;
+            for i in 0..8 {
+                n = (n << 8) | b[p + 1 + i] as u64;
+            }
+            (n, 8)
+        }
+        other => panic!("indefinite or reserved CBOR head 0x{other:02x} — illegal in DAG-CBOR"),
+    };
+    (major, arg, p + 1 + extra)
+}
+
+fn decode_at(b: &[u8], p: usize) -> (Cbor, usize) {
+    let (major, arg, mut p) = read_head(b, p);
+    match major {
+        0 => (Cbor::Int(arg as i64), p),
+        1 => (Cbor::Int(-(arg as i64) - 1), p),
+        2 => {
+            let end = p + arg as usize;
+            (Cbor::Bytes(b[p..end].to_vec()), end)
+        }
+        3 => {
+            let end = p + arg as usize;
+            (
+                Cbor::Text(std::str::from_utf8(&b[p..end]).unwrap().to_string()),
+                end,
+            )
+        }
+        4 => {
+            let mut items = Vec::new();
+            for _ in 0..arg {
+                let (v, next) = decode_at(b, p);
+                items.push(v);
+                p = next;
+            }
+            (Cbor::Array(items), p)
+        }
+        5 => {
+            let mut m = BTreeMap::new();
+            let mut prev_key: Option<String> = None;
+            for _ in 0..arg {
+                let (k, next) = decode_at(b, p);
+                p = next;
+                let key = k.text().to_string();
+                // Canonical DAG-CBOR order: shorter keys first, then
+                // bytewise. Asserted here because a consumer written against
+                // a strict codec (cborg's `dag-cbor` decoder, for one) will
+                // reject a frame whose keys are out of order.
+                if let Some(prev) = &prev_key {
+                    let ordered = (prev.len(), prev.as_bytes()) < (key.len(), key.as_bytes());
+                    assert!(ordered, "map keys out of canonical order: {prev:?} then {key:?}");
+                }
+                prev_key = Some(key.clone());
+                let (v, next) = decode_at(b, p);
+                p = next;
+                m.insert(key, v);
+            }
+            (Cbor::Map(m), p)
+        }
+        6 => {
+            assert_eq!(arg, 42, "DAG-CBOR permits only tag 42");
+            let (inner, next) = decode_at(b, p);
+            let raw = match inner {
+                Cbor::Bytes(v) => v,
+                other => panic!("tag 42 must wrap bytes, got {other:?}"),
+            };
+            assert_eq!(
+                raw.first(),
+                Some(&0x00),
+                "a binary CID link must start with the 0x00 multibase identity prefix"
+            );
+            (Cbor::Link(raw[1..].to_vec()), next)
+        }
+        7 => match arg {
+            20 => (Cbor::Bool(false), p),
+            21 => (Cbor::Bool(true), p),
+            22 => (Cbor::Null, p),
+            other => panic!("unsupported CBOR simple value {other}"),
+        },
+        other => panic!("unsupported CBOR major type {other}"),
+    }
+}
+
+/// Split one binary frame into its header and body values, and assert that
+/// the two together consume the whole message — a frame with trailing bytes
+/// would silently desynchronise a streaming consumer.
+fn parse_frame(bytes: &[u8]) -> (Cbor, Cbor) {
+    let (header, after_header) = decode_at(bytes, 0);
+    let (body, end) = decode_at(bytes, after_header);
+    assert_eq!(end, bytes.len(), "frame must be exactly two CBOR values");
+    (header, body)
+}
+
+/// Minimal CAR v1 reader: returns the root CIDs and the block CIDs, both as
+/// `bafy…` strings.
+fn parse_car(bytes: &[u8]) -> (Vec<String>, Vec<String>) {
+    fn varint(b: &[u8], p: &mut usize) -> u64 {
+        let (mut v, mut shift) = (0u64, 0u32);
+        loop {
+            let byte = b[*p];
+            *p += 1;
+            v |= ((byte & 0x7f) as u64) << shift;
+            if byte & 0x80 == 0 {
+                return v;
+            }
+            shift += 7;
+        }
+    }
+    let mut p = 0usize;
+    let header_len = varint(bytes, &mut p) as usize;
+    let header = &bytes[p..p + header_len];
+    p += header_len;
+
+    // The CAR header is a CBOR map; walk it with the same head reader. Note
+    // that this server's CAR header tags its roots *without* the 0x00
+    // identity prefix (a documented deviation in `car.rs`), so the roots are
+    // read as plain tagged byte strings rather than through `decode_at`.
+    let mut hp = 0usize;
+    let (major, n, next) = read_head(header, hp);
+    assert_eq!(major, 5, "CAR header must be a map");
+    hp = next;
+    let mut roots = Vec::new();
+    for _ in 0..n {
+        let (m, len, next) = read_head(header, hp);
+        assert_eq!(m, 3);
+        hp = next;
+        let key = std::str::from_utf8(&header[hp..hp + len as usize]).unwrap().to_string();
+        hp += len as usize;
+        if key == "version" {
+            let (m, v, next) = read_head(header, hp);
+            assert_eq!(m, 0);
+            assert_eq!(v, 1, "CAR must be v1");
+            hp = next;
+        } else if key == "roots" {
+            let (m, count, next) = read_head(header, hp);
+            assert_eq!(m, 4);
+            hp = next;
+            for _ in 0..count {
+                let (m, tag, next) = read_head(header, hp);
+                assert_eq!((m, tag), (6, 42), "root must be a tag-42 link");
+                hp = next;
+                let (m, len, next) = read_head(header, hp);
+                assert_eq!(m, 2);
+                hp = next;
+                // A DAG-CBOR link wraps `0x00 || <binary CID>`; the 0x00 is
+                // the multibase identity prefix and is not part of the CID.
+                // Assert on it rather than skipping it silently — this
+                // reader stands in for a foreign consumer, and dropping the
+                // check would let the header regress unnoticed.
+                let raw = &header[hp..hp + len as usize];
+                assert_eq!(
+                    raw.first(),
+                    Some(&0x00),
+                    "CAR root link must carry the multibase identity prefix"
+                );
+                roots.push(cid::Cid::read_bytes(&raw[1..]).unwrap().to_string());
+                hp += len as usize;
+            }
+        } else {
+            panic!("unexpected CAR header key {key}");
+        }
+    }
+
+    let mut blocks = Vec::new();
+    while p < bytes.len() {
+        let section_len = varint(bytes, &mut p) as usize;
+        let section = &bytes[p..p + section_len];
+        let cid = cid::Cid::read_bytes(section).unwrap();
+        blocks.push(cid.to_string());
+        p += section_len;
+    }
+    (roots, blocks)
+}
+
+// -- tests -----------------------------------------------------------------
+
+/// A live subscriber receives a `#commit` frame for a record created after it
+/// connected, and every field of that frame says what it should.
+#[tokio::test]
+async fn live_subscriber_receives_a_commit_frame() {
+    if !wait_for_pds().await {
+        eprintln!("pds not running, skipping");
+        return;
+    }
+    let (c, did, jwt) = fresh_user("fhlive").await;
+
+    let mut socket = subscribe(None).await;
+    settle().await;
+
+    let created = create_post(&c, &did, &jwt, "hello firehose").await;
+    let record_cid = created["cid"].as_str().expect("createRecord cid").to_string();
+    let commit_cid = created["commit"]["cid"]
+        .as_str()
+        .expect("createRecord commit.cid")
+        .to_string();
+    let commit_rev = created["commit"]["rev"].as_str().unwrap().to_string();
+    let uri = created["uri"].as_str().unwrap().to_string();
+    let rkey = uri.rsplit('/').next().unwrap().to_string();
+
+    // The account was created moments ago and has never written before, so
+    // the first frame we see for it is this post's. Other accounts may be
+    // writing concurrently, so filter by DID rather than taking frame 1.
+    let (header, body) = loop {
+        let bytes = next_frame(&mut socket)
+            .await
+            .expect("expected a #commit frame within the timeout");
+        let (header, body) = parse_frame(&bytes);
+        if header.opt("t").map(|t| t.text()) == Some("#commit")
+            && body.get("repo").text() == did
+        {
+            break (header, body);
+        }
+    };
+
+    assert_eq!(header.get("op").int(), 1, "regular frames carry op = 1");
+    assert_eq!(header.get("t").text(), "#commit");
+
+    assert!(body.get("seq").int() > 0, "seq must be a real cursor value");
+    assert_eq!(body.get("repo").text(), did);
+    assert_eq!(body.get("rev").text(), commit_rev);
+    assert_eq!(body.get("commit").link_cid(), commit_cid);
+    assert!(!body.get("rebase").bool());
+    assert!(!body.get("tooBig").bool());
+    assert!(body.get("blobs").array().is_empty());
+    // First-ever commit on a brand new repo — there is no predecessor.
+    assert_eq!(*body.get("since"), Cbor::Null, "first commit has since = null");
+    // RFC 3339 with a Z offset.
+    let time = body.get("time").text();
+    assert!(
+        chrono_like_rfc3339(time),
+        "time must be RFC 3339, got {time}"
+    );
+
+    // ops
+    let ops = body.get("ops").array();
+    assert_eq!(ops.len(), 1, "one record write means one op");
+    assert_eq!(ops[0].get("action").text(), "create");
+    assert_eq!(ops[0].get("path").text(), format!("app.twi.post/{rkey}"));
+    assert_eq!(ops[0].get("cid").link_cid(), record_cid);
+
+    // blocks: a CAR rooted at the commit, containing the commit block and the
+    // new record block.
+    let (roots, blocks) = parse_car(body.get("blocks").bytes());
+    assert_eq!(roots, vec![commit_cid.clone()], "CAR root is the commit");
+    assert!(
+        blocks.contains(&commit_cid),
+        "CAR must carry the commit block itself; got {blocks:?}"
+    );
+    assert!(
+        blocks.contains(&record_cid),
+        "CAR must carry the new record block; got {blocks:?}"
+    );
+
+    let _ = socket.send(Message::Close(None)).await;
+}
+
+/// The same event, fetched again from the durable log with a cursor, is
+/// byte-identical to the live frame.
+///
+/// Byte-identity is the strong form of the claim and the one that matters: a
+/// consumer that deduplicates by hashing frames, or that verifies a signature
+/// over them, must not see two different representations of one event.
+#[tokio::test]
+async fn cursor_replay_reproduces_the_live_frame_exactly() {
+    if !wait_for_pds().await {
+        eprintln!("pds not running, skipping");
+        return;
+    }
+    let (c, did, jwt) = fresh_user("fhreplay").await;
+
+    let mut live = subscribe(None).await;
+    settle().await;
+    create_post(&c, &did, &jwt, "replay me").await;
+
+    let live_bytes = loop {
+        let bytes = next_frame(&mut live)
+            .await
+            .expect("expected a live #commit frame");
+        let (header, body) = parse_frame(&bytes);
+        if header.opt("t").map(|t| t.text()) == Some("#commit")
+            && body.get("repo").text() == did
+        {
+            break bytes;
+        }
+    };
+    let (_h, live_body) = parse_frame(&live_bytes);
+    let seq = live_body.get("seq").int();
+    let _ = live.send(Message::Close(None)).await;
+
+    // Reconnect asking for everything after the event *before* ours, so the
+    // replay's first matching frame is the one we just saw.
+    let mut replayed = subscribe(Some(seq - 1)).await;
+    let replay_bytes = loop {
+        let bytes = next_frame(&mut replayed)
+            .await
+            .expect("expected the event to come back from the replay");
+        let (header, body) = parse_frame(&bytes);
+        if header.opt("t").map(|t| t.text()) == Some("#commit")
+            && body.get("seq").int() == seq
+        {
+            break bytes;
+        }
+    };
+
+    assert_eq!(
+        replay_bytes, live_bytes,
+        "a replayed frame must be byte-identical to the live one"
+    );
+    let _ = replayed.send(Message::Close(None)).await;
+}
+
+/// Handing over from replay to live loses nothing and duplicates nothing.
+///
+/// Connect with a cursor at the current head (so the replay is empty), then
+/// write twice: both events must arrive, in order, exactly once each.
+#[tokio::test]
+async fn replay_to_live_handover_has_no_gap_and_no_duplicate() {
+    if !wait_for_pds().await {
+        eprintln!("pds not running, skipping");
+        return;
+    }
+    let (c, did, jwt) = fresh_user("fhhandover").await;
+
+    // Establish where the log currently ends by writing one event and reading
+    // its seq off the live stream.
+    let mut probe = subscribe(None).await;
+    settle().await;
+    create_post(&c, &did, &jwt, "probe").await;
+    let head_seq = loop {
+        let bytes = next_frame(&mut probe).await.expect("probe frame");
+        let (header, body) = parse_frame(&bytes);
+        if header.opt("t").map(|t| t.text()) == Some("#commit")
+            && body.get("repo").text() == did
+        {
+            break body.get("seq").int();
+        }
+    };
+    let _ = probe.send(Message::Close(None)).await;
+
+    // Now reconnect at that exact cursor: nothing to replay, straight to live.
+    let mut socket = subscribe(Some(head_seq)).await;
+    settle().await;
+
+    create_post(&c, &did, &jwt, "after handover one").await;
+    create_post(&c, &did, &jwt, "after handover two").await;
+
+    let mut seen: Vec<i64> = Vec::new();
+    while seen.len() < 2 {
+        let bytes = next_frame(&mut socket)
+            .await
+            .expect("expected both post-handover frames");
+        let (header, body) = parse_frame(&bytes);
+        if header.opt("t").map(|t| t.text()) != Some("#commit") {
+            continue;
+        }
+        let seq = body.get("seq").int();
+        assert!(
+            seq > head_seq,
+            "the cursor said we already had seq {head_seq}; got {seq} again"
+        );
+        if body.get("repo").text() == did {
+            assert!(!seen.contains(&seq), "event {seq} delivered twice");
+            seen.push(seq);
+        }
+    }
+    assert_eq!(seen.len(), 2);
+    assert!(seen[0] < seen[1], "events must arrive in seq order: {seen:?}");
+    let _ = socket.send(Message::Close(None)).await;
+}
+
+/// A cursor past the end of the log is a terminal error frame, not silence.
+#[tokio::test]
+async fn future_cursor_gets_an_error_frame() {
+    if !wait_for_pds().await {
+        eprintln!("pds not running, skipping");
+        return;
+    }
+    let mut socket = subscribe(Some(i64::MAX / 2)).await;
+    let bytes = next_frame(&mut socket)
+        .await
+        .expect("expected an error frame for a future cursor");
+    let (header, body) = parse_frame(&bytes);
+    assert_eq!(header.get("op").int(), -1, "error frames carry op = -1");
+    assert!(
+        header.opt("t").is_none(),
+        "an error header has no `t`, only `op`"
+    );
+    assert_eq!(body.get("error").text(), "FutureCursor");
+    assert!(
+        !body.get("message").text().is_empty(),
+        "the error should say what went wrong"
+    );
+}
+
+/// A record deleted through `deleteRecord` produces a `delete` op with a null
+/// CID — the one op shape that is not a link.
+#[tokio::test]
+async fn delete_produces_a_delete_op_with_a_null_cid() {
+    if !wait_for_pds().await {
+        eprintln!("pds not running, skipping");
+        return;
+    }
+    let (c, did, jwt) = fresh_user("fhdelete").await;
+    let created = create_post(&c, &did, &jwt, "to be deleted").await;
+    let rkey = created["uri"]
+        .as_str()
+        .unwrap()
+        .rsplit('/')
+        .next()
+        .unwrap()
+        .to_string();
+
+    let mut socket = subscribe(None).await;
+    settle().await;
+
+    let resp = c
+        .post(format!("{PDS_URL}/xrpc/com.atproto.repo.deleteRecord"))
+        .bearer_auth(&jwt)
+        .json(&json!({
+            "repo": did,
+            "collection": "app.twi.post",
+            "rkey": rkey,
+        }))
+        .send()
+        .await
+        .unwrap();
+    assert!(resp.status().is_success(), "deleteRecord: {:?}", resp.status());
+
+    let (_header, body) = loop {
+        let bytes = next_frame(&mut socket).await.expect("expected a delete frame");
+        let (header, body) = parse_frame(&bytes);
+        if header.opt("t").map(|t| t.text()) == Some("#commit")
+            && body.get("repo").text() == did
+        {
+            break (header, body);
+        }
+    };
+
+    let ops = body.get("ops").array();
+    assert_eq!(ops.len(), 1);
+    assert_eq!(ops[0].get("action").text(), "delete");
+    assert_eq!(ops[0].get("path").text(), format!("app.twi.post/{rkey}"));
+    assert_eq!(*ops[0].get("cid"), Cbor::Null, "a delete has no resulting CID");
+
+    // The second commit on this repo, so `since` is the previous revision.
+    assert!(
+        matches!(body.get("since"), Cbor::Text(_)),
+        "a follow-up commit must name its predecessor's rev, got {:?}",
+        body.get("since")
+    );
+    let _ = socket.send(Message::Close(None)).await;
+}
+
+/// Shape check for the frame's `time`: RFC 3339, UTC, with a `Z` suffix.
+fn chrono_like_rfc3339(s: &str) -> bool {
+    chrono::DateTime::parse_from_rfc3339(s).is_ok() && s.ends_with('Z')
+}

crates/pds-server/tests/pds_integration.rs

@@ -435,7 +435,17 @@ fn parse_car(bytes: &[u8]) -> ParsedCar {
                 assert_eq!(maj, 2, "root CID must be a byte string");
                 p += c;
                 let cid_bytes = &bytes[p..p + ln as usize];
-                let cid_hex: String = cid_bytes
+                // A DAG-CBOR link wraps `0x00 || <binary CID>`. The 0x00 is
+                // the multibase identity prefix, not part of the CID, so it
+                // comes off before the hex comparison against a real CID's
+                // bytes. Asserted rather than skipped: this helper is the
+                // only place the header's wire shape is checked.
+                assert_eq!(
+                    cid_bytes.first(),
+                    Some(&0x00),
+                    "root link must carry the multibase identity prefix"
+                );
+                let cid_hex: String = cid_bytes[1..]
                     .iter()
                     .map(|b| format!("{:02x}", b))
                     .collect();
@@ -1056,42 +1066,86 @@ async fn sync_list_repos_keyset_pagination() {
         assert!(resp["uri"].is_string(), "createRecord: {:?}", resp);
         created_dids.push(did);
     }
-    let min_did = created_dids.iter().min().unwrap().clone();
-    let start_cursor = did_cursor_lt(&min_did);
+    // Two separate properties, deliberately not tested by one long walk
+    // from the top of the table: `repos` grows without bound on a
+    // long-lived instance (a few thousand rows here), the seeded DIDs are
+    // random `did:plc:bafy…` hashes scattered across that range, and a
+    // full scan at two rows per page ran into its own iteration cap —
+    // failing for table size rather than for anything about pagination.
 
-    let mut seen: std::collections::HashSet<String> = std::collections::HashSet::new();
-    let mut cursor: Option<String> = Some(start_cursor);
-    let mut pages = 0;
-    loop {
-        pages += 1;
-        assert!(pages < 2000, "pagination did not terminate");
+    // 1. Every seeded DID is reachable: anchor the cursor immediately
+    //    before it and it must be on the first page.
+    for did in &created_dids {
         let url = format!(
             "{}/xrpc/com.atproto.sync.listRepos?limit=2&cursor={}",
             PDS_URL,
-            urlencode(cursor.as_deref().unwrap_or(""))
+            urlencode(&did_cursor_just_before(did))
         );
         let resp = client().await.get(&url).send().await.unwrap();
         assert_eq!(resp.status().as_u16(), 200);
         let body: Value = resp.json().await.unwrap();
         let repos = body["repos"].as_array().expect("repos array");
+        assert!(
+            repos.iter().any(|r| r["did"].as_str() == Some(did.as_str())),
+            "DID not on the page starting immediately before it: {did}"
+        );
+    }
+
+    // 2. The keyset itself: walking forward from the lowest seeded DID
+    //    yields strictly increasing DIDs, never a duplicate, and the
+    //    cursor the server hands back is always the last DID of the page.
+    //    A bounded number of pages is enough — these are properties of
+    //    every step, not of the whole table.
+    let min_did = created_dids.iter().min().unwrap().clone();
+    let mut cursor = did_cursor_just_before(&min_did);
+    let mut seen: std::collections::HashSet<String> = std::collections::HashSet::new();
+    let mut last: Option<String> = None;
+    for _ in 0..25 {
+        let url = format!(
+            "{}/xrpc/com.atproto.sync.listRepos?limit=2&cursor={}",
+            PDS_URL,
+            urlencode(&cursor)
+        );
+        let resp = client().await.get(&url).send().await.unwrap();
+        assert_eq!(resp.status().as_u16(), 200);
+        let body: Value = resp.json().await.unwrap();
+        let repos = body["repos"].as_array().expect("repos array");
+        if repos.is_empty() {
+            break;
+        }
         for r in repos {
             let did = r["did"].as_str().unwrap().to_string();
             assert!(
                 seen.insert(did.clone()),
                 "duplicate DID across pages: {did}"
             );
-        }
-        if created_dids.iter().all(|d| seen.contains(d)) {
-            break;
+            if let Some(prev) = &last {
+                assert!(
+                    &did > prev,
+                    "listRepos must be strictly ascending by DID: {prev} then {did}"
+                );
+            }
+            last = Some(did);
         }
         match body["cursor"].as_str() {
-            Some(c) => cursor = Some(c.to_string()),
-            None => panic!(
-                "pagination exhausted before all created DIDs were seen; missing {:?}",
-                created_dids.iter().filter(|d| !seen.contains(*d)).collect::<Vec<_>>()
-            ),
+            Some(c) => {
+                assert_eq!(
+                    Some(c),
+                    last.as_deref(),
+                    "cursor must be the last DID of the page just served"
+                );
+                cursor = c.to_string();
+            }
+            // Fewer rows than the limit: the end of the table, and the
+            // server correctly stops handing out a cursor.
+            None => break,
         }
     }
+    assert!(
+        seen.len() >= 2,
+        "expected the walk to cover at least two pages, saw {}",
+        seen.len()
+    );
 }
 
 fn urlencode(s: &str) -> String {

crates/tauri-app/src-tauri/src/lib.rs

@@ -250,7 +250,12 @@ async fn repost_post(
     });
     let resp = state
         .pds
-        .create_record_with(&sess.did, "app.bsky.feed.repost", record, false, &sess.access_jwt)
+        // `true`: `app.bsky.feed.repost` is in the PDS's lexicon registry
+        // and this record passes it (verified against a live PDS). The
+        // `false` that stood here was inert — the flag was dropped before
+        // the request — so validating is what has actually been happening
+        // all along; saying so keeps the behaviour and drops the fiction.
+        .create_record_with(&sess.did, "app.bsky.feed.repost", record, true, &sess.access_jwt)
         .await
         .map_err(|e| e.to_string())?;
     Ok(serde_json::json!({

crates/tauri-app/src-tauri/src/pds_client.rs

@@ -41,6 +41,10 @@ pub struct CreateRecordReq {
     pub repo: String,
     pub collection: String,
     pub record: serde_json::Value,
+    /// Omitted rather than sent as `null` when the caller has no
+    /// opinion — the PDS's own default (`true`) then applies.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub validate: Option<bool>,
 }
 
 /// Strong reference as defined by
@@ -184,12 +188,19 @@ impl PdsHttpClient {
         Ok(r.json().await?)
     }
 
+    /// `validate` is forwarded to the PDS, which defaults it to `true`.
+    ///
+    /// It used to be `_validate` — accepted and silently dropped, so a
+    /// caller asking for `false` still got server-side validation. That
+    /// made no difference in practice (every collection the client writes
+    /// is in the PDS's lexicon registry and passes), but a parameter that
+    /// does nothing is a trap for the next caller who relies on it.
     pub async fn create_record_with(
         &self,
         repo: &str,
         collection: &str,
         record: serde_json::Value,
-        _validate: bool,
+        validate: bool,
         jwt: &str,
     ) -> Result<CreateRecordResp> {
         let r = self
@@ -200,6 +211,7 @@ impl PdsHttpClient {
                 repo: repo.to_string(),
                 collection: collection.to_string(),
                 record,
+                validate: Some(validate),
             })
             .send()
             .await?;

docs/architecture.md

@@ -25,7 +25,8 @@ auf welchem Weg kommt ein Post vom Client bis in die Timeline zurück.
     │  /healthz                        │   │  GET /api/post|thread/*uri   │
     │  /.well-known/did.json ──────────┼───┼─▶ Schlüssel für 🔒           │
     │                                  │   │  GET /api/notifications…  🔒 │
-    │                                  │   │  GET /api/followers|following│
+    │  /xrpc/…sync.subscribeRepos ─────┼───┼─▶ pds_firehose.rs (WS)       │
+    │      (WebSocket, seq-Cursor)     │   │  GET /api/followers|following│
     │                                  │   │  GET /healthz                │
     │  at-lexicon  Validierung (160)   │   │                              │
     │  at-repo/at-mst  MST + Commit    │   │  indexer.rs  Upserts         │
@@ -41,13 +42,13 @@ auf welchem Weg kommt ein Post vom Client bis in die Timeline zurück.
   ┌───────────┐  ┌────────┐              ┌─────────────────┐     │
   │ Postgres  │  │ MinIO  │              │ Postgres        │     │
   │ pds :5434 │  │ :9100  │              │ appview :5435   │     │
-  └───────────┘  └────────┘              └─────────────────┘     │
-        │                                                        │
-        │  (heute: kein eigener Firehose-Ausgang)                 │
-        ▼                                                        │
+  │ +firehose │  └────────┘              │ +cursor         │     │
+  │  _events  │                          └─────────────────┘     │
+  └───────────┘                                                  │
+                                                                 │
    ┌──────────────────────────────────────────┐                  │
    │ Jetstream-Relay (extern, WebSocket)      │──────────────────┘
-   │ JETSTREAM_URL                            │   at-firehose
+   │ JETSTREAM_URL — kennt diese PDS nicht    │   at-firehose
    └──────────────────────────────────────────┘   JetstreamConsumer
 ```
 
@@ -56,27 +57,46 @@ entsprechen. Die AppView verifiziert die ES256-Signatur mit dem öffentlichen
 Schlüssel, den die PDS in ihrem DID-Dokument veröffentlicht — `PDS_JWT_SECRET`
 verlässt die PDS nie. Details in [`deployment.md`](deployment.md), Abschnitt 6.
 
-Zwei Wege führen in die AppView, und das ist Absicht:
+Drei Wege führen in die AppView, und das ist Absicht:
 
 1. **Direkter Push (schnell, lokal).** Jeder erfolgreiche Commit auf der PDS
    wird per `POST /internal/ingest-commit` an die AppView geschoben
    (`crates/pds-server/src/appview_push.rs`). Best effort, 5 s Timeout, blockiert
    den Record-Write nie. Damit sieht der Nutzer seinen eigenen Post sofort.
-2. **Jetstream (global, verzögert).** `at-firehose::JetstreamConsumer` hängt an
+2. **PDS-Firehose (lokal, garantiert).** Die PDS führt in derselben
+   Transaktion wie den Commit ein Event in `firehose_events` und liefert es
+   über `com.atproto.sync.subscribeRepos` als WebSocket aus
+   (`crates/pds-server/src/firehose.rs`). Die AppView konsumiert das mit
+   persistiertem Cursor (`crates/appview/src/pds_firehose.rs`).
+3. **Jetstream (global, verzögert).** `at-firehose::JetstreamConsumer` hängt an
    einem externen Jetstream-Relay und liefert alles, was in den konfigurierten
    Collections weltweit passiert.
 
-Wichtig für das Verständnis der Topologie: **die eigene PDS speist den
-Jetstream nicht.** Es gibt keinen `com.atproto.sync.subscribeRepos`-Endpoint im
-PDS-Router. Der Firehose-Weg ist ein reiner Konsum-Pfad für fremde Repos; die
-eigenen Records erreichen die AppView ausschließlich über den Push aus
-Punkt 1 (noch offen).
+Warum 1 **und** 2: Der Push ist der schnelle Weg, der Firehose der
+verlässliche. Ein verlorener Push (AppView kurz weg, Netzwerkfehler) war
+früher endgültig — der öffentliche Jetstream kennt diese PDS nicht, also
+wäre der Post nie angekommen. Jetzt holt der Cursor-Replay ihn nach.
+Dass beide Wege denselben Commit liefern, ist unkritisch: die Indexer-Pfade
+sind Upserts, und der Dedupe-Index der Notifications fängt den Rest.
+
+Wichtig für das Verständnis der Topologie bleibt: **die eigene PDS speist den
+*öffentlichen* Jetstream nicht.** Weg 2 ist ein lokaler Firehose zwischen den
+eigenen zwei Diensten; ein fremder Relay erfährt von dieser PDS weiterhin
+nichts.
+
+Zur Spec-Treue: Die Frame-Hülle ist konformes DAG-CBOR mit Tag-42-CID-Links.
+Die Blöcke *darin* tragen die Konvention dieses Codebases — CIDs innerhalb von
+Commit-Blöcken sind Strings, nicht Links (`at-repo/src/commit.rs`). Ein
+fremder atproto-Consumer kann die Frames also lesen, scheitert aber beim
+Validieren der Blockinhalte. Das zu ändern hieße, die Blockkodierung zu
+ändern, und damit ändern sich sämtliche CIDs inklusive der
+`did:plc:`-Ableitung — eine eigene, bewusste Migration.
 
 ## Crates
 
 | Crate | Typ | Aufgabe |
 |---|---|---|
-| `at-lexicon` | lib | Lexicon-Schemas laden (`Lex::from_json`) und Records validieren. `LexRegistry` in der PDS kennt `app.twi.post` (160 Zeichen), `app.bsky.feed.like`, `app.bsky.feed.repost`, `app.bsky.actor.profile` — alle vier per `include_str!` einkompiliert |
+| `at-lexicon` | lib | Lexicon-Schemas laden (`Lex::from_json`) und Records validieren. `LexRegistry` in der PDS kennt `app.twi.post` (160 Zeichen), `app.bsky.feed.like`, `app.bsky.feed.repost`, `app.bsky.graph.follow`, `app.bsky.actor.profile` — alle per `include_str!` einkompiliert |
 | `at-crypto` | lib | secp256k1/P-256-Keypairs, DAG-CBOR-CIDs, multibase/base58btc, JWT (`issue_jwt` / `verify_jwt`), PLC-Operationen inkl. `did_plc_from_op` |
 | `at-identity` | lib | Handle- und DID-Auflösung. Drei Resolver hinter dem Trait `DidHandleResolver`: `PlcClient` (PLC-Directory), `WebResolver` (`.well-known/did.json`), `PdsHandleResolver` (fragt die lokale PDS) |
 | `at-mst` | lib | Merkle-Search-Tree: Knoten, `split_around`, `wrap_with_split`, spec-konformes `encode_key` |
@@ -164,9 +184,11 @@ Alle Schreibpfade sind Upserts, das Replay nach einem Reconnect ist damit
 unschädlich. Fehlerhafte Events rücken den Cursor **nicht** vor.
 
 `ingest.rs` bedient denselben Indexer über HTTP, mit den Aktionen
-`create` / `delete`; für `app.bsky.graph.follow`-Deletes braucht der Aufrufer
-`subject_did` im Body, weil der Record-Wert bei Deletes nicht garantiert
-mitkommt.
+`create` / `delete`. Für `app.bsky.graph.follow`-Deletes gibt es zwei Wege:
+der Push schickt `subject_did` im Body mit, der Firehose kennt nur `did` +
+`rkey` — deshalb speichert `follows` seit Migration 0011 den rkey des
+Follow-Records und löst darüber auf. Ohne den rkey war ein Unfollow über den
+Firehose nicht anwendbar und hing allein am Push.
 
 ## Datenbanken und Tabellen
 

docs/deployment.md

@@ -287,7 +287,7 @@ Hinweise:
 * Es gibt **keinen** Signal-Handler für graceful Shutdown. `systemctl stop`
   beendet den Prozess hart; bei der AppView bedeutet das, dass der letzte
   Cursor-Flush nur passiert, wenn der Kanal regulär geschlossen wird —
-  praktisch also mit bis zu 100 Events Verlust (siehe Abschnitt 9). Das ist
+  praktisch also mit bis zu 100 Events Verlust (siehe Abschnitt 10). Das ist
   unkritisch, weil der Cursor beim Resume ohnehin leicht in die Vergangenheit
   zeigt und Events idempotent verarbeitet werden.
 * Eine Abhängigkeit `After=` auf Postgres/MinIO ist nur nötig, wenn diese auf
@@ -344,7 +344,33 @@ Instanz hinter VPN und für die fail-open-Integrationstests. Die AppView warnt
 beim Start in Großbuchstaben. Öffentlich erreichbar heißt das: jeder kann die
 Notifications jeder DID lesen und als gelesen markieren.
 
-## 7. Reverse-Proxy
+## 7. Firehose
+
+Die PDS liefert `com.atproto.sync.subscribeRepos` als WebSocket aus, die
+AppView konsumiert ihn. Betrieblich wichtig:
+
+* **Das Event liegt in derselben Transaktion wie der Commit.** Es kann keinen
+  Commit ohne Event geben und umgekehrt.
+* **Die `seq` ist lückenfrei.** Ein globaler `pg_advisory_xact_lock` sorgt
+  dafür, dass Commit-Reihenfolge und `seq`-Reihenfolge übereinstimmen — sonst
+  könnte ein Consumer eine Nummer überspringen, die erst danach sichtbar wird,
+  und sie nie nachholen. Preis: das Ende jeder schreibenden Transaktion ist
+  über alle Accounts hinweg serialisiert.
+* **Cursor:** `?cursor=<seq>` liefert alles mit `seq > cursor` aus der
+  Datenbank nach und geht dann nahtlos live weiter. Ohne Cursor nur live. Ein
+  Cursor aus der Zukunft ist ein Fehler-Frame, ein zu alter ein
+  `#info`/`OutdatedCursor`.
+* **`firehose_events` wächst unbegrenzt.** Es gibt keine Retention. Beschneiden
+  ist sicher, weil ein zu alter Cursor sauber behandelt wird — wer die Tabelle
+  aufräumt, sollte aber wissen, wie weit die eigenen Consumer zurückhängen
+  dürfen (`pds_firehose_seq` in `/healthz` der AppView gegen `MAX(seq)`).
+* **Reverse-Proxy:** die Route braucht ein WebSocket-Upgrade (`Upgrade`/
+  `Connection`-Header durchreichen) und einen Read-Timeout, der längere
+  Ruhephasen überlebt.
+* `PDS_FIREHOSE_ENABLED=false` schaltet den Consumer in der AppView ab; lokale
+  Commits hängen dann wieder allein am Best-Effort-Push.
+
+## 8. Reverse-Proxy
 
 ### PDS
 
@@ -462,7 +488,7 @@ HTTP-Aufrufe an PDS/AppView laufen über den Rust-IPC-Layer
 (`src-tauri/src/pds_client.rs`, `appview_client.rs`), nicht aus dem Webview —
 die CSP muss also für neue Backend-URLs nicht angefasst werden.
 
-## 8. Health-Checks und Logs
+## 9. Health-Checks und Logs
 
 ### PDS
 
@@ -526,7 +552,7 @@ Log-Zeilen, auf die es sich lohnt zu achten:
 | `s3 ping failed at startup` | MinIO beim PDS-Start nicht erreichbar |
 | `plc submit failed (dev ok)` | PLC-Directory nicht erreichbar; die DID bleibt lokal gültig, ist aber global nicht registriert |
 
-## 9. Neustart-Verhalten
+## 10. Neustart-Verhalten
 
 **PDS.** Zustandslos bis auf Postgres und MinIO. Der In-Memory-Blockstore
 (`MemoryBlockstore` in `state.rs`) wird beim Start neu aufgebaut; persistent
@@ -569,7 +595,7 @@ nach — zuerst über die lokale PDS (`PdsHandleResolver`, 2 s Timeout), dann PL
 bzw. `did:web`. Nach einem Neustart holt der erste Durchlauf das nach; der
 Zustand ist reine Anzeigekosmetik.
 
-## 10. Was noch offen ist
+## 11. Was noch offen ist
 
 * Kein Compose-Service für `pds-server` / `appview` — das Compose-File deckt nur
   Postgres und MinIO ab. Es gibt kein Dockerfile im Repo.
@@ -581,3 +607,10 @@ Zustand ist reine Anzeigekosmetik.
   einsetzt. Signatur, Ablauf, `scope` und `sub` werden geprüft.
 * Notifications werden nie gelöscht; ein Unlike/Unfollow lässt die Zeile stehen.
 * Kein Backfill-Werkzeug für Jetstream-Lücken.
+* **Keine Retention für `firehose_events`.** Die Tabelle wächst mit jedem
+  Commit und wird nie beschnitten. Pruning ist sicher — ein Consumer mit zu
+  altem Cursor bekommt `#info`/`OutdatedCursor` und läuft ab der ältesten
+  überlebenden Zeile weiter — aber es gibt weder Job noch Policy dafür.
+* Der globale Advisory-Lock, der die `seq`-Vergabe ordnet, serialisiert das
+  Ende jeder schreibenden Transaktion über alle Accounts hinweg. Das
+  begrenzt den Schreibdurchsatz auf ein COMMIT nach dem anderen.

lexicons/app/bsky/graph/follow.json

@@ -0,0 +1,18 @@
+{
+  "lexicon": 1,
+  "id": "app.bsky.graph.follow",
+  "defs": {
+    "main": {
+      "type": "record",
+      "key": "tid",
+      "record": {
+        "type": "object",
+        "required": ["subject", "createdAt"],
+        "properties": {
+          "subject": { "type": "string", "format": "did" },
+          "createdAt": { "type": "datetime" }
+        }
+      }
+    }
+  }
+}

migrations/appview/0010_pds_firehose_cursor.sql

@@ -0,0 +1,44 @@
+-- AppView database schema 0010: cursor for the local PDS firehose.
+--
+-- Why a second cursor table
+--
+-- The AppView now consumes two event streams, and they are numbered in
+-- completely different spaces:
+--
+--   * `jetstream_cursor.cursor` is a Jetstream `time_us` — microseconds
+--     since the epoch, produced by a public relay we do not control.
+--   * this table's `cursor` is the `seq` of our own PDS's
+--     `com.atproto.sync.subscribeRepos` — a small monotonic counter
+--     that starts at 1 in a fresh PDS database.
+--
+-- Sharing one row between them would mean the larger of the two values
+-- (always the Jetstream timestamp) permanently swallowing the other:
+-- `cursor_advance` uses GREATEST, so the very first Jetstream event
+-- would push the PDS cursor to ~1.7e15 and every subsequent
+-- subscribeRepos connect would ask for a sequence the PDS will never
+-- reach. Hence a table of its own, deliberately in the same shape as
+-- `jetstream_cursor` so both read/advance the same way.
+--
+-- Shape
+--   id          pinned to 1 by a CHECK — a single-row table, the same
+--               pattern `jetstream_cursor` uses. It makes "advance the
+--               cursor" a plain UPDATE with no upsert dance and makes a
+--               second row impossible to create by accident.
+--   cursor      the last `seq` we durably applied. 0 means "nothing
+--               yet": the consumer then subscribes without a `cursor`
+--               query parameter, which the PDS reads as "start from the
+--               current head" rather than replaying the entire repo
+--               history into a fresh index.
+--   updated_at  observability only — how stale the stream is can be
+--               read straight off the row.
+--
+-- The row is inserted here so `cursor_advance`'s UPDATE always has a
+-- target; `pds_firehose::cursor_get` still tolerates a missing row and
+-- returns 0.
+
+CREATE TABLE pds_firehose_cursor (
+    id             INTEGER PRIMARY KEY DEFAULT 1 CHECK (id = 1),
+    cursor         BIGINT NOT NULL DEFAULT 0,
+    updated_at     TIMESTAMPTZ NOT NULL DEFAULT now()
+);
+INSERT INTO pds_firehose_cursor (id, cursor) VALUES (1, 0);

migrations/appview/0011_follows_rkey.sql

@@ -0,0 +1,79 @@
+-- AppView database schema 0011: remember which record a follow came from.
+--
+-- Why
+--
+-- A `follows` row was addressable only as `(follower_did, subject_did)`.
+-- That is the right identity for the *relationship*, but it is not the
+-- identity a delete event carries. A firehose / Jetstream delete op is
+-- just `did` + `rkey`:
+--
+--   {"action": "delete", "path": "app.bsky.graph.follow/3lmnop"}
+--
+-- There is no record body on a delete — the record is gone, that is the
+-- whole point of the event — so the subject DID is nowhere in it. With
+-- no rkey stored, the indexer had no way from `3lmnop` back to
+-- "did:plc:bob" and logged-and-skipped the op
+-- (`indexer::apply_commit`, `app.bsky.graph.follow` arm).
+--
+-- The practical consequence: unfollows only ever landed through the
+-- PDS's best-effort `POST /internal/ingest-commit` push, which knows the
+-- subject from its own snapshot. That push has no retry and no
+-- acknowledgement (see `pds_firehose`'s module docs). If it was lost —
+-- AppView restarting, request timing out — the follow stayed in the
+-- index forever, and the firehose, the stream that exists precisely to
+-- repair such gaps, could not repair this one. Storing the rkey closes
+-- that hole: the firehose replay can now apply the unfollow on its own.
+--
+-- The primary key deliberately stays `(follower_did, subject_did)`
+-- ------------------------------------------------------------------
+-- It is what makes `upsert_follow` idempotent. The same follow reaches
+-- us over both transports (push *and* firehose) and again after any
+-- replay, and every one of those must converge on one row. Keying on
+-- the rkey instead — or adding it to the key — would make a re-follow
+-- under a fresh rkey a *second* row for the same relationship, and then
+-- `follower_count` would count the same follower twice.
+--
+-- So `rkey` is not identity here; it is a second *access path* to a row
+-- the primary key already identifies.
+--
+-- Nullable, because history has no rkey
+-- -------------------------------------
+-- Every row written before this migration was inserted without one, and
+-- there is nothing to backfill it from: the AppView never stored the
+-- follow record itself. A NOT NULL column would need a fabricated
+-- placeholder that a later delete could accidentally match. NULL says
+-- exactly what is true — "we do not know which record this came from" —
+-- and a delete-by-rkey simply finds nothing for those rows, which is the
+-- documented no-op path in `indexer::delete_follow_by_rkey`. Those rows
+-- keep working through the push path (which sends `subject_did`) and
+-- heal on their own the next time the follow is re-created.
+--
+-- Re-follow under a new rkey
+-- --------------------------
+-- Follow → unfollow → follow again produces a *different* rkey each
+-- time (rkeys are TIDs; the client never reuses one). The upsert
+-- therefore hits the primary key and overwrites `rkey` with the newer
+-- record's: the youngest record wins. That ordering is what makes a
+-- late or replayed delete for the *old* rkey harmless — it matches no
+-- row and is skipped, instead of tearing down a follow that is
+-- currently live.
+--
+-- The index is NOT unique
+-- -----------------------
+-- `(follower_did, rkey)` is unique in practice — an rkey identifies one
+-- record inside one repo's collection — but a unique index would turn
+-- the one situation this migration exists for into a *write failure*:
+-- if a delete was lost and a create later reused that rkey, the insert
+-- would abort instead of the stale row being cleaned up. An index whose
+-- only job is to serve a lookup should not be able to reject a write.
+-- Partial (`WHERE rkey IS NOT NULL`) because a lookup key is never
+-- NULL, so the pre-migration rows have no business bloating it.
+
+ALTER TABLE follows ADD COLUMN IF NOT EXISTS rkey TEXT;
+
+-- Serves `DELETE FROM follows WHERE follower_did = $1 AND rkey = $2
+--         RETURNING subject_did` — the delete path for a firehose
+-- unfollow, which is the only lookup this column exists for.
+CREATE INDEX IF NOT EXISTS follows_follower_rkey_idx
+    ON follows (follower_did, rkey)
+    WHERE rkey IS NOT NULL;

migrations/pds/0003_firehose.sql

@@ -0,0 +1,112 @@
+-- PDS database schema 0003: the firehose event log.
+--
+-- Why
+--
+-- Until now the PDS produced no `com.atproto.sync.subscribeRepos` stream at
+-- all. The only way a local record reached the AppView was the best-effort
+-- HTTP push in `appview_push.rs` — a fire-and-forget `tokio::spawn` that is
+-- explicitly documented as "the Jetstream replay will catch up". There is no
+-- Jetstream replay for records that only exist on this PDS, so a dropped push
+-- meant the post was simply never indexed. Nothing retried it, and nothing
+-- could: the commit lived in `repos` / `repo_blocks` but there was no ordered
+-- log of *what changed* for a consumer to walk.
+--
+-- This table is that log. Every repo write appends exactly one row, in the
+-- same transaction as the head-pointer update, so the sequence and the repo
+-- head can never disagree. A consumer that reconnects with a cursor replays
+-- from here; a consumer that is live gets the same rows pushed over a
+-- broadcast channel.
+--
+-- Column choices
+--
+--   seq         BIGSERIAL PRIMARY KEY — the cursor. It has to be a single
+--               monotonically increasing integer because that is what the
+--               `subscribeRepos` wire contract hands the client and takes
+--               back as `?cursor=`. BIGSERIAL (not an `(timestamp, id)`
+--               keyset like the AppView's notifications table) because the
+--               protocol's cursor is opaque-but-numeric and clients compare
+--               it with `>`.
+--
+--               Sequence values are handed out at INSERT time, which by
+--               itself does NOT guarantee that they become *visible* in seq
+--               order — two transactions can grab 5 and 6 and commit in the
+--               opposite order, leaving a reader that polls in between with a
+--               gap it would never fill. The write path therefore takes
+--               `pg_advisory_xact_lock` on a fixed key immediately before
+--               this INSERT (see `routes::helpers::apply_repo_write`), which
+--               serialises the tail of every firehose-writing transaction so
+--               commit order == seq order. That is what makes "give me
+--               everything with seq > N" an exact, gap-free replay rather
+--               than a best guess.
+--
+--   did         the repo the event belongs to. Not a FK to `users(did)`:
+--               the log outlives the account. If a user is deleted we still
+--               want consumers that are mid-replay to see the events that
+--               already happened rather than have the rows cascade out from
+--               under their cursor.
+--
+--   rev         the new commit's revision (TID string), mirrored from
+--               `repos.rev`. Goes out as the frame's `rev`.
+--
+--   since       the *previous* commit's rev, or NULL for the first commit on
+--               a repo. The frame's `since` field; a consumer uses it to
+--               detect that it missed an intermediate commit.
+--
+--   commit_cid  BYTEA holding the raw binary CID of the new commit, stored
+--               the same way `repos.head_cid` stores it so the two are
+--               directly comparable with `=` and no text/binary conversion
+--               is needed to join them.
+--
+--   blocks      BYTEA holding a complete CAR v1 file: the commit block as the
+--               root plus every block this commit newly created (MST nodes
+--               and record values). Stored pre-serialised rather than
+--               reassembled from `repo_blocks` at read time because the
+--               *diff* — which blocks were new for this particular commit —
+--               is only knowable at write time. Recomputing it later would
+--               mean diffing two MST snapshots on every replayed event.
+--
+--   ops         JSONB array of `{action, path, cid}`, the same objects that
+--               go into the frame's `ops` field. JSONB rather than a child
+--               table because it is always read as a whole, is never queried
+--               by content, and a child table would need its own ordering
+--               column to reproduce the array faithfully.
+--
+--   created_at  when the event was appended. This is what the frame's `time`
+--               field carries, so a replayed frame is byte-identical to the
+--               live one that was broadcast at commit time — a consumer that
+--               deduplicates by hashing frames does not see two different
+--               frames for one event.
+--
+-- Retention: there is none
+-- ------------------------
+-- Nothing prunes this table. It grows by one row per repo write, and each row
+-- carries a CAR of the commit's new blocks (a few hundred bytes for a plain
+-- post, more when a record is large). At the volume this deployment sees that
+-- is fine for a long time, but it is unbounded, and an operator who wants a
+-- bound has to add one. Deleting the oldest rows is safe: a client whose
+-- cursor points before the surviving range gets an `#info`/`OutdatedCursor`
+-- frame and resumes from the oldest row that still exists. See the module
+-- header of `crates/pds-server/src/firehose.rs`.
+
+CREATE TABLE IF NOT EXISTS firehose_events (
+    seq            BIGSERIAL PRIMARY KEY,
+    did            TEXT NOT NULL,
+    rev            TEXT NOT NULL,
+    since          TEXT,
+    commit_cid     BYTEA NOT NULL,
+    blocks         BYTEA NOT NULL,
+    ops            JSONB NOT NULL,
+    created_at     TIMESTAMPTZ NOT NULL DEFAULT now()
+);
+
+-- Cursor replay is `WHERE seq > $1 ORDER BY seq LIMIT $2`, which the
+-- BIGSERIAL primary key's own index already serves — no second index for
+-- that, on purpose: an extra index on `seq` would be pure write amplification
+-- on the hottest path in this table.
+--
+-- What the PK does *not* serve is "replay one repo", which is how an operator
+-- re-drives a single account into the AppView after an ingest bug, and how
+-- `getRepo`-style backfills are debugged. `(did, seq)` covers that and keeps
+-- the per-repo scan in seq order.
+CREATE INDEX IF NOT EXISTS firehose_events_did_seq_idx
+    ON firehose_events (did, seq);