diff --git a/docs/superpowers/specs/2026-05-25-native-feed-detection-design.md b/docs/superpowers/specs/2026-05-25-native-feed-detection-design.md new file mode 100644 index 0000000..2bb30b4 --- /dev/null +++ b/docs/superpowers/specs/2026-05-25-native-feed-detection-design.md @@ -0,0 +1,178 @@ +# Detect a newsletter's native Atom/RSS/JSON feed — design + +_Date: 2026-05-25 · Backlog item: TODO.md "Detect a newsletter's native Atom/RSS feed" (P2·S)_ + +## Goal + +When an incoming newsletter email's HTML advertises its own syndication feed via +``, detect it and surface it to the admin: +"this newsletter already publishes a feed — you can subscribe to it directly." + +A genuine differentiator: it is the top item on upstream kill-the-newsletter's +own TODO and is not built there. Per the user, detect **Atom, RSS, and JSON Feed**. + +## Approach + +Reuse the existing, proven pipeline — identical to the **confirmation-detection** +feature and the **per-sender unsubscribe** storage: + +infra parses the HTML → a **pure domain detector** decides which links count → +the result rides into the aggregate via `IngestOptions` (like `unsub`) → the +aggregate stores it per-sender → admin surfaces it (detail + list badge + +dashboard pill + dismiss) → exposed read-only on the REST API. + +Alternative considered and rejected: an ad-hoc detector living in infrastructure. +The "which MIME types are a feed" rule is business knowledge, so it belongs in +`domain/`, mirroring `domain/confirmation.ts`. + +## Data model + +```ts +// New value shape +type NativeFeed = { url: string; type: "rss" | "atom" | "json" }; + +// FeedMetadata (src/types/index.ts) — additive, like `unsubscribe` / `pendingConfirmation` +nativeFeeds?: Record; // key = senderKey (same scheme as `unsubscribe`) +nativeFeedDismissed?: boolean; // dismiss: hides pill + badge, keeps the URLs + +// FeedListItem (src/types/index.ts) — projected flag for the dashboard, like pendingConfirmation +hasNativeFeed?: boolean; +``` + +Update semantics (chosen by the user: "accumulation, but latest-per-sender"): + +- Storage is **per sender**, keyed by the same `senderKey` used for `unsubscribe` + (`input.senders[0] || iconDomain || input.from`). +- **Latest non-empty wins per sender**: the most recent email from a given sender + that declares feeds overwrites _that sender's_ list; other senders are preserved. + Mirrors how `ingest` updates `unsubscribe` only when an unsubscribe URL is present. +- The aggregate exposes `nativeFeeds(): NativeFeed[]` = the **union across senders, + deduped by URL** (returns a copy). +- Projected flag `hasNativeFeed = nativeFeeds().length > 0 && !nativeFeedDismissed`. + +Smart re-notify (avoid nagging on every email): + +- On ingest, if a **previously-unseen URL** appears (not in the current union), + clear `nativeFeedDismissed` (re-raise the notice). +- If ingestion only re-discovers already-known URLs, leave `nativeFeedDismissed` + untouched, so a dismiss sticks until a genuinely new native feed shows up. + +## Detection + +**Infra — `src/infrastructure/html-processor.ts`**: new +`extractFeedLinks(content): { href: string; type: string }[]`. + +- Parse `` elements whose `rel` token-list contains `alternate` and that + carry a `type` attribute (`link[rel~="alternate"][type]`). +- Return the raw `href` + `type` tuples; absolutize a relative `href` best-effort + via the existing `toAbsolute` helper; http(s) only (drop others). +- Plain-text bodies have no `` → returns `[]`. + +**Domain — `src/domain/native-feed.ts`** (pure, no DOM/IO, mirrors `confirmation.ts`): +`detectNativeFeeds(links): NativeFeed[]`. + +- Owns the recognized MIME → kind table (strict, the three canonical types only): + - `application/atom+xml` → `"atom"` + - `application/rss+xml` → `"rss"` + - `application/feed+json` → `"json"` +- Ignores any other type (no `application/json` — too broad, would capture non-feeds). +- Dedupes by URL; preserves first-seen kind for a URL. + +## Ingestion wiring + +**`src/application/email-processor.ts`**: alongside the existing confirmation + +unsubscribe extraction, call `extractFeedLinks(input.content)` → +`detectNativeFeeds(...)`. When non-empty, pass into `feed.ingest` via +`IngestOptions`: + +```ts +nativeFeeds?: { senderKey: string; feeds: NativeFeed[] }; +``` + +`senderKey` is the same value already computed for `unsub`. + +**`src/domain/feed.aggregate.ts`**: + +- `ingest`: when `opts.nativeFeeds` is present, set + `_metadata.nativeFeeds[senderKey] = feeds`; if any feed URL is new vs the + pre-update union, set `_metadata.nativeFeedDismissed = false`. +- Getter `nativeFeeds(): NativeFeed[]` — union deduped by URL (copy). +- Getter `hasNativeFeed(): boolean` — `nativeFeeds().length > 0 && !dismissed`. +- `dismissNativeFeed(): void` — sets `nativeFeedDismissed = true` (lower-only, + mirrors `dismissConfirmation`). +- `removeEmails` does **not** touch native feeds (the data is per-sender, not + per-email; deleting emails should not drop a discovered native feed). + +**`src/infrastructure/feed-mapper.ts`**: `toListItemDTO` gains a `hasNativeFeed` +parameter (like `pendingConfirmation`), projected into `FeedListItem`. The +repository passes `feed.hasNativeFeed()` when saving. `nativeFeeds` / +`nativeFeedDismissed` persist as part of `FeedMetadata` (stored directly in KV — +additive, no mapper change for the metadata blob itself). + +## REST API + +**`src/routes/api/schemas.ts`** — `FeedSchema` gains a read-only field: + +```ts +nativeFeeds: z.array(z.object({ + url: z.string(), + type: z.enum(["rss", "atom", "json"]), +})), +``` + +Populated from `feed.nativeFeeds()` in the feed-read handler so an API client can +choose which native feed to subscribe to. Read-only — not accepted on +`FeedCreate`/`FeedUpdate`. + +## Admin UI + +Mirror the confirmation surfaces (detail + badge + pill + dismiss). + +- **Detail (per-feed view, `src/routes/admin/emails.tsx`)**: next to the existing + `FeedFormats` "Subscribe" block (the **KTN feeds**), render a second group + **"Native feeds"** when `nativeFeeds()` is non-empty. Each native feed is a + copyable chip (type label RSS/Atom/JSON + copy + open-in-new-tab), reusing the + existing copyable/chip styling. Net result: KTN feeds on one side, native feeds + on the other, both copy-pasteable into a reader. Include a "dismiss" control + (POST to the dismiss route) to clear the dashboard/list notice. +- **List badge (`src/routes/admin.tsx` feed row)**: a discreet badge when + `hasNativeFeed`. +- **Dashboard pill**: `pill-native` (styled like `pill-confirmation`) on the + dashboard feed list when `hasNativeFeed`. +- **Dismiss route**: `POST /admin/feeds/:feedId/native-feed/dismiss` → load + aggregate → `feed.dismissNativeFeed()` → save → JSON ok. Client script wired + like the existing `confirmation-dismiss` (in `src/scripts/client/`). +- **Styles**: add `.native-feeds` group + `.pill-native` + badge rules in + `src/styles/components.css`, matching the format-chip / confirmation styling. + +## Out of scope (v1) + +- No change to the **public XML/JSON feed output** (user decision: native feeds + live in admin + REST, not in the rendered feeds). +- No anchor-text heuristics ("Subscribe via RSS" links) — `` + only, to keep false positives near zero. + +## Testing + +- `src/domain/native-feed.test.ts` — MIME mapping, dedupe, ignores unknown types. +- `src/infrastructure/html-processor.test.ts` — `extractFeedLinks`: rel/type + parsing, relative-href absolutization, http(s)-only, plain-text → `[]`. +- `src/domain/feed.aggregate.test.ts` — ingest per-sender latest-wins, union + getter, re-notify on new URL, dismiss lower-only, removeEmails leaves it intact. +- `src/application/email-processor.test.ts` — end-to-end: an email with a + `` populates `nativeFeeds`; one without leaves it alone. +- `src/infrastructure/feed-mapper.test.ts` / repository — `hasNativeFeed` + projection into `feeds:list`. +- `src/routes/admin.test.ts` — detail "native-feeds" group, list badge, dashboard + `pill-native`, dismiss route clears the flag. +- REST API test — `FeedSchema.nativeFeeds` present in the feed-read response. + +End green: `npx tsc --noEmit`, `npm test`, `npm run build`. + +## Docs + +- `README.md` / `INSTALL.md` — mention native-feed detection. +- `docs/index.html` (marketing landing) — add a feature card (it's a + differentiator we ship before upstream). +- `CLAUDE.md` — add `domain/native-feed.ts` to the source layout; note the new + `FeedMetadata.nativeFeeds` / `nativeFeedDismissed` fields.