From edc1183e6122b9906ec10958a00f9f597f6c4293 Mon Sep 17 00:00:00 2001 From: Julien Herr Date: Fri, 22 May 2026 15:51:53 +0200 Subject: [PATCH] docs: rewrite CLAUDE.md to match actual codebase, remove .cursor CLAUDE.md now reflects the real route set (atom, entries, files, hub, email handler), src/lib/ layout, admin sub-modules, client script pipeline, full Env bindings, and WebSub KV schema. Co-Authored-By: Claude Sonnet 4.6 --- .cursor/rules/project-implementation-plan.mdc | 57 -------- AGENTS.md | 99 ------------- CLAUDE.md | 137 ++++++++++++------ 3 files changed, 95 insertions(+), 198 deletions(-) delete mode 100644 .cursor/rules/project-implementation-plan.mdc delete mode 100644 AGENTS.md diff --git a/.cursor/rules/project-implementation-plan.mdc b/.cursor/rules/project-implementation-plan.mdc deleted file mode 100644 index f7a14b4..0000000 --- a/.cursor/rules/project-implementation-plan.mdc +++ /dev/null @@ -1,57 +0,0 @@ ---- -description: Project Implementation Plan -globs: * -alwaysApply: false ---- -# Detailed Project Implementation Plan - -This section is a top-level "blueprint" describing the solution architecture and how each component fits together. - -## Overview - -**Goal:** Build a service that turns email newsletters into RSS feeds, so you can subscribe in an RSS reader like Reeder. The service should provide unique email addresses per feed, a front-end admin panel, indefinite (or long-term) storage of newsletters, and minimal cost—preferably using Cloudflare services plus ForwardEmail.net. - -## Key Components - -1. **ForwardEmail.net** - - Accept incoming newsletters on your custom domain’s email addresses. - - Forward them (via webhook) to your API endpoint for processing. - - Free inbound plan includes JSON + raw MIME data. -2. **Cloudflare Workers** - - **Inbound Worker:** Receives the webhook from ForwardEmail.net, parses/stores newsletter data in KV (or R2). - - **RSS Worker:** Serves RSS feeds by reading from KV and outputting XML. - - **Admin Worker (potential):** Could serve a small UI or JSON API for feed management. -3. **Cloudflare KV** - - Key-value store for storing newsletter items (subject, date, HTML, etc.). - - Minimal cost for text data. - - Indefinite retention if you keep usage under limits. -4. **Cloudflare Pages (Optional)** - - Could host a separate front-end for admin tasks. - - Alternatively, build a simple admin UI directly within the Worker. -5. **Admin Dashboard** - - Basic login and feed creation (generate random email addresses). - - List newsletters and optionally delete them or rename feed titles. - - For a simple approach, implement a minimal password-protected area or JSON endpoints. -6. **Domain / DNS Setup** - - Use your custom domain (e.g. `mynewsletters.dev`). - - Add DNS records so ForwardEmail.net is the MX handler. - - Configure Cloudflare for general DNS (with “Orange Cloud” or not, depending on your proxying preferences). - - Verify your domain following ForwardEmail.net’s instructions. - -## Data Flow - -1. A newsletter arrives at `newsletterXYZ@mynewsletters.dev`. -2. ForwardEmail.net triggers a webhook to `https://your-worker.example.com/api/inbound?feed=XYZ` with JSON + raw MIME. -3. The Worker parses the email, extracts relevant information (date, subject, HTML body), and stores it in KV under a key like `feed:XYZ:timestamp`. -4. When your RSS reader (e.g. Reeder) requests `GET https://your-worker.example.com/rss/XYZ`, the Worker fetches all items from KV for that feed, builds an RSS XML response, and returns it. -5. *(Optional)* The Admin Dashboard (via a password-protected route or a separate Cloudflare Pages front-end) can create new feed IDs, display items, etc. - -## Summary of Implementation Steps - -1. Set up the Domain and ForwardEmail.net for inbound mail. -2. Create a Cloudflare Worker to handle the inbound webhook. -3. Parse the email (using ForwardEmail.net’s parsed data or parsing the raw MIME if necessary). -4. Store the data in KV. -5. Create an RSS Worker endpoint to retrieve the data and output XML. -6. *(Optional)* Develop an Admin UI to create new feeds, list items, and manage them. -7. Deploy and test the solution. Subscribe to the feed with Reeder. diff --git a/AGENTS.md b/AGENTS.md deleted file mode 100644 index 75e7afa..0000000 --- a/AGENTS.md +++ /dev/null @@ -1,99 +0,0 @@ -# AGENTS.md - -This file gives coding agents fast context for working in this repository. - -## Project summary - -kill-the-news is a Cloudflare Worker that ingests email newsletters and exposes them as private RSS feeds. - -Core goals: - -- Self-hosted and private -- Free-tier-friendly (Cloudflare + ForwardEmail) -- Minimal operational overhead - -## Runtime and stack - -- Runtime: Cloudflare Workers -- Framework: Hono (`src/index.ts` + `src/routes/*`) -- Storage: Cloudflare KV (`EMAIL_STORAGE` binding) -- Typescript + Vitest for development/testing - -## Important files - -- `setup.sh`: bootstraps local setup, KV namespaces, secrets, and local Wrangler config -- `wrangler-example.toml`: template used by setup -- `src/index.ts`: app boot + CORS + inbound IP allowlist middleware -- `src/routes/inbound.ts`: email ingestion endpoint -- `src/routes/rss.ts`: RSS rendering endpoint -- `src/routes/admin.ts`: admin UI and feed/email management -- `src/test/setup.ts`: test runtime mocks (KV + Cache) - -## KV data model - -Current keys used by routes: - -- `feeds:list` -> `{ feeds: Array<{ id, title }> }` -- `feeds:list.feeds[].description` -> optional description (used to keep the dashboard fast; older data may omit it) -- `feed::config` -> feed config object -- `feed::config.allowed_senders` -> optional sender allowlist (email or domain) -- `feed::metadata` -> `{ emails: Array<{ key, subject, receivedAt }> }` -- `feed::` -> stored email body/metadata - -Notes: - -- Some utility files contain alternate key helpers not used by routes (`src/utils/storage.ts`). -- Keep route behavior and key schema consistent when refactoring. - -## Setup/deploy workflow - -1. `npx wrangler login` -2. `bash setup.sh` -3. Configure ForwardEmail DNS records in Cloudflare -4. `npm run deploy` - -`setup.sh` assumes Wrangler v4 command syntax (`wrangler kv namespace ...`). - -## Development workflow - -- Install: `npm install` -- Test: `npm test` -- Build (dry-run deploy bundle): `npm run build` -- Dev server: `npm run dev` - -## Testing notes - -- Tests run in Node environment (`vitest.config.ts`), not DOM. -- Hono v4 test requests pass env as the 3rd arg: `app.request(path, init, env)`. -- Some tests intentionally hit validation errors; stderr logs are expected. - -## Security assumptions - -- Inbound endpoint only accepts requests from ForwardEmail source IPs. -- Admin access uses a signed cookie gate and password stored in Worker secret (`ADMIN_PASSWORD`). -- Admin pages set `Cache-Control: no-store`. -- Prefer setting `allowed_senders` on legitimate feeds to reduce inbound spam. -- Do not hardcode credentials or domain-specific secrets into tracked files. - -## Spam cleanup workflow - -- First choice: use dashboard bulk actions (`/admin`) with search + checkbox selection. -- Use **Table** view for bulk delete. -- Table columns are resizable and sortable; widths persist per-browser via localStorage. -- **Select Results** selects all rows currently shown by the search filter; **Clear Selection** unselects everything. -- Bulk deletes are performed asynchronously (batched requests) so the UI stays responsive. -- Avoid wildcard deletion; prefer search + small batches to reduce risk of deleting legitimate feeds. - -## Cloudflare/Wrangler conventions - -- `wrangler.toml` is generated locally from `wrangler-example.toml`. -- Keep `compatibility_date` current on meaningful runtime upgrades. -- Prefer explicit `--env production` for deploy/secret commands. - -## If you change behavior - -Update all of the following together: - -- `README.md` -- `setup.sh` (if setup/deploy assumptions changed) -- tests under `src/routes/*.test.ts` and `src/test/setup.ts` diff --git a/CLAUDE.md b/CLAUDE.md index 9cfac76..5a82a27 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,18 +1,19 @@ # CLAUDE.md -This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. +This file provides guidance to Claude Code when working in this repository. ## Commands ```bash -npm install # Install dependencies -npm run dev # Start local dev server (wrangler dev) -npm test # Run tests once -npm run test:watch # Run tests in watch mode +npm install # Install dependencies (also builds client scripts via prepare) +npm run dev # Start local dev server (wrangler dev) +npm test # Run all tests once +npm run test:watch # Run tests in watch mode npm run test:coverage # Run tests with coverage report -npm run build # Dry-run deploy bundle (wrangler deploy --dry-run) -npm run deploy # Deploy to Cloudflare production -npm run format # Format with Prettier +npm run build # Dry-run deploy bundle (wrangler deploy --dry-run) +npm run build:client # Compile client scripts only (src/scripts/client → src/scripts/generated) +npm run deploy # Deploy to Cloudflare production +npm run format # Format with Prettier ``` Run a single test file: @@ -21,41 +22,101 @@ Run a single test file: npx vitest run src/routes/admin.test.ts ``` +## Project summary + +kill-the-news is a Cloudflare Worker that ingests email newsletters and exposes them as private RSS/Atom feeds. Self-hosted, free-tier-friendly (Cloudflare + ForwardEmail). + ## Architecture -Cloudflare Worker built with Hono. A single Worker handles three route groups: +Single Cloudflare Worker built with Hono. Routes: -- `POST /api/inbound` — ForwardEmail webhook; IP-restricted to ForwardEmail MX sources (verified dynamically via `https://forwardemail.net/ips/v4.json`, with in-memory cache + static fallback) -- `GET /rss/:feedId` — public RSS feed rendered from KV -- `/admin` — password-protected admin UI (server-rendered HTML with inline scripts) +| Method | Path | Purpose | +| ------------------------------------ | ---------------------------------------------------------------------- | ------- | +| `POST /api/inbound` | Webhook from ForwardEmail; IP-allowlisted to their MX sources | +| `GET /rss/:feedId` | Public RSS 2.0 feed | +| `GET /atom/:feedId` | Public Atom feed (with WebSub hub header) | +| `GET /entries/:feedId/:entryId` | Individual email HTML view | +| `GET /files/:attachmentId/:filename` | R2 attachment serving | +| `GET /admin` | Password-protected admin UI | +| `/hub` | WebSub hub (subscribe/publish) | +| `GET /health` | Health check | +| `email` | Cloudflare Email routing handler (alternative to ForwardEmail webhook) | -### Key files +### Source layout -| File | Purpose | -| ----------------------- | ------------------------------------------------------------------------ | -| `src/index.ts` | App entrypoint: CORS middleware, IP allowlist middleware, route mounting | -| `src/routes/inbound.ts` | Email ingestion: validates, parses, stores to KV | -| `src/routes/rss.ts` | Reads KV and renders RSS XML | -| `src/routes/admin.ts` | Admin UI (HTML) and feed/email CRUD API | -| `src/types/index.ts` | Shared TypeScript types (`Env`, `FeedConfig`, `EmailData`, etc.) | -| `src/test/setup.ts` | Test mocks for KV (`MockKV`) and Cache; exports `createMockEnv()` | +``` +src/ + index.ts # App entrypoint: CORS, IP middleware, route mounting, email handler export + config/constants.ts # Shared constants (TTLs, limits) + types/index.ts # Env, FeedConfig, EmailData, WebSubSubscription, etc. + routes/ + inbound.ts # ForwardEmail webhook handler + rss.ts # RSS feed renderer + atom.ts # Atom feed renderer + entries.ts # Single email HTML view + files.ts # R2 attachment serving + hub.ts # WebSub hub + admin.tsx # Admin UI entrypoint (hono/jsx) + admin/ # Admin sub-modules + feeds.tsx # Feeds CRUD UI + emails.tsx # Emails list/delete UI + ui.tsx # Shared UI components + helpers.ts # Shared admin helpers + lib/ + cloudflare-email.ts # Cloudflare Email routing handler + email-parser.ts # Email parsing (mailparser) + email-processor.ts # Core ingestion logic (parse → store) + feed-fetcher.ts # KV feed/email fetch helpers + feed-generator.ts # RSS/Atom XML generation + forwardemail.ts # ForwardEmail webhook types/parsing + id-generator.ts # Feed/entry ID generation + logger.ts # JSON structured logger + storage.ts # KV key helpers + websub.ts # WebSub subscription management + worker.ts # Typed worker export helper + scripts/ + client/ # TypeScript client scripts (compiled by esbuild) + dashboard.ts # Admin dashboard interactions + emails-page.ts # Emails page interactions + generated/ # Compiled output (gitignored, rebuilt on npm install) + styles/ # CSS files bundled into the Worker + variables.css + layout.css + components.css + utilities.css + data/nouns.ts # Word list for ID generation + test/setup.ts # Test mocks: MockKV, createMockEnv() +``` ### KV schema All data lives in the `EMAIL_STORAGE` KV namespace: -| Key | Value | -| --------------------------- | ------------------------------------------------- | -| `feeds:list` | `{ feeds: Array<{ id, title, description? }> }` | -| `feed::config` | `FeedConfig` object | -| `feed::metadata` | `{ emails: Array<{ key, subject, receivedAt }> }` | -| `feed::` | Full `EmailData` object | +| Key | Value | +| -------------------------------- | ------------------------------------------------------------------------ | +| `feeds:list` | `{ feeds: Array<{ id, title, description? }> }` | +| `feed::config` | `FeedConfig` | +| `feed::metadata` | `{ emails: Array<{ key, subject, receivedAt, size?, attachmentIds? }> }` | +| `feed::` | Full `EmailData` | +| `websub::` | `WebSubSubscription` | -`src/utils/storage.ts` contains alternate key helpers not used by routes — keep route key usage consistent with the schema above. +`src/lib/storage.ts` contains key-builder helpers — use them; don't inline key strings in routes. -### Admin UI architecture +### Worker bindings (`Env`) -The admin UI is server-rendered HTML returned by `src/routes/admin.ts`. Interactive behavior (toast notifications, clipboard, bulk delete, table column resizing/sorting) is implemented as inline scripts in `src/scripts/`. Styles are composed from `src/styles/`. These are bundled into the Worker — there is no separate frontend build step. +```ts +EMAIL_STORAGE: KVNamespace; // All feed/email data +ADMIN_PASSWORD: string; // Worker secret — never in config files +DOMAIN: string; // e.g. "getmynews.app" +ATTACHMENT_BUCKET?: R2Bucket; // R2 for email attachments +FEED_MAX_SIZE_BYTES?: string; // Optional email size cap +PROXY_TRUSTED_IPS?: string; // Trusted reverse-proxy IPs +PROXY_AUTH_SECRET?: string; // Shared secret for proxy auth +``` + +### Client scripts + +`src/scripts/client/` contains TypeScript that runs in the browser. It is compiled by esbuild into `src/scripts/generated/` (gitignored) and bundled into the Worker as inline `