mirror of
https://github.com/juherr/kill-the-news.git
synced 2026-06-20 22:03:48 +00:00
docs: rewrite CLAUDE.md to match actual codebase, remove .cursor
CLAUDE.md now reflects the real route set (atom, entries, files, hub, email handler), src/lib/ layout, admin sub-modules, client script pipeline, full Env bindings, and WebSub KV schema. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Julien Herr
@@ -1,57 +0,0 @@
|
|||||||
---
|
|
||||||
description: Project Implementation Plan
|
|
||||||
globs: *
|
|
||||||
alwaysApply: false
|
|
||||||
---
|
|
||||||
# Detailed Project Implementation Plan
|
|
||||||
|
|
||||||
This section is a top-level "blueprint" describing the solution architecture and how each component fits together.
|
|
||||||
|
|
||||||
## Overview
|
|
||||||
|
|
||||||
**Goal:** Build a service that turns email newsletters into RSS feeds, so you can subscribe in an RSS reader like Reeder. The service should provide unique email addresses per feed, a front-end admin panel, indefinite (or long-term) storage of newsletters, and minimal cost—preferably using Cloudflare services plus ForwardEmail.net.
|
|
||||||
|
|
||||||
## Key Components
|
|
||||||
|
|
||||||
1. **ForwardEmail.net**
|
|
||||||
- Accept incoming newsletters on your custom domain’s email addresses.
|
|
||||||
- Forward them (via webhook) to your API endpoint for processing.
|
|
||||||
- Free inbound plan includes JSON + raw MIME data.
|
|
||||||
2. **Cloudflare Workers**
|
|
||||||
- **Inbound Worker:** Receives the webhook from ForwardEmail.net, parses/stores newsletter data in KV (or R2).
|
|
||||||
- **RSS Worker:** Serves RSS feeds by reading from KV and outputting XML.
|
|
||||||
- **Admin Worker (potential):** Could serve a small UI or JSON API for feed management.
|
|
||||||
3. **Cloudflare KV**
|
|
||||||
- Key-value store for storing newsletter items (subject, date, HTML, etc.).
|
|
||||||
- Minimal cost for text data.
|
|
||||||
- Indefinite retention if you keep usage under limits.
|
|
||||||
4. **Cloudflare Pages (Optional)**
|
|
||||||
- Could host a separate front-end for admin tasks.
|
|
||||||
- Alternatively, build a simple admin UI directly within the Worker.
|
|
||||||
5. **Admin Dashboard**
|
|
||||||
- Basic login and feed creation (generate random email addresses).
|
|
||||||
- List newsletters and optionally delete them or rename feed titles.
|
|
||||||
- For a simple approach, implement a minimal password-protected area or JSON endpoints.
|
|
||||||
6. **Domain / DNS Setup**
|
|
||||||
- Use your custom domain (e.g. `mynewsletters.dev`).
|
|
||||||
- Add DNS records so ForwardEmail.net is the MX handler.
|
|
||||||
- Configure Cloudflare for general DNS (with “Orange Cloud” or not, depending on your proxying preferences).
|
|
||||||
- Verify your domain following ForwardEmail.net’s instructions.
|
|
||||||
|
|
||||||
## Data Flow
|
|
||||||
|
|
||||||
1. A newsletter arrives at `newsletterXYZ@mynewsletters.dev`.
|
|
||||||
2. ForwardEmail.net triggers a webhook to `https://your-worker.example.com/api/inbound?feed=XYZ` with JSON + raw MIME.
|
|
||||||
3. The Worker parses the email, extracts relevant information (date, subject, HTML body), and stores it in KV under a key like `feed:XYZ:timestamp`.
|
|
||||||
4. When your RSS reader (e.g. Reeder) requests `GET https://your-worker.example.com/rss/XYZ`, the Worker fetches all items from KV for that feed, builds an RSS XML response, and returns it.
|
|
||||||
5. *(Optional)* The Admin Dashboard (via a password-protected route or a separate Cloudflare Pages front-end) can create new feed IDs, display items, etc.
|
|
||||||
|
|
||||||
## Summary of Implementation Steps
|
|
||||||
|
|
||||||
1. Set up the Domain and ForwardEmail.net for inbound mail.
|
|
||||||
2. Create a Cloudflare Worker to handle the inbound webhook.
|
|
||||||
3. Parse the email (using ForwardEmail.net’s parsed data or parsing the raw MIME if necessary).
|
|
||||||
4. Store the data in KV.
|
|
||||||
5. Create an RSS Worker endpoint to retrieve the data and output XML.
|
|
||||||
6. *(Optional)* Develop an Admin UI to create new feeds, list items, and manage them.
|
|
||||||
7. Deploy and test the solution. Subscribe to the feed with Reeder.
|
|
||||||
@@ -1,99 +0,0 @@
|
|||||||
# AGENTS.md
|
|
||||||
|
|
||||||
This file gives coding agents fast context for working in this repository.
|
|
||||||
|
|
||||||
## Project summary
|
|
||||||
|
|
||||||
kill-the-news is a Cloudflare Worker that ingests email newsletters and exposes them as private RSS feeds.
|
|
||||||
|
|
||||||
Core goals:
|
|
||||||
|
|
||||||
- Self-hosted and private
|
|
||||||
- Free-tier-friendly (Cloudflare + ForwardEmail)
|
|
||||||
- Minimal operational overhead
|
|
||||||
|
|
||||||
## Runtime and stack
|
|
||||||
|
|
||||||
- Runtime: Cloudflare Workers
|
|
||||||
- Framework: Hono (`src/index.ts` + `src/routes/*`)
|
|
||||||
- Storage: Cloudflare KV (`EMAIL_STORAGE` binding)
|
|
||||||
- Typescript + Vitest for development/testing
|
|
||||||
|
|
||||||
## Important files
|
|
||||||
|
|
||||||
- `setup.sh`: bootstraps local setup, KV namespaces, secrets, and local Wrangler config
|
|
||||||
- `wrangler-example.toml`: template used by setup
|
|
||||||
- `src/index.ts`: app boot + CORS + inbound IP allowlist middleware
|
|
||||||
- `src/routes/inbound.ts`: email ingestion endpoint
|
|
||||||
- `src/routes/rss.ts`: RSS rendering endpoint
|
|
||||||
- `src/routes/admin.ts`: admin UI and feed/email management
|
|
||||||
- `src/test/setup.ts`: test runtime mocks (KV + Cache)
|
|
||||||
|
|
||||||
## KV data model
|
|
||||||
|
|
||||||
Current keys used by routes:
|
|
||||||
|
|
||||||
- `feeds:list` -> `{ feeds: Array<{ id, title }> }`
|
|
||||||
- `feeds:list.feeds[].description` -> optional description (used to keep the dashboard fast; older data may omit it)
|
|
||||||
- `feed:<feedId>:config` -> feed config object
|
|
||||||
- `feed:<feedId>:config.allowed_senders` -> optional sender allowlist (email or domain)
|
|
||||||
- `feed:<feedId>:metadata` -> `{ emails: Array<{ key, subject, receivedAt }> }`
|
|
||||||
- `feed:<feedId>:<timestamp>` -> stored email body/metadata
|
|
||||||
|
|
||||||
Notes:
|
|
||||||
|
|
||||||
- Some utility files contain alternate key helpers not used by routes (`src/utils/storage.ts`).
|
|
||||||
- Keep route behavior and key schema consistent when refactoring.
|
|
||||||
|
|
||||||
## Setup/deploy workflow
|
|
||||||
|
|
||||||
1. `npx wrangler login`
|
|
||||||
2. `bash setup.sh`
|
|
||||||
3. Configure ForwardEmail DNS records in Cloudflare
|
|
||||||
4. `npm run deploy`
|
|
||||||
|
|
||||||
`setup.sh` assumes Wrangler v4 command syntax (`wrangler kv namespace ...`).
|
|
||||||
|
|
||||||
## Development workflow
|
|
||||||
|
|
||||||
- Install: `npm install`
|
|
||||||
- Test: `npm test`
|
|
||||||
- Build (dry-run deploy bundle): `npm run build`
|
|
||||||
- Dev server: `npm run dev`
|
|
||||||
|
|
||||||
## Testing notes
|
|
||||||
|
|
||||||
- Tests run in Node environment (`vitest.config.ts`), not DOM.
|
|
||||||
- Hono v4 test requests pass env as the 3rd arg: `app.request(path, init, env)`.
|
|
||||||
- Some tests intentionally hit validation errors; stderr logs are expected.
|
|
||||||
|
|
||||||
## Security assumptions
|
|
||||||
|
|
||||||
- Inbound endpoint only accepts requests from ForwardEmail source IPs.
|
|
||||||
- Admin access uses a signed cookie gate and password stored in Worker secret (`ADMIN_PASSWORD`).
|
|
||||||
- Admin pages set `Cache-Control: no-store`.
|
|
||||||
- Prefer setting `allowed_senders` on legitimate feeds to reduce inbound spam.
|
|
||||||
- Do not hardcode credentials or domain-specific secrets into tracked files.
|
|
||||||
|
|
||||||
## Spam cleanup workflow
|
|
||||||
|
|
||||||
- First choice: use dashboard bulk actions (`/admin`) with search + checkbox selection.
|
|
||||||
- Use **Table** view for bulk delete.
|
|
||||||
- Table columns are resizable and sortable; widths persist per-browser via localStorage.
|
|
||||||
- **Select Results** selects all rows currently shown by the search filter; **Clear Selection** unselects everything.
|
|
||||||
- Bulk deletes are performed asynchronously (batched requests) so the UI stays responsive.
|
|
||||||
- Avoid wildcard deletion; prefer search + small batches to reduce risk of deleting legitimate feeds.
|
|
||||||
|
|
||||||
## Cloudflare/Wrangler conventions
|
|
||||||
|
|
||||||
- `wrangler.toml` is generated locally from `wrangler-example.toml`.
|
|
||||||
- Keep `compatibility_date` current on meaningful runtime upgrades.
|
|
||||||
- Prefer explicit `--env production` for deploy/secret commands.
|
|
||||||
|
|
||||||
## If you change behavior
|
|
||||||
|
|
||||||
Update all of the following together:
|
|
||||||
|
|
||||||
- `README.md`
|
|
||||||
- `setup.sh` (if setup/deploy assumptions changed)
|
|
||||||
- tests under `src/routes/*.test.ts` and `src/test/setup.ts`
|
|
||||||
@@ -1,18 +1,19 @@
|
|||||||
# CLAUDE.md
|
# CLAUDE.md
|
||||||
|
|
||||||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
This file provides guidance to Claude Code when working in this repository.
|
||||||
|
|
||||||
## Commands
|
## Commands
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
npm install # Install dependencies
|
npm install # Install dependencies (also builds client scripts via prepare)
|
||||||
npm run dev # Start local dev server (wrangler dev)
|
npm run dev # Start local dev server (wrangler dev)
|
||||||
npm test # Run tests once
|
npm test # Run all tests once
|
||||||
npm run test:watch # Run tests in watch mode
|
npm run test:watch # Run tests in watch mode
|
||||||
npm run test:coverage # Run tests with coverage report
|
npm run test:coverage # Run tests with coverage report
|
||||||
npm run build # Dry-run deploy bundle (wrangler deploy --dry-run)
|
npm run build # Dry-run deploy bundle (wrangler deploy --dry-run)
|
||||||
npm run deploy # Deploy to Cloudflare production
|
npm run build:client # Compile client scripts only (src/scripts/client → src/scripts/generated)
|
||||||
npm run format # Format with Prettier
|
npm run deploy # Deploy to Cloudflare production
|
||||||
|
npm run format # Format with Prettier
|
||||||
```
|
```
|
||||||
|
|
||||||
Run a single test file:
|
Run a single test file:
|
||||||
@@ -21,41 +22,101 @@ Run a single test file:
|
|||||||
npx vitest run src/routes/admin.test.ts
|
npx vitest run src/routes/admin.test.ts
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## Project summary
|
||||||
|
|
||||||
|
kill-the-news is a Cloudflare Worker that ingests email newsletters and exposes them as private RSS/Atom feeds. Self-hosted, free-tier-friendly (Cloudflare + ForwardEmail).
|
||||||
|
|
||||||
## Architecture
|
## Architecture
|
||||||
|
|
||||||
Cloudflare Worker built with Hono. A single Worker handles three route groups:
|
Single Cloudflare Worker built with Hono. Routes:
|
||||||
|
|
||||||
- `POST /api/inbound` — ForwardEmail webhook; IP-restricted to ForwardEmail MX sources (verified dynamically via `https://forwardemail.net/ips/v4.json`, with in-memory cache + static fallback)
|
| Method | Path | Purpose |
|
||||||
- `GET /rss/:feedId` — public RSS feed rendered from KV
|
| ------------------------------------ | ---------------------------------------------------------------------- | ------- |
|
||||||
- `/admin` — password-protected admin UI (server-rendered HTML with inline scripts)
|
| `POST /api/inbound` | Webhook from ForwardEmail; IP-allowlisted to their MX sources |
|
||||||
|
| `GET /rss/:feedId` | Public RSS 2.0 feed |
|
||||||
|
| `GET /atom/:feedId` | Public Atom feed (with WebSub hub header) |
|
||||||
|
| `GET /entries/:feedId/:entryId` | Individual email HTML view |
|
||||||
|
| `GET /files/:attachmentId/:filename` | R2 attachment serving |
|
||||||
|
| `GET /admin` | Password-protected admin UI |
|
||||||
|
| `/hub` | WebSub hub (subscribe/publish) |
|
||||||
|
| `GET /health` | Health check |
|
||||||
|
| `email` | Cloudflare Email routing handler (alternative to ForwardEmail webhook) |
|
||||||
|
|
||||||
### Key files
|
### Source layout
|
||||||
|
|
||||||
| File | Purpose |
|
```
|
||||||
| ----------------------- | ------------------------------------------------------------------------ |
|
src/
|
||||||
| `src/index.ts` | App entrypoint: CORS middleware, IP allowlist middleware, route mounting |
|
index.ts # App entrypoint: CORS, IP middleware, route mounting, email handler export
|
||||||
| `src/routes/inbound.ts` | Email ingestion: validates, parses, stores to KV |
|
config/constants.ts # Shared constants (TTLs, limits)
|
||||||
| `src/routes/rss.ts` | Reads KV and renders RSS XML |
|
types/index.ts # Env, FeedConfig, EmailData, WebSubSubscription, etc.
|
||||||
| `src/routes/admin.ts` | Admin UI (HTML) and feed/email CRUD API |
|
routes/
|
||||||
| `src/types/index.ts` | Shared TypeScript types (`Env`, `FeedConfig`, `EmailData`, etc.) |
|
inbound.ts # ForwardEmail webhook handler
|
||||||
| `src/test/setup.ts` | Test mocks for KV (`MockKV`) and Cache; exports `createMockEnv()` |
|
rss.ts # RSS feed renderer
|
||||||
|
atom.ts # Atom feed renderer
|
||||||
|
entries.ts # Single email HTML view
|
||||||
|
files.ts # R2 attachment serving
|
||||||
|
hub.ts # WebSub hub
|
||||||
|
admin.tsx # Admin UI entrypoint (hono/jsx)
|
||||||
|
admin/ # Admin sub-modules
|
||||||
|
feeds.tsx # Feeds CRUD UI
|
||||||
|
emails.tsx # Emails list/delete UI
|
||||||
|
ui.tsx # Shared UI components
|
||||||
|
helpers.ts # Shared admin helpers
|
||||||
|
lib/
|
||||||
|
cloudflare-email.ts # Cloudflare Email routing handler
|
||||||
|
email-parser.ts # Email parsing (mailparser)
|
||||||
|
email-processor.ts # Core ingestion logic (parse → store)
|
||||||
|
feed-fetcher.ts # KV feed/email fetch helpers
|
||||||
|
feed-generator.ts # RSS/Atom XML generation
|
||||||
|
forwardemail.ts # ForwardEmail webhook types/parsing
|
||||||
|
id-generator.ts # Feed/entry ID generation
|
||||||
|
logger.ts # JSON structured logger
|
||||||
|
storage.ts # KV key helpers
|
||||||
|
websub.ts # WebSub subscription management
|
||||||
|
worker.ts # Typed worker export helper
|
||||||
|
scripts/
|
||||||
|
client/ # TypeScript client scripts (compiled by esbuild)
|
||||||
|
dashboard.ts # Admin dashboard interactions
|
||||||
|
emails-page.ts # Emails page interactions
|
||||||
|
generated/ # Compiled output (gitignored, rebuilt on npm install)
|
||||||
|
styles/ # CSS files bundled into the Worker
|
||||||
|
variables.css
|
||||||
|
layout.css
|
||||||
|
components.css
|
||||||
|
utilities.css
|
||||||
|
data/nouns.ts # Word list for ID generation
|
||||||
|
test/setup.ts # Test mocks: MockKV, createMockEnv()
|
||||||
|
```
|
||||||
|
|
||||||
### KV schema
|
### KV schema
|
||||||
|
|
||||||
All data lives in the `EMAIL_STORAGE` KV namespace:
|
All data lives in the `EMAIL_STORAGE` KV namespace:
|
||||||
|
|
||||||
| Key | Value |
|
| Key | Value |
|
||||||
| --------------------------- | ------------------------------------------------- |
|
| -------------------------------- | ------------------------------------------------------------------------ |
|
||||||
| `feeds:list` | `{ feeds: Array<{ id, title, description? }> }` |
|
| `feeds:list` | `{ feeds: Array<{ id, title, description? }> }` |
|
||||||
| `feed:<feedId>:config` | `FeedConfig` object |
|
| `feed:<feedId>:config` | `FeedConfig` |
|
||||||
| `feed:<feedId>:metadata` | `{ emails: Array<{ key, subject, receivedAt }> }` |
|
| `feed:<feedId>:metadata` | `{ emails: Array<{ key, subject, receivedAt, size?, attachmentIds? }> }` |
|
||||||
| `feed:<feedId>:<timestamp>` | Full `EmailData` object |
|
| `feed:<feedId>:<timestamp>` | Full `EmailData` |
|
||||||
|
| `websub:<feedId>:<callbackHash>` | `WebSubSubscription` |
|
||||||
|
|
||||||
`src/utils/storage.ts` contains alternate key helpers not used by routes — keep route key usage consistent with the schema above.
|
`src/lib/storage.ts` contains key-builder helpers — use them; don't inline key strings in routes.
|
||||||
|
|
||||||
### Admin UI architecture
|
### Worker bindings (`Env`)
|
||||||
|
|
||||||
The admin UI is server-rendered HTML returned by `src/routes/admin.ts`. Interactive behavior (toast notifications, clipboard, bulk delete, table column resizing/sorting) is implemented as inline scripts in `src/scripts/`. Styles are composed from `src/styles/`. These are bundled into the Worker — there is no separate frontend build step.
|
```ts
|
||||||
|
EMAIL_STORAGE: KVNamespace; // All feed/email data
|
||||||
|
ADMIN_PASSWORD: string; // Worker secret — never in config files
|
||||||
|
DOMAIN: string; // e.g. "getmynews.app"
|
||||||
|
ATTACHMENT_BUCKET?: R2Bucket; // R2 for email attachments
|
||||||
|
FEED_MAX_SIZE_BYTES?: string; // Optional email size cap
|
||||||
|
PROXY_TRUSTED_IPS?: string; // Trusted reverse-proxy IPs
|
||||||
|
PROXY_AUTH_SECRET?: string; // Shared secret for proxy auth
|
||||||
|
```
|
||||||
|
|
||||||
|
### Client scripts
|
||||||
|
|
||||||
|
`src/scripts/client/` contains TypeScript that runs in the browser. It is compiled by esbuild into `src/scripts/generated/` (gitignored) and bundled into the Worker as inline `<script>` tags. The `prepare` npm hook rebuilds them on `npm install`. Run `npm run build:client` to rebuild manually.
|
||||||
|
|
||||||
### Testing
|
### Testing
|
||||||
|
|
||||||
@@ -65,25 +126,17 @@ Tests run in Node (not a Worker runtime). Hono test requests pass the mock env a
|
|||||||
const res = await app.request("/path", init, createMockEnv());
|
const res = await app.request("/path", init, createMockEnv());
|
||||||
```
|
```
|
||||||
|
|
||||||
MSW (`msw/node`) handles external HTTP mocks. Tests that hit validation paths intentionally produce stderr output — this is expected.
|
MSW (`msw/node`) handles external HTTP mocks. Tests that hit validation paths intentionally produce stderr output — expected.
|
||||||
|
|
||||||
### Worker environment bindings (`Env`)
|
|
||||||
|
|
||||||
```ts
|
|
||||||
EMAIL_STORAGE: KVNamespace; // KV namespace for all data
|
|
||||||
ADMIN_PASSWORD: string; // Cloudflare Worker secret (not in wrangler.toml)
|
|
||||||
DOMAIN: string; // e.g. "yourdomain.com"
|
|
||||||
```
|
|
||||||
|
|
||||||
## Configuration
|
## Configuration
|
||||||
|
|
||||||
- `wrangler.toml` is generated locally from `wrangler-example.toml` by `setup.sh` — do not commit `wrangler.toml`
|
- `wrangler.toml` is generated locally from `wrangler-example.toml` by `setup.sh` — do not commit it
|
||||||
- `ADMIN_PASSWORD` is a Cloudflare Worker secret set via `wrangler secret put`; it is never in config files
|
- `ADMIN_PASSWORD` is set via `wrangler secret put` — never in config files
|
||||||
- Keep `compatibility_date` current on runtime upgrades
|
- Keep `compatibility_date` current on runtime upgrades
|
||||||
|
|
||||||
## When changing behavior
|
## When changing behavior
|
||||||
|
|
||||||
Update these together:
|
Update together:
|
||||||
|
|
||||||
- `README.md`
|
- `README.md`
|
||||||
- `AGENTS.md`
|
- `AGENTS.md`
|
||||||
|
|||||||
Reference in New Issue
Block a user