diff --git a/.gitignore b/.gitignore index 062db28..a9f696c 100644 --- a/.gitignore +++ b/.gitignore @@ -25,4 +25,7 @@ yarn-error.log* # System .DS_Store -Thumbs.db \ No newline at end of file +Thumbs.db + +# Cloudflare +wrangler.toml diff --git a/README.md b/README.md index 4c44bf5..b85b163 100644 --- a/README.md +++ b/README.md @@ -1,45 +1,88 @@ # Email-to-RSS -A modern service that turns email newsletters into RSS feeds, built with Cloudflare Workers. This service provides unique email addresses per feed, a front-end admin panel, and long-term storage of newsletters. +A modern service that turns email newsletters into RSS feeds, built with Cloudflare Workers and ForwardEmail.net. This service provides unique email addresses per feed, a front-end admin panel, and long-term storage of newsletters. + +## Why Email to RSS? + +I love consolidating my newsletters into a centralized reading app like [Reeder](https://reederapp.com). They make for a better reading experience and prevents my email inbox from getting clogged with newsletters. However, Reeder requires RSS support and many email newsletters don't support native RSS feeds. + +There are some free services online that do the same thing (e.g. [kill-the-newsletter.com](kill-the-newsletter.com)), but there are several downsides to this approach: + +- **No long-term retention**: old RSS posts are deleted to save space. +- **Risk of blocklisting**: being forced to use the same domain (@kill-the-newsletter.com) as everyone else increases the likelihood that an email newsletter can blocklist you from signing up. +- **Self-hosting is non-trivial**: Kill The Newsletter is also [open source](https://github.com/leafac/kill-the-newsletter), but the self-hosting steps seem neither straightforward nor does it focus on exclusively leveraging free services. + +On the other hand, while Email-to-RSS isn't necessarily a one-click-deploy solution, it works just as well, is customized to your domain, and is completely free (except for the custom domain itself)! ## Features -- **Minimal Dependencies**: Built using modern, web-friendly libraries without Node.js-specific dependencies -- **Lightweight**: Entire worker bundle is only ~360KB (gzipped: ~65KB) -- **Email Processing**: Handles emails from ForwardEmail.net webhook -- **RSS Generation**: Serves standards-compliant RSS feeds -- **Admin Interface**: Simple management UI for feeds and emails -- **Storage**: Uses Cloudflare KV for efficient, low-cost storage -- **Deletion Support**: Email content can be removed from feeds, with cache updates +- **Autogenerate Custom Emails**: Creates custom email addresses in the format `noun1.noun2.XY@yourdomain.com` for each feed +- **ForwardEmail.net Integration**: Processes incoming emails via webhook with robust IP verification +- **Minimalist Email Parser**: Custom-built lightweight parser that works efficiently in edge environments +- **RSS Feed Generation**: Serves standards-compliant RSS feeds using the modern Feed library +- **Admin Dashboard**: Complete web UI for managing feeds and viewing emails +- **Secure Authentication**: Password-protected admin interface +- **Cloudflare KV Storage**: Efficient, low-cost storage solution for feed data +- **Minimal Dependencies**: Built using modern, web-friendly libraries +- **Lightweight**: Entire worker bundle is optimized for edge deployment +- **Deletion Support**: Email content can be removed from feeds in the admin UI, with automatic cache updates + +## Setup + +1. Buy your custom domain. You can buy it directly from Cloudflare for convenience. If purchased elsewhere, add the domain to your Cloudflare account to manage the DNS there. +2. Clone this repository. +3. Open your command line in the cloned repo folder and run: `bash setup.sh` – this will install dependencies and set up the Cloudflare Worker/KV with your admin password. +4. Set up your ForwardEmail.net account and configure it to forward to Cloudflare (replace `yourdomain.com` with your custom domain): + + 1. Sign up for a free account using any email (doesn't necessarily have to be the one used for the newsletter). + 2. Add your custom domain. + 3. To verify your domain, add the following DNS records to your Cloudflare DNS configuration: + + | Type | Name | Content | TTL | Proxy Status | Notes | + | ---- | ---- | -------------------------------------------------- | ---- | ------------ | ---------------------------------------- | + | MX | @ | mx1.forwardemail.net | Auto | DNS only | Set Priority to 10. | + | MX | @ | mx2.forwardemail.net | Auto | DNS only | Set Priority to 10. | + | TXT | @ | "forward-email=https://yourdomain.com/api/inbound" | Auto | DNS only | This forwards your emails to the webhook | + | TXT | @ | "v=spf1 include:spf.forwardemail.net -all" | Auto | DNS only | Email security | + +5. Deploy with `npm run deploy`. +6. Go to yourdomain.com to open up the admin panel and log in! + +Tip: If you're unsure about any of these steps, ask ChatGPT or Cursor to guide you through them. ## Architecture ### Email Flow -1. A newsletter arrives at `apple.mountain.42@yourdomain.com` (feed ID format: noun1.noun2.XX) -2. ForwardEmail.net forwards it to your Cloudflare Worker -3. The Worker parses the email, extracts content, and stores it in KV -4. The RSS feed is updated with the new content +1. A newsletter email arrives at `apple.mountain.42@yourdomain.com` (feed ID format: noun1.noun2.XY). +2. ForwardEmail.net forwards it to your Cloudflare Worker endpoint via webhook. +3. The Worker validates the request is from ForwardEmail.net based on IP address. +4. The email is parsed, content extracted, and stored in Cloudflare KV. +5. Feed metadata is updated to include the new email. +6. The RSS feed endpoint dynamically generates the feed from stored emails. ### Key Components -- **Email Parser**: Lightweight custom parser that works in edge environments -- **Feed Generator**: Modern RSS feed generator with minimal dependencies -- **Admin UI**: Simple interface to manage feeds and view emails -- **ID Generator**: Creates memorable, collision-resistant feed IDs -- **Data Store**: Organized module for common nouns used in ID generation +- **Email Parser**: Extracts content from ForwardEmail.net webhook payload +- **Feed Generator**: Creates standard-compliant RSS feeds from stored emails +- **Admin UI**: Interface for creating, viewing, and managing feeds +- **ID Generator**: Creates memorable, collision-resistant feed IDs using common nouns +- **Security Layer**: Validates webhook requests against ForwardEmail.net IP addresses +- **Storage Manager**: Organized module for storing and retrieving data from KV ### Code Structure -- **src/routes/**: API and UI route handlers -- **src/utils/**: Utility functions including email parsing and ID generation -- **src/data/**: Data files like the nouns list for feed IDs -- **src/types/**: TypeScript type definitions -- **src/index.ts**: Main application entry point +- `src/routes/`: API and UI route handlers for inbound emails, RSS feeds, and admin panel +- `src/utils/`: Utility functions including email parsing, feed generation, and ID creation +- `src/data/`: Data files including the nouns list used in ID generation +- `src/types/`: TypeScript type definitions +- `src/scripts/`: Client-side JavaScript for the admin interface +- `src/styles/`: CSS styling for the admin interface +- `src/index.ts`: Main application entry point with middleware and routing configuration ## Development -This project uses a modern build process with Wrangler's built-in bundling (powered by esbuild): +This project uses a modern build process with Cloudflare Wrangler's built-in bundling (powered by `esbuild`): ```bash # Install dependencies @@ -55,27 +98,14 @@ npm run build npm run deploy ``` -## Environment Variables - -- `ADMIN_PASSWORD`: Password for the admin interface -- `DOMAIN`: Your custom domain for receiving emails - ## Technology Stack -- **Cloudflare Workers**: Edge computing platform -- **Cloudflare KV**: Key-value storage -- **Hono**: Lightweight web framework -- **TypeScript**: Type-safe JavaScript +- **Cloudflare Workers**: Edge computing platform for running the service +- **Cloudflare KV**: Key-value storage for email and feed data +- **Hono**: Lightweight web framework for routing and middleware +- **TypeScript**: Type-safe JavaScript for reliable code - **Feed**: Modern RSS feed generator -- **Zod**: Schema validation - -## Setup - -1. Clone this repository -2. Install dependencies with `npm install` -3. Copy `wrangler.toml.example` to `wrangler.toml` and set your values -4. Run `npm run dev` to start the development server -5. Deploy with `npm run deploy` +- **Zod**: Schema validation for input data ## Minimalist Approach @@ -90,17 +120,28 @@ This project follows a minimalist approach: ## Feed ID System -The system generates memorable, user-friendly feed IDs in the format `noun1.noun2.XX` where: +The system generates memorable, user-friendly feed IDs in the format `noun1.noun2.XY` where: -- `noun1` and `noun2` are randomly selected from a curated list of ~500 common nouns -- `XX` is a random two-digit number between 10 and 99 +- `noun1` and `noun2` are randomly selected from a curated list of ~450 common, neutral nouns +- `XY` is a random two-digit number between 10 and 99 + +This is inspired by iCloud's Hide My Email feature. This format provides: - Easy to read and share email addresses - Low collision probability (can handle thousands of feeds) - Simple to remember for users -- ~22.5 million possible combinations +- ~20 million possible combinations + +### Noun Selection + +The noun list has been carefully curated to: + +- Include only common, everyday objects and concepts +- Exclude any potentially problematic terms +- Ensure appropriate combinations when nouns are randomly paired +- Maintain a professional appearance for all generated feed IDs ## License diff --git a/package.json b/package.json index a79b739..0664857 100644 --- a/package.json +++ b/package.json @@ -7,7 +7,7 @@ "build": "wrangler deploy --dry-run --outdir=dist", "format": "prettier --write '**/*.{js,ts,css,json,md}'", "dev": "wrangler dev", - "deploy": "wrangler deploy" + "deploy": "wrangler deploy --env production" }, "author": "", "license": "MIT", diff --git a/setup.sh b/setup.sh index ff1622d..9215316 100755 --- a/setup.sh +++ b/setup.sh @@ -4,26 +4,189 @@ echo "🚀 Setting up Email to RSS service..." +# Check if npm and npx are installed +if ! command -v npm &> /dev/null || ! command -v npx &> /dev/null; then + echo "❌ Error: npm and npx are required but not found." + echo "Please install Node.js from https://nodejs.org/en/download/" + exit 1 +fi + +# Check if wrangler-example.toml exists early +if [ ! -f "wrangler-example.toml" ]; then + echo "❌ Error: wrangler-example.toml not found." + exit 1 +fi + # Install dependencies echo "đŸ“Ļ Installing dependencies..." npm install -# Create KV namespaces +# Check if user is logged in to Cloudflare +echo "🔒 Checking Cloudflare authentication..." +if ! npx wrangler whoami &>/dev/null; then + echo "❌ You are not logged in to Cloudflare. Please run:" + echo "npx wrangler login" + echo "After login completes, run this setup script again." + exit 1 +fi +echo "✅ Cloudflare authentication verified" + +# Function to get KV namespace IDs +get_kv_namespace_ids() { + echo "ℹī¸ Retrieving KV namespace IDs..." + + # Get the complete KV namespace list + local output + output=$(npx wrangler kv:namespace list 2>/dev/null) + + if [ $? -ne 0 ]; then + echo "❌ Error listing KV namespaces. Please check your Cloudflare authentication." + return 1 + fi + + # Try the direct approach first (most reliable) + MAIN_ID=$(echo "$output" | grep -o '"id": *"[^"]*"' | head -1 | cut -d'"' -f4) + PREVIEW_ID=$(echo "$output" | grep -o '"id": *"[^"]*"' | head -2 | tail -1 | cut -d'"' -f4) + + # If the direct approach failed, try to match by namespace title + if [ -z "$MAIN_ID" ] || [ -z "$PREVIEW_ID" ]; then + # Save the output to a file for more complex processing + local temp_file=$(mktemp) + echo "$output" > "$temp_file" + + # Try with different patterns + if [ -z "$MAIN_ID" ]; then + MAIN_ID=$(grep -A3 "email-to-rss-EMAIL_STORAGE\"" "$temp_file" | grep -o '"id": "[^"]*"' | head -1 | cut -d'"' -f4) + if [ -z "$MAIN_ID" ]; then + MAIN_ID=$(grep -A3 "email-to-rss-EMAIL_STORAGE\"" "$temp_file" | grep -o '"id":"[^"]*"' | head -1 | cut -d'"' -f4) + fi + fi + + if [ -z "$PREVIEW_ID" ]; then + PREVIEW_ID=$(grep -A3 "email-to-rss-EMAIL_STORAGE_preview\"" "$temp_file" | grep -o '"id": "[^"]*"' | head -1 | cut -d'"' -f4) + if [ -z "$PREVIEW_ID" ]; then + PREVIEW_ID=$(grep -A3 "email-to-rss-EMAIL_STORAGE_preview\"" "$temp_file" | grep -o '"id":"[^"]*"' | head -1 | cut -d'"' -f4) + fi + fi + + # Clean up + rm -f "$temp_file" + fi + + # Check if we found both IDs + if [ -z "$MAIN_ID" ] || [ -z "$PREVIEW_ID" ]; then + echo "❌ Failed to extract KV namespace IDs. Please run manually:" + echo "npx wrangler kv:namespace list" + echo "And update the IDs in wrangler.toml" + return 1 + fi + + return 0 +} + +# Create KV namespaces (suppressing output) echo "🗄ī¸ Creating KV namespaces..." -echo "You'll need to update wrangler.toml with these IDs." -npx wrangler kv:namespace create EMAIL_STORAGE -npx wrangler kv:namespace create EMAIL_STORAGE --preview +npx wrangler kv:namespace create EMAIL_STORAGE > /dev/null 2>&1 || true +npx wrangler kv:namespace create EMAIL_STORAGE --preview > /dev/null 2>&1 || true + +# Get KV namespace IDs +get_kv_namespace_ids +if [ $? -ne 0 ]; then + echo "⚠ī¸ Will continue without KV namespace IDs" + KV_ID="" + KV_PREVIEW_ID="" +else + KV_ID="$MAIN_ID" + KV_PREVIEW_ID="$PREVIEW_ID" +fi + +# Summarize KV namespace status +echo "📊 KV Namespace Status:" +if [ -z "$KV_ID" ]; then + echo " ❌ Main KV namespace ID: Not found" + SETUP_SUCCESS=false +else + echo " ✅ Main KV namespace ID: $KV_ID" + SETUP_SUCCESS=true +fi + +if [ -z "$KV_PREVIEW_ID" ]; then + echo " ❌ Preview KV namespace ID: Not found" + SETUP_SUCCESS=false +else + echo " ✅ Preview KV namespace ID: $KV_PREVIEW_ID" +fi # Set up admin password echo "🔐 Setting up admin password..." read -p "Enter admin password: " admin_password -echo "$admin_password" | npx wrangler secret put ADMIN_PASSWORD + +echo "Setting admin password for production environment..." +# Initialize SETUP_SUCCESS if not already set +if [ -z "$SETUP_SUCCESS" ]; then + SETUP_SUCCESS=true +fi + +# Try to set the secret without redirecting stderr to see any errors +if [ -z "$admin_password" ]; then + echo "⚠ī¸ No admin password provided. Skipping secret creation." +else + # Run the command and capture its output + SECRET_OUTPUT=$(echo "$admin_password" | npx wrangler secret put ADMIN_PASSWORD --env production --name email-to-rss 2>&1) + SECRET_STATUS=$? + + if [ $SECRET_STATUS -ne 0 ]; then + echo "⚠ī¸ Failed to set admin password for production environment" + echo "Error: $SECRET_OUTPUT" + SETUP_SUCCESS=false + else + echo "✅ Admin password set for production environment" + fi +fi # Prompt for domain read -p "Enter your domain (e.g., yourdomain.com): " domain -echo "📝 Please update your domain in wrangler.toml" +if [ -z "$domain" ]; then + echo "❌ No domain provided. Cannot continue." + SETUP_SUCCESS=false +else + echo "✅ Domain: $domain" +fi +# Create and update wrangler.toml only if everything is successful +if [ "$SETUP_SUCCESS" = false ]; then + echo "⚠ī¸ Some parts of the setup failed. Will not create wrangler.toml." + echo "Please fix the issues and run the script again." + exit 1 +fi + +# Create and configure wrangler.toml +echo "📝 Creating and configuring wrangler.toml..." +cp wrangler-example.toml wrangler.toml + +# Update wrangler.toml with domain and KV IDs +if [[ "$OSTYPE" == "darwin"* ]]; then + # macOS requires empty string for -i + sed -i '' "s/REPLACE_WITH_YOUR_DOMAIN/$domain/g" wrangler.toml + if [ ! -z "$KV_ID" ]; then + sed -i '' "s/REPLACE_WITH_YOUR_KV_NAMESPACE_ID/$KV_ID/g" wrangler.toml + fi + if [ ! -z "$KV_PREVIEW_ID" ]; then + sed -i '' "s/REPLACE_WITH_YOUR_PREVIEW_KV_NAMESPACE_ID/$KV_PREVIEW_ID/g" wrangler.toml + fi +else + # Linux and others + sed -i "s/REPLACE_WITH_YOUR_DOMAIN/$domain/g" wrangler.toml + if [ ! -z "$KV_ID" ]; then + sed -i "s/REPLACE_WITH_YOUR_KV_NAMESPACE_ID/$KV_ID/g" wrangler.toml + fi + if [ ! -z "$KV_PREVIEW_ID" ]; then + sed -i "s/REPLACE_WITH_YOUR_PREVIEW_KV_NAMESPACE_ID/$KV_PREVIEW_ID/g" wrangler.toml + fi +fi + +echo "✅ wrangler.toml has been created and configured successfully!" +echo "" echo "✅ Setup complete! Next steps:" -echo "1. Update wrangler.toml with your KV namespace IDs and domain" -echo "2. Set up MX records for your domain with ForwardEmail.net" -echo "3. Deploy with 'npm run deploy'" \ No newline at end of file +echo "1. Set up MX records for your domain with ForwardEmail.net (see README for more details)" +echo "2. Deploy with 'npm run deploy'" \ No newline at end of file diff --git a/wrangler-example.toml b/wrangler-example.toml new file mode 100644 index 0000000..88dc4c4 --- /dev/null +++ b/wrangler-example.toml @@ -0,0 +1,33 @@ +name = "email-to-rss" +main = "src/index.ts" +compatibility_date = "2024-09-23" +compatibility_flags = ["nodejs_compat"] + +# Global KV Namespace bindings +kv_namespaces = [ + { binding = "EMAIL_STORAGE", id = "REPLACE_WITH_YOUR_KV_NAMESPACE_ID", preview_id = "REPLACE_WITH_YOUR_PREVIEW_KV_NAMESPACE_ID" } +] + +# Global Environment variables +[vars] +DOMAIN = "REPLACE_WITH_YOUR_DOMAIN" # Your custom domain for emails + +# Development environment +[env.dev] +workers_dev = true + +# Production environment +[env.production] +workers_dev = false + +kv_namespaces = [ + { binding = "EMAIL_STORAGE", id = "REPLACE_WITH_YOUR_KV_NAMESPACE_ID" } +] + +routes = [ + { pattern = "REPLACE_WITH_YOUR_DOMAIN", custom_domain = true }, + { pattern = "www.REPLACE_WITH_YOUR_DOMAIN", custom_domain = true } +] + +[env.production.vars] +DOMAIN = "REPLACE_WITH_YOUR_DOMAIN" \ No newline at end of file diff --git a/wrangler.toml b/wrangler.toml deleted file mode 100644 index 59c1b09..0000000 --- a/wrangler.toml +++ /dev/null @@ -1,28 +0,0 @@ -name = "email-to-rss" -main = "src/index.ts" -compatibility_date = "2024-09-23" -compatibility_flags = ["nodejs_compat"] - -# KV Namespace bindings -kv_namespaces = [ - { binding = "EMAIL_STORAGE", id = "721e2789af9a41eba56a77e7891fd85a", preview_id = "b741d3713dd0416ca80b34ae6539736e" } -] - -# Environment variables -[vars] -ADMIN_PASSWORD = "" # Set this using wrangler secret -DOMAIN = "getmynews.app" # Your custom domain for emails - -# Development environment -[env.dev] -# Add any development-specific configuration here -workers_dev = true - -# Production environment -[env.production] -# Add any production-specific configuration here -workers_dev = false -routes = [ - "https://getmynews.app/*", - "https://www.getmynews.app/*" -] \ No newline at end of file