Completely streamline setup experience for new users, add more documentation

.gitignore

@@ -25,4 +25,7 @@ yarn-error.log*
 
 # System
 .DS_Store
-Thumbs.db 
+Thumbs.db 
+
+# Cloudflare
+wrangler.toml

README.md

@@ -1,45 +1,88 @@
 # Email-to-RSS
 
-A modern service that turns email newsletters into RSS feeds, built with Cloudflare Workers. This service provides unique email addresses per feed, a front-end admin panel, and long-term storage of newsletters.
+A modern service that turns email newsletters into RSS feeds, built with Cloudflare Workers and ForwardEmail.net. This service provides unique email addresses per feed, a front-end admin panel, and long-term storage of newsletters.
+
+## Why Email to RSS?
+
+I love consolidating my newsletters into a centralized reading app like [Reeder](https://reederapp.com). They make for a better reading experience and prevents my email inbox from getting clogged with newsletters. However, Reeder requires RSS support and many email newsletters don't support native RSS feeds.
+
+There are some free services online that do the same thing (e.g. [kill-the-newsletter.com](kill-the-newsletter.com)), but there are several downsides to this approach:
+
+- **No long-term retention**: old RSS posts are deleted to save space.
+- **Risk of blocklisting**: being forced to use the same domain (@kill-the-newsletter.com) as everyone else increases the likelihood that an email newsletter can blocklist you from signing up.
+- **Self-hosting is non-trivial**: Kill The Newsletter is also [open source](https://github.com/leafac/kill-the-newsletter), but the self-hosting steps seem neither straightforward nor does it focus on exclusively leveraging free services.
+
+On the other hand, while Email-to-RSS isn't necessarily a one-click-deploy solution, it works just as well, is customized to your domain, and is completely free (except for the custom domain itself)!
 
 ## Features
 
-- **Minimal Dependencies**: Built using modern, web-friendly libraries without Node.js-specific dependencies
+- **Autogenerate Custom Emails**: Creates custom email addresses in the format `noun1.noun2.XY@yourdomain.com` for each feed
-- **Lightweight**: Entire worker bundle is only ~360KB (gzipped: ~65KB)
+- **ForwardEmail.net Integration**: Processes incoming emails via webhook with robust IP verification
-- **Email Processing**: Handles emails from ForwardEmail.net webhook
+- **Minimalist Email Parser**: Custom-built lightweight parser that works efficiently in edge environments
-- **RSS Generation**: Serves standards-compliant RSS feeds
+- **RSS Feed Generation**: Serves standards-compliant RSS feeds using the modern Feed library
-- **Admin Interface**: Simple management UI for feeds and emails
+- **Admin Dashboard**: Complete web UI for managing feeds and viewing emails
-- **Storage**: Uses Cloudflare KV for efficient, low-cost storage
+- **Secure Authentication**: Password-protected admin interface
-- **Deletion Support**: Email content can be removed from feeds, with cache updates
+- **Cloudflare KV Storage**: Efficient, low-cost storage solution for feed data
+- **Minimal Dependencies**: Built using modern, web-friendly libraries
+- **Lightweight**: Entire worker bundle is optimized for edge deployment
+- **Deletion Support**: Email content can be removed from feeds in the admin UI, with automatic cache updates
+
+## Setup
+
+1. Buy your custom domain. You can buy it directly from Cloudflare for convenience. If purchased elsewhere, add the domain to your Cloudflare account to manage the DNS there.
+2. Clone this repository.
+3. Open your command line in the cloned repo folder and run: `bash setup.sh` – this will install dependencies and set up the Cloudflare Worker/KV with your admin password.
+4. Set up your ForwardEmail.net account and configure it to forward to Cloudflare (replace `yourdomain.com` with your custom domain):
+
+   1. Sign up for a free account using any email (doesn't necessarily have to be the one used for the newsletter).
+   2. Add your custom domain.
+   3. To verify your domain, add the following DNS records to your Cloudflare DNS configuration:
+
+      | Type | Name | Content                                            | TTL  | Proxy Status | Notes                                    |
+      | ---- | ---- | -------------------------------------------------- | ---- | ------------ | ---------------------------------------- |
+      | MX   | @    | mx1.forwardemail.net                               | Auto | DNS only     | Set Priority to 10.                      |
+      | MX   | @    | mx2.forwardemail.net                               | Auto | DNS only     | Set Priority to 10.                      |
+      | TXT  | @    | "forward-email=https://yourdomain.com/api/inbound" | Auto | DNS only     | This forwards your emails to the webhook |
+      | TXT  | @    | "v=spf1 include:spf.forwardemail.net -all"         | Auto | DNS only     | Email security                           |
+
+5. Deploy with `npm run deploy`.
+6. Go to yourdomain.com to open up the admin panel and log in!
+
+Tip: If you're unsure about any of these steps, ask ChatGPT or Cursor to guide you through them.
 
 ## Architecture
 
 ### Email Flow
 
-1. A newsletter arrives at `apple.mountain.42@yourdomain.com` (feed ID format: noun1.noun2.XX)
+1. A newsletter email arrives at `apple.mountain.42@yourdomain.com` (feed ID format: noun1.noun2.XY).
-2. ForwardEmail.net forwards it to your Cloudflare Worker
+2. ForwardEmail.net forwards it to your Cloudflare Worker endpoint via webhook.
-3. The Worker parses the email, extracts content, and stores it in KV
+3. The Worker validates the request is from ForwardEmail.net based on IP address.
-4. The RSS feed is updated with the new content
+4. The email is parsed, content extracted, and stored in Cloudflare KV.
+5. Feed metadata is updated to include the new email.
+6. The RSS feed endpoint dynamically generates the feed from stored emails.
 
 ### Key Components
 
-- **Email Parser**: Lightweight custom parser that works in edge environments
+- **Email Parser**: Extracts content from ForwardEmail.net webhook payload
-- **Feed Generator**: Modern RSS feed generator with minimal dependencies
+- **Feed Generator**: Creates standard-compliant RSS feeds from stored emails
-- **Admin UI**: Simple interface to manage feeds and view emails
+- **Admin UI**: Interface for creating, viewing, and managing feeds
-- **ID Generator**: Creates memorable, collision-resistant feed IDs
+- **ID Generator**: Creates memorable, collision-resistant feed IDs using common nouns
-- **Data Store**: Organized module for common nouns used in ID generation
+- **Security Layer**: Validates webhook requests against ForwardEmail.net IP addresses
+- **Storage Manager**: Organized module for storing and retrieving data from KV
 
 ### Code Structure
 
-- **src/routes/**: API and UI route handlers
+- `src/routes/`: API and UI route handlers for inbound emails, RSS feeds, and admin panel
-- **src/utils/**: Utility functions including email parsing and ID generation
+- `src/utils/`: Utility functions including email parsing, feed generation, and ID creation
-- **src/data/**: Data files like the nouns list for feed IDs
+- `src/data/`: Data files including the nouns list used in ID generation
-- **src/types/**: TypeScript type definitions
+- `src/types/`: TypeScript type definitions
-- **src/index.ts**: Main application entry point
+- `src/scripts/`: Client-side JavaScript for the admin interface
+- `src/styles/`: CSS styling for the admin interface
+- `src/index.ts`: Main application entry point with middleware and routing configuration
 
 ## Development
 
-This project uses a modern build process with Wrangler's built-in bundling (powered by esbuild):
+This project uses a modern build process with Cloudflare Wrangler's built-in bundling (powered by `esbuild`):
 
 ```bash
 # Install dependencies
@@ -55,27 +98,14 @@ npm run build
 npm run deploy
 ```
 
-## Environment Variables
-
-- `ADMIN_PASSWORD`: Password for the admin interface
-- `DOMAIN`: Your custom domain for receiving emails
-
 ## Technology Stack
 
-- **Cloudflare Workers**: Edge computing platform
+- **Cloudflare Workers**: Edge computing platform for running the service
-- **Cloudflare KV**: Key-value storage
+- **Cloudflare KV**: Key-value storage for email and feed data
-- **Hono**: Lightweight web framework
+- **Hono**: Lightweight web framework for routing and middleware
-- **TypeScript**: Type-safe JavaScript
+- **TypeScript**: Type-safe JavaScript for reliable code
 - **Feed**: Modern RSS feed generator
-- **Zod**: Schema validation
+- **Zod**: Schema validation for input data
-
-## Setup
-
-1. Clone this repository
-2. Install dependencies with `npm install`
-3. Copy `wrangler.toml.example` to `wrangler.toml` and set your values
-4. Run `npm run dev` to start the development server
-5. Deploy with `npm run deploy`
 
 ## Minimalist Approach
 
@@ -90,17 +120,28 @@ This project follows a minimalist approach:
 
 ## Feed ID System
 
-The system generates memorable, user-friendly feed IDs in the format `noun1.noun2.XX` where:
+The system generates memorable, user-friendly feed IDs in the format `noun1.noun2.XY` where:
 
-- `noun1` and `noun2` are randomly selected from a curated list of ~500 common nouns
+- `noun1` and `noun2` are randomly selected from a curated list of ~450 common, neutral nouns
-- `XX` is a random two-digit number between 10 and 99
+- `XY` is a random two-digit number between 10 and 99
+
+This is inspired by iCloud's Hide My Email feature.
 
 This format provides:
 
 - Easy to read and share email addresses
 - Low collision probability (can handle thousands of feeds)
 - Simple to remember for users
-- ~22.5 million possible combinations
+- ~20 million possible combinations
+
+### Noun Selection
+
+The noun list has been carefully curated to:
+
+- Include only common, everyday objects and concepts
+- Exclude any potentially problematic terms
+- Ensure appropriate combinations when nouns are randomly paired
+- Maintain a professional appearance for all generated feed IDs
 
 ## License
 

package.json

@@ -7,7 +7,7 @@
     "build": "wrangler deploy --dry-run --outdir=dist",
     "format": "prettier --write '**/*.{js,ts,css,json,md}'",
     "dev": "wrangler dev",
-    "deploy": "wrangler deploy"
+    "deploy": "wrangler deploy --env production"
   },
   "author": "",
   "license": "MIT",

setup.sh

@@ -4,26 +4,189 @@
 
 echo "🚀 Setting up Email to RSS service..."
 
+# Check if npm and npx are installed
+if ! command -v npm &> /dev/null || ! command -v npx &> /dev/null; then
+  echo "❌ Error: npm and npx are required but not found."
+  echo "Please install Node.js from https://nodejs.org/en/download/"
+  exit 1
+fi
+
+# Check if wrangler-example.toml exists early
+if [ ! -f "wrangler-example.toml" ]; then
+  echo "❌ Error: wrangler-example.toml not found."
+  exit 1
+fi
+
 # Install dependencies
 echo "📦 Installing dependencies..."
 npm install
 
-# Create KV namespaces
+# Check if user is logged in to Cloudflare
+echo "🔒 Checking Cloudflare authentication..."
+if ! npx wrangler whoami &>/dev/null; then
+  echo "❌ You are not logged in to Cloudflare. Please run:"
+  echo "npx wrangler login"
+  echo "After login completes, run this setup script again."
+  exit 1
+fi
+echo "✅ Cloudflare authentication verified"
+
+# Function to get KV namespace IDs
+get_kv_namespace_ids() {
+  echo "ℹ️ Retrieving KV namespace IDs..."
+  
+  # Get the complete KV namespace list
+  local output
+  output=$(npx wrangler kv:namespace list 2>/dev/null)
+  
+  if [ $? -ne 0 ]; then
+    echo "❌ Error listing KV namespaces. Please check your Cloudflare authentication."
+    return 1
+  fi
+  
+  # Try the direct approach first (most reliable)
+  MAIN_ID=$(echo "$output" | grep -o '"id": *"[^"]*"' | head -1 | cut -d'"' -f4)
+  PREVIEW_ID=$(echo "$output" | grep -o '"id": *"[^"]*"' | head -2 | tail -1 | cut -d'"' -f4)
+  
+  # If the direct approach failed, try to match by namespace title
+  if [ -z "$MAIN_ID" ] || [ -z "$PREVIEW_ID" ]; then
+    # Save the output to a file for more complex processing
+    local temp_file=$(mktemp)
+    echo "$output" > "$temp_file"
+    
+    # Try with different patterns
+    if [ -z "$MAIN_ID" ]; then
+      MAIN_ID=$(grep -A3 "email-to-rss-EMAIL_STORAGE\"" "$temp_file" | grep -o '"id": "[^"]*"' | head -1 | cut -d'"' -f4)
+      if [ -z "$MAIN_ID" ]; then
+        MAIN_ID=$(grep -A3 "email-to-rss-EMAIL_STORAGE\"" "$temp_file" | grep -o '"id":"[^"]*"' | head -1 | cut -d'"' -f4)
+      fi
+    fi
+    
+    if [ -z "$PREVIEW_ID" ]; then
+      PREVIEW_ID=$(grep -A3 "email-to-rss-EMAIL_STORAGE_preview\"" "$temp_file" | grep -o '"id": "[^"]*"' | head -1 | cut -d'"' -f4)
+      if [ -z "$PREVIEW_ID" ]; then
+        PREVIEW_ID=$(grep -A3 "email-to-rss-EMAIL_STORAGE_preview\"" "$temp_file" | grep -o '"id":"[^"]*"' | head -1 | cut -d'"' -f4)
+      fi
+    fi
+    
+    # Clean up
+    rm -f "$temp_file"
+  fi
+  
+  # Check if we found both IDs
+  if [ -z "$MAIN_ID" ] || [ -z "$PREVIEW_ID" ]; then
+    echo "❌ Failed to extract KV namespace IDs. Please run manually:"
+    echo "npx wrangler kv:namespace list"
+    echo "And update the IDs in wrangler.toml"
+    return 1
+  fi
+  
+  return 0
+}
+
+# Create KV namespaces (suppressing output)
 echo "🗄️ Creating KV namespaces..."
-echo "You'll need to update wrangler.toml with these IDs."
+npx wrangler kv:namespace create EMAIL_STORAGE > /dev/null 2>&1 || true
-npx wrangler kv:namespace create EMAIL_STORAGE
+npx wrangler kv:namespace create EMAIL_STORAGE --preview > /dev/null 2>&1 || true
-npx wrangler kv:namespace create EMAIL_STORAGE --preview
+
+# Get KV namespace IDs
+get_kv_namespace_ids
+if [ $? -ne 0 ]; then
+  echo "⚠️ Will continue without KV namespace IDs"
+  KV_ID=""
+  KV_PREVIEW_ID=""
+else
+  KV_ID="$MAIN_ID"
+  KV_PREVIEW_ID="$PREVIEW_ID"
+fi
+
+# Summarize KV namespace status
+echo "📊 KV Namespace Status:"
+if [ -z "$KV_ID" ]; then
+  echo "  ❌ Main KV namespace ID: Not found"
+  SETUP_SUCCESS=false
+else
+  echo "  ✅ Main KV namespace ID: $KV_ID"
+  SETUP_SUCCESS=true
+fi
+
+if [ -z "$KV_PREVIEW_ID" ]; then
+  echo "  ❌ Preview KV namespace ID: Not found"
+  SETUP_SUCCESS=false
+else
+  echo "  ✅ Preview KV namespace ID: $KV_PREVIEW_ID"
+fi
 
 # Set up admin password
 echo "🔐 Setting up admin password..."
 read -p "Enter admin password: " admin_password
-echo "$admin_password" | npx wrangler secret put ADMIN_PASSWORD
+
+echo "Setting admin password for production environment..."
+# Initialize SETUP_SUCCESS if not already set
+if [ -z "$SETUP_SUCCESS" ]; then
+  SETUP_SUCCESS=true
+fi
+
+# Try to set the secret without redirecting stderr to see any errors
+if [ -z "$admin_password" ]; then
+  echo "⚠️ No admin password provided. Skipping secret creation."
+else
+  # Run the command and capture its output
+  SECRET_OUTPUT=$(echo "$admin_password" | npx wrangler secret put ADMIN_PASSWORD --env production --name email-to-rss 2>&1)
+  SECRET_STATUS=$?
+  
+  if [ $SECRET_STATUS -ne 0 ]; then
+    echo "⚠️ Failed to set admin password for production environment"
+    echo "Error: $SECRET_OUTPUT"
+    SETUP_SUCCESS=false
+  else
+    echo "✅ Admin password set for production environment"
+  fi
+fi
 
 # Prompt for domain
 read -p "Enter your domain (e.g., yourdomain.com): " domain
-echo "📝 Please update your domain in wrangler.toml"
+if [ -z "$domain" ]; then
+  echo "❌ No domain provided. Cannot continue."
+  SETUP_SUCCESS=false
+else
+  echo "✅ Domain: $domain"
+fi
 
+# Create and update wrangler.toml only if everything is successful
+if [ "$SETUP_SUCCESS" = false ]; then
+  echo "⚠️ Some parts of the setup failed. Will not create wrangler.toml."
+  echo "Please fix the issues and run the script again."
+  exit 1
+fi
+
+# Create and configure wrangler.toml
+echo "📝 Creating and configuring wrangler.toml..."
+cp wrangler-example.toml wrangler.toml
+
+# Update wrangler.toml with domain and KV IDs
+if [[ "$OSTYPE" == "darwin"* ]]; then
+  # macOS requires empty string for -i
+  sed -i '' "s/REPLACE_WITH_YOUR_DOMAIN/$domain/g" wrangler.toml
+  if [ ! -z "$KV_ID" ]; then
+    sed -i '' "s/REPLACE_WITH_YOUR_KV_NAMESPACE_ID/$KV_ID/g" wrangler.toml
+  fi
+  if [ ! -z "$KV_PREVIEW_ID" ]; then
+    sed -i '' "s/REPLACE_WITH_YOUR_PREVIEW_KV_NAMESPACE_ID/$KV_PREVIEW_ID/g" wrangler.toml
+  fi
+else
+  # Linux and others
+  sed -i "s/REPLACE_WITH_YOUR_DOMAIN/$domain/g" wrangler.toml
+  if [ ! -z "$KV_ID" ]; then
+    sed -i "s/REPLACE_WITH_YOUR_KV_NAMESPACE_ID/$KV_ID/g" wrangler.toml
+  fi
+  if [ ! -z "$KV_PREVIEW_ID" ]; then
+    sed -i "s/REPLACE_WITH_YOUR_PREVIEW_KV_NAMESPACE_ID/$KV_PREVIEW_ID/g" wrangler.toml
+  fi
+fi
+
+echo "✅ wrangler.toml has been created and configured successfully!"
+echo ""
 echo "✅ Setup complete! Next steps:"
-echo "1. Update wrangler.toml with your KV namespace IDs and domain"
+echo "1. Set up MX records for your domain with ForwardEmail.net (see README for more details)"
-echo "2. Set up MX records for your domain with ForwardEmail.net"
+echo "2. Deploy with 'npm run deploy'" 
-echo "3. Deploy with 'npm run deploy'" 

wrangler-example.toml

@@ -0,0 +1,33 @@
+name = "email-to-rss"
+main = "src/index.ts"
+compatibility_date = "2024-09-23"
+compatibility_flags = ["nodejs_compat"]
+
+# Global KV Namespace bindings
+kv_namespaces = [
+  { binding = "EMAIL_STORAGE", id = "REPLACE_WITH_YOUR_KV_NAMESPACE_ID", preview_id = "REPLACE_WITH_YOUR_PREVIEW_KV_NAMESPACE_ID" }
+]
+
+# Global Environment variables
+[vars]
+DOMAIN = "REPLACE_WITH_YOUR_DOMAIN" # Your custom domain for emails
+
+# Development environment
+[env.dev]
+workers_dev = true
+
+# Production environment
+[env.production]
+workers_dev = false
+
+kv_namespaces = [
+  { binding = "EMAIL_STORAGE", id = "REPLACE_WITH_YOUR_KV_NAMESPACE_ID" }
+]
+
+routes = [
+  { pattern = "REPLACE_WITH_YOUR_DOMAIN", custom_domain = true },
+  { pattern = "www.REPLACE_WITH_YOUR_DOMAIN", custom_domain = true }
+]
+
+[env.production.vars]
+DOMAIN = "REPLACE_WITH_YOUR_DOMAIN"

wrangler.toml

@@ -1,28 +0,0 @@
-name = "email-to-rss"
-main = "src/index.ts"
-compatibility_date = "2024-09-23"
-compatibility_flags = ["nodejs_compat"]
-
-# KV Namespace bindings
-kv_namespaces = [
-  { binding = "EMAIL_STORAGE", id = "721e2789af9a41eba56a77e7891fd85a", preview_id = "b741d3713dd0416ca80b34ae6539736e" }
-]
-
-# Environment variables
-[vars]
-ADMIN_PASSWORD = ""  # Set this using wrangler secret
-DOMAIN = "getmynews.app"  # Your custom domain for emails
-
-# Development environment
-[env.dev]
-# Add any development-specific configuration here
-workers_dev = true
-
-# Production environment
-[env.production]
-# Add any production-specific configuration here
-workers_dev = false
-routes = [
-  "https://getmynews.app/*",
-  "https://www.getmynews.app/*"
-]