feat: store email attachments in R2 and expose as RSS enclosures

Attachments from incoming emails are uploaded to an optional Cloudflare R2
bucket and exposed as <enclosure> elements in RSS and <link rel="enclosure">
in Atom feeds, served at /files/{id}/{filename} with immutable caching.

R2 is opt-in: if ATTACHMENT_BUCKET is not bound the feature is a no-op.
Attachments are cleaned up from R2 on email/feed deletion and during
size-based feed trimming. Adds MockR2 to the test setup.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Julien Herr
2026-05-21 09:09:37 +02:00
parent 3e28246c61
commit e93bbb8d3e
15 changed files with 615 additions and 19 deletions
+35
View File
@@ -21,6 +21,8 @@ Email-to-RSS keeps the same workflow while avoiding shared domains and shared da
- ForwardEmail webhook ingestion with source-IP verification (optional alternative) - ForwardEmail webhook ingestion with source-IP verification (optional alternative)
- Optional per-feed sender allowlist (`email@domain.com` or `domain.com`) - Optional per-feed sender allowlist (`email@domain.com` or `domain.com`)
- RSS generation on demand (`/rss/:feedId`) - RSS generation on demand (`/rss/:feedId`)
- Atom feed at `/atom/:feedId`
- Email attachments stored in Cloudflare R2 and exposed as RSS enclosures (optional)
- Cloudflare KV storage for feed config + email metadata/content - Cloudflare KV storage for feed config + email metadata/content
- Password-protected admin UI - Password-protected admin UI
@@ -45,6 +47,8 @@ Main routes:
- `src/lib/cloudflare-email.ts`: Cloudflare Email Workers ingestion - `src/lib/cloudflare-email.ts`: Cloudflare Email Workers ingestion
- `src/routes/inbound.ts`: ForwardEmail webhook ingestion - `src/routes/inbound.ts`: ForwardEmail webhook ingestion
- `src/routes/rss.ts`: RSS rendering - `src/routes/rss.ts`: RSS rendering
- `src/routes/atom.ts`: Atom feed rendering
- `src/routes/files.ts`: attachment file serving from R2
- `src/routes/admin.ts`: admin UI + feed CRUD - `src/routes/admin.ts`: admin UI + feed CRUD
## Requirements ## Requirements
@@ -141,6 +145,32 @@ To override the threshold, add to `wrangler.toml` under `[vars]`:
FEED_MAX_SIZE_BYTES = "524288" # 512 KB — adjust as needed FEED_MAX_SIZE_BYTES = "524288" # 512 KB — adjust as needed
``` ```
### Email attachments (R2)
When an incoming email contains attachments, the Worker can store them in a Cloudflare R2 bucket and expose them as `<enclosure>` elements in the RSS feed (and `<link rel="enclosure">` in Atom). Each attachment is served at `/files/{id}/{filename}` with an immutable cache header.
This feature is **optional**. If no R2 bucket is bound, attachments are silently ignored and nothing else changes.
**Setup:**
1. Create an R2 bucket in the Cloudflare dashboard (*R2 Object Storage → Create bucket*), or with Wrangler:
```bash
npx wrangler r2 bucket create your-bucket-name
```
2. In `wrangler.toml`, uncomment and fill in the R2 binding (the commented block from `wrangler-example.toml`):
```toml
r2_buckets = [
{ binding = "ATTACHMENT_BUCKET", bucket_name = "your-bucket-name", preview_bucket_name = "your-bucket-name-preview" }
]
```
Do the same under `[env.production]` (without `preview_bucket_name`).
3. Redeploy:
```bash
npm run deploy
```
Attachments are deleted from R2 automatically when the corresponding email is deleted from the admin UI, or when an email is dropped during feed size trimming.
### External auth provider (Authelia / Authentik / reverse proxy) ### External auth provider (Authelia / Authentik / reverse proxy)
Instead of the built-in password login you can delegate admin authentication to a reverse proxy that sets a trusted user header (`Remote-User` or `X-Forwarded-User`). Instead of the built-in password login you can delegate admin authentication to a reverse proxy that sets a trusted user header (`Remote-User` or `X-Forwarded-User`).
@@ -188,6 +218,11 @@ npm run build
Then update `compatibility_date` and redeploy. Then update `compatibility_date` and redeploy.
## Acknowledgements
- [kill-the-newsletter](https://github.com/leafac/kill-the-newsletter) by Leandro Facchinetti — the inspiration for this project and the reference implementation for feature ideas (Atom feeds, attachment enclosures, entry HTML views, and more).
- [Email-to-RSS](https://github.com/yl8976/Email-to-RSS) by yl8976 — the initial codebase this project is based on.
## License ## License
MIT MIT
+1 -1
View File
@@ -20,6 +20,6 @@ Feature gaps identified by comparing with [kill-the-newsletter](https://github.c
## Heavy ## Heavy
- [ ] **Email attachments as RSS enclosures** — store attachments in Cloudflare R2 and expose them as `<enclosure>` elements in the feed. kill-the-newsletter serves them at `/files/{enclosureId}/{filename}`. - [x] **Email attachments as RSS enclosures** — store attachments in Cloudflare R2 and expose them as `<enclosure>` elements in the feed. kill-the-newsletter serves them at `/files/{enclosureId}/{filename}`.
- [ ] **WebSub (PubSubHubbub) push notifications** — notify subscribers in real time when a new email arrives, instead of requiring them to poll the feed. Requires either integrating a public WebSub hub or implementing the hub protocol directly. - [ ] **WebSub (PubSubHubbub) push notifications** — notify subscribers in real time when a new email arrives, instead of requiring them to poll the feed. Requires either integrating a public WebSub hub or implementing the hub protocol directly.
+6
View File
@@ -4,6 +4,7 @@ import { handle as handleRSS } from "./routes/rss";
import { handle as handleAtom } from "./routes/atom"; import { handle as handleAtom } from "./routes/atom";
import { handle as handleAdmin } from "./routes/admin"; import { handle as handleAdmin } from "./routes/admin";
import { handle as handleEntry } from "./routes/entries"; import { handle as handleEntry } from "./routes/entries";
import { handle as handleFiles } from "./routes/files";
import { handleCloudflareEmail } from "./lib/cloudflare-email"; import { handleCloudflareEmail } from "./lib/cloudflare-email";
import { Env } from "./types"; import { Env } from "./types";
@@ -105,6 +106,7 @@ const api = new Hono();
const rss = new Hono(); const rss = new Hono();
const atom = new Hono(); const atom = new Hono();
const entries = new Hono(); const entries = new Hono();
const files = new Hono();
const admin = new Hono(); const admin = new Hono();
// Webhook security middleware for /inbound - verify ForwardEmail.net IP // Webhook security middleware for /inbound - verify ForwardEmail.net IP
@@ -141,6 +143,9 @@ atom.get("/:feedId", handleAtom);
// Email entry HTML view (public) // Email entry HTML view (public)
entries.get("/:feedId/:entryId", handleEntry); entries.get("/:feedId/:entryId", handleEntry);
// Attachment file serving (public)
files.get("/:attachmentId/:filename", handleFiles);
// Admin routes (protected) // Admin routes (protected)
admin.route("/", handleAdmin); admin.route("/", handleAdmin);
@@ -149,6 +154,7 @@ app.route("/api", api);
app.route("/rss", rss); app.route("/rss", rss);
app.route("/atom", atom); app.route("/atom", atom);
app.route("/entries", entries); app.route("/entries", entries);
app.route("/files", files);
app.route("/admin", admin); app.route("/admin", admin);
// Root path redirects to admin dashboard // Root path redirects to admin dashboard
+10 -1
View File
@@ -1,6 +1,6 @@
import PostalMime from "postal-mime"; import PostalMime from "postal-mime";
import { Env } from "../types"; import { Env } from "../types";
import { processEmail } from "./email-processor"; import { processEmail, RawAttachment } from "./email-processor";
export async function handleCloudflareEmail( export async function handleCloudflareEmail(
message: ForwardableEmailMessage, message: ForwardableEmailMessage,
@@ -21,6 +21,14 @@ export async function handleCloudflareEmail(
headers[h.key] = h.value; headers[h.key] = h.value;
} }
const rawAttachments: RawAttachment[] = (email.attachments ?? [])
.filter((a) => a.content instanceof ArrayBuffer)
.map((a) => ({
filename: a.filename || "attachment",
contentType: a.mimeType || "application/octet-stream",
content: a.content as ArrayBuffer,
}));
await processEmail( await processEmail(
{ {
toAddress: message.to, toAddress: message.to,
@@ -30,6 +38,7 @@ export async function handleCloudflareEmail(
content: email.html ?? email.text ?? "", content: email.html ?? email.text ?? "",
receivedAt: email.date ? new Date(email.date).getTime() : Date.now(), receivedAt: email.date ? new Date(email.date).getTime() : Date.now(),
headers, headers,
attachments: rawAttachments,
}, },
env, env,
); );
+125 -2
View File
@@ -1,7 +1,7 @@
import { describe, it, expect, beforeEach } from "vitest"; import { describe, it, expect, beforeEach } from "vitest";
import "../test/setup"; import "../test/setup";
import { createMockEnv } from "../test/setup"; import { createMockEnv, MockR2 } from "../test/setup";
import { processEmail, ProcessEmailInput } from "./email-processor"; import { processEmail, ProcessEmailInput, RawAttachment } from "./email-processor";
const VALID_FEED_ID = "apple.mountain.42"; const VALID_FEED_ID = "apple.mountain.42";
const VALID_TO = `${VALID_FEED_ID}@test.getmynews.app`; const VALID_TO = `${VALID_FEED_ID}@test.getmynews.app`;
@@ -179,3 +179,126 @@ describe("processEmail", () => {
expect(metadata.emails).toHaveLength(2); expect(metadata.emails).toHaveLength(2);
}); });
}); });
describe("processEmail — attachments", () => {
const pdfContent = new TextEncoder().encode("PDF bytes").buffer as ArrayBuffer;
const pdfAttachment: RawAttachment = {
filename: "report.pdf",
contentType: "application/pdf",
content: pdfContent,
};
it("skips R2 upload when ATTACHMENT_BUCKET is not configured", async () => {
const env = createMockEnv();
await env.EMAIL_STORAGE.put(
`feed:${VALID_FEED_ID}:config`,
JSON.stringify({}),
);
const res = await processEmail(
makeInput({ attachments: [pdfAttachment] }),
env as any,
);
expect(res.status).toBe(200);
const metadata = await env.EMAIL_STORAGE.get(
`feed:${VALID_FEED_ID}:metadata`,
"json",
);
const emailData = await env.EMAIL_STORAGE.get(metadata.emails[0].key, "json");
expect(emailData.attachments).toBeUndefined();
});
it("uploads attachments to R2 and stores AttachmentData in emailData", async () => {
const env = createMockEnv({ withR2: true });
const mockR2 = (env as any).ATTACHMENT_BUCKET as unknown as MockR2;
await env.EMAIL_STORAGE.put(
`feed:${VALID_FEED_ID}:config`,
JSON.stringify({}),
);
const res = await processEmail(
makeInput({ attachments: [pdfAttachment] }),
env as any,
);
expect(res.status).toBe(200);
const metadata = await env.EMAIL_STORAGE.get(
`feed:${VALID_FEED_ID}:metadata`,
"json",
);
const emailData = await env.EMAIL_STORAGE.get(metadata.emails[0].key, "json");
expect(emailData.attachments).toHaveLength(1);
expect(emailData.attachments[0].filename).toBe("report.pdf");
expect(emailData.attachments[0].contentType).toBe("application/pdf");
expect(emailData.attachments[0].size).toBe(pdfContent.byteLength);
const id = emailData.attachments[0].id;
expect(mockR2._has(id)).toBe(true);
});
it("stores attachmentIds in EmailMetadata for trim-time cleanup", async () => {
const env = createMockEnv({ withR2: true });
await env.EMAIL_STORAGE.put(
`feed:${VALID_FEED_ID}:config`,
JSON.stringify({}),
);
await processEmail(makeInput({ attachments: [pdfAttachment] }), env as any);
const metadata = await env.EMAIL_STORAGE.get(
`feed:${VALID_FEED_ID}:metadata`,
"json",
);
expect(metadata.emails[0].attachmentIds).toHaveLength(1);
expect(typeof metadata.emails[0].attachmentIds[0]).toBe("string");
});
it("deletes R2 objects when a trimmed email had attachments", async () => {
const env = createMockEnv({ withR2: true });
const mockR2 = (env as any).ATTACHMENT_BUCKET as unknown as MockR2;
await env.EMAIL_STORAGE.put(
`feed:${VALID_FEED_ID}:config`,
JSON.stringify({}),
);
// Store an old email with attachment in KV and metadata
const oldKey = `feed:${VALID_FEED_ID}:111`;
const oldAttachmentId = "old-attachment-uuid";
const bigContent = "x".repeat(200);
const oldEmail = JSON.stringify({
subject: "Old",
from: "a@b.com",
content: bigContent,
receivedAt: 111,
headers: {},
attachments: [{ id: oldAttachmentId, filename: "old.pdf", contentType: "application/pdf", size: 100 }],
});
await env.EMAIL_STORAGE.put(oldKey, oldEmail);
// Also put the attachment in mock R2
await mockR2.put(oldAttachmentId, new ArrayBuffer(100));
await env.EMAIL_STORAGE.put(
`feed:${VALID_FEED_ID}:metadata`,
JSON.stringify({
emails: [
{
key: oldKey,
subject: "Old",
receivedAt: 111,
size: oldEmail.length,
attachmentIds: [oldAttachmentId],
},
],
}),
);
// Process with tight size budget to force trimming
const tinyEnv = { ...env, FEED_MAX_SIZE_BYTES: "50" };
const res = await processEmail(makeInput({ subject: "New" }), tinyEnv as any);
expect(res.status).toBe(200);
// Old attachment should be deleted from R2
expect(mockR2._has(oldAttachmentId)).toBe(false);
});
});
+54 -6
View File
@@ -1,5 +1,11 @@
import { EmailParser } from "../utils/email-parser"; import { EmailParser } from "../utils/email-parser";
import { Env, FeedConfig, FeedMetadata } from "../types"; import { AttachmentData, EmailMetadata, Env, FeedConfig, FeedMetadata } from "../types";
export interface RawAttachment {
filename: string;
contentType: string;
content: ArrayBuffer;
}
export interface ProcessEmailInput { export interface ProcessEmailInput {
toAddress: string; toAddress: string;
@@ -9,6 +15,7 @@ export interface ProcessEmailInput {
content: string; content: string;
receivedAt: number; receivedAt: number;
headers?: Record<string, string>; headers?: Record<string, string>;
attachments?: RawAttachment[];
} }
function normalizeEmail(value: string): string { function normalizeEmail(value: string): string {
@@ -34,6 +41,29 @@ function senderMatchesAllowlist(
return senderDomain === normalizedDomain; return senderDomain === normalizedDomain;
} }
async function uploadAttachments(
attachments: RawAttachment[],
bucket: R2Bucket,
): Promise<AttachmentData[]> {
return Promise.all(
attachments.map(async (att) => {
const id = crypto.randomUUID();
await bucket.put(id, att.content, {
httpMetadata: {
contentType: att.contentType,
contentDisposition: `attachment; filename="${att.filename}"`,
},
});
return {
id,
filename: att.filename,
contentType: att.contentType,
size: att.content.byteLength,
};
}),
);
}
export async function processEmail( export async function processEmail(
input: ProcessEmailInput, input: ProcessEmailInput,
env: Env, env: Env,
@@ -74,12 +104,18 @@ export async function processEmail(
} }
} }
const storedAttachments: AttachmentData[] =
env.ATTACHMENT_BUCKET && input.attachments?.length
? await uploadAttachments(input.attachments, env.ATTACHMENT_BUCKET)
: [];
const emailData = { const emailData = {
subject: input.subject, subject: input.subject,
from: input.from, from: input.from,
content: input.content, content: input.content,
receivedAt: input.receivedAt, receivedAt: input.receivedAt,
headers: input.headers ?? {}, headers: input.headers ?? {},
...(storedAttachments.length > 0 ? { attachments: storedAttachments } : {}),
}; };
const emailKey = `feed:${feedId}:${Date.now()}`; const emailKey = `feed:${feedId}:${Date.now()}`;
@@ -104,24 +140,36 @@ export async function processEmail(
const serialised = JSON.stringify(emailData); const serialised = JSON.stringify(emailData);
const serialisedSize = new TextEncoder().encode(serialised).byteLength; const serialisedSize = new TextEncoder().encode(serialised).byteLength;
feedMetadata.emails.unshift({ const newEntry: EmailMetadata = {
key: emailKey, key: emailKey,
subject: emailData.subject, subject: emailData.subject,
receivedAt: emailData.receivedAt, receivedAt: emailData.receivedAt,
size: serialisedSize, size: serialisedSize,
}); ...(storedAttachments.length > 0
? { attachmentIds: storedAttachments.map((a) => a.id) }
: {}),
};
feedMetadata.emails.unshift(newEntry);
let totalSize = feedMetadata.emails.reduce((sum, e) => sum + (e.size ?? 0), 0); let totalSize = feedMetadata.emails.reduce((sum, e) => sum + (e.size ?? 0), 0);
const toDelete: string[] = []; const toDelete: EmailMetadata[] = [];
while (totalSize > maxBytes && feedMetadata.emails.length > 1) { while (totalSize > maxBytes && feedMetadata.emails.length > 1) {
const dropped = feedMetadata.emails.pop()!; const dropped = feedMetadata.emails.pop()!;
totalSize -= dropped.size ?? 0; totalSize -= dropped.size ?? 0;
toDelete.push(dropped.key); toDelete.push(dropped);
} }
const r2Deletions =
env.ATTACHMENT_BUCKET && toDelete.length > 0
? toDelete
.flatMap((e) => e.attachmentIds ?? [])
.map((id) => env.ATTACHMENT_BUCKET!.delete(id))
: [];
await Promise.all([ await Promise.all([
env.EMAIL_STORAGE.put(feedMetadataKey, JSON.stringify(feedMetadata)), env.EMAIL_STORAGE.put(feedMetadataKey, JSON.stringify(feedMetadata)),
...toDelete.map((k) => env.EMAIL_STORAGE.delete(k)), ...toDelete.map((e) => env.EMAIL_STORAGE.delete(e.key)),
...r2Deletions,
]); ]);
console.log(`Successfully processed email for feed ${feedId}`); console.log(`Successfully processed email for feed ${feedId}`);
+37 -2
View File
@@ -1,6 +1,16 @@
import { EmailParser } from "../utils/email-parser"; import { EmailParser } from "../utils/email-parser";
import { Env } from "../types"; import { Env } from "../types";
import { processEmail } from "./email-processor"; import { processEmail, RawAttachment } from "./email-processor";
export interface ForwardEmailAttachment {
filename?: string;
contentType?: string;
size?: number;
content?:
| { type: "Buffer"; data: number[] }
| ArrayBuffer
| ArrayBufferView;
}
export interface ForwardEmailPayload { export interface ForwardEmailPayload {
recipients?: string[]; recipients?: string[];
@@ -17,7 +27,7 @@ export interface ForwardEmailPayload {
headerLines?: Array<{ key: string; line: string }>; headerLines?: Array<{ key: string; line: string }>;
headers?: string; headers?: string;
raw?: string; raw?: string;
attachments?: Array<any>; attachments?: ForwardEmailAttachment[];
} }
function normalizeEmail(value: string): string { function normalizeEmail(value: string): string {
@@ -41,12 +51,36 @@ function extractSenderAddresses(payload: ForwardEmailPayload): string[] {
return Array.from(new Set(matches.map(normalizeEmail))); return Array.from(new Set(matches.map(normalizeEmail)));
} }
function toArrayBuffer(
content: ForwardEmailAttachment["content"],
): ArrayBuffer | null {
if (!content) return null;
if (content instanceof ArrayBuffer) return content;
if (ArrayBuffer.isView(content)) return (content as ArrayBufferView).buffer as ArrayBuffer;
if (typeof content === "object" && content.type === "Buffer" && Array.isArray(content.data)) {
return Uint8Array.from(content.data).buffer as ArrayBuffer;
}
return null;
}
export async function handleForwardEmail( export async function handleForwardEmail(
payload: ForwardEmailPayload, payload: ForwardEmailPayload,
env: Env, env: Env,
): Promise<Response> { ): Promise<Response> {
const emailData = EmailParser.parseForwardEmailPayload(payload); const emailData = EmailParser.parseForwardEmailPayload(payload);
const rawAttachments: RawAttachment[] = (payload.attachments ?? [])
.map((a) => {
const buffer = toArrayBuffer(a.content);
if (!buffer) return null;
return {
filename: a.filename || "attachment",
contentType: a.contentType || "application/octet-stream",
content: buffer,
};
})
.filter((a): a is RawAttachment => a !== null);
return processEmail( return processEmail(
{ {
toAddress: payload.recipients?.[0] || "", toAddress: payload.recipients?.[0] || "",
@@ -56,6 +90,7 @@ export async function handleForwardEmail(
content: emailData.content, content: emailData.content,
receivedAt: emailData.receivedAt, receivedAt: emailData.receivedAt,
headers: emailData.headers, headers: emailData.headers,
attachments: rawAttachments,
}, },
env, env,
); );
+58 -6
View File
@@ -1908,7 +1908,7 @@ async function deleteFeedFastDetailed(
async function purgeFeedKeysStep( async function purgeFeedKeysStep(
emailStorage: KVNamespace, emailStorage: KVNamespace,
feedId: string, feedId: string,
options: { cursor?: string; limit?: number } = {}, options: { cursor?: string; limit?: number; bucket?: R2Bucket } = {},
): Promise<{ ): Promise<{
deletedKeys: string[]; deletedKeys: string[];
failedKeys: string[]; failedKeys: string[];
@@ -1921,6 +1921,33 @@ async function purgeFeedKeysStep(
const listed = await emailStorage.list({ prefix, cursor, limit }); const listed = await emailStorage.list({ prefix, cursor, limit });
const keys = (listed.keys || []).map((k) => k.name); const keys = (listed.keys || []).map((k) => k.name);
// Collect R2 attachment IDs from email entries before deleting
if (options.bucket && keys.length > 0) {
const emailKeys = keys.filter((k) => {
const suffix = k.slice(prefix.length);
return suffix !== "config" && suffix !== "metadata";
});
if (emailKeys.length > 0) {
const emailDataResults = await Promise.allSettled(
emailKeys.map((k) =>
emailStorage.get(k, { type: "json" }) as Promise<EmailData | null>,
),
);
const attachmentIds = emailDataResults
.filter(
(r): r is PromiseFulfilledResult<EmailData | null> =>
r.status === "fulfilled",
)
.flatMap((r) => r.value?.attachments?.map((a) => a.id) ?? []);
if (attachmentIds.length > 0) {
await Promise.allSettled(
attachmentIds.map((id) => options.bucket!.delete(id)),
);
}
}
}
const { ok, failed } = await deleteKeysWithConcurrency( const { ok, failed } = await deleteKeysWithConcurrency(
emailStorage, emailStorage,
keys, keys,
@@ -1949,7 +1976,7 @@ app.post("/feeds/:feedId/delete", async (c) => {
// Best-effort cleanup in the background so the request stays fast. // Best-effort cleanup in the background so the request stays fast.
// Use the UI purge endpoint for full, user-visible progress. // Use the UI purge endpoint for full, user-visible progress.
waitUntilSafe(c, purgeFeedKeysStep(emailStorage, feedId)); waitUntilSafe(c, purgeFeedKeysStep(emailStorage, feedId, { bucket: env.ATTACHMENT_BUCKET }));
if (wantsJson) { if (wantsJson) {
return c.json({ ok: true, feedId }); return c.json({ ok: true, feedId });
} }
@@ -1987,6 +2014,7 @@ app.post("/feeds/:feedId/purge", async (c) => {
const step = await purgeFeedKeysStep(emailStorage, feedId, { const step = await purgeFeedKeysStep(emailStorage, feedId, {
cursor, cursor,
limit, limit,
bucket: env.ATTACHMENT_BUCKET,
}); });
return c.json({ return c.json({
@@ -3420,15 +3448,18 @@ app.post("/emails/:emailKey/delete", async (c) => {
return c.text("Feed ID is required", 400); return c.text("Feed ID is required", 400);
} }
// Delete the email // Load metadata first to collect attachment IDs for R2 cleanup
await emailStorage.delete(emailKey);
// Remove the email from the feed metadata
const feedMetadataKey = `feed:${feedId}:metadata`; const feedMetadataKey = `feed:${feedId}:metadata`;
const feedMetadata = (await emailStorage.get(feedMetadataKey, { const feedMetadata = (await emailStorage.get(feedMetadataKey, {
type: "json", type: "json",
})) as FeedMetadata | null; })) as FeedMetadata | null;
const attachmentIds =
feedMetadata?.emails.find((e) => e.key === emailKey)?.attachmentIds ?? [];
// Delete the email
await emailStorage.delete(emailKey);
if (feedMetadata) { if (feedMetadata) {
// Filter out the deleted email // Filter out the deleted email
feedMetadata.emails = feedMetadata.emails.filter( feedMetadata.emails = feedMetadata.emails.filter(
@@ -3439,6 +3470,13 @@ app.post("/emails/:emailKey/delete", async (c) => {
await emailStorage.put(feedMetadataKey, JSON.stringify(feedMetadata)); await emailStorage.put(feedMetadataKey, JSON.stringify(feedMetadata));
} }
// Best-effort R2 attachment cleanup
if (env.ATTACHMENT_BUCKET && attachmentIds.length > 0) {
await Promise.allSettled(
attachmentIds.map((id) => env.ATTACHMENT_BUCKET!.delete(id)),
);
}
if (wantsJson) { if (wantsJson) {
return c.json({ ok: true, emailKey, feedId }); return c.json({ ok: true, emailKey, feedId });
} }
@@ -3508,6 +3546,13 @@ app.post("/feeds/:feedId/emails/bulk-delete", async (c) => {
} }
const candidates = emailKeys.filter((key) => allowedKeys.has(key)); const candidates = emailKeys.filter((key) => allowedKeys.has(key));
// Collect attachment IDs from metadata before deleting (no extra KV reads needed)
const candidateSet = new Set(candidates);
const r2AttachmentIds = feedMetadata.emails
.filter((e) => candidateSet.has(e.key))
.flatMap((e) => e.attachmentIds ?? []);
const { ok: deletedOk, failed: failedEmailKeys } = const { ok: deletedOk, failed: failedEmailKeys } =
await deleteKeysWithConcurrency(emailStorage, candidates, 35); await deleteKeysWithConcurrency(emailStorage, candidates, 35);
@@ -3517,6 +3562,13 @@ app.post("/feeds/:feedId/emails/bulk-delete", async (c) => {
); );
await emailStorage.put(feedMetadataKey, JSON.stringify(feedMetadata)); await emailStorage.put(feedMetadataKey, JSON.stringify(feedMetadata));
// Best-effort R2 attachment cleanup
if (env.ATTACHMENT_BUCKET && r2AttachmentIds.length > 0) {
await Promise.allSettled(
r2AttachmentIds.map((id) => env.ATTACHMENT_BUCKET!.delete(id)),
);
}
return c.json({ return c.json({
ok: failedEmailKeys.length === 0, ok: failedEmailKeys.length === 0,
deletedEmailKeys: deletedOk, deletedEmailKeys: deletedOk,
+103
View File
@@ -0,0 +1,103 @@
import { describe, it, expect, beforeEach } from "vitest";
import "../test/setup";
import { createMockEnv, MockR2 } from "../test/setup";
import { Hono } from "hono";
import { handle as handleFiles } from "./files";
async function request(
env: ReturnType<typeof createMockEnv>,
path: string,
): Promise<Response> {
const app = new Hono();
const files = new Hono();
files.get("/:attachmentId/:filename", handleFiles);
app.route("/files", files);
return app.request(path, {}, env as any);
}
describe("GET /files/:attachmentId/:filename", () => {
let envNoR2: ReturnType<typeof createMockEnv>;
let envWithR2: ReturnType<typeof createMockEnv>;
let mockR2: MockR2;
beforeEach(() => {
envNoR2 = createMockEnv();
envWithR2 = createMockEnv({ withR2: true });
mockR2 = (envWithR2 as any).ATTACHMENT_BUCKET as unknown as MockR2;
});
it("returns 404 when ATTACHMENT_BUCKET is not configured", async () => {
const res = await request(envNoR2, "/files/some-id/file.pdf");
expect(res.status).toBe(404);
expect(await res.text()).toContain("not configured");
});
it("returns 404 when attachment ID is not found in R2", async () => {
const res = await request(envWithR2, "/files/unknown-id/file.pdf");
expect(res.status).toBe(404);
expect(await res.text()).toBe("Not found");
});
it("returns 200 with stored content when attachment exists", async () => {
const content = new TextEncoder().encode("PDF content").buffer as ArrayBuffer;
await mockR2.put("test-uuid", content, {
httpMetadata: { contentType: "application/pdf" },
});
const res = await request(envWithR2, "/files/test-uuid/report.pdf");
expect(res.status).toBe(200);
});
it("returns correct Content-Type from stored httpMetadata", async () => {
const content = new TextEncoder().encode("data").buffer as ArrayBuffer;
await mockR2.put("img-uuid", content, {
httpMetadata: { contentType: "image/png" },
});
const res = await request(envWithR2, "/files/img-uuid/photo.png");
expect(res.status).toBe(200);
expect(res.headers.get("Content-Type")).toContain("image/png");
});
it("sets Cache-Control immutable header", async () => {
const content = new TextEncoder().encode("data").buffer as ArrayBuffer;
await mockR2.put("cache-uuid", content, {
httpMetadata: { contentType: "application/pdf" },
});
const res = await request(envWithR2, "/files/cache-uuid/doc.pdf");
expect(res.headers.get("Cache-Control")).toBe(
"public, max-age=31536000, immutable",
);
});
it("sets Content-Disposition from httpMetadata when present", async () => {
const content = new TextEncoder().encode("data").buffer as ArrayBuffer;
await mockR2.put("disp-uuid", content, {
httpMetadata: {
contentType: "application/pdf",
contentDisposition: 'attachment; filename="stored.pdf"',
},
});
const res = await request(envWithR2, "/files/disp-uuid/other.pdf");
expect(res.headers.get("Content-Disposition")).toBe(
'attachment; filename="stored.pdf"',
);
});
it("falls back to URL filename for Content-Disposition when not in httpMetadata", async () => {
const content = new TextEncoder().encode("data").buffer as ArrayBuffer;
await mockR2.put("fallback-uuid", content, {
httpMetadata: { contentType: "text/plain" },
});
const res = await request(
envWithR2,
"/files/fallback-uuid/hello%20world.txt",
);
expect(res.headers.get("Content-Disposition")).toBe(
'attachment; filename="hello world.txt"',
);
});
});
+33
View File
@@ -0,0 +1,33 @@
import { Context } from "hono";
import { Env } from "../types";
export async function handle(c: Context): Promise<Response> {
const env = c.env as unknown as Env;
if (!env.ATTACHMENT_BUCKET) {
return new Response("Attachment storage not configured", { status: 404 });
}
const attachmentId = c.req.param("attachmentId");
const filename = c.req.param("filename");
const object = await env.ATTACHMENT_BUCKET.get(attachmentId);
if (!object) {
return new Response("Not found", { status: 404 });
}
const headers = new Headers();
object.writeHttpMetadata(headers);
headers.set("etag", object.httpEtag);
headers.set("Cache-Control", "public, max-age=31536000, immutable");
if (!headers.get("Content-Disposition")) {
headers.set(
"Content-Disposition",
`attachment; filename="${decodeURIComponent(filename)}"`,
);
}
return new Response(object.body, { headers });
}
+81 -1
View File
@@ -157,12 +157,92 @@ afterEach(() => {
server.resetHandlers(); server.resetHandlers();
}); });
/**
* Mock R2 bucket implementation
* Simulates Cloudflare Workers R2 storage using an in-memory Map
*/
export class MockR2 {
private store: Map<
string,
{
body: ArrayBuffer;
httpMetadata?: { contentType?: string; contentDisposition?: string };
httpEtag: string;
}
> = new Map();
async put(
key: string,
value: ReadableStream | ArrayBuffer | ArrayBufferView | string | null | Blob,
options?: {
httpMetadata?: { contentType?: string; contentDisposition?: string };
},
) {
let buffer: ArrayBuffer;
if (value instanceof ArrayBuffer) {
buffer = value;
} else if (typeof value === "string") {
buffer = new TextEncoder().encode(value).buffer as ArrayBuffer;
} else if (ArrayBuffer.isView(value)) {
buffer = value.buffer as ArrayBuffer;
} else {
buffer = new ArrayBuffer(0);
}
this.store.set(key, {
body: buffer,
httpMetadata: options?.httpMetadata,
httpEtag: `"mock-etag-${key}"`,
});
}
async get(key: string) {
const entry = this.store.get(key);
if (!entry) return null;
return {
key,
body: new ReadableStream({
start(controller) {
controller.enqueue(new Uint8Array(entry.body));
controller.close();
},
}),
httpEtag: entry.httpEtag,
httpMetadata: entry.httpMetadata,
size: entry.body.byteLength,
arrayBuffer: async () => entry.body,
text: async () => new TextDecoder().decode(entry.body),
writeHttpMetadata: (headers: Headers) => {
if (entry.httpMetadata?.contentType) {
headers.set("Content-Type", entry.httpMetadata.contentType);
}
if (entry.httpMetadata?.contentDisposition) {
headers.set("Content-Disposition", entry.httpMetadata.contentDisposition);
}
},
};
}
async delete(keys: string | string[]) {
const arr = Array.isArray(keys) ? keys : [keys];
for (const k of arr) {
this.store.delete(k);
}
}
_has(key: string) {
return this.store.has(key);
}
}
/** /**
* Create a mock environment for testing * Create a mock environment for testing
* @returns Mock environment with KV storage and configuration * @returns Mock environment with KV storage and configuration
*/ */
export const createMockEnv = () => ({ export const createMockEnv = (options: { withR2?: boolean } = {}) => ({
EMAIL_STORAGE: new MockKV(), EMAIL_STORAGE: new MockKV(),
DOMAIN: "test.getmynews.app", DOMAIN: "test.getmynews.app",
ADMIN_PASSWORD: "test-password", ADMIN_PASSWORD: "test-password",
...(options.withR2
? { ATTACHMENT_BUCKET: new MockR2() as unknown as R2Bucket }
: {}),
}); });
+11
View File
@@ -3,11 +3,20 @@ export interface Env {
EMAIL_STORAGE: KVNamespace; EMAIL_STORAGE: KVNamespace;
ADMIN_PASSWORD: string; ADMIN_PASSWORD: string;
DOMAIN: string; DOMAIN: string;
ATTACHMENT_BUCKET?: R2Bucket;
FEED_MAX_SIZE_BYTES?: string; FEED_MAX_SIZE_BYTES?: string;
PROXY_TRUSTED_IPS?: string; PROXY_TRUSTED_IPS?: string;
PROXY_AUTH_SECRET?: string; PROXY_AUTH_SECRET?: string;
} }
// Stored attachment metadata (bytes live in R2, keyed by id)
export interface AttachmentData {
id: string;
filename: string;
contentType: string;
size: number;
}
// Email interface for stored emails // Email interface for stored emails
export interface EmailData { export interface EmailData {
subject: string; subject: string;
@@ -15,6 +24,7 @@ export interface EmailData {
content: string; content: string;
receivedAt: number; receivedAt: number;
headers: Record<string, string>; headers: Record<string, string>;
attachments?: AttachmentData[];
} }
// Feed configuration interface // Feed configuration interface
@@ -41,6 +51,7 @@ export interface EmailMetadata {
subject: string; subject: string;
receivedAt: number; receivedAt: number;
size?: number; size?: number;
attachmentIds?: string[];
} }
// Feed list interface // Feed list interface
+43
View File
@@ -21,6 +21,18 @@ const mockEmails: EmailData[] = [
}, },
]; ];
const mockEmailWithAttachment: EmailData = {
...mockEmails[0],
attachments: [
{
id: "550e8400-e29b-41d4-a716-446655440000",
filename: "report.pdf",
contentType: "application/pdf",
size: 12345,
},
],
};
const BASE_URL = "https://test.getmynews.app"; const BASE_URL = "https://test.getmynews.app";
const FEED_ID = "abc123"; const FEED_ID = "abc123";
@@ -31,6 +43,25 @@ describe("generateRssFeed", () => {
expect(result).toContain("<title>Test Newsletter</title>"); expect(result).toContain("<title>Test Newsletter</title>");
}); });
it("includes <enclosure> element for email with attachment", () => {
const result = generateRssFeed(mockFeedConfig, [mockEmailWithAttachment], BASE_URL, FEED_ID);
expect(result).toContain("<enclosure");
expect(result).toContain("550e8400-e29b-41d4-a716-446655440000");
expect(result).toContain("report.pdf");
expect(result).toContain("application/pdf");
expect(result).toContain("12345");
});
it("does not include <enclosure> for email without attachments", () => {
const result = generateRssFeed(mockFeedConfig, mockEmails, BASE_URL, FEED_ID);
expect(result).not.toContain("<enclosure");
});
it("enclosure URL uses /files/{id}/{filename} scheme", () => {
const result = generateRssFeed(mockFeedConfig, [mockEmailWithAttachment], BASE_URL, FEED_ID);
expect(result).toContain(`${BASE_URL}/files/550e8400-e29b-41d4-a716-446655440000/report.pdf`);
});
it("includes rss self-link in RSS output", () => { it("includes rss self-link in RSS output", () => {
const result = generateRssFeed(mockFeedConfig, mockEmails, BASE_URL, FEED_ID); const result = generateRssFeed(mockFeedConfig, mockEmails, BASE_URL, FEED_ID);
expect(result).toContain(`${BASE_URL}/rss/${FEED_ID}`); expect(result).toContain(`${BASE_URL}/rss/${FEED_ID}`);
@@ -104,4 +135,16 @@ describe("generateAtomFeed", () => {
const result = generateAtomFeed(configWithAuthor, mockEmails, BASE_URL, FEED_ID); const result = generateAtomFeed(configWithAuthor, mockEmails, BASE_URL, FEED_ID);
expect(result).toContain("Bob"); expect(result).toContain("Bob");
}); });
it("includes enclosure link for email with attachment in Atom feed", () => {
const result = generateAtomFeed(mockFeedConfig, [mockEmailWithAttachment], BASE_URL, FEED_ID);
expect(result).toContain('rel="enclosure"');
expect(result).toContain("550e8400-e29b-41d4-a716-446655440000");
expect(result).toContain("report.pdf");
});
it("does not include enclosure link for email without attachments in Atom feed", () => {
const result = generateAtomFeed(mockFeedConfig, mockEmails, BASE_URL, FEED_ID);
expect(result).not.toContain('rel="enclosure"');
});
}); });
+8
View File
@@ -42,6 +42,7 @@ function buildFeed(
for (const email of emails) { for (const email of emails) {
const uniqueId = `${email.receivedAt}-${Buffer.from(email.subject).toString("base64").substring(0, 10)}`; const uniqueId = `${email.receivedAt}-${Buffer.from(email.subject).toString("base64").substring(0, 10)}`;
const firstAttachment = email.attachments?.[0];
feed.addItem({ feed.addItem({
title: email.subject, title: email.subject,
id: uniqueId, id: uniqueId,
@@ -50,6 +51,13 @@ function buildFeed(
content: email.content, content: email.content,
author: [parseFromAddress(email.from)], author: [parseFromAddress(email.from)],
date: new Date(email.receivedAt), date: new Date(email.receivedAt),
enclosure: firstAttachment
? {
url: `${baseUrl}/files/${firstAttachment.id}/${encodeURIComponent(firstAttachment.filename)}`,
type: firstAttachment.contentType,
length: firstAttachment.size,
}
: undefined,
}); });
} }
+10
View File
@@ -8,6 +8,11 @@ kv_namespaces = [
{ binding = "EMAIL_STORAGE", id = "REPLACE_WITH_YOUR_KV_NAMESPACE_ID", preview_id = "REPLACE_WITH_YOUR_PREVIEW_KV_NAMESPACE_ID" } { binding = "EMAIL_STORAGE", id = "REPLACE_WITH_YOUR_KV_NAMESPACE_ID", preview_id = "REPLACE_WITH_YOUR_PREVIEW_KV_NAMESPACE_ID" }
] ]
# Optional: R2 bucket for storing email attachment enclosures
# r2_buckets = [
# { binding = "ATTACHMENT_BUCKET", bucket_name = "REPLACE_WITH_YOUR_BUCKET_NAME", preview_bucket_name = "REPLACE_WITH_YOUR_PREVIEW_BUCKET_NAME" }
# ]
# Workers Observability (keeps config in sync with dashboard toggle) # Workers Observability (keeps config in sync with dashboard toggle)
[observability.logs] [observability.logs]
enabled = true enabled = true
@@ -42,6 +47,11 @@ kv_namespaces = [
{ binding = "EMAIL_STORAGE", id = "REPLACE_WITH_YOUR_KV_NAMESPACE_ID" } { binding = "EMAIL_STORAGE", id = "REPLACE_WITH_YOUR_KV_NAMESPACE_ID" }
] ]
# Optional: R2 bucket for storing email attachment enclosures
# r2_buckets = [
# { binding = "ATTACHMENT_BUCKET", bucket_name = "REPLACE_WITH_YOUR_BUCKET_NAME" }
# ]
routes = [ routes = [
{ pattern = "REPLACE_WITH_YOUR_DOMAIN", custom_domain = true }, { pattern = "REPLACE_WITH_YOUR_DOMAIN", custom_domain = true },
{ pattern = "www.REPLACE_WITH_YOUR_DOMAIN", custom_domain = true } { pattern = "www.REPLACE_WITH_YOUR_DOMAIN", custom_domain = true }