Initial commit: read social posts back without the app
antisocial is the other half of a StopTheMadness redirect rule. Links to X, Threads, Instagram, TikTok and Bluesky get rewritten to /<prefix>/<original path>, and this resolves the post and shows the media and the words, with a badge saying where it came from and a button to copy the original URL. Every request drives a real headless Chromium, logged out, from a residential IP. One code path, and it survives markup changes better than parsing from the outside would. Extraction is layered, most structured first: the platform's own API response caught in flight, then an inline payload, then the rendered DOM, then Open Graph tags. Media is never linked straight at a CDN. Instagram and TikTok reject requests without a matching Referer and cookies, and proxying keeps the viewer's browser from talking to the platform at all. Range is forwarded so the native video scrubber can seek. HLS is the exception, since proxying it would mean rewriting playlists. TikTok sometimes answers with a slider puzzle. Rather than reporting that as a failure, the page is parked and the viewer is handed the puzzle: screenshots stream out, pointer events are replayed back. Solving it leaves the cookie in the shared browser context, so the retry is an ordinary request. A failed resolve is never a blank error page. The card carries the platform, the original URL and the copy button, so a broken adapter still leaves the link one tap away. Verified end to end against real shared links on all five platforms, in the container, including multi-image carousels, reels, TikTok short links and photo posts. 49 tests run the adapters against captured payloads with no network. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01BGkRmLfiWuJHx6tQ12EELY
This commit is contained in:
Claude Opus 5
@@ -0,0 +1,143 @@
|
||||
# antisocial — working notes
|
||||
|
||||
A self-hosted page that shows a social post without the app. StopTheMadness rewrites
|
||||
links to X, Threads, Instagram, TikTok and Bluesky into `/<prefix>/<original path>`;
|
||||
this resolves the post by driving a real headless Chromium and renders the media, the
|
||||
text, a platform badge and a copy-the-original button.
|
||||
|
||||
Single user, no auth, tailnet only, running on Kallone (lab3). `README.md` has the
|
||||
rewrite rules and the user-facing description. This file is the working context.
|
||||
|
||||
## The main job: "this link didn't work"
|
||||
|
||||
That is the recurring task. Work it in this order and do not skip step 1.
|
||||
|
||||
**1. Reproduce against the real browser.**
|
||||
|
||||
```sh
|
||||
npm run resolve -- 'https://www.instagram.com/user/reel/ABC123/'
|
||||
```
|
||||
|
||||
Prints the resolved `Post` as JSON, or the error. This is the ground truth.
|
||||
|
||||
**2. Decide what kind of failure it is.** These are genuinely different and get
|
||||
different fixes:
|
||||
|
||||
| Symptom | What it usually is |
|
||||
| --- | --- |
|
||||
| Works now, failed before | Rate limiting. TikTok especially. Not a bug — verify by waiting and retrying before changing anything. |
|
||||
| Verification puzzle | Expected. Should redirect to `/challenge/<id>`. If it reports an error instead, the detection raced the render. |
|
||||
| Right media, wrong count | The structured payload was missed and it fell through to the DOM, which only shows the first carousel item. |
|
||||
| Poster image where a video belongs | Same fall-through, plus the `<video>` had not hydrated when the DOM was read. |
|
||||
| Nothing at all | The platform refused this specific post logged out, or a key moved. |
|
||||
|
||||
**3. Look at what the page actually served.**
|
||||
|
||||
```sh
|
||||
npm run probe -- 'https://www.instagram.com/reel/ABC123/embed/captioned/' 5000
|
||||
```
|
||||
|
||||
Pass the URL *the adapter loads*, not the post URL — for Instagram that is the
|
||||
`/embed/captioned/` form. It reports which payload keys are present, the DOM's
|
||||
`<video>`/`<img>`, the OG tags, whether a challenge is up, and the API traffic with
|
||||
bundle noise filtered out. Every adapter fix so far started here.
|
||||
|
||||
**4. Fix the adapter, then capture a fixture** for the case that broke, and assert on it
|
||||
in `test/<platform>.test.ts`. Fixtures are real captured payloads — see the ones in
|
||||
`test/fixtures/`. A fix without a fixture will silently regress.
|
||||
|
||||
**5. Re-run the whole suite** (`npm test`), and re-resolve the link end to end.
|
||||
|
||||
## Architecture, briefly
|
||||
|
||||
Request → `src/routes/post.ts` → `src/platforms/index.ts` maps prefix to adapter →
|
||||
`src/resolve.ts` (cache + timeout + concurrency gate) → `withPage` from
|
||||
`src/browser/pool.ts` → the adapter → a `Post` → `src/render/post.ts`.
|
||||
|
||||
Adding a platform is one file in `src/platforms/` plus one row in the table in
|
||||
`index.ts`. Everything downstream already handles a `Post`.
|
||||
|
||||
Things worth knowing before editing:
|
||||
|
||||
- **One Chromium, one context, persistent.** Cookies and dismissed banners accumulate on
|
||||
purpose — that is what makes traffic look like ordinary browsing. `MAX_CONCURRENT`
|
||||
page loads, the rest queue.
|
||||
- **Images, video and fonts are aborted** at the route layer. The URL is all we need; the
|
||||
bytes get fetched later through our own proxy. Aborting does not remove `src`
|
||||
attributes, so DOM extraction still works.
|
||||
- **Extraction is layered**, most structured first: the platform's own API response
|
||||
caught in flight → an inline payload → the rendered DOM → OG tags. Keep that order
|
||||
when you touch an adapter; each layer is the fallback for the one above.
|
||||
- **`src/platforms/scan.ts`** pulls a balanced JSON object out of a page by key,
|
||||
including when it arrives escaped inside a JS string. Instagram needs this. If a
|
||||
payload looks present but parses to nothing, suspect an extra encoding level.
|
||||
- **Media is never linked straight at a CDN.** Assets are registered in
|
||||
`src/media/registry.ts` and served from `/m/<id>` with the `Referer`/`Cookie` the CDN
|
||||
demands. `Range` is forwarded — without it the native video scrubber cannot seek.
|
||||
The exception is HLS (`direct: true`), because proxying would mean rewriting playlists.
|
||||
- **Never a bare error page.** A failed resolve renders a card carrying the platform, the
|
||||
original URL and the copy button. A broken adapter must still leave the link one tap
|
||||
away.
|
||||
|
||||
## Per-platform notes
|
||||
|
||||
- **Bluesky** — asks the public API directly (still through the browser context), so it
|
||||
is the most reliable. The web app calls `getPostThreadV2` now; the page fallback
|
||||
deliberately matches only V1.
|
||||
- **X** — the `platform.twitter.com` embed calls the syndication endpoint; we catch that
|
||||
response. A quote post carries no media of its own, so the quoted post's media is used.
|
||||
- **Instagram** — the least reliable. It ships the structured payload only some of the
|
||||
time, and refuses some posts from the embed entirely ("the link may be broken"). The
|
||||
DOM is the floor and only ever shows the first carousel item. Reels hydrate their
|
||||
`<video>` roughly 0.6–1.3s after `domcontentloaded`, so the DOM read waits when a video
|
||||
is expected.
|
||||
- **TikTok** — the post lives under whichever `__UNIVERSAL_DATA_FOR_REHYDRATION__` scope
|
||||
carries an `itemStruct`; do not hardcode the key, photo posts use a different one.
|
||||
Its CDN needs `Referer` *and* cookies. Short `vm.`/`vt.` links arrive as a bare path
|
||||
segment and are rebuilt in `buildOriginalUrl`.
|
||||
- **Threads** — same media schema as Instagram (`src/platforms/meta-media.ts`). Its
|
||||
payloads are full of empty stub nodes, so the finder only accepts a node with actual
|
||||
candidates in it.
|
||||
|
||||
## Verification puzzles
|
||||
|
||||
TikTok challenges sometimes. The page is parked (`src/challenge/registry.ts`), kept open
|
||||
but outside the pool holding no concurrency permit, and the viewer is redirected to
|
||||
`/challenge/<id>`, which streams screenshots of the puzzle and replays their pointer
|
||||
events onto it. Solving it deposits the cookie in the shared context, so the retry is an
|
||||
ordinary request.
|
||||
|
||||
Login walls are not passed through — those need an account and cannot be solved this way.
|
||||
|
||||
## Conventions
|
||||
|
||||
- TypeScript 7, ESM, `.ts` import specifiers (tsc rewrites them to `.js` on emit).
|
||||
Node's type stripping runs the tree directly for dev, tests and the CLIs.
|
||||
- **No ESLint** — `typescript-eslint` does not support TS 7 yet. `npm run typecheck` is
|
||||
the lint step.
|
||||
- Tests are `node:test` against captured fixtures. No network in the test suite.
|
||||
- Comments explain *why*, especially where the code looks odd because a platform is odd.
|
||||
Match that; do not add narration of what the next line does.
|
||||
- Post text comes from strangers: everything goes through the `html` tagged template or
|
||||
`linkify`, both of which escape. Do not build markup by concatenation.
|
||||
|
||||
## Commands
|
||||
|
||||
```sh
|
||||
npm run resolve -- '<url>' # resolve one post, print JSON <- start here
|
||||
npm run probe -- '<url>' # what the page actually served <- then here
|
||||
npm test # fixtures, no network
|
||||
npm run typecheck
|
||||
npm run dev # http://localhost:8080
|
||||
container build --tag antisocial:dev . # Apple container, not Docker
|
||||
```
|
||||
|
||||
## Deployment
|
||||
|
||||
CI (`.gitea/workflows/publish.yml`) builds on main and pushes
|
||||
`git.unsupervised.ca/unsupervised/antisocial:latest` and `:sha-<short>`. Manifests live
|
||||
in the **`infra` repo** at `k8s/kallone/antisocial.unsupervised.studio/`; Flux syncs
|
||||
`k8s/kallone` every 10 minutes. The DNS CNAME is in `infra/cloudflare/unsupervised.studio.tf`.
|
||||
|
||||
Chromium needs the memory-backed `/dev/shm` volume in the deployment — it crashes against
|
||||
the 64Mi default. Measured 623Mi resident.
|
||||
Reference in New Issue
Block a user