Skip to content

Scope — CMS-lane batch runner (the Slice 6 corpus vehicle)

Date: 2026-07-18 Status: IN PROGRESS — scope confirmed (EC2 corpus box; production R2 bucket OK'd). Step 0 PROVEN; runner shipped (scripts/run-cms-batch.mjs); positive control PASSED 2026-07-18: local CMS serve of WCP scores S 82.1 / T 100 / G_det 84.4 → composite 87.2, HOLD on videoDebt — the deployed-worker baseline to a tenth of a point, same single veto (done criterion a). batch-scorecard consumed the output — 2 sites, 0 skipped (criterion c). Zero production writes (criterion d — no deploy/remote-D1 path exists in the runner; images stage idempotent, 0 uploads on the control run). Remaining: criterion (b) — ≥3 sites unattended incl. ≥1 never-before-crawled (needs Cathal's site picks; team-built .ie sites only per the matcher-tuning-scope rule); ~~criterion (e) runbook below is written, EC2 re-run of the control pending~~ criterion (e) COMPLETE 2026-07-18 (pt2): EC2 re-run of the positive control PASSED — S 82.3 / T 100 / G_det 84.4 → composite 87.3 HOLD(videoDebt) vs the Windows baseline 82.1/100/84.4 → 87.2: within re-crawl drift (S +0.2), same sole veto, no new veto classes. batch-scorecard consumed it on EC2 (1 row; the two blind sites loud-skip there — their artifacts live on the Windows box). EC2 is the corpus box; see the EC2 runbook below.

Runbook (criterion e)

node scripts/run-cms-batch.mjs --hosts <file>          # one domain per line, # comments
node scripts/run-cms-batch.mjs --site <domain>         # single site (e.g. positive control)
  [--port 8931] [--limit N] [--force <stage,..|all>] [--stop-after <stage>]
node scripts/batch-scorecard.mjs <hosts-file>          # aggregate → feature matrix
  • Stages + skip rules are in the runner header. A stage is skipped iff its artifact exists; --force redoes. Failed sites are recorded in builds/cms-batch-report.json and the batch continues.
  • Needs on the box: node ≥18, Playwright (with browsers), apps/cms/.dev.vars (R2 + Wix creds for the images stage), npm workspaces installed. Sequential by design — apps/cms build state is shared.
  • Reference sites: a committed apps/cms/seed/<site>.json is never overwritten (the reseed-revert landmine); the runner serves the committed seed.
  • v1 constraint: bare <site>.ie domains only (resolve-site's themePathFor).
  • Wall-clock: ~4 min/site with crawl artifacts cached; ~15–25 min/site from scratch (crawl + live capture dominate).

EC2 runbook (the corpus box — proven 2026-07-18 pt2)

  • Box: cathals-demo (52.48.145.217, SSH alias, user admin), repo at ~/replatform-dashboard, Debian 12, Node 22.22.0, 2 GB RAM + 6 GB swap (/swapfile 2G pre-existing + /swapfile2 4G added for the astro build). Superseded 2026-07-20 by the 8 GB resize — see "Instance identity" below.
  • Instance identity (resolved 2026-07-20, was the long-open unknown). It is Lightsail, not EC2 — the 172.26/16 private IP was the tell.
  • Current: Cathal-Dev3, bundle large_3_0 (8 GB RAM / 2 vCPU), zone eu-west-1a, blueprint LAMP, ARN arn:aws:lightsail:eu-west-1:371676573654:Instance/aa36c447-5889-4a8e-a41e-5645e0fcb147, private IP 172.26.4.31, static public IP 52.48.145.217 (retained).
  • Previous: Cathal-Dev2, small_3_0 (2 GB), private IP 172.26.9.17, now stopped. Keep it as a read-only escape hatch for corpus artifacts only — it is NOT a rollback target. Reverting the box to Dev2 would also revert the WordPress dev sites below to their 2026-07-20 08:27 state and destroy any WP work done since. Pull files off it; never promote it.
  • Account 371676573654 (shared with ~25 client production instances — tread carefully). Local aws creds are user fcrchatbot_uploader, which has Lightsail read only (get-instance* yes; GetInstancePortStates, OpenInstancePublicPorts, and all EC2 describes are denied). Firewall and power changes need IT.
  • A Lightsail resize is snapshot → NEW instance, and the firewall does not come with it. The 2026-07-20 upgrade (snapshot Cathal-Dev2-bckup-before-upgrade 08:27 UTC → Cathal-Dev3 08:39 UTC) left the box reachable on 80/443 but dropped the port 22 and port 3001 rules, which read from outside exactly like the networkd death of 07-19. Distinguish them by port, not by ping: ICMP is closed on this box by default, so ping timing out proves nothing.
  • 80/443 answer + 22 times out → firewall, box is healthy.
  • Nothing answers on any port → network/instance genuinely down.
  • 22 refuses fast (RST) rather than timing out → port open, sshd down. After any resize, re-request: 22 (SSH) and 3001 (dashboard API).
  • Co-tenant: THIS BOX IS NOT OURS ALONE. It also serves live WordPress dev sites on nginx (not the blueprint's Apache): cathaldev1.fcrweb.ie (FCR Media), cathaldev2 / cathaldev3.fcrweb.ie (BM Parts & Tools), TLS via a Let's Encrypt SAN cert expiring 2026-09-01. Verified serving 200 on 2026-07-20 after the resize.
  • Safe by construction: ports don't collide (WP on 80/443; ours on 3001 + wrangler/workerd), the kill recipe below is pattern-matched to our own process names, and the trees are separate (~/replatform-dashboard vs the nginx webroot).
  • NOT safe by construction — contention. Only 2 vCPU (the resize raised RAM only), so a corpus run saturates both cores for hours and the dev sites crawl. Don't run the corpus during a client-facing WP demo.
  • Memory is the shared failure surface — and it has bitten WordPress, not us. Checked 2026-07-20: the journal holds exactly two OOM events, 2026-02-26 and 2026-04-03, and both killed mariadbd (the WordPress DB) on the 2 GB instances — mariadb.service: Failed with result 'oom-kill'. So the co-tenancy genuinely was memory-starved, WP took the damage, and the 8 GB upgrade is the fix for it. The OOM theory for the 07-19 networkd incident is DISPROVEN — there are no OOM events in July at all. Don't re-raise it.
  • The networkd self-heal cron restarts networking box-wide — if it fires, the WP sites blip too.
  • Sync is git-bundle (no GitHub auth on the box): locally git bundle create ec2-sync.bundle master ^<ec2-head> → scp → git fetch <bundle> master && git merge --ff-only FETCH_HEAD. The box's pre-sync dirty tree (May-era ad-hoc copies) is preserved on branch rescue/pre-0718-sync; repo-local git identity is set. apps/cms/.dev.vars is gitignored — scp it separately.
  • NODE_OPTIONS=--max-old-space-size=2560 is REQUIRED on this box. The apps/cms astro build exceeds Node's default ~1 GB heap cap on a 2 GB host and dies with SIGABRT/134 (measured; the failure is loud and lands in the batch report). The raised cap + swap completes the build (~5 min, swap-bound). Post-8 GB resize (2026-07-20): 2560 was sized for the 2 GB host. Raising it lets the build go RAM-resident — but this box is shared (see "Co-tenant" below), so do NOT hand Node most of the machine. ~4096 is the sane target, leaving headroom for MySQL + php-fpm + headless Chrome + OS; 6144+ re-creates the exact overcommit the upgrade was meant to escape. Unmeasured either way — time one build before trusting it, and keep the flag set (Node's ~1 GB default still kills the build).
  • Launch (survives ssh disconnect): 
    cd ~/replatform-dashboard && NODE_OPTIONS=--max-old-space-size=2560 \
  nohup bash -c "node scripts/run-cms-batch.mjs --hosts calibration/corpus-hosts.txt; echo EXIT:\$?" \
  > ~/corpus-run.log 2>&1 </dev/null & disown
    Poll the LOG/ARTIFACTS, never the process; EXIT: in the log is the completion marker. Measured from-scratch wall-clock on this box: ~22 min/site (crawl 7.5 + capture-live 5.4 + build ~5 + capture-ours 2.4 + rest) → 30–50 sites ≈ 11–18 h sequential; resume semantics absorb an interrupted night.
  • Kill + resume: pkill -f run-cms-batch, then sweep leftovers (pkill -f workerd, pkill -f "wrangler dev", pkill -f headless_shell), verify with pgrep -af "run-cms-batch|workerd|chrom". Completed sites keep their scorecards + report rows; relaunching the same command skips them. Caveat: the site mid-flight at kill time (last ===== header in the log with no scorecard ✓) may hold partial artifacts that pass presence-based skips — re-run it with --force all --site <domain>. A leftover workerd on the port makes the runner refuse loudly (port-interception assertion): kill it, don't switch ports.
  • Stale-crawl trap for the 18 legacy builds/<domain>/ dirs on the box (static-lane-era artifacts, May 2026): the crawl skip rule is presence-only (body.html present), so a legacy site would silently pair a two-month-old crawl with a fresh live capture. For any corpus site whose builds/ dir predates the run: --force crawl,theme (or clear the dir). The 18 (bonus corpus candidates, team-built): airconditioningandrefrigeration, amsecurity, askeatyres, assuredqualityplumbing, coachhiredublin, drainunblockingrathmines, europalletsolutions, greenfelltreeservices, jkterrazzo, jrpianolessons, mcdonaghfuneraldirectors, mcguigansculptors, molansmotors, patriciaokeeffe, protechsecurity, rsplumbingandheating, seanmurrayplumbingandheating, webb (all .ie).

2026-07-19 incident — the box's NETWORK dies under sustained load (run 1 lost 34/42 sites; watchdog now installed)

The first corpus run (launched 2026-07-18 ~14:20, ~/corpus-run-0718.log) did not survive the night, and the failure was the box, not the sites. Measured chain, from the journal of the dying boot:

  1. 16:32:00 — systemd-networkd: ens5: Could not set route: Connection timed out → ens5: Failed. Under the run's load a routine networkd operation (route programming at DHCP renewal) timed out; networkd declared the primary interface Failed and stopped managing it.
  2. 16:32→17:30 — kernel routes limped on while things expired: discover 1s→30–71s, crawls to 40 min (brianlarkinsolicitor 2427s), cms-build spawnSync npm ETIMEDOUT (bmac), theme-extractor exit-1s, one raw ✗ spawn.
  3. ~17:30 — DHCP lease expired with nobody to renew → total network loss. sshd's last inbound (even internet bot noise) is 17:30:07; every remaining site instant-failed discover ("No site …"); run exited 17:32:57 with 8/42 ok.
  4. Overnight — box healthy but netless for 20 h (cron at normal speed all night; networkd-wait-online failures 03:44/06:35/06:50). The kernel never crashed and never OOM'd — the journal boot list shows one continuous boot until IT's console stop/start at 13:48/13:51 next day (clean systemd poweroff; the public IP survived).

Box changes made in response (both live as of 2026-07-19):

  • /etc/cron.d/networkd-selfheal — every 5 min, if the metadata IP (169.254.169.254) doesn't answer, restart systemd-networkd and log to /var/log/networkd-selfheal.log. Converts this failure class from a lost night into a ≤5-min blip. Check that log when triaging any future gap.
  • /swapfile2 is now in /etc/fstab. It never was — the reboot silently dropped swap 6 G → 2 G, which would have re-broken the NODE_OPTIONS=--max-old-space-size=2560 build requirement above. A reboot no longer degrades the box.

Recovery discipline (applied on relaunch, generalises): any site whose stage "succeeded" during a degradation window holds poisoned artifacts that pass the presence-based skips — run 1's five degradation-window crawls (braylaunderette, brayupholstery, brianlarkinsolicitor, brightstartcreche, brownsfruitandveg) were relaunched --force crawl,theme; braylaunderette's clean re-crawl took 31.8s vs 858s degraded, which is the proof the force was needed. Run 2 log: ~/corpus-run-0719.log (5 forced suspects → full batch resume; 8 banked sites skip, bmac resumes at cms-build).

Still unrecorded (surfaced by the outage): the instance's ID / type / account are in no repo file — diagnosis was blind until IT arrived, and the hostname (ip-172-26-9-17, a 172.26/16 address) plus the static IP surviving stop/start suggest Lightsail rather than plain EC2 (unconfirmed). Record the instance identity in this runbook when IT confirms it.

Parent: ADR-0013 §Decision 3 (the corpus must score the CMS lane's render); ADR-0009 Slice 6 (steps 1–2 of five: pick sites → batch-build → scorecard → labels → fit)

Goal

One command that takes a hosts file (30–50 FCR-built Wix domains, Cathal-picked) and produces, per site, a builds/<domain>/fidelity-scorecard.json scored against the CMS lane's rendered output, feeding the existing scripts/batch-scorecard.mjs → calibration/feature-matrix.json → labels → fit-calibration chain unchanged.

The per-site chain (all pieces exist except the serve step and the orchestration)

  1. Crawl live — dom-pipeline (body.html + assets), theme-extractor (theme.json), capture-section-backgrounds (live capture, current schemaVersion; --slides OFF by default — corpus speed beats carousel completeness).
  2. Seed — seed-from-build + transformer (capture stamps ride along exactly as on the reference sites).
  3. Image migration — transform-seed-images (ADR-0005 path; uploads confined to the site's own <domain>/ R2 prefix; idempotent).
  4. Local CMS serve — SITE=<site> emit-seed + emit-theme-css + production astro build, then serve locally (wrangler dev / miniflare) against a local D1 seeded out-of-band. Public pages only — no Access, no admin, no deploy.
  5. Ours-capture + scorecard — capture-section-backgrounds --base http://localhost:<port> → lib/scorecard.js → fidelity-scorecard.json.
  6. Batch report — per-site {site, stage, outcome} rows; a failed site is recorded loudly and the batch continues.

Runner: scripts/run-cms-batch.mjs (node, cross-platform — EC2 preferred for wall-clock, must also run on the Windows box). Resumable: a stage is skipped iff its artifact self-asserts freshness (schemaVersion / site name / mtime-vs-capturedAt — poll the artifact, never the process; house pattern).

Step 0 — prove headless local tenant bring-up — PROVEN 2026-07-18

The load-bearing unknown is resolved. The working recipe, proven on waterfordcountypainters end-to-end (public pages 200, real content, images on the /_image?href=<public R2 URL> ship path, hero video slide included):

  1. SITE=<site> npm run build in apps/cms — the vanilla ship config. The Cloudflare Access adapter does NOT gate public routes locally (measured — see trap 1 below for why it briefly appeared to), so no Access-free build variant is needed: zero config divergence from the deployed workers.
  2. npx wrangler dev --port <port> — plain, never -c (trap 2). First boot on an empty local D1 runs the emdash migrations and 302-redirects every public route to /_emdash/admin/setup — that redirect is the runner's "not seeded yet" signature.
  3. npx emdash seed -d .wrangler/state/v3/d1/miniflare-D1DatabaseObject/<hash>.sqlite (server stopped while writing — WAL lock). The emdash CLI operates on the sqlite file directly; it applied WCP's baked seed (26 content entries) and can run the migrations itself if the pre-boot is skipped (it reported "Database up to date"). The <hash> derives deterministically from the D1 database_id; a glob on that directory finds it.
  4. Restart wrangler dev → public pages serve with real content. No admin user, no Access, no deploys, no remote writes.

Three traps measured on the way, all runner requirements now:

  1. Port interception. A leftover workerd from a different repo (quick-spike) shared 127.0.0.1:8787 and answered every probe with a plausible JSON 403 ("protected by Cloudflare Access…") that exists nowhere in this workspace — two builds were misdiagnosed before checking the port owner. The runner must (i) use a unique high port per site, and (ii) assert responder identity before capturing — fetch the page and require the site's own name in the HTML, never trust an HTTP 200/403 alone.
  2. wrangler dev -c <file> bypasses .wrangler/deploy/config.json (the Astro adapter's redirect to dist/server/wrangler.json, which has main: entry.mjs + no_bundle). With -c, wrangler tries to bundle src/worker.ts itself and dies on Astro build-time virtuals. Plain wrangler dev only.
  3. Corpus sites have no fleet descriptor. emit-wrangler/emit-seed key off sites/<site>.jsonc, which corpus sites won't have (and whose database_id is a remote fact). The runner must synthesize a local-only descriptor per corpus site (any UUID works as database_id — it only namespaces the local sqlite) without polluting the committed fleet descriptors.

In scope

  • Step 0 spike; the runner; the local-serve mechanism; the batch report; the runbook (one command + hosts file + creds list + where it runs).
  • Positive control (done criterion a below).
  • Concurrency/chunking sufficient for a 30–50-site overnight run.

Out of scope (named)

  • Site selection (Cathal), labels, the fit, calibration promotion — the corpus lane's human half.
  • Fixing any defect the batch surfaces (that backlog is the point of the batch — file, don't chase).
  • Worker deploys, Access apps, cross-account anything (separate ADR-0008 Slice 3 spike).
  • The emdash save race (separate product-blocker track).
  • Static-lane anything (void per ADR-0013).
  • --slides / video acquisition per corpus site (VOD is blocked at source anyway).

Done criteria

  • (a) Positive control: the runner, on waterfordcountypainters.ie from a fresh crawl, produces a scorecard whose S/T/G_det sit within re-crawl drift of the deployed-worker baseline (composite within ~2 points; no new veto classes beyond the known videoDebt). Local render ≡ deployed render, proven not assumed.
  • (b) Unattended: ≥3 sites including ≥1 never-before-crawled site run end-to-end unattended; each yields a complete scorecard or a loud, named, per-site failure; the batch continues past failures.
  • (c) Downstream unchanged: batch-scorecard.mjs consumes the outputs as-is (feature-matrix rows appear; its loud-skip on partial scorecards still fires).
  • (d) Zero production writes: the runner contains no deploy path and no remote-D1 path; R2 writes confined to <domain>/ prefixes. Verified by inspection + by running a full site with remote wrangler creds absent.
  • (e) Runbook committed (command, hosts-file format, creds, runtime, resume semantics).

Time estimate (honest order of magnitude)

  • Step 0: ~half a session (real-stop possible).
  • Runner + positive control: 1–1.5 sessions.
  • The 30–50-site run itself: wall-clock, ~15–25 min/site → chunked overnight on EC2; plus Cathal's labelling time (separate, after).

Risks

  1. Step 0 fails headless — the named real stop. Most likely resolution is path (b); if both fail, ADR-0013 must hear it.
  2. Capture robustness on unfamiliar templates — ADR-0009 itself predicts a 70–85% initial clean rate. Per-site loud failures are corpus data (they measure capture robustness), not slice failure.
  3. Wall-clock underestimate — 25-page sites with full capture may exceed 25 min; chunking + resume is the mitigation, not tighter estimates.
  4. Runtime drift between boxes (Windows vs EC2 Playwright/node) — the positive control runs on whichever box the corpus will run on.
  5. R2 credential availability on the corpus box (EC2 pre-stage was confirmed reachable in the ADR-0005 era; re-verify, don't assume).
  6. A local-serve render that silently diverges from the deployed worker (env vars, PUBLIC_MEDIA_BASE, image endpoint config) — this is exactly what done-criterion (a) exists to catch; any divergence is a finding, not a tolerance.