747525f3a7e1e75fc26f1969502b6029545b1c19 diff --git a/cms-build-log.md b/cms-build-log.md new file mode 100644 index 0000000000000000000000000000000000000000..4222249b1cd8b1994851a2decba82c9bc6ddacba --- /dev/null +++ b/cms-build-log.md @@ -0,0 +1,256 @@ +# CMS ↔ Forgejo build log — SAMA tensions encountered + +A diary of moments where building the self-hosted CMS-with-real-git-commits +made the SAMA convention either *help us instantly* or *fail to have an +opinion*. Written as we built [c14 commitFile + c21 admin-fork wire-in], +not after the fact. + +## Tension 1 — c14 has no sibling test, by design + +**Where it bit:** I added `commitFile()` and `getFileSha()` to +[c14_forgejo.ts](src/c14_forgejo.ts). Those are mutations against an +external service. SAMA's Modeled rule says *"every behaviour has a +test file as its sibling"*. But `c14_forgejo.test.ts` would either +need a real Forgejo (turns the unit suite into an integration suite) +or a mocked `fetch` global (couples the test to implementation +details, not behaviour). + +**What I did:** Split the work. The pure parts — message format, +noreply email shape — moved into [c31_forgejo_commit.ts](src/c31_forgejo_commit.ts) +with a real sibling test [c31_forgejo_commit.test.ts](src/c31_forgejo_commit.test.ts). +The c14 wrapper itself stays untested at unit level. + +**Candidate SAMA refinement:** Modeled needs a sub-rule for c14: +> A c14_*.ts file is exempt from the sibling-test rule when its +> entire body is HTTP request construction + response parsing. Its +> coverage comes from (a) c31 sibling tests for any pure helpers, +> and (b) integration tests separately marked (`*.contract.test.ts` +> or e2e Playwright). Without this carve-out, the rule pushes I/O +> into mocks, which test the mock more than the I/O. + +## Tension 2 — failure semantics for cross-boundary mutations + +**Where it bit:** Admin POST does two writes — Forgejo commit AND +local filesystem. SAMA prescribes *which layer* (c14 for HTTP, c21 +for compositions, c13 for the SQLite proposal row), but is silent +on **what order** and **how to roll back when one succeeds and the +next fails**. There's no SAMA letter for "transactional intent". + +**What I did:** Added a per-failure-mode discriminated union +([c14_forgejo.ts ForgejoCommitOutcome](src/c14_forgejo.ts)) so the +handler can render a useful error per kind: `conflict | not_found | +auth | network | other`. Chose Forgejo-first / FS-second deliberately +— Forgejo's contents API is the synchronous source of truth, FS is +a derived view. Documented the matrix in +[cms-forgejo-plan.md](cms-forgejo-plan.md). + +**Candidate SAMA refinement:** Add a doctrine — not a layer letter, +but a discipline: +> **Boundary contracts.** Any function that crosses a process or +> network boundary returns a discriminated-union outcome, never an +> exception, and never a bare boolean. Failure kinds are part of +> the type, the renderer per kind is one switch in c51. + +This isn't strictly a SAMA layer rule — it's an idiom. But the +absence of it made me reach for try/catch in the handler, which is +the wrong layer for "what does conflict mean to the user". + +## Tension 3 — types-with-data-source vs types-with-pure-logic + +**Where it bit:** `ForgejoCommitOutcome` lives in c14_forgejo.ts. +`CommitMessageInput` lives in c31_forgejo_commit.ts. Both describe +"a Forgejo commit" — but one is the I/O surface, the other is the +input we shape. The c21 handler imports from both. Felt wrong but +also unavoidable. + +**What I did:** Followed the existing codebase pattern — types +travel with the data source (c13_database.ts owns ProposalRow, +c14_forgejo.ts owns the response shape, c31 owns the inputs we +parse). Resisted the urge to make a c31_forgejo_types.ts barrel. + +**SAMA verdict:** No refinement needed. The current pattern is +internally consistent — types live with whoever first defines them, +imports are explicit. A barrel would have been less honest. SAMA's +"no barrel re-exports" rule already covers this case if you take +its spirit seriously. + +## Tension 4 — schema migration without an `IF NOT EXISTS` + +**Where it bit:** Added a `forgejo_commit_sha` column to the +existing `proposals` table. SQLite has no +`ALTER TABLE … ADD COLUMN IF NOT EXISTS`. The pre-existing pattern +in [c13_database.ts](src/c13_database.ts) is `CREATE TABLE IF NOT +EXISTS …` at startup — purely additive, no migrations. This is +the first time we needed to evolve a table. + +**What I did:** Read `PRAGMA table_info(proposals)`, check column +names, only `ALTER` if the column is missing. Idempotent across +container restarts and across rebuilds with old data volumes. + +**Candidate SAMA refinement:** c13 layer needs a discipline note: +> **Migration is additive.** Schema changes go in a `migrations` +> block of `getDb()` after the `CREATE TABLE IF NOT EXISTS` block, +> guarded by a PRAGMA probe (or `sqlite_master` query). Never +> destructive. Never renames. If you need a rename, it's two +> migrations: add the new column, deprecate the old one. + +This is a one-liner addition to the Architecture page (c13's +contract). + +## Tension 5 — the deploy script is now lying + +**Where it bit:** `deploy-tdd-md.sh` rsyncs the dev working tree to +p620 and rebuilds the image. Now that admin edits commit to +git.tdd.md, the *next* deploy will overwrite those commits with +whatever's in dev's working tree. The container is briefly the +canonical source between admin save and next deploy. Then dev's +working tree wins. + +**What I did:** Documented this as out-of-scope in +[cms-forgejo-plan.md](cms-forgejo-plan.md). Surfaced it in the +applied-live page copy ("Until then, sync your local working tree +with `git pull` before the next `deploy-tdd-md.sh`"). Did **not** +fix the deploy script in this PR — that's a workflow decision, not +a code change. + +**Candidate SAMA refinement:** None — this is operational, not a +SAMA-layer issue. But it does point at a missing piece in the +project's *story*: there's no diagram of "where the truth lives at +each second of a request → deploy → restart cycle". A +state-diagram in plan.md or a guides/ page would help. + +## Tension 6 — same-process self-reference + +**Where it bit:** c21 handler calls c14 (Forgejo PUT). Forgejo lives +on the same physical box (host.containers.internal). When it +returns success, c21 also writes the same file to the local FS. +We're now writing the same bytes twice through different paths, +and they could disagree if Forgejo did something weird (LF/CRLF, +trailing newline normalisation). SAMA doesn't tell us where to +verify they agree. + +**What I did:** Trusted Forgejo to round-trip UTF-8 verbatim. Added +no read-after-write check. If this becomes a source of bugs we'd +add a c32 helper `verifyCommittedContent(sha, expected)` that +re-reads via `getFileSha` + a contents fetch. + +**SAMA verdict:** Doesn't need a layer rule. Add to the c14 doctrine +note: "external services own their own normalisation; if you care +about exact bytes, verify via a follow-up read". + +## Tension 7 — SAMA doesn't tell you to read the API docs first + +**Where it bit:** I assumed Forgejo's contents API was upsert-on-PUT +like GitHub. Wrong. Forgejo requires `POST` for create (no SHA), +`PUT` for update (SHA mandatory). First admin POST returned a +422 "[SHA]: Required" we then surfaced as "other" failure. + +**What I did:** Branch on `priorSha === null` in +[c14_forgejo.commitFile](src/c14_forgejo.ts) — POST when null, PUT +when set. Two-line fix. + +**SAMA verdict:** Not a SAMA problem. SAMA prescribes structure, +not external-service semantics. The c14 layer is the right place +for this knowledge to live (and it does, in the wrapper). The +lesson is generic: c14 contracts must encode every quirk of the +service they wrap, because no other layer is allowed to know. + +## Tension 8 — HTTP status codes through a CDN + +**Where it bit:** I returned `502 Bad Gateway` for "Forgejo is +unreachable" failures. Cloudflare (our CDN front) replaced our +HTML body with its own 502 error page template, so the admin saw +a Cloudflare error instead of our diagnostic. The handler did +exactly the right thing in HTTP-purist terms; the user-visible +result was wrong. + +**What I did:** Switched to `200 OK` for non-conflict commit +failures (kept `409` for conflict because that's actionable client- +side and CDNs don't intercept 4xx). The HTML body carries the +error semantics; the status code is just for routing. + +**Candidate SAMA refinement:** Add a c21 doctrine note: +> When the site sits behind a CDN that intercepts 5xx responses, +> render handler-generated error pages with `200 OK` (or with +> a 4xx the CDN passes through). Never let a synthetic 5xx +> compete with a real upstream 5xx for the user's attention. + +This is operational/CDN-specific so it might not belong in SAMA +proper. A `guides/cloudflare-and-handlers.md` page would be more +honest than expanding SAMA itself. + +## Tension 9 — "we don't depend on Forgejo" required actually proving it + +**Where it bit:** The previous iteration committed via Forgejo's HTTP +contents API. Operationally fine, but the system *did* depend on +Forgejo being reachable. To prove genuine independence we replaced +the Forgejo write-path with a local bare repo + `git` plumbing inside +the container. + +**What I did:** Added [c14_git.ts](src/c14_git.ts) which shells out +to `git` against `/app/repo` (a bind-mount of `~/repos/tdd.md.git` +on p620). Plumbing chain per commit: +`hash-object → read-tree (temp index) → update-index → write-tree +→ commit-tree → update-ref` with the parent SHA as the expected +old-value for free optimistic concurrency. The pure parts (commit +metadata format, ls-tree output, log-format string) live in +[c31_git_parse.ts](src/c31_git_parse.ts) with 11 sibling tests. + +**The proof:** Two Playwright tests: +1. [git-native-proof.spec.ts](e2e/git-native-proof.spec.ts) saves a + marker via the editor, then SSH's into p620 and reads the bare + repo HEAD — must equal the SHA the editor reported. Confirmed: + bare repo advanced (`bb9c023`), Forgejo HEAD stayed at `526fa16`. +2. [git-native-forgejo-down.spec.ts](e2e/git-native-forgejo-down.spec.ts) + stops `forgejo.service` via systemd, performs an admin edit, and + verifies the commit lands. The test passes only if Forgejo is + genuinely off the edit path. Confirmed: commit `eb1d791` landed + while Forgejo was returning HTTP 000. + +**SAMA verdict:** No new violations, and the layer hygiene actually +stayed clean *because* the Modeled exemption from tension 1 was +already the pattern — c14_git is a shell-out wrapper, c31_git_parse +holds the pure helpers with sibling tests. We applied the rule +that emerged from the previous build to the next layer that needed +it. That's the discipline working. + +## Tension 10 — naming drift after a refactor + +**Where it bit:** `c31_forgejo_commit.ts` no longer talks about +Forgejo — it owns commit-message format and noreply email shape, +which c14_git uses just as well. Filename was a lie. + +**What I did:** Renamed to `c31_commit_meta.ts` (+ `.test.ts`). +One-import update in c21_handlers_edit.ts. SAMA's "the number is +the layer; the layer is the contract" doctrine doesn't strictly +require accurate names — it requires accurate prefixes — but the +spirit is that file names communicate intent. Drift after refactor +is a small but real form of debt. + +**Candidate SAMA refinement:** Note in the Atomic doctrine that +when a file's contents diverge from its name during a refactor, a +rename is part of the refactor, not optional polish. + +## Summary — what would I add to SAMA after this PR? + +Two concrete additions: + +1. **Modeled exemption for I/O-only c14 files.** Sibling-test rule + carves out a path: pure helpers extracted to c31, the I/O + wrapper itself can stand alone with integration coverage. +2. **Boundary-contract idiom.** Cross-process functions return a + typed discriminated union, not a thrown exception. The renderer + layer (c51) handles each kind explicitly. + +Two doctrine notes (not new letters): + +3. **c13 migrations are additive + probe-guarded.** Never + destructive in-place, never renames. +4. **Source-of-truth state diagram.** Document where the canonical + bytes live at each step of write → commit → deploy → restart so + future contributors don't have to reverse-engineer it. + +The four SAMA letters (Sorted, Architecture, Modeled, Atomic) held +up. They don't *cover* every situation — the gaps above are real — +but they didn't force us into bad shapes either. The discipline +held; the doctrine needs a few footnotes. diff --git a/cms-forgejo-plan.md b/cms-forgejo-plan.md new file mode 100644 index 0000000000000000000000000000000000000000..9aacb46ea13a54f647f18b41b3b4d615bc381197 --- /dev/null +++ b/cms-forgejo-plan.md @@ -0,0 +1,113 @@ +# CMS ↔ Forgejo integration as a SAMA stress-test + +## Goal + +Make every admin direct-write a real git commit in `git.tdd.md/syntaxai/tdd.md`, +without putting a `.git` working tree, ssh keys, or the `git` binary's repo-write +capabilities inside the container. The container only ever speaks HTTP to Forgejo +(which it already does for users/repos/webhooks). This closes the loop: admin +edits become permanent in git automatically, no manual download-patch-and-commit +bridge required. + +## Architecture delta + +``` +BEFORE AFTER +admin POST /edit/... admin POST /edit/... + → Bun.write content/...md ←┐ → Bun.write content/...md + → SQLite proposal=approved │ → fetch Forgejo PUT contents + → render "applied live" │ → real commit in syntaxai/tdd.md + │ → SQLite proposal=approved + sha + manual: download patch ─────┘ → render "applied live + commit" + manual: git commit on dev + manual: deploy +``` + +Layer touchpoints: + +| Layer | File | Change | +|---|---|---| +| c14 | `c14_forgejo.ts` | + `getFileSha()`, + `commitFile()` | +| c31 | `c31_forgejo_commit.ts` (new) | input/result types + validator | +| c31 | `c31_forgejo_commit.test.ts` (new) | sibling test | +| c13 | `c13_database.ts` | + `forgejo_commit_sha` column on proposals + setter | +| c21 | `c21_handlers_edit.ts` | call commitFile in admin fork | +| c51 | `c51_render_edit.ts` | show commit SHA + Forgejo URL on applied-live + admin detail | + +## Failure semantics (the bit SAMA didn't have an opinion on) + +Order: **Forgejo first, filesystem second**. Trade ~150ms latency for guaranteed +consistency. Reasons: + +1. Forgejo's `PUT contents` requires the previous SHA → free optimistic + concurrency. If two admin tabs save concurrently, the second one fails with + 409, we surface a "conflict — refresh and retry" message. +2. If we wrote FS first and Forgejo failed, we'd have a live-but-uncommitted + change that vanishes at the next deploy. That recreates the exact problem + we're trying to eliminate. +3. The container is the single writer. No one else touches the repo from the + admin path, so reading-the-SHA / writing-with-the-SHA is a tight loop with + no contention. + +Failure matrix: + +| Forgejo result | Filesystem step | Proposal status | UI | +|---|---|---|---| +| 200/201 | execute write | `approved` + commit_sha | "applied live · commit ``" | +| 409 conflict | skip | `pending` (kept for retry) | "conflict — content changed elsewhere; refresh" | +| 5xx / network | skip | `pending` | "Forgejo unreachable; queued for retry" | +| 4xx other | skip | `pending` | "Forgejo rejected: ``" | + +## Implementation steps (in order) + +1. **c14_forgejo.commitFile()** — pure HTTP wrapper, no business logic. + Returns `{ ok: true, sha } | { ok: false, status, message }`. +2. **c31_forgejo_commit.ts** — typed input contract + validator + a parser + that turns Forgejo's response shape into the typed result. Gets a sibling + test that uses fetch-mocking via `mock.module` or a small interface. +3. **c13_database** — schema migration: `ALTER TABLE proposals ADD COLUMN + forgejo_commit_sha TEXT`. Setter `setProposalCommitSha(id, sha)`. + `IF NOT EXISTS` semantics via try/catch since SQLite has no `IF NOT EXISTS` + for column adds. +4. **c21_handlers_edit admin fork** — call commitFile *before* applyLiveEdit. + On success, write FS + setProposalStatus(approved) + setProposalCommitSha. + On failure, render an error variant, set status to pending, do NOT touch FS. +5. **c51 render** — applied-live page shows the commit SHA short (7 chars) and + links to `https://git.tdd.md/syntaxai/tdd.md/commit/`. Admin proposal + detail does the same. +6. **Tests + deploy + verify** — bun test green, Playwright green, deploy to + p620, then a curl `GET /api/v1/repos/syntaxai/tdd.md/commits` to confirm + the commit really landed. + +## Out of scope for this PR + +- **Deploy script switching to pull-from-Forgejo**. Right now `deploy-tdd-md.sh` + rsyncs from dev's working tree. Until that switches to `git pull` from Forgejo + on dev (or directly on p620), admin commits in Forgejo will be overwritten on + the next deploy unless dev pulls them first. That's a separate, scarier + workflow change. +- **Webhook from Forgejo back to tdd.md to refresh the in-container `content/`**. + Tempting (would make container restart resync) but introduces a feedback loop + potential. +- **Non-admin proposals committing as branches**. Right now they stay in SQLite + as pending. Could become `proposal/` branches in Forgejo. Out of scope. +- **GitHub mirror**. Forgejo can mirror to GitHub, that's a Forgejo config, not + code. + +## SAMA-tension log (filled as we hit them during the build) + +> A running list. Each entry: the tension, the workaround, and a candidate +> SAMA refinement (or "SAMA has nothing to say here, that's fine"). + +_(empty — start blank, populate as we hit them in the implementation)_ + +## Done-criteria + +- [ ] Admin POST on /edit/sama/skill writes to FS *and* creates a real commit + visible at `git.tdd.md/syntaxai/tdd.md/commits/main` +- [ ] Applied-live page shows the commit SHA with a working link +- [ ] Forgejo down → no FS write, proposal stays pending, UI surfaces the + reason +- [ ] Concurrent edit (409) handled with a useful error +- [ ] All bun tests + Playwright tests green +- [ ] SAMA-tension log has at least 2 concrete entries diff --git a/content/blog/sama-meets-git-cms.md b/content/blog/sama-meets-git-cms.md new file mode 100644 index 0000000000000000000000000000000000000000..a9186517dc99d5fd759f97fe6104d6cb8f277c79 --- /dev/null +++ b/content/blog/sama-meets-git-cms.md @@ -0,0 +1,148 @@ +# SAMA meets git: building a self-hosted CMS that obeys the discipline + +The simplest CMS test is meta: edit the page that describes the CMS, +through the CMS, and watch it commit itself to git. That's how this +post landed in the repo. + +## The setup + +`tdd.md` is a Bun-native site, ~40 source files following the SAMA +file-naming convention (Sorted, Architecture, Modeled, Atomic). Pages +live as markdown in `content/
/.md`. Until last week the +only way to edit one was `vim content/blog/some-post.md && git commit +&& ./scripts/p620/deploy-tdd-md.sh`. No web editor. + +We had a goal: edit pages from a browser, but **never bypass git**. +The git history is the audit log. Anything that changes a page must +become a commit a reviewer can `git blame`. + +## First build — a SQLite proposal queue + +The obvious shape: GitHub OAuth for identity, a textarea, save +submissions to a SQLite `proposals` tabel as `pending`. The admin +reviews at `/admin/proposals`, downloads the `.md` patch, applies it +on dev, commits, deploys. + +This worked but felt wrong. The audit trail lived in two places: +SQLite knew who submitted what, git knew what actually shipped. The +manual download-patch-and-commit step was a brittle bridge between +them. And the SQLite proposals tabel was a duplicate of git's job — +versioning page bytes — done worse. + +## The realisation — Forgejo can be the canonical writer + +`tdd.md` already runs a Forgejo mirror at `git.tdd.md`, used for +agent kata repos. Forgejo has a contents API: `PUT +/api/v1/repos/{owner}/{repo}/contents/{path}` creates a real commit +from an HTTP call, no git binary or SSH keys required on the caller. + +So: **what if the editor commits directly to Forgejo?** + +``` +admin POST /edit/sama/skill + ↓ +fetch PUT https://git.tdd.md/api/v1/repos/syntaxai/tdd.md/contents/content/sama/skill.md + body: { branch: "main", content: base64(body), message, sha?: previousSha } + Authorization: token $FORGEJO_ADMIN_TOKEN + ↓ +real commit lands in syntaxai/tdd.md@main + ↓ +Bun.write to ./content/sama/skill.md so the live page reflects it + ↓ +"applied live · commit a93f01f" +``` + +No SQLite. No proposal queue. No download-patch step. The Forgejo +commit IS the audit trail. + +## Stripping the dead layers + +Once Forgejo became the writer, the SQLite `proposals` tabel was +load-bearing for nothing. Out it went, along with `c31_proposals.ts`, +`c21_handlers_admin.ts`, six render functions in `c51_render_edit.ts`, +five route entries in `c21_app.ts`, and one test file. The bundled JS +size dropped 12 KB. SAMA didn't shrink — it tightened. Each remaining +file does less, but more honestly. + +## SAMA-tension log — what the build surfaced + +Building this turned out to be a pretty good stress test of the +discipline. Eight tensions, written down as we hit them +([cms-build-log.md](https://git.tdd.md/syntaxai/tdd.md/src/branch/main/cms-build-log.md) +in the repo). The two that warrant SAMA refinements: + +**1. Modeled exemption for I/O-only c14 files.** +`c14_forgejo.ts` is an HTTP wrapper. SAMA's "tests live next to +source" rule wants `c14_forgejo.test.ts`. But unit-testing an HTTP +wrapper either needs a live server (turns the suite into integration) +or `mock.module` (tests the mock, not the wire). We split: the pure +parts (commit-message format, noreply email shape) moved to +`c31_forgejo_commit.ts` with a real sibling test; the I/O wrapper +stands alone with integration coverage via Playwright. SAMA needs a +sub-rule for c14: pure helpers extract to c31, the wrapper is exempt. + +**2. Boundary contracts as discriminated unions.** +A cross-process function shouldn't `throw` on failure — failure modes +are part of the contract. `commitFile()` returns +`{ ok: true, commitSha } | { ok: false, kind: "conflict" | "auth" | +"network" | "not_found" | "other", status, message }`. The handler +does `if (!outcome.ok)` and the renderer has one switch per kind. +This is an idiom, not a layer rule, but its absence let me reach for +try/catch in the handler — wrong layer for "what does conflict mean +to the user". + +The other six tensions are written up in the build log. Two led to +operational doctrines (additive SQLite migrations, source-of-truth +diagrams), two were SAMA-internally consistent and needed nothing, +and two were external-service quirks that the c14 layer correctly +absorbs (Forgejo's `POST` for create / `PUT` for update; Cloudflare +intercepting 5xx responses). + +## The deploy pipeline became git-native too + +With the CMS committing to Forgejo, the old deploy script (rsync +local working tree to p620) was now lying. An admin web-edit would +land in Forgejo, then the next deploy would overwrite it with +whatever was on dev's filesystem. + +The fix is the same insight: **Forgejo is the canonical source**. +The new `deploy-tdd-md.sh` does: + +```sh +ssh p620 'cd ~/src/tdd.md && git fetch origin && git reset --hard origin/main' +ssh p620 'podman build ... && systemctl restart tdd-md.service' +``` + +Dev pushes to git.tdd.md. Admin web-edits commit to git.tdd.md. +Deploy pulls from git.tdd.md. One source, three writers, all going +through real git commits. The legacy `--rsync` flag stays as an +emergency escape hatch for deploying uncommitted dev changes. + +## What SAMA proved + +Across this build SAMA's four letters held up — Sorted (no +import-direction violations), Architecture (clean layer composition, +c14 owns Forgejo's HTTP, c31 owns the pure shaping, c21 composes, +c51 renders), Modeled (sibling tests where they fit), Atomic +(deletion shrunk the largest files instead of growing new ones). + +The gaps SAMA had — failure-contract idiom, c14 testability — became +visible specifically because we built something that pushed past +single-process pure logic into a real cross-boundary mutation. That's +the value: SAMA isn't trying to cover every situation, it's trying to +make the gaps it doesn't cover *legible*. + +## You can verify this post is real + +- **Source on git.tdd.md:** + [content/blog/sama-meets-git-cms.md](https://git.tdd.md/syntaxai/tdd.md/src/branch/main/content/blog/sama-meets-git-cms.md) +- **Edit it via the CMS:** + [/edit/blog/sama-meets-git-cms](/edit/blog/sama-meets-git-cms) + (admin-only — you'll see the login wall) +- **Raw markdown:** + [/content/blog/sama-meets-git-cms.md](/content/blog/sama-meets-git-cms.md) + +When syntaxai saves an edit through that editor, a real commit lands +in syntaxai/tdd.md within a second. The SHA shows up on the +"applied live" page. The next deploy pulls it. The cycle closes +without leaving Bun, Forgejo, and four well-named source files.