Your pipeline is lying by staying silent: the append-only ledger that catches broken publish packages

Three separate failures sat in this repo for days, and not one left a trace a tool could see.

• The rewrite of the strongest article never got new LinkedIn, Medium, or Substack packages. Three of four platforms simply were not generated, and no list anywhere said so.
• The X package shipped its text but not its images. The runner wrapped the render step in a bare try/except: pass, so a headless-Chrome failure vanished.
• The audiobook, quiz, and read-along files for the rewritten article all existed on disk. Which version of the article produced them? Undecidable. Nothing recorded it.

The second one deserves the exhibit. This shape was live in my syndication runner (condensed):

try:
    subprocess.run(["node", "gen-assets.mjs", "--slug", slug], capture_output=True)
except Exception:
    pass

A package that would post without its images, and the run reports success. The failure was not rare. It was unreportable.

A file on disk answers exactly one question, "does something exist", and pipeline state needs three more: is it current, is it complete, and was it ever posted.

Every generator I had used presence as its skip signal. File exists, skip the step. That heuristic cannot distinguish:

When I computed a fresh plan over the rewritten article, every present artifact came back with the same honest verdict: skip, question mark, asterisk. Present, but freshness undecidable. That asterisk is the whole disease.

flowchart LR
  A["file exists?"] -->|"the only question<br/>presence can answer"| B["skip it"]
  B --> C["stale quiz kept"]
  B --> D["missing packages invisible"]
  B --> E["swallowed render failure"]
  F["event ledger"] -->|"records status, inputs hash,<br/>failure, post id"| G["skip / stale / run / failed<br/>decided, not guessed"]

Every pipeline action becomes one appended line in a committed NDJSON file, and current state is always a fold over those lines, never a second stored copy.

Six event kinds cover the lifecycle: lifecycle, artifact, review, post, metric, decision. The files live in content/ledger/<slug>.ndjson, in git, greppable by any agent with zero credentials. A backfill pass seeded the history: 159 presence-grade events across 32 pieces of content, each honestly marked as "this existed at backfill, provenance unknown".

Here is a real failure record, from forcing the read-along aligner to run on a piece with no audio:

{"kind": "artifact", "artifact": "align", "status": "failed",
 "error": "exit 1: no manifest (run gen-audiobook first): ...",
 "retry_cmd": "node --experimental-strip-types scripts/align-narration.mjs --slug btc-bot",
 "cost": {"wall_s": 0.3}}

Compare that to the old world, where the same failure was an exit code somebody's except clause ate. The record carries the error, the exact retry command, and what the attempt cost. A failure is now a row you can query, not an absence you have to notice.

Posted-ness finally has a home too. When a package goes out, a post event records the platform, the post id, and the URL. That id is the join key the whole measurement loop hangs off later.

A declarative registry turns the pipeline into a dependency graph, and a content-hash key per artifact decides skip, stale, or run.

I counted fourteen separate commands in the old runbook for one fully-processed article: the write, the illustration review, the quiz, the voice server, the audiobook, the narration check, the alignment, the upload, the syndication runner, the studio compile, the asset render, the screenshot, the short, the deploy check. In order, from memory, with the right environment for each.

Now the graph lives in one registry file. For my flagship article that expands to a 28-artifact plan. Each artifact declares its inputs, its command, its outputs, and a validation check. The key is a hash of the artifact id, the generator version, the input contents, and the parameters:

flowchart TD
  O["object"] --> Q["quiz"]
  O --> N["narration script"] --> V["numbers check"] --> AU["audio"] --> AL["align"]
  O --> SX["synd-x"] --> PX["post-x"] --> AX["assets-x"] --> SH["screenshot"] --> PUB["publish (human)"]
  K["key = sha256(id, generator version,<br/>inputs, params)"] -.->|"same key + valid output = SKIP<br/>changed inputs = STALE<br/>anything else = RUN"| O

Run it twice, the second run skips everything. Edit the article, only the artifacts that consume the changed fields go stale. A step that fails writes a failed event with the retry command and the run exits nonzero. The bare try/except in the syndication runner is gone, replaced by the same loud path.

Two guardrails ride along. Expensive stale artifacts on an already-published piece do not silently regenerate, they wait for an explicit flag, because republishing is a decision, not a cache event. And a budget config in the ledger holds paid or GPU stages once a daily ceiling is hit, recording budget-held instead of overspending.

A ledger only stays true if every writer records into it, so the failure mode to watch is the tool that bypasses it.

The honest limits, stated plainly:

• Backfilled history is presence-grade. It says "this existed", not "this was correct". Real keys arrive as each artifact is regenerated through the new path.
• The human gates stay. Posting is still a person, and the post id is recorded by hand for now. The ledger makes the queue visible, it does not remove the judgment.
• The audit command compares ledger claims against disk both ways: artifacts recorded done but missing, and files present but never recorded. That two-way diff is the standing tripwire for bypass drift.

The next payoff is already wired: metrics checkpoints join on the recorded post ids, so "what did this post actually do" becomes one more appended event instead of a spreadsheet. The loop closes where the silence used to be.