Signed delivery reference
The GitHub CLI verifies the original bundle against an explicit provenance policy. Failed verification prevents local CLI promotion or rollback.
ENGINEERING CASE STUDY / SECURE DELIVERY REFERENCE
Verify the build.
Rehearse recovery.
Check signed artifact provenance before changing local release state. Then use a separate HTTP lab to make failure detection and recovery observable.
- My contribution
- Design, implementation and failure diagnosis. Package verification, local promotion state, real-verifier regression coverage and an HTTP recovery rehearsal.
- Stack
- Python · GitHub Actions · GitHub attestation verification · Docker · CodeQL
- Project scope
- Independent local demonstrations. The signed delivery reference and the unsigned recovery lab retain separate evidence and trust boundaries.
01 / THE PROBLEM & MY APPROACH
Know what you promote.
Check what you restore.
Make release trust explicit.
A package checksum identifies bytes. The delivery CLI also verifies that the signed provenance meets the approved repository, workflow, issuer, runner and commit policy before it changes local promotion or rollback state.
I kept the state model small so the decision and its failure behavior could be examined without a cloud account.
Make recovery measurable.
The separate reliability lab records real HTTP requests to owned local workers. A deliberately injected health failure creates a known incident, an alert triggers the harness runbook, and the previous package must pass recovery checks.
The retained samples show which requests failed and what was checked before the rehearsal was closed.
Separate recovery lab
The lab uses unsigned synthetic packages and checksum-only state helpers. It demonstrates HTTP detection and a recovery runbook; it does not exercise the attestation gate.
02 / ARCHITECTURE & DECISIONS
Inspect the boundary
behind each decision.
Choose an exercise, then a component to read its purpose, tradeoff and implementation.
SIGNED RELEASE REFERENCE
Bind the bytes to the approved build.
A manually dispatched main-branch release builds and attests the package. The promotion gate checks those bytes against an independently approved source commit.
BUILDReproducible packageapp.py + commit/version manifest
SIGNGitHub attestationExact ZIP digest · Actions identity
OPERATOR INPUTApproved source commitIndependent 40-character commit
PRESERVERetained release filesZIP, checksum and original Sigstore bundle
ACCEPTLocal promotion stateActive + retained previous package
VERIFY BEFORE PROMOTIONProvenance gateBytes, manifest, signature and identity
COMPONENT NOTES
Reproducible package
- Purpose
- Create an allowlisted ZIP with fixed member metadata and a SHA-256 sidecar. The release job requires passing test and CodeQL jobs, workflow_dispatch and the main ref before it builds.
- Tradeoff
- A small package is easy to inspect and reproduce. A checksum detects changed bytes but does not authenticate their origin; signing is a separate step.
GitHub attestation
- Purpose
- Use the release job’s GitHub Actions identity to attest the exact ZIP digest. Signing permissions are scoped to that release job; pull requests cannot reach it.
- Tradeoff
- Provenance binds an artifact to a build identity. It does not certify that the application is free of defects or suitable for every environment.
Approved source commit
- Purpose
- Supply the expected commit independently of the package. The verifier requires the manifest, source digest and signer digest to agree with it under the fixed repository/workflow/main identity.
- Tradeoff
- The operator must choose the intended build. Reading an expected commit from the untrusted package would remove this independent acceptance check.
Retained release files
- Purpose
- Keep the signed bytes and copy the original attestation bundle to provenance.jsonl. The release workflow also retains the verifier’s demonstration evidence alongside them.
- Tradeoff
- Co-locating files makes later inspection easier, but JSON claims alone are not signature proof. Verification still depends on the GitHub CLI and its trust checks.
Local promotion state
- Purpose
- Call the full provenance gate before atomically changing local state. Public CLI rollback reauthorizes the retained previous package with its own approved commit and provenance.
- Tradeoff
- This records a local release choice. It does not deploy a service, switch live traffic or provide a transaction against a hostile filesystem.
Provenance gate
- Purpose
- Check the hash and ZIP structure, match the approved commit, then call gh attestation verify. Require the exact workflow/main certificate identity, GitHub OIDC issuer, source and signer digests, SLSA v1 predicate and a hosted runner. Rehash after the external verifier.
- Tradeoff
- Rejected signatures, missing tooling, timeouts and verifier failures all block promotion. The real-CLI regression check covers a boundary that mocked wrapper tests previously missed.
DISPOSABLE LOOPBACK REHEARSAL
Observe. Recover. Verify the result.
Real HTTP samples drive a small alert state machine. An explicit harness runbook restores the previous package and checks that recovery holds.
SYNTHETIC SERVICEOwned local workerHealthy baseline → injected HTTP 503
MEASUREHTTP probesStatus, response body and duration
HARNESS ACTIONRecovery runbookStop candidate · restore previous package
OBSERVEAlert state machineFive requests · two breached windows
CHECKRecovery verificationHealth, version, traffic and alert clearing
RECORDRetained evidenceRaw samples, incident report and metrics
COMPONENT NOTES
Owned local worker
- Purpose
- Run verified fixture packages as disposable loopback workers. The harness deliberately configures the candidate to return HTTP 503, after confirming its version and failed health response.
- Tradeoff
- The fault is injected behavior, not an application defect. Sequential workers use different local ports and do not measure traffic during startup or shutdown gaps.
HTTP probes
- Purpose
- Request /health on the owned loopback endpoint, bypass inherited proxies and measure through the response-body read. HTTP errors, transport failures and invalid bodies remain unsuccessful observations.
- Tradeoff
- Keeping failed requests in the denominator exposes fast failures. The sample excludes readiness/version checks and worker-transition gaps, so it is not a production SLO.
Recovery runbook
- Purpose
- After observing the alert, stop the owned candidate, reverify the previous package with the local checksum helper, restore it and confirm its expected identity before starting the replacement worker.
- Tradeoff
- This explicit rehearsal action is separate from the monitor. It tests local package recovery; persistent data, production traffic and signature verification are outside this lab.
Alert state machine
- Purpose
- Evaluate a rolling five-request window. Two consecutive windows with at least 60% errors or p95 of at least 500 ms fire one alert; two full zero-error windows below the latency threshold clear it.
- Tradeoff
- Short windows keep the lab quick and repeatable. They are demonstration thresholds, not tuned production objectives; the class emits state transitions without paging or changing releases.
Recovery verification
- Purpose
- Require the restored 1.0.0 package identity, healthy /health and /version responses, twelve successful measured requests and a cleared alert. Check health/version again before declaring recovery verified.
- Tradeoff
- Multiple checks provide more evidence than a successful restart. This remains one local rehearsal and does not establish aggregate MTTR or production recovery targets.
Retained evidence
- Purpose
- Save samples.jsonl, incident.json, incident.md and a final metrics.prom snapshot in a fresh directory. Retain policy, timeline, phase metrics and source/file hashes; stop all owned workers and remove temporary state.
- Tradeoff
- The capture is unsigned: hashes identify bytes without authenticating execution. A saved Prometheus-text snapshot is inspectable evidence, not live monitoring infrastructure.
03 / A REAL VERIFIER FAILURE
The release was signed.
The invocation was wrong.
The version 1.0.0 reference workflow built and signed its package, then stopped at verification. The gate contained the failure: it did not accept an unverified package. This was a portfolio release workflow, not a customer outage.
ROOT CAUSE
The mock accepted an invalid command.
- What failed
- The wrapper passed both
--signer-workflowand--cert-identity. The real GitHub CLI treated them as mutually exclusive and rejected the invocation before signature verification. - Why tests missed it
- Unit tests checked the expected arguments and simulated subprocess outcomes. An assertion actually required the incompatible combination.
- What slowed diagnosis
- A generic error hid the CLI’s explanation, leaving signature, identity, connectivity and compatibility failures difficult to distinguish.
CORRECTIVE DECISIONS
Fix the command. Retain the policy.
- Keep the exact identity
- Remove the redundant workflow selector. Retain the exact certificate identity, approved source and signer commit, repository, issuer, runner and predicate restrictions.
- Make refusal diagnosable
- Include the exit code and bounded CLI diagnostics, with configured tokens redacted and control characters flattened. Errors still prevent state changes.
- Exercise the real dependency
- Add a repeatable check against a genuine signed fixture, with successful controls before and after the negative cases.
The unit tests checked wrapper behavior. The added integration check covers the external CLI’s invocation contract and verifies that a genuine signed package can pass.
04 / SIGNED RELEASE EVIDENCE
A fresh release
passed the corrected gate.
Version 1.0.1
September 10, 2026
The corrected workflow built and signed version 1.0.1 from source commit 556a405, then retained the package, checksum, original attestation bundle and verification observations. These counts describe that release run.
29
Automated tests passed
The same release run also passed real-signature checks, the HTTP recovery rehearsal, a Docker build and CodeQL analysis.
Inspect the release workflow run2
Trusted controls accepted
The genuine signed package passed before and after the negative cases. A verifier that rejects everything cannot satisfy this exercise.
Read the retained release observations7
Negative cases rejected
Artifact, bundle and origin-policy checks refused the constructed invalid cases. Attempted promotions in the first three cases preserved state.
Inspect the integration methodWhat the negative checks exercised
- Altered artifact bytes with a recomputed checksum
- A missing attestation bundle
- An invalid attestation bundle
- An unexpected repository requirement
- An unexpected workflow identity requirement
- An unexpected ref requirement
- An unexpected source-commit requirement
The origin cases change verifier expectations against the genuine bundle; they do not manufacture certificates from other repositories. The retained JSON is transcribed from the successful job log, rather than being an independently verified signature bundle.
05 / RECORDED RECOVERY LAB
Detect the failure.
Verify the recovery.
Recorded local rehearsal
September 11, 2026 · Linux / Python 3.12.14
A candidate service returns injected HTTP 503 errors. A monitor detects sustained failure, and the rehearsal follows a documented runbook to restore the previous package. Every measured request is retained.
- 0.060 s
Establish the baseline
Version 1.0.0 serves twelve healthy measured requests.
- 1.238 s
Observe the failure
The candidate’s injected health failure produces four HTTP 503 responses.
- 1.547 s
Raise one alert
Two breached windows trigger the response. Four of the latest five requests have failed.
- 2.791 s
Verify recovery
The previous package is reverified and restored. Twelve measured requests pass, the alert clears, and health and version are checked again.
What the requests showed
| Phase | Requests | Errors | p95 latency |
|---|---|---|---|
| Baseline | 12 | 0 | 1.273 ms |
| Incident | 4 | 4 | 1.859 ms |
| Recovery | 12 | 0 | 6.728 ms |
The failed requests returned quickly. Their HTTP status still exposed the fault: latency alone would not have described this incident.
DETECTION & RESPONSE
Make the alert actionable.
The rolling window contains five requests. Two consecutive windows with at least 60% errors or p95 latency of at least 500 ms trigger an alert. Clearing requires two full healthy windows below the latency threshold.
- Observed fault to alert
- 0.309 seconds
- Alert to verified recovery
- 1.245 seconds
Inspect the recovery runbook
- Inspect the failed health response and candidate version.
- Stop the owned candidate process.
- Reverify and restore the retained previous package.
- Check health, version and twelve measured requests; require the alert to clear.
- Stop all owned workers and remove temporary packages and state.
Captured in the build workspace using synthetic traffic and sequential loopback workers. Readiness checks and worker-transition gaps are excluded from the traffic sample. These are timings for one rehearsal; production SLOs, aggregate MTTR, live traffic switching and cloud recovery remain unmeasured. The unsigned capture uses checksum-only local package verification. Inspect the original metadata and file hashes.
06 / EVIDENCE & LIMITS
Make the scope
as clear as the result.
Release verification
- The evidence covers a genuine signed reference release, explicit origin expectations and refusal of seven constructed cases.
- Promotion changes local, single-operator state. Cloud deployment, production traffic switching and concurrent state safety are outside this reference.
- CodeQL completed in the release run. This does not establish an absence of vulnerabilities or a severity-based blocking policy.
Recovery rehearsal
- The injected fault is known: a health-response flag produces HTTP 503 while both fixture versions retain the same application source.
- The monitor reports alert state; the harness explicitly runs the documented recovery procedure. There is no general remediation agent or external paging.
- The saved capture is unsigned. Its hashes identify bytes, and its single-run timings do not establish production SLOs or aggregate recovery performance.
Architecture links pin the inspected implementation at 3cc9d33. The release and local capture retain their own source identities and dates.
07 / LET’S TALK
Build delivery
your team can understand.
I’m looking for Senior DevOps and Platform Engineering opportunities where I can improve cloud delivery, build useful automation and help teams do their best work.
bailey.n.fitchett@gmail.com