ENGINEERING CASE STUDY / AZURE PLATFORM FOUNDATION
AI review with engineering guardrails.
A local language model selects Terraform review questions. Python checks the response against the plan evidence and keeps control of the review decision.
Independent local demonstration using synthetic plans. The output supports human review; it does not deploy infrastructure.
01 / THE PROBLEM & MY APPROACH
A plausible answer still needs to meet the contract.
Make the review inspectable.
A model can produce valid-looking JSON while omitting a resource or asking the wrong question. Even replacement order matters: creation before deletion calls for cutover checks; deletion before creation calls for an interruption window.
I built a bounded question selector around a deterministic Terraform action reviewer. The local implementation made it possible to exercise the model contract without an Azure subscription.
Keep the decision in code.
Send generated evidence IDs and action metadata to the model. Keep resource addresses, attribute values and arbitrary plan strings outside the request.
Give every resource its own allowed and required question IDs.
Reject missing resources, changed facts, extra fields and invalid questions before accepting a report.
Carry the deterministic policy result into the JSON and Markdown output unchanged.
02 / ARCHITECTURE & DECISIONS
Separate model choices from policy authority.
Choose a component to inspect its purpose, tradeoff and implementation.
LOCAL OLLAMA REVIEW
Two paths. Python keeps the decision.
The plan supplies deterministic policy evidence and a bounded model request. Only a validated response reaches the advisory report.
The plan feeds policy → report and bounded evidence → local model → response validation → report. The model path cannot change the policy decision; invalid responses stop the review with exit 2.
COMPONENT NOTES
Saved plan JSON
Purpose
Read saved-plan JSON locally, validate the consumed fields and preserve a hash of the input bytes. The AI review accepts at most 20 resource objects without truncation.
Tradeoff
The contract rejects unsupported actions and malformed plans. This is action-level evidence, so a passing check does not establish that a change is safe or authenticate the input.
Send generated evidence IDs, mode, actions, category and unknown-value status, plus the question catalogue and per-resource constraints. Addresses, before/after values, variables and arbitrary plan strings stay out of the request.
Tradeoff
Limiting the outbound evidence makes the contract inspectable. It deliberately prevents the model from performing attribute-level or dependency analysis.
Classify known action combinations and flag deletion, replacement, incomplete plans and deferred changes for human review. Carry that result unchanged into the final report.
Tradeoff
An explicit rule is repeatable and testable. The no_destructive_changes result only describes this policy; it is not deployment approval or a full risk assessment.
Call the installed Qwen model through the loopback Ollama API using a schema keyed by every evidence ID. Each resource gets its own allowed and required question IDs.
Tradeoff
A small local model can still return omissions or irrelevant questions. Constrained generation narrows those failures but does not replace application validation.
Join accepted findings to the local evidence map, render catalogue text and save JSON and Markdown in a fresh directory. Exit 1 means human review is required; exit 0 means this policy found no destructive changes.
Tradeoff
The saved report preserves an inspectable result, not authorization to apply Terraform. Its hashes identify bytes without proving who produced them.
Require every evidence ID exactly once, unchanged category and unknown flags, one to four distinct allowed questions, and every required question. Reject extra fields and missing coverage before rendering.
Tradeoff
Failing closed makes a bad answer visible and can require another attempt. The validator does not silently fill omissions, repair facts or establish broader model quality.
R001 included an interruption question for create-before-delete. R004 chose cutover and omitted interruption for delete-before-create. Local validation rejected the response.
What I changed
Request revision v2 supplied explicit allowed and mandatory question IDs for each resource.
What stayed enforced
Irrelevant questions and missing mandatory questions still caused exit 2. The application did not repair the model’s answer.
OBSERVED FAILURE / 02
Missing resource evidence.
What happened
A subsequent attempt failed with Response omits resource evidence. The omitted IDs were not captured, so the record does not identify which resources were missing.
What I changed
Revision v3 made the native findings an object with a required property for every evidence ID, with resource-specific facts and allowed question IDs.
What stayed enforced
Local code checked exact coverage and identity before converting the response to the public report format. Missing entries remained an error.
The schema makes the contract easier for the model to follow; local validation remains authoritative. These submitted failures explain the iteration history, rather than establish a controlled comparison of model accuracy.
The engineering notes retain the observed behavior, linked fixes and evidence limits.
One plan. Four resources. A review you can inspect.
Captured September 11, 2026 Local Qwen 2.5 3B · Synthetic plan
Replacement order changes the review question. Step through the saved findings, then inspect the decision. The guided walkthrough uses an edited presentation timeline; the original recording is available separately.
Walk through the saved review
GUIDED
STEP 1 OF 6
One plan. Four resources.
An edited walkthrough of the saved synthetic plan and review. Each resource has its own action order and review questions.
WALKTHROUGH: one local Ollama attempt; synthetic Terraform evidence.
The model selects questions; Python owns the review decision.
R001: create -> delete
R002: no-op
R003: delete
R004: delete -> create
The saved findings and original transcript are available below.
The 30-second guided walkthrough is an edited reading sequence. Original recording preserves all 60.375 seconds, including the model’s wait and buffered CLI output. The measured review time remains 60.203 seconds. Neither view autoplays.
The response covered all four resources. Exit 1 is the expected review decision for this destructive plan. The report is advisory and does not authorize an apply.
This is one saved run, separate from the 12-case evaluation. The unsigned capture and matching file hashes establish internal consistency; they do not independently attest execution. Azure deployment, broad model quality and reviewer time savings are outside the measured scope. Read the capture notes and reproduction details.
05 / VERIFIED RESULTS
Three checks. Three distinct scopes.
12 / 12
Synthetic evaluation cases passed
One local Qwen 2.5 3B run met the corpus contract. Nine cases contained resource evidence; three legitimately returned empty findings, with separate policy warnings.
Separately constructed, deliberately corrupted responses exercised the validator. These were application checks, not thirteen attacks generated or resisted by Qwen.
A later saved attempt included every resource, including the unchanged one. Python accepted the response contract and returned review_required with exit 1.
The recording’s 60.203-second CLI time and 1,688 tokens belong to that four-resource attempt. Its totals are not combined with the earlier evaluation, and neither run measures human review time saved.
06 / EVIDENCE & LIMITS
Keep the claims inside the evidence.
What the evidence establishes
Contract compliance on a small, synthetic development corpus under the recorded settings.
Rejection of the constructed invalid responses and two observed local model failures.
A retained four-resource review with matching artifact hashes and an inspectable transcript.
What remains unmeasured
General model accuracy, unseen-plan reliability and benefit compared with a deterministic question baseline.
Reviewer time savings, cost savings or production readiness.
Attribute-level risk, live dependencies, outage prediction and Azure deployment. Unsigned captures and hashes do not independently attest execution.
Implementation links show revision 1c24b17. The capture notes identify the separate source revision used for the recorded run.
I’m looking for Senior DevOps and Platform Engineering opportunities where I can improve cloud delivery, build useful automation and help teams do their best work.