How a build works, end to end

Compose

An environment is a composition: blocks, wires, groups

Where data repo compositions/marshal-app/composition.yaml capabilities in capabilities/*.yaml modules in modules/terraform/aws/*

A block names a capability (network, sql-database, container-platform) and, where more than one module implements it, the module. Inputs are plain values; {env} is the one substitution.
A wire is <block>.<output>, optionally .<key> for one entry of a map. It can point at an earlier block in the same group or at a published output of an earlier group. It cannot index a list, which is why the bastion takes a subnet list.
A group (stack) is one Terraform state. Marshal's own environment has three, in order: aws-foundation, data, marshal-app, plus the on-demand eks group added that day.
Anything company-specific (an image tag, a model API key, a Cloudflare zone) has no default: a person fills it in once as a stack input, sensitive ones never leave SSM.

Groups apply in order. Each publishes its outputs as SSM parameters, and the next group reads them by path, so no group needs another's state file.

Render

Marshal writes the Terraform a person would have written

Where backend/app/services/provisioning/render.py workdir under the runner state in the environment's own S3 bucket

Each block becomes a module call pinned to a module commit; each capability input is resolved against the module's own variables first (an identical name needs no mapping), then through the capability's mapping table.
Each wire to an earlier group becomes an aws_ssm_parameter data source; each output the group publishes becomes an SSM write plus a local_file so outputs.json exists even before terraform output runs.
The backend block points at the environment's state bucket and key (<env>/<group>/terraform.tfstate): the same key the hand-written stack in infrastructure/environments/production/<group> uses, so either side can plan against the other's applies.
A group that needs the Cloudflare provider is refused before Terraform if the environment has no Cloudflare connection, with a sentence saying where to connect one.

Same resources, two authors

The property this whole design keeps: the composition and the hand-written stacks produce the same resource names at the same state keys. A test (test_marshal_composition_diff_zero) renders the composition and diffs it against the folder.

infrastructure/ environments/production/ aws-foundation/  data/  marshal-app/  eks/ modules/            # vendored from the data repo Makefile            # make plan STACK=data

Plan

A run is planned, scanned, priced and given a verdict

Where runs.py::run_plan_phase runner: local (dev), hosted (a Fargate task per run), customer (their CI)

The runner executes, in order: init, validate, plan -out=plan.bin, show -json plan.bin, and since 6 September 2026 output -json, so a read-only refresh can restore an environment's published outputs if the apply's artifact ever goes missing.
plan.json is parsed into counts (create / update / replace / destroy) and a risk tier; an unreadable plan is never "zero changes".
Review rules and IaC scanners run over the plan: blocking findings must be fixed or explicitly acknowledged with a reason. A cost delta is estimated from provider prices; the model writes an advisory verdict that cannot block or approve anything.
Artifacts (log, plan, outputs) are encrypted at rest under ~/.marshal/run-artifacts locally or the artifact bucket in production. Before that they lived under the OS temp dir, which is how they once vanished.

The runner's plan steps and the artifact each leaves. The last step is the change that lets a refresh heal a lost outputs file.

Seen on 2026-09-06

Three refreshes started in the same second installed the AWS provider into three work dirs at once; one lost the race and its plan failed with "Failed to load plugin schemas". Run alone, it passed. The run reads as PlanFailed, never as "no changes".

Approve

A person approves; the tier decides who, the plan decides what they type

Where approvals.py Slack notice on awaiting_approval expiry by risk tier

Production needs the admin role; the org policy can require two approvers. A destroy or a replace of a stateful resource needs the exact phrase typed.
Blocking review findings stop approval unless each is acknowledged with a written reason; the run page then shows an "Accepted risks" card for as long as the run exists.
A plan expires (hours for low risk, less for high) so nobody applies a stale picture of the account. Cancel races the plan and wins.
The model's tools can propose a plan run or a stack-input change, attributed to the agent. They can never approve, apply, destroy or import: the approval route checks the caller's membership, not a prompt.

A deploy parked here, 2026-09-06

Run d4fdd189 (image upload-57869eb0cbf3) is awaiting_approval. A first proposal was auto-failed after 20 minutes by a reaper that treated "waiting on a person" as "wedged process"; that is fixed, and the re-proposal waits until someone approves it.

Apply

Apply the saved plan, then read the account back and disagree loudly

Where runs.py::run_apply_phase restore points before anything stateful outputs → SSM + artifact

The runner applies plan.bin and nothing else, so what was reviewed is what runs. A restore point (snapshot) is taken first where the plan touches something with data in it.
After the apply, a plan -refresh-only -detailed-exitcode asks whether the account already differs from what was just applied. If it does, the run is VerifyDrift: failed, but its outputs are kept because the change is live.
terraform output -json is stored as the run's outputs_ref; the built explorer, the deploy targets and the smoke check all read from it. A failed apply queues a refresh so the next plan starts from truth.
A group's published values land at /{env}/marshal/<group>/<block>/<output> as JSON strings; the app's entrypoint strips one layer of quotes when it reads one.

Every apply ends in one of three states, and only the first is called done. Drift after an apply is a failed run with real outputs.

Build the image

Your source, zipped, built in your own account, tagged in your registry

Where delivery/image_builds.py CodeBuild project from the image_build block ECR marshal-app

Two sources: a connected GitHub repo at a ref, or a zip uploaded through the product (git archive --format=zip HEAD, 6 MB for this repo, 200 MB cap). The zip is streamed to disk and hashed, never held in memory; the Dockerfile path is checked against traversal.
The zip goes to builds/<run>.zip in the environment's own state bucket (customer-owned, KMS, expiring), and CodeBuild starts with sourceLocationOverride, IMAGE_TAG=upload-<sha12>, DOCKERFILE_PATH.
The project runs on ARM (amazonlinux-aarch64-standard:3.0, privileged for buildx), logs in to ECR, builds linux/arm64, pushes. One build at a time per environment; a duplicate tag is refused before anything starts.
Marshal polls the build, records each phase, then confirms the tag exists in the registry (describe_images). A finished build whose image is missing is a failure, not a success.

Nothing on a laptop builds or pushes. Marshal hands CodeBuild a zip from a bucket the customer owns and reads the tag back from the registry.

What the image is

One image, one container, two processes: alembic upgrade head under a Postgres advisory lock (so two tasks starting together do not race the migrations), uvicorn on 127.0.0.1:6001, then the Next.js standalone server on 0.0.0.0:3000 proxying /backend. /up answers 200 only when the API's /health does. Since 2026-09-06 it also carries terraform 1.15.7 for the local runner.

Deploy the app

One button proposes a deploy per service; a person approves; ECS rolls it

Where delivery/deploy_runs.py targets from the app group's outputs POST .../deploy-app, POST /runs/{id}/approve

Targets are derived, not typed: cluster name, service names and region come from the environment's published outputs (create_targets_from_outputs). On Marshal's own environment: marshal-cluster-production / marshal-app-production, container app.
Propose picks the tag (given, or the newest in the registry), resolves the digest, records the family's newest task definition as the base (not the running one, so an infrastructure change to the task definition is never silently dropped), and checks the service's error budget from its own monitor. One live run per target.
Approve re-checks role against the environment tier, the typed phrase when there is no undo, and the budget acknowledgement; then a background thread registers a new revision with the new image, updates the service, and waits for it to be stable.
Verify runs a smoke check against the published address (the load balancer with a host header, or the tunnel's public_url after the cut-over) through the SSRF guard, records real status and latency, proposes a health monitor, and keeps the previous revision as the rollback.

Deploy states. Waiting on a person is ended by expiry; a running step is ended by a missed heartbeat. Conflating the two was a bug fixed on 2026-09-06.

Failures this pipeline has already caught and named

An arm64 image on an amd64 task platform (fixed with cpu_architecture: ARM64 everywhere the task is described).
A start-up crash on alembic upgrade head against real Postgres, read as kind crash with the log's error lines rather than "could not tell".
A zip sealed with a bucket key the build role could not decrypt; a log group published as an ARN; a tag lookup in the wrong region; a deploy copying the running revision instead of the newest.
A Spot interruption at 19:13 took the single task down for about two minutes until ECS replaced it. With base: 0 that is the price of all-Spot.

Verify and keep verifying

What is live is read back, never assumed

Where drift sweep (refresh runs) uptime monitor on /up alarms from the monitoring block Cost Explorer

The drift job starts read-only refresh runs; a refresh that finds a hand-edited value is drift, shown per resource, and since 6 September 2026 it also re-records the group's outputs.
The health monitor probes the public address on its own schedule and feeds the error budget the next deploy is checked against.
Missing or stale data is unknown: a smoke check that could not be made, a monitor with no result yet, a cost read with no billing rows. Nothing green stands on nothing.
The $50 number is verified from the account's own Cost Explorer after two full days, Mumbai usage types only, and recorded in plan 45.

Item (Mumbai list price, 730 h)	Before	After cut-over
NAT gateway + its public IPv4	$44.53	removed
Application load balancer + 2 public IPv4	$24.75	removed (Cloudflare Tunnel)
RDS db.t4g.micro + 20 GB gp3	$17.95	$17.95
Valkey cache.t4g.micro	$11.68	$11.68
Fargate 0.5 vCPU / 1 GB, on-demand ceiling (Spot in practice)	$10.60	≤ $10.60
App task public IPv4	–	$3.65
Bastion t4g.nano + 8 GB gp3	–	$2.77
KMS keys × 3, ECR, logs, S3, SSM, CodeBuild minutes	≈ $4	≈ $4
Total at one running copy	≈ $113	≈ $50.7 ceiling · ≈ $43 on Spot

The shape after plan 45 §2: nothing inside the network listens to the internet. The connector dials Cloudflare and requests arrive back down that link; the database and cache never leave the private subnets.

Two ways to run it

Marshal or the folder, same names, same state

Through app.marshalcloud.com

Build → environment → group → Plan. Read the counts, the findings, the cost delta.
Approve (admin on production; type the phrase if there is no undo; acknowledge blockers with a reason).
Marshal applies on a runner, verifies, publishes outputs.
Build the image (zip or repo) → Deploy the app → approve → smoke check.

Production executes runs on a hosted Fargate task per run (plan 45 §5); a developer's own machine uses the local runner.

From `infrastructure/` with Terraform

cd infrastructure make check                 # guardrails, fmt, validate make plan  STACK=aws-foundation make apply STACK=aws-foundation   # human only make plan  STACK=data make plan  STACK=marshal-app make plan  STACK=eks       # $0.10/h while it stands deploy/scripts/deploy.sh   # new image revision

A plan from the folder after a Marshal apply shows no changes, and the reverse, because both write the same state keys. The tunnel cut-over order is written in the README: foundation first, app group straight after.

Compose

An environment is a composition: blocks, wires, groups

Where data repo compositions/marshal-app/composition.yaml capabilities in capabilities/*.yaml modules in modules/terraform/aws/*

A block names a capability (network, sql-database, container-platform) and, where more than one module implements it, the module. Inputs are plain values; {env} is the one substitution.
A wire is <block>.<output>, optionally .<key> for one entry of a map. It can point at an earlier block in the same group or at a published output of an earlier group. It cannot index a list, which is why the bastion takes a subnet list.
A group (stack) is one Terraform state. Marshal's own environment has three, in order: aws-foundation, data, marshal-app, plus the on-demand eks group added that day.
Anything company-specific (an image tag, a model API key, a Cloudflare zone) has no default: a person fills it in once as a stack input, sensitive ones never leave SSM.

Groups apply in order. Each publishes its outputs as SSM parameters, and the next group reads them by path, so no group needs another's state file.

Render

Marshal writes the Terraform a person would have written

Where backend/app/services/provisioning/render.py workdir under the runner state in the environment's own S3 bucket

Each block becomes a module call pinned to a module commit; each capability input is resolved against the module's own variables first (an identical name needs no mapping), then through the capability's mapping table.
Each wire to an earlier group becomes an aws_ssm_parameter data source; each output the group publishes becomes an SSM write plus a local_file so outputs.json exists even before terraform output runs.
The backend block points at the environment's state bucket and key (<env>/<group>/terraform.tfstate): the same key the hand-written stack in infrastructure/environments/production/<group> uses, so either side can plan against the other's applies.
A group that needs the Cloudflare provider is refused before Terraform if the environment has no Cloudflare connection, with a sentence saying where to connect one.

Same resources, two authors

infrastructure/ environments/production/ aws-foundation/  data/  marshal-app/  eks/ modules/            # vendored from the data repo Makefile            # make plan STACK=data

Plan

A run is planned, scanned, priced and given a verdict

Where runs.py::run_plan_phase runner: local (dev), hosted (a Fargate task per run), customer (their CI)

The runner executes, in order: init, validate, plan -out=plan.bin, show -json plan.bin, and since 6 September 2026 output -json, so a read-only refresh can restore an environment's published outputs if the apply's artifact ever goes missing.
plan.json is parsed into counts (create / update / replace / destroy) and a risk tier; an unreadable plan is never "zero changes".
Review rules and IaC scanners run over the plan: blocking findings must be fixed or explicitly acknowledged with a reason. A cost delta is estimated from provider prices; the model writes an advisory verdict that cannot block or approve anything.
Artifacts (log, plan, outputs) are encrypted at rest under ~/.marshal/run-artifacts locally or the artifact bucket in production. Before that they lived under the OS temp dir, which is how they once vanished.

The runner's plan steps and the artifact each leaves. The last step is the change that lets a refresh heal a lost outputs file.

Seen on 2026-09-06

Approve

A person approves; the tier decides who, the plan decides what they type

Where approvals.py Slack notice on awaiting_approval expiry by risk tier

Production needs the admin role; the org policy can require two approvers. A destroy or a replace of a stateful resource needs the exact phrase typed.
Blocking review findings stop approval unless each is acknowledged with a written reason; the run page then shows an "Accepted risks" card for as long as the run exists.
A plan expires (hours for low risk, less for high) so nobody applies a stale picture of the account. Cancel races the plan and wins.
The model's tools can propose a plan run or a stack-input change, attributed to the agent. They can never approve, apply, destroy or import: the approval route checks the caller's membership, not a prompt.

A deploy parked here, 2026-09-06

Apply

Apply the saved plan, then read the account back and disagree loudly

Where runs.py::run_apply_phase restore points before anything stateful outputs → SSM + artifact

The runner applies plan.bin and nothing else, so what was reviewed is what runs. A restore point (snapshot) is taken first where the plan touches something with data in it.
After the apply, a plan -refresh-only -detailed-exitcode asks whether the account already differs from what was just applied. If it does, the run is VerifyDrift: failed, but its outputs are kept because the change is live.
terraform output -json is stored as the run's outputs_ref; the built explorer, the deploy targets and the smoke check all read from it. A failed apply queues a refresh so the next plan starts from truth.
A group's published values land at /{env}/marshal/<group>/<block>/<output> as JSON strings; the app's entrypoint strips one layer of quotes when it reads one.

Every apply ends in one of three states, and only the first is called done. Drift after an apply is a failed run with real outputs.

Build the image

Your source, zipped, built in your own account, tagged in your registry

Where delivery/image_builds.py CodeBuild project from the image_build block ECR marshal-app

Two sources: a connected GitHub repo at a ref, or a zip uploaded through the product (git archive --format=zip HEAD, 6 MB for this repo, 200 MB cap). The zip is streamed to disk and hashed, never held in memory; the Dockerfile path is checked against traversal.
The zip goes to builds/<run>.zip in the environment's own state bucket (customer-owned, KMS, expiring), and CodeBuild starts with sourceLocationOverride, IMAGE_TAG=upload-<sha12>, DOCKERFILE_PATH.
The project runs on ARM (amazonlinux-aarch64-standard:3.0, privileged for buildx), logs in to ECR, builds linux/arm64, pushes. One build at a time per environment; a duplicate tag is refused before anything starts.
Marshal polls the build, records each phase, then confirms the tag exists in the registry (describe_images). A finished build whose image is missing is a failure, not a success.

Nothing on a laptop builds or pushes. Marshal hands CodeBuild a zip from a bucket the customer owns and reads the tag back from the registry.

What the image is

Deploy the app

One button proposes a deploy per service; a person approves; ECS rolls it

Where delivery/deploy_runs.py targets from the app group's outputs POST .../deploy-app, POST /runs/{id}/approve

Targets are derived, not typed: cluster name, service names and region come from the environment's published outputs (create_targets_from_outputs). On Marshal's own environment: marshal-cluster-production / marshal-app-production, container app.
Propose picks the tag (given, or the newest in the registry), resolves the digest, records the family's newest task definition as the base (not the running one, so an infrastructure change to the task definition is never silently dropped), and checks the service's error budget from its own monitor. One live run per target.
Approve re-checks role against the environment tier, the typed phrase when there is no undo, and the budget acknowledgement; then a background thread registers a new revision with the new image, updates the service, and waits for it to be stable.
Verify runs a smoke check against the published address (the load balancer with a host header, or the tunnel's public_url after the cut-over) through the SSRF guard, records real status and latency, proposes a health monitor, and keeps the previous revision as the rollback.

Deploy states. Waiting on a person is ended by expiry; a running step is ended by a missed heartbeat. Conflating the two was a bug fixed on 2026-09-06.

Failures this pipeline has already caught and named

An arm64 image on an amd64 task platform (fixed with cpu_architecture: ARM64 everywhere the task is described).
A start-up crash on alembic upgrade head against real Postgres, read as kind crash with the log's error lines rather than "could not tell".
A zip sealed with a bucket key the build role could not decrypt; a log group published as an ARN; a tag lookup in the wrong region; a deploy copying the running revision instead of the newest.
A Spot interruption at 19:13 took the single task down for about two minutes until ECS replaced it. With base: 0 that is the price of all-Spot.

Verify and keep verifying

What is live is read back, never assumed

Where drift sweep (refresh runs) uptime monitor on /up alarms from the monitoring block Cost Explorer

The drift job starts read-only refresh runs; a refresh that finds a hand-edited value is drift, shown per resource, and since 6 September 2026 it also re-records the group's outputs.
The health monitor probes the public address on its own schedule and feeds the error budget the next deploy is checked against.
Missing or stale data is unknown: a smoke check that could not be made, a monitor with no result yet, a cost read with no billing rows. Nothing green stands on nothing.
The $50 number is verified from the account's own Cost Explorer after two full days, Mumbai usage types only, and recorded in plan 45.

Item (Mumbai list price, 730 h)	Before	After cut-over
NAT gateway + its public IPv4	$44.53	removed
Application load balancer + 2 public IPv4	$24.75	removed (Cloudflare Tunnel)
RDS db.t4g.micro + 20 GB gp3	$17.95	$17.95
Valkey cache.t4g.micro	$11.68	$11.68
Fargate 0.5 vCPU / 1 GB, on-demand ceiling (Spot in practice)	$10.60	≤ $10.60
App task public IPv4	–	$3.65
Bastion t4g.nano + 8 GB gp3	–	$2.77
KMS keys × 3, ECR, logs, S3, SSM, CodeBuild minutes	≈ $4	≈ $4
Total at one running copy	≈ $113	≈ $50.7 ceiling · ≈ $43 on Spot

Two ways to run it

Marshal or the folder, same names, same state

Through app.marshalcloud.com

Build → environment → group → Plan. Read the counts, the findings, the cost delta.
Approve (admin on production; type the phrase if there is no undo; acknowledge blockers with a reason).
Marshal applies on a runner, verifies, publishes outputs.
Build the image (zip or repo) → Deploy the app → approve → smoke check.

Production executes runs on a hosted Fargate task per run (plan 45 §5); a developer's own machine uses the local runner.

From `infrastructure/` with Terraform

cd infrastructure make check                 # guardrails, fmt, validate make plan  STACK=aws-foundation make apply STACK=aws-foundation   # human only make plan  STACK=data make plan  STACK=marshal-app make plan  STACK=eks       # $0.10/h while it stands deploy/scripts/deploy.sh   # new image revision

From a YAML composition to a task answering on app.marshalcloud.com

An environment is a composition: blocks, wires, groups

Marshal writes the Terraform a person would have written

Same resources, two authors

A run is planned, scanned, priced and given a verdict

A person approves; the tier decides who, the plan decides what they type

Apply the saved plan, then read the account back and disagree loudly

Your source, zipped, built in your own account, tagged in your registry

One button proposes a deploy per service; a person approves; ECS rolls it

What is live is read back, never assumed

Marshal or the folder, same names, same state

Through app.marshalcloud.com

From `infrastructure/` with Terraform

From a YAML composition to a task answering on app.marshalcloud.com

An environment is a composition: blocks, wires, groups

Marshal writes the Terraform a person would have written

Same resources, two authors

A run is planned, scanned, priced and given a verdict

A person approves; the tier decides who, the plan decides what they type

Apply the saved plan, then read the account back and disagree loudly

Your source, zipped, built in your own account, tagged in your registry

One button proposes a deploy per service; a person approves; ECS rolls it

What is live is read back, never assumed

Marshal or the folder, same names, same state

Through app.marshalcloud.com

From `infrastructure/` with Terraform

An environment is a composition: blocks, wires, groups

Marshal writes the Terraform a person would have written

Same resources, two authors

A run is planned, scanned, priced and given a verdict

A person approves; the tier decides who, the plan decides what they type

Apply the saved plan, then read the account back and disagree loudly

Your source, zipped, built in your own account, tagged in your registry

One button proposes a deploy per service; a person approves; ECS rolls it

What is live is read back, never assumed

Marshal or the folder, same names, same state

Through app.marshalcloud.com

From infrastructure/ with Terraform

An environment is a composition: blocks, wires, groups

Marshal writes the Terraform a person would have written

Same resources, two authors

A run is planned, scanned, priced and given a verdict

A person approves; the tier decides who, the plan decides what they type

Apply the saved plan, then read the account back and disagree loudly

Your source, zipped, built in your own account, tagged in your registry

One button proposes a deploy per service; a person approves; ECS rolls it

What is live is read back, never assumed

Marshal or the folder, same names, same state

Through app.marshalcloud.com

From infrastructure/ with Terraform

From `infrastructure/` with Terraform

From `infrastructure/` with Terraform