UAI Deployment and CD: High-Level Design

Scope. Staging and prod for the UAI core services: what runs where, how an image becomes a running deployment, and what has to be green before prod. Orchestrator and OpenWebUI are staging-only for now. This is the what; the exact compose file, deploy script, workflows, and alert rules are the LLD (section 16). Per-service internals live in the service LLDs; the system shape lives in the UAI High-Level Design.

Status. Draft v0.2 — 7 September 2026. Owner: Aditya.

Index

1. Goal
2. What we are deploying
3. Shape of each environment
4. Images
5. Sandbox e2e in CI
6. How a deploy reaches a VM
7. The deploy script
8. Release flow
9. Migrations and rollback
10. Secrets
11. TLS
12. Backups
13. Observability
14. Maintenance
15. Not in scope now
16. Next step
17. Open items to confirm

1. Goal

Make deploying, updating, and rolling back UAI easy and mostly automatic. Nothing reaches prod unless it has passed a full test run on staging.

2. What we are deploying

3. Shape of each environment

One VM per environment. Everything runs with docker compose on that VM.

Staging and prod use the same compose.yml. Only the .env file differs. This keeps the two environments identical except for secrets, hostnames, and sizes.

VM (staging or prod)
  Caddy (TLS, routing)
    Registry   ---> Postgres (registry)
    Trust      ---> Postgres (trust)      Redis
    Ledger     ---> Postgres (ledger)
    Orchestrator (staging only) ---> OpenWebUI (staging only)
  Prometheus, Grafana
  Backup cron
  GitHub Actions self-hosted runner

Admin ports are never exposed by Caddy. Only the edge routes named in the service LLDs are public.

4. Images

One image per service, built from the monorepo. Built once on every merge to main, tagged with the git SHA, and pushed to GitHub Container Registry.

The same image goes to staging and then to prod. We never rebuild for prod.

5. Sandbox e2e in CI

Every PR and every merge to main runs the full e2e suite in a temporary sandbox before anything touches staging.

The sandbox is the same compose.yml started on the GitHub-hosted runner with a ci env file. Postgres, Redis, and all services run inside the job. The suite runs against it, then the job ends and the sandbox is gone. No extra VM, nothing to clean up.

Staging still runs the same suite after deploy. The sandbox catches broken code early; staging catches problems that only show up on the real VM, network, and TLS.

6. How a deploy reaches a VM

Each VM runs a GitHub Actions self-hosted runner. The runner only makes outbound HTTPS calls to GitHub, so no inbound SSH is needed. Deploy jobs run on the VM through this runner.

Runners are labelled uai-staging and uai-prod. A job for prod can only run on the prod runner.

7. The deploy script

One script, deploy.sh <sha>, used for staging, prod, and rollback. In order:

1. Pull images for the given SHA.
2. Take a pg_dump of each database. Keep it next to the nightly backups, named with the SHA.
3. Run migrations for each service as a one-shot container. Stop if any fail.
4. On staging only, run the seed step. This registers the test agents and test seeker used by the e2e suite.
5. Start or update the services with compose up -d.
6. Wait for /readyz on every service, up to a timeout.
7. If any check fails, redeploy the previous SHA and exit non-zero.
8. Record the new SHA as current.

Rollback is deploy.sh <older sha>. Nothing special.

8. Release flow

flowchart TD
    A[Merge to main] --> B[CI: lint, unit tests, build images tagged by SHA]
    B --> C[deploy.sh on staging runner]
    C --> D[Staging verification: integration, e2e, smoke, load check]
    D -->|green| E[Mark SHA prod ready]
    D -->|red| F[Stop. Fix and merge again]
    E --> G[git tag v1.x.y on a prod ready SHA]
    G --> H[Manual approve in GitHub]
    H --> I[deploy.sh on prod runner]
    I --> J[Prod smoke tests]
    J -->|fail| K[Auto rollback to previous SHA]

Rules

• Only SHAs marked prod ready can be tagged for prod. The tag job checks this before asking for approval.
• The staging verification suite runs from GitHub against the staging URLs. It does not run on the VM.
• The same suite runs with one command and also works against dev, so people can run it locally.
• No direct hotfix to prod. Every prod change goes through staging first.

9. Migrations and rollback

goose, plain SQL, one change per file, as in the Registry and Audit Ledger LLDs. Forward migrations run in the deploy script before the app starts. A failed migration blocks the deploy. Fail closed.

Every migration must work with both the new and the previous app version. This is the expand-and-contract rule:

• Allowed in a release: add table, add nullable column, add index, add new enum value, backfill data.
• Not allowed in the same release as the code that stops using them: drop column, drop table, rename, make a column not null. These go in a later release, after the old code is gone from prod.

Because of this rule, rollback is only deploy.sh <previous sha>. The old app runs fine on the new schema, so no down migration is needed.

If a migration itself is broken and cannot be fixed forward, the pre-migration pg_dump from step 2 is restored by hand, then the previous SHA is deployed. This is the only manual path, and it should be rare.

Down migrations are still written and tested in CI so they exist if ever needed.

10. Secrets

Secrets live in GitHub Environments, one environment per VM: staging and prod. GitHub Environment secrets are encrypted at rest, masked in logs, and prod secrets can require approval before a job reads them.

At deploy time the runner reads the secrets for its environment and writes the .env file on the VM with mode 600. The file is never committed. Rotating a secret is one change in GitHub followed by a redeploy.

Service signing keys enter this store from the identity ceremony, not from the pipeline.

11. TLS

Caddy terminates TLS 1.3 only and gets certificates from Let’s Encrypt on its own. No manual cert work.

12. Backups

A cron container runs pg_dump nightly for all three databases. Keep 30 days on the VM. Copy each dump to a second location inside India. Restore is tested once per milestone on staging.

13. Observability

Prometheus scrapes every service on the internal listener. Grafana shows one dashboard per service using the metrics named in the LLDs. Container logs stay in Docker’s JSON log files with rotation. No Loki for Phase 1.

Alerts go to Slack, on:

Prometheus and Grafana run on the same VM as the services, so if the VM dies they die with it. To still know when that happens, a scheduled GitHub Actions workflow runs every 5 minutes on GitHub’s own runners. It calls /readyz on each prod service and posts to Slack if any call fails or times out. It sits fully outside our VMs and needs nothing installed.

Prometheus data is included in the nightly off-VM backup, so history is not lost with the VM.

When the 5-minute check is too slow, or metrics and logs need to be kept off the VM, the next step is Grafana Cloud: Prometheus remote-writes to it, and its synthetic probe replaces the GitHub ping. Logs come after that if needed. Nothing in the Phase 1 setup has to be redone for either step.

14. Maintenance

Dependabot opens PRs for Go modules and base images. Updates flow through the same staging gate. There is no separate patching process.

15. Not in scope now

• Prod Orchestrator and OpenWebUI. Waits for BharatGrid clarity.
• Kubernetes, load balancers, multi-VM. Revisit when a real HA or DR need is triggered.
• AI in the pipeline.
• Blue-green or canary deploys. Rollback by SHA is enough for Phase 1.

16. Next step

Write the LLD: exact compose file, deploy script, GitHub workflows, test suite layout, seed data, alert rules.