UAI Orchestrator: Low-Level Design (LLD)

Scope. This is the how of the Orchestrator. The what — components, flows, stages, and where we deliberately stop — lives in the parent UAI x BharatGrid Integration HLD; the platform-wide UAI High-Level Design remains the general system shape. Phase 1 delivery tracking (sub-epics, issue lists) lives on epic #268, not in this LLD.

Index

1. Purpose and scope
2. Responsibilities
3. Module layout and boundaries
4. State posture and infrastructure
5. Domain model
6. Persistence model (no database)
7. Core interfaces
8. API specification
   • 8.1 Edge (OpenAI-compatible)
   • 8.2 Health
   • 8.3 Internal admin listener (September)
   • 8.4 Filter-based API (OpenWebUI pre/post filter ↔ Orchestrator)
9. Key flows
   • 9.1 Request lifecycle
   • 9.2 Snapshot sync
   • 9.3 Token acquisition and cache
   • 9.4 Receipt build, sign, and emit
   • 9.5 Selection
   • 9.6 Streaming
10. Concurrency model
11. Validation
12. Security summary
13. Basic logging pattern (debugging)
14. Configuration
15. Error model
16. Open items to confirm
17. Decisions (O-P series)

1. Purpose and scope

The Orchestrator is the only UAI component in the BharatGrid request path. It receives each user query from OpenWebUI, selects one of the 7 registered agents, runs the token gate (observe posture at launch, enforcing from the September flip), routes the request through CDAC’s LLM infrastructure, streams the response back, and emits a hash-only receipt to the Audit Ledger. It is dual role by design: a UAI Seeker toward the providers and the provider-side verification gate toward the user channel.

Out of scope here: agent business logic (agents answer, this service selects, applies the gate, routes), the Registry write path, token minting, and receipt storage.

2. Responsibilities

Does:

• Serve an OpenAI-compatible edge API so OpenWebUI’s auto mode points at it with a config change and nothing else.
• Discovery: match the query to the single launch capability and select the agent. At launch: route-to-default among the 7. September: the selection engine with its golden-set bar.
• Seeker side: hold the Orchestrator’s did:webvh identity, obtain tokens from the Trust Server per capability and selected provider, cache one token per provider per TTL window. Live at launch.
• Provider gate: verify the token offline before routing. Signature, expiry, capability, audience matches the selected agent, agent active in the snapshot. Launch posture: log allow or deny with the token id and always pass through. From the September flip: reject fail-closed on any failure.
• Route via vllm / llm-connect with CDAC’s existing API credentials, stream the response through, accumulate hashes as bytes pass.
• Build, sign, and emit the receipt after the response completes, async, spool-backed. Zero latency added to the user path.

Does not:

• Persist payloads anywhere, ever. The spool holds receipts, which are hash-only.
• Trust the selection. A bad pick is a quality problem; every security property is re-derived at the gate. Under observe posture the deny is recorded, not enforced; the property still holds because nothing silent happens.
• Hold a database. In-memory snapshot and token cache, disk spool, nothing else.
• Hardcode endpoints or capability IDs. Endpoints come from the Registry record, the capability from discovery output.

3. Module layout and boundaries

One binary in its own repository, bharatgrid-orchestrator, with its own release cadence. The layout mirrors the monorepo service pattern; pkg/ is consumed from uai-core as a pinned Go module dependency, never vendored, never forked.

bharatgrid-orchestrator/
  cmd/main.go        wiring + config + graceful shutdown; builds backends, injects ports
  api/
    openapi.yaml     edge contract: OpenAI-compatible surface + health
  internal/
    model/           domain types (AgentView, Snapshot, Selection, TokenEntry, ReceiptDraft)
                     + sentinel errors + the port interfaces: RegistrySource, Selector,
                     TokenSource, Gate, ModelClient, ReceiptSink, Spool, Clock
    httpapi/         edge listener: routing, middleware, OpenAI-compat handlers, SSE streaming
    adminapi/        internal listener: snapshot refresh, spool status (September; seam only at launch)
    discover/        capability match + agent selection + route-to-default
    seeker/          Trust Server client: proof-of-control, token acquisition, per-provider cache
    gate/            provider handler: offline token verification (wraps pkg/jwtverify),
                     observe decorator at launch, enforce from the September flip
    llm/             vllm / llm-connect client (OpenAI upstream dialect), streaming
    registrysync/    snapshot fetch loop: startup fetch, interval refresh, last-good, seed fallback
    receipt/         ReceiptDraft assembly, hash accumulation, signing, async emitter + disk spool
    config/          typed config from environment, startup validation
  tools/mockllm/     OpenAI-dialect mock upstream for CI and the steel thread (O-P16),
                     excluded from the production build
uai-core pkg/ (module dependency, pinned tag)
                     jcs, hashing, ed25519x, proofcontrol (seeker/'s proof-of-control, #449),
                     jwtverify, receipt (#68), transport + uaierr (#63),
                     clients (Registry, Trust Server, Audit Ledger) (#70)

Boundary rules, enforced by depguard: internal/... imports only github.com/SoulVisionCreations/uai-core/pkg/... from the uai-core module, never its services/... internals. Feature packages never import a backend client directly: only cmd/ and the adapter packages (seeker/, llm/, registrysync/, receipt/) touch the network, each behind a port in model/. model/ imports nothing internal; everything else depends on model/, never the reverse. The primitives it depends on are specified in the Shared Primitives LLD.

Two listeners, same pattern as the other services: the edge API and a separate internal admin listener, never gateway-exposed. The admin listener is a September item; at launch a snapshot refresh is the interval loop or a restart.

One contract, generated both ways: api/openapi.yaml is the source of truth; oapi-codegen generates the server interface and request/response types. The OpenAI-compatible subset is pinned there so drift against OpenWebUI expectations is a CI failure, not a runtime surprise.

4. State posture and infrastructure

Stateless service with three in-process caches and one durable artifact:

• Snapshot: the last-good verified agent list, in memory behind an atomic pointer.
• Token cache: up to one entry per (capability, provider), in memory. Live at launch.
• Selection assets: the taxonomy seed, loaded at boot.
• Receipt spool: append-only JSONL on a mounted volume, fsync on append. Proposed companion file: spool.checkpoint bookmark for async Ledger emit (see 9.4; open item 6 — not finalized). Launch intent: one spool file; September: rotation, alerting, backpressure. Receipts only, hash-only, so a disk compromise leaks no content (O-P7).

No Postgres, no Redis. Horizontal scaling is per-replica state: each replica keeps its own snapshot, token cache, and spool; correctness does not depend on sharing any of them.

5. Domain model

type AgentView struct {
    DID        string   // did:webvh:...
    Capability string   // taxonomy id
    Endpoint   string   // resolved from the Registry record
    Status     string   // active | suspended | deregistered
    ETag       string
}

type Snapshot struct {
    Agents    []AgentView
    FetchedAt time.Time
    Source    string // registry | seed | last-good
}

type Selection struct {
    Agent      AgentView
    Capability string
    Method     string // matched | fallback-default
}

type TokenEntry struct { // live at launch
    Capability string
    Audience   string // agent DID
    JWT        string
    TokenID    string // jti, lands in the receipt
    ExpiresAt  time.Time
}

type ReceiptDraft struct {
    SeekerDID, AgentDID, Capability string
    RequestHash, ResponseHash       string // sha256 over canonical bytes
    TokenID                         string // empty only when issuance failed under observe posture,
                                           // recorded and visible, never silent (HLD flow D.12)
    RequestAt, ResponseAt           time.Time
    Status                          string // ok | upstream_error | client_aborted | rejected
}

Sentinel errors in model/: ErrNoSnapshot, ErrSnapshotStale, ErrNoAgent, ErrAgentSuspended, ErrTokenMissing, ErrTokenInvalid, ErrTokenAudienceMismatch, ErrUpstream, ErrSpoolFull. Under observe posture the token sentinels are logged with the gate decision, not returned to the handler; from the September flip they surface as rejections.

6. Persistence model (no database)

Decision (O-P3): the Orchestrator has no Postgres and no Redis. There is no SQL/NoSQL schema to migrate. Durable state is files on a mounted volume; hot state is in-process memory. Other UAI services (Registry, Trust Server, Audit Ledger) own their own databases — this binary only calls their HTTP APIs.

6.1 What we deliberately do not persist

6.2 In-memory stores (rebuilt after restart)

Logical record shapes match the domain model (AgentView, Snapshot, TokenEntry). No separate ORM schema.

Recovery when in-memory data is lost (no Redis)

There is no Redis (or other shared cache) to restore from. Process crash, OOM kill, deploy restart, or clearing process memory means all three in-memory stores are gone. Recovery is rebuild-from-source, not restore-from-cache:

What is not lost on process death (because it is on disk, not in Redis/memory):

• spool.jsonl — hash-only receipts already appended + fsynced survive crash; async worker resumes (via proposed checkpoint or by re-reading unemitted lines) and POSTs to Audit Ledger (idempotent).
• Last-good snapshot file — feeds step (2) above after restart when Registry is slow/down.
• Seed file — cold-start fallback.

What we accept losing: in-flight request state in the crashed process (that client’s HTTP connection dies); cached tokens (reminted); any snapshot only held in RAM that was never persisted as last-good (mitigated by persisting last-good on every successful Registry fetch).

Multi-replica note: replicas do not share memory. Losing one replica’s RAM does not affect another’s snapshot/token cache/spool. There is no cluster recovery protocol at launch — each instance rebuilds independently.

Process restart / memory loss
        │
        ├─► Snapshot:  Registry GET /agents
        │                 └─ fail → last-good file
        │                              └─ fail → SEED_PATH
        │                 then fail-closed if still unusable / too stale
        │
        ├─► Tokens:    empty cache → mint from Trust Server on demand
        │                 (observe: tokenless + empty TokenID if mint fails)
        │
        ├─► Taxonomy:  reload SEED_PATH at boot
        │
        └─► Receipts:  spool.jsonl on disk → worker retries Ledger POST

6.3 On-disk artifacts (the only “schemas”)

Durable spool + optional checkpoint below are the working proposal for emit sync / retention (open item 6) — not finalized. JSONL receipts themselves are required (O-P3 / O-P12); exact checkpoint/retention mechanics may change before freeze.

spool.jsonl line: one JSON object per line, frozen receipt fields (seeker DID, agent DID, capability, request/response hashes, token id, timestamps, status, signature) — same schema the Audit Ledger LLD accepts on intake. No message content.

spool.checkpoint (proposed): bookmark only. Day-1 recommendation: store an accepted line index / count (“last processed record was line N; next is N+1”) because it matches the record-by-record Ledger POST model and is easy to reason about in tests. Byte offset is an equivalent alternative (O(1) seek on large files). Pick one convention and stick to it. Worker rules:

1. One owner goroutine (or single worker) advances the checkpoint — no concurrent writers.
2. After a successful POST /audit/v1/receipts (202), fsync the new checkpoint past that receipt.
3. On Ledger 5xx / 429 / network error: leave the checkpoint unchanged and retry the same record (Ledger intake is idempotent on (transaction_id, message_id)).
4. On Orchestrator restart: reload the checkpoint and resume; never skip ahead; never delete unemitted lines to “catch up.”

Lag vs loss (day-1): a slow worker or unreachable Ledger increases lag (depth / oldest age). That is not data loss while lines remain fsynced on a durable SPOOL_DIR volume. Real loss is disk full / volume gone / crash before append fsync / outlet never reaching Orchestrator. Day-1 must not truncate or drop unemitted spool lines under pressure.

Soft lag thresholds (day-1): when unemitted depth exceeds SPOOL_ALERT or the oldest unemitted receipt exceeds a configured age, emit a structured warn log (depth, oldest age, last emit error class) with no payloads, JWTs, or receipt bodies. Prometheus / paging on those signals is September. Hard reject of new traffic (RECEIPT_BACKPRESSURE) is also September — launch logs-and-continues.

Launch intent: one spool file, no rotation required until decided. September: rotation / size caps with backpressure (RECEIPT_BACKPRESSURE) rather than dropping unemitted receipts.

6.4 Later (not launch)

Redis (or similar) appears only if/when DPoP jti guard or other shared anti-replay state is required — that is a September+ seam, not part of the 1.1.1 schema. Do not introduce Redis “just in case” at launch.

7. Core interfaces

type RegistrySource interface { Fetch(ctx context.Context) (Snapshot, error) }

type Selector interface {
    Select(ctx context.Context, req ChatRequest, snap Snapshot) (Selection, error)
}

type TokenSource interface { // real at launch (seeker/); failure handling per GATE_MODE
    Get(ctx context.Context, capability, audienceDID string) (TokenEntry, error)
}

type Gate interface { // observe decorator at launch, enforcing implementation behind it
    Verify(tok TokenEntry, sel Selection, snap Snapshot) error
}

type ModelClient interface {
    Stream(ctx context.Context, sel Selection, req ChatRequest) (UpstreamStream, error)
}

type ReceiptSink interface { Emit(d ReceiptDraft) error } // spool-backed, non-blocking

Why interfaces at the boundaries: handlers stay testable with fakes, and the September enforcement flip is removing the observe decorator around the same enforcing Gate, selected by GATE_MODE, with no handler change. The seeker/ and gate/ implementations are later replaceable by the Seeker Kit and Provider Kit (BG-D2 seam). The observe decorator verifies with the real verifier, logs the decision and token id, swallows the error, and returns nil. Enforcement was in the binary from day one; only the decorator leaves.

8. API specification

Implementer contract for the Orchestrator HTTP APIs. Use this section (plus api/openapi.yaml in the service repo) when coding handlers and OpenWebUI integration tests. OpenAPI remains the generated source of truth; the subsections below freeze the launch behavior reviewers and developers need without opening the YAML.

8.1 Edge (OpenAI-compatible, consumed by OpenWebUI)

Docs name: Auto Completion API. Wire paths stay OpenAI-shaped so OpenWebUI can call us with connection config only (no OpenWebUI fork for launch). System placement of OpenWebUI (multi-connection dropdown, Auto vs other providers) is in the HLD §3.2.

How OpenWebUI connects (ops)

Base URL must end with /v1 because OpenWebUI calls {base}/models and {base}/chat/completions.

Auth — UAI Orchestrator API key (proposal — not finalized)

Status: Open / proposal. Edge auth was a launch blocker; the approach below is the working proposal to unblock design and coding against mocks. It is not a closed decision until CDAC / BharatGrid confirm the connection auth contract (header, key issuance, rotation owner).

For BharatGrid we propose not reusing OpenAI/Claude/Gemini keys. UAI would issue a dedicated UAI Orchestrator API key for the OpenWebUI → Orchestrator hop only.

Proposed wire format on every edge call: Authorization: Bearer <UAI_ORCHESTRATOR_API_KEY>. Orchestrator compares to ORCH_API_KEY (alias of former ORCH_SERVICE_TOKEN) with a constant-time check; missing/wrong → 401 OpenAI-shaped unauthorized. Reject before body parse when practical. Proposed rotation: generate new key → update Orchestrator secret → update OpenWebUI connection key → retire old. Still to confirm with CDAC: rotation owner and cadence; whether Bearer shared-secret is accepted as-is or another header/scheme is required.

Per-connection keys in OpenWebUI never mix: the Orchestrator connection key authenticates only to us; selecting gpt-4o uses the OpenAI connection key.

Optional headers OpenWebUI may send (HTTP-Referer, X-Title, user-info forward headers): tolerate, do not require. No Data Principal fields in receipts at launch.

Endpoints

Do not implement /v1/embeddings, /v1/images/*, /v1/audio/* for this integration.

GET /v1/models (launch policy)

User journey: the user opens OpenWebUI or refreshes the page. OpenWebUI calls GET /v1/models on every configured connection to build the model dropdown. The user sees “UAI Auto” appear in the list alongside GPT-4o, Claude, etc. (if those connections are configured). This call happens before any chat — it populates the UI. The user never triggers it consciously; it fires on page load / connection refresh.

Return only auto / display name Auto on this connection. That forces the UAI path when this connection is used. Listing the 7 agent DIDs here is a product decision — if listed, users can pick an agent id and bypass Orchestrator selection unless those ids also resolve through us. data[].id is what OpenWebUI sends back as model on chat.

Example 200:

{
  "object": "list",
  "data": [
    {
      "id": "auto",
      "name": "Auto",
      "object": "model",
      "owned_by": "uai",
      "created": 1722470400
    }
  ]
}

POST /v1/chat/completions

User journey: this is the proxy-style integration path (not used at BharatGrid launch — see §8.4 for the filter-based production path). In this model, the user selects “UAI Auto” and sends a message. OpenWebUI routes the entire chat to the Orchestrator as a single HTTP call. The Orchestrator handles everything end-to-end: select the agent, get a token, run the gate, proxy the request to the agent’s LLM endpoint, stream the response back to the user, and build the receipt inline. The user sees tokens streaming in the chat bubble — indistinguishable from a direct OpenAI/Claude call. This path exists for environments where agents sit behind a single LLM endpoint rather than OpenWebUI pipe functions.

Example request:

{
  "model": "auto",
  "messages": [
    { "role": "user", "content": "What is the leave policy?" }
  ],
  "stream": true
}

• Accept model: "auto" for discovery mode. Accept a registered agent DID as model for direct mode (discovery skipped, gate and receipt still apply). Unknown model value → 404 model_not_found.
• messages[] required; used for upstream prompt (and later for selection engine input). Pass AS-IS to CDAC (no Orchestrator chat memory; O-P7).
• Streaming (stream: true): SSE (text/event-stream), one JSON chunk per data: line, flush per upstream chunk, terminal data: [DONE]. model in chunks may echo auto.
• Non-streaming (stream: false): single chat.completion JSON with full message.content. usage may be zeros if upstream omits counts.
• Mid-stream upstream failure after SSE headers: OpenAI-style error event, then [DONE]; receipt status: upstream_error.
• Client disconnect: cancel upstream; still emit receipt with status: client_aborted.

Caps / HTTP codes on oversize requests: §11 Validation. Upstream timeout defaults and edge effect: §9.6 Streaming.

api/openapi.yaml remains the generated source of truth; the shapes above are the launch contract OpenWebUI expects.

8.2 Health

User journey: the user never calls these directly. The container orchestrator (docker-compose / Kubernetes) probes /healthz and /readyz to decide whether to route traffic. If /readyz fails (no snapshot, spool full), new chats return 503 until the issue clears — the user sees an error in OpenWebUI. The user’s fix: wait (snapshot refresh) or alert ops.

• GET /healthz: process up.
• GET /readyz: snapshot present and within max age, spool writable. Deliberately does not probe the LLM upstream (avoids readiness flapping on upstream blips).

8.3 Internal admin listener (September)

• POST /admin/snapshot/refresh: force a Registry pull.
• GET /admin/spool: depth, oldest entry age, emit error counts.
• Separate port, service token, never gateway-exposed. Seam only at launch; the module boundary exists, the listener does not start.

8.4 Filter-based API (OpenWebUI pre/post filter ↔ Orchestrator)

The UAI Filter (uai_filter) is a global OpenWebUI filter that calls the Orchestrator on every chat via two endpoints: inlet (pre-filter, before the agent runs) and outlet (post-filter, after the agent responds). This is the production integration path for BharatGrid – not the proxy-style POST /v1/chat/completions approach. The full design rationale and end-to-end pipeline are in the HLD §4.C.

Auth on both endpoints: Authorization: Bearer <UAI_ORCHESTRATOR_API_KEY> – same key and validation as the edge API (§8.1).

Hashing ownership (filter path): the Orchestrator computes request_hash and response_hash with pkg/jcs + pkg/hashing (same primitives the Audit Ledger and auditors use). The UAI Filter must not implement canonical hashing in Python. Inlet already carries messages; outlet carries the assistant response bytes/text so the Orchestrator can hash them. The filter stays thin: call APIs, rewrite model, stash Orchestrator-returned fields.

POST /v1/inlet

Called by uai_filter.inlet (pre-filter) on every chat, before the agent pipe runs.

User journey — what the user sees when this API fires

Auto mode:
  1. User opens OpenWebUI, selects "UAI Auto" from the dropdown, types a question, hits Send.
  2. The chat input locks. A status line appears: "selecting best model…"
  3. ── POST /v1/inlet fires here ──
     Behind the scenes: filter forwards messages to the Orchestrator.
     Orchestrator hashes the request (pkg/jcs + pkg/hashing), picks the best
     agent, gets a token, runs the gate.
  4. The status line disappears. The chat bubble starts streaming the answer
     from the selected agent — the user never knows which agent was picked
     unless the UI chooses to show it.

Direct mode:
  1. User selects a specific agent (e.g. "Varya Video") from the dropdown, types a question, hits Send.
  2. The chat input locks. No "selecting best model" status — the user already chose.
  3. ── POST /v1/inlet fires here ──
     Behind the scenes: filter calls the Orchestrator with mode=direct.
     Orchestrator hashes the request, skips discovery, validates the agent
     is active, gets a token, runs the gate.
  4. The chat bubble starts streaming the answer from the user's chosen agent.

In both modes the user is waiting during step 3. Inlet latency (hash + snapshot
lookup + token cache hit or mint + gate) adds to the time before the first token
appears. Target: < 200ms on cache hit, < 1s on cold mint.

Request body

{
  "mode": "auto",
  "model": "uai_auto",
  "messages": [
    { "role": "user", "content": "What is the leave policy?" }
  ]
}

request_hash is not accepted from the filter as a trusted input. If a legacy client still sends it, the Orchestrator recomputes from messages and returns its own value.

Success 200

{
  "agent_model_id": "varya.video",
  "token_id": "jti-abc123",
  "request_hash": "sha256:<hex>",
  "status": "ok"
}

Error responses

Error envelope follows the same OpenAI shape as the edge API (§15).

What the Orchestrator does on inlet (ordered)

1. Authenticate Bearer key
2. Parse and validate request (mode, model, messages)
3. Compute request_hash over the canonical request with pkg/jcs + pkg/hashing
4. Snapshot freshness check
5. If mode=auto: run selection (launch: route-to-default). If mode=direct: look up agent in snapshot, reject if not found/suspended
6. Token acquisition: TokenSource.Get(capability, agent.DID) – cache hit or singleflight mint
7. Gate verify (observe at launch): log decision, pass through
8. Return agent_model_id, token_id, request_hash, status

No spool write, no Ledger write on inlet. Receipt is built only on the outlet call (single-phase receipt model).

POST /v1/outlet

Called by uai_filter.outlet (post-filter) after the agent pipe has responded to the user.

User journey — what the user sees when this API fires

Both modes (Auto and Direct):
  1. The agent has finished responding. The user sees the complete answer
     in the chat bubble. Streaming is done — the last token has arrived.
  2. ── POST /v1/outlet fires here ──
     Behind the scenes: the filter sends token_id, stashed request_hash,
     agent_model_id, status, and the assistant response text/bytes.
     Orchestrator computes response_hash (pkg/jcs + pkg/hashing), builds and
     signs one receipt, appends to the spool, returns ack.
  3. The user is already reading the answer or typing their next question.
     They never notice this call — it is invisible and non-blocking.

The outlet is pure accountability. Nothing the user sees depends on it.
If the outlet fails (Orchestrator down, network blip), the filter logs
the error and moves on — the user's conversation is never interrupted.
The receipt is lost for that query, which is a spool/audit concern, not a UX one.

Timing: outlet fires after the full response is delivered. The user has
already received their answer, so outlet latency (typically < 50ms) adds
zero perceived delay. The ack returns before the async Ledger POST.

Request body

{
  "token_id": "jti-abc123",
  "request_hash": "sha256:<hex>",
  "response": "Sure — here is the leave policy summary…",
  "agent_model_id": "varya.video",
  "status": "ok"
}

response_hash from the filter is not trusted. If present for transitional clients, the Orchestrator recomputes from response and uses its own digest. Raw response is held only in request memory for hashing — never written to spool, logs, or disk (O-P7).

Success 200

{
  "ack": true
}

The outlet response is deliberately minimal. The user has already received their streamed answer – the outlet must never block or delay the UX.

Error responses

What the Orchestrator does on outlet (ordered)

1. Authenticate Bearer key
2. Parse and validate request fields
3. Compute response_hash from response with pkg/jcs + pkg/hashing (drop raw response from memory after hashing)
4. Build ReceiptDraft: combine request_hash, response_hash, token_id, agent_model_id, status, timestamps (hash-only)
5. Sign the receipt with the Orchestrator’s Ed25519 key (same key used for proof-of-control)
6. Append to spool.jsonl + fsync
7. Return ack immediately – user is never blocked
8. Background: single async worker reads past spool.checkpoint → POST {LEDGER_URL}/audit/v1/receipts record-by-record with backoff (see 9.4)

Outlet never rejects a chat that already completed. Even on validation errors, the Orchestrator should log the issue and attempt a best-effort receipt rather than silently dropping accountability.

Relationship to the edge API (POST /v1/chat/completions)

Both APIs share the same auth (ORCH_API_KEY), the same snapshot, the same gate, and the same receipt spool. The filter API separates the control plane (inlet/outlet) from the data plane (agent pipe), which is the correct architecture when agents are OpenWebUI pipe functions rather than a single proxied LLM endpoint.

9. Key flows

9.1 Request lifecycle (ordered)

0. Authenticate the edge call (proposal, see 8.1): until the auth contract is finalized, implement against the proposed Authorization: Bearer ↔ ORCH_API_KEY check; fail closed with 401 before parse when practical. Swap if CDAC requires a different header/scheme.
1. Parse and cap the request (body size, message count). Reject oversize. Request hash precomputed over the canonical body.
2. Snapshot check: present and within SNAPSHOT_MAX_AGE. Stale beyond the bound: reject ErrSnapshotStale. This row stays fail-closed at every stage; without an endpoint there is nothing to route to (HLD failure matrix, Registry row).
3. Select: if model is auto, run discovery – match the capability and pick the agent. Launch: route-to-default (O-P15). On selector failure or timeout: the named default model (O-P10), Method = fallback-default. If model is a registered agent DID, skip discovery – look up the agent in the snapshot, verify it is active (reject if not found or suspended), proceed directly to token acquisition.
4. Token: TokenSource.Get(capability, agent.DID). Cache hit or a single-flight mint. Launch posture on failure: log, continue tokenless, the receipt records the empty TokenID. From the September flip: reject once the cached token expires.
5. Gate: Gate.Verify(token, selection, snapshot). Signature (EdDSA only, via pkg/jwtverify), expiry, capability match, audience equals the selected agent DID, agent status active. Launch posture: the observe decorator logs allow or deny with the token id and passes. From the September flip: any failure rejects with the SDE error code, no pass-through.
6. Route: ModelClient.Stream to the agent’s endpoint with CDAC credentials. Forward messages[] AS-IS.
7. Stream through to the client while accumulating the response hash.
8. On completion: finalize ReceiptDraft, sign, ReceiptSink.Emit, return. Emit never blocks the response.

9.2 Snapshot sync

Startup: blocking fetch from the Registry with a bounded deadline. On transport / HTTP failure, load the persisted last-good if present and still within SNAPSHOT_MAX_AGE; if last-good is missing or corrupt, load SEED_PATH (first-boot / recovery). A present but stale last-good fails closed (ErrSnapshotStale) and must not be revived via seed. Mark degraded loads Source: last-good | seed.

Runtime: interval refresh with ETag / If-None-Match, which Registry now serves on the list endpoint (see Day-1 fetch walk and Registry list ETag support below). When every page is unchanged (all-304), the agent list is left as-is; the Orchestrator advances freshness (FetchedAt) and may rewrite last-good so staleness tracks the last successful Registry confirmation, not the last content change. A content-changing successful fetch atomically swaps the in-memory snapshot and persists last-good to disk.

Empty catalog (successful walk, zero mapped agents): clear the in-memory snapshot, invalidate (delete) the active last-good file, and fail closed (ErrNoSnapshot / /readyz not ready). Do not fall through to seed on that bootstrap — live empty is source of truth. A subsequent all-304 while unready must not bump freshness (no forever-fresh empty). After last-good was invalidated, a later cold boot with Registry down may use seed only via the normal missing-last-good path.

Staleness policy: serve a published snapshot up to SNAPSHOT_MAX_AGE then reject; the September verified-only mode keeps the same rule with a tighter default (HLD failure matrix).

Day-1 fetch walk

Each refresh walks GET /registry/v1/agents (active-only feed per HLD) with page_size = 100 (Registry maximum). The walk is capped at 10 pages (1000 agents day-1). A per-page ETag cache is keyed by the request cursor (empty string for the first page). Every page sends If-None-Match when a prior ETag exists for that cursor; a 304 reuses the cached page body and the walk continues to the next cursor. Only when every visited page is 304 is the walk treated as unchanged (Changed = false): the in-memory agent list is left as-is, FetchedAt may still advance, and last-good may be rewritten so staleness tracks the last successful Registry confirmation rather than the last content change. Overlapping refresh callers are coalesced with singleflight; the shared walk runs on a detached 15s timeout so one caller’s cancel does not abort waiters. Registry URL must be https only; outbound TLS floor is 1.3, no 1.2 fallback (§12).

Registry list ETag support (Phase 1)

Registry serves a per-page ETag on GET /registry/v1/agents and answers If-None-Match with a 304 (Registry LLD §9.8). The blocker this section previously recorded is gone, so the walk above has a validator to send and the all-304 no-op path can run rather than sitting dormant.

Two properties of the Registry tag the walk depends on. It is scoped to the page, computed over that page’s response body, so caching it per cursor is correct and one tag can never satisfy another page. And it changes on a suspension, because the body changes when a row leaves the active set — which is what makes O-P6 freshness-from-confirmation safe: an all-304 walk cannot be hiding a suspended agent still present in the snapshot.

A 304 saves the Orchestrator bandwidth, not Registry database work: Registry still runs the query and marshals the body before comparing. So a tight refresh interval costs Registry the same either way, and the interval should be chosen on freshness need, not on an assumption that 304s are free upstream.

Still deferred on the Registry side, and not needed here: If-None-Match on the single-record GET /agents/{did}. The Orchestrator reads the list, never one record at a time.

Not yet confirmed: whether the no-op path has been exercised end-to-end against a live Registry. The two sides are specified to match; a G1 staging run should verify an unchanged catalog produces an all-304 walk with Changed = false.

9.3 Token acquisition and cache (live at launch)

Per (capability, provider) entry. Get returns the cached token if ExpiresAt - now > REFRESH_SKEW; otherwise a singleflight mint: challenge and proof-of-control to the Trust Server with the Orchestrator’s did:webvh identity, mint, cache. Trust Server unreachable at launch: cached tokens serve until expiry, then requests continue tokenless under the observe posture with the miss logged and the receipt showing the empty TokenID (HLD failure matrix, Trust Server row). From the September flip: new requests reject once caches expire, blast radius one TTL window. Tokens live only in memory; never logged, never spooled.

9.4 Receipt build, sign, and emit

Filter path (BharatGrid launch): on inlet the Orchestrator computes request_hash once over the canonical request (pkg/jcs + pkg/hashing). On outlet it computes response_hash over the assistant response bytes/text with the same primitives, builds a hash-only draft (raw response discarded after hashing), signs with the Orchestrator’s Ed25519 key via pkg/receipt (#68), appends to spool.jsonl (fsync), returns ack, then a single background worker POSTs to the Audit Ledger.

Proxy path (not launch): if POST /v1/chat/completions is used later, request hash is computed at accept and response hash accumulates over streamed chunks; finalize/sign/spool/emit is the same.

Audit Ledger call (emit worker): for each unemitted spool line, POST {LEDGER_URL}/audit/v1/receipts with the signed UAIReceipt JSON body. No bearer token — the receipt signature is authentication (Audit Ledger LLD §8.1). One HTTP call per receipt (record-by-record). Status handling:

Day-1 spool layout (proposal — not finalized; open item 6): spool.jsonl (append-only receipt lines, fsync on append) and spool.checkpoint (accepted line index/count or byte offset — see §6.3). The outlet handler appends + fsync then returns; the worker seeks past the checkpoint, POSTs new lines, advances on success, leaves unchanged on failure. Ledger idempotent resubmit covers crash after 202 before checkpoint fsync. Retention “delete vs keep N days” remains open; the proposal is “keep the file; checkpoint tracks sync” until decided. No external queue required to start. Soft lag: when depth exceeds SPOOL_ALERT or oldest unemitted age is too high, warn-log (no payloads); do not drop lines. September: depth beyond a hard cap applies backpressure (RECEIPT_BACKPRESSURE=on) rather than dropping receipts.

Client abort / upstream failure on the agent hop: outlet still emits with status: client_aborted or upstream_error and the hash of the response bytes actually available.

One identity, two uses: the same did:webvh key proves control to the Trust Server and signs receipts. The Ledger verifies against the resolved DID document (#172, #173).

9.5 Selection

Launch (O-P15): deterministic and minimal. The single capability from the seed taxonomy, the agent from DEFAULT_MODEL_DID, which must be active in the snapshot; if it is not, reject rather than guess. Method = fallback-default on every request is expected and honest at this stage. Under the observe posture a wrong default is a quality problem made visible in receipts, not a security hole; the September flip tightens it.

September: taxonomy match produces the candidate set, ranking inside the set is the selection engine behind Selector with a hard SELECT_TIMEOUT. Engine choice (pure heuristic vs a small LLM-assisted classifier) is open item 1; either way O-P8 holds. The golden selection set (query to expected agent) runs in CI with precision and recall thresholds, same pattern as the Registry discovery accuracy suite.

9.6 Streaming

SSE passthrough with flush per upstream chunk. Response hash accumulates as chunks pass. Upstream error mid-stream: terminate the SSE with an OpenAI-style error event, receipt Status: upstream_error. Client disconnect cancels the upstream call.

Upstream timeouts → user (config: UPSTREAM_* in §14):

Receipt status: upstream_error when the failure is on the LLM hop. Do not mask upstream failures as success.

Ops note: if tokens arrive as one delayed dump on staging, check proxy buffering on the OpenWebUI → Orchestrator hop (e.g. nginx proxy_buffering off). That is a deploy/path concern, not an Orchestrator handler redesign. Proxies in front of Orchestrator must disable response buffering or SSE stalls in OpenWebUI.

10. Concurrency model

Per-request goroutine from the HTTP server. Snapshot behind atomic.Pointer[Snapshot], readers never lock. Token cache: mutex-guarded map plus singleflight per key so one mint serves concurrent requests. Spool: single writer goroutine fed by a bounded channel; emit workers are a small fixed pool. Graceful shutdown drains in-flight requests, flushes the spool channel, leaves unemitted lines on disk.

11. Validation

Edge validation for OpenWebUI requests. Caps are config (§14); this table is the edge-visible contract (HTTP + error.code). Error envelope shape: §15.

Also: role whitelist (system, user, assistant, tool if tools enabled later); reject unknown critical UAI extension headers (fail-closed); tolerate unknown OpenAI body fields (compatibility). Launch: accept model: "auto" for discovery or a registered agent DID for direct mode; selection ignores the value for agent pick in Auto mode.

12. Security summary

• The Orchestrator is the only UAI component that sees payloads. Its cardinal rule is payload blindness at rest (O-P7): no content on disk, no content in logs, spool is hash-only, crash dumps disabled in production.
• TLS 1.3 on every hop, no 1.2 fallback. CDAC LLM credentials from environment or mounted secret, never logged, never in receipts.
• Gate verification pins EdDSA via pkg/jwtverify (LLD-P10); none and HS256 are unrepresentable. Audience and capability are exact matches. The observe posture changes what happens after verification, never the verification itself.
• The observe posture is a named availability exception (HLD security posture), written as a decision, flipped by GATE_MODE, and it must never be extended past the September flip without a new decision.
• The cnf claim is carried but not enforced (DPoP seam, BG-D8). Enforcing it is a config flip plus a jti guard, not a redesign.
• Logging hygiene: follow §13 Basic logging pattern. Never message content, never raw tokens, never signatures.
• Admin listener isolated, service token, separate port, September.
• Rate limiting: inherited from the platform edge in front of OpenWebUI at launch; per-DID limits become meaningful only with external seekers and land with that trigger.

13. Basic logging pattern (debugging)

System signals (health, steel thread) live in the HLD §10–§11. This section is the Orchestrator implementer contract for structured logs so one chat can be debugged without payloads.

Use structured logs (key/value, e.g. slog), not free-form prose. Goal: from one request_id, reconstruct what happened on the UAI path.

Correlation

One summary line per completed (or failed) request, at least:

Optional step logs (same request_id, higher verbosity or debug level): snapshot stale reject, Trust mint failure, SSE client disconnect — still no payloads.

Levels (launch)

Hygiene (non-negotiable)

How to debug a stuck or wrong chat

1. Get request_id from OpenWebUI/Orchestrator error body or Orchestrator info log
2. Grep logs for that request_id
3. Read: selection_method + agent_did → gate_decision + token_id → status + latencies
4. If status upstream_error → check CDAC/LLM side with time window (not message content)
5. If receipt missing → check spool depth / emit errors, then Ledger intake

Launch: this pattern + /healthz//readyz + cheap counters (requests by status, spool depth) is enough. September: add Prometheus/OTel on the internal listener (request/selection/upstream/mint/spool/emit) without changing the log field contract.

14. Configuration

Typed config with startup validation; the process refuses to start on a missing required value (fail-closed at boot). GATE_MODE=enforce plus RECEIPT_BACKPRESSURE=on together are the September flip.

15. Error model

Internal: pkg/uaierr canonical errors mapped from the sentinels; codes follow the SDE Technical Reference vocabulary exactly (invalid_did, unresolvable_did, proof_of_control_failed, invalid_token, conflict, …). Edge: because the surface is OpenAI-compatible, errors render in the OpenAI shape with the SDE code carried inside:

{ "error": { "message": "agent suspended", "type": "invalid_request_error",
             "code": "agent_suspended", "request_id": "..." } }

One mapping table in httpapi/, tested against golden fixtures so the edge dialect cannot drift. Under the observe posture, gate and token failures are logged with their SDE codes but do not reach the edge; the mapping table is exercised in tests either way so the September flip changes behavior, not code.

16. Open items to confirm

1. Selection engine: pure heuristic vs small LLM-assisted classifier. Cost, latency, and the golden-set bar decide. September decision. Owner: Aditya.
2. Default model DID and its owner (HLD open question 4). Provisional per O-P10; pick the stopgap now, revisit before the September enforcement flip.
3. OpenWebUI edge auth contract — still open (proposal phase). Former blocker. Working proposal: dedicated UAI Orchestrator API key via Authorization: Bearer, stored as ORCH_API_KEY (see 8.1). Not finalized until CDAC / BharatGrid confirm scheme, key issuance, and rotation owner/cadence. Implement against the proposal for local/mock work; treat production wiring as blocked on confirmation.
4. SSE through the C-DAC edge: confirm proxy buffering is off on the OpenWebUI to Orchestrator path, or streaming stalls. Test in integration week; owner: deploy issue. See 9.6.
5. User or department identifier passthrough into receipts (HLD open question 6); needs a frozen receipt field name before freeze, and must hold the DPDP position (no Data Principal data). Launch default: do not require / do not put user identity in receipts.
6. Spool retention / emit sync — still open (proposal phase). Working proposal: spool.jsonl + spool.checkpoint (line index/count or byte offset); single worker; keep file; checkpoint tracks Ledger sync; day-1 soft lag = warn logs on SPOOL_ALERT / oldest age (see §6.3 and 9.4). Delete-vs-N-days and final retention policy not finalized. Use the proposal to unblock local coding; confirm before production freeze.
7. Filter-path hashing — decided for launch coding: Orchestrator owns request_hash / response_hash via pkg/jcs + pkg/hashing (filter does not implement canonical hash). Confirm OpenAPI field names (response on outlet) before freeze.
8. GET /v1/models naming — resolved for launch: advertise auto / Auto only on the Orchestrator connection (see 8.1). Listing the 7 agents is a later product decision.
9. HA posture — resolved for launch coding: single replica (HLD one-VM topology). Stay stateless-compatible (per-replica spool/cache). LB/multi-replica is a later deploy topic with CDAC, not a Phase 1 feature.
10. CDAC LLM wire config (still open): real base URL, auth to vllm/llm-connect, agent addressing (Registry endpoint vs model id), request dialect, SSE behavior on Orchestrator → LLM. Local build uses OpenAI-compatible client + mockllm; staging/prod blocked until CDAC shares config.

17. Decisions (O-P series)