Operations & observability
Know what ran.
Know when to stop.
An operating model connects signals to decisions. Record what the system attempted, distinguish known from unknown outcomes, and make intervention possible.
Measure the execution, not just the endpoint.
Define event semantics before aggregating them. A successful network response is not necessarily a successful task, and a timeout is not necessarily a failed external action.
| Signal family | Measurement definition | Decision supported |
|---|---|---|
| Run outcomes | Accepted, completed, failed, cancelled and outcome-unknown runs; preserve the denominator and observation window. | Separate execution reliability from business acceptance. |
| Latency & queue age | Queue delay, execution duration and end-to-end elapsed time; distinguish unfinished from completed runs. | Locate backpressure and decide when to stop accepting work. |
| Policy decisions | Allow, deny and approval-required decisions, grouped by reason and policy version. | Detect boundary attempts and unexpected access changes. |
| Resource use | Tool calls, provider usage, execution time, retry work and storage use attributable to a workload. | Identify loops, budget pressure and inefficient paths. |
| Evidence delivery | Audit delivery lag, dropped or missing records, buffer pressure and redaction failures. | Pause evidence-dependent actions when history cannot be preserved. |
Keep identifiers out of metric labels.
Use bounded dimensions such as workload, environment, tool and outcome class. Keep per-run correlation IDs in access-controlled logs or traces. Raw prompts, payloads and secrets are not logged by default; any additional capture requires an approved purpose and retention rule.
We define in-scope instrumentation. The customer owns approved collection infrastructure and access; signal interpretation and thresholds are agreed together.
Page for consequences. Route for ownership.
Severity reflects potential harm and the ability to contain it—not simply the number of errors. Each alert needs an owner, context and an actionable runbook.
| Priority | Trigger concept | Handling principle |
|---|---|---|
| Contain | Suspected boundary breach, unauthorized action or uncontrolled consequential execution. | Stop affected dispatch, restrict credentials and preserve evidence. Escalate to the designated incident owner. |
| Restore | A supported path cannot complete safely; queues or dependencies threaten accepted work. | Limit new work, inspect unknown outcomes and apply the approved recovery procedure. |
| Review | Usage trends, drift, noisy failures or a reduction in evidence quality without an immediate unsafe action. | Assign a review owner and validate the finding before changing policy or thresholds. |
We propose actionable alert conditions. The customer designates responders and business escalation; routing and priority tests are shared.
Make the stop path a first-class path.
- Classify and assign authority.
Identify the affected workload, boundary and potential external effects. Establish who can stop work and who owns the incident.
- Contain the affected path.
Pause dispatch, cancel bounded work and restrict relevant credentials or routes. Preserve unrelated safe workloads where the boundary permits.
- Preserve and reconcile.
Capture approved configuration and event references. Distinguish completed actions, pending work and outcomes that require downstream confirmation.
- Recover against evidence.
Validate compatible configuration and state. Re-enable only the approved scope after the incident authority accepts the checks.
- Review and update.
Record causes, contributing conditions and assigned follow-up actions. Update tests and runbooks before expanding the scope again.
Attribute usage before enforcing budgets.
Associate model usage, tool calls, execution resources and storage with workload and run references. Separate estimated consumption from provider-reconciled usage; provider reports may arrive after execution.
- Enforcement
- Set concurrency, step, token and tool-call limits. Reserve budget for in-flight work and retries; check authority before dispatch. Use configured ceilings to stop new work.
- Failure handling
- Quota exhaustion returns an explicit outcome. Do not silently route to another provider or account. Delayed metering and already-dispatched work can produce additional consumption.
We implement agreed attribution and local limits. The customer owns spending authority and provider-account controls; reconciliation and acceptable overrun handling are shared. A local quota is not a guarantee about a provider invoice.
Detect changes beyond the code.
Track a reviewed baseline for runtime artifacts, policies, prompts, tool schemas, source permissions and provider routes. Changed behavior without a local release is still a change that needs investigation.
- Compare configuration and dependency references with the accepted release record.
- Exercise repeatable evaluation inputs after model, tool or policy changes.
- Review shifts in denied actions, outcome classes, resource use and retrieval provenance.
- Quarantine an affected route when a change invalidates its accepted boundary; do not automatically widen permissions.
We define baseline checks in scope. The customer notifies relevant environment changes; acceptance and safe restart are shared decisions.