> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tensorcost.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Observability

> OpenTelemetry across the TensorCost backend and agent, structured logs with trace correlation, customer-side dashboards, and the audit trail.

# Observability

TensorCost is built so that any platform team running it can answer "what's happening right now?" in their existing observability stack. We don't ask you to adopt our tooling — we ship OTLP and let you point it at whatever you already use.

## What's instrumented

| Layer                  | Stack                                                                                | Notes                                                                |
| ---------------------- | ------------------------------------------------------------------------------------ | -------------------------------------------------------------------- |
| Backend (NestJS)       | OpenTelemetry auto-instrumentation for HTTP, Express/Fastify, gRPC, Sequelize, Redis | Spans cover REST, gRPC ingress, DB queries, Redis ops, OTLP outbound |
| Agent (Python)         | OpenTelemetry auto-instrumentation for `grpc` and `requests`                         | Trace context propagates into the gRPC stream metadata               |
| Microfrontends (React) | OpenTelemetry web SDK with route + RTK / TanStack Query span correlation             | Optional; tenant-side env var controls export                        |

## Distributed tracing

Tracing is **OFF by default on self-hosted installs** and **ON in our managed environment**. Enable with one env var.

### Backend

```
OTEL_ENABLED=true
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
OTEL_SERVICE_NAME=tensorcost-<service>
```

The OTel SDK loads **before** Nest bootstraps, so auto-instrumentation patches the HTTP client, Sequelize, gRPC, and Redis stacks. The default endpoint targets `localhost` because the recommended deployment shape is the **AWS Distro for OpenTelemetry collector** running as an ECS sidecar that forwards to X-Ray + CloudWatch (or any OTLP backend).

If the `@opentelemetry/*` packages are missing from the image, the tracing module degrades to a no-op log and the service still starts. No hard dependency.

### Agent

```
OTEL_ENABLED=true
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
OTEL_SERVICE_NAME=tensorcost-agent
```

Spans propagate trace context into the gRPC metadata, so a request initiated by the dashboard can be followed all the way through: REST gateway → service-to-service gRPC → agent execution → command result back over the stream.

## Structured logs and trace correlation

Backend logs are JSON (Winston) with one object per line. When OTel is active, every line is automatically tagged with the active `trace_id` and `span_id`:

```json theme={null}
{
  "level": "info",
  "service": "ai-service",
  "timestamp": "2026-04-29T16:41:38.412Z",
  "tenantId": "...",
  "message": "model-routing recommender produced 12 recommendations",
  "trace_id": "7a0db7e045...",
  "span_id": "2e757533..."
}
```

This lets you jump from a CloudWatch Logs (or Datadog, Loki, Grafana Loki) line straight to the full distributed trace in your tracing UI.

Agent logs use the same pattern when OTel is enabled.

## Log levels

```
LOG_LEVEL=error|warn|info|debug
```

| Level   | What you see                                                                   |
| ------- | ------------------------------------------------------------------------------ |
| `error` | Failures only                                                                  |
| `warn`  | Failures + non-fatal warnings ("Redis unavailable, falling back to in-memory") |
| `info`  | Default. Startup, scheduled-job summaries, request/response lines              |
| `debug` | Everything, including event publishes and individual metric sync counts        |

## Graceful shutdown

Fargate sends `SIGTERM` and gives the container **30s** before `SIGKILL`. Each backend service handles it the same way:

1. Arm a 30s watchdog (tunable via `SHUTDOWN_TIMEOUT_MS`).
2. Re-entry guard — a second SIGTERM forces an immediate `exit(1)` rather than corrupt the sequence.
3. Each step runs in its own try/catch:
   * Stop scheduled jobs (cron + `@tensorcost/jobs` runners).
   * Drain in-flight gRPC streams (gpu-service is the only one with long-lived streams).
   * Flush + close the LaunchDarkly client.
   * Close Redis pub/sub.
   * Close the Postgres connection pool.
4. `exit(0)` on success; `exit(1)` if the watchdog fires, with a log line identifying the hung step.

The agent uses the same pattern: stop the gRPC stream, drain the send queue to disk atomically, send a final `health: shutting_down` report, exit.

## Customer-side dashboards

You don't need a TensorCost-specific observability tool — point your existing one at the OTLP endpoints we expose.

### AWS — ECS + ADOT + X-Ray

Add the AWS Distro for OpenTelemetry as a sidecar container in each task definition. Backend points at `http://localhost:4318` (the default). The ADOT sidecar forwards traces to X-Ray and logs to CloudWatch with no code changes. Customer-visible deliverables:

* **CloudWatch dashboard** templates per service (request rate, p95 latency, error rate, DB query rate).
* **CloudWatch alarms** wired to your existing on-call.
* **X-Ray service map** showing the inter-service call graph for your tenant.

### Kubernetes — OpenTelemetry Collector

Deploy the OpenTelemetry Collector as a DaemonSet or sidecar. Point `OTEL_EXPORTER_OTLP_ENDPOINT` at the collector's service DNS.

### Datadog / Honeycomb / Grafana Cloud / New Relic

Any OTLP-compatible SaaS works. Point the collector (or backend directly) at the SaaS's OTLP endpoint with the appropriate API key in `OTEL_EXPORTER_OTLP_HEADERS`.

## SLOs and error budgets

We track per-service SLOs internally. A public status page is planned but not live yet — ask support for current uptime data. Customer-facing SLAs:

| Tier                  | Uptime      | First-byte p95 | Recommendation refresh |
| --------------------- | ----------- | -------------- | ---------------------- |
| Free / design partner | Best-effort | —              | Daily                  |
| Growth                | 99.9%       | `<800ms`       | Hourly                 |
| Enterprise            | 99.95%      | `<500ms`       | 5-minute               |

Error-budget policies: when 50% of the monthly budget is burned, non-critical deploys freeze.

## Audit trail

Every state-changing action writes to the cross-tenant audit ledger:

| Field              | Detail                                             |
| ------------------ | -------------------------------------------------- |
| `actor`            | User UUID + email, or service name                 |
| `action`           | Resource + verb (e.g. `enforcement.policy.create`) |
| `resource_id`      | Affected resource UUID                             |
| `before` / `after` | JSON diff of the change                            |
| `metadata`         | IP, user-agent, tenant ID                          |
| `trace_id`         | OTel trace correlation                             |
| `occurred_at`      | UTC timestamp                                      |

Audit rows are immutable. Retention defaults to **7 years** per the SOC 2 readiness posture and survive tenant offboarding (audit-trail preservation is documented in the [SOC 2 readiness guide](/soc2-readiness-guide#tenant-offboarding)).

Export via:

```
GET /v1/identity/audit?format=csv&from=2026-01-01&to=2026-04-01
```

## Health endpoints

Every service exposes:

| Path               | What it returns                                                         |
| ------------------ | ----------------------------------------------------------------------- |
| `/health`          | Liveness — `200 OK` when the process is up                              |
| `/health/ready`    | Readiness — checks DB, Redis, downstream gRPC peers; `503` if not ready |
| `/health/detailed` | Full per-dependency status; `admin`-scoped                              |

Wire `/health/ready` to your load balancer's health check; reserve `/health/detailed` for paged investigations.

## Per-tenant observability views

Admins get a **tenant observability** page that shows:

* Last-seen timestamps for each agent.
* `last_sync_status` for each managed-inference connection, with the per-step error blob from the last sync.
* Recommendation freshness — when the last batch ran per recommender.
* Daily ingest counts vs the rolling baseline (silent ingest failures stand out as a sudden zero).

The same data is queryable via REST under `/v1/integration/connections/:id/sync-history` and via the MCP `getConnectionHealth` tool.