Skip to main content

​
Observability

TensorCost is built so that any platform team running it can answer “what’s happening right now?” in their existing observability stack. We don’t ask you to adopt our tooling — we ship OTLP and let you point it at whatever you already use.

​
What’s instrumented

​
Distributed tracing

Tracing is OFF by default on self-hosted installs and ON in our managed environment. Enable with one env var.

​
Backend

The OTel SDK loads before Nest bootstraps, so auto-instrumentation patches the HTTP client, Sequelize, gRPC, and Redis stacks. The default endpoint targets localhost because the recommended deployment shape is the AWS Distro for OpenTelemetry collector running as an ECS sidecar that forwards to X-Ray + CloudWatch (or any OTLP backend). If the @opentelemetry/* packages are missing from the image, the tracing module degrades to a no-op log and the service still starts. No hard dependency.

​
Agent

Spans propagate trace context into the gRPC metadata, so a request initiated by the dashboard can be followed all the way through: REST gateway → service-to-service gRPC → agent execution → command result back over the stream.

​
Structured logs and trace correlation

Backend logs are JSON (Winston) with one object per line. When OTel is active, every line is automatically tagged with the active trace_id and span_id:
This lets you jump from a CloudWatch Logs (or Datadog, Loki, Grafana Loki) line straight to the full distributed trace in your tracing UI. Agent logs use the same pattern when OTel is enabled.

​
Log levels

​
Graceful shutdown

Fargate sends SIGTERM and gives the container 30s before SIGKILL. Each backend service handles it the same way:
  1. Arm a 30s watchdog (tunable via SHUTDOWN_TIMEOUT_MS).
  2. Re-entry guard — a second SIGTERM forces an immediate exit(1) rather than corrupt the sequence.
  3. Each step runs in its own try/catch:
    • Stop scheduled jobs (cron + @tensorcost/jobs runners).
    • Drain in-flight gRPC streams (gpu-service is the only one with long-lived streams).
    • Flush + close the LaunchDarkly client.
    • Close Redis pub/sub.
    • Close the Postgres connection pool.
  4. exit(0) on success; exit(1) if the watchdog fires, with a log line identifying the hung step.
The agent uses the same pattern: stop the gRPC stream, drain the send queue to disk atomically, send a final health: shutting_down report, exit.

​
Customer-side dashboards

You don’t need a TensorCost-specific observability tool — point your existing one at the OTLP endpoints we expose.

​
AWS — ECS + ADOT + X-Ray

Add the AWS Distro for OpenTelemetry as a sidecar container in each task definition. Backend points at http://localhost:4318 (the default). The ADOT sidecar forwards traces to X-Ray and logs to CloudWatch with no code changes. Customer-visible deliverables:
  • CloudWatch dashboard templates per service (request rate, p95 latency, error rate, DB query rate).
  • CloudWatch alarms wired to your existing on-call.
  • X-Ray service map showing the inter-service call graph for your tenant.

​
Kubernetes — OpenTelemetry Collector

Deploy the OpenTelemetry Collector as a DaemonSet or sidecar. Point OTEL_EXPORTER_OTLP_ENDPOINT at the collector’s service DNS.

​
Datadog / Honeycomb / Grafana Cloud / New Relic

Any OTLP-compatible SaaS works. Point the collector (or backend directly) at the SaaS’s OTLP endpoint with the appropriate API key in OTEL_EXPORTER_OTLP_HEADERS.

​
SLOs and error budgets

We track per-service SLOs internally. A public status page is planned but not live yet — ask support for current uptime data. Customer-facing SLAs: Error-budget policies: when 50% of the monthly budget is burned, non-critical deploys freeze.

​
Audit trail

Every state-changing action writes to the cross-tenant audit ledger: Audit rows are immutable. Retention defaults to 7 years per the SOC 2 readiness posture and survive tenant offboarding (audit-trail preservation is documented in the SOC 2 readiness guide). Export via:

​
Health endpoints

Every service exposes: Wire /health/ready to your load balancer’s health check; reserve /health/detailed for paged investigations.

​
Per-tenant observability views

Admins get a tenant observability page that shows:
  • Last-seen timestamps for each agent.
  • last_sync_status for each managed-inference connection, with the per-step error blob from the last sync.
  • Recommendation freshness — when the last batch ran per recommender.
  • Daily ingest counts vs the rolling baseline (silent ingest failures stand out as a sudden zero).
The same data is queryable via REST under /v1/integration/connections/:id/sync-history and via the MCP getConnectionHealth tool.