Skip to main content

​
ML, anomaly detection, and forecasting

TensorCost’s ML surface covers four jobs:
  1. Anomaly detection across GPU metrics and AI spend (statistical baseline + optional ML layer).
  2. Burn-rate alerting against budget hierarchies (in flight).
  3. Cost forecasting at tenant, application, and team scope.
  4. Recommendations — the four shipped Bedrock recommenders, plus their GPU-side counterparts.
This page is the customer-facing tour. Engineers and integrators looking for the algorithm details can dig into ADR-0011 (event store) and the per-recommender source under apps-new/services/ai-service/src/recommenders/.

​
Anomaly detection — statistical layer

Every metric stream — GPU utilization, memory, temperature, daily cost, hourly cost, agent invocation count, cache-hit rate — runs through three independent statistical detectors: An anomaly is confirmed when at least 2 of 3 methods agree (composite confidence ≥ 0.5). Day-of-week seasonal baselines avoid false positives on predictable patterns (training jobs that always spike Tuesday morning, etc.). Every detection writes a row to the analysis audit log with the inputs, scores, decision, and reason — useful for tuning thresholds and for SOC 2 processing-integrity evidence.

​
Anomaly detection — ML layer (opt-in)

Customers on growth and enterprise plans can enable a per-tenant Isolation Forest layer that composes with the statistical scores.

​
Training

MLTrainingJob runs daily. For each tenant + model type, retrain triggers on: Minimum samples default to 1,000. For ai_spend_anomaly, one sample = one filled hour bucket (so a 14-day lookback caps at 336 samples; new tenants need to accumulate hours, or operators set ML_MIN_TRAINING_POINTS=200 until volume builds).

​
Inference and ensemble

For each incoming metric:
  1. The statistical scorer runs.
  2. If ml-enabled is on and the model is loaded, the sidecar predicts.
  3. The composite score is a weighted average — ML_ENSEMBLE_WEIGHT defaults to 0.2 (20% ML, 80% statistical).
  4. An alert fires when the composite crosses the severity threshold.
The ML score alone is never a sole trigger. The statistical layer always runs.

​
Circuit breaker

All sidecar calls are wrapped in a circuit breaker. After 3 consecutive failures or 30s timeouts, the breaker opens for 60s and detection falls back to the statistical layer. The ML models admin page surfaces breaker state — CLOSED, HALF_OPEN, or OPEN.

​
Runaway-loop and retry-storm detection

For agent workloads — where one bug can burn $10,000 overnight — TensorCost ships a dedicated runaway-loop detector alongside the generic anomaly detection. It triggers on:
  • Sudden invocation-count spike (>3σ from the per-agent baseline).
  • Unusually long invocation chains (same conversation_id exceeding N calls).
  • Cost-per-user-session spike vs the rolling baseline.
Alert payloads include the top 5 offending request IDs and a one-click Pause agent action where the customer has wired in their orchestrator. Routes through PagerDuty, Slack, email, or webhook per configuration.

​
Cost forecasting

cost-service produces 30-, 60-, and 90-day forecasts at three scopes:
  • Tenant total
  • Application
  • Team
Forecasts are produced by an ensemble of seasonal-naïve and ARIMA-style models, blended by recent error rates. Each forecast carries a confidence interval and a “drift score” — high drift means the forecast is less reliable, typically because a workload pattern just changed (a new agent rolled out, a fine-tune started, a Bedrock provisioned-throughput contract activated). When the forecasted month-end exceeds the budget, a budget breach predicted event fires (see real-time events) and the configured channels are notified.

​
Recommendations

Recommendations are computed by per-domain recommenders. Four are shipped today against Bedrock; the same patterns apply to Azure OpenAI / Vertex / OpenAI / Anthropic as those adapters land.

​
Bedrock recommenders (shipped)

Each recommendation carries:
  • A specific dollar impact estimate.
  • The evidence (sample request IDs, cost breakdown, A/B plan).
  • An accept / dismiss-with-reason / snooze action.
Acceptance writes a row to the savings ledger; verified savings populate after the 30-day verification window. Verification methodology is documented in the Savings methodology PDF (downloadable from the dashboard).

​
GPU recommenders

​
Live training and recommendation events

Every job and recommender publishes Socket.IO events:
  • ml.training.started, ml.training.completed, ml.training.failed
  • recommendation.created, recommendation.accepted, recommendation.dismissed
  • forecast.generated, forecast.budget_breach_predicted
  • anomaly.detected, anomaly.resolved
  • runaway_loop.detected, runaway_loop.paused
See real-time events for the full catalog and consumption patterns.

​
The ML models admin page

Admins on plans with the ML layer get an ML models page that shows:
  • Whether ml-enabled is active for the tenant.
  • Active models per type with version, sample count, accuracy metrics.
  • Ensemble weight.
  • Circuit breaker state.
  • Per-model retraining status with reason labels.
  • A train now button per model.
Live training progress streams in via Socket.IO so the page reflects starts, completions, and failures without a page reload.

​
Data minimization

The recommenders see model IDs, token counts, latency, and (where the customer enabled invocation-log capture) request IDs and hashes of prompts/responses. Raw prompts and responses are never stored. Customers concerned about content can disable request-metadata capture entirely and still get model/cost/latency-level recommendations.