Skip to main content

​
Budgets and alerts

Budgets are TensorCost’s spend caps; alerts are what fires when one is crossed, or when a policy is violated. This page documents both, and is deliberately specific about which alert paths actually run on a schedule versus which only work when you check them by hand — that distinction matters more here than almost anywhere else in the product.

​
Budgets

A budget is defined against one dimension — tenant, environment, team, project, provider, model, instance family, tag, or a custom key — for one period (week, month, quarter, or year), with a cap amount and an enforcement mode:
soft_cap and hard_cap do not currently gate or block anything — there is no admission-control path in the product yet that stops spend when a hard-cap budget is exceeded. All three modes alert; only alert is honestly named for what it does. Don’t rely on hard_cap as an enforcement mechanism until this ships.
Budgets can be nested — a child budget’s cap can’t exceed its parent’s, and reducing a parent’s cap below what its children already sum to is rejected — so a tenant-level budget can have team-level budgets underneath it that must stay within the total.

​
Thresholds — actual spend, fixed percentages

Every budget carries a list of alert thresholds — percentages of the cap that, once actual spend crosses them, fire an alert. The budget-creation form in the console sets these to 80% and 100% by default. The underlying model allows any set of percentages up to 200% of the cap, configurable per budget via the API, but the console form doesn’t expose that — what you create from the UI today is fixed at 80/100. These thresholds compare current utilization (actual spend so far in the period, as a percentage of the cap) against a fixed number. They do not move as a team’s normal spend grows, and they do not fire on a forecast — only on spend that has already happened.

​
What the forecast fields do — and don’t do

Every budget’s status computation includes two forward-looking figures:
  • days_to_overrun — at the current burn rate, how many days until this budget exceeds its cap.
  • projected_overage_cents — the projected dollar amount over cap by period end, at the current rate.
Both are computed and shown in the console — the Budgets list and the per-budget detail page both surface “projected to exceed cap in about N days, roughly $X over by period end” when the projection is close enough to be worth a call-out. But neither figure drives an alert. Crossing 80% or 100% of actual spend fires an alert; a budget that’s on pace to blow through its cap in three days, while still under 80% today, does not.
If you want an alert that can fire before the cap is reached, see the pace-based budget-burn-rate rule described below — it exists, but it isn’t creatable from the console today.

​
Budget detail

Each budget has a dedicated detail page — the enforcement mode, current-period utilization, a burn-down chart (cumulative spend against a linear path to the cap), and a 12-period trend table showing how this budget performed in prior periods. Useful for a bookmark or an alert-email deep link, not just the list view.

​
Alerts (Rules · Alerts)

The Alerts tab shows two live feeds: currently open alerts and alerts resolved in the last 24 hours. This is the surface a policy violation or a budget breach lands on.

​
What actually fires an alert

There are two genuinely working alert paths today:
  1. Budget threshold crossings. When a budget’s actual utilization crosses one of its alert thresholds (see above), an alert is created automatically — severity critical at 100%, warning at ≥ 90%, info below that. This runs every time new cost data is ingested; no manual step is needed.
  2. Burn-rate rules (budget-burn-rate). A pace-based rule: actual spend so far, compared against a pace-adjusted expectation for elapsed time in the period (e.g. “you’re at 60% of your monthly budget, but only 40% of the month has passed”). This is the one alert type in the product that reasons about pace rather than a flat percentage, and it is swept on a schedule (weekly) and does fire real alerts.

​
What does not run automatically — even though the form lets you create it

The ”+ New rule” form on the Alerts page lets you build a simple metric/operator/threshold condition — for example, “GPU utilization below 10% for 2 hours.” Creating one of these gives you a working Test button, which previews how many times the rule would have fired against historical data. It does not give you a live rule. There is no scheduled job anywhere in the product that evaluates a plain metric-threshold rule and creates a real alert from it — only the budget-burn-rate rule kind is actually swept, and that kind isn’t buildable from this form at all (it’s API-only today).
If you create a metric-threshold alert rule from the console today, it will never fire on its own. The only way to see whether it would have caught something is to re-run Test by hand. This is a real gap, not a UI bug — document it for anyone relying on these rules for live monitoring.

​
Other honest gaps in the alerting model

  • Severity is fixed, not configurable. Budget-threshold severity is a hardcoded mapping (100% → critical, ≥ 90% → warning, else info); there’s no way for a tenant to redefine what counts as “high” or to subscribe only to critical alerts.
  • No owner or routing field. Alerts and policy violations route to notification channels (Slack, PagerDuty, webhook) — destinations, not accountable people. There’s no field that assigns an open alert to a person or team; “acknowledged by” records who reacted after the fact, not who was on the hook before it fired.
  • Detection runs on raw metrics, not cost-to-serve. Rule conditions and the anomaly detector both key on raw values (GPU utilization, memory, temperature, or raw spend) — nothing in the alerting or anomaly pipeline normalizes against usage or revenue, so a spend increase that tracks legitimate growth looks identical to a real problem.
Hardware anomalies (Rules · Hardware anomalies) is a separate surface: a statistical outlier detector over raw GPU telemetry — utilization, memory, temperature, power draw — meant to catch a genuinely broken instance, not a spend problem. It shares the same three gaps listed above (fixed severity bands, no ownership, no cost-to-serve context), and it’s deliberately not folded into the Alerts feed — a spend-alert reader and a hardware-telemetry reader are different audiences, and merging them was tried and reverted for exactly that reason. Actions (Rules · Actions) is the operational queue for what enforcement has done or is about to do — approve or reject pending autonomous actions (see Policies, guardrails, and routing), and audit what already ran. It’s downstream of a policy firing, not an alerting surface itself.