Skip to main content

​
Policies, guardrails, and routing

The Rules section of the console holds three control systems that all get called “policies” in casual conversation but do different jobs, act on different things, and are configured on different tabs. Confusing them is easy because the names are close; this page exists to keep them separate.

​
Enforcement policies — GPU-fleet control

This is the oldest and most literal use of “policy” in the product: a condition on GPU-fleet telemetry that triggers an action against a running instance. Triggers (trigger_type): idle-threshold, cost-cap, stale-heartbeat, utilization-low. Actions (action_type): stop_instance, restart_instance, send_alert. A policy runs in dry-run mode (evaluate and log, don’t act) or enforce mode (actually fire the action). The Policies tab (Rules · Policies) is one resource with three views, switched by a control at the top of the page rather than three separate tabs:
  • List — every policy, its trigger, action, mode, and how many times it has fired.
  • Editor — author or edit a single policy’s condition and action. Opened by clicking a row or ”+ New policy.”
  • Pre-block readiness — before flipping a policy from dry-run to enforce, this view shows which real events over a trailing window (7/14/30 days) would have blocked something if the policy had been live — ranked by frequency, with the engineers behind those events named where the underlying request carried a user id. This is the safety check for turning enforcement on without surprising anyone.
Policies stop or restart instances directly; they don’t touch model selection or in-flight inference calls — that’s the other two systems below.

​
AI guardrails — predictive intervention on inference calls

Guardrails are a different mechanism entirely: a trigger condition evaluated before a risky inference or tool call completes, based on a forecast (for example, “this call’s projected cost exceeds 5”) rather than a GPU metric. Actions: downgrade_tier (route to a cheaper model tier for this call), prune_tool_call (drop a tool invocation from the call), hard_stop (block the call outright). Modes: auto (act automatically), suggest, alert, or disabled. Each guardrail’s trigger is a simple condition list (a field/operator/value structure, not a full expression language) — the console shows it as a plain-language summary like “cost > 5” rather than exposing the underlying format as jargon. Guardrails live on their own tab (Rules · Guardrails); they aren’t nested inside the GPU-enforcement editor because the two systems don’t share a data model.

​
Routing policies — which model handles a request

Routing policies are the newest surface here and answer a third, unrelated question: given a request, which model should actually serve it. Each policy is scoped to a model key (or all models, if left blank) and an environment, and constrains routing with four floors:
  • Min quality — a quality score the routed model must clear.
  • Latency budget — a maximum acceptable latency, in milliseconds.
  • Min savings % — the routing candidate must save at least this much versus the default, or it isn’t used.
  • Rollout % — what fraction of matching traffic actually gets routed under this policy, for a gradual ramp.
A routing policy’s effective state is derived from its own fields rather than a separate status flag:
  • Live — enabled, not in dry-run, and rollout % is above zero. Actually routing production traffic.
  • Shadow — enabled but either in dry-run mode or at 0% rollout. Evaluated, but not affecting real traffic yet.
  • Disabled — turned off.
Creating a routing policy requires an approving user (approved_by), resolved automatically from your signed-in identity — there’s no separate approval workflow to click through, but the record of who approved it is kept.

​
Why these live in one section but not one page

All three systems get grouped under “Rules” because they’re all forms of automated cost control a tenant admin authors and reviews. But an enforcement policy stopping an idle GPU, a guardrail blocking an over-budget tool call mid-request, and a routing policy silently sending 20% of traffic to a cheaper model are different enough in mechanism, blast radius, and review workflow that folding them into one editor would hide more than it would simplify. Each keeps its own tab, its own data model, and its own list of triggers and actions.

​
Downstream: the action queue

Whatever an enforcement policy or the autonomous ladder decides to do shows up in Rules · Actions — the operational queue for approving, rejecting, or auditing an action before or after it runs. It’s a review surface, not a fourth control system: everything in that queue was produced by one of the three systems above.