Token optimization

Token optimization is an optional layer that reduces the number of tokens a Sentinel exchanges with the model — lowering cost and latency. It works in two directions: Headroom compresses the prompt on the way in, and Caveman keeps the model's response terse on the way out. Each is off by default and enabled per Sentinel by an admin, from the Sentinel's Token optimization tab.

Headroom — input compression

Headroom rewrites large blocks of prompt content — tool outputs, logs, files, and retrieved context — to say the same thing in fewer tokens. It's content-aware: it detects what each block actually is (JSON, a diff, a build log, source code, …) and compresses each appropriately, and it only touches blocks above a minimum size.

It runs after your input guardrails pass, before the request reaches the model, on inbound prompts for OpenAI, Anthropic, and Gemini.

Turn on Headroom

1. Open a Sentinel and go to the Token optimization tab.
2. Enable Headroom — once on, it starts by compressing system prompts.
3. Adjust what it's allowed to touch:
   • Compress system prompts — on by default once Headroom is enabled; turn it off to keep your cache prefix stable.
   • Compress user messages — off by default (user messages are usually short).
   • Min tokens — the minimum block size before a block is compressed.

What should I know before turning it on?

• Lossy — compression rewrites wording to save tokens. It aims to preserve meaning and includes an inflation guard: if a rewrite would come out larger, the original is kept.
• Best-effort — savings vary per request and can be zero. Prompts below Min tokens are left untouched, and if compression fails the original prompt is forwarded unchanged.

Caveman — output optimization

Caveman reduces the tokens the model generates. It steers the model to answer in a terse register that keeps the technical substance — code, numbers, identifiers, commands, and error strings — and drops the conversational filler, restated questions, and over-explanation that most of an answer's tokens go to. Because output tokens are typically the most expensive part of a request, trimming them is often where the biggest savings are.

It works by appending a short instruction to the request before it reaches the model, so it applies across OpenAI, Anthropic, and Gemini without changing your application.

Turn on Caveman

1. Open a Sentinel and go to the Token optimization tab.
2. Enable Caveman.
3. Choose an intensity — how aggressively responses are shortened:
   • Lite — trims filler while keeping answers conversational.
   • Full — noticeably terser; the default.
   • Ultra — maximally compact, closest to notes and snippets.

What should I know before turning it on?

• It shapes the answer, not just its length — at higher intensities responses drop most prose and formatting. Keep it Lite, or off, for Sentinels that serve end-user-facing chat or anything that needs polished, verbose replies.
• Best-effort — Caveman instructs the model rather than editing its output, so the exact reduction varies by model and prompt. Output-token savings are estimated from the response, not measured exactly.

Privacy

Only token-savings metadata (tokens before, after, and saved) is reported to SUPERWISE®. Your prompts and the model's responses go to your chosen LLM provider, never to SUPERWISE®. See Data privacy and the FinOps dashboard for where the savings show up.

Get Sentinel: Sentinel on superwise.ai.

See also

• FinOps dashboard: see token consumption and savings across your fleet.
• Manage guardrails: the other per-Sentinel settings.
• Architecture: where the gateway sits in the request path.