Budget-Obedient Hysteretic Gating for LLM Inference

Abstract

The preprint introduces BOSC, an audit-first controller for LLM inference that enforces tail-latency, time-to-first-token, and KV-cache budgets via virtual queues and hysteretic gating under fail-closed telemetry contracts.

Keywords

• LLM inference
• budget obedience
• tail latency
• time to first token
• KV cache
• hysteretic gating
• telemetry contract
• virtual queues
• auditability
• controller verification
• serving

Identifiers and source records

• DOI
• Works entry
• ORCID