98$\times$ Faster LLM Routing Without a Dedicated GPU: Flash Attention, Prompt Compression, and Near-Streaming for the vLLM Semantic Router

Dimensions overall score 8.0

GitHub Code Pulse

No public code linked for this paper yet.

• The vLLM Semantic Router achieves a 98x improvement in end-to-end latency for LLM routing.result 95%
• Stage 1 optimization, a custom CK Flash Attention operator for ONNX Runtime on ROCm, reduces attention memory from O(n^2) to O(n).method 95%
• Stage 1 optimization reduces end-to-end latency from 4,918ms to 127ms, a 38.7x improvement.result 95%
• Stage 2 optimization, using classical NLP prompt compression techniques, reduces input tokens to approximately 512 without neural inference.method 95%
• Stage 2 optimization reduces end-to-end latency from 127ms to 62ms, a 2.0x improvement.result 95%
• Stage 3 optimization, near-streaming body processing with adaptive chunking and zero-copy JSON, eliminates serialization overhead.method 95%
• The total router GPU footprint is under 800MB, enabling it to share a GPU with LLM serving instances.technical 95%
• The proposed router removes the need for a dedicated accelerator.market 95%

Competitive landscape

Competitor map is still being generated for this paper. Enable generation or check back soon.

Startup potential card