TIDE: Token-Informed Depth Execution for Per-Token Early Exit in LLM Inference

Dimensions overall score 8.0

• TIDE requires no model retraining and works with any HuggingFace causal LM.method 90%
• TIDE achieves 100% prefill exit rate on DeepSeek R1 Distill 8B, with 5% of tokens exiting at layer 11 and the remaining at layer 31.result 95%
• TIDE reduces prefill latency by 7.2% and increases single-batch throughput by 6.6% on NVIDIA A100 with DeepSeek R1 Distill 8B.result 95%
• During autoregressive decoding, TIDE achieves 98-99% early token exit rate while maintaining correct multi-step math problem solving.result 90%
• TIDE improves throughput by 8.1% at batch size 8 on Qwen3 8B (36 layers).result 95%
• Calibration on 2,000 WikiText samples takes under 3 minutes and produces a ~4 MB router checkpoint.technical 95%
• TIDE auto-detects GPU architecture and supports float32, float16, and bfloat16 through fused CUDA kernels.technical 90%
• The TIDE system comprises 1,308 lines of Python and 1,081 lines of CUDA/C++ with 74 passing tests.technical 95%

Competitive landscape

Competitor map is still being generated for this paper. Enable generation or check back soon.

Startup potential card