Rocks, Pebbles and Sand: Modality-aware Scheduling for Multimodal Large Language Model Inference

Dimensions overall score 7.0

GitHub Code Pulse

No public code linked for this paper yet.

• RPS-Serve reduces average time-to-first-token (TTFT) by 54% overall for multimodal large language models (MLLMs).result 95%
• RPS-Serve reduces average time-to-first-token (TTFT) by 78.5% for latency-critical requests for MLLMs.result 95%
• Existing LLM serving systems optimized for text-only workloads fail under multimodality due to resource monopolization by large requests.limitation 90%
• Multimodal requests differ by orders of magnitude in resource demands, with videos being the most demanding, followed by images, and then text.technical 90%
• RPS-Serve is a modality-aware scheduler designed to improve MLLM inference responsiveness and avoid starvation.method 95%
• Image and video requests occupy one to three orders of magnitude more memory than text requests.technical 90%
• Text requests, despite being lightweight, suffer significant delays and dominate SLO violation counts in multimodal workloads due to head-of-line blocking by larger requests.limitation 85%
• RPS-Serve reduces average time-to-first-token (TTFT) by 54% overall for multimodal large language models (MLLMs).result 95%
• RPS-Serve reduces TTFT by 78.5% for latency-critical requests in MLLMs.result 95%
• Existing LLM serving systems fail under multimodality due to large requests (e.g., videos) monopolizing resources, causing head-of-line blocking.limitation 90%
• Videos behave like 'rocks', images like 'pebbles', and text like 'sand' in terms of resource demands for MLLMs.technical 90%
• RPS-Serve is a modality-aware scheduler designed to prioritize requests based on their modality.method 95%

Competitive landscape

Competitor map is still being generated for this paper. Enable generation or check back soon.

Startup potential card