MiroEval: Benchmarking Multimodal Deep Research Agents in Process and Outcome

Dimensions overall score 7.0

• Existing benchmarks for deep research systems predominantly assess final reports using fixed rubrics, failing to evaluate the underlying research process.limitation 95%
• MiroEval's benchmark comprises 100 tasks, with 70 being text-only and 30 being multimodal.method 95%
• Process quality serves as a reliable predictor of overall outcome while revealing weaknesses invisible to output-level metrics.result 90%
• Multimodal tasks pose substantially greater challenges for deep research systems than text-only tasks, with most systems declining by 3 to 10 points.result 90%
• The MiroThinker series achieves the most balanced performance, with MiroThinker-H1 ranking the highest overall in both text-only and multimodal settings.result 90%
• MiroEval's process-centric evaluation audits research trajectories across five intrinsic dimensions: search breadth, analytical depth, progressive refinement, critical thinking, and efficiency.method 85%
• The benchmark tasks are constructed via a dual-path pipeline that supports periodic updates, enabling a live and evolving setting.method 85%
• User-derived queries are consistently harder for deep research systems than auto-generated ones.result 80%

Competitive landscape

Competitor map is still being generated for this paper. Enable generation or check back soon.

Startup potential card