Dimensions overall score 8.0

• Omanic provides decomposed sub-questions and intermediate answers as structural annotations for analyzing reasoning processes in multi-hop QA.method 95%
• State-of-the-art LLMs achieve only 73.11% multiple-choice accuracy on OmanicBench, confirming its high difficulty.result 92%
• Supervised fine-tuning on OmanicSynth brings substantial transfer gains (7.41 average points) across six reasoning and math benchmarks.result 91%
• CoT's performance hinges on factual completeness, with its gains diminishing under knowledge gaps and errors amplifying in later hops.technical 88%
• Omanic contains 10,296 machine-generated training examples (OmanicSynth) and 967 expert-reviewed human-annotated evaluation examples (OmanicBench).method 93%
• The human-annotated evaluation set is small (967 examples), which could lead to overfitting or biased benchmarks.limitation 85%
• Step-wise evaluation adds computational overhead, potentially slowing down real-time applications.limitation 82%
• The dataset's synthetic training examples may not generalize to all real-world domains, limiting effectiveness in niche industries.limitation 80%

Competitive landscape

Competitor map is still being generated for this paper. Enable generation or check back soon.

Startup potential card