Dimensions overall score 8.0

GitHub Code Pulse

No public code linked for this paper yet.

• SommBench is a multilingual benchmark designed to assess sommelier expertise in language models across three tasks: Wine Theory Question Answering, Wine Feature Completion, and Food-Wine Pairing.method 95%
• SommBench is available in eight languages: English, Slovak, Swedish, Finnish, German, Danish, Italian, and Spanish.method 95%
• The benchmark datasets were developed in collaboration with a professional sommelier and native speakers, resulting in 1,024 WTQA questions, 1,000 WFC examples, and 1,000 FWP examples.method 95%
• The most capable models perform well on wine theory question answering, with up to 97% correct answers achieved by a closed-weights model.result 90%
• Feature completion is more challenging than wine theory question answering, with performance peaking at 65%.result 90%
• Food-wine pairing is particularly challenging for language models, with Matthews correlation coefficient (MCC) scores ranging between 0 and 0.39.result 90%
• SommBench tests whether textual grounding is sufficient for language models to emulate expert-level sensory judgment in the domain of smell and taste.technical 85%
• Previous cultural evaluation benchmarks focus mainly on basic cultural knowledge that can be encoded in linguistic form, while SommBench focuses on sensory domains.limitation 85%

Competitive landscape

Competitor map is still being generated for this paper. Enable generation or check back soon.

Startup potential card