ALBA: A European Portuguese Benchmark for Evaluating Language and Linguistic Dimensions in Generative LLMs

Dimensions overall score 4.0

• ALBA is a new benchmark specifically designed to evaluate Large Language Models (LLMs) on European Portuguese (pt-PT) across eight linguistic dimensions.method 95%
  Evidencepartial

  we introduce ALBA, a linguistically grounded benchmark designed from the ground up to assess LLM proficiency in linguistic-related tasks in pt-PT across eight linguistic dimensions

  Implicationpartial

  The abstract explicitly states the introduction of ALBA and its purpose for evaluating LLMs in pt-PT across eight linguistic dimensions.

  Verificationpartial

  partial

• ALBA was manually constructed by language experts and utilizes an LLM-as-a-judge framework for scalable evaluation.method 95%
  Evidencepartial

  ALBA is manually constructed by language experts and paired with an LLM-as-a-judge framework for scalable evaluation of pt-PT generated language.

  Implicationpartial

  The abstract clearly describes the manual construction by experts and the use of an LLM-as-a-judge framework for evaluation.

  Verificationpartial

  partial

• Experiments on ALBA reveal performance variability among LLMs across different linguistic dimensions.result 95%
  Evidencepartial

  Experiments on a diverse set of models reveal performance variability across linguistic dimensions

  Implicationpartial

  The abstract mentions experiments revealing performance variability across linguistic dimensions, and the paper structure indicates a results section.

  Verificationpartial

  partial

• Fully open LLM models generally achieve lower scores on the ALBA benchmark compared to proprietary state-of-the-art systems.result 90%
  Evidencepartial

  Overall, fully open models achieve lower scores.

  Implicationpartial

  The 'Overall Results' section explicitly states that 'fully open models achieve lower scores.'

  Verificationpartial

  partial

• AMALIA, a model tailored for European Portuguese, achieves the strongest results among fully open models on ALBA, outperforming larger models in specific dimensions.result 90%
  Evidencepartial

  AMALIA, tailored for European Portuguese (pt-PT), achieves the strongest results among fully open models, even outperforming the larger Gemma 3-12B in culturally bound semantics and lexicology

  Implicationpartial

  The text directly compares AMALIA's performance to other models, highlighting its strengths.

  Verificationpartial

  partial

• LLMs tend to perform well on syntactic, lexical, and discourse-level tasks but exhibit substantially lower performance in phonology and morphology.result 90%
  Evidencepartial

  LLMs tend to perform well on syntactic, lexical, and discourse-level tasks, while exhibiting substantially lower performance in other linguistic dimensions.

  Implicationpartial

  The paper explicitly contrasts LLM performance across different linguistic dimensions, noting strengths and weaknesses.

  Verificationpartial

  partial

• LLMs exhibit limitations in tasks involving phonology, morphology, and wordplay, potentially due to tokenization segmentation.limitation 90%
  Evidencepartial

  LLMs exhibit limitations in tasks involving phonology, morphology, and wordplay, particularly in tasks involving rhyme, scansion, syllable segmen- tation, morphological composition, and character- level manipulations such as reordering or counting.

  Implicationpartial

  The paper directly states these limitations and offers a potential technical reason.

  Verificationpartial

  partial

Competitive landscape

A new benchmark and evaluation framework to assess Large Language Model proficiency in European Portuguese across eight linguistic dimensions.