Automating Clinical Information Retrieval from Finnish Electronic Health Records Using Large Language Models

Dimensions overall score 7.0

• The Llama-3.1-70B model achieved 95.3% accuracy and 97.3% consistency in free-text generation for clinical information retrieval.result 95%
  Evidencepartial

  In free-text generation, Llama-3.1-70B achieved 95.3% accuracy and 97.3% consistency across semantically equivalent question variants

  Implicationpartial

  This claim is directly stated in the abstract with specific performance metrics.

  Verificationpartial

  partial

• The Qwen3-30B-A3B-2507 model demonstrated comparable performance to Llama-3.1-70B in free-text generation.result 90%
  Evidencepartial

  while the smaller Qwen3-30B-A3B-2507 model achieved comparable performance.

  Implicationpartial

  The abstract states that the smaller Qwen3-30B-A3B-2507 model achieved comparable performance to Llama-3.1-70B.

  Verificationpartial

  partial

• Low-precision quantization (4-bit and 8-bit) preserved predictive performance while reducing GPU memory requirements.technical 90%
  Evidencepartial

  Low-precision quantization (4-bit and 8-bit) preserved predictive performance while reducing GPU memory requirements and improving deployment feasibility.

  Implicationpartial

  The abstract explicitly mentions the benefits of low-precision quantization on performance and memory.

  Verificationpartial

  partial

• Clinically significant errors were identified in 2.9% of the outputs from the clinical evaluation.result 95%
  Evidencepartial

  Clinical evaluation identified clinically significant errors in 2.9% of outputs

  Implicationpartial

  This is a specific quantitative result reported in the abstract regarding clinical evaluation.

  Verificationpartial

  partial

• Semantically equivalent questions occasionally yielded discordant responses, with 0.96% of cases containing a clinically significant error in one formulation.result 95%
  Evidencepartial

  and semantically equivalent questions occasionally yielded discordant responses, including instances where one formulation was correct and the other contained a clinically significant error (0.96% of cases).

  Implicationpartial

  The abstract provides a specific percentage for discordant responses with clinically significant errors.

  Verificationpartial

  partial

• The study evaluated open-source large language models ranging from 4B to 70B parameters under fully offline conditions.method 95%
  Evidencepartial

  Open-source large language models (LLMs) ranging from 4B to 70B parameters were benchmarked under fully offline conditions

  Implicationpartial

  The abstract clearly defines the scope of the models evaluated and the deployment condition.

  Verificationpartial

  partial

• The dataset used for benchmarking consisted of 1,664 expert-annotated question-answer pairs derived from 183 patients, predominantly in Finnish.method 95%
  Evidencepartial

  using 1,664 expert-annotated question-answer pairs derived from records of 183 patients. The dataset consisted predominantly of Finnish clinical text.

  Implicationpartial

  The abstract provides specific details about the dataset size, annotation, and origin.

  Verificationpartial

  partial

• Large models required peak memory allocations exceeding 130 GB, while smaller models operated below approximately 25 GB.technical 90%
  Evidencepartial

  Large models required peak memory allocations exceeding 130 GB, whereas smaller models operated below approximately 25 GB.

  Implicationpartial

  This claim is supported by a direct statement in the text regarding memory consumption differences between model sizes.

  Verificationpartial

  partial

Competitive landscape

A locally deployable framework using open-source LLMs to accurately retrieve patient-specific information from Finnish EHRs, with a low rate of clinically significant errors.