Cognitively Layered Data Synthesis for Domain Adaptation of LLMs to Space Situational Awareness

Dimensions overall score 8.0

• Using BD-FDG, a domain dataset named SSA-SFT was constructed, containing approximately 230,000 samples, and used to fine-tune Qwen3-8B into SSA-LLM-8B.result 70%
  Evidencepartial

  Using BD-FDG, we construct SSA-SFT, a domain dataset of approximately 230K samples, and fine-tune Qwen3-8B to obtain SSA-LLM-8B.

  Implicationpartial

  The abstract directly states the dataset size and the resulting fine-tuned model.

  Verificationpartial

  partial

• BD-FDG is a framework designed to address challenges in creating high-quality supervised fine-tuning (SFT) datasets for domain adaptation of LLMs, specifically targeting incomplete knowledge coverage, shallow cognitive depth, and limited quality controllability.method 70%
  Evidencepartial

  To this end, we propose BD-FDG (Bloom's Taxonomy-based Domain-specific Fine-tuning Data Generation), a framework that addresses incomplete knowledge coverage, shallow cognitive depth, and limited quality controllability through three mechanisms: structured knowledge organization, cognitively layered question modeling, and automated quality control.

  Implicationpartial

  The abstract explicitly states the purpose and mechanisms of the BD-FDG framework in addressing these specific challenges.

  Verificationpartial

  partial

• The BD-FDG framework utilizes a knowledge tree for structured corpus coverage and a question generation scheme spanning nine categories and six cognitive levels (Remember to Create) to produce samples with a continuous difficulty gradient.method 70%
  Evidencepartial

  The framework uses a knowledge tree to ensure structured corpus coverage, designs a question generation scheme spanning nine categories and six cognitive levels from Remember to Create to produce samples with a continuous difficulty gradient...

  Implicationpartial

  The abstract clearly describes the components and their functions within the BD-FDG framework.

  Verificationpartial

  partial

• The fine-tuned model, SSA-LLM-8B, achieved relative BLEU-1 improvements of 144% (no-think) and 176% (think) on the domain test set compared to a baseline.result 70%
  Evidencepartial

  Experiments show that SSA-LLM-8B achieves relative BLEU-1 improvements of 144% (no-think) and 176% (think) on the domain test set...

  Implicationpartial

  The abstract provides specific quantitative results for the performance improvement of the fine-tuned model.

  Verificationpartial

  partial

• SSA-LLM-8B demonstrated a win rate of 82.21% over the baseline in arena comparisons.result 70%
  Evidencepartial

  ...and a win rate of 82.21% over the baseline in arena comparisons...

  Implicationpartial

  The abstract provides a specific win rate percentage from arena comparisons.

  Verificationpartial

  partial

• SSA-LLM-8B largely preserved general benchmark performance on MMLU-Pro and MATH-500.result 70%
  Evidencepartial

  ...while largely preserving general benchmark performance (MMLU-Pro, MATH-500).

  Implicationpartial

  The abstract explicitly mentions the preservation of general benchmark performance.

  Verificationpartial

  partial

• SFT data construction driven by cognitive layering is an effective paradigm for complex engineering domain adaptation of LLMs.result 70%
  Evidencepartial

  These results validate SFT data construction driven by cognitive layering as an effective paradigm for complex engineering domains...

  Implicationpartial

  This is a concluding statement in the abstract that summarizes the main finding and its broader applicability.

  Verificationpartial

  partial

Competitive landscape

A framework for generating high-quality fine-tuning datasets for LLMs in space situational awareness.