REKD

Gold definitionUpdated Apr 2, 2026

Rationale Extraction with Knowledge Distillation (REKD) is a novel approach designed to improve the performance of Rationale Extraction (RE) models, particularly when these models are based on less capable or smaller neural networks. Rationale Extraction itself is an interpretable-by-design framework for deep neural networks, employing a select-predict architecture where two networks jointly learn feature selection (to identify "rationales") and prediction. However, learning to select optimal feature subsets with only remote supervision is computationally challenging, especially for smaller models. REKD addresses this by integrating knowledge distillation: a student RE model learns not only through its own optimization but also by mimicking the rationales and predictions of a more powerful "teacher" or "rationalist" model. This mechanism allows the student to leverage the teacher's expertise, thereby boosting its predictive performance and making interpretable AI more accessible for diverse applications.

Understanding REKD: Rationale Extraction with Knowledge Distillation

Rationale Extraction (RE): Rationale Extraction provides an interpretable-by-design framework for deep neural networks, utilizing a select-predict architecture. In this setup, two neural networks are jointly trained to perform feature selection, identifying subsets of features or "rationales," and subsequent prediction.
Challenges in Rationale Extraction: Learning to select optimal feature subsets (rationales) is computationally challenging, especially when relying solely on remote supervision from the final task prediction. This difficulty is exacerbated when the base neural networks are not sufficiently capable or are smaller in size.

The Mechanism of REKD

Teacher-Student Learning in REKD

At a glance

Executive summary

REKD improves how smaller AI models explain their decisions by having them learn from a more expert AI model. It combines "rationale extraction," which helps AI show *why* it made a decision, with "knowledge distillation," where a small model learns from a big one, leading to better performance for the smaller, explainable AI.

TL;DR

REKD helps small AI models better explain their decisions by learning from a big, expert AI model's explanations and predictions.

Key points

A student Rationale Extraction (RE) model learns from a teacher's rationales and predictions via knowledge distillation, in addition to its own RE optimization.
Improves the predictive performance of less capable or smaller Rationale Extraction models, addressing computational challenges in learning rationales.
Used by researchers and engineers working on explainable AI (XAI), model interpretability, and efficient deployment of interpretable deep learning models.
Unlike standard Rationale Extraction which can struggle with smaller networks, REKD leverages a teacher to boost student performance and interpretability.
Combines model interpretability (Rationale Extraction) with model efficiency and performance enhancement techniques (Knowledge Distillation).

Use cases

Deploying interpretable AI in high-stakes domains (e.g., medical diagnosis, financial fraud detection) using smaller, more efficient models.
Enabling explainable AI on edge devices or mobile platforms where computational resources are limited, by improving the performance of compact RE models.
Developing more robust and accurate "select-predict" architectures for feature selection and interpretation in scenarios with complex data.
Improving the training efficiency of Rationale Extraction models by providing a strong supervisory signal from a pre-trained teacher.

Also known as

Rationale Extraction with Knowledge Distillation