linear probe classifiers

A linear probe classifier is a diagnostic tool in machine learning, specifically for evaluating the quality of learned representations from pre-trained models. It consists of a simple linear layer (e.g., a logistic regression or a single fully connected layer) trained on top of frozen features extracted from a deeper neural network. The key aspect is that the weights of the pre-trained feature extractor are kept fixed, and only the linear layer's weights are optimized. The process involves taking a pre-trained model (e.g., a vision transformer or a large language model), passing input data through it to obtain intermediate feature vectors (embeddings) from a specific layer, and then training a linear classifier on these fixed feature vectors to perform a downstream task (e.g., image classification, sentiment analysis). The performance of this linear classifier serves as a proxy for the quality and task-relevance of the underlying representations. It matters because it provides a computationally efficient and interpretable way to assess the information encoded within a model's representations without the complexity and cost of full fine-tuning. This helps researchers understand what knowledge a pre-trained model has implicitly acquired, identify useful layers for transfer learning, and compare different self-supervised learning methods. It is widely used in representation learning research, self-supervised learning, transfer learning, and interpretability studies across computer vision (e.g., evaluating CLIP, DINO features) and natural language processing (e.g., evaluating BERT, GPT embeddings).