DevRev Search is a novel passage retrieval benchmark specifically designed for technical customer support, introduced to tackle critical challenges in large-scale multi-tenant retrieval systems. The primary problems it addresses are the 'dark data' issue—a severe lack of curated relevance labels in vast user query logs—and the prohibitive operational cost of model updates, particularly the re-indexing required when jointly fine-tuning query and document encoders. The benchmark is constructed via a fully automatic pipeline, employing a fusion-based candidate generation strategy that pools results from diverse sparse and dense retrievers. Furthermore, it leverages an LLM-as-a-Judge for rigorous consistency filtering and relevance assignment. DevRev Search also proposes an 'Index-Preserving Adaptation' strategy, where only the query encoder is fine-tuned using Low-Rank Adaptation (LoRA), thereby achieving performance improvements while keeping the document index frozen. This benchmark and its associated strategies are crucial for researchers and ML engineers working on efficient domain adaptation, information retrieval, and cost-effective model deployment in complex multi-tenant environments.
DevRev Search is a new benchmark for AI systems that help with technical customer support. It's designed to overcome common issues like not having enough labeled data and the high cost of updating AI models in systems used by many different companies. It uses smart techniques like AI judges and efficient model tuning to create a useful dataset and adaptation strategy.
Was this definition helpful?