Audit LLMs: Detect Model Substitution & Overbilling (No Trust Needed)

Dimensions overall score 8.0

• IMMACULATE can detect economically motivated deviations in LLM API services including model substitution, quantization abuse, and token overbilling.method 95%
  Evidencepartial

  detects economically motivated deviations-such as model substitution, quantization abuse, and token overbilling

  Implicationpartial

  Directly stated in abstract as the core purpose of the framework

  Verificationpartial

  partial

• IMMACULATE achieves detection without requiring trusted hardware or access to model internals.technical 90%
  Evidencepartial

  without trusted hardware or access to model internals

  Implicationpartial

  Explicitly stated in abstract as a key feature of the approach

  Verificationpartial

  partial

• IMMACULATE selectively audits a small fraction of requests using verifiable computation to amortize cryptographic overhead.method 90%
  Evidencepartial

  selectively audits a small fraction of requests using verifiable computation, achieving strong detection guarantees while amortizing cryptographic overhead

  Implicationpartial

  Directly stated in abstract as the core technical approach

  Verificationpartial

  partial

• IMMACULATE reliably distinguishes benign and malicious executions with under 1% throughput overhead.result 85%
  Evidencepartial

  IMMACULATE reliably distinguishes benign and malicious executions with under 1% throughput overhead

  Implicationpartial

  Directly stated in abstract with performance metric, though specific experimental conditions not detailed

  Verificationpartial

  partial

• IMMACULATE has been tested on both dense and MoE (Mixture of Experts) models.result 90%
  Evidencepartial

  Experiments on dense and MoE models show

  Implicationpartial

  Explicitly stated in abstract as part of experimental validation

  Verificationpartial

  partial

• IMMACULATE is a practical auditing framework for commercial LLM API services.method 85%
  Evidencepartial

  IMMACULATE: A Practical LLM Auditing Framework via Verifiable Computation

  Implicationpartial

  Directly stated in title and abstract as the framework's purpose

  Verificationpartial

  partial

• Commercial LLM APIs are typically deployed as black-box services requiring users to trust providers.market 80%
  Evidencepartial

  Commercial large language models are typically deployed as black-box API services, requiring users to trust providers to execute inference correctly and report token usage honestly

  Implicationpartial

  Directly stated in abstract as the problem context, though 'typically' implies some generalization

  Verificationpartial

  partial

Competitive landscape

IMMACULATE provides a practical framework for auditing LLM API services to detect economic abuses like model substitution and token overbilling.