High-Fidelity Pruning for Large Language Models

Dimensions overall score 8.0

GitHub Code Pulse

No public code linked for this paper yet.

• Common pruning approaches using Taylor expansion with one-hot cross entropy loss have a key limitation of narrowly assessing neuron importance based only on the probability assigned to the single predicted next token.limitation 90%
  Evidencepartial

  its reliance on one-hot cross entropy loss, a key limitation is that it narrowly assesses importance based only on the probability assigned to the single predicted next token

  Implicationpartial

  Directly and explicitly stated in the abstract with clear technical explanation

  Verificationpartial

  partial

• Using self-distillation for importance evaluation introduces significant computational overhead by requiring a separate teacher model for supervision.limitation 85%
  Evidencepartial

  this approach introduces significant computational overhead by requiring a separate teacher model for supervision

  Implicationpartial

  Directly stated in the abstract as a limitation of an alternative approach

  Verificationpartial

  partial

• The proposed method uses information entropy of the model's output distribution to efficiently evaluate importance scores of neurons with Taylor pruning without requiring an additional teacher model.method 90%
  Evidencepartial

  we propose a simple but effective criterion, information entropy of the model's output distribution, to efficiently evaluate importance scores of neurons with Taylor pruning without requirement of additional teacher

  Implicationpartial

  Directly and explicitly stated as the core method contribution in the abstract

  Verificationpartial

  partial

• Compared to plain cross entropy criterion, the information entropy approach provides a more holistic criterion for Taylor pruning to prune neurons with the least impact on the model's predictions in a global manner.technical 85%
  Evidencepartial

  Compared to plain cross entropy criterion, it provides a more holistic criterion for Taylor pruning to prune neurons with the least impact on the prediction of model in a global manner

  Implicationpartial

  Directly stated in the abstract as a comparative advantage of the proposed method

  Verificationpartial

  partial

• The proposed method preserves the fidelity of the model's predictive capabilities better than existing approaches.result 80%
  Evidencepartial

  thereby preserving the fidelity of the model's predictive capabilities

  Implicationpartial

  Directly stated in the abstract as a benefit of the method, though specific fidelity metrics are not provided

  Verificationpartial

  partial

• Experimental results on extensive zero-shot benchmarks demonstrate that the proposed method consistently outperforms existing pruning methods across the LLaMA and Qwen series models.result 90%
  Evidencepartial

  Experimental results on extensive zero-shot benchmarks demonstrate that our method consistently outperforms existing pruning methods across the LLaMA and Qwen series models

  Implicationpartial

  Directly stated in the abstract with specific model families mentioned and claims of consistent outperformance

  Verificationpartial

  partial

• The source code and trained weights for the proposed method are publicly available.technical 95%
  Evidencepartial

  The source code and trained weights are availabel at https://github.com/visresearch/HFPrune

  Implicationpartial

  Explicitly stated with a specific GitHub URL provided in the abstract

  Verificationpartial

  partial

Startup potential card