Efficient ML

1. Step 1
   Quantization
   Quantization stores or computes with lower-precision numbers, such as INT8 or 4-bit values, instead of full-precision floats. It reduces memory bandwidth and can speed inference, but accuracy depends on how well the lower-precision representation preserves weights and activations.
2. Step 2
   Post-Training Quantization (PTQ)
   Post-training quantization converts a trained model to lower precision after optimization is finished, usually using a calibration set to estimate activation ranges. It is easy and cheap to apply, but accuracy can drop more than with quantization-aware training at very low bit widths.
3. Step 3
   Quantization-Aware Training (QAT)
   Quantization-aware training simulates low-precision arithmetic during training so the model learns weights that remain accurate after quantization. It usually outperforms post-training quantization at low bit widths, but it adds training cost and implementation complexity.
4. Step 4
   GPTQ Quantization
   A one-shot, layer-wise post-training quantization that pushes LLM weights to 3–4 bits while preserving generation quality. GPTQ reformulates quantization as a per-row error-minimisation problem solved greedily using the inverse Hessian of a small calibration set, achieving 3-bit LLaMA-65B with <1% perplexity loss.
5. Step 5
   AWQ (Activation-aware Weight Quantization)
   AWQ is a post-training quantization method that preserves quality by protecting the weights attached to the largest activation channels before rounding. It targets low-bit LLM inference with small accuracy loss and is popular because it is simpler to deploy than Hessian-based methods such as GPTQ.
6. Step 6
   Low-Rank Adaptation (LoRA)
   LoRA fine-tunes a model by expressing each weight update as a low-rank product \( \Delta W = BA \) while keeping the original weight matrix frozen. This dramatically cuts trainable parameters and optimizer state, which is why LoRA became the default PEFT method for LLMs.
7. Step 7
   LoRA Adapter
   A LoRA adapter is the task-specific pair of low-rank matrices inserted around a frozen base weight matrix to produce a learned update at inference or training time. Because adapters are small, many tasks can be stored, swapped, and merged without copying the full base model.
8. Step 8
   QLoRA (Quantized LoRA)
   QLoRA combines 4-bit quantization of the frozen base model with LoRA adapters trained in higher precision. This makes fine-tuning very large models feasible on modest hardware because the base weights stay compressed while only the small adapter parameters receive gradient updates.

All paths · Wiki home