Skip to content
dhii.ai

fine-tuning

LoRA vs QLoRA vs Full Fine-Tuning: Choosing the Right Method

How LoRA, QLoRA and full fine-tuning work, what they cost in memory and compute, how quality compares, and a decision guide for fine-tuning open-weight LLMs.

By Amit Valecha · · 6 min read

If you’ve decided that fine-tuning an open-weight language model is the right move — rather than prompting or retrieval — the next question is how. The three methods you’ll hear about most are full fine-tuning, LoRA and QLoRA. They differ in memory requirements, cost, speed, flexibility and, in some cases, quality. This article explains each in practical terms and offers a decision guide.

A quick refresher: what fine-tuning changes

A language model is a very large set of numerical weights. Pre-training sets those weights by learning from vast amounts of text. Fine-tuning continues training on a smaller, targeted dataset — examples of the inputs your application receives and the outputs you want — so the model’s behaviour shifts toward your task.

The methods below differ in which weights are updated and how precisely the base model is stored during training.

Full fine-tuning

In full fine-tuning, every weight in the model is updated. This gives the most flexibility: the model can change its behaviour in any way the data demands.

Memory. Training needs memory not only for the weights but also for gradients and optimiser states, which together can require several times the memory of the model itself. For models in the billions of parameters, that means multiple high-memory GPUs, often with techniques such as sharding across devices.

Output. A complete new copy of the model. If you fine-tune for five tasks, you have five full-size models to store and serve.

Risks. With small datasets, full fine-tuning can overfit or cause “catastrophic forgetting”, where the model loses general abilities it had before.

When it makes sense. Large, high-quality datasets; substantial domain shifts (for example, adapting to a new language or a highly specialised field); or teams with the compute budget and the need to squeeze out the last bit of quality.

LoRA (Low-Rank Adaptation)

LoRA freezes the original model weights and adds small trainable matrices — “adapters” — to selected layers. Instead of updating a huge weight matrix directly, training learns a low-rank update to it, expressed as the product of two much smaller matrices. Only these adapter weights are trained.

Memory. Because the base weights are frozen, there are no gradients or optimiser states for them. The trainable parameter count is typically a small fraction of a percent to a few percent of the model. Memory use drops dramatically, though the full-precision base model still needs to fit in GPU memory.

Output. A small adapter file, often megabytes to a few hundred megabytes, that is applied on top of the base model. You can keep many adapters for different tasks and swap them, or merge one into the base weights for deployment. Some serving frameworks can serve multiple LoRA adapters on one base model simultaneously, which is efficient for multi-tenant or multi-task setups.

Quality. For most instruction-following, classification, extraction and style tasks, LoRA achieves quality close to full fine-tuning. The gap tends to grow when the task requires the model to absorb large amounts of new knowledge or shift substantially away from its pre-training distribution.

Key settings. The rank (size of the adapter matrices), alpha (scaling), which layers receive adapters (attention projections only, or also MLP layers), learning rate and number of epochs. Applying adapters to more layers generally improves quality at modest extra cost.

QLoRA (Quantised LoRA)

QLoRA combines LoRA with quantisation of the frozen base model. The base weights are loaded in 4-bit precision (using a data type designed for normally distributed weights), while the LoRA adapters are trained in higher precision. Additional tricks, such as paged optimisers, help manage memory spikes.

Memory. Much lower again. Models that would need a multi-GPU setup for LoRA in 16-bit precision can often be fine-tuned on a single GPU with QLoRA. This is what made fine-tuning mid-sized models accessible on a single consumer or workstation GPU.

Speed. Training can be somewhat slower per step than 16-bit LoRA because of the de-quantisation overhead, though optimised libraries reduce this.

Quality. In many practical cases, QLoRA’s results are close to 16-bit LoRA. For quality-critical work, it’s worth running a comparison on your own evaluation set when hardware allows.

Deployment note. After training, you can merge adapters into a higher-precision copy of the base model and then quantise for serving with whatever method suits your inference stack. Be careful to evaluate the final deployed artefact, not just the training checkpoint, because quantisation at serving time can affect quality.

Side-by-side

Full fine-tuning LoRA QLoRA
Weights trained All Small adapters Small adapters
Base model precision in training 16/32-bit 16-bit typically 4-bit
GPU memory Very high Moderate Low
Output size Full model Small adapter Small adapter
Multiple tasks Multiple full models Swap adapters Swap adapters
Typical quality Highest ceiling Close to full for most tasks Close to LoRA for most tasks
Risk of forgetting Higher Lower Lower

A decision guide

Start with QLoRA when you’re exploring: it’s the cheapest way to test whether fine-tuning helps, compare base models and tune hyperparameters. Many projects never need to go further.

Use LoRA in 16-bit when you have the GPU memory and want to remove quantisation as a variable, or when QLoRA results are just short of your quality bar.

Consider full fine-tuning when you have a large, high-quality dataset, the task represents a big shift from the base model’s abilities, and parameter-efficient methods have plateaued on your evaluation set.

Consider not fine-tuning if a stronger prompt, a few examples or retrieval gets you there. Fine-tuning creates an asset you must maintain and re-train as base models improve.

Practical tips that matter more than the method

  • Data quality dominates. A thousand consistent, correct examples beat ten thousand noisy ones. Invest in guidelines and review.
  • Hold out a test set that the model never sees in training, built from real cases.
  • Compare against baselines: the base model with a good prompt, and a frontier API. Fine-tuning must beat them on your metric to justify itself.
  • Watch for regressions on general abilities and safety behaviour, not just your target task.
  • Track every run with its data version and configuration so results are reproducible.
  • Check licences. Open-weight models come with different licence terms; some restrict certain uses or require attribution.

Tooling

The Hugging Face ecosystem — Transformers, PEFT and TRL — supports all three approaches. Unsloth offers memory- and speed-optimised LoRA/QLoRA training. Axolotl provides configuration-driven training recipes. For serving, vLLM and similar engines support LoRA adapters, and llama.cpp handles quantised models efficiently on smaller hardware.

Common mistakes we see

A few errors recur across fine-tuning projects regardless of method. Training with a chat template that differs from the one used at inference time quietly degrades results. Learning rates copied from a full fine-tuning recipe are often wrong for adapters, and vice versa. Training for too many epochs on a small dataset produces a model that memorises examples and generalises poorly — watch validation loss and, more importantly, task metrics on held-out data. And teams frequently evaluate the adapter in 16-bit during development, then deploy a 4-bit quantised merge without re-testing. Each of these is easy to avoid once you know to look for it.

Summary

LoRA and QLoRA have made fine-tuning practical for most organisations: smaller hardware, faster experiments and small, swappable adapters, with quality close to full fine-tuning for typical business tasks. Full fine-tuning remains the tool for large-scale adaptation. In every case, the method matters less than the data and the evaluation.

Our LLM fine-tuning service starts with a baseline and feasibility check, and our fine-tuning cost guide breaks down where the budget actually goes.

Services related to this article

Keep reading

Start a project

Have a workflow that should run itself?

Tell us what you want automated or built. You get a written assessment of feasibility, risks and a fixed quote — usually within two business days.