How to finetune LLMs

On this page
To finetune an LLM, you need a set of high-quality examples (we've had
success with as few as 100) in a conversational or prompt/completion format, a base model from Hugging
Face, and a GPU. With QLoRA, you load the base model in 4-bit precision
with bitsandbytes, train small LoRA adapters on top of it with TRL's
SFTTrainer, and then load the result with AutoPeftModelForCausalLM
for inference. This guide walks through each step using a Llama 3
model, from setting up the environment
and creating the dataset to running the training and using the finetuned
model.
Finetuning has become a surprisingly efficient and accessible process. It makes models more accurate and relevant for specific tasks, and a finetuned open model can beat many closed-source models on use-case specific performance.
Update (November 2026): The code in this post is written for the pinned
versions below (TRL 0.8.6, huggingface_hub 0.22.2). Current TRL releases moved
max_seq_length (now max_length) and packing into SFTConfig, renamed
the tokenizer argument of SFTTrainer to processing_class, and replaced
setup_chat_format with clone_chat_template; see the SFTTrainer
docs. Current huggingface_hub
releases also replaced the huggingface-cli command with hf, so you log in
with hf auth login (CLI guide).
What is finetuning and why do we want to finetune LLMs?
Model finetuning is a process in machine learning where a pre-trained model is trained further on a specific dataset or task to improve its performance in a particular domain. For Large Language Models, finetuning lets us adapt general-purpose models to specialized applications without having to train a new AI model from scratch.
Finetuning adjusts the model's parameters to better fit the new data and task. The idea is to reuse the knowledge and patterns the model learned during its initial training on vast amounts of data, and refine them for a narrower application.
Why finetuning is important
Finetuning can significantly improve a model's performance on specific tasks or domains, often beyond both the original pre-trained model and smaller models trained from scratch on that task.
It is also far cheaper. Training LLMs from scratch requires enormous compute and vast amounts of data, while finetuning gets excellent results with much less of both. Instead of spending months or years building a new model, you can have a task-specific model in days or weeks.
Finetuning adapts general-purpose models to niche domains and specialized tasks. It can also address some limitations of pre-trained models, such as biases or outdated information, by introducing new, carefully curated data. And as new data arrives or requirements change, you can finetune the model again to keep it current.
When to consider finetuning your own AI model
- When you have a specific task or domain that differs from the general use case of the pre-trained model.
- When you have a dataset that represents your specific use case or contains information not present in the original training data.
- When you need to improve the model's performance on particular types of inputs or outputs.
- When you want to adapt the model's style, tone, or domain-specific language.
If you'd rather not run the training yourself, OpenAI also offers hosted finetuning; see our guide on finetuning GPT-4o mini. And if your problem is mostly about instructions, try better prompts first: DSPy optimizes prompts automatically from a small set of examples, without changing the model.
Pretraining a foundation model, then finetuning it on specific data
Setting up your environment for finetuning LLMs
This section covers the hardware and software requirements and the installation, step by step. It sounds like a lot, but it isn't. We only need these tools and accounts:
-
A Hugging Face account to download the base models and (optionally) some training datasets.
-
A handful of Python packages to run the training process. These are
pytorch,transformers,datasets,accelerate,peft,huggingface_hub,bitsandbytes, andtrl. -
Hardware: while it's theoretically possible to finetune LLMs on CPUs, at the time of writing we highly recommend a CUDA-enabled NVIDIA GPU. It speeds up training significantly.
It's possible to use AMD, Intel or even Mac GPUs for model finetuning. However, because of NVIDIA's dominance in AI, most tools are optimized for NVIDIA, so this guide assumes NVIDIA GPUs.
Hardware requirements
The main requirement for finetuning LLMs is the GPU, and the limiting factor is the amount of VRAM. The model itself needs to fit in GPU memory, with some headroom for the adapters and the training process.
If you have a GPU with Ampere architecture or later, like the NVIDIA RTX 4090, RTX 3090 or A10, you can also use Flash Attention. Flash Attention both reduces the memory footprint of training and drastically speeds it up. (In short, it rearranges parts of the computation to make better use of the GPU's architecture. Read more in the Flash Attention repository.)
If you don't have your own GPUs, use one of the many GPU cloud providers. Two we use regularly and can recommend are runpod.io and latitude.sh.
Setting up your Hugging Face account
-
Use the form to create a new account.
-
Once logged in, go to Settings -> Access Tokens.
-
Create a new access token. Note the token, as we'll need it in a minute.
Installing the Python packages
The last preparation step is installing the Python packages. The
following assumes python 3.11.
First, set up a virtual environment, using either venv or anaconda. We regularly use the latter and therefore recommend it.
-
Activate your virtual environment
-
Install the packages:
Note: Pin the library versions. The Python ecosystem is known for introducing incompatibilities in minor version upgrades. Better safe than sorry.
Finally, log in with the Hugging Face CLI:
That's it for the setup.
What did we actually install?
Here is a short introduction to each library:
-
torch: PyTorch is an open-source machine learning library with a flexible and efficient framework for building and training all sorts of machine learning and deep learning models. -
tensorboard: TensorBoard is a visualization toolkit for machine learning experiments. It tracks and visualizes metrics such as loss and accuracy, model architecture, and more. -
transformers: The Hugging Face Transformers library provides a wide range of pre-trained models for natural language processing tasks, as well as tools for finetuning and deploying these models. -
datasets: The Hugging Face Datasets library provides a collection of datasets for natural language processing tasks, plus tools for loading, processing and working with them. -
accelerate: The Hugging Face Accelerate library provides tools for distributed training and mixed-precision training of deep learning models. -
bitsandbytes: The bitsandbytes library provides CUDA optimizers and quantization. -
huggingface_hub: The Hugging Face Hub library provides tools for sharing, discovering and using models and datasets from the Hugging Face Hub. -
trl: The Hugging Face TRL library (Transformer Reinforcement Learning) provides tools for training language models with supervised finetuning and reinforcement learning methods. ItsSFTTraineris a convenient interface for finetuning. -
peft: The Hugging Face PEFT library (Parameter-Efficient Finetuning) adapts large pretrained models to downstream applications without finetuning all of the model's parameters.
How to create a dataset for finetuning Large Language Models?
The main question beginners have is: what data do I need to finetune my model, and how do I prepare it?
There is no general answer to exactly what data you need, let alone how much. But some guidelines apply:
-
Use domain-specific data. If you finetune a model for a specific domain, gather data from that domain: legal documents for legal text, medical records for medical text.
-
Use task-specific data. If you finetune a model for sentiment analysis, use a dataset with examples of the sentiments you want to detect.
-
Include diverse examples. Diversity helps the model generalize to new examples, and this point matters most of all.
-
Put quality before quantity. A smaller dataset of high-quality, diverse examples beats a large dataset with wrong or repeated examples. LLMs fit to specific examples very quickly. That cuts both ways: a model is quickly "destroyed" by wrong examples (it might "remember" them forever), but it also learns fast from good ones. We created successful finetunes with just 100 rows of high-quality data.
How do you get this data?
-
Create it manually. This is still one of the best approaches and gives the highest quality, but it is also the most time-consuming.
-
Use LLMs to generate synthetic finetuning data. Many LLMs are now trained on synthetic data, and LLMs generate quite good datasets when instructed well. This helps when you want to extend a small manually created dataset, or when you want to finetune your model for a field another model is already good at (for example, using GPT-4 to make smaller models better).
Another approach is to include your internal documentation, emails or other text sources in the generation prompt. That way the LLM grounds the dataset in your own language, domain and facts.
-
Use existing, public datasets. There are hundreds of thousands of datasets on the Hugging Face dataset hub. One of them might be a good starting point, especially for general purpose applications like
text-to-sql. There's not much sense in creating your own when established ones exist.
Format of a finetuning dataset
The trl library made formatting finetuning datasets much easier. Before
looking at the formats, recall the goal: we want to teach a model
a specific domain or task so it generates correct answers to our input
question or prompt.
That tells us how to structure the dataset: we need a "question" and an "answer".
trl supports two ways to express this:
-
The conversational format:
-
The instruction format:
Both formats are very close to what LLM APIs expect at inference time.
The conversational format assumes a chatbot-like application, so you create artificial conversations. The instruction format assumes an instruction -> answer type of system, so you create question/answer pairs.
Pick one format and make sure each row of your dataset follows it.
What about test data?
How to split training and test data depends on the use case, but it's best to keep 10-20% of your data for testing and validation. Either assign the rows to training or testing by hand (distribute the data "uniformly") or use a random split. A random split works well for larger datasets. The smaller the dataset, the more you want to create the split by hand.
Save the training and test datasets as separate files. Especially with larger datasets, keeping them only in memory gets annoying, because you might need to restart your Python environment while setting up training.
Hands-on: Execute the finetuning process with QLoRA
One more piece of theory first. Most models are enormous and would need hundreds of GBs of VRAM to finetune. Thanks to a lot of work by the open-source community, there are ways around this. In our experience, they don't noticeably reduce the quality of the finetuned result.
-
QLoRA: QLoRA was introduced in May 2023 and was a major breakthrough for finetuning. Quoting from the paper: "QLoRA is an efficient finetuning approach that reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance. QLoRA introduces a number of innovations to save memory without sacrificing performance: (a) 4-bit NormalFloat (NF4), a new data type that is information theoretically optimal for normally distributed weights (b) double quantization to reduce the average memory footprint by quantizing the quantization constants, and (c) paged optimizers to manage memory spikes."
Hugging Face did a great job of integrating QLoRA into its
transformerslibrary, so we can use it without much hassle, as you'll see in a moment.For more on QLoRA, see the Hugging Face intro.
-
Model quantization with
bitsandbytes: QLoRA reduces memory usage during training, but the model weights are still as big as they are. Withbitsandbytes, we can quantize the model to lower floating point precision.In short, we reduce the model's size by using fewer bits to represent its weights. This reduces the memory footprint and can speed up training. In theory, it can also reduce the model's performance, but in practice that is surprisingly rare.
Floating point formats, from the Hugging Face blog post on 4-bit quantization with bitsandbytes
Both optimizations are explained in detail in this excellent blog post from Hugging Face.
Now to the finetuning itself.
First, import the required modules.
Then load the training dataset. Make sure it's in the jsonl format
discussed above.
Next, load the model with 4-bit quantization, using bitsandbytes:
If you use the conversational format for your training data (which is
most likely), you need to add special tokens the LLM usually doesn't know
yet. These tokens mark the start and end of a conversation and of the
messages from each role (system, user, assistant). trl provides
a convenient function for that:
Next, define the QLoRA configuration.
Check the tips in the line comments. The LoRA config differs between models and even datasets, so you need to experiment. Try different values as indicated in the comments and compare the model results.
The LoRA parameters explained
-
lora_alphasets the scaling of the low-rank matrices, which is how much influence they have in finetuning. In theory, the higher this value, the more the "new" knowledge influences the model. The lower the value, the more the existing knowledge prevails. -
ris the mathematical rank of the low-rank matrices, which sets their size. A higher rank means bigger matrices and longer finetuning, but also more influence of the low-rank adapters on the final model. So this is another parameter where you balance efficiency against model performance. In our experience, a rank of16produces good results for most current-gen models. -
lora_dropout: during finetuning, a random portion of parameters is "dropped out", mainly to prevent overfitting. High values can lead to underfitting, meaning the model won't "remember" what you told it. Low values can reduce the model's ability to generalize over your data (because it "clings" to the training data).
Before running the training, define the training hyperparameters. The code comments cover the simpler parameters, and the section below the code explains the rest.
The Hugging Face TrainingArguments explained
-
num_train_epochs: Total number of training epochs. 3 is good for most use cases. As a rule of thumb, stop training when the training loss stops declining. Check the loss curve in TensorBoard: if the training loss stays roughly constant after 1 or 2 epochs, reduce this parameter. -
per_device_train_batch_size: Batch size per GPU core, meaning how many examples are processed at once. In general, set it as high as your GPU memory allows, because training gets faster. Combine it withgradient_accumulation_steps. -
gradient_accumulation_steps: Number of update steps to accumulate the gradients for before performing back propagation. In other words, after how many batches do you want to back propagate? While it is advised to max out GPU usage as much as possible, a high number of gradient accumulation steps can result in a more pronounced training slowdown. Consider the following example. Let's say the per_device_train_batch_size=4 without gradient accumulation hits the GPU's limit. If you would like to train with batches of size 64, do not set the per_device_train_batch_size to 1 and gradient_accumulation_steps to 64. Instead, keep per_device_train_batch_size=4 and set gradient_accumulation_steps=16. This results in the same effective batch size while making better use of the available GPU resources. (From Hugging Face) -
optim: Which optimizer to use. In our experience,adamw_torchoradamw_torch_fusedwork best. -
lr_scheduler_type: The learning rate scheduler.constantapplies the same learning rate throughout training.cosinedecays the learning rate over the course of training. For QLoRA applications, keepconstant.
Now we can put it all together, instantiate the trainer and run the training:
The training script summarized
Here is the complete Python code for LLM finetuning:
Use the finetuned Large Language Model
With the model trained, we want to use (and, in fact, evaluate) it. Load the model and adapters from the output directory and generate a response:
For our example, the results looked promising.
To recap the process: prepare high-quality rows (as few as 100 can be
enough) in the conversational or instruction format, load the base model in
4-bit with bitsandbytes, train LoRA adapters with TRL's SFTTrainer,
and evaluate the result on your held-out test set before using it.
If you want to deploy this model to production, read our follow-up piece on How to securely self host your own LLM with vLLM and Caddy.
Further reading
- Advanced RAG: Recursive Retrieval using llamaindex
- Use LLMs to extract data from documents
- AI directly from within your Postgres database
- Saving costs with LLM routing: send easy prompts to a smaller model
- Deploying open models with the Vertex AI Model Garden
Interested in building high-quality AI agent systems?
We prepared a comprehensive guide based on cutting-edge research for how to build robust, reliable AI agent systems that actually work in production. This guide covers:
- Understanding the 14 systematic failure modes in multi-agent systems
- Evidence-based best practices for agent design
- Structured communication protocols and verification mechanisms



