BitNet: LLM Quantization at its Extreme

Sascha Gstir
Sascha Gstir · Co-Founder, Pondhouse Data
Published Updated 10 min read
BitNet: LLM Quantization at its Extreme
On this page

BitNet is Microsoft's answer to the growing resource demands of large language models (LLMs). Instead of compressing a trained model afterwards, BitNet trains the model with 1-bit weights from the start. Its successor, BitNet b1.58, uses ternary weights (-1, 0 and 1) and, according to the paper, reaches almost the same benchmark results as a reproduced FP16 LLaMA model. At 70B parameters it needs 7.16 times less memory and 41.2 times less end-to-end energy.

Why does this matter? Since 2022, model developers have competed on output quality, benchmark scores and context length, and the most reliable lever has been size. Scaling up means an exponential increase in both the number of parameters and the amount of training data.

GPT-3 (2020) was trained on 300 billion tokens, drawn from a dataset of about 0.5 trillion tokens, for 175 billion parameters. GPT-4 is estimated at around 13 trillion tokens for an MoE model with (rumored) 16x111 billion parameters; OpenAI does not publish official figures. Models with known figures show the same trend: Llama2 with 2 trillion tokens and up to 70 billion parameters, PaLM 2 with 3.6 trillion tokens and 340 billion parameters, Llama 3 with 15 trillion tokens and up to 405 billion parameters (the 405B model was released as Llama 3.1). Storage, memory and compute requirements have grown with them.

To reduce these demands, model weights (parameters) are often quantized, meaning their resolution is reduced. This usually happens after training, in what is called post-training quantization. It costs a little accuracy, but it is simple and shrinks the model proportionally. Most models people run locally today are post-training quants, like the Q4 GGUF in our guide to running GLM-4.7-Flash on a 16GB GPU.

Microsoft took a different route and made the model efficient by design. Rather than quantizing after training, their study reduced the resolution of the parameters to the absolute minimum from the beginning, implementing the feed-forward networks with binary (1-bit!) parameters. In this post we explain how this works and how the model compares to a reproduced Llama model with the same parameter count but higher weight resolution.

The drawbacks of current quantization methods*

There are two common methods to quantize an LLM: Quantization-Aware Training (QAT) and Post-Training Quantization (PTQ). The quotes below are from "A Survey of Quantization Methods for Efficient Neural Network Inference" by Gholami et al.

Comparison of quantization-aware training and post-training quantization

Quantization-aware training (QAT)

"Given a trained model, quantization may introduce a perturbation to the trained model parameters, and this can push the model away from the point to which it had converged when it was trained with floating point precision. It is possible to address this by re-training the NN model with quantized parameters so that the model can converge to a point with better loss. One popular approach is to use Quantization-Aware Training (QAT), in which the usual forward and backward pass are performed on the quantized model in floating point, but the model parameters are quantized after each gradient update (similar to projected gradient descent). In particular, it is important to do this projection after the weight update is performed in floating point precision. Performing the backward pass with floating point is important, as accumulating the gradients in quantized precision can result in zero-gradient or gradients that have high error, especially in low-precision"

"QAT has been shown to work despite the coarse approximation of STE. However, the main disadvantage of QAT is the computational cost of re-training the NN model. This re-training may need to be performed for several hundred epochs to recover accuracy, especially for low-bit precision quantization. If a quantized model is going to be deployed for an extended period, and if efficiency and accuracy are especially important, then this investment in re-training is likely to be worth it. However, this is not always the case, as some models have a relatively short lifetime."

Quantization-aware training using the straight-through estimator (STE)

Post-training quantization (PTQ)

"An alternative to the expensive QAT method is Post-Training Quantization (PTQ) which performs the quantization and the adjustments of the weights, without any fine-tuning. As such, the overhead of PTQ is very low and often negligible. Unlike QAT, which requires a sufficient amount of training data for retraining, PTQ has an additional advantage that it can be applied in situations where data is limited or unlabeled. However, this often comes at the cost of lower accuracy as compared to QAT, especially for low-precision quantization."

"In PTQ, all the weights and activations quantization parameters are determined without any re-training of the NN model. As such, PTQ is a very fast method for quantizing NN models. However, this often comes at the cost of lower accuracy as compared to QAT."

Both methods have real drawbacks. QAT is expensive because it needs additional fine-tuning after pre-training. PTQ is simple, but it lowers the model's accuracy by an uncertain amount.

BitNet's approach: train with 1-bit weights from the start

Microsoft avoided both drawbacks with a different approach that makes transformer-based LLMs much more efficient, especially at inference time. The core idea is to skip the separate quantization step by training the model with highly quantized parameters in the first place. They chose the most extreme option: parameters with a resolution of just 1 bit. The model, named "BitNet", was introduced in the paper "BitNet: Scaling 1-bit Transformers for Large Language Models" in October 2023.

How BitNet works

BitNet follows the same structure as other transformers, with stacked blocks of self-attention and feed-forward networks. Unlike a standard transformer, BitNet uses the newly introduced BitLinear blocks instead of conventional matrix multiplications. BitLinear works with binarized (1-bit) model weights.

With the floating-point parameters reduced to the values 1 and -1, matrix multiplications (combinations of multiplications and additions) become plain integer additions. That lowers both compute and energy demands considerably.

Not every part of the BitNet transformer is reduced to 1 bit, though. The following components stay in high precision (8-bit):

  1. The residual connections and layer normalizations were not changed, as they contribute negligibly to the computational costs in LLMs.
  2. The attention heads were also not reduced in resolution, since the QKV transformation is significantly smaller than the parametric projection.
  3. The precision of the input/output embeddings was not reduced, as language models require high-precision probabilities for effective sampling.

BitNet architecture with BitLinear blocks

How BitNet is trained

Straight-through estimator (STE)

The forward path of BitNet contains non-differentiable functions (Sign and Clip). To train the model anyway, the authors used the Straight-Through Estimator (STE), which bypasses these functions during backpropagation.

Mixed precision training

The heavily reduced parameter resolution required a few additional measures during training.

For stability and accuracy, gradients and optimizer states were kept in high precision. The weights were also stored in high precision, so parameter updates could accumulate at high resolution. In the forward pass, these weights were binarized on the fly to measure the effect of the 1-bit parameters.

Large learning rate

With such low resolution, small update steps often don't change the weights at all, which makes optimization harder. The authors found that the simplest and most effective fix is a much higher learning rate. BitNet needs it to converge faster, while the same high learning rates make an FP16 Transformer diverge during training.

BitNet training stability at different learning rates

BitNet compared with an FP16 Transformer

To measure the performance and efficiency gains, the authors compared BitNet with a Llama-based FP16 Transformer trained on the same data, with the same parameter counts, tokenizer and vocabulary size.

The results show that BitNet's accuracy also follows a power law, so it can be predicted as the model grows. Scaling up BitNet brings its loss close to that of the FP16 Transformer.

Since BitNet was designed to make inference much more efficient, the authors also plotted loss against inference energy consumption. BitNet reaches the same loss and accuracy levels as the FP16 Transformer with much lower energy consumption.

BitNet scaling curves

BitNet accuracy vs. energy consumption of zero-shot and few-shot inference

BitNet compared with post-training quantization methods

The authors also compared BitNet with state-of-the-art post-training quantization methods at different quantization levels: weight-only quantization (GPTQ and QuIP) and weight-and-activation quantization (Absmax and SmoothQuant).

"The results demonstrate the effectiveness of BitNet in achieving competitive performance levels compared to the baseline approaches, particularly for lower bit levels. The zero-shot scores of BitNet are comparable with the 8-bit models, while the inference cost is much lower. For the 4-bit models, the weight-only quantization methods outperform the weight-and-activation quantizers, mainly because the activation is more difficult to quantify. BitNet, as a 1-bit model, significantly achieves better results than both the weight-and-activation quantization methods and the weight-only methods. As for the lower-bit models, BitNet has consistently superior scores over all baselines. This proves the advantages of the quantization-aware training approaches over the post-training quantization methods."

BitNet benchmarks compared to PTQ methods at various quantization levels

"Figure 6 summarizes both the zero-shot accuracy and few-shot accuracy of our method and the baselines while scaling up the model size from 1.3B to 6.7B. It proves that the advantage is consistent across different scales."

BitNet zero-shot and few-shot accuracy compared to common PTQ methods

BitNet b1.58: ternary weights for more efficient LLMs

In the next iteration, Microsoft extended the BitNet concept to ternary weights with the possible values {-1, 0, 1}. Three states correspond to 1.58 bits, which is why the model is called 'BitNet b1.58'.

Adding 0 to the possible weights explicitly supports feature filtering, which significantly improves the modeling capability of 1-bit LLMs. BitNet b1.58 keeps the BitNet architecture but needs some adjustments to the quantization function because of the ternary weights.

The results are remarkable. With the usual 16-bit floating-point values reduced to these three states, the matrix calculations need no complex multiplications, only integer additions of the input values.

'BitNet b1.58' Pareto improvement

The study compared BitNet b1.58 with a reproduced FP16 LLaMA LLM. Despite the drastic reduction in weight resolution, BitNet b1.58 achieved almost identical results to the LLaMA model across various benchmarks.

'BitNet b1.58' vs FP16 LLaMA

BitNet b1.58 only applies this quantization in the FFNs, not in other parts of the Transformer, so the attention layers and some other components still compute at higher resolution. That's why the model size doesn't shrink in direct proportion to the weight resolution. The effect grows with the parameter count:

  • For a 3B parameter model, the required memory drops by a factor of 3.55.
  • For a 70B model, the memory requirement drops by a factor of 7.16, and the model is 4.1 times faster.
  • The 70B model reaches a throughput of 2977 tokens/s, an 8.9-fold increase.

Less computation means a smaller memory footprint and higher throughput, and it also cuts the energy required. According to the study, the 70B BitNet b1.58 model reduces end-to-end energy consumption by 41.2 times compared to the LLaMA 70B model. (For why energy use matters so much in GenAI, see our post GenAI: Technological Masterpiece or Ecological Disaster?)

'BitNet b1.58' energy consumption 'BitNet b1.58' resource reduction and speed improvements

The savings are large enough that, in terms of resources, BitNet models compete with LLaMA models that have far fewer parameters. The paper states:

  • 13B BitNet b1.58 is more efficient, in terms of latency, memory usage and energy consumption, than 3B FP16 LLM.
  • 30B BitNet b1.58 is more efficient, in terms of latency, memory usage and energy consumption, than 7B FP16 LLM.
  • 70B BitNet b1.58 is more efficient, in terms of latency, memory usage and energy consumption, than 13B FP16 LLM.

Conclusion

The race for better benchmark scores and output quality has made models bigger, and with them the resources needed for training and inference. That means higher costs, lower throughput and more energy consumption. Quantization reduces model size and resource demand, but the usual methods have a price: Quantization-Aware Training (QAT) complicates the training process, and the simpler Post-Training Quantization (PTQ) can degrade quality.

With BitNet, Microsoft showed that these resource savings are possible without extra post-training work and without losing performance, as long as the quantization is part of the model's structure from the start.

BitNet b1.58 reaches the quality level of a comparable FP16 LLM while needing less energy and memory, with higher throughput and lower latency. The approach also scales well: the benefits grow with model size.

This opens up new use cases, such as running LLMs on edge or mobile devices. Combined with other techniques like Mixture-of-Experts (MoE) or hardware built for 1-bit LLMs, these models could push LLM performance further. If you want to cut costs with the models you run today, our post on strategies to reduce LLM costs covers quantization alongside ten other strategies.

Further reading

Interested in building high-quality AI agent systems?

We prepared a comprehensive guide based on cutting-edge research for how to build robust, reliable AI agent systems that actually work in production. This guide covers:

  • Understanding the 14 systematic failure modes in multi-agent systems
  • Evidence-based best practices for agent design
  • Structured communication protocols and verification mechanisms
Sascha Gstir

About the author

Sascha Gstir

Co-Founder, Pondhouse Data

Sascha worked for almost a decade in data analytics and data science. At Pondhouse Data he builds and tunes AI solutions, from model selection to production.