KAN: Is this the end of feedforward networks?

Sascha Gstir
Sascha Gstir · Co-Founder, Pondhouse Data
Published Updated 9 min read
KAN: Is this the end of feedforward networks?
On this page

Kolmogorov-Arnold Networks (KANs) are unlikely to replace feedforward networks anytime soon, mainly because they train much more slowly than Multi-Layer Perceptrons (MLPs). But in the 2024 paper KAN: Kolmogorov-Arnold Networks by Ziming Liu, Max Tegmark and colleagues, small KANs reached higher accuracy than much larger MLPs and were far easier to interpret. For solving partial differential equations, a 2-layer KAN of width 10 was 100 times more accurate than a 4-layer MLP of width 100, with 100 times fewer parameters.

MLPs, also known as fully-connected feedforward networks, are the backbone of the most powerful AI models available today. They are used in classic machine learning tasks (feature extraction, regression, classification) and inside advanced deep learning models such as Convolutional Neural Networks (CNNs) and Transformers.

KANs are a rediscovered type of network. They use spline functions instead of static weights to reach high accuracy and interpretability at very small network sizes. This post explains how the two architectures differ, what the paper shows about grid extension and interpretability, and where KANs still fall short.

Difference between MLP and KAN

Comparison of MLP and KAN: theorems, formulas and network structure

Multi-Layer Perceptrons (MLPs)

A Multi-Layer Perceptron is an artificial neural network with an input layer, several hidden layers and an output layer. Each layer consists of nodes, usually called "neurons", which are connected to the nodes in the layers directly before and after them.

Data flows from the input layer through the hidden layers to the output layer. Each node first multiplies its input values by learned weights (determined during training), then sums these products and passes the result through a fixed activation function. The activation function introduces non-linearity into the system, which the network needs to deal with complex, non-linear problems.

This way, MLPs can model complex relationships between input features and outputs. The ability rests on the Universal Approximation Theorem: a neural network with at least one hidden layer can approximate any continuous function, given enough neurons and the right set of weights. That is why MLPs work well in fields like image recognition, speech processing and predictive analytics, where they find patterns in data that are not immediately obvious.

Kolmogorov-Arnold Networks (KANs)

Where MLPs are based on the Universal Approximation Theorem, KANs are founded on the Kolmogorov-Arnold Representation Theorem, also known as the Superposition Theorem, developed by Andrey Kolmogorov and Vladimir Arnold. Kolmogorov's work on representing continuous functions by superpositions of functions with fewer variables was published in 1956. Arnold's closely related work on functions of three variables followed in 1957. Together they established that any continuous multivariate function can be represented as a superposition of continuous univariate functions and addition.

Like MLPs, KANs consist of an input layer, an output layer and one or more hidden layers in between. But in a KAN, the "nodes" only sum the incoming "signals", without a fixed activation function, and the "edges" use univariate functions instead of static weights. These univariate functions are learned during training, much like the weights in MLPs. In the paper's experiments, the functions are B-splines, and instead of weights, the coefficients of the B-spline basis functions are learned. The grid of knots is not trained by gradient descent: the paper updates it from the input activations and can make it finer later (grid extension). B-splines have several advantages for this:

  • Splines struggle with high-dimensional functions (the curse of dimensionality, COD) because they can't exploit compositional structures. For univariate functions, which is all a KAN edge needs, splines are very accurate.
  • The resolution of the functions, and with it the accuracy of the model, is easy to adjust by increasing or decreasing the grid size.
  • B-spline basis functions are local, so each coefficient changes only a small part of the curve. This helps the learning process, particularly in continuous learning.

To learn a function accurately, which is the goal of any model, a model must capture the compositional structure (external degrees of freedom) and approximate the univariate functions well (internal degrees of freedom). KANs do both by combining MLPs and splines: an MLP-like structure on the outside, splines on the inside. The MLP-like structure lets KANs learn features, and the splines let them optimize these learned features with high accuracy.

The paper reports four main advantages of KANs.

The first is accuracy. Because every edge has its own learned function, KANs can model the influence of individual features very well and reach high accuracy even with small networks.

The second is efficiency, at least in parameters. In the paper's experiments, KANs needed significantly longer to train than MLPs with the same number of parameters. They still count as much more efficient, because their accuracy allows a disproportionate reduction in network size and parameter count for the same result. The authors state that "for PDE solving, a 2-Layer width-10 KAN is 100 times more accurate than a 4-Layer width-100 MLP (10^−7 vs 10^−5 MSE) and 100 times more parameter efficient (10^2 vs 10^4 parameters)."

Third, KANs are interpretable. The functions on the edges make the influence of individual features clear and easy to interpret. In an MLP, these relationships are spread over large parts of the network and many nodes and edges, which makes it much harder to interpret. This is why KANs could also be useful in scientific fields like physics and mathematics.

Finally, thanks to the local independence of the B-spline functions, KANs appear to support continuous learning. Training one part of the network does not cause "catastrophic forgetting" in the parts already learned, unlike in MLPs.

Basing neural networks on the Kolmogorov-Arnold representation theorem is not entirely new. Earlier studies used the approach but stuck with the original depth-2, width-(2n + 1) representation and did not use modern techniques such as backpropagation to train the networks. What the 2024 paper adds is the generalization to arbitrary network sizes and training with backpropagation.

Results of the KAN paper

The paper runs many tests on the potential of KANs and the factors that influence them. The authors vary the grid size of the spline functions and train KANs on a range of combined and specialized mathematical functions. They pay special attention to the internal representation and interpretability of the networks. They also demonstrate continuous learning, a pruning technique and an alternative to symbolic regression, show the scaling laws of KANs, and give examples of KANs in supervised and unsupervised learning.

Covering all of this is beyond the scope of this article. We focus on two points: grid extension and interpretability.

Grid extension

A spline can reach high accuracy by making its grid finer, and KANs inherit this property. MLPs have no concept of "fine-graining". KANs, on the other hand, can improve accuracy by refining their spline grids without retraining the model from scratch. Making an MLP wider and deeper also improves performance, but it is slow and costly. With a KAN, you can start with few parameters and extend to more later by refining the spline grids.

Grid extension of univariate B-spline functions

The research team uses the function f(x, y) = exp(sin(πx) + y^2) as a toy example to show how grid extension affects the training and test RMSE of a KAN.

Effect of extending the spline grid on training and test RMSE

The number of grid points starts small and increases in steps up to 1000. Training loss drops immediately and consistently every time the grid is extended, except at the highest point, where poor optimization makes the RMSE rise again. The test loss follows a U-shape, which reflects the bias-variance tradeoff (underfitting vs. overfitting). The optimal test loss occurs when the number of parameters matches the number of data points, the expected interpolation threshold. In this example, there are 1000 training samples and a [2, 5, 1] KAN has 15G parameters in total (G is the number of grid intervals). The interpolation threshold is therefore expected at G = 1000/15 ≈ 67, which roughly matches the observed value of G ∼ 50.

Knowing the function being learned, the ideal network would be a [2, 1, 1] KAN rather than the chosen [2, 5, 1] KAN. As the graph of training and test RMSE shows, this smaller network with fewer nodes would need less computation and training effort and still achieve even better results.

NOTE: The notation '[2, 5, 1]' used in the paper describes the structure of the KANs, indicating the layers of the KANs and the number of nodes in each layer. In this example, there are 2 nodes in the input layer, 5 nodes in the (single) hidden layer, and 1 node in the output layer. An illustration of this network can be found in the next section, see Figure "Symbolic Regression with KAN".

To avoid this kind of unnecessary and problematic oversizing in practice, the paper also introduces a technique to shrink the KAN structure to what the problem requires.

Simplifying KANs and making them interactive

Reducing the nodes of a KAN towards the ideal structure for a problem can cut training and computation effort and improve the model's accuracy at the same time. In practice, this ideal structure is unknown, so you need an automated process to decide which nodes can or should be removed.

The paper introduces a pruning technique that adjusts the KAN structure accordingly. It evaluates the importance of each node and removes those that contribute least to the network's performance. This reduces the network size step by step while keeping or even improving accuracy, so the network stays efficient and suited to the problem at hand.

Symbolic Regression with KAN

To make the network sparse, a sparsification penalty is added during training. The penalty considers the average magnitude of an activation function over all inputs, the sum of these values for a layer, and the layer's entropy. After this special training, the network is pruned at node level by removing nodes with low incoming and outgoing scores.

Because the spline functions make the relationships between individual features directly visible and interpretable, the researchers went a step further. They visualized the network with its splines as an interactive graph. In this graph, the learned functions can be replaced with symbolic functions (with some automatic adjustment for shifts and scaling). This makes KANs a strong alternative to traditional symbolic regression methods, and they can help identify internal structures in various mathematical and physical fields.

Conclusion: will KANs replace MLPs?

Despite their sophisticated mathematical foundation, KANs are essentially combinations of splines and MLPs that use the strengths of each while avoiding their weaknesses.

Will KANs replace MLPs completely? Probably not anytime soon, especially because MLPs train significantly faster. But we expect substantial improvements and a growing number of applications for KANs.

The authors of the paper are very confident about the potential of KANs, as their decision tree "Should I use KANs or MLPs?" shows. The only drawback they see is slow training, and they note: "we did not try hard to optimize KANs' efficiency though, so we deem KANs' slow training more as an engineering problem to be improved in the future rather than a fundamental limitation."

KAN vs. MLP decision tree provided by paper authors

Training speed is already improving. Just a few weeks after the paper's publication, the freely available GitHub repository "efficient-kan" appeared with an improved implementation.

Whether KANs will ultimately replace MLPs, as the authors predict, remains to be seen. If you are considering KANs for your own problem, the authors' decision tree above is a sensible first check. For another approach to making neural networks smaller and cheaper, see our post on BitNet and 1-bit LLM quantization.

Further reading

Interested in building high-quality AI agent systems?

We prepared a comprehensive guide based on cutting-edge research for how to build robust, reliable AI agent systems that actually work in production. This guide covers:

  • Understanding the 14 systematic failure modes in multi-agent systems
  • Evidence-based best practices for agent design
  • Structured communication protocols and verification mechanisms
Sascha Gstir

About the author

Sascha Gstir

Co-Founder, Pondhouse Data

Sascha worked for almost a decade in data analytics and data science. At Pondhouse Data he builds and tunes AI solutions, from model selection to production.