While following a neural networks course from Lightning AI, I wanted to understand one of the simplest trainable models: the perceptron.
I implemented it in Ruby, since most examples I found were written in Python.
A perceptron as a mathematical model
A perceptron with two inputs is essentially:
z = w1*x1 + w2*x2 + bias
The result is then converted into a binary prediction such as 0 or 1.
The interesting part is that this simple mathematical function can represent some problems accurately.
Considering the AND gate, if we plot the four inputs as points on a graph, we can draw a single straight line separating (1,1) from the other three points.
The same is true for OR.
These problems are linearly separable, so a single perceptron can learn them.
Why XOR fails with perceptron
Now consider XOR, when we plot four points, the limitation becomes visible.
There is no single straight line that correctly separates the class 0 points from the class 1 points.
This is why a single perceptron cannot solve XOR.
It is not an implementation problem. It is a limitation of the mathematical model itself.
From perceptron to neural network
The solution is to add another layer.
Instead of one perceptron, we can use a tiny network with:
2 inputs → 2 hidden neurons → 1 output
With the hidden layer, the network is no longer restricted to one linear boundary and can learn XOR.
Implementation of this network is in Ruby as well, including sigmoid activation and backpropagation.
I also wanted to add visualization during training. The program captures the decision surface every few hundred epochs. This makes it possible to watch the network gradually reshape its decision regions until XOR is classified correctly.
Starting with no decision boundary at the beginning of the training:
Slowly separating boundaries:
Until the end of the training to have the final separated boundaries for the high accuracy prediction.
A historical connection
The limitations of single-layer perceptrons also played an important role in the early history of neural networks.
In 1969, Marvin Minsky and Seymour Papert published Perceptrons, showing important limitations of single-layer networks, including their inability to represent problems such as XOR.
Later multilayer networks showed that these limitations could be overcome, but interest and funding in neural-network research declined significantly during this period, contributing to what became known as the first AI winter.
The history must be more complicated than saying that XOR alone caused the AI winter, but the example remains useful because it clearly demonstrates why model architecture matters.
Try it yourself
The Ruby implementation is available here:
Perceptron and neural network in Ruby
For an interactive way to experiment with hidden layers and decision boundaries, TensorFlow Playground is also excellent:
Inspiration: Lightning AI course
Source: Deep Learning for Coders