Signal theory of intelligence

2  Learning in Neural Networks

Intelligence requires knowledge. Without knowledge, a system—biological or artificial—cannot recognize patterns, make decisions, or generate predictions. Knowledge is acquired through learning, which involves permanent changes in the synaptic strengths within a neural network.

In biological systems, learning occurs through interaction with the environment. Experiences modify synaptic strengths, strengthening useful connections and weakening less helpful ones. This process results in a growing, structured knowledge network that enables perception, behavior, and cognition.

Artificial neural networks acquire knowledge differently: they are trained with large amounts of existing data. Mathematical optimization algorithms adjust the weights so that the network produces the desired output behavior. Despite differing mechanisms, both systems are based on the same fundamental principle:

Knowledge arises from the adjustment of synaptic strengths.

Below, we examine four fundamental learning mechanisms that play a central role in both biological and artificial neural networks.

2.1  Hebbian and Anti-Hebbian Learning

In 1949, Donald Hebb formulated the principle now known as Hebb’s learning rule: b>"Neurons that fire together, wire together."

When two neurons are active simultaneously, the connection between them is strengthened. This allows the network to reinforce frequently co-occurring patterns. Hebbian learning underpins:

• association formation
• pattern recognition
• memory formation
• the development of stable representations

The complementary Anti-Hebbian learning weakens connections when the activity of two neurons is systematically opposite. This suppresses disruptive or redundant correlations. Together, these mechanisms enable biological networks to enhance relevant patterns and eliminate irrelevant ones.

2.2  Gradient-Based Methods

In artificial neural networks, learning primarily occurs through > gradient-based methods. A loss function is defined to measure how far the current network behavior deviates from the desired behavior. The weights are then adjusted to minimize this error.

The most well-known method is backpropagation, where the error is propagated backward through the network. The weights are modified proportionally to the gradient of the error function.

Gradient methods are:

• mathematically precise
• efficient for large datasets
• flexible for various tasks

ThThey form the backbone of modern AI systems, although they differ significantly from biological learning mechanisms in terms of biological plausibility.

2.3  Lateral Inhibition, Neural Competition, and Symmetry Breaking

Biological neural networks consist of many neurons interconnected with others. Without additional mechanisms, multiple neurons would learn the same complex signals, wasting resources. The nervous system must ensure that neurons specialize rather than develop redundant representations.

In artificial networks, this problem is known as symmetry breaking: if all neurons have identical inputs and initial weights, they would learn the same functions without additional measures. AI systems address this through random initialization or explicit regularization.

Biological networks employ a more elegant mechanism: neural competition via lateral inhibition.

• Active output neurons inhibit their neighbors.
• The neuron with the strongest signal prevails.
• Weaker neurons are suppressed and can specialize in other patterns.

This competition enforces functional differentiation among neurons and prevents redundant learning. Without lateral inhibition, symmetry breaking would not occur in natural networks—neurons would encode the same patterns, preventing the development of complex representations.

Therefore, lateral inhibition is a key mechanism for:

• feature selection
• contrast enhancement
• formation of specialized detectors
• efficient utilization of neural resources

It It provides the biological basis for learning rules that extract statistical structures from input data.

2.4  Oja’s and Sanger’s Rules in Biological Networks

Oja’s rule is a biologically plausible extension of Hebb’s rule. It strengthens correlations but also prevents weights from growing without bound, leading to a stable, normalized representation of the most significant input features.>

Sanger’s rule advances this further by combining Hebbian learning with lateral inhibition. This results in a network where multiple neurons learn different, orthogonal principal components.

The critical point is: Sanger’s rule only works because lateral competition enforces symmetry breaking.

• Without competition, all neurons would learn the same principal component.
• Through lateral inhibition, the first neuron becomes the “winner” of the strongest component.
• The remaining neurons are inhibited and learn subsequent, orthogonal components.

This makes Sanger’s rule a biologically plausible form of Principal Component Analysis (PCA), which operates entirely locally without global optimization.

It demonstrates how biological networks:

• extract statistical structures,
• avoid redundant representations,

• and efficiently span complex feature spaces.