Spiking Neural Networks

The biological metaphor behind the Artificial Neural Networks (ANNs) used in today's language models, classifiers, and reinforcement learning algorithms can be traced back to Alan Turing's B-type machines. Similar to biological neurons, artificial neurons integrate weighted signals from input neurons, apply an activation function, and transmit this signal to connected output neurons. This coarse approximation of the biological neuron has found tremendous success. Students, researchers, and corporations alike have created incremental improvements to this model that brought us from Turing's speculation to the vibrant machine learning landscape of today. Now, with hardware restricted by both Moore's Law and supply issues, interest in the next improvement is at a peak. Alternative systems are being explored, both more or less biologically informed, and one such model is the spiking neural network.

What They Are

Neuron anatomy is an incredibly complex system involving the interplay between thousands of genes and proteins. At a high level, the signal of a real biological neuron is a time series of action potentials, or spikes. A neuron's action potential travels to synapses with other neurons, where it triggers the release of neurotransmitters. These neurotransmitters lead to a change in the voltage of the downstream neuron, and when this voltage depolarizes beyond a threshold, it spikes. This chain of activity gives rise to the central dogma of neuroscience: neurons that fire together wire together. Neurons are constantly adjusting the strength of their synapses through a variety of mechanisms, and we describe this as plasticity.

The standard artificial neuron retains no such voltage, instead producing an output based solely on its input at any given timestep. Furthermore, all artificial neurons are synchronized in time. Spiking neural networks aim to better approximate the biological neuron. They maintain a voltage at each node, are typically subject to sparse activations, and often have a model of plasticity built in.

Implementations and Applications

A number of organizations have developed research platforms for spiking neural networks. Intel has the Loihi 2, IBM created TrueNorth, and the Universities of Manchester and Dresden built SpiNNaker and SpiNNaker 2 respectively. Because neuron activity is typically sparse, these organizations have been able to drive spiking workloads with small fractions of the power required by traditional ANNs. They accomplish this by multiplexing a large number of cores, each representing a collection of neurons. As long as activity remains sparse and there is little bus contention, a route between any two neurons on any two cores in the chip is easily accessible.

Creating models of and programming spiking networks is an area of active research. Chris Eliasmith from the University of Waterloo has one such solution called Nengo. Nengo provides tools to build networks, deploy to hardware, and leverage a variety of neuron, plasticity, and network models.

BrainChip, a company specializing in neuromorphic hardware, has found commercial success monetizing low power learning at the edge. Rather than the research platforms like Loihi 2 that seek to enable co-development of spiking models and hardware, BrainChip focuses on the translation from traditional convolutional neural networks (think TensorFlow) into spiking equivalents. By taking advantage of preexisting silicon manufacturing technology and model development practices, BrainChip has become a frontrunner in the neuromorphic computing race.