Neuromorphic AI might not just help bring
AI to the edge, but also reduce carbon emissions. Generated by author
with ImageGen 3.
There are periodic proclamations of the coming neuromorphic computing
revolution. While there remain substantial challenges in the field,
there are solid successes that have come out of the field and there
continues to be steady progress in using spiking neural network
algorithms with neuromorphic hardware. In this article, I will cover
some neuromorphic computing and engineering basics, training, the
advantages of neuromorphic systems, and the remaining challenges.
The classical use case of neuromorphic systems is for edge devices
that need to perform the computation locally and are energy-limited, for
example, battery-powered devices. However, one of the recent interests
in using neuromorphic systems is to reduce energy usage at data centers,
such as the energy needed by large language models (LLMs). For example,
OpenAI signed a letter of intent to purchase $51 million of neuromorphic
chips from Rain AI in December 2023. This makes sense since OpenAI
spends a lot on inference, with one estimate of around $4
billion on running inference in 2024. It also appears that both
Intel’s Loihi 2 and IBM’s NorthPole (successor to TrueNorth)
neuromorphic systems are designed for use in servers.
The promises of neuromorphic computing can broadly be divided into 1)
pragmatic, near-term successes that have already found successes and 2)
more aspirational, wacky neuroscientist fever-dream ideas of how spiking
dynamics might endow neural networks with something closer to real
intelligence. Of course, it’s group 2 that really excites me, but I’m
going to focus on group 1 for this post. And there is no more exciting
way to start than to dive into terminology.
Terminology
Neuromorphic computation is often defined as
computation that is brain-inspired, but that definition leaves a lot to
the imagination. Neural networks are more neuromorphic than classical
computation, but these days neuromorphic computation is specifically
interested in using event-based spiking neural networks (SNNs) for their
energy efficiency. Even though SNNs are a type of artificial neural
network, the term “artificial neural networks” (ANNs) is reserved for
the more standard non-spiking artificial neural networks in the
neuromorphic literature. Schuman and colleagues (2022) define
neuromorphic computers as non-von Neuman computers where both processing
and memory are collocated in artificial neurons and synapses, as opposed
to von Neuman computers that separate processing and memory.
von Neumann Computers operate on digital
information, have separate processors and memory, and are synchronized
by clocks, while neuromorphic computers operate on event-driven spikes,
combine compute and memory, and are asynchronous. Created by author with
inspiration from Schuman et al 2022.
Neuromorphic engineering means designing the
hardware while “neuromorphic computation” is focused on what is being
simulated rather than what it is being simulated on. These are tightly
intertwined since the computation is dependent on the properties of the
hardware and what is implemented in hardware depends on what is
empirically found to work best.
Another related term is NeuroAI, the goal of which
is to use AI to gain a mechanistic understanding of the brain and is
more interested in biological realism. Neuromorphic computation is
interested in neuroscience as a means to an end. It views the brain as a
source of ideas that can be used to achieve objectives such as energy
efficiency and low latency in neural architectures. A decent amount of
the NeuroAI research relies on spike averages rather than spiking neural
networks, which allows closer comparison of the majority of modern ANNs
that are applied to discrete tasks.
Event-Driven Systems
Generated by author using ImageGen
3.
Neuromorphic systems are event-based, which is a paradigm shift from
how modern ANN systems work. Even real-time ANN systems typically
process one frame at a time, with activity synchronously propagated from
one layer to the next. This means that in ANNs, neurons that carry no
information require the same processing as neurons that carry critical
information. Event-driven is a different paradigm that often starts at
the sensor and applies the most work where information needs to be
processed. ANNs rely on matrix operations that take the same amount of
time and energy regardless of the values in the matrices. Neuromorphic
systems use SNNs where the amount of work depends on the number
of spikes.
A traditional deployed ANN would often be connected to a camera that
synchronously records a frame in a single exposure. The ANN then
processes the frame. The results of the frame might then be fed into a
tracking algorithm and further processed.
Event-driven systems may start at the sensor with an event camera.
Each pixel sends updates asynchronously whenever a change crosses a
threshold. So when there is movement in a scene that is otherwise
stationary, the pixels that correspond to the movement send events or
spikes immediately without waiting for a synchronization signal. The
event signals can be sent within tens of microseconds, while a
traditional camera might collect at 24 Hz and could introduce a latency
that’s in the range of tens of milliseconds. In addition to receiving
the information sooner, the information in the event-based system would
be sparser and would focus on the movement. The traditional system would
have to process the entire scene through each network layer
successively.
Learning in Spiking Neural
Networks
One way to teach a spiking neural network
is to have a teacher [ANN]. Generated by author with ImageGen
3.
One of the major challenges of SNNs is training them. Backpropagation
algorithms and stochastic gradient descent are the go-to solutions for
training ANNs, however, these methods run into difficulty with SNNs. The
best way to train SNNs is not yet established and the following methods
are some of the more common approaches that are used:
ANN to SNN conversion
Backpropagation-like
Synaptic plasticity
Evolutionary
ANN to SNN conversion
One method of creating SNNs is to bypass training the SNNs directly
and instead train ANNs. This approach limits the types of SNNs and
hardware that can be used. For example, Sengupta et al (2019) converted
VGG and ResNets to ANNs using an integrate-and-fire (IF) neuron that
does not have a leaking or refractory period. They introduce a novel
weight-normalization technique to perform the conversion, which involves
setting the firing threshold of each neuron based on its pre-synaptic
weights. Dr. Priyadarshini Panda goes into more detail in her ESWEEK
2021 SNN Talk.
Benefits:
Enables deep SNNs.
Allows reuse of deep ANN knowledge, such as training, architecture,
etc.
Drawbacks:
Limits architectures to those suited to ANNs and the conversion
procedures.
Network doesn’t learn to take advantage of SNN properties, which can
lead to lower accuracy and longer latency.
Backpropagation-like
approaches and surrogate gradient descent
The most common methods currently used to train SNNs are
backpropagation-like approaches. Standard backpropagation does not work
to train SNNs because 1) the spiking threshold function’s gradient is
nonzero except at the threshold where it is undefined and 2) the credit
assignment problem needs to be solved in the temporal dimension in
addition spatial (or color etc).
In ANNs, the most common activation function is the ReLU. For SNNs,
the neuron will fire if the membrane potential is above some threshold,
otherwise, it will not fire. This is called a Heaviside function. You
could use a sigmoid function instead, but it is not a spiking neural
network. The solution of using surrogate gradients is to use the
standard threshold function in the forward pass, but then use the
derivative from a “smoothed” version of the Heaviside function, such as
the sigmoid function, in the backward pass (Neftci et al. 2019, Bohte,
2011).
Advantages:
Connects to well-known methods.
Compared to conversion, can result in a more energy efficient
network (Li et al 2022)
Disadvantages:
Can be computationally intensive to solve both spatially and through
time
Synaptic Plasticity
Spike-timing-dependent plasticity (STDP) is the most well-known form
of synaptic plasticity. In most cases, STDP increases the strength of a
synapse when a presynaptic (input) spike comes immediately before the
postsynaptic spike. Early models have shown promise with STDP on simple
unsupervised tasks, although getting it to work well for more complex
models and tasks has proven more difficult.
Other biological learning mechanisms include the pruning and creation
of both neurons and synapses, homeostatic plasticity, neuromodulators,
astrocytes, and evolution. There is even some recent evidence that some
primitive types of knowledge can be passed down by epigenetics.
Advantages:
Unsupervised
Can take advantage of temporal properties
Biologically inspired
Disadvantages:
Synaptic plasticity is not well understood, especially at different
timescales
Difficult to get to work with non-trivial networks
Evolutionary Optimization
Evolutionary optimization is another approach that has some cool
applications that works well with small networks. Dr. Catherine Schuman
is a leading expert and she gave a fascinating talk on neuromorphic
computing to the ICS lab that is available on YouTube.
Advantages:
Applicable to many tasks, architectures, and devices.
Can learn topology and parameters (requiring less knowledge of the
problem).
Learns small networks which results in lower latency.
Disadvantages:
Not effective for problems that require deep or large
architectures.
Advantages of Neuromorphic
Systems
Energy Efficiency
Neuromorphic systems have two main advantages: 1) energy efficiency
and 2) low latency. There are a lot of reasons to be excited about the
energy efficiency. For example, Intel claimed
that their Loihi 2 Neural Processing Unit (NPU) can use 100 times less
energy while being as much as 50 times faster than conventional ANNs.
Chris Eliasmith compared the energy efficiency of an SNN on neuromorphic
hardware with an ANN with the same architecture on standard hardware in
a
presentation available on YouTube. He found that the SNN is 100
times more energy efficient on Loihi compared to the ANN on a standard
NVIDIA GPU and 20 times more efficient than the ANN on an NVIDIA Jetson
GPU. It is 5-7 times more energy efficient than the Intel Neural Compute
Stick (NCS) and NCS 2. At the same time the SNN achieves a 93.8%
accuracy compared to the 92.7% accuracy of the ANN.
Figure recreated by author from Chris
Eliasmith’s slides at
https://www.youtube.com/watch?v=PeW-TN3P1hk&t=1308s which shows the
neuromorphic processor being 5-100x more efficient while achieving a
similar accuracy.
Neuromorphic chips are more energy efficient and allow complex deep
learning models to be deployed on low-energy edge devices. In October
2024, BrainChip introduced the Akida Pico NPU which uses less than 1 mW
of power, and Intel Loihi 2 NPU uses 1 W. That’s a lot less power than
NVIDIA Jetson modules that use between 10-50 watts which is often used
for embedded ANNs and server GPUs can use around 100 watts.
Comparing the energy efficiency between ANNs and SNNs are difficult
because: 1. energy efficiency is dependent on hardware, 2. SNNs and ANNs
can use different architectures, and 3. they are suited to different
problems. Additionally, the energy used by SNNs scales with the number
of spikes, so the number of spikes needs to be minimized to achieve the
best energy efficiency.
Theoretical analysis is often used to estimate the energy needed by
SNNs and ANNs, however, this doesn’t take into account all of the
differences between the CPUs and GPUs used for ANNs and the neuromorphic
chips for SNNs.
Looking into nature can give us an idea of what might be possible in
the future and Mike Davies provided a great anecdote in an Intel Architecture All
Access YouTube video:
Consider the capabilities of a tiny cockatiel parrot brain, a
two-gram brain running on about 50 mW of power. This brain enables the
cockatiel to fly at speeds up to 20 mph, to navigate unknown
environments while foraging for food, and event to learn to manipulate
objects as tools and utter human words.
In current neural networks, there is a lot of wasted computation. For
example, an image encoder takes the same amount of time encoding a blank
page as a cluttered page in a “Where’s Waldo?” book. In spiking neural
networks, very few units would activate on a blank page and very little
computation would be used, while a page containing a lot of features
would fire a lot more units and use a lot more computation. In real
life, there are often regions in the visual field that contain more
features and require more processing than other regions that contain
fewer features, like a clear sky. In either case, SNNs only perform work
when work needs to be performed, whereas ANNs depend on matrix
multiplications that are difficult to use sparsely.
This in itself is exciting. A lot of deep learning currently involves
uploading massive amounts of audio or video to the cloud, where the data
is processed in massive data centers, spending a lot of energy on the
computation and cooling the computational devices, and then the results
are returned. With edge computing, you can have more secure and more
responsive voice recognition or video recognition, that you can keep on
your local device, with orders of magnitude less energy consumption.
Low Latency
When a pixel receptor of an event camera changes by some threshold,
it can send an event or spike within microseconds. It doesn’t need to
wait for a shutter or synchronization signal to be sent. This benefit is
seen throughout the event-based architecture of SNNs. Units can send
events immediately, rather than waiting for a synchronization signal.
This makes neuromorphic computers much faster, in terms of latency, than
ANNs. Hence, neuromorphic processing is better than ANNs for real-time
applications that can benefit from low latency. This benefit is reduced
if the problem allows for batching and you are measuring speed by
throughput since ANNs can take advantage of batching more easily.
However, in real-time processing, such as robotics or user interfacing,
latency is more important.
Disadvantages and Challenges
Everything Everywhere All at
Once
One of the challenges is that neuromorphic computing and engineering
are progressing at multiple levels at the same time. The details of the
models depend on the hardware implementation and empirical results with
actualized models guide the development of the hardware. Intel
discovered this with their Loihi 1 chips and built more flexibility into
their Loihi 2 chips, however, there will always be tradeoffs and there
are still many advances to be made on both the hardware and software
side.
Limited
Availability of Commercial Hardware
Hopefully, this will change soon, but commercial hardware isn’t very
available. BrainChip’s Akida was the first neuromorphic chip to be
commercially available, although apparently,
it does not even support the standard leaky-integrate and fire (LIF)
neuron. SpiNNaker boards used to be for sale, which was part of the EU
Human Brain Project but are no longer
available. Intel makes Loihi 2 chips available to some academic
researchers via the Intel
Neuromorphic Research Community (INRC)program.
Datasets
The number of neuromorphic datasets is much less than traditional
datasets and can be much larger. Some of the common smaller computer
vision datasets, such as MNIST (NMNIST, Li et al 2017) and CIFAR-10
(CIFAR10-DVS, Orchard et al 2015), have been converted to event streams
by displaying the images and recording them using event-based cameras.
The images are collected with movement (or “saccades”) to increase the
number of spikes for processing. With larger datasets, such as
ES-ImageNet (Lin et al 2021), simulation of event cameras has been
used.
The dataset derived from static images might be useful in comparing
SNNs with conventional ANNs and might be useful as part of the training
or evaluation pipeline, however, SNNs are naturally temporal, and using
them for static inputs does not make a lot of sense if you want to take
advantage of SNNs temporal properties. Some of the datasets that take
advantage of these properties of SNNs include:
- DvsGesture (Amir et al. 2017) - a dataset of people performing a
set of 11 hand and arm gestures - Bullying10K (Dong et al. 2024) - a
privacy-preserving dataset for bullying recognition
Synthetic data can be generated from standard visible camera data
without the use of expensive event camera data collections, however
these won’t exhibit the high dynamic range and frame rate that event
cameras would capture.
Tonic is an example python library that makes it easy to access at
least some of these event-based datasets. The datasets themselves can
take up a lot more space than traditional datasets. For example, the
training images for MNIST is around 10 MB, while in N-MNIST, it is
almost 1 GB.
Another thing to take into account is that visualizing the datasets
can be difficult. Even the datasets derived from static images can be
difficult to match with the original input images. Also, the benefit of
using real data is typically to avoid a gap between training and
inference, so it would seem that the benefit of using these datasets
would depend on their similarity to the cameras used during deployment
or testing.
Conclusion
Neuromorphic Computers are the Wave of
the Future! Created by author with ImageFx and GIMP.
We are in an exciting time with neuromorphic computation. There are
still challenges for adoption, but there are proven cases where they are
more energy efficient, especially standard server GPUs while having
lower latency and similar accuracy as traditional ANNs. A lot of
companies, including Intel, IBM, Qualcomm, Analog Devices, Rain AI, and
BrainChip have been investing in neuromorphic systems. BrainChip is the
first company to make their neuromorphic chips commercially available
while both Intel and IBM are on the second generations of their research
chips (Loihi 2 and NorthPole respectively).
References
Amir, A., Taba, B., Berg, D., Melano, T., McKinstry, J., Di Nolfo,
C., Nayak, T., Andreopoulos, A., Garreau, G., Mendoza, M., Kusnitz, J.,
Debole, M., Esser, S., Delbruck, T., Flickner, M., & Modha, D.
(2017). A Low Power, Fully Event-Based Gesture Recognition
System. 7243–7252. https://openaccess.thecvf.com/content_cvpr_2017/html/Amir_A_Low_Power_CVPR_2017_paper.html
Bohte, S. M. (2011). Error-Backpropagation in Networks of
Fractionally Predictive Spiking Neurons. In Artificial Neural
Networks and Machine Learning https://doi.org/10.1007/978-3-642-21735-7_8
Dong, Y., Li, Y., Zhao, D., Shen, G., & Zeng, Y. (2023).
Bullying10K: A Large-Scale Neuromorphic Dataset towards
Privacy-Preserving Bullying Recognition. Advances in Neural
Information Processing Systems, 36, 1923–1937.
Li, C., Ma, L., & Furber, S. (2022). Quantization Framework for
Fast Spiking Neural Networks. Frontiers in Neuroscience,
16. https://doi.org/10.3389/fnins.2022.918793
Li, H., Liu, H., Ji, X., Li, G., & Shi, L. (2017). CIFAR10-DVS:
An Event-Stream Dataset for Object Classification. Frontiers in
Neuroscience, 11. https://doi.org/10.3389/fnins.2017.00309
Lin, Y., Ding, W., Qiang, S., Deng, L., & Li, G. (2021).
ES-ImageNet: A Million Event-Stream Classification Dataset for Spiking
Neural Networks. Frontiers in Neuroscience, 15.
[https://doi.org/10.3389/fnins.2021.726582](https://doi.org/10.3389/fnins.2021.726582
Neftci, E. O., Mostafa, H., & Zenke, F. (2019). Surrogate
Gradient Learning in Spiking Neural Networks: Bringing the Power of
Gradient-Based Optimization to Spiking Neural Networks. IEEE Signal
Processing Magazine. https://doi.org/10.1109/MSP.2019.2931595
Orchard, G., Jayawant, A., Cohen, G. K., & Thakor, N. (2015).
Converting Static Image Datasets to Spiking Neuromorphic Datasets Using
Saccades. Frontiers in Neuroscience, 9. https://doi.org/10.3389/fnins.2015.00437
Schuman, C. D., Kulkarni, S. R., Parsa, M., Mitchell, J. P., Date,
P., & Kay, B. (2022). Opportunities for neuromorphic computing
algorithms and applications. Nature Computational Science,
2(1), 10–19. https://doi.org/10.1038/s43588-021-00184-y
Sengupta, A., Ye, Y., Wang, R., Liu, C., & Roy, K. (2019). Going
Deeper in Spiking Neural Networks: VGG and Residual Architectures.
Frontiers in Neuroscience, 13. https://doi.org/10.3389/fnins.2019.00095
Resources
Open Neuromorphic (ONM)
Collective
Event-Based Vision Resources
(https://github.com/uzh-rpg/event-based_vision_resources) - Upcoming
workshops, papers, companies, neuromorphic systems, etc.
Talks on Youtube
[[Neuromorphic Computing from the Computer Science Perspective video
from ICAS Lab with Dr Catherine Schuman]]
[[Cosyne 2022 Tutorial on Spiking Neural Networks]]
ESWEEK 2021
Dr. Priyadarshini Panda’s SNN Talk
Intel Architecture All Access: Neuromorphic Computing Part 1 and Part 2 by Mike
Davies.
Spiking Neural
Networks for More Efficient AI Algorithms Talk by Professor Chris
Eliasmith at University of Waterloo
Read More ›
