Blogs

I’ve recently started to look at a wider set of technologies in Machine Learning and AI. This is born out of the fact that, as good as deep learning is, I don’t see that it can answer all of the questions we need to answer to create the next level of machine intelligence.

Before I talk in more depth about one of the alternatives, I wanted to give an insight into why I think the current deep learning approaches have so much momentum:

1. There is a readily-available set of training data. I’m not sure if this is chicken-and-egg, but it seems that the algorithms used currently are designed to work well with the data that is available (think labelled datasets for image recognition, speech, text, video, medical scans, etc.). It could be that these datasets are also designed to work with the current algorithms also. In any case, the outcome is that the datasets available work well with the algorithms that are getting significant focus.
2. There are a number of very good frameworks for deep learning, such as Tensorflow, Keras, PyTorch, Caffe2, and more coming available almost each week. Large-scale tech companies are also layering additional tools on top of these to improve scalability, training efficiency, etc. such as Horovod and Michelangelo by Uber.
3. These technologies are taught heavily in leading AI and ML institutes around the World. You only need to look at YouTube to find courses by Stanford and many others, as well as online courses by Coursera and Udacity. For me, this sets up a virtuous circle of learning skills from courses backed by large tech companies (Google, Facebook, etc.) so that you get a job doing the same thing for those companies, and they then reinvest their knowledge into the next generation coming through those same training establishments. This is good, but I also think it doesn’t necessarily give a wider perspective of the technologies and approaches in this space.
4. These deep-learning technologies are getting lots of press, for example Deepmind’s AlphaGo and Agents with Imagination have been the topic of many Facebook and blog posts over recent months. This encourages others to leverage these technologies to try the next thing, to apply them in new ways, or to take them to the next level.
5. Subjectively at least, there’s a perspective that having “deep learning” in your company brochure will get you huge investment, or a buy-out for millions (or billions!) of dollars. So understandably, if you want to create a great company doing great stuff and sell for lots of money, that’s where I’d be putting my bets too. Or is it…

So, as good as the current deep learning approaches are, I think they fall short in a number of areas:

1. They are computationally expensive and require big hardware, whereas the brain is computationally (and therefore also energy) efficient.
2. They require lots of training, whereas often we can learn with a single or relatively small number of examples.
3. They don’t fully mimic the way we learn. We don’t have a training cycle then a “run” cycle. We continually learn as we go.
4. The brain seems to have many more “feedback” connections than feedforward connections (which is in effect all that deep learning does), so it helps us predict what might happen. Perhaps this is a way for us to learn more efficiently?
5. Deep learning currently is very focused on one task. It is hard to generalise it to take what it learns in one area and apply it to another in the same neural network.
6. And there are probably others too!

In my research for alternatives, one of the technologies I came across is the Spiking Neural Network (or SNN for short). They are designed to mimic the brain in a more biologically plausible way. From Wikipedia:

Spiking neural networks (SNNs) fall into the third generation of neural network models, increasing the level of realism in a neural simulation.[1] In addition to neuronal and synaptic state, SNNs also incorporate the concept of time into their operating model. The idea is that neurons in the SNN do not fire at each propagation cycle (as it happens with typical multi-layer perceptron networks), but rather fire only when a membrane potential — an intrinsic quality of the neuron related to its membrane electrical charge — reaches a specific value. When a neuron fires, it generates a signal which travels to other neurons which, in turn, increase or decrease their potentials in accordance with this signal.

Although this sounds complicated, and I won’t get into the details too much in this post, in effect spikes (or electrical pulses in the brain) travel between connected neurons, with only those neurons related to that particular activity activated at any particular time. This happens asynchronously, or depending on the implementation, is aligned to a clock signal (so that all of the spikes in a cycle are processed at the same time).

However, as far as I can tell, SNNs are mostly used in brain research at this point in time, with a relatively small number of researchers applying them to what I would deem as commercially-related concepts (such as computer vision). From my research so far, they do appear to have some interesting advantages over their somewhat-related deep-learning cousins:

1. With the right type of hardware (for example, neuromorphic chips specifically designed for this), they are more energy efficient, for example not all neurons need to be processed in a layer before the next layer of neurons is processed. This deep-learning characteristic means that in a typical training approach, every layer is processed in sequence before an error is calculated, then through back-propogation, that error adjusts all of the weights and biases in the previous layers in the network. This is computationally expensive and requires big hardware, hence the large GPU-powered computing farms employed by many of the companies investing heavily in this space.
2. They appear to get similarly impressive results as deep neural networks.
3. They cater for the ability to feed-forward and feed-backward in the same network meaning predicting events, or perhaps more accurately predicting possible multiple future events based on the current state of the agent, can become possible.
4. They can work with temporal data much more naturally, they are naturally designed to handle spikes over time.

There are however, some disadvantages:

1. They’re typically used as a tool to learn how the brain works rather than in any “practical” use cases (as in with a commercial interest), so examples are few and far between.
2. It’s harder to get the input data into a spiking format, so for example you need expensive (?) cameras that operate very differently than the way a normal camera works.
3. Following on from 3, there’s limited datasets out there to train them on, or the ones that are out there require significant pre-processing to turn them into a format suitable for an SNN to process.
4. To get the most out of them, they need to run on specifically designed chips that cater for large-scale parallel computations. Although this sounds similar to the way GPUs work, intrinsically SNNs require a different approach.

I’ve collated a selection of research papers that I’ve come across recently that start to bring their applicability to life, and also to show how they compare (favourably or otherwise) with DNNs:

1. LEARNING IN SPIKING NEURAL NETWORKS by Sergio Davis, School of Computer Science, University of Manchester
2. Article Source: Unsupervised Learning of Visual Features through Spike Timing Dependent Plasticity
   Masquelier T, Thorpe SJ (2007) Unsupervised Learning of Visual Features through Spike Timing Dependent Plasticity. PLOS Computational Biology 3(2): e31.https://doi.org/10.1371/journal.pcbi.0030031
3. Pani D, Meloni P, Tuveri G, Palumbo F, Massobrio P, Raffo L. An FPGA Platform for Real-Time Simulation of Spiking Neuronal Networks. Frontiers in Neuroscience. 2017;11:90. doi:10.3389/fnins.2017.00090.
4. Spike-Timing-Dependent Hebbian Plasticity as Temporal Difference Learning by Rajesh P. N. Rao, Department of Computer Science and Engineering, University of Washington, Seattle, WA 98195-2350, U.S.A. and Terrence J. Sejnowski, Howard Hughes Medical Institute, The Salk Institute for Biological Studies, La Jolla, CA 92037, U.S.A., and Department of Biology, University of California at San Diego, La Jolla, CA 92037, U.S.A.
5. Article Source: A Neuromorphic Architecture for Object Recognition and Motion Anticipation Using Burst-STDP
   Nere A, Olcese U, Balduzzi D, Tononi G (2012) A Neuromorphic Architecture for Object Recognition and Motion Anticipation Using Burst-STDP. PLOS ONE 7(5): e36958.https://doi.org/10.1371/journal.pone.0036958
6. Article Source: Supervised Learning in Spiking Neural Networks for Precise Temporal Encoding: Gardner B, Grüning A (2016) Supervised Learning in Spiking Neural Networks for Precise Temporal Encoding. PLOS ONE 11(8): e0161335.https://doi.org/10.1371/journal.pone.0161335
7. Spiking Deep Convolutional Neural Networks for Energy-Efficient Object Recognition: Cao, Y., Chen, Y. & Khosla, D. Int J Comput Vis (2015) 113: 54. https://doi.org/10.1007/s11263-014-0788-3
8. Spiking neural networks for vision tasks: By Henry Martin, and Prof. Dr Jörg Conrad, NEUROSCIENTIFIC SYSTEM THEORY, Technische Universität München
9. Event-based Visual Data Sets for Prediction Tasks in Spiking Neural Networks by Tingting (Amy) Gibson, Scott Heath, Robert P. Quinn, Alexia H. Lee, Joshua T. Arnold, Tharun S. Sonti, Andrew Whalley, George P. Shannon, Brian T. Song, James A. Henderson, and Janet Wiles
10. MNIST-DVS and FLASH-MNIST-DVS Databases