SyNAPSE is not a project DARPA undertook lightly. Many attempts at large-scale neuromorphic engineering have been made in the past. None met their goals. As such, SyNAPSE owes its existence to a number of recent game-changing developments. From HP Labs, the discovery of the memristor was one such keystone innovation. It took Greg Snider’s 2007 work in Nanotechnology, however, to establish memristors as a viable platform for the implementation of self-organizing recurrent neural networks.
In the self-organizing literature, Christoph von der Malsburg’s 1973 work on adaptive, recurrent networks in visual cortex is seminal. It certainly isn’t the root of all things self-organizing, but the key result is simple and compelling. In the paper, von der Malsburg demonstrated development of orientation-selective cells, a crucial neuron type for visual processing, but used only a simple two-layer neural network with a uniform pattern of interconnects. Through deterministic evolution of the interconnect strengths in response to visual input, the network naturally produced orientation-selective cells. Given the significance of the results, replicating von der Malsburg’s network is typically one of the first non-trivial tests a new adaptive, recurrent neural architecture must pass. In his 2007 paper, Snider did just this.
The simulated memristor implementation is remarkable in its ordinariness. Snider not only easily replicated the key findings from von der Malsburg’s work, but did so in a way conducive to extremely efficient implementation on real hardware. Like most of the memristor hardware designed by HP Labs, this design used a hybrid of the conventional CMOS process and a nanowire lattice with memristor interconnects. CMOS, the technology used to manufacture standard microprocessors today, was used for the “neuron” parts of the model. Nanowires with memristor interconnects, which can be built at vastly higher density than standard CMOS components, were used to wire the simulated neurons together. The memristors themselves were used to store the simulated synaptic weights. This density disparity is actually a key part of the argument for using hybrid memristor hardware in neuromorphic systems.
Again, the Snider simulation is significant not just because it replicated the von der Malsburg work, but because it implies that such networks could be implemented extremely efficiently on actual hardware. This is a direct product of the high manufacturing density of memristors. In similar work that uses CMOS for both neurons and synapses, synapses consume roughly the same surface area as neurons. In biological systems, however, synapses outnumber neurons by a factor of 1000 on average. Since memristors can be manufactured at similarly high ratio compared to CMOS features, they are an ideal candidate for simulated synapses.
Such a high-density approach is not without drawbacks. Memristors on a nanoware lattice are very small, but also very difficult to manufacture reliably. A modern microprocessor, for instance, contains on the order of a billion transistors. Typically, a single manufacturing defect renders the whole chip inoperable. Thus, the defect rate must be on the order of one per billion or less to manufacture microprocessors economically. The memristor/nanowire architecture, in comparison, may carry defect rates as high as tens per hundred. Not only this, but the memristor chips decay rapidly in comparison to CMOS chips. Multiple new faults are likely per second. Some fraction of these higher defect and fault rates are a product of basic physical limitations, so better manufacturing techniques are unlikely to solve the problem. Snider notes that the technical term for this type of hardware is “crummy.”
Implementing standard boolean logic on crummy hardware is very difficult, but brains operate with a reasonable degree of accuracy under very similar conditions. Brain-like computation systems typically degrade gracefully as the error rate increases, rather than catastrophically failing in response to a small number of errors. This graceful fall-off of accuracy is precisely what Snider goes on to demonstrate. As he increased the simulated manufacturing defect rate, the networks continued to converge to the desired results. Accuracy fell slowly as a function of the error rate.
Being a simulation, Snider’s work does come with some caveats. Most significantly, he used a highly simplified differential equation to describe memristor behavior. Weighing the benefits against the potential drawbacks, however, Snider’s self-organizing proof-of-concept is a crucial building block for SyNAPSE.
Nanotechnology, 2007. DOI: 10.1088/0957-4484/18/36/365202