Neural Networks Need Naps, Just Like You
Researchers at Los Alamos National Laboratory have discovered that neural networks benefit from periods of downtime, just like humans need a good night's rest.
After long periods of work, a neural net can become unstable, so the scientists exposed the AI to waves that are very similar to the ones we experience while sleeping.
They presented their paper at the Conference on Computer Vision and Pattern Recognition.
Neural networks would like a day off. Between powering facial recognition systems, filtering email spam, and even aiding in cancer research, the specialized branch of machine learning deserves a bit of rest and relaxation.
And according to new research from the Los Alamos National Laboratory, neural nets might actually need that shuteye—or at least a machine's version of it.
Yijing Watkins, a computer scientist at Los Alamos, said her team had been studying "spiking neural networks," or systems that learn as much as our own living brains do, when they became inspired to try something a bit unusual.
🤖 The robots are coming—are you prepared? Learn how to master machines and take control of your world.
Because neural net simulations can become unstable after prolonged periods of unsupervised learning—the processing time during which the system looks for patterns in a dataset—Watkins and her team posited that a digital kind of "nap" could return the system to stasis.
"We were fascinated by the prospect of training a [neural net] in a manner analogous to how humans and other biological systems learn from their environment during childhood development," she said in a press release.
In June, Watkins and her team published their findings in the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops. In their paper, they describe incorporating "epochs of sinusoidally-modulated noise that we hypothesize are analogous to slow-wave sleep."
📩 Make your inbox more awesome.
That means to achieve some semblance of artificial sleep, the researchers had to inject noise into the neural network. They experimented with a few different types of static, but ultimately settled on Gaussian noise waves, which include an array of different frequencies and amplitudes.
Watkins and her colleagues believe their new algorithm, which induces the noise, can mimic the slow wave sleep that rejuvenates the biological neurons in our own brains. "It was as though we were giving the neural networks the equivalent of a good night’s rest," she said.
Moving forward, the team wants to test its algorithm on the Intel Loihi neuromorphic chip, which emulates the neural structure of the human brain. The chip includes 130,000 neurons "optimized for spiking neural networks," according to an Intel blog post, making the hardware suitable for sensing and perception tasks, like analyzing the contents of a video frame.
The Los Alamos researchers hope that by letting the Loihi chip "sleep" here and there with the team's noise-generating algorithm, it will be able to process data from a silicon retina camera (basically an artificial eye) in real time.
You Might Also Like
