Second, sparseness is demanded because the world is sparse.
In our environment—and, correspondingly, in our heads—things happen in bursts, rather than at a low level all the time. This includes events, objects, attention, action, meaning, and decisions.
David Field, working with UC Berkeley neuroscientist Bruno Olshausen, followed this line of thinking in a landmark 1996 paper about sparseness in the visual system. They used computational models to show that our brains assume sparse structure in the spatial patterns of the world around us.
Essentially, they built a computational system that was forced to learn a sparse code. The code or “alphabet” is much like a written language, except it encodes pictures (small chunks of natural images) rather than words. What they found was that the visual alphabet that the computer model learned strongly resembled the alphabet used in our visual system.
In other words, in a system trained to simply be sparse, the brain’s basic strategy for analyzing the visual world pops out “for free” and without being explicitly pre-programmed. As it turns out, sparse coding also has strong connections to the current “deep learning” revolution in artificial intelligence, which I will discuss in future posts.
Getting back to the recent study led by Saskia de Vries of the Allen Institute, which I mentioned in the first post, one of the researchers’ main goals was to measure sparseness. With this information, we can estimate what proportion of the brain is active at the same time.
Across the visual areas, they sampled, and in different layers of the cortex, the de Vries data imply that around 20% of neurons are typically active at the same time. Although about three-quarters of visual neurons respond regularly, only about one in five is active at a time. And over time, individual neurons were active only during about 20% of the length of the recordings. Clearly, these values are closer to the 10% value, which I suggested as a rule of thumb than they are to the 100% myth.
What does all this have to do with your internet brain? The key is that the internet is also sparse. It is active in bursts, both across the network at a given time and within communication channels over time.
You can get a sense of the activity by monitoring your computer’s wifi signals. Macintosh users can open the application Activity Monitor (in the Utilities folder) and examine the Network tab. You will see a plot of message chunks (packets) sent and received by your computer over time. Unless you are using a great deal of bandwidth, the trace will generally appear sparse and bursty, like this:
Brains and the internet share sparse operating conditions. But we can go further. Another lesson of the internet metaphor for the brain is that occasional brief signals are crucial for making the system work. We are naturally attracted to strong, consistent signals in the brain, and to the bright flashes when the brain “lights up.” But following the internet metaphor, short spurts of activity are also crucial.
On the internet, there are a variety of brief signal bursts that allow routers to stay in contact. Importantly, these signal bursts don’t carry any message content. The signals include ACKs or acknowledgments, which tell a sending router that a set of messages was received at its destination. Routers also periodically send keep-alives, which are small messages that tell network neighbors that a router is ready to transmit messages.
If the brain used a similar strategy, these kinds of tiny messages would be missed or misunderstood if we were only concerned with situations where lots of neurons are active together. Like the internet, many signals in the brain probably relate to keeping the communication system working, rather than performing a specific task or behavior. It is about keeping the whole brain network “reachable,” which is something the internet excels at.