A history of neural networks (part 1)
Artificial intelligence (AI) and its underlying code are now ubiquitous. It is all too easy to become disoriented in this vast field of research that lies at the crossroads of neuroscience, computer science and mathematics. AI algorithms are powerful, innovative and even computer scientists consider them deeply mysterious. To understand these algorithms, let’s break them down into their basic elements. From personalized suggestion tools to cancer detection tools to newsfeed, image and data generation tools, AI is drastically changing our approach to problem resolution not only in the field of computer science but also in other fields, some of them unrelated to computers.
AI in a nutshell
Programming involves writing lengthy, detailed instructions to tell a computer what to do and how to do it in order to achieve a desired outcome. But artificial intelligence attempts to do the opposite: you write an algorithm to enable a computer to figure out a new algorithm to solve a given problem. Clear as mud? Read on.
Imagine a programme that recognizes the letters and numbers on license plates. A traditional programmer would write code providing a myriad rules to cover hundreds or thousands of exceptions in order to obtain an effective algorithm. An intelligent algorithm, on the other hand, would enable the computer to learn these rules on its own to complete the required task.
A short history
The perceptron: the first building block
The field of artificial intelligence is not new. In fact, it’s been explored for decades more or less intensely, depending on the availability of technical resources and funding for primary research. Since the ’50s, researchers have been positing what intelligence is and how to replicate it artificially. The goal isn’t so much to copy the human brain, but rather to emulate its cognitive abilities to find new solutions to a wide variety of problems. Research in the field of artificial intelligence is intrinsically tied to that of neuroscience, and both fields are progressing independently yet simultaneously, working apace to reproduce intelligence.
The first model of an artificial neuron was the Rosenblatt perceptron, developed in 1957. Rosenblatt’s impressive and innovative work, which is still considered the beginning of intelligent algorithms, was quickly set aside due to the limited concrete applications of the actual algorithm. The Rosenblatt artificial neuron is in fact little more than a linear, binary sorter for classification. However, it was a ground-breaking development that would set the stage for future research.
John McCarthy and LISP — a world of functions
Less than a year later, at the Massachusetts Institute of Technology (MIT), John McCarthy, one of the pioneers of AI, invented a purely functional programming language called LISP, which quickly became the language of choice for AI research. LISP was first made public in a paper in Communications of the ACM, in which it was stated that computers should be able to handle declarative as well as imperative sentences and exhibit “common sense”.
LISP was nothing short of a revolution for computer science, becoming the tool of choice for research and experimentation in the field of artificial intelligence.
The first occurrence of “machine learning” and artificial neural networks
British scientist Arthur L. Samuel first coined the term “machine learning” in one of his pioneering articles on neural networks. Samuel explained the representation of an algorithm for a machine to learn to play checkers. Samuel was also a significant contributor to TeX, a system for the writing of scientific and mathematics documents.
Learning, the brain and algorithms that can learn
Of course, there’s more to the history of neural networks; take, for example, the appearance of gradient backpropagation, which will be the subject of another feature. But all the fundamental research was predicated on the basic idea of artificial intelligence, which is to get a computer to find the optimal way to perform the simplest of tasks (for example, binary classification in the perceptron example) or the most advanced demonstrations of the principles of cognitive learning (for example, learning to play checkers).
What about the human brain?
Neuroscience is the scientific study of the functioning of the nervous system. Artificial intelligence does not seek to replicate the functioning of the human brain, but rather to emulate its principles to find innovative solutions. The human brain is perpetually learning, and the electrical discharges between neurons direct all our actions. The brain is constantly stimulated by external and internal signals, coming from the five senses or from an empty stomach for example, and neural networks are responsible for the transmission of this information.
A few facts
- There are 86 to 100 billion neurons in the human brain (10¹¹).
- The brain weighs between 1.3 and 1.4 kg.
- There are 200 million more neurons in the left hemisphere than in the right.
- Some impulses can reach speeds of 400 km/h.
- There are over 160,000 km of axons in the brain.
How does the brain learn?
The brain is bombarded with information, which it processes at lightning speed. Neurons communicate with each other through energy signals called nerve impulses, which trigger a chain reaction of synaptic activity. A signal strong enough to stimulate any given neuron triggers it to relay the message to the next neuron, and so on.
What is a neuron?
Let’s look at the biology of a neuron; this will help us understand the equations that we’ll be looking at later.
From left to right, the various parts of a neuron are represented thus:
The green circle points to a cellular body called the perikaryon. This is the nucleus of the neuron, the region that analyzes signals and sends them to the axon for transmission to other neurons.
The blue circle points to dendrites. Dendrites are limb-like extensions to the perikaryon, responsible for picking up external signals, determining their intensity, and relaying them.
The yellow circles indicate synapses, the physical contacts between neurons.
The pink circle shows the axon. This is an extension of the neuron that generates the nerve impulses, or action potential, to send to the next neuron. The intensity of the action potential determines whether the next neuron will also fire.
Nerve impulses
Neurons communicate through impulses, or action potential. These are very low-intensity electrical signals (about 100 millivolts). Transmission speeds can reach 150 metres per second.
A neuron fires when a nerve impulse reaches its nucleus through its dendrites. The cellular body absorbs this information and relays it to the axon through an electrical signal. The signal travels to the collateral axon then to the synapses, the site of inter-neural communication. If the signal is strong enough, the next neuron also fires, and so on. A single neuron can handle the nerve impulses of over 1,000 other neurons.
Neuroplasticity
At first, neurobiologists believed that links between neurons and the resulting networks were fixed and couldn’t be “rewired”. However, science has shown that brain connections are constantly evolving in response to many factors, including learning. This ability of the human brain to change throughout its life is called neuroplasticity and this recognition has transformed the field of neuroscience.
AI neurons
Let’s look at a digital neural network to see what happens in each of its parts, and how they can perform impressive feats by working together.
Input values
This is the equivalent of the information provided to neurons for processing. Each input is given a weight indicating its “importance” for processing purposes. Input values and their respective weights are used in the combination function.
Combination function
The combination function considers every input, with its unique value and weight.
Let’s take a basic example of a neuron: the perceptron. The combination function makes a simple equation factoring the weight and value of every input to determine if the result passes the activation function threshold, triggering the perceptron. As mentioned, the perceptron is the most straightforward way to represent “artificial intelligence”. The basic function of a perceptron is to perform a binary classification of an input.
Activation function
The activation function looks at the result of the combination function and determines whether the neuron needs triggering. There are several types of activation functions, and their relative merits are constantly examined and compared to improve the accuracy of neural networks.
Perceptrons: simplicity and limits
Though ingenious, perceptrons nevertheless have limited practical applications, since they are binary in nature, i.e. either positive or negative. Remember that at their inception, available computing power was a major issue in basic research.
Multi-layer perceptrons and non-linear results
With the passage of time and breakthroughs in computer science, new researchers built on their predecessors’ work and improved upon it. For example, they revisited Frank Rosenblatt’s work, developing and extending the basic concept as well as its theoretical and practical applications.
Perceptrons might only be used to solve linear problems, but later research combined perceptrons to provide greater flexibility and a wider field of application in problem resolution.
Based on neurobiological principles, computer scientists have posited the basic model of what is today called artificial intelligence. Neural networks create thousands of links between different layers, each link relaying a signal according to a given set of rules
In the next installment, we’ll analyze multilayer perceptrons, the various layers of neural networks and how they learn.
Final thoughts until the next installment…
Neural networks are rife with possibilities. Universities across the world and the GAFAM are working intently in the field, coming up with new ideas and breaking down barriers.
The development of artificial intelligence is a watershed moment in the history of computer science. Computers will no longer follow instructions blindly; they will take the initiative to figure out the best algorithm to solve any given problem.
