All You Need to Know About Neural Networks — Part 2

  • The Evolution of Neural Networks
  • Sensor Models
  • Feed-forward Neural Networks
  • Back-propagation
  • Deep Learning Basics

5. Deep Learning Basics

From the second low point in development in 1990s to 2006, neural networks once again entered the consciousness of the masses, this time in even more force than before. A monumental occurrence during this rise of neural networks was the two theses on multi-layer neural networks (now called “deep learning”) submitted by Hinton in a number of places including Salahundinov.

Convolutional Neural Networks

There are two core concepts to Convolutional Neural Networks. One is convolution and the other is pooling. At this point, some may ask why we don’t simply use feed-forward neural networks rather than CNN. Taking a 1000x1000 image for example, a neural network would have 1 million nodes on the hidden layer. A feed-forward neural network, then, would have 10¹² parameters. At this point it’s nearly impossible for the system to learn since it would require an absolutely massive number of estimations.

Recursive Neural Networks

As for the foundational principles behind recursive neural networks, we can see from the image below that the output from such a network relies not only on output x, but the status of the hidden layer, which is updated according to the previous input x. The expanded image shows the entire process. The hidden layer from the first input is S(t-1), which influences the next input, X(t). The main advantage of the recursive neural network model is that we can use it in sequential data operations like text, language, and speech where the state of the current data is influenced by previous data states. This type of data is very difficult to handle using a feed-forward neural network.


In this article, we talked about the evolution of neural networks and introduced several basic concepts and approaches in this field. The above article is based on a speech delivered by Dr. Sun Fei at the annual Alibaba Cloud Computing Conference (speech in Chinese). He is currently involved in researching recommendation systems and methods of text generation.



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Alibaba Cloud

Alibaba Cloud

Follow me to keep abreast with the latest technology news, industry insights, and developer trends. Alibaba Cloud website: