Introduction To The Basics Of Neural Networks - Reshared Knol

Author: Christian Eder,

Software-Engineer

Munich, Germany

The human brain is a complex structure consisting of billions of cells, the neurons. These neurons are connected by synapses. The neurons communicate with each other by sending electric pulses to their neighbouring neurons. Since the synapses can have different sizes, these electric pulses can be of varying power. The electricity floating through our brain makes us remember things, react on outside influences or makes our muscles move.

This already simplified image of our brain has been adopted by artificial neural networks. They are a kind of abstract, "mathematical" brain.

Neural networks could be formally described as a weighted graph G = (V, E). The set of vertices V can be seen as the neurons in the network. The edges E, that connect the vertices V are the synapses connecting the neurons. And the weights W of the edges describe the thickness or size of the synapses. A subset of vertices are called input neurons, and another subset of V is called output neurons. All vertices that are not part of the input or output subsets, are called "hidden" neurons.

The picture shows a simple, so called feedforward network - the neurons are only connected from one layer to the next. There exist more complex architectures for neural networks, e.g. with recurrent connections going from the hidden layer back to the input layer, but these networks are not discussed by this article, which wants to give an overview about how artificial neural networks work generally.

The electric pulses which are generated by the neurons in our brain are represented as numbers in neural networks - the higher the number, the higher the voltage.

The input layer of a neural network is where the electric pulses enter the network. In nature, an example is the set of neurons connected to our eyes through the optic nerve. The output layer can be seen as a set of neurons connected to actors like our legs - the eyes send an electric pulse to the input neurons, and our legs move because the output neurons tell them that there is a wall in front of us.

Artificial neural net works can be used to learn nonlinear functions - in other words, they can learn a mapping

where n is the number of input neurons and m the number of output neurons. Each of the neurons has an input and an output value. The values in an n-dimensional input vector are handed over to the input layer, and the output values of the i-th input layer neuron is set to the value in the i-th dimension of the input vector. The neurons in the hidden layer compute their input as the weighted sum of all ingoing connections from the input layer: they sum up all output values of the input neurons they are connected with, and multiply each of these values with the weight of the connecting synapse. After computing their input, the hidden layer neurons compute their output by applying an function to their input. The neurons in the output layer compute their input and output analog.

Formally, each of the neurons works as follows:

The weights of the synapses, combined with the structure of the net (number of neuons, number of neurons per layer, ...) determines the mapping that the neural network represents. Thus, learning a specific mapping means learning the correct weights of the synapses. There are several algorithms that can be used to accomplish this. One of them is called "error backpropagation".

When using error backpropagation, teaching the network works as follows:

An input vector is given to the net, and the correct output vector expected from the net is known. The actual output vector of the net is compared with the expected, and the difference vector between both defines the error vector. This error vector is set to the corresponding output neurons, and the error is propagated backwards through the net, in order to find out, how the synapse's weights would have to be changed to minimize the error. This gradient based approach needs the activation functions of all neurons to be differentiable. A concrete implementation of the backpropagation algorithm can be found on Wikipedia.

Source Knol: http://knol.google.com/k/introduction-to-the-basics-of-neural-networks#

Knol Nrao - 5192

Ud. 16.1.2024

Pub. 29.4.2012

Artificial Neural Networks - Introduction

Author: Marek Libra

Posted under creative commons from Knol

The Artificial Neural Network (NN later) is a topic in artificial intelligence methods and techniques.  It was successfully applied in a wide range of problem domains like finance, engineering, medicine, geology, physics or control.

Neural networks are useful especially for solving problems of prediction, classification or control. They are also a good alternative to classical statistical approaches like regression analysis.

The artificial neural networks techniques were developed based on the model of biological neural networks. The biological neural networks are the basis of functioning of the nervous system of biological organisms. This inspiration is commonly known fact and it is mentioned in most of neural networks publications.

 The NN is built from a large number of simple processing units called artificial neurons (called
just neurons later).

The interface of an artificial neuron stays from n numeric inputs and one numeric output. Some models of neurons consider one next special input called bias. Each  input is evaluated by its numeric weight. The neuron can perform two operations: compute and adapt.

The compute operation transforms inputs to output. The compute operation takes numerical
inputs and computes their weighted sum. It performs a so called activation function to this sum
(a mathematical transformation) afterwards. The result of the activation function is set as a value
to the output interface.

The adapt operation, based on a pair of inputs and awaited outputs specified by the user,
tunes the weights of an NN for a better approximation of the computed output compared to
the awaited output for considered input.

The neurons in an NN are ordered and numerically signed (from N) according to the order.
    A lot of models of NNs are known. These models differs to each other by different usage of

    • the domain of numeric input, output and weights (real, integer or finite set like {0,1}),

    • the presence of bias (yes or no),

    • the definition of an activation function (sigmoid, hyperbolic tangents, discrete threshold,
      etc),

    • the topology of interconnected neurons (feed-forward or recurrent),

    • the ability to change the number of neurons or the network topology during the lifetime of
      the network,

    • the algorithm of the computation flow through the network over neurons,

    • the simulation time (discrete or continuous) or

    • the adaptation algorithm (none, back propagation, perceptron rule, genetic, simulated annealing etc.).

 A good taxonomy of NN models can be found i.e. in [1]

More detailed general descriptions, which are formal and well readable, can be found in [2] .

References

• [1] Šíma and P. Orponen. General purpose computation with neural
  

• [2] David M Skapura. Building Neural Networks. Addison-Wesley, 1995

Source Knol: /knol.google.com/   marek-libra/artificial-neural-networks/5rqq7q8930m0/12#
Knol Nrao - 5193

Further Reading

Simple mathematical steps in Neural Network problem solving

https://brilliant.org/wiki/feedforward-neural-networks/

Mode detailed reading

Artificial Neural Networks: Mathematics of Backpropagation (Part 4)
October 28, 2014 in ml primers, neural networks
http://briandolhansky.com/blog/2013/9/27/artificial-neural-networks-backpropagation-part-4

But what is a neural network? | Chapter 1, Deep learning

9,774,444 views 5 Oct 2017 - 19 sep 2021

https://www.youtube.com/watch?v=aircAruvnKk

Further Reading

Knols

• Feed-Forward Neural Networks
• Adaptation of Feed-Forward Neural Networks
• The Perceptron Rule
• The Back Propagation

Updated 21 November 2018,  8 June 2017, 8 May 2017, 28 April 2012.

Feed Forward Neural Networks

Author: Marek Libra

A brief note prior reading: I suggest to read introduction to Artificial Neural Networks before reading that Knol.

---

The Feed Forward Neural Networks (FFNNs) represent a subclass of perceptron networks.Same as other perceptron networks, the FFNNs are computed with discrete time. The models using the discrete time are used for solving practical problems very often due to generally good theoretical understanding of those models and especially due to their ability to be easily simulated on von Neumann machines.

A feed-forward network is an acyclic graph of neurons unambiguously signed (weighted) by numbers from R (real numbers). The neurons are grouped into a sequence of d + 1 pair wise disjoint layers. The neurons are connected only to the neurons in a subsequent or previous layer.

The layers are called input, output and hidden in a feed-forward NN. The input and output layers must be present exactly once per each in a network. These layers build the network interface. The hidden layers are optional and there can be more than one.

The number of layers excluding the input one is called the depth and equals to the d. The connection (sometimes called synapse) between neuron i and j is evaluated by the weight w_ij . The w_ij = 0 means there is no connection from the i to the j neuron.

FFNNs are notated according to counts of neurons in their layers. For example the 3-1-2-4 FFNN notation points to an FFNN with 3 input neurons, 1 neuron in the first hidden layer, 2 neurons in the second hidden layer and 4 output neurons, as depicted on the following figure:

The computation of an FFNN is done gradually, layer by layer, where the outputs of one layer neurons serve as the inputs for the subsequent layer neurons.
Initially, the states (means outputs) of neurons in the input layer are set to represent the external input.
The states of neurons in a subsequent hidden or output layer are computed in parallel. The neuron computing its output takes the output of the preceding neurons as its input. The neuron j computes a weighted sum of the inputs called the inner potential ε _j ∈ R:

where the j← is a set of input neurons for neuron j (means from preceding layer) and y _i is the output of the neuron i.
The nonlinearity is introduced to an NN using an activation function σ computed on the inner potential ε:

σ : R −→ (0, 1).

The activation function must be differentiable on R or at least on a subset of R which is used by the neurons to solve a particular problem. The differentiability is necessary especially for the back-propagation algorithm.

The commonly used activation function is the standard sigmoid:

The λ is a parameter of this function used for optimization of the network’s behavior. The λ determines the function steepness and its value can differ for neurons in an NN in general.

The actual computation of σ is problem-specific, but the standard sigmoid on the interval (0, 1) is commonly used for computation of σ(ε).
Another also popular activation function seems to be the hyperbolic tangents, where σ = tangh:

A discussion about the activation function’s behavior can be found in [1] .

Further Reading

Knol

• Artificial Neural Networks
• Adaptation of Feed-Forward Neural Networks (contains farther links)
  
• My Knol Directory

External

• [1] David M Skapura. Building Neural Networks. Addison-Wesley, 1995.

Source Url: Feed Forward Neural Networks

Knol - 5195

Adaptation of Feed Forward Neural Networks

Author: Marek Libra

A brief note prior reading of this Knol: I suggest to read

• introduction to Artificial Neural Networks
  
• and Feed Forward Neural Networks

---

There are two main purposes for considering artificial NNs
    • the study of biological decision mechanisms (like human brain) and/or
    • the machine learning paradigm useful for implementing of decision logic in artificial agents or as a tool for statistical analysis as an alternative to classical approaches.

The main advantage of the NN paradigm compared to the deterministic Turing-like approaches are in absence of precise description for solving a particular problem like it is done in some general-purpose programming language (i.e. C++).

On the contrary, we consider the supervised learning where a pair of a training set and an FFNN architecture is used to describe the problem which is solved by the FFNN. The training set stays of pairs (given input, awaited output). These pairs are called patterns. The training set is provided by the FFNN user (human or machine).

For the completeness, the training set is not mandatory for computation with FFNNs. The adaptation function of an FFNN can be omitted (never used). Well specified weights for an FFNN are sufficient for proper computation of a network. Unfortunately, it is very difficult to properly set the weights by the NN designer.

For this reason, there are introduced multiple learning mechanisms which can adapt the weights automatically. Each the learning algorithm has different ability to provide generalization to the network.

The ability of network to solve a particular problem is measured by the network error on a set of patterns L of size p:

where the E_k is called the partial error of network on the pattern k:

where the Y is a set of output neurons (determined by the FFNN topology), the y _j (x_k ) is the computed output of the neuron j after network computation on the input vector x_k . The d_kj is the awaited output of the neuron j after network computation on the input x _k .

The learning algorithm adapts the weights to minimize the network error on a training set. However the training set is used for the error evaluation during the adaptation, the testing set is used to evaluate the efficiency of the network for solving the problem. More precisely, the testing set stays from patterns similarly as the training set. The testing set can be disjoint to the training set. This simple methodology shows better the network capability to handle an unknown input.

The goal of the adaptation process must not be necessarily in the zero network error on the training set but in a reasonable low error on the testing set. Moreover, a low value of a network error must not necessarily mean a good network for solving a problem. There is a big issue in the over training, when the network doesn’t generalize the given data but it just memorizes them. Such an over trained network lacks the ability to classify correctly the output on newly (not previously given) input. Some aspects of the over training issue are mentioned in [1].

Further Reading

Knols

• Artificial Neural Networks
• Feed-Forward Neural Networks
• The Perceptron Rule
• The Back Propagation
• My Knol Directory

External

• [1] S. Amari, N. Murata, K.-R. M ̈ ller, M. Finke, and H. H. Yang. Asymptotic statistical theory of overtraining and cross-validation. IEEE Transactions on Neural Networks, 8(5):985–996, September 1997.
  

Source Url:  http://knol.google.com/k/marek-libra/adaptation-of-feed-forward-neural/5rqq7q8930m0/15#

Knol Nrao - 5196

The Back Propagation - Algorithm

Author: Marek Libra

A brief note prior reading of this Knol: I suggest to read

• introduction to Artificial Neural Networks
  
• and Feed Forward Neural Networks
• and Adaptation of Feed-Forward Neural Networks
• and The Perceptron Rule
  

---

The Back Propagation (called BP later) is an algorithm used for adaptation of a multi-layered feed-forward neural network. It is based on the perceptron rule, but propagating the effect of the overall network error into inner parts of the network.

The network error computation on the training set is based on the complicated nonlinear function. It is a classical hard optimization problem to minimize the network error value and the BP tries to deal with it.

The BP uses gradient descent method which demands the differentiability of the network error function. Obviously, the differentiability of the network error function relies on the differentiability of the activation function (due to the feed-forward computation mechanism described in the preceeding Adaptation of Feed-Forward Neural Networks Knol).

The Knol you are reading aims for the standard sigmoid as the activation function in the description of the following equations. This function is commonly used in practical experiments.
                         
Initially, the weight w_ij⁰ from the neuron i to the neuron j is set randomly for each interconnected neurons i and j at the time t=0.
The adaptation is computed in a loop of discrete steps (begin at time t = 1, increment t in each subsequent step).

The weights w_t of time t ensue from the weights w_t−1 of time t−1 by incrementing a negative gradient of the network error at the point w_t−1 multiplied by a learning speed parameter e ∈ R:

Eq. 1

The parameter e is set accordingly in interval 0 < e < 1 by the user or by some supporting algorithm. Its value can be changed during the learning process to tune the adaptation behavior (learning speed or skipping over local extremes).

The gradient

in Eq. 1 is computed from

Eq. 4

The function of the network can be considered as a composite function of neurons. We can differentiate the gradient in previous equotation using the rule for a composite function:

Eq. 2

By usage of standard sigmoid and inner potential definition (described in Feed-Forward Neural Networks):

Eq. 3

Next, the back propagation strategy is used for computation of the partial derivation:

• for a neuron j in output layer:
  
• for a neuron j in hidden layer:
  

where j^→ is a set of neurons with j as its input.

Same as other gradient descent methods, the main disadvantage of the BP is in locating of a local
minimum. It is very difficult to reach the global minimum with such approaches. Some heuristics
based a stochastic extension of BP or on tuning of the learning speed parameter exist but they
are not optimal in general.

Following simplified BP algorithm is provided for better description. The ∆E is a matrix used for step-by-step computation
of the network error gradient.

The claim to discovery of the BP is controversial. Initially, it was accepted to have been discovered independently by Rumelhart, Hinton and Williams (1986), Le Cun (1985), and Parker (1985). But it was mainly the work by Rumelhart et. al. who made the model popular. He published his original article as [1].

Further Reading

Knols

• Artificial Neural Networks
• Feed-Forward Neural Networks
• Adaptation of Feed-Forward Neural Networks
• The Perceptron Rule
• My Knol Directory

External

• [1] D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning internal representations by error propagation. In Parallel Distributed Processing: Explorations in the Microstructure of Cognition. Bradford Books/MIT Press, Cambridge, 1986.
  

Source Knol: The Back Propagation

Knol Narayana rao - 5198

Artificial Neural Networks - Interesting Web Pages

1. 15 Steps to implement a neural network

http://code-spot.co.za/2009/10/08/15-steps-to-implemented-a-neural-net/
Good explanation, simple mathematical functions and some code to use are given

Neural Networks - Introduction - Video Lecture

An interesting introductory lecture

http://www.youtube.com/watch?v=6ixqKw7uK6o

Artificial Neural Networks - Video Lectures - Playlist

_______________ _______________
Play list

http://www.youtube.com/watch?v=xbYgKoG4x2g&playnext=1&list=PL3EA65335EAC29EE8