CNN feed forward or back propagtion model - neural-network

Is convolutional neural network (CNN) a feed forward model or back propagation model. I get this confusion by comparing the blog of DR.Yann and Wikipedia definition of CNN.

A convolutional neural net is a structured neural net where the first several layers are sparsely connected in order to process information (usually visual).
A feed forward network is defined as having no cycles contained within it. If it has cycles, it is a recurrent neural network. For example, imagine a three layer net where layer 1 is the input layer and layer 3 the output layer. A feed forward network would be structured by layer 1 taking inputs, feeding them to layer 2, layer 2 feeds to layer 3, and layer 3 outputs. A recurrent neural net would take inputs at layer 1, feed to layer 2, but then layer two might feed to both layer 1 and layer 3. Since the "lower" layer feeds its outputs into a "higher" layer, it creates a cycle inside the neural net.
Back propagation, however, is the method by which a neural net is trained. It doesn't have much to do with the structure of the net, but rather implies how input weights are updated.
When training a feed forward net, the info is passed into the net, and the resulting classification is compared to the known training sample. If the net's classification is incorrect, the weights are adjusted backward through the net in the direction that would give it the correct classification. This is the backward propagation portion of the training.
So a CNN is a feed-forward network, but is trained through back-propagation.

In short,
CNN is feed forward Neural Network.
Backward propagation is a technique that is used for training neural network.

Similar to tswei's answer but perhaps more concise.
A convolutional Neural Network is a feed forward nn architecture that uses multiple sets of weights (filters) that "slide" or convolve across the input-space to analyze distance-pixel relationship opposed to individual node activations.
Backward propagation is a method to train neural networks by "back propagating" the error from the output layer to the input layer (including hidden layers).

Related

Customized neural network Matlab

I am struggling to create a customized neural network in MatLab. I've made a sketch of my intended neural network.
To explain better how the network should work:
Two input parameters (features) connected to the first hidden layer with two neurons (exactly equal to the number of input parameters)
Each input parameter is connected to one neuron only.
No bias in the first hidden layer.
All the neurons in the first hidden layer are connected to the neurons in the second layer. There is a bias term in the second layer.
The neurons from the second hidden layer is connected to one output.
For simplicity, I did not show the projection functions in the plots.
Could somebody help me with creating this (probably) simple customized network?
I appreciate your help.
You want a feedforwardnet, in your example you have one layer of 3 neurons and an output layer but no bias on the neurons. For this you'll need to set up your network and change the net.biasConnect element
net = feedforwardnet(3);
net.biasConnect(1) = false;
view(net)
Once you train the network the rest will come into place.

Keras explanation: number of nodes in input layer

I'm trying to understand the relationship between a simple Perceptron and a neural network one gets when using the keras Sequence class.
I learned that the neural network perceptron looks as such:
Each "node" in the first layer is one of the features of a sample x_1, x_2,...,x_n
Could somebody explain the jump to the neural network I find in the Keras package below?
Since the input layer has four nodes, does that mean that network consists of four of the perceptron networks?
There is seem to be misunderstanding on what a perceptron is. A perceptron is a single unit that multiplies the inputs with weights, sums them up and applies an activation function:
Now the diagrams you have are called multi-layer perceptrons (MLP) and consist of a stack of perceptrons organised in layers, wiki. In Keras, there is no explicit notion of a perceptron but of a layer of perceptrons implemented as a Dense layer because the layers are densely connected, ie every output is connected to every input between layers. The second diagram would correspond to:
model = Sequential()
model.add(Dense(4, activation='sigmoid', input_dim=3))
model.add(Dense(4, activation='sigmoid'))
model.add(Dense(1, activation='sigmoid'))
assuming you have sigmoid activation. In this case, the input layer is implicit by specifying the input_dim=3 and the final layer would be the output layer.

What is a convolutional ply?

I want to recreate the result of this paper. They use the term convolutional ply for the neural network they apply on the audio spectogram. I am not sure I understand what a convolutional ply is, and how it differs from an ordinary convolutional neural network (cnn).
The paper states this as being the difference:
A convolution ply differs from a standard, fully connected hidden
layer in two important aspects, however. First, each convolutional
unit receives input only from a local area of the input. This means
that each unit represents some features of a local region of the
input. Second, the units of the convolution ply can themselves be
organized into a number of feature maps, where all units in the same
feature map share the same weights but receive input from different
locations of the lower layer
Which to me sound like a ordinary cnn network. What is the difference?

What's the difference between convolutional and recurrent neural networks? [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about programming within the scope defined in the help center.
Closed 2 years ago.
Improve this question
I'm new to the topic of neural networks. I came across the two terms convolutional neural network and recurrent neural network.
I'm wondering if these two terms are referring to the same thing, or, if not, what would be the difference between them?
Difference between CNN and RNN are as follows:
CNN:
CNN takes a fixed size inputs and generates fixed-size outputs.
CNN is a type of feed-forward artificial neural network - are variations of multilayer perceptrons which are designed to use minimal amounts of preprocessing.
CNNs use connectivity pattern between its neurons and is inspired by the organization of the animal visual cortex, whose individual neurons are arranged in such a way that they respond to overlapping regions tiling the visual field.
CNNs are ideal for images and video processing.
RNN:
RNN can handle arbitrary input/output lengths.
RNN unlike feedforward neural networks - can use their internal memory to process arbitrary sequences of inputs.
Recurrent neural networks use time-series information. i.e. what I spoke last will impact what I will speak next.
RNNs are ideal for text and speech analysis.
Convolutional neural networks (CNN) are designed to recognize images. It has convolutions inside, which see the edges of an object recognized on the image. Recurrent neural networks (RNN) are designed to recognize sequences, for example, a speech signal or a text. The recurrent network has cycles inside that implies the presence of short memory in the net. We have applied CNN as well as RNN choosing an appropriate machine learning algorithm to classify EEG signals for BCI: http://rnd.azoft.com/classification-eeg-signals-brain-computer-interface/
These architectures are completely different, so it is rather hard to say "what is the difference", as the only thing in common is the fact, that they are both neural networks.
Convolutional networks are networks with overlapping "reception fields" performing convolution tasks.
Recurrent networks are networks with recurrent connections (going in the opposite direction of the "normal" signal flow) which form cycles in the network's topology.
Apart from others, in CNN we generally use a 2d squared sliding window along an axis and convolute (with original input 2d image) to identify patterns.
In RNN we use previously calculated memory. If you are interested you can see, LSTM (Long Short-Term Memory) which is a special kind of RNN.
Both CNN and RNN have one point in common, as they detect patterns and sequences, that is you can't shuffle your single input data bits.
Convolutional neural networks (CNNs) for computer vision, and recurrent neural networks (RNNs) for natural language processing.
Although this can be applied in other areas, RNNs have the advantage of networks that can have signals travelling in both directions by introducing loops in the network.
Feedback networks are powerful and can get extremely complicated. Computations derived from the previous input are fed back into the network, which gives them a kind of memory. Feedback networks are dynamic: their state is changing continuously until they reach an equilibrium point.
First, we need to know that recursive NN is different from recurrent NN.
By wiki's definition,
A recursive neural network (RNN) is a kind of deep neural network created by applying the same set of weights recursively over a structure
In this sense, CNN is a type of Recursive NN.
On the other hand, recurrent NN is a type of recursive NN based on time difference.
Therefore, in my opinion, CNN and recurrent NN are different but both are derived from recursive NN.
This is the difference between CNN and RNN
Convolutional Neural NEtwork:
In deep learning, a convolutional neural network (CNN, or ConvNet) is a class of deep neural networks, most commonly applied to analyzing visual imagery. ... They have applications in image and video recognition, recommender systems, image classification, medical image analysis, and natural language processing.
Recurrent Neural Networks:
A recurrent neural network (RNN) is a class of artificial neural networks where connections between nodes form a directed graph along a temporal sequence. This allows it to exhibit temporal dynamic behavior. Unlike feedforward neural networks, RNNs can use their internal state (memory) to process sequences of inputs.
It is more helpful to describe the convolution and recurrent layers first.
Convolution layer:
Includes input, one or more filters (as well as subsampling).
The input can be one-dimensional or n-dimensional (n>1), for example, it can be a two-dimensional image. One or more filters are also defined in each layer. Inputs are convolving with each filter. The method of convolution is almost similar to the convolution of filters in image processing. In general, the purpose of this section is to extract the features of each filter from the input. The output of each convolution is called a feature map.
For example, a filter is considered for horizontal edges, and the result of its convolution with the input is the extraction of the horizontal edges of the input image. Usually, in practice and especially in the first layers, a large number of filters (for example, 60 filters in one layer) are defined. Also, after convolution, the subsampling operation is usually performed, for example, their maximum or average of each of the two neighborhood values ​​is selected.
The convolution layer allows important features and patterns to be extracted from the input. And delete input data dependencies (linear and nonlinear).
[The following figure shows an example of the use of convolutional layers and pattern extraction for classification.][1]
[1]: https://i.stack.imgur.com/HS4U0.png [Kalhor, A. (2020). Classification and Regression NNs. Lecture.]
Advantages of convolutional layers:
Able to remove correlations and reduce input dimensions
Network generalization is increasing
Network robustness increases against changes because it extracts key features
Very powerful and widely used in supervised learning
...
Recurrent layers:
In these layers, the output of the current layer or the output of the next layers can also be used as the input of the layer. It also can receive time series as input.
The output without using the recurrent layer is as follows (a simple example):
y = f(W * x)
Where x is input, W is weight and f is the activator function.
But in recurrent networks it can be as follows:
y = f(W * x)
y = f(W * y)
y = f(W * y)
... until convergence
This means that in these networks the generated output can be used as an input and thus have memory networks. Some types of recurrent networks are Discrete Hopfield Net and Recurrent Auto-Associative NET, which are simple networks or complex networks such as LSTM.
An example is shown in the image below.
Advantages of Recurrent Layers:
They have memory capability
They can use time series as input.
They can use the generated output for later use.
Very used in machine translation, voice recognition, image description
...
Networks that use convolutional layers are called convolutional networks (CNN). Similarly, networks that use recurrent layers are called recurrent networks. It is also possible to use both layers in a network according to the desired application!

Elman and Jordan context values during training for neural network

I am experimenting with Elman and/or Jordan ANN's using the Encog framework. I am trying to code my own but studying how Encog has it implemented. I see how the backpropagation through time updates the weights, but how do the context neurons get updated? There values seem to fluctuate somewhat randomly as the output of the neural network is calculated. How is it that these values actually allow the simple recurrent neural network to actually recognize patterns in input data over time?
The context neurons neuron values themselves are not updated as training progresses. The weights between them and the next layer ARE updated during training. The context values will be updated as the neural network runs.