Related
I am novice to deep learning so I begin with the simplest test case: XOR learning.
In the new edition of Digital Image Processing by G & W the authors give an example of XOR learning by a deep net with 3 layers: input, hidden and output (each layer has 2 neurons.), and a sigmoid as the network activation function.
For network initailization they say: "We used alpha = 1.0, an inital set of Gaussian random weights of zero mean and standard deviation of 0.02" (alpha is the gradient descent learning rate).
Training was made with 4 labeled examples:
X = [1 -1 -1 1;1 -1 1 -1];%MATLAB syntax
R = [1 1 0 0;0 0 1 1];%Labels
I have written the following MATLAB code to implement the network learing process:
function output = neuralNet4e(input,specs)
NumPat = size(input.X,2);%Number of patterns
NumLayers = length(specs.W);
for kEpoch = 1:specs.NumEpochs
% forward pass
A = cell(NumLayers,1);%Output of each neuron in each layer
derZ = cell(NumLayers,1);%Activation function derivative on each neuron dot product
A{1} = input.X;
for kLayer = 2:NumLayers
B = repmat(specs.b{kLayer},1,NumPat);
Z = specs.W{kLayer} * A{kLayer - 1} + B;
derZ{kLayer} = specs.activationFuncDerive(Z);
A{kLayer} = specs.activationFunc(Z);
end
% backprop
D = cell(NumLayers,1);
D{NumLayers} = (A{NumLayers} - input.R).* derZ{NumLayers};
for kLayer = (NumLayers-1):-1:2
D{kLayer} = (specs.W{kLayer + 1}' * D{kLayer + 1}).*derZ{kLayer};
end
%Update weights and biases
for kLayer = 2:NumLayers
specs.W{kLayer} = specs.W{kLayer} - specs.alpha * D{kLayer} * A{kLayer - 1}' ;
specs.b{kLayer} = specs.b{kLayer} - specs.alpha * sum(D{kLayer},2);
end
end
output.A = A;
end
Now, when I am using their setup (i.e., weights initalizaion with std = 0.02)
clearvars
s = 0.02;
input.X = [1 -1 -1 1;1 -1 1 -1];
input.R = [1 1 0 0;0 0 1 1];
specs.W = {[];s * randn(2,2);s * randn(2,2)};
specs.b = {[];s * randn(2,1);s * randn(2,1)};
specs.activationFunc = #(x) 1./(1 + exp(-x));
specs.activationFuncDerive = #(x) exp(-x)./(1 + exp(-x)).^2;
specs.NumEpochs = 1e4;
specs.alpha = 1;
output = neuralNet4e(input,specs);
I'm getting (after 10000 epoches) that the final output of the net is
output.A{3} = [0.5 0.5 0.5 0.5;0.5 0.5 0.5 0.5]
but when I changed s = 0.02; to s = 1; I got output.A{3} = [0.989 0.987 0.010 0.010;0.010 0.012 0.0.98 0.98] as it should.
Is it possible to get these results with `s=0.02;' and I am doing something wrong in my code? or is standard deviation of 0.02 is just a typo?
Based on your code, I don't see any errors. In my knowledge, the result that you got,
[0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5]
That is a typical result of overfitting. There are many reasons for this to happen, such as too many epochs, too large learning rate, too small sample data, and others.
On your example, s=0.02 limits the values of randomized weights and biases. Changing that to s=1 makes the randomized values unchanged/unscaled.
To make the s=0.02 one work, you can try minimizing the number of epochs or maybe lowering the alpha.
Hope this helps.
In my research area (meteorology) graphs within graphs are commonly produced.
more information about it can be found here.
Each of those lines joints up data points that have:
An x-value, between 0 and 1 (values greater than 1 should not be represented in the graph).
A y-value, between 0 and 1.
A PSS value, between 1 and -1.
A Frequency Bias value, ranging from 0 to +∞, but values higher than 4 are not displayed.
A False Alarm Ratio (FAR) value, ranging from 0.0 to 0.9. The False Alarm Ratio value is held constant at a particular value for each data point on any given line.
EDIT: To make things really concrete, I've drawn a pink dot on the graph. That dot represents a data point for which x=0.81, y=0.61, PSS=-0.2, B=3.05, FAR=0.8.
I am trying to reproduce something similar in MATLAB. Googling turned up a lot of answers like this, which feature inset figures rather than what I'm looking for.
I have the data organized in a 3D array, where each page refers to a different level of False Alarm Ratio. The page with a FAR of 0.8 (data here) starts out like this
Then there are other pages on the 3D array devoted to FARs of 0.7, 0.6, and so on.
Questions
1. Is it even possible to create such an graph in MATLAB?
2. If so, what function should I use, and what approach should I take? EDIT: I have working code (below) that creates a somewhat similar figure using the linear plot function, but the documentation for this function does not indicate any way to insert a graph inside another graph. I am not sure how helpful this code is, but have inserted it in response to the downvoter.
H = [0:0.01:1];
figure; hold on
fill([0 1 1],[0 0 1],[0 0.2 0.4]) % Deep blue
fill([0 1 0],[0 1 1],[0.4 0 0]) % Purple
low_colours = {[0 0.501 1],[0 0.8 0.4], [0.4 0.8 0], [0.8 0.8 0]};
high_colours = {[0.6 0 0],[0.8 0 0], [1 0.5019 0], [0.988 0.827 0.196]};
colour_counter = 0;
for ii = -0.8:0.2:0
colour_counter = colour_counter + 1;
if colour_counter < 5
colour_now = low_colours{colour_counter};
end
ORSS = ones(1,size(H,2))*ii;
F = (H .* (1-ORSS)) ./ ((1-2.*H) .* ORSS + 1);
plot(F,H)
fill(F,H,colour_now);
end
colour_counter = 0;
for ii = 0.8:-0.2:0
colour_counter = colour_counter + 1;
if colour_counter < 5
colour_now = high_colours{colour_counter};
end
ORSS = ones(1,size(H,2))*ii;
F = (H .* (1-ORSS)) ./ ((1-2.*H) .* ORSS + 1);
plot(F,H)
fill(F,H,colour_now);
end
I think I got what you want, but before you go to the code below, notice the following:
I didn't need any of your functions in the link (and I have no idea what they do).
I also don't really use the x and y columns in the data, they are redundant coordinates to the PSS and B.
I concat all the 'pages' in your data to one long table (FAR below) with 5 columns (FAR,x,y,PSS,FB).
If you take closer look at the data you see that some areas that supposed to be colored in the graph has no representation in it (i.e. no values). So in order to interpolate the color to there we need to add the corners:
FAR{end+1,:} = [0.8 0 0 0 4];
FAR{end+1,:} = [0.9 0 0 -0.66 3.33];
FAR{end+1,:} = [1 0 0 0 0];
FAR{end+1,:} = [1 0 0 -1 3];
Next, the process has 2 parts. First we make a matrix for each variable, that ordered in columns by the corresponding FAR value, so for instance, in the PSS matrix the first column is all PSS values where FAR is 0, the second column is all PSS values where FAR is 0.1, and so on. We make such matrices for FAR(F), PSS and FreqBias(B), and we initialize them with NaNs so we can have columns with different number of values:
F = nan(max(histcounts(FAR.FAR,10)),10);
PSS = F;
B = F;
c = 1;
f = unique(FAR.FAR).';
for k = f
valid = FAR.FAR==k & FAR.x<=1;
B(1:sum(valid),c) = FAR.FB(valid);
B(sum(valid):end,c) = B(sum(valid),c);
PSS(1:sum(valid),c) = FAR.PSS(valid);
PSS(sum(valid):end,c) = PSS(sum(valid),c);
F(:,c) = k;
c = c+1;
end
Then we set the colors for the colormap (which I partially took from you), and set the labels position:
colors = [0 0.2 0.4
0 0.501 1;
0 0.8 0.4;
0.4 0.8 0;
0.8 0.8 0;
0.988 0.827 0.196;
1 0.5019 0;
0.8 0 0;
0.6 0 0.2;
0.4 0.1 0.5];
label_pos =[0.89 0.77
1.01 0.74
1.14 0.69
1.37 0.64
1.7 0.57
2.03 0.41
2.65 0.18
2.925 -0.195
2.75 -0.55];
And we use contourf to plot everything together, and set all kind of properties to make it look good:
[C,h] = contourf(B,PSS,F);
xlim([0 4])
ylim([-1 1])
colormap(colors)
caxis([0 1])
xlabel('Frequency Bias B')
ylabel('Pierce Skill Score PSS')
title('False Alarm Ratio')
ax = h.Parent;
ax.XTick = 0:4;
ax.YTick = -1:0.5:1;
ax.FontSize = 20;
for k = 1:numel(f)-2
text(label_pos(k,1),label_pos(k,2),num2str(f(k+1)),...
'FontSize',12+k)
end
And here is the result:
Getting the labels position:
If you wonder what is a fast way to obtain the variable label_pos, then here is how I made it...
You run the code above without the last for loop. Then you run the following code:
clabel(C,'manual')
f = gcf;
label_pos = zeros(numel(f.Children.Children)-1,2);
for k = 1:2:size(label_pos,1)
label_pos(k,:) = f.Children.Children(k).Position(1:2);
end
label_pos(2:2:size(label_pos,1),:) = [];
After the first line the script will pause and you will see this message in the command window:
Carefully select contours for labeling.
When done, press RETURN while the Graph window is the active window.
Click on the figure where you want to have a label, and press Enter.
That's it! Now the variable label_pos has the positions of the labels, just as I used it above.
Here is a question about whether we can use vectorization type of operation in matlab to avoid writing for loop.
I have a vector
Q = [0.1,0.3,0.6,1.0]
I generate a uniformly distributed random vector over [0,1)
X = [0.11,0.72,0.32,0.94]
I want to know whether each entry of X is between [0,0.1) or [0.1,0.3) or [0.3,0.6), or [0.6,1.0) and I want to return a vector which contains the index of the maximum element in Q that each entry of X is less than.
I could write a for loop
Y = zeros(length(X),1)
for i = 1:1:length(X)
Y(i) = find(X(i)<Q, 1);
end
Expected result for this example:
Y = [2,4,3,4]
But I wonder if there is a way to avoid writing for loop? (I see many very good answers to my question. Thank you so much! Now if we go one step further, what if my Q is a matrix, such that I want check whether )
Y = zeros(length(X),1)
for i = 1:1:length(X)
Y(i) = find(X(i)<Q(i), 1);
end
Use the second output of max, which acts as a sort of "vectorized find":
[~, Y] = max(bsxfun(#lt, X(:).', Q(:)), [], 1);
How this works:
For each element of X, test if it is less than each element of Q. This is done with bsxfun(#lt, X(:).', Q(:)). Note each column in the result corresponds to an element of X, and each row to an element of Q.
Then, for each element of X, get the index of the first element of Q for which that comparison is true. This is done with [~, Y] = max(..., [], 1). Note that the second output of max returns the index of the first maximizer (along the specified dimension), so in this case it gives the index of the first true in each column.
For your example values,
Q = [0.1, 0.3, 0.6, 1.0];
X = [0.11, 0.72, 0.32, 0.94];
[~, Y] = max(bsxfun(#lt, X(:).', Q(:)), [], 1);
gives
Y =
2 4 3 4
Using bsxfun will help accomplish this. You'll need to read about it. I also added a Q = 0 at the beginning to handle the small X case
X = [0.11,0.72,0.32,0.94 0.01];
Q = [0.1,0.3,0.6,1.0];
Q_extra = [0 Q];
Diff = bsxfun(#minus,X(:)',Q_extra (:)); %vectorized subtraction
logical_matrix = diff(Diff < 0); %find the transition from neg to positive
[X_categories,~] = find(logical_matrix == true); % get indices
% output is 2 4 3 4 1
EDIT: How long does each method take?
I got curious about the difference between each solution:
Test Code Below:
Q = [0,0.1,0.3,0.6,1.0];
X = rand(1,1e3);
tic
Y = zeros(length(X),1);
for i = 1:1:length(X)
Y(i) = find(X(i)<Q, 1);
end
toc
tic
result = arrayfun(#(x)find(x < Q, 1), X);
toc
tic
Q = [0 Q];
Diff = bsxfun(#minus,X(:)',Q(:)); %vectorized subtraction
logical_matrix = diff(Diff < 0); %find the transition from neg to positive
[X_categories,~] = find(logical_matrix == true); % get indices
toc
Run it for yourself, I found that when the size of X was 1e6, bsxfun was much faster, while for smaller arrays the differences were varying and negligible.
SAMPLE: when size X was 1e3
Elapsed time is 0.001582 seconds. % for loop
Elapsed time is 0.007324 seconds. % anonymous function
Elapsed time is 0.000785 seconds. % bsxfun
Octave has a function lookup to do exactly that. It takes a lookup table of sorted values and an array, and returns an array with indices for values in the lookup table.
octave> Q = [0.1 0.3 0.6 1.0];
octave> x = [0.11 0.72 0.32 0.94];
octave> lookup (Q, X)
ans =
1 3 2 3
The only issue is that your lookup table has an implicit zero which be fixed easily with:
octave> lookup ([0 Q], X) # alternatively, just add 1 at the results
ans =
2 4 3 4
You can create an anonymous function to perform the comparison, then apply it to each member of X using arrayfun:
compareFunc = #(x)find(x < Q, 1);
result = arrayfun(compareFunc, X, 'UniformOutput', 1);
The Q array will be stored in the anonymous function ( compareFunc ) when the anonymous function is created.
Or, as one line (Uniform Output is the default behavior of arrayfun):
result = arrayfun(#(x)find(x < Q, 1), X);
Octave does a neat auto-vectorization trick for you if the vectors you have are along different dimensions. If you make Q a column vector, you can do this:
X = [0.11, 0.72, 0.32, 0.94];
Q = [0.1; 0.3; 0.6; 1.0; 2.0; 3.0];
X <= Q
The result is a 6x4 matrix indicating which elements of Q each element of X is less than. I made Q a different length than X just to illustrate this:
0 0 0 0
1 0 0 0
1 0 1 0
1 1 1 1
1 1 1 1
1 1 1 1
Going back to the original example you have, you can do
length(Q) - sum(X <= Q) + 1
to get
2 4 3 4
Notice that I have semicolons instead of commas in the definition of Q. If you want to make it a column vector after defining it, do something like this instead:
length(Q) - sum(X <= Q') + 1
The reason that this works is that Octave implicitly applies bsxfun to an operation on a row and column vector. MATLAB will not do this until R2016b according to #excaza's comment, so in MATLAB you can do this:
length(Q) - sum(bsxfun(#le, X, Q)) + 1
You can play around with this example in IDEOne here.
Inspired by the solution posted by #Mad Physicist, here is my solution.
Q = [0.1,0.3,0.6,1.0]
X = [0.11,0.72,0.32,0.94]
Temp = repmat(X',1,4)<repmat(Q,4,1)
[~, ind]= max( Temp~=0, [], 2 );
The idea is that make the X and Q into the "same shape", then use element wise comparison, then we obtain a logical matrix whose row tells whether a given element in X is less than each of the element in Q, then return the first non-zero index of each row of this logical matrix. I haven't tested how fast this method is comparing to other methods
iI was given this solution to a problem in my course material.
Problem:
a signal x(t) sampled at 10 sample/sec. consider the first 10 samples of x(t)
x(t) = 0.3 cos(2*pi*t);
using a 8-bit quantiser find the quantisation error.
solution:
(256 quantisation levels)
t=1:10;
x=(0.3)*cos(2*pi*(t-1)/10);
mx=max(abs(x));
q256=mx*(1/128)*floor(128*(x/mx));
stem(q256)
e256=(1/10)*sum(abs(x-q256))
Error: e256 = 9.3750e-04
There was no explanation on this, can you explain how this was calculated in detail?
For the first two code lines I prefer,
Fs = 10;
L = 10;
t = (0 : L - 1) / Fs;
x = 0.3 * cos(2 * pi * t);
where Fs is sampling frequency, L number of samples and t shows the time.
Note that x is sinusoidal with frequency of Fx = 1 Hz or we can say that it's periodic with Tx = 1 sec.
For 8-bit quantization we have 256 levels. Since L / Fs = [10 sample] / [10 sample/sec] = 1 sec is equal to Tx (a whole period of x) we can work with positive samples.
mx = max(abs(x));
mx is defined because in order to use floor we need to scale the x.
q256 = mx*(1/128)*floor(128*(x/mx));
mx shows the maximum value for x so x / mx will take values over [-1 1] and 128*x/mx over [-128 128] will cover all 256 levels.
So we will quantize it with floor and scale it back (mx*1/128).
e256 = (1/L)*sum(abs(x-q256))
e256 simply shows the mean error over 10 samples.
Note that if L / Fs < Tx then this quantization won't be the optimum one.
Have in mind
The answer that you are given has some problems!
suppose x = [-1 -.2 0 .7 1]; and we want to quantize it with 2 bits.
mx = max(abs(x));
q4 = mx * (1/2) * floor(2*(x/mx));
Will give q4 = [-1 -0.5 0 0.5 1] which has 5 levels (instead of 2^2 = 4).
It might not be a big problem, you can delete the level x=1 and have q4 = [-1 -0.5 0 0.5 0.5], still the code needs some improvements and of course the error will increase.
A simple solution is to add
[~,ind] = max(x);
x(ind) = x(ind) - 1e-10;
after definition of mx so the maximum values of x will be quantized in one level lower.
The error will increase to 0.0012.
I am using matlab for part of my final year project. I am solving a geometric series such as the sum of x^j, starting from j=0 up to n-1. I have the following code so far:
$Variable dictionary
%N Number of terms to sum
%alpha Sum of series
%x Vector of constants
%n Loop counter
N = input('Enter the number of terms to sum: ');
alpha = 0;
x = [0.9 0.99 0.999 0.9999 0.99999 0.999999];
for n = 0:N-1
alpha = alpha + (x.^(n));
end
format long
alpha
When I run this script it is allowing me to put in the values of x in the script as a vector but asks the user for values of n. Is there anyway I can amend my code so that I can put the n in myself? And make it more than one value of n?
Thanks
Maybe this is what you want (no loops needed):
x = [0.9 0.99 0.999 0.9999 0.99999 0.999999];
n = [1 2 5];
alphas = sum(bsxfun(#power, x(:), n(:).')); %'// one result for each value of n
Modify this part of code:
for n = 1: length(N)
alpha = alpha + (x.^(N(n)));
end
And pass the N as vector [10 100 1000]
Here a solution:
x = [0.9 0.99 0.999 0.9999 0.99999 0.999999];
nlist = [10,100,1000];
for elm = nlist
alpha = alpha + (x.^(elm));
end