I have been stuck trying trying to write a MATLAB algorithm that computes a recursion in reverse, or that is what it seems like to me.
y_n = (1/n)−10*y_n−1 for n = 1,...,30 works in MATLAB, but because of the (*10), the round-off error makes the algorithm unstable and it is useless. Just by manipulating the recursion, y_n-1 = (1/10)(1/n - y_n) will work and the round-off errors will be reduced 10 fold at each step, potentially making this a stable algorithm.
After a couple days, I still cannot understand the logic needed to code this. Evaluating at y_n-1 is really throwing me in a loop. I was able to tackle the unstable algorithm, but I cannot think of the logic to manipulate the code to make it work. My question lies with how do you code this in MATLAB? I am truly stumped.
% Evaluate the integral yn = integral from 0 to 1 of x^n/(x+10).
% Unstable algorithm:
y(1) = log(11) - log(10);
k = 30;
for n = 1:k
y(n+1) = (1/n) - 10*y(n);
end
n_vector = 0:k;
[n_vector;y]
By manipulating the recursion, the results will be close to true values because of the bound on the error. The current output:
0.0953101798043248
0.0468982019567523
0.0310179804324768
0.0231535290085650
0.0184647099143501
0.0153529008564988
0.0131376581016785
0.0114805618403582
0.0101943815964183
0.00916729514692832
0.00832704853071678
0.00763860560192312
0.00694727731410218
0.00745030378205516
-0.00307446639198020
0.0974113305864686
-0.911613305864686
9.17495658805863
-91.6940103250307
916.992734829255
-9169.87734829255
91698.8211019731
-916988.165565185
9169881.69913012
-91698816.9496345
916988169.536345
-9169881695.32499
91698816953.2869
-916988169532.833
9169881695328.37
-91698816953283.7
What is expected, with the round-off errors taken care of is the results to stay between 0and1.
This output you are getting is correct, and as pointed out in the comments by Mad Physicist, the recursive function you have should behave this way.
If you look at the behavior of the two terms, as n gets bigger the initial subtraction will have less of an effect on the 10*y(n) term. So for large n, we can ignore 1/n.
At large n we then expect each step will increase our value by roughly a factor of 10. This is what you see in your output.
As far as writing a backward recursion. By definition you need a starting value, so you would need to assume y(30) and run the recursion backward as suggested in the comments.
So, I was able to answer by own question. The code needed would look like this:
% This function calculates the value of y20 with a guarantee to have an
% absolute error less than 10^-5
% The yn1 chosen to be high enough to guarantee this is n1 = 25
% Returns the value of y(20)
function [x]= formula(k)
% RECURSION APPROXIMATION
y(k) = 0;
n = k:-1:20;
y(n-1) = (1./10)*(1./n - y(n));
x = y(20);
% FURTHER: I needed to guarantee y20 to have <= 10^-5 magnitude error
% I determined n=25 would be my starting point, approximating y25=0 and working
% backwards to n=20 as I did above.
% y(n-1)=1/10(1/n-yn) “exact solution”
% (yn-1)*=1/10(1/n-(yn)*) “approximate solution with error”
% y(n-1)-(y(n-1))*=1/10(1/n-yn)-1/10(1/n-(yn)*) calculating the error
% = 1/10((yn)*-yn)
% So,
% E(n-1)=1/10(En)
% E(n-2)=1/100(E(n-1))
% E(n-3)=1/1000(E(n-2))
% E(n-4)=1/10000(E(n-3))
% E(n-5)=1/100000(E(n-4)) ⇒ 10^(-5)
% En20=(10^-5)En25
% Therefore, if we start with n1=25, it guarantees that y20 will have 10^-5 magnitude of % the initial propagating error.
Related
This is my first go with ML (and Matlab) and I'm following "Learning From Data" by Yaser S. Abu-Mostafa.
I'm trying to implement the Perceptron algorithm, after trying to go through the pseudocode, using other people's solutions I can't seem to fix my problem (I went through other threads too).
The algorithm separates the data fine, it works. However, I want to plot a single line, but it seems as it separates them in a way so the '-1' cluster is divided to a second cluster or more.
This is the code:
iterations = 100;
dim = 3;
X1=[rand(1,dim);rand(1,dim);ones(1,dim)]; % class '+1'
X2=[rand(1,dim);1+rand(1,dim);ones(1,dim)]; % class '-1'
X=[X1,X2];
Y=[-ones(1,dim),ones(1,dim)];
w=[0,0,0]';
% call perceptron
wtag=weight(X,Y,w,iterations);
% predict
ytag=wtag'*X;
% plot prediction over origianl data
figure;hold on
plot(X1(1,:),X1(2,:),'b.')
plot(X2(1,:),X2(2,:),'r.')
plot(X(1,ytag<0),X(2,ytag<0),'bo')
plot(X(1,ytag>0),X(2,ytag>0),'ro')
legend('class -1','class +1','pred -1','pred +1')
%Why don't I get just one line?
plot(X,Y);
The weight function (Perceptron):
function [w] = weight(X,Y,w_init,iterations)
%WEIGHT Summary of this function goes here
% Detailed explanation goes here
w = w_init;
for iteration = 1 : iterations %<- was 100!
for ii = 1 : size(X,2) %cycle through training set
if sign(w'*X(:,ii)) ~= Y(ii) %wrong decision?
w = w + X(:,ii) * Y(ii); %then add (or subtract) this point to w
end
end
sum(sign(w'*X)~=Y)/size(X,2); %show misclassification rate
end
I don't think the problem is in the second function but I added it regardless
I'm pretty sure the algorithm separates it to more than one cluster but I can't tell why most of the learning I've done so far was math and theory and not actual coding so I'm probably missing something obvious..
I have an integrated error expression E = int[ abs(x-p)^2 ]dx with limits x|0 to x|L. The variable p is a polynomial of the form 2*(a*sin(x)+b(a)*sin(2*x)+c(a)*sin(3*x)). In other words, both coefficients b and c are known expressions of a. An additional equation is given as dE/da = 0. If the upper limit L is defined, the system of equations is closed and I can solve for a, giving the three coefficients.
I managed to get an optimization routine to solve for a purely based on maximizing L. This is confirmed by setting optimize=0 in the code below. It gives the same solution as if I solved the problem analytically. Therefore, I know the equations to solve for the coefficent a are correct.
I know the example I presented can be solved with pencil and paper, but I'm trying to build an optimization function that is generalized for this type of problem (I have a lot to evaluate). Ideally, polynomial is given as an input argument to a function which then outputs xsol. Obviously, I need to get the optimization to work for the polynomial I presented here before I can worry about generalizations.
Anyway, I now need to further optimize the problem with some constraints. To start, L is chosen. This allows me to calculate a. Once a is know, the polynomial is a known function of x only i.e p(x). I need to then determine the largest INTERVAL from 0->x over which the following constraint is satisfied: |dp(x)/dx - 1| < tol. This gives me a measure of the performance of the polynomial with the coefficient a. The interval is what I call the "bandwidth". I would like to emphasis two things: 1) The "bandwidth" is NOT the same as L. 2) All values of x within the "bandwidth" must meet the constraint. The function dp(x)/dx does oscillate in and out of the tolerance criteria, so testing the criteria for a single value of x does not work. It must be tested over an interval. The first instance of violation defines the bandwidth. I need to maximize this "bandwidth"/interval. For output, I also need to know which L lead to such an optimization, hence I know the correct a to choose for the given constraints. That is the formal problem statement. (I hope I got it right this time)
Now my problem is setting this whole thing up with MATLAB's optimization tools. I tried to follow ideas from the following articles:
Tutorial for the Optimization Toolbox™
Setting optimize=1 for the if statement will work with the constrained optimization. I thought some how nested optimization is involved, but I couldn't get anything to work. I provided known solutions to the problem from the IMSL optimization library to compare/check with. They are written below the optimization routine. Anyway, here is the code I've put together so far:
function [history] = testing()
% History
history.fval = [];
history.x = [];
history.a = [];
%----------------
% Equations
polynomial = #(x,a) 2*sin(x)*a + 2*sin(2*x)*(9/20 -(4*a)/5) + 2*sin(3*x)*(a/5 - 2/15);
dpdx = #(x,a) 2*cos(x)*a + 4*cos(2*x)*(9/20 -(4*a)/5) + 6*cos(3*x)*(a/5 - 2/15);
% Upper limit of integration
IC = 0.8; % initial
LB = 0; % lower
UB = pi/2; % upper
% Optimization
tol = 0.003;
% Coefficient
% --------------------------------------------------------------------------------------------
dpda = #(x,a) 2*sin(x) + 2*sin(2*x)*(-4/5) + 2*sin(3*x)*1/5;
dEda = #(L,a) -2*integral(#(x) (x-polynomial(x,a)).*dpda(x,a),0,L);
a_of_L = #(L) fzero(#(a)dEda(L,a),0); % Calculate the value of "a" for a given "L"
EXITFLAG = #(L) get_outputs(#()a_of_L(L),3); % Be sure a zero is actually calculated
% NL Constraints
% --------------------------------------------------------------------------------------------
% Equality constraint (No inequality constraints for parent optimization)
ceq = #(L) EXITFLAG(L) - 1; % Just make sure fzero finds unique solution
confun = #(L) deal([],ceq(L));
% Objective function
% --------------------------------------------------------------------------------------------
% (Set optimize=0 to test coefficent equations and proper maximization of L )
optimize = 1;
if optimize
%%%% Plug in solution below
else
% Optimization options
options = optimset('Algorithm','interior-point','Display','iter','MaxIter',500,'OutputFcn',#outfun);
% Optimize objective
objective = #(L) -L;
xsol = fmincon(objective,IC,[],[],[],[],LB,UB,confun,options);
% Known optimized solution from IMSL library
% a = 0.799266;
% lim = pi/2;
disp(['IMSL coeff (a): 0.799266 Upper bound (L): ',num2str(pi/2)])
disp(['code coeff (a): ',num2str(history.a(end)),' Upper bound: ',num2str(xsol)])
end
% http://stackoverflow.com/questions/7921133/anonymous-functions-calling-functions-with-multiple-output-forms
function varargout = get_outputs(fn, ixsOutputs)
output_cell = cell(1,max(ixsOutputs));
[output_cell{:}] = (fn());
varargout = output_cell(ixsOutputs);
end
function stop = outfun(x,optimValues,state)
stop = false;
switch state
case 'init'
case 'iter'
% Concatenate current point and objective function
% value with history. x must be a row vector.
history.fval = [history.fval; optimValues.fval];
history.x = [history.x; x(1)];
history.a = [history.a; a_of_L(x(1))];
case 'done'
otherwise
end
end
end
I could really use some help setting up the constrained optimization. I'm not only new to optimizations, I've never used MATLAB to do so. I should also note that what I have above does not work and is incorrect for the constrained optimization.
UPDATE: I added a for loop in the section if optimizeto show what I'm trying to achieve with the optimization. Obviously, I could just use this, but it seems very inefficient, especially if I increase the resolution of range and have to run this optimization many times. If you uncomment the plots, it will show how the bandwidth behaves. By looping over the full range, I'm basically testing every L but surely there's got to be a more efficient way to do this??
UPDATE: Solved
So it seems fmincon is not the only tool for this job. In fact I couldn't even get it to work. Below, fmincon gets "stuck" on the IC and refuses to do anything...why...that's for a different post! Using the same layout and formulation, fminbnd finds the correct solution. The only difference, as far as I know, is that the former was using a conditional. But my conditional is nothing fancy, and really unneeded. So it's got to have something to do with the algorithm. I guess that's what you get when using a "black box". Anyway, after a long, drawn out, painful, learning experience, here is a solution:
options = optimset('Display','iter','MaxIter',500,'OutputFcn',#outfun);
% Conditional
index = #(L) min(find(abs([dpdx(range(range<=L),a_of_L(L)),inf] - 1) - tol > 0,1,'first'),length(range));
% Optimize
%xsol = fmincon(#(L) -range(index(L)),IC,[],[],[],[],LB,UB,confun,options);
xsol = fminbnd(#(L) -range(index(L)),LB,UB,options);
I would like to especially thank #AndrasDeak for all their support. I wouldn't have figured it out without the assistance!
I have to construct the following function in MATLAB and am having trouble.
Consider the function s(t) defined for t in [0,4) by
{ sin(pi*t/2) , for t in [0,1)
s(t) = { -(t-2)^3 , for t in [1,3)*
{ sin(pi*t/2) , for t in [3,4)
(i) Generate a column vector s consisting of 512 uniform
samples of this function over the interval [0,4). (This
is best done by concatenating three vectors.)
I know it has to be something of the form.
N = 512;
s = sin(5*t/N).' ;
But I need s to be the piecewise function, can someone provide assistance with this?
If I understand correctly, you're trying to create 3 vectors which calculate the specific function outputs for all t, then take slices of each and concatenate them depending on the actual value of t. This is inefficient as you're initialising 3 times as many vectors as you actually want (memory), and also making 3 times as many calculations (CPU), most of which will just be thrown away. To top it off, it'll be a bit tricky to use concatenate if your t is ever not as you expect (i.e. monotonically increasing). It might be an unlikely situation, but better to be general.
Here are two alternatives, the first is imho the nice Matlab way, the second is the more conventional way (you might be more used to that if you're coming from C++ or something, I was for a long time).
function example()
t = linspace(0,4,513); % generate your time-trajectory
t = t(1:end-1); % exclude final value which is 4
tic
traj1 = myFunc(t);
toc
tic
traj2 = classicStyle(t);
toc
end
function trajectory = myFunc(t)
trajectory = zeros(size(t)); % since you know the size of your output, generate it at the beginning. More efficient than dynamically growing this.
% you could put an assert for t>0 and t<3, otherwise you could end up with 0s wherever t is outside your expected range
% find the indices for each piecewise segment you care about
idx1 = find(t<1);
idx2 = find(t>=1 & t<3);
idx3 = find(t>=3 & t<4);
% now calculate each entry apprioriately
trajectory(idx1) = sin(pi.*t(idx1)./2);
trajectory(idx2) = -(t(idx2)-2).^3;
trajectory(idx3) = sin(pi.*t(idx3)./2);
end
function trajectory = classicStyle(t)
trajectory = zeros(size(t));
% conventional way: loop over each t, and differentiate with if-else
% works, but a lot more code and ugly
for i=1:numel(t)
if t(i)<1
trajectory(i) = sin(pi*t(i)/2);
elseif t(i)>=1 & t(i)<3
trajectory(i) = -(t(i)-2)^3;
elseif t(i)>=3 & t(i)<4
trajectory(i) = sin(pi*t(i)/2);
else
error('t is beyond bounds!')
end
end
end
Note that when I tried it, the 'conventional way' is sometimes faster for the sampling size you're working on, although the first way (myFunc) is definitely faster as you scale up really a lot. In anycase I recommend the first approach, as it is much easier to read.
Suppose I have a vector J of jump sizes and an initial starting point X_0. Also I have boundaries 0, B (assume 0 < X_0 < B). I want to do a random walk where X_i = [min(X_{i-1} + J_i,B)]^+. (positive part). Basically if it goes over a boundary, it is made equal to the boundary. Anyone know a vectorized way to do this? The current way I am doing it consists of doing cumsums and then finding places where it violates a condition, and then starting from there and repeating the cumsum calculation, etc until I find that I stop violating the boundaries. It works when the boundaries are rarely hit, but if they are hit all the time, it basically becomes a for loop.
In the code below, I am doing this across many samples. To 'fix' the ones that go out of the boundary, I have to loop through the samples to check...(don't think there is a vectorized 'find')
% X_init is a row vector describing initial resource values to use for
% each sample
% J is matrix where each col is a sequence of Jumps (columns = sample #)
% In this code the jumps are subtracted, but same thing
X_intvl = repmat(X_init,NumJumps,1) - cumsum(J);
X = [X_init; X_intvl];
for sample = 1:NumSamples
k = find(or(X_intvl(:,sample) > B, X_intvl(:,sample) < 0),1);
while(~isempty(k))
change = X_intvl(k-1,sample) - X_intvl(k,sample);
X_intvl(k:end,sample) = X_intvl(k:end,sample)+change;
k = find(or(X_intvl(:,sample) > B, X_intvl(:,sample) < 0),1);
end
end
Interesting question (+1).
I faced a similar problem a while back, although slightly more complex as my lower and upper bound depended on t. I never did work out a fully-vectorized solution. In the end, the fastest solution I found was a single loop which incorporates the constraints at each step. Adapting the code to your situation yields the following:
%# Set the parameters
LB = 0; %# Lower bound
UB = 5; %# Upper bound
T = 100; %# Number of observations
N = 3; %# Number of samples
X0 = (1/2) * (LB + UB); %# Arbitrary start point halfway between LB and UB
%# Generate the jumps
Jump = randn(N, T-1);
%# Build the constrained random walk
X = X0 * ones(N, T);
for t = 2:T
X(:, t) = max(min(X(:, t-1) + Jump(:, t-1), UB), 0);
end
X = X';
I would be interested in hearing if this method proves faster than what you are currently doing. I suspect it will be for cases where the constraint is binding in more than one or two places. I can't test it myself as the code you provided is not a "working" example, ie I can't just copy and paste it into Matlab and run it, as it depends on several variables for which example (or simulated) values are not provided. I tried adapting it myself, but couldn't get it to work properly?
UPDATE: I just switched the code around so that observations are indexed on columns and samples are indexed on rows, and then I transpose X in the last step. This will make the routine more efficient, since Matlab allocates memory for numeric arrays column-wise - hence it is faster when performing operations down the columns of an array (as opposed to across the rows). Note, you will only notice the speed-up for large N.
FINAL THOUGHT: These days, the JIT accelerator is very good at making single loops in Matlab efficient (double loops are still pretty slow). Therefore personally I'm of the opinion that every time you try and obtain a fully-vectorized solution in Matlab, ie no loops, you should weigh up whether the effort involved in finding a clever solution is worth the slight gains in efficiency to be made over an easier-to-obtain method that utilizes a single loop. And it is important to remember that fully-vectorized solutions are sometimes slower than solutions involving single loops when T and N are small!
I'd like to propose another vectorized solution.
So, first we should set the parameters and generate random Jumpls. I used the same set of parameters as Colin T Bowers:
% Set the parameters
LB = 0; % Lower bound
UB = 20; % Upper bound
T = 1000; % Number of observations
N = 3; % Number of samples
X0 = (1/2) * (UB + LB); % Arbitrary start point halfway between LB and UB
% Generate the jumps
Jump = randn(N, T-1);
But I changed generation code:
% Generate initial data without bounds
X = cumsum(Jump, 2);
% Apply bounds
Amplitude = UB - LB;
nsteps = ceil( max(abs(X(:))) / Amplitude - 0.5 );
for ii = 1:nsteps
ind = abs(X) > (1/2) * Amplitude;
X(ind) = Amplitude * sign(X(ind)) - X(ind);
end
% Shifting X
X = X0 + X;
So, instead of for loop I'm using cumsum function with smart post-processing.
N.B. This solution works significantly slower than Colin T Bowers's one for tight bounds (Amplitude < 5), but for loose bounds (Amplitude > 20) it works much faster.
Not quite sure what this means.
"Warning: Matrix is singular to working precision."
I have a 3x4 matrix called matrix bestM
matrix Q is 3x3 of bestM and matrix m is the last column of bestM
I would like to do C = -Inverse matrix of Q * matrix m
and I get that warning
and C =[Inf Inf Inf] which isn't right because i am calculating for the camera center in the world
bestM = [-0.0031 -0.0002 0.0005 0.9788;
-0.0003 -0.0006 0.0028 0.2047;
-0.0000 -0.0000 0.0000 0.0013];
Q = bestM(1:3,1:3);
m = bestM(:,4);
X = inv(Q);
C = -X*m;
disp(C);
A singular matrix can be thought of as the matrix equivalent of zero, when you try to invert 0 it blows up (goes to infinity) which is what you are getting here. user 1281385 is absolutely wrong about using the format command to increase precision; the format command is used to change the format of what is shown to you. In fact the very first line of the help command for format says
format does not affect how MATLAB computations are done.
As found here, a singular matrix is one that does not have an inverse. As dvreed77 already pointed out, you can think of this as 1/0 for matrices.
Why I'm answering, is to tell you that using inv explicitly is almost never a good idea. If you need the same inverse a few hundred times, it might be worth it, however, in most circumstances you're interested in the product C:
C = -inv(Q)*m
which can be computed much more accurately and faster in Matlab using the backslash operator:
C = -Q\m
Type help slash for more information on that. And even if you happen to find yourself in a situation where you really need the inverse explicitly, I'd still advise you to avoid inv:
invQ = Q\eye(size(Q))
Below is a little performance test to demonstrate one of the very few situations where the explicit inverse can be handy:
% This test will demonstrate the one case I ever encountered where
% an explicit inverse proved useful. Unfortunately, I cannot disclose
% the full details without breaking the law, but roughly, it came down
% to this: The (large) design matrix A, a result of a few hundred
% co-registrated images, needed to be used to solve several thousands
% of systems, where the result matrices b came from processing the
% images one-by-one.
%
% That means the same design matrix was re-used thousands of times, to
% solve thousands of systems at a time. To add to the fun, the images
% were also complex-valued, but I'll leave that one out of consideration
% for now :)
clear; clc
% parameters for this demo
its = 1e2;
sz = 2e3;
Bsz = 2e2;
% initialize design matrix
A = rand(sz);
% initialize cell-array to prevent allocating memory from consuming
% unfair amounts of time in the first loop.
% Also, initialize them, NOT copy them (as in D=C,E=D), because Matlab
% follows a lazy copy-on-write scheme, which would influence the results
C = {cellfun(#(~) zeros(sz,Bsz), cell(its,1), 'uni', false) zeros(its,1)};
D = {cellfun(#(~) zeros(sz,Bsz), cell(its,1), 'uni', false) zeros(its,1)};
E = {cellfun(#(~) zeros(sz,Bsz), cell(its,1), 'uni', false) zeros(its,1)};
% The impact of rand() is the same in both loops, so it has no
% effect, it just gives a longer total run time. Still, we do the
% rand explicitly to *include* the indexing operation in the test.
% Also, caching will most definitely influence the results, because
% any compiler (JIT), even without optimizations, might recognize the
% easy performance gain when the code computes the same array over and
% over again. It probably will, but we have no control over when and
% wherethat happens. So, we prevent that from happening at all, by
% re-initializing b at every iteration.
% The assignment to cell is a necessary part of the demonstration;
% it is the desired output of the whole calculation. Assigning to cell
% instead of overwriting 'ans' takes some time, which is to be included
% in the demonstration, again for cache reasons: the extra time is now
% guaranteed to be equal in both loops, so it really does not matter --
% only the total run time will be affected.
% Direct computation
start = tic;
for ii = 1:its
b = rand(sz,Bsz);
C{ii,1} = A\b;
C{ii,2} = max(max(abs( A*C{ii,1}-b )));
end
time0 = toc(start);
[max([C{:,2}]) mean([C{:,2}]) std([C{:,2}])]
% LU factorization (everyone's
start = tic;
[L,U,P] = lu(A, 'vector');
for ii = 1:its
b = rand(sz,Bsz);
D{ii,1} = U\(L\b(P,:));
D{ii,2} = max(max(abs( A*D{ii,1}-b )));
end
time1 = toc(start);
[max([D{:,2}]) mean([D{:,2}]) std([D{:,2}])]
% explicit inv
start = tic;
invA = A\eye(size(A)); % NOTE: DON'T EVER USE INV()!
for ii = 1:its
b = rand(sz,Bsz);
E{ii,1} = invA*b;
E{ii,2} = max(max(abs( A*E{ii,1}-b )));
end
time2 = toc(start);
[max([E{:,2}]) mean([E{:,2}]) std([E{:,2}])]
speedup0_1 = (time0/time1-1)*100
speedup1_2 = (time1/time2-1)*100
speedup0_2 = (time0/time2-1)*100
Results:
% |Ax-b|
1.0e-12 * % max. mean st.dev.
0.1121 0.0764 0.0159 % A\b
0.1167 0.0784 0.0183 % U\(L\b(P,;))
0.0968 0.0845 0.0078 % invA*b
speedup0_1 = 352.57 % percent
speedup1_2 = 12.86 % percent
speedup0_2 = 410.80 % percent
It should be clear that an explicit inverse has its uses, but just as a goto construct in any language -- use it sparingly and wisely.