svmtain() is a function in MATLAB for SVM learning. The help doc is here:
http://www.mathworks.com/help/bioinfo/ref/svmtrain.html
How can I use it with a custom kernel? In the help doc, it says:
#kfun — Function handle to a kernel function. A kernel function must be of the form
function K = kfun(U, V)
The returned value, K, is a matrix of size M-by-N, where U and V have M and N rows respectively.
It mentions nothing about what U and V are and what M and N mean. I just don't know how to use it in the right format.
Can anyone tell me what U and V are and what M and N mean?
For example, the training data are 5-dimensional vectors and the kernel function is the sum of the length of the vectors. How can I write the kernel function?
Thank you!
just a guess:
according to: http://www.tech.dmu.ac.uk/~hseker/Statistics%20in%20Genetics/Statistical%20Learning%20and%20Visualization%20in%20MATLAB.doc , U, V should be just parameters in your functional prescription K, e.g. if your kernel is tanh, then:
function K = kfun(U,V,P1,P2)
K = tanh(U*V');
And P1, P2 are for some additional features of your respective kernel. But as I wrote in the comment, you need to be good mathematician to reach better results than the ones acquired by already defined kernels.
Kernel functions are one of the most common techniques used at Machine Learning algorithms. Here is definition of it from Wikipedia:
For machine learning algorithms, the kernel trick is a way of mapping
observations from a general set S into an inner product space V
(equipped with its natural norm), without ever having to compute the
mapping explicitly, in the hope that the observations will gain
meaningful linear structure in V.
i.e. this kernel is used at RBF:
K(x,y) = (x*y + c)^d
Here is an detailed explanation of Kernels: http://www.youtube.com/watch?v=bUv9bfMPMb4 by Andrew Ng.
There are some Kernels (i.e. Gaussian Kernel), kernels have same convention thats why it is generalized as K(u,v). You can try different kernels performances or you can search about related works about what you work on and try to use that kind of kernels.
Related
Spefifically, the following two kinds of code can get the same S and V idealy. However, the second one's speed is usually faster than the first one in Matlab. Can someone tell me the reason?
Moreover, which method is more numerically stable?
Thanks.
[~,S,V] = svd(B,'econ');
[Qc,Rc] = qr(B',0);
[U,S,~] = svd(Rc,'econ');
V = Qc*U;
The second method does not have to be faster. For almost squared matrices it can be slower. Consider as example the Golub-Reinsch SVD-algorithm:
Its work depends on the output you want to calculate (only S, Sand V or S,V and U).
If you want to calculate Sand V without performing any preprocessing the required work is 4mn^2+8n^3.
If you perform QR-decomposition before this the needed amount of work is: 2/3n^3+n^2+1/3n-2 for the Housholder transformation. Now if your Matrix was almost squared, i.e m=n, you will have gained not much as R is still m x n. However if m is larger than n you can reduce R to an n x n matrix (called thin QR factorization). Now you want to calculate Uand S which will add 12n^3 for your SVD-algorithm.
So only SVD: 4mn^2+8n^3
SVD with QR: (12+2/3)n^3+n^2+1/3n-2
However most SVD-algorithms should inculde some (R-) bidiagonalizations which will reduce the work to: 2mn^2+11n^3
You can also apply QR, the R-bifactorization and then SVD to make it even faster but it all depends on your matrix dimensions.
Matlab uses for SVD the Lapack libraries. You can look up the exact runtimes here. They're approximately the same as above algorithm.
Hope this helps.
I need the algorithm to compute the smith normal form of a matrix M. Usually it is described as M = USV, see e.g. SNF on WP, where my matrices are always of full rank, i.e. det(S) ~= 0.
I started in Mathematica, where a nice package provides this algorithm. While there are some implementations around, for example this one, this one by Gilbert or this one using maple, I don't have access to maple (I think) and both other implementations either don't compute the matrices U and V or compute them wronly (try [5,0;0,5] in the second implementation, where U and V should just be unit matrices)
So are there any other implementations available?
Perhaps one following the approach of this Mathematica package (of course analogously in MatLab), because that worked quite well for my needs (when working with Mathematica), but as long as U and V are provided correctly I don't mind.
Since Matlab 2015b there's SmithForm in the Symbolic Math Toolbox documented here
https://mathworks.com/help/symbolic/smithform.html
which resolves the mentioned problems.
My Matlab program has multiple inputs as a struct (in.a, in.b, etc.)
and multiple outputs (out.a, out.b, etc.)
I would like to use the genetic algorithm solver from teh optimization toolbox to find the best input in.a, while all the other inputs are constant. The fitness is one of the outputs, e.g. out.b(2,3).
How do I "tell" the solver this?
Thanks
Daniel
It is not uncommon in programming to have a situation where what is most convenient for your function and what some library call expects of it don't agree. The normal resolution to such a problem is to write a small layer in between that allows the two to talk; an interface.
From help ga:
X = GA(FITNESSFCN,NVARS) finds a local unconstrained minimum X to the
FITNESSFCN using GA. [...] FITNESSFCN accepts a vector X of size
1-by-NVARS, and returns a scalar evaluated at X.
So, ga expects vector input, scalar output, whereas you have a structure going in and out. You would have to write the following (sub)function:
function Y = wrapper_Objfun(X, in)
in.a = X; %# variable being optimized
out = YOUR_REAL_FUNCTION(in); %# call to your actual function
Y = out.b(2,3); %# objective value
end
and then the call to ga will look like
X = ga(#(x) wrapper_Objfun(x,in), N);
where N is however large in.a should be.
Also have a read about it in Matlab's own documentation on the subject.
I'd like to find the principal components of a data matrix X in Matlab by solving the optimization problem min||X-XBB'||, where the norm is the Frobenius norm, and B is an orthonormal matrix. I'm wondering if anyone could tell me how to do that. Ideally, I'd like to be able to do this using the optimization toolbox. I know how to find the principal components using other methods. My goal is to understand how to set up and solve an optimization problem which has a matrix as the answer. I'd very much appreciate any suggestions or comments.
Thanks!
MJ
The thing about Optimization is that there are different methods to solve a problem, some of which can require extensive computation.
Your solution, given the constraints for B, is to use fmincon. Start by creating a file for the non-linear constraints:
function [c,ceq] = nonLinCon(x)
c = 0;
ceq = norm((x'*x - eye (size(x))),'fro'); %this checks to see if B is orthonormal.
then call the routine:
B = fmincon(#(B) norm(X - X*B*B','fro'),B0,[],[],[],[],[],[],#nonLinCon)
with B0 being a good guess on what the answer will be.
Also, you need to understand that this algorithms tries to find a local minimum, which may not be the solution you ultimately want. For instance:
X = randn(1,2)
fmincon(#(B) norm(X - X*B*B','fro'),rand(2),[],[],[],[],[],[],#nonLinCon)
ans =
0.4904 0.8719
0.8708 -0.4909
fmincon(#(B) norm(X - X*B*B','fro'),rand(2),[],[],[],[],[],[],#nonLinCon)
ans =
0.9864 -0.1646
0.1646 0.9864
So be careful, when using these methods, and try to select a good starting point
The Statistics toolbox has a built-in function 'princomp' that does PCA. If you want to learn (in general, without the optimization toolbox) how to create your own code to do PCA, this site is a good resource.
Since you've specifically mentioned wanting to use the Optimization Toolbox and to set this up as an optimization problem, there is a very well-trusted 3rd-party package known as CVX from Stanford University that can solve the optimization problem you are referring to at this site.
Do you have the optimization toolbox? The documentation is really good, just try one of their examples: http://www.mathworks.com/help/toolbox/optim/ug/brg0p3g-1.html.
But in general the optimization function look like this:
[OptimizedMatrix, OptimizedObjectiveFunction] = optimize( (#MatrixToOptimize) MyObjectiveFunction(MatrixToOptimize), InitialConditionsMatrix, ...optional constraints and options... );
You must create MyObjectiveFunction() yourself, it must take the Matrix you want to optimize as an input and output a scalar value indicating the cost of the current input Matrix. Most of the optimizers will try to minimise this cost. Note that the cost must be a scalar.
fmincon() is a good place to start, once you are used to the toolbox you and if you can you should choose a more specific optimization algorithm for your problem.
To optimize a matrix rather than a vector, reshape the matrix to a vector, pass this vector to your objective function, and then reshape it back to the matrix within your objective function.
For example say you are trying to optimize the 3 x 3 matrix M. You have defined objective function MyObjectiveFunction(InputVector). Pass M as a vector:
MyObjectiveFunction(M(:));
And within the MyObjectiveFunction you must reshape M (if necessary) to be a matrix again:
function cost = MyObjectiveFunction(InputVector)
InputMatrix = reshape(InputVector, [3 3]);
%Code that performs matrix operations on InputMatrix to produce a scalar cost
cost = %some scalar value
end
I would use a multi dimensional gaussian modell for regression. Rasmussen has a book with an algoritm, but it is only for one dimension output. Any idea to modify it?
First, I presume that you know about http://www.gaussianprocess.org/gpml/code/matlab/doc/regression.html#ard, and this is not what you want.
Second, I consequently presume that your problem involves several functions. In this case, for most purposes, you can just run your regression on each function separately; that is, unless you have some weird norm on the output space prescribed to you.
Lets say you want to model f(x,y) = [u , v]^T. You could model u and v separately:
f1(x,y) = u
f2(x,y) = v
This is making the assumption that u and v are conditionally independent given x, y. However, GPML advises that u and v can remain correlated because of a correlating noise process. Consult Chapter 9 of GPML for approaches in this case.