The title of this post may be a bit confusing. Please allow me to provide a bit of context and then elaborate on what I'm asking. For your reference, the question I'm asking is toward the end and is denoted by bold letters. I provide some code, outlining where I'm currently at in solving the problem, immediately beforehand.
Essentially what I'm trying to do is Kernel Regression, which is usually done using a single test point x and a set of training instances . A reference to this can be found on wikipedia here. The kernel I'm using is the RBF kernel, a Wikipedia reference for which can be found here.
Anyway, I have some code written in Matlab so that this can be done quickly for a single instance of x, which is 1 x p in size. What I'd like to do is make it so I can estimate for numerous points very quickly, say m x p.
For the sake of avoiding notational mixups, I'll let the training instances be denoted Train and the instances I want estimates for as Test: and . It also needs to be mentioned that I want to estimate a vector of numbers for each of the m points. For a single point this vector would be 1 x v in size. Now I need it to be m x v. Therefore, Train will also have a vector of these know values associated with it called TS: . Lastly, we need a vector of sigmas that is 1 x v in size. This is denoted as Sig.
Here's the code I have so far:
%First, we have to get the matrices to equivalent size so we can subtract Train from Test
tm0 = kron(ones(size(Train,1),1),Test) - kron(ones(size(Test,1),1),Train);
%Secondly, we apply the Euclidean norm sq by row and then multiply each of these results by each element (j) in Sig times 1/2j^2
tm3 = exp(-kron(sum((tm0).^2,2),1/2./(Sig.^2)));
Now, at this point tm3 is an (m*n) x v matrix. This is where my question is: I now need to multiply TS' (TS transpose) times each of the n x v-sized segments in tm3 (there are m of these segments), get the diagonal elements of each of these resulting segments (after multiplication one of the m segments will be v x v, so each chunk of diagonal elements will be 1 x v meaning the resulting matrix is m x v) and sum these diagonal elements together to produce an m x 1 sized matrix. Lastly, I will need to divide each entry i in this m x 1 matrix by each of the v elements in the ith row of the diagonal-holding m x v-sized matrix, producing an m x v-sized result matrix.
I hope all of that makes sense. I'm sure there's some kind of trick that can be employed, but I'm just not coming up with it. Any help is greatly appreciated.
Edit 1: I was asked to provide more of an example to help demonstrate what it is that I would like done. The following represent that two matrices I'm talking about, TS and tm3:
As you can see, TS'(TS transpose) is v x n and tm3 is mn x v. In tm3 there are blocks that are of size n x v -- there are m blocks of this size. Notice that the size of TS' is of size v x n. This means that I can multiply TS' by a single block of tm3, which again is of size n x v. This would result in a matrix that is v x v in size. I would like to do this operation -- individually multiplying TS' by each of the n x v-sized blocks of tm3, which would produce m v x v matrices.
From here, though, I would like to obtain the diagonal elements from each of these v x v matrices. So, for a single v x v matrix, denoted using a:
Ultimately, I would to do this for each of the m v x v matrices giving me something that looks like the following, where s is the mth v x v matrix:
If I denote this last matrix as Q, which is m x v in size, it is trivial to sum the elements across the rows to produce the m x 1 vector I was looking for. I will refer to this vector as C. However, I would then like to divide each of these m scalar values by the corresponding row of matrix Q, to produce another m x v matrix:
This is the final matrix I'm looking for. Hopefully this helps make it clear what I'm looking for. Thanks for taking the time to read this!
Thought: I'm pretty sure I could accomplish this by converting tm3 to a cell matrix by doing tc1 = mat2cell(tm3,repmat(length(Train),1,m),length(Sig)), and then put replicate TS m times in another cell matrix tc2 = mat2cell(TS',length(indirectSigma),repmat(length(Train),1,m))'. Finally, I could do operations like tc3 = cellfun(#(a,b) a*b, tc2,tc1,'UniformOutput',false), which would give me m cells filled with the v x v matrices I was looking for. I could proceed from there. However, I'm not sure how fast these cell operations are. Can anybody comment? I'm afraid they might be slow, so I would prefer operations be performed on normal matrices, which I know to be fast. Thanks!
Related
I have a set of climate data (temperature, pressure and moisture for example), X, Y, Z which are matricies with dimensions (n x p) where n is the number of observations and p is the number of spatial points.
Previously, to investigate modes of variability in dataset X, I simply performed a empirical orthogonal function (EOF) analysis OR Principle component Analysis (PCA) on X. This involved decomposing (via SVD), the matrix X.
To investigate the coupling of the modes of variability of X and Y, I used maximum covariance analysis (MCA) which involved decomposing a covariance matrix proportional to XY^{T}. (T is transpose)
However, if I wish to looked at all three datasets, how do I go about doing this? One idea I had was to form a fourth matrix, L, which will be the 'feature' concatenation of the three datasets:
L = [X, Y, Z]
so that my matrix L will have dimensions (n x 3p).
I would then use standard PCA/EOF analysis and use SVD to decompose this matrix L and then I would obtain modes of variabiilty with size (3p x 1) and thus subsequently the mode associated with X is the first p values, the mode associated with Y is the second set of p values and the mode associated with Z is the last p values.
Is this correct? Or can anyone suggest a better way of looking at the coupling of all three (or more) datasets?
Thank you so much!
I'd recommend to treat spatial points as extra dimension, i.e. f x n x p, where 'f' is your number of features. At this point you should use multilinear extension of PCA that can work on tensor data.
I have multiple linear equations in the form of Zi=ai*Xi+bi*Yi for i = 1..30.
How can I calculate every pair of regression coefficient values, or those 30 values of a and b for each (Z,X,Y) combination using MATLAB?
I've tried the following code:
A=Z; B=[Xs Ys];
C = B \ A;
A are my Z points while B is a matrix of my X and Y points. However, I seem to only get one pair of regression coefficients for all of the points.
Thanks in advance!
What you have set up there is unfortunately not the right way to solve it if I understand your problem formulation. That assumption assumes that you are trying to fit all of the points on a single line. Each row of B would thus serve as one point on only one line that you are trying to find the linear regression of. If you want to solve for multiple lines simultaneously, you are going to need to change your formulation.
That is actually very simple. I'm going to assume that you have 30 (x,y) points where each point denotes one equation of a line. You have these set as Xs and Ys respectively. The outputs for each of these equations is also in Zs. I'm also going to assume these are column vectors, and therefore, you have a system set up such that:
a_i and b_i are the coefficients for each line. You know the (x,y) for each line and your goal is to solve for each corresponding a and b. As such, you would need to reformulate your system so that you're solve for a and b.
Rewriting that problem in matrix form, it can be done like so:
The right hand side vector of a_1, b_1, a_2, b_2, ... is what you are ultimately solving for. You can see that we have a matrix equation of Y = M*X where M and Y are known and X is what we need to solve for by doing X = M\Y. As such, all you need to rearrange your x and y values into a block matrix like the above. First we need to find the correct linear indices so that we can place our x and y values into this matrix, then solve the system by least squares with the ldivide operator. The matrix is a N x 2N matrix where N is the total number of equations or constraints that we have (so in your case, 30):
N = numel(Xs);
M = zeros(N, 2*N);
xind = sub2ind(size(M), 1:N, 1:2:2*N);
yind = sub2ind(size(M), 1:N, 2:2:2*N);
M(xind) = Xs;
M(yind) = Ys;
sub2ind allows you to place multiple values into a matrix with a single line of code. Specifically, sub2ind determine the linear indices from a set of row and column coordinates to access into a matrix. If you don't already know, you can access values (and set values) in a matrix using a single number instead of a pair of row and columns. sub2ind will allow you to set multiple values in a matrix at once by specifying a set of linear indices to access in the matrix with a corresponding vector.
In our case, we need two sets of linear indices - one for the x values and one for the y values. Note that the x values start from the first column and skip every other column. The same behaviour can be said for the y values but we start at the second column. Once we have those indices, we set the x and y values in this matrix and we now we simply solve for the coefficients:
coeff = M \ Z;
coeff will now be 2N x 1 vector, so if you want, you can reshape this into a matrix:
coeff = reshape(coeff, 2, []);
Now, coeff will be shaped such that each column will give you the pair of a,b for each equation that you had. As such, the first column denotes a_1, b_1, the second column denotes a_2, b_2 and so on. The first row of coeff is all of the a coefficients for each constraint while the second row is all of the b coefficients for each constraint.
I have just started working using CCA in Matlab. I have two vectors X and Y of dimension 60x1920 and 60x1536 with the number of samples being 60 and variables in the different set of vectors being 1920 and 1536 respectively. I want to know do CCA for reducing them to the subspace and then do feature matching.
I am using this commands.
%% DO CCA
[A,B,r,U,V] = canoncorr(X,Y);
The output I get is this :
Name Size Bytes Class Attributes
A 1920x58 890880 double
B 1536x58 712704 double
U 60x58 27840 double
V 60x58 27840 double
r 1x58 464 double
Can anyone please tell me what these variables mean. I have gone over the documentation several times and still is unclear about them. As I understand CCA finds two linear projection matrices Wx and Wy such that the projection of X and Y on Wx and Wy are maximally correlated.
1) Could anyone please tell me which of the following matrices are these?
2) Also how can I find the projected vectors in the learned subspace of CCA?
Any help will be appreciated. Thanks in advance.
As I understand it, with X and Y being your original data matrices, A and B are the sets of coefficients that perform a change of basis to maximally correlate your original data. Your data is represented in the new bases as the matrices U and V.
So to answer your questions:
The projection matrices you are looking for would be A and B since they transform X and Y into the new space.
The resulting projections of X and Y into the new space would be U and V, respectively. (The r vector represents the entries of the correlation matrix between U and V, which is a diagonal matrix.)
The The MATLAB documentation says this transformation can be done with the following formulae, where N is the number of observations:
U = (X-repmat(mean(X),N,1))*A
V = (Y-repmat(mean(Y),N,1))*B
This page lays out the process nicely so you can see what each coefficient means in the transformation process.
This is the equation I am trying to solve:
h = (X'*X)^-1*X'*y
where X is a matrix and y is a vector ((X'X)^-1 is the inverse of X-transpose times X). I have coded this in Matlab as:
h = (X'*X)\X'*y
which I believe is correct. The problem is that X is around 10000x10000, and trying to calculate that inverse is crashing Matlab on even the most powerful computer I can find (16 cores, 24GB RAM). Is there any way to split this up, or a library designed for doing such large inversions?
Thank you.
That looks like a pseudo inverse. Are you perhaps looking for just
h = X \ y;
I generated a random 10,000 by 10,000 matrix X and a random 10,000 by 1 vector y.
I just broke up my computation step by step. (Code shown below)
Computed the transpose and held it in matrix K
Then I computed Matrix A by multiplying K by X
Computed vector b by multiplying K by vector y
Lastly, I used the backslash operator on A and b to solve
I didn't have a problem with the computation. It took a while, but breaking up the operations into the smallest groups possible helped to prevent the computer from being overwhelmed. However, it could be the composition of the matrix that you are using (ie. Sparse, decimals, etc.).
X = randi(2000, [10000, 10000]);
y = randi(2000, 10000, 1);
K = X';
A = K*X;
b = K*y;
S = A\b;
If you have multiple machines at your disposal, and you can recast your problem into the form h = X\y as proposed by #Ben, then you could use distributed arrays. This demo shows how you can do that.
Jordan,
Your equation is exactly the definition for "Moore-Penrose Matrix Inverse".
Check: http://mathworld.wolfram.com/Moore-PenroseMatrixInverse.html
Directly using h = X \ y; should help.
Or check Matlab pinv(X)*y
I have an mxm kernel matrix, K, which is for the sake of simplicity, a linear kernel computed as pdist2(X,X), where X is mxn and the m dimension relates to feature vectors with n dimensions.
since n is large, I save computation time by precalculating K for all X.
Later on, I need to swap two of the features in X, say X_1 and X_5.
Can I somehow rearrange K, without having to recompute the entire matrix?
If pv is your permutation vector and J0=pdist2(X,X), then
Y=X(pv,:); J1=pdist2(Y,Y);
should get you the same answer as
J1=J0(pv,pv);
If you are permuting the columns (I couldn't quite tell from your question), then it seems like J1 and J0 should be equal...