I have a very large sparse matrix of the type 'scipy.sparse.coo.coo_matrix'. I can convert to csr with .tocsr(), however .todense() will not work since the array is too large. I want to be able to extract elements from the matrix as I would do with a regular array, so that I may pass row elements to a function.
For reference, when printed, the matrix looks as follows:
(7, 0) 0.531519363001
(48, 24) 0.400946334437
(70, 6) 0.684460955022
...
Make a matrix with 3 elements:
In [550]: M = sparse.coo_matrix(([.5,.4,.6],([0,1,2],[0,5,3])), shape=(5,7))
It's default display (repr(M)):
In [551]: M
Out[551]:
<5x7 sparse matrix of type '<class 'numpy.float64'>'
with 3 stored elements in COOrdinate format>
and print display (str(M)) - looks like the input:
In [552]: print(M)
(0, 0) 0.5
(1, 5) 0.4
(2, 3) 0.6
convert to csr format:
In [553]: Mc=M.tocsr()
In [554]: Mc[1,:] # row 1 is another matrix (1 row):
Out[554]:
<1x7 sparse matrix of type '<class 'numpy.float64'>'
with 1 stored elements in Compressed Sparse Row format>
In [555]: Mc[1,:].A # that row as 2d array
Out[555]: array([[ 0. , 0. , 0. , 0. , 0. , 0.4, 0. ]])
In [556]: print(Mc[1,:]) # like 2nd element of M except for row number
(0, 5) 0.4
Individual element:
In [560]: Mc[1,5]
Out[560]: 0.40000000000000002
The data attributes of these format (if you want to dig further)
In [562]: Mc.data
Out[562]: array([ 0.5, 0.4, 0.6])
In [563]: Mc.indices
Out[563]: array([0, 5, 3], dtype=int32)
In [564]: Mc.indptr
Out[564]: array([0, 1, 2, 3, 3, 3], dtype=int32)
In [565]: M.data
Out[565]: array([ 0.5, 0.4, 0.6])
In [566]: M.col
Out[566]: array([0, 5, 3], dtype=int32)
In [567]: M.row
Out[567]: array([0, 1, 2], dtype=int32)
Related
I am using scala spark. I have a dataframe that 2 column each containing a Vector with the same cardinality/size. I want to find the distance between each element of the 2 Vectors and put the results in a vector in another column of the dataframe.
Example: [1, 3, 5, -2] - [-2, 5, 0, 1] = [3, 2, 5, 3]
I found sqdist method that can get me the sum of the square distances between 2 Vectors but how do I get the individual distances of each elements in the vector.
How can I reshape efficiently and scipy.sparse csr_matrix?
I need to add zero rows at the end.
Using:
from scipy.sparse import csr_matrix
data = [1,2,3,4,5,6]
col = [0,0,0,1,1,1]
row = [0,1,2,0,1,2]
a = csr_matrix((data, (row, col)))
a.reshape(3,5)
I get this error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.5/dist-packages/scipy/sparse/base.py", line 129, in reshape
self.__class__.__name__)
NotImplementedError: Reshaping not implemented for csr_matrix.
If you can catch the problem early enough, just include a shape parameter:
In [48]: a = csr_matrix((data, (row, col)))
In [49]: a
Out[49]:
<3x2 sparse matrix of type '<class 'numpy.int64'>'
with 6 stored elements in Compressed Sparse Row format>
In [50]: a = csr_matrix((data, (row, col)),shape=(3,5))
In [51]: a
Out[51]:
<3x5 sparse matrix of type '<class 'numpy.int64'>'
with 6 stored elements in Compressed Sparse Row format>
In [52]: a.A
Out[52]:
array([[1, 4, 0, 0, 0],
[2, 5, 0, 0, 0],
[3, 6, 0, 0, 0]], dtype=int64)
You could also hstack on a pad. Make sure it's the sparse version:
In [59]: z = sparse.coo_matrix(np.zeros((3,3)))
In [60]: z
Out[60]:
<3x3 sparse matrix of type '<class 'numpy.float64'>'
with 0 stored elements in COOrdinate format>
In [61]: sparse.hstack((a,z))
Out[61]:
<3x5 sparse matrix of type '<class 'numpy.float64'>'
with 6 stored elements in COOrdinate format>
In [62]: _.A
Out[62]:
array([[1., 4., 0., 0., 0.],
[2., 5., 0., 0., 0.],
[3., 6., 0., 0., 0.]])
hstack uses sparse.bmat. That combines the coo attributes of the 2 arrays, and makes a new coo matrix.
The reshape() method will work with csr_matrix objects in scipy 1.1, which is close to being released. In the meantime, you can try the code at Reshape sparse matrix efficiently, Python, SciPy 0.12 for reshaping a sparse matrix.
Your example won't work, however, because you are trying to reshape an array with shape (3, 2) into an array with shape (3, 5). The code linked to above and the sparse reshape() method follow the same rules as the reshape() method of numpy arrays: you can't change the total size of the array.
If you want to change the total size, you will eventually be able to use the resize() method (which operates in-place), but that is also a new feature of scipy 1.1, so it is not yet released.
Instead, you can construct a new sparse matrix as follows:
In [57]: b = csr_matrix((a.data, a.indices, a.indptr), shape=(3, 5))
In [58]: b.shape
Out[58]: (3, 5)
In [59]: b.A
Out[59]:
array([[1, 4, 0, 0, 0],
[2, 5, 0, 0, 0],
[3, 6, 0, 0, 0]], dtype=int64)
So say I have the below matrix
[1, 2, 3,
4, 5, 6,
7, 8, 9]
And I have a vector [1,3]
I want to access the 1st and 3rd row which would return
[1,2,3
7,8,9]
I need to be able to scale this up to about 1000 rows being grabbed based on values in the vector.
if A is your matrix and v your vector of index, you just have to do A(v,:)
I have a matrix like this:
A = [1, 2, 3, 4, 5, NaN, NaN, NaN, NaN, NaN;
1, 2, 3, 4, 5, 6, 7, NaN, NaN, NaN;
1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
I would like to know how I can count the number of values in each row excluding any NaNs.
So I would get an output like:
output = [5;
7;
10;]
If A is a 2D array, e.g.
A = [1, 2, 3, 4, 5, NaN, NaN, NaN, NaN, NaN;
1, 2, 3, 4, 5, 6, 7, NaN, NaN, NaN;
1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
and you want to count the number of NaN entries on each row of A, you can simply use
>> sum(~isnan(A), 2)
ans =
5
7
10
Breakdown
isnan(A) returns a logical array of the same size as A, in which (logical1 indicates a NaN and 0 a non-NaN.
Note that you have to use the isnan function, here. In particular, the expression A == ~NaN is useless: it would simply return a logical array of the same size as A but full of (logical) 0's. Why? Because, according to floating-point arithmetic, NaN == NaN always returns "false" (i.e. logical 0, in MATLAB).
Then, by applying MATLAB's not operator (~) to that, you get a logical array of the same size as A, in which 1 indicates a non-NaN and 0 a NaN.
Finally, sum(~isnan(A), 2) returns a column vector in which the i-th entry corresponds to the number of logical 1's on the i-th row of ~isnan(A).
The resulting column vector is exactly what you want: a count, row by row, of the non-NaN entries in A.
I have a 3 Dimensional array Val 4xmx2 dimension. (m can be variable)
Val{1} = [1, 280; 2, 281; 3, 282; 4, 283; 5, 285];
Val{2} = [2, 179; 3, 180; 4, 181; 5, 182];
Val{3} = [2, 315; 4, 322; 5, 325];
Val{4} = [1, 95; 3, 97; 4, 99; 5, 101];
I have a subscript vector:
subs = {1,3,4};
What i want to get as output is the average of column 2 in the above 2D Arrays (only 1,3 an 4) such that the 1st columns value is >=2 and <=4.
The output will be:
{282, 318.5, 98}
This can probably be done by using a few loops, but just wondering if there is a more efficient way?
Here's a one-liner:
output = cellfun(#(x)mean(x(:,1)>=2 & x(:,1)<=4,2),Val(cat(1,subs{:})),'UniformOutput',false);
If subs is a numerical array (not a cell array) instead, i.e. subs=[1,3,4], and if output doesn't have to be a cell array, but can be a numerical array instead, i.e. output = [282,318.5,98], then the above simplifies to
output = cellfun(#(x)mean(x(x(:,1)>=2 & x(:,1)<=4,2)),Val(subs));
cellfun applies a function to each element of a cell array, and the indexing makes sure only the good rows are being averaged.