in my fortran code i am outputting the results into a binary file.
open(21,file=anum('press',itime),form=format_mode)
write(21) rtime,itime,dt,nx0,ny0,nz,deltax,deltay,rlenz
write(21) rw
close(21)
the above is the fortran code that writes and saves the file.
i now want to open and analyse it in matlab:
fid = open('press.420000');
A = fread(fid);
close(fid);
this however, only creates a 1d array which i am guessing includes all the header information too.
i want Matlab to read the header values but not include them into the final array. i intend to reshape the array in to a 3d array as the data is from a cfd simulation which has a grid of 256x512x390 = 51,180,80
the Matlab code gives me a 1d array of 411,343,976, which cannot be correct.
thus i am struggling how to read the binary file. I need some guidance on how i should code a Matlab script to read the binary file
You can read data in byte vector:
bytevec = fread(fid, inf, 'uint8');
Then you can look at and manually arrange elements by their indices, for example - single precision (float) data:
vec = typecast(bytevec(i1:i2), 'single');
And then convert it to default matlab double type without changing data values:
vec = cast(vec, 'double');
Finally, you can reshape raw vector to 3d matrix:
M = reshape(vec, [d1, d2, d3]);
Related
I have a binary square matrix with complex values, stored in a .bin format file. I have tried to read this 100-by-100 matrix with a Matlab script:
i=fopen('matrix.bin','r')
A=fread(i,[100 100]
This code does not correctly read the complex values contained in A. I only get a 100-by-100 matrix of integers.
MATLAB fread support ANSI C types, but there is no native ANSI C types that represent complex numbers. Most likely, a complex number is stored as a pair of real and imaginary numbers.
Without information as to how the binary file is saved, you can still perform some test to figure this out. If the complex number is represented as a real part and an imaginary part, and both in double precision, then a single complex number would take up 8 + 8 = 16 bytes. We can test this by navigating to the end of the file, and see how many bytes there are.
fID = fopen('matrix.bin','r')
fseek(fID, 0, 'eof') % Go to the end of file
ftell(fID) % Tell current position in the open file
fclose(fID)
If this number is equal to 16 * 100 * 100 = 160000, then you're in luck. There is no extra stuff saved in this file, and you can simply read the data by this code:
fID = fopen('matrix.bin','r')
data = []
for ii = 1:10000
data = [data; fread(fID, 2, 'double')']
end
fclose(fID)
You'll end up with a 10000*2 array, with each row representing a complex number. If the file size is 80000, then both real and imaginary part could be saved in single data type. If file size is some other number, then it probably means some additional information is stored in the binary. You'll have to know what additional information is stored so you can read the file correctly.
I have a very large matrix (M X N). I want to divide matrix into 10 equal parts (almost) and save each of them into a separate file say A1.txt, A2.txt, etc. or .mat format. How can I do this ?
Below is a code to divide a matrix into 10 equal parts and data_size is (M / 10).
for i=1:10
if i==1
data = DATA(1:data_size,:);
elseif i==10
data = DATA((i-1)*data_size+1:end,:);
else
data = DATA((i-1)*data_size+1: i*data_size,:);
end
save data(i).mat data
% What should I write here in order to save data into separate file data1.mat, data2.mat etc.
end
You said you wanted it in either txt format or mat format. I'll provide both solutions, and some of this is attributed to Daniel in his comment in your post above.
Saving as a text file
You can use fopen to open a file up for writing. This returns an ID to the file that you want to write to. After this, use fprintf and specify the ID to the file that you want to write to, and the data you want to write to this file. As such, with sprintf, generate the text file name you want, then use fprintf to write data to your file. It should be noted that writing matrices to fprintf in MATLAB assume column major format. If you don't want your data written this way and want it done in row-major, you need to transpose your data before you write this to file. I'll provide both methods in the code depending on what you want.
After you're done, use fclose to close the file noting that you have finished writing to it. Therefore, you would do this:
for i=1:10
if i==1
data = DATA(1:data_size,:);
elseif i==10
data = DATA((i-1)*data_size+1:end,:);
else
data = DATA((i-1)*data_size+1: i*data_size,:);
end
filename = sprintf('A%d.txt', i); %// Generate file name
fid = fopen(filename, 'w'); % // Open file for writing
fwrite(fid, '%f ', data.'); %// Write to file - Transpose for row major!
%// fwrite(fid, '%f ', data); %// Write to file - Column major!
fclose(fid); %// Close file
end
Take note that I space separated the numbers so you can open up the file and see how these values are written accordingly. I've also used the default precision and formatting by just using %f. You can play around with this by looking at the fprintf documentation and customizing the precision and leading zero formatting to your desire.
Saving to a MAT file
This is actually a more simpler approach. You would still use sprintf to save your data, then use the save command to save your workspace variables to file. Therefore, your loop would be this:
for i=1:10
if i==1
data = DATA(1:data_size,:);
elseif i==10
data = DATA((i-1)*data_size+1:end,:);
else
data = DATA((i-1)*data_size+1: i*data_size,:);
end
filename = sprintf('A%d.mat', i); %// Generate file name
save(filename, 'data');
end
Take note that the variable you want to save must be a string. This is why you have to put single quotes around the data variable as this is the variable you are writing to file.
You can use
save(['data' num2str(i) '.mat'], 'data');
where [ ] is used to concatenate strings and num2str to convert an integer to a string.
I have a MatLab program that generates a large 1000x1000 matrix. How can I save this matrix for use in future programs. Ideally, I want to save it as a particular variable. Here is the code that I am using.
function A = generateSPDmatrix(n)
A = rand(n,n); % generate a random n x n matrix
A = A+A';
A = A*A';
A = A + n*eye(n);
end
If you want to use it in future Matlab programs, you could do it like this:
save('A.mat', 'A');
To load, just do it like this:
load('A.mat');
% the file path is current path.
save('A.txt', 'A','-ascii');
% save to your file path
save('D:\test.txt','m','-ascii')
'D:\test.txt': file name and path
'm': your matrix
'-ascii': 8-digit ASCII format
See the matlab Help. Search the save(Save workspace variables to file) function
save(filename, variables, format) saves in the specified format: '-mat' or '-ascii'. You can specify the format option with additional inputs such as variables, '-struct' , '-append', or version.
I am using mex bridge to perform some operations on Sparse matrices from Matlab.
For that I need to convert input matrix into CSR (compressed row storage) format, since Matlab stores the sparse matrices in CSC (compressed column storage).
I was able to get value array and column_indices array. However, I am struggling to get row_pointer array for CSR format.Is there any C library that can help in conversion from CSC to CSR ?
Further, while writing a CUDA kernel, will it be efficient to use CSR format for sparse operations or should I just use following arrays :- row indices, column indices and values?
Which on would give me more control over the data, minimizing the number for-loops in the custom kernel?
Compressed row storage is similar to compressed column storage, just transposed. So the simplest thing is to use MATLAB to transpose the matrix before you pass it to your MEX file. Then, use the functions
Ap = mxGetJc(spA);
Ai = mxGetIr(spA);
Ax = mxGetPr(spA);
to get the internal pointers and treat them as row storage. Ap is row pointer, Ai is column indices of the non-zero entries, Ax are the non-zero values. Note that for symmetric matrices you do not have to do anything at all! CSC and CSR are the same.
Which format to use heavily depends on what you want to do with the matrix later. For example, have a look at matrix formats for Sparse matrix vector multiplication. That is one of the classic papers, research has moved since then so you can look around further.
I ended up converting CSC format from Matlab to CSR using CUSP library as follows.
After getting the matrix A from matlab and I got its row,col and values vectors and I copied them in respective thrust::host_vector created for each of them.
After that I created two cusp::array1d of type Indices and Values as follows.
typedef typename cusp::array1d<int,cusp::host_memory>Indices;
typedef typename cusp::array1d<float,cusp::host_memory>Values;
Indices row_indices(rows.begin(),rows.end());
Indices col_indices(cols.begin(),cols.end());
Values Vals(Val.begin(),Val.end());
where rows, cols and Val are thrust::host_vector that I got from Matlab.
After that I created a cusp::coo_matrix_view as given below.
typedef cusp::coo_matrix_view<Indices,Indices,Values>HostView;
HostView Ah(m,n,NNZa,row_indices,col_indices,Vals);
where m,n and NNZa are the parameters that I get from mex functions of sparse matrices.
I copied this view matrix to cusp::csr_matrixin device memory with proper dimensions set as given below.
cusp::csr_matrix<int,float,cusp::device_memory>CSR(m,n,NNZa);
CSR = Ah;
After that I just copied the three individual content arrays of this CSR matrix back to the host using thrust::raw_pointer_cast where arrays with proper dimension are already mxCalloced as given below.
cudaMemcpy(Acol,thrust::raw_pointer_cast(&CSR.column_indices[0]),sizeof(int)*(NNZa),cudaMemcpyDeviceToHost);
cudaMemcpy(Aptr,thrust::raw_pointer_cast(&CSR.row_offsets[0]),sizeof(int)*(n+1),cudaMemcpyDeviceToHost);
cudaMemcpy(Aval,thrust::raw_pointer_cast(&CSR.values[0]),sizeof(float)*(NNZa),cudaMemcpyDeviceToHost);
Hope this is useful to anyone who is using CUSP with Matlab
you can do something like this:
n = size(M,1);
nz_num = nnz(M);
[col,rowi,vals] = find(M');
row = zeros(n+1,1);
ll = 1; row(1) = 1;
for l = 2:n
if rowi(l)~=rowi(l-1)
ll = ll + 1;
row(ll) = l;
end
end
row(n+1) = nz_num+1;`
It works for me, hope it can help somebody else!
I am thinking of writing the data to a file. Does anyone have an example of how to write a big amount of data to a file?
Edit: Most elements in the matrix are zeroes, others are uint32. I guess the simplest save() and load() would work, as #Jonas suggested.
I guess nobody's seen the edit about the zeroes :)
If they're mostly zeroes, you should convert your matrix to its sparse representation and then save it. You can do that with the sparse function.
Code
z = zeros(10000,10000);
z(123,456) = 1;
whos z
z = sparse(z);
whos z
Output
Name Size Bytes Class Attributes
z 10000x10000 800000000 double
Name Size Bytes Class Attributes
z 10000x10000 40016 double sparse
I don't think the sparse implementation is designed to handle uint32.
If you're concerned with keeping the size of the data file as small as possible, here are some suggestions:
Write the data to a binary file (i.e. using FWRITE) instead of to a text file (i.e. using FPRINTF).
If your data contains all integer values, convert it to or save it as a signed or unsigned integer type instead of the default double precision type MATLAB uses.
If your data contains floating point values, but you don't need the range or resolution of the default double precision type, convert it to or save it as a single precision type.
If your data is sufficiently sparse (i.e. there are many more zeroes than non-zeroes in your matrix), then you can use the FIND function to get the row and column indices of the non-zero values, then just save these to your file.
Here are a couple of examples to illustrate:
data = double(rand(16,2^20) <= 0.00001); %# A large but very sparse matrix
%# Writing the values as type double:
fid = fopen('data_double.dat','w'); %# Open the file
fwrite(fid,size(data),'uint32'); %# Write the matrix size (2 values)
fwrite(fid,data,'double'); %# Write the data as type double
fclose(fid); %# Close the file
%# Writing the values as type uint8:
fid = fopen('data_uint8.dat','w'); %# Open the file
fwrite(fid,size(data),'uint32'); %# Write the matrix size (2 values)
fwrite(fid,data,'uint8'); %# Write the data as type uint8
fclose(fid); %# Close the file
%# Writing out only the non-zero values:
[rowIndex,columnIndex,values] = find(data); %# Get the row and column indices
%# and the non-zero values
fid = fopen('data_sparse.dat','w'); %# Open the file
fwrite(fid,numel(values),'uint32'); %# Write the length of the vectors (1 value)
fwrite(fid,rowIndex,'uint32'); %# Write the row indices
fwrite(fid,columnIndex,'uint32'); %# Write the column indices
fwrite(fid,values,'uint8'); %# Write the non-zero values
fclose(fid); %# Close the file
The files created above will differ drastically in size. The file 'data_double.dat' will be about 131,073 KB, 'data_uint8.dat' will be about 16,385 KB, and 'data_sparse.dat' will be less than 2 KB.
Note that I also wrote the data\vector sizes to the files so that the data can be read back in (using FREAD) and reshaped properly. Note also that if I did not supply a 'double' or 'uint8' argument to FWRITE, MATLAB would be smart enough to figure out that it didn't need to use the default double precision and would only use 8 bits to write out the data values (since they are all 0 and 1).
How is the data generated? How do you need to access the data?
If I calculate correctly, the variable is less than 200MB if it's all double. Thus, you can easily save and load it as a single .mat file if you need to access it from Matlab only.
%# create data
data = zeros(16,2^20);
%# save data
save('myFile.mat','data');
%# clear data to test everything works
clear data
%# load data
load('myFile.mat')