I am currently taking a Neural Network course, and I am trying to load a .mat file from Python using scipy.io.loadmat(filename), but I keep getting the following error message:
ValueError: Unknown mat file type, version 101, 58
The same message occured when I tried loading a .m file, rather than .mat file, using scipy.io.loadmat().
I don't know how to resolve this issue, so I would really appreciate if someone on here could help me out.
(converting my comments into an answer)
A .m file is a matlab / octave script. You don't "load" a script. Not even in matlab / octave. You can only run it, and this may or may not result in a particular workspace that you can then save into a .mat file.
There are "matlab interfaces" for python that you could use to run an .m file from within a python session, e.g. the official MATLAB Engine API for Python, or if you have an older matlab version you could try finding other python matlab interfaces online (a simple google search gave me pymatlab, mlab, mlabwrap etc). If your .m file is octave-compatible (and you have octave installed) you could also try the oct2py interface which is known to work well. Once you have one of those interfaces installed, you could either attempt to run the script and then transfer the workspace variables onto python, or save to a .mat file using the interface, and then load it back onto python using scipy.io.loadmat as normal.
As for the .mat file giving you an error, the most likely reason for this is that it's a newer version of the .mat specification, which python doesn't recognise yet (what matlab version are you using?). Try loading the .mat file in matlab (or octave) and saving it again with the -v7 option. Python should then open this without problems.
EDIT: Here's an example running an octave script via oct2py, saving the resulting octave workspace onto a .mat file, and loading the .mat file into python.
%% file myscript.m located at /home/tasos/Desktop
a = 1
b = 2
c = 3
# from python:
from oct2py import octave as oct
oct.eval("cd /home/tasos/Desktop")
oct.eval("myscript")
oct.eval("save -v7 myworkspace.mat")
from scipy.io import loadmat
D = loadmat("/home/tasos/Desktop/myworkspace.mat")
print D.keys()
#>>> ['a', 'c', 'b', '__header__', '__globals__', 'ans', '__version__']
print D['a'], D['b'], D['c']
#>>> [[ 1.]] [[ 2.]] [[ 3.]]
For people coming over this issue again. You can use numpy.loadtxt() to read .mat files generated by Octave.
For example.
In Octave:
octave:1> arr1 = [1,2]
octave:2> arr2 = [2,3]
octave:3> save mydata.mat arr1 arr2
Then read mydata.mat in Python Interpreter:
>>> import numpy as np
>>> data = np.loadtxt('mydata.mat')
>>> data
array([[1., 2.],
[2., 3.]])
You cannot load .m script-files. These are MATLAB's own script-files and they contain MATLAB code. The function scipy.io.loadmat cannot parse this code.
On the other hand, the function can load data from a MATLAB .mat-file. This is MATLAB's own data-format within which you store variables and data - but not code.
Reference: https://docs.scipy.org/doc/scipy/reference/generated/scipy.io.loadmat.html.
Much easier is to just load your data with:
numpy.loadtxt('your dot mat file')
You can use MATLAB Engine API to run .m files and MATLAB functions. it can also load .mat files as well, but could be a bit slow.
https://www.mathworks.com/help/matlab/matlab-engine-for-python.html
Related
there are datasets in .mat format in the this site: http://www.cs.nyu.edu/~roweis/data.html
I want to change the format to .csv.
Can someone tell me how to change the format to create the .csv file.
Thanks!
Suppose that the .mat files from the site are available already. In the command window in Matlab, you may write, for example:
load('C:\Users\YourUserName\Downloads\mnist_all.mat');
to load the .mat file; the result should be a set of matrices test0, test1, ..., train0, train1 ... created in your workspace, which you want saved as CSV files. Because they're different size, you need to save one CSV per variable, e.g. (also in the command window):
csvwrite('C:\Users\YourUserName\Downloads\mnist_test0.csv', test0);
Repeat the command for each variable, and do not forget to change also the name of the output file to avoid overwriting.
Did you tried the csvwrite function in Matlab?
Just load your .mat files with the load function and then write them with csvwrite!
I do not have a Matlab license so I installed GNU Octave 4.2.1 (2017) on Windows 10 (thank you to John W. Eaton and others). I was not fully successful using the csvwrite so I used the following workaround. (BTW, I am totally incompetent in the Octave world. csvwrite worked for simple data structures).
In the Command Window I used the following two commands
load myfile.mat
save("-text","myfile.txt","variablename")
When the "myfile.mat" is loaded, the variable names for the data vectors loaded are displayed in the workspace window. This is the name(s) to use in the save command. Some .mat files will load several data structures.
The "-text" option is the default, so you may not need to include this option in the command.
The output file lists the .mat file contents in text format as single column (of potentially sequential variables). It should be easy to use you text editor to massage this data into the original matrix structure for use in whatever app you are comfortable with.
Had a similar issue. Needed to convert a series of .mat files that had two columns of numerical data into standard data files (ascii text). Note that I don't really ever use csv, but everything here could be adapted by using csvwrite instead of the standard save.
Using Octave 4.2.1 ....
load myfile.mat
LI = [L, I] ## L and I are column vectors representing my data
save myfile.txt LI
Note that L and I appear to be default variable names chosen by Octave for the two columns vectors in my original data file. Ideally a script that iterated over all files with the .mat extension in my directory would be ideal, but this got the job done. It saves the data as two space separated columns of data.
*** Update
The following script works on Octave 4.2.1 for a series of data files with the .mat extension that are in the same directory. It will iterate over them and write the data out to text files with the same name but with the extension .dat . Note that this is not efficient, so if you have a lot of files or if they are large it can take a while to run. I would suggest that you run it from the command line using octave mat2dat.m so you can actually watch it go.
I make no guarantees that this will work for you, but it did for me. I also am NOT proficient in Octave or Matlab, so I'm sure a better solution exists.
# mat2dat.m
dirlist = glob("*.mat")
for i=1:length(dirlist)
filename = dirlist{i,1}
load(filename, "L", "I")
LI = [L,I]
tmpname = filename(1:length(filename)-3)
txtname = strcat(tmpname, 'dat')
save(txtname, "LI")
end
Background
Say I compile the following simple function in MATLAB
function foo(path_to_m_file)
disp([' Running ' path_to_m_file])
run(path_to_m_file);
end
The function foo just takes a path to an .m file and tries to run it.
However, when I actually try to run foo after compiling it:
./run_foo.sh $path_to_run_time $path_to_m_file
where path_to_m_file is a simple .m file with a statement such as:
a = 2;
I get the following error:
Error using ==> run
MATLAB:run:FileNotFound
However, I know that foo gets the correct path. For example, if I try replacing the line with run by the following two lines in foo
fID = fopen(conf_file, 'rt');
first_line = textscan(fID, '%s', Inf, 'Delimiter', '\n');
foo reads the corresponding line of the .m file. So the .m file is there, and the MATLAB engine can "see" it. Indeed I can even run eval on strings read with textscan.
So my questions are:
Why do I get the error above? Why doesn't foo run the .m file?
Update: See #strictlyrude27's answer below for what seems to be an answer to this question.
If the above doesn't work. Is there a way to get a MATLAB-compiled function to run an .m file that may have changed after compiling the original function?
The motivation for my second question:
I would like to have the ability to "update" an .m file that is part of the project without having to re-compile the full project. Any ideas for this would be greatly appreciated.
From the MATLAB Compiler's documentaton:
Compiled Applications Do Not Process MATLAB Files at Runtime
The MATLAB Compiler was designed so that you can deploy locked down functionality. Deployable MATLAB files are suspended or frozen at the time MATLAB Compiler encrypts them—they do not change from that point onward. This does not mean that you cannot deploy a flexible application—it means that you must design your application with flexibility in mind. If you want the end user to be able to choose between two different methods, for example, they both must be compiled in.
The MCR only works on MATLAB code that was encrypted when the component was built. Any function or process that dynamically generates new MATLAB code will not work against the MCR.
Some MATLAB toolboxes, such as the Neural Network Toolbox™ product, generate MATLAB code dynamically. Because the MCR only executes encrypted MATLAB files, and the Neural Network Toolbox generates unencrypted MATLAB files, some functions in the Neural Network Toolbox cannot be deployed.
Similarly, functions that need to examine the contents of a MATLAB function file cannot be deployed. HELP, for example, is dynamic and not available in deployed mode. You can use LOADLIBRARY in deployed mode if you provide it with a MATLAB function prototype.
Instead of compiling the function that generates the MATLAB code and attempting to deploy it, perform the following tasks:
Run the code once in MATLAB to obtain your generated function.
Compile the MATLAB code with MATLAB Compiler, including the generated function.
Tip: Another alternative to using EVAL or FEVAL is using anonymous function handles.
If you require the ability to create MATLAB code for dynamic run time processing, your end users must have an installed copy of MATLAB.
You can read read an m file, line by line and execute each line with the eval() function. There are restrictions on the format of the m file (no line breaks for example, each line must contain a complete MATLAB statement) but it does work and can add to your run time environment inside the compiled application. I use this technique to allow users to define configuration and data files for a compiled application I have developed.
Clearly, if your end user provides a poorly formed m file to evaluate, you will end up with difficult to resolve bugs.
I've downloaded a data set that I am interested in. However, it is in .mat format and I do not have access to Matlab.
I've done some googling and it says I can open it in SciLab.
I tried a few things, but I haven't found any good tutorials on this.
I did
fd = matfile_open("file.mat")
matfile_listvar(fd)
and that prints out the filename without the extension. I tried
var1 = matfile_varreadnext(fd)
and that just gives me "var1 = "
I don't really know how the data is organized. The repository described the data it contains, but not how it is organized.
So, my question is, what am I doing wrong in extracting/viewing this data? I'm not committed to SciLab, if there is a better tool for this I am open to that.
One options is to use Octave, which can read .mat files and run most Matlab .m files. Octave is open source with binaries available for Linux, Mac, and Windows. Inside of Octave you can load the file using:
load file
See Octave's manual section 14.1.3 Simple File I/O for more details.
In Scilab:
loadmatfile('file.mat');
(Source)
I had this same interest a few years back. I used this question as a guide. It uses Python and SciPy. There are options for NumPy and hd5f as well. Another option is to write your own reader for the .mat format in whatever language you need. Here is the link to the mat file format definition.
Background
Say I compile the following simple function in MATLAB
function foo(path_to_m_file)
disp([' Running ' path_to_m_file])
run(path_to_m_file);
end
The function foo just takes a path to an .m file and tries to run it.
However, when I actually try to run foo after compiling it:
./run_foo.sh $path_to_run_time $path_to_m_file
where path_to_m_file is a simple .m file with a statement such as:
a = 2;
I get the following error:
Error using ==> run
MATLAB:run:FileNotFound
However, I know that foo gets the correct path. For example, if I try replacing the line with run by the following two lines in foo
fID = fopen(conf_file, 'rt');
first_line = textscan(fID, '%s', Inf, 'Delimiter', '\n');
foo reads the corresponding line of the .m file. So the .m file is there, and the MATLAB engine can "see" it. Indeed I can even run eval on strings read with textscan.
So my questions are:
Why do I get the error above? Why doesn't foo run the .m file?
Update: See #strictlyrude27's answer below for what seems to be an answer to this question.
If the above doesn't work. Is there a way to get a MATLAB-compiled function to run an .m file that may have changed after compiling the original function?
The motivation for my second question:
I would like to have the ability to "update" an .m file that is part of the project without having to re-compile the full project. Any ideas for this would be greatly appreciated.
From the MATLAB Compiler's documentaton:
Compiled Applications Do Not Process MATLAB Files at Runtime
The MATLAB Compiler was designed so that you can deploy locked down functionality. Deployable MATLAB files are suspended or frozen at the time MATLAB Compiler encrypts them—they do not change from that point onward. This does not mean that you cannot deploy a flexible application—it means that you must design your application with flexibility in mind. If you want the end user to be able to choose between two different methods, for example, they both must be compiled in.
The MCR only works on MATLAB code that was encrypted when the component was built. Any function or process that dynamically generates new MATLAB code will not work against the MCR.
Some MATLAB toolboxes, such as the Neural Network Toolbox™ product, generate MATLAB code dynamically. Because the MCR only executes encrypted MATLAB files, and the Neural Network Toolbox generates unencrypted MATLAB files, some functions in the Neural Network Toolbox cannot be deployed.
Similarly, functions that need to examine the contents of a MATLAB function file cannot be deployed. HELP, for example, is dynamic and not available in deployed mode. You can use LOADLIBRARY in deployed mode if you provide it with a MATLAB function prototype.
Instead of compiling the function that generates the MATLAB code and attempting to deploy it, perform the following tasks:
Run the code once in MATLAB to obtain your generated function.
Compile the MATLAB code with MATLAB Compiler, including the generated function.
Tip: Another alternative to using EVAL or FEVAL is using anonymous function handles.
If you require the ability to create MATLAB code for dynamic run time processing, your end users must have an installed copy of MATLAB.
You can read read an m file, line by line and execute each line with the eval() function. There are restrictions on the format of the m file (no line breaks for example, each line must contain a complete MATLAB statement) but it does work and can add to your run time environment inside the compiled application. I use this technique to allow users to define configuration and data files for a compiled application I have developed.
Clearly, if your end user provides a poorly formed m file to evaluate, you will end up with difficult to resolve bugs.
I have some data generated in MATLAB that I want to process using Perl. I saved the data from MATLAB in a .mat file. Is there any way to read it in Perl?
One option would be to save the binary MAT file as ASCII from inside MATLAB using something like:
load('test_data.mat');
save('test_data.asc', 'var1', 'var2', '-ascii');
Then you would have ASCII data to process in Perl.
If you need a solution completely written in Perl, then you should be able to automate the process using the Math::MATLAB package on CPAN.
NOTE: If Python is an option, you could use the loadmat function in the SciPy Python library.
The Java library JMatIO has worked well for me. Maybe you can try using inline Java.