I have a lot of .txt files with sizes from 100MB to 300MB and I want to transfer all the data that is in them to .mat files. The data is mostly numbers and I am looking to make the storage space as small as possible. Right now what I've done is read the data in the .txt files, put them in a struct and then save it using a v7.3 compression scheme, but each variable then comes out to be almost 9 GB. Would anyone have an idea about how I can make this better ?
You can save to MAT file v7 as noted in the documentation:
Version 7.3 MAT-files use an HDF5 based format that requires some overhead storage to describe the contents of the file. For cell arrays, structure arrays, or other containers that can store heterogeneous data types, Version 7.3 MAT-files are sometimes larger than Version 7 MAT-files.
Related
I have a pretty large HDF5 dataset which is of size [1 12672 1 228020] following the format:[height width channel N]. This file occupies about 22G on hard disk.
I want to partition this file in to smaller parts, say 2G files.
h5repart has been tried out but it does not work well, because I'm not able to display partitioned files in MATLAB using h5disp('...').
One solution would be for you to use the 'chunk' capability of the HDF5 format.
Using the MATLAB low-level HDF5 functions you should be able to read the chunks you require.
I know ILNumerics handles mat file saved using -v6 or -v7.3 option quite nicely but anyone knows how to make it recognize files saved using -v7 option?
The -v7 option by default uses compression and hence ILNumerics will give me a message saying it does not support compressed file and suggests me using -v6 option to save the file.
I guess nowdays mat file saved using latest releases of matlab (like 2010 - 2014) prefer the -v7 file format against the -v7.3 format which internally uses HDF5 because of the overhead HDF5 causeds. And since compression it's also smaller in size than the -v6 file which also is a big advantage.
So can ILNumerics fill this gap? Or is there already a solution that I missed?
Thanks.
I have identical code in Matlab, identical data that was analyzed using two different computers. Both are Win 7 64 bit. Both Matlabs are 2014-a version. After the code finishes its run, I save the variables using save command and it outputs .mat file.
Is it possible to have two very different memory sizes for these files? Like one being 170 MB, and the other being 2.4 GB? This is absurd because when I check the variables in matlab they add up to maybe 1.5 GB at most. What can be the reason for this?
Does saving to .mat file compress the variables (still with the regular .mat extension)? I think it does because when I check the individual variables they add up to around 1.5 GB.
So why would one output smaller file size, but the other just so huge?
Mat in recent versions is HDF5, which includes gzip compression. Probably on one pc the default mat format is changed to an old version which does not support compression. Try saving specifying the version, then both PCs should result in the same size.
I found the reason for this based on the following stackoverflow thread: MATLAB: Differences between .mat versions
Apparently one of the computers was using -v7 format which produces much smaller files. - v7.3 just inflates the files significantly. But this is ironical in my opinion since -v7.3 enables saving files larger than 2 GB, which means they will be much much larger when saved in .mat file.
Anyway this link is very useful.
Update:
I implemented the serialization mentioned in the above link, and it increased the file size. In my case the best option will be using -v7 format since it provides the smallest file size, and is also able to save structures and cell arrays that I use a lot.
I'm finding that writing and reading the native mat file format becomes very, very slow with larger data structures of about 1G in size. In addition we have other, non-matlab, software that should be able to read and write these files. So I would to find an alternative format to use to serialize matlab data structures. Ideally this format would ...
be able to represent an arbitrary matlab structure to a file.
have faster I/O than than mat files.
have I/O libraries for other languages like Java, Python and C++.
Simplifying your data structures and using the new v7.3 MAT file format, which is a variant of HDF5, might actually be the best approach. The HDF5 format is open and already has I/O libraries for your other languages. And depending on your data structure, they may be faster than the old binary mat files.
Simplify the data structures you're saving, preferring large arrays of primitives to complex container structures.
Try turning off compression if your data structures are still complex.
Try the v7.3 MAT file format using "-v7.3"
If using a network file system, consider saving and loading to a temporary dir on a fast local drive and copying to/from the network
For large data structures, your MAT file I/O speed may be determined more by the internal structure of the data you're writing out than the size of the resulting MAT file itself. (In my experience, this has usually been the major factor in slow MAT files.) When you say "arbitrary Matlab structure", that suggests you might be using cells, structs, or objects to make complex data structures. That slows down MAT I/O because there is per-array overhead in MAT file I/O, and the members of cell and struct arrays (container types) all count as separate arrays. For example, 5,000 strings stored in a cellstr are much, much slower than the same 5,000 strings stored in a 2-D char array. And objects have even more overhead. As a test, try writing out a 1 GB file that contains just a 1 GB primitive array of random uint8s, and see how long that takes. From there, see if you can simplify your data to reduce the total mxarray count, even if that means reshaping it for serialization. (My experience with this is mostly with the v7 format; the newer HDF5 format may have less per element overhead.)
If your data files live on the network, you could also try doing the save and load operations on temporary files on fast local drives, and separately using copy operations to move them back and forth between the network. At least on Windows networks, I've seen speedups of up to 2x from doing this. Possibly due to optimizations the full-file copy operation can do that the MAT I/O code can't.
It would probably be a substantial effort to come up with an alternate file format that supported fully arbitrary Matlab data structures and was portable to other languages. I'd try making smaller changes around your use of the existing format first.
mat format has changed with Matlab versions. v7.3 uses HDF5 format, which has builtin compression and other features and it can take a large time to read/write. However, you can force Matlab to use previous formats which are faster (but might take more space).
See here:
http://www.mathworks.com/help/matlab/import_export/mat-file-versions.html
I would like to take a 512x512 image and convert it into a png byte array in Matlab so that I can stream it via a socket.
At the moment I take the array, write it to a png file using imwrite(I,'file.png'), then read it as a binary file and send it through the socket. This is obviously horribly inefficient because I first write to disk and then read from disk. I want to skip the and write to disk.
Is there anyway to do this in Matlab?
Probably not directly using the base MATLAB toolbox since the PNG file itself is created by the PNGWRITEC MEX-function. However, there may be some Java classes that can help, such as those in the javax.imageio package.