Efficiency loading a file into Matlab - matlab

I am trying to load a file in Matlab. But I am a bit confused about the best way to do it.
The file has 3 columns and looks like the screenshot below:
This file I can load very quickly by doing load('c').
However, I had to add 2 NaNs on the bottom row.
The original file actually looks like the file below:
Now if I do load('c') on the file below I get the error:
Error using load
Unable to read file 'c'. Input must be a MAT-file or an ASCII file containing numeric
data with same number of columns in each row.
Of course I can use ImportData to import this file, but it is just soooo slow to import it.
Any suggestions?

You should be able to use c = readtable('c'). This should automatically change the empty entries to "NaN" by default, but if not, there is a way to set that in the options.
If I have a file that is tricky to import (prior to readtable()...that made things a lot easier in the last few years), I will often use the Import Data tool (if its a really big file you can make a mock-up of the complicated file so it loads faster) then change all the import settings as I would want it, then where the green check says "Import Selection" use the black drop down arrow to select "Generate Function." This will give you the coded way of setting everything up to get the file in just the way you want it.
load() is better suited for reading in previously saved '.mat' files that were created in Matlab.

Here's a low-level approach, which might be faster than other methods:
filename = 'c'; % name of the file
N = 3; % number of columns
fid = fopen(filename, 'r'); % open file for reading
x = fscanf(fid, '%f'); % read all values as a column vector
fclose(fid); % close file
x = [x; NaN(N-mod(numel(x)-1,N)-1, 1)]; % include NaN's to make length a multiple of N
x = reshape(x, N, []).'; % reshape to N columns in row-major order

Related

Numerical Matrix CSV -> Covariance Matrix

I have a CSV file containing a numerical matrix with over two thousand New York Stock Exchange listed companies' value over two years.
It seems like it should be really simple - I want to attain a covariance matrix formed from the CSV matrix.
As far as I'm aware I simply need to:
Import the data as numerical matrix (just the data no headings etc) using the MATLAB Import Data button.
Press save as on the workspace variable and make e.g. NYSE.mat.
In my function call cov(NYSE.mat);
This should access the matrix and return a large covariance matrix from my data. The cov() function works when i manually input an example matrix of for example:
[5 0 3 7; 1 -5 7 3; 4 9 8 10];
But for some reason whenever I try to call cov(NYSE.mat); only one number is returned, rather than a covariance matrix.
Can anyone tell me where I'm going wrong? I've been trying to figure this out for a while now and I feel the answer should be really simple.
I'm running on MATLAB R2016a.
Not sure you need to manually save the workspace name in step 2. As part of your import process you once you click the import button the variable should be loaded into workspace with the name of the file (maybe NYSE)
Try Load('NYSE.mat') and see what appears in your workspace.
Once you figure out the name of the variable, calling the function with that.
If your CSV file is without any header lines (not sure if any CSV file needs them) and contains only numbers, you can simply call
x = importdata('<nameofyourcsv>.csv');
and then
c = cov(x);
or even do it in one step:
c = cov(importdata('nameofcsv.csv'));
This function is equivalent of clicking on the button you mention. However, if your file contains header lines, you have to specify its number and call this function with more parameters:
x = importdata('nameofcsv.csv',',', nh); % nh is a number of header lines
x will no longer be a regular MATLAB matrix but a structure with following fields: data, textdata, and colheaders. cov(x.data) will calculate covariance. Not sure why you would need to save anything in a mat file, though. If you wish, you would probably need to assign a value from a data field to a regular variable and then save such variable. Otherwise, you will save the whole x structure.
For instance, the following file, test.csv:
95.01,76.21,61.54,40.57,5.79,20.28,1.53
23.11,45.65,79.19,93.55,35.29,19.87,74.68
60.68,1.85,92.18,91.69,81.32,60.38,44.51
48.60,82.14,73.82,41.03,0.99,27.22,93.18
89.13,44.47,17.63,89.36,13.89,19.88,46.60
can be imported by calling importdata:
>> x = importdata('test.csv')
x =
95.0100 76.2100 61.5400 40.5700 5.7900 20.2800 1.5300
23.1100 45.6500 79.1900 93.5500 35.2900 19.8700 74.6800
60.6800 1.8500 92.1800 91.6900 81.3200 60.3800 44.5100
48.6000 82.1400 73.8200 41.0300 0.9900 27.2200 93.1800
89.1300 44.4700 17.6300 89.3600 13.8900 19.8800 46.6000
EDIT
I downloaded the file and copy pasted this command from your answer:
c = importdata('nyse_data_matrix_no_tags.csv');
Then, I opened the file in a text editor and copied 1st and 20th row of numbers in the file and created a matrix in MATLAB:
x = [46.09,24.69,156.78,5.95,21.14,76.17,55.51,7.04,38.87,19.58,47.57,7.73,119.44,1.61,44.55,24.9,50.89,4.87,26.25,15.95,15.96,39.14,121.27,15.7,25.91,25.8,69.7,16.32,12.86,7.89,247.4,27.11,41.6,5.14,47.77,40.98,13.78,26.058,14.56,41.05,24.27,47.13,40.92,27,85.04,15.06,5.05,29.14,7.51,67.5,67.79,42.68,124.86,24.28,27.82,18.08,100.67,109.07,12.59,50.34,18.64,4.75,6.06,16.63,109.86,14.1,54.48,13.9,59.98,16.24,45.53,4.06,67.99,18.22,4.2,117.23,158.75,5.7,2.46,76.39,77.79,6.17,16.95,5.27,12.74,14.7,19.14,14.4,42.77,26.4,14.17,76.57,12.4,42.07,77.47,41.27,60.53,16.97,65.71,56.13,258.4,28.96,760.07,60.46,34.77,133.1,14.19,29.41,11.78,154.55,44.75,15.18,17.52,24.86,36.18,14.1,30.1,47.65,22.92,47.37,61.84,226.52,10.2,65.14,7.88,16.34,170.23,9.6,34.42,15.0999,20.34,58.7,16.6,14.72,14.32,20.89,14.32,13.2,6.05,422.96,21.43,47.55,13.28,27.98,8.16,45.15,15.1665,41.65,17.45,25.73,63.03,16.07,17.56,11.54,6.54,33.18,15.01,357.96,74.88,15.24,27.24,69.08,36.29,76.55,65.72,50.93,72.73,16.13,23.81,52.14,16.05,12.19,71.77,14.97,11.82,33.04,7.86,15.26,72.46,15.42,24.49,15.75,63.78,32.43,36.57,16.24,7.08,26.08,15.74,14.78,16.22,16.85,23.32,31.14,12.72,6.7,26.65,24,12.95,129.92,20.07,11.28,34.66,12.86,17.5,26.5,0.7038,1.22,128.98,23.33,19.2,15.64,8.34,45.01,38.14,50.31,13.11,17.4,47.15,79.29,8.24,28.92,24.97,1.92,77.16,126.5,8.97,3.97,12.85,30.75,37.82,2.36,10.37,52.28,1.56,53,47.33,41.32,10.65,15.24,39.5,94.42,12,54.25,48.03,3.72,6.34,18.5,23.47,26.74,8.74,6.7,7.08,70.92,27.3,65.43,18.776,7.82,16.8,12.0625,124.24,29.79,22.88,29.4,66.44,28.88,1.505,0.8642,10.37,47.14,99.83,133.87,46.41,4.88,57.63,14.45,11.2,34.99,11.06,129.25,7.71,35.2,2.29,2.53,1.65,13.74,15.6,11.84,59.79,33.85,62.48,72.34,25.15,131.2,61.13,2.17,5.1,53.35,26.9,43.15,10.8,63.61,130.91,81.42,45.29,17.58,415.79,118.64,73.88,9.85,79.22,43.39,4.86,32.18,68.36,60.21,34.15,19.47,23.39,29.96,39.58,14.66,5.98,70.87,25.9,38.64,90.45,165.2,46.57,83.18,16.48,134.15,54.68,62.51,12.25,24.33,17.69,60.73,11.51,15.6,85.99,80.81,59.9,17.48,30.73,104.34,0.94,86.03,82.73,14.41,34.05,17.2,11.98,13,52.5,39.2,19.58,101.59,26.38,44.13,18.72,23.3,32.05,27,26.12,13.13,18.58,6.07,73.74,3.53,42.4,12.21,15.72,16.71,26.27,26.69,4.52,35.99,26.1,17.31,45.61,67.9,11.15,13.08,1.1,9.7001,17.51,59.8,26.27,86.95,18.95,26.75,34.33,66.72,107.98,9.7,18.27,24.16,56.8,46.56,91.69,21.43,78.75,3.37,13.71,31.73,100.39,5.65,6.98,85.21,97.84,13.5,26.19,42.43,26.73,10.11,47.95,102.84,66.95,28.35,14.07,5.25,23.68,127.43,11.41,10.43,4.19,25.48,19.78,72.08,53.59,17.41,36.57,92.37,127.48,23.98,16.46,5.28,24.76,9.1,68.33,64.45,8.15,18.54,8.85,119.72,3.62,20.49,2.57,94.05,16.99,25.93,25.12,25.71,9.68,81.26,16.7,77.21,37.45,80.43,6.95,24.95,87.1,20.74,31.82,25.44,25.48,25.5515,46.96,19.11,43.49,10,8.68,16.99,120.79,3.95,1.91,76.54,7.8,33.71,14.59,13.5,15.99,42.6,38.86,46.76,8.1,23.14,23.04,17.39,15.4399,13.62,14,125.94,13.92,22.98,48.68,4.62,68.17,1.14,9.82,29.75,73.66,6.51,92.28,25.53,25.6,25.63,7.59,72.9,25.23,10.41,27.87,10.81,48.76,11.38,3.76,71.78,13.21,53.59,25.55,42.43,68.21,66.25,37.2,6.1,6.24,84.58,13.17,13.2,22.92,82.95,28.9,6.07,74.57,29.35,74.89,64.46,77.91,30.21,11.09,6.34,21.69,56.88,15.25,41.44,1.77,67.42,209.26,11.24,15.8,13.53,15.09,32.8,9.78,12.99,62.77,21.8,39.11,78.68,14.75,10.67,20.6,10.57,36.39,7.24,6.1,13.61,14.58,51.04,20.36,102.48,35.14,12.31,8.98,3.7,81.22,89.67,27.32,13.26,1.54,37.84,11.29,8.98,24.08,4.43,8.54,15.51,36.26,9.33,6.56,43.38,13.04,10.9,54.68,160,164.11,34.25,15.87,14.65,18.72,10.57,13.25,21.08,62.55,6.08,22.72,17.25,14.1,11.91,16.5,17.36,4.93,31.46,75.14,32.84,55.65,21.39,18.76,53.36,52.27,150,11.51,50.35,4.77,16.2,13.83,43.24,94.4,37.89,13.18,35.9,70.65,14.16,11.8273,12.67,22.93,11.11,13.55,9.55,14.24,26.44,13.49,69.92,9.88,157.12,14.93,6.53,13.5,6.83,28.27,13.07,48.33,59,9.67,2.04,28.2,5.82,31.87,65.9,33.14,36.14,30.08,8.49,64.92,4.76,141.98,8.63,10.11,17.27,19.65,30.24,26.53,40.67,26.87,26.25,39.18,1.54,33.93,16.59,7.33,15.36,13.91,1.1,17.12,4.47,15.67,18.68,1.79,15.54,81.13,21.34,27.59,7.46,10.77,503.01,19.74,18.46,15.26,47.57,30.24,6.55,65.34,4.13,13.2,21.55,21.03,29.36,27.27,47.72,10,26.17,7.45,7.62,3.48,18.15,8.2,20.66,97.23,59.46,13.37,7.8,76.6,19.31,9.21,24.18,77.53,0.9,10.8,153.17,23.32,1.18,25.83,42.06,1.48,11.49,24.78,18.24,21.24,6.59,44.54,33.28,64.61,227.17,30,48.14,29.72,45.5,79.45,26.83,4.96,81.03,30.76,72.15,34.48,129.3,65.48,33.48,67.43,34.31,15.05,60.38,32.59,26.84,1.7,32.48,2.31,113.96,1.13,7.3,31.13,12.27,44.48,163.75,4.55,4.93,49.64,7,0.42,4.66,62.44,41.56,21.6,8.18,26.89,33.78,3.79,47.23,27.86,0.5199,45.15,117.21,9.51,74.52,1.8,67.24,22.28,22.735,28.48,13.84,19.61,26.62,25.78,18.95,33.53,21.54,51.22,13.79,34.9,81.22,6.86,15.59,96.42,18.49,31.21,24.57,21.23,22.93,16.32,10.63,11.18,188.35,16.08,18.39,18.17,43.96,62.54,8.82,1.87,33.38,14.9,10.47,16.69,3.2,8.89,3.95,50.02,153.42,7.65,35.53,0.54,18.36,5.24,262.52,4.07,74.98,12.53,13.66,86.82,129.8,24.78,9.8,7.03,21.68,8.1,17.1,7.5,38.4,118.45,6.47,26.92,17.2,7.6,35.32,1.72,25.83,9.11,26.27,13,12.18,12.29,44.75,26.22,7.44,44.5,43.59,24.19,1.35,13.57,65.69,13.29,37.11,6.3,16.42,15.98,25.62,25.8,25.93,101.16,45.95,17.1,35.85,108.8,11.53,13.84,14.7,29.07,68.14,19.1,63.34,13.98,44.89,8.42,11.42,6.6,52.51,7.77,10.61,14.67,11.98,0.4476,15.65,14.54,21.3,122.3,22.92,13.03,7.38,5.41,22.4,117,14.97,17.58,9.98,22.08,10.13,23.79,64.75,9.23,16.0197,8.25,16.56,10.61,15.97,11.8,14.82,39.57,77.69,50.96,41.52,30.2,14.56,15.27,13.66,19.89,0.3292,2.93,26.22,72.39,12.44,33.06,26.14,26.96,5.15,29.29,5.31,8.85,29.28,15.21,21,13.64,130.69,15.04,26.49,18.08,8.27,24.92,51.81,15.55,12.13,24.95,26.79,45.76,82.08,29.35,5.82,36.58,27.79,62.53,26.16,5.64,16.33,13.71,37.44,11.68,90.18,90.17,14.71,14.2,18.3425,20.81,91.59,10.86,21.05,49.61,40.1,77.37,33.53,69.02,8.1,4.47,24.11,8.38,8.9401,39.17,24.28,40.52,117.73,1.99,50.87,47.66,9.32,11.02,6.59,21.86,13.2401,131.19,23.07,140.8,14.33,147.9,74.43,33.26,240.67,44.33,74.18,39.42,9.31,78.67,8.66,12.41,10.29,38.6,18.01,11.96,11.94,49.05,4.85,17.69,44.19,53.67,47.27,37.6,9.86,14.96,79.92,4.09,24.39,26.39,33.6,97.02,98.48,78.25,79.19,31.84,13.91,7.42,8.95,26.08,16.16,6.51,122.79,17.785,186.88,7.77,101.07,8.48,20.3,53.09,70.02,24.55,51.81,4.93,85.78,23.6,32.98,17.93,29.19,12.71,8.9,43.78,7.35,14.59,12.75,3.05,15.88,7.48,15.32,5.4967,24.88,41.18,5.37,25.09,26.12,22.12,18.47,8.23,17.21,14.3799,197.03,15.16,19.27,4.71,73.13,10.714,4.618,15.6,83.55,99.95,944.3,31.95,186.34,22,66.36,20.31,26.93,168.13,72.451,57.37,6.05,15.05,8.8074,15.5,11.38,60.84,17.25,12.82,65.95,10.1,51.415,109.8275,26.9,19.825,15.4293,36.63,16.29,14.34,15.065,8.44,14.4,16.77,14.84,56.78,13.945,25.58,51.74,10.52,9.3,13.52,68.56,25.73,10.75,73.77,11.61,5.34,116.61,116.86,381.835,6.41,37.74,1.75,1.79,140.44,9.2,25.3,4.81,5.585,58.08,22.97,15.12,16,14.62,16.65,15.09,15.83,30.81,14.73,2.89,8.24,10.74,55.84,11.03,16.31,30.76,16.54,4.64,6.11,5.3,14.99,9.62,14.83,123.98,35.9,29.06,410.2,16.19,56.83,12.49,41.27,72.28,18.24,23.03,21.98,71.37,25.02,13.86,18.36,10.773,13.66,37.26,4.16,54.94,14.14,16.5,3.22,20.7,10.8,1.1,12.84,1.42,14.92,4.03,15.25,13.8336,1.93,14.55,36.2,47.92,10.22,2.8,218.58,5.68,75.56,72.75,2.44,14.85,15.12,92.25,15.14,50.15,16.29,14.64,4.39,16.36,15.1,15.25,5.36,13.9,16.59,14.05,14.05,9.32,13.9,52.42,85.6,25.92,6.15,12.62,75.39,24.26,25.27,13.7,18.84,21.41,5.65,44.37,15.54,50.55,14.99,16.52,41.85,10.55,18.78,56.87,1738.1,80.51,60.58,48.96,59.1,1.49,19.69,16.42,14.9,14.375,15.15,14.76,15.4,12.04,63.98,90.63,46.89,10.77,13.28,50.63,1.76,33.79,16.16,27.71,8.73,31.48,62.99,33.13,19.92,7.89,33.58,35.78,46.53,38.53,23.63,23.77,84.4,19.83,28.76,38.37,7.02,3.73,9.61,15.47,43.98,10.45,19.1,5.11,46.7,23.07,61.76,75.6,3.99,11.95,27.51,99.32,37.51,10.17,13.98,26.18,4.71,20.98,53.83,31.34,25.83,53.61,18.87,6.89,7.39,7.81,62.98,18.95,10.9,9.62,15.03,16.79,27.26,20.91,5.66,16.08,27.44,26.23,44.62,22.25,20.49,103.66,2.53,4.44,43.92,15.08,35.31,43.14,10.1,9.07,11.97,20.15,20.48,83.17,1.94,20.18,32.86,18,115.36,11.46,26.27,14.23,44.33,9.84,19.58,10.13,16.95,84.53,4.44,5.49,40.79,26.17,2.54,16.71,69.64,54.29,23.04,46.29,17.98,49.87,45.06,101.78,27.09,16.72,13.53,7.91,13.27,16.12,12.66,7.08,88.33,13.53,14.1,34.57,60.16,77.17,36.77,60.23,37.78,42.56,17.22,110.43,39.25,1.91,5.22,58.17,4.8,26.87,3.34,52.44,26.2,57.87,16.53,75.15,245.43,103.59,27.06,3.17,11.88,81.13,72.92,14.26,37.51,37.51,8.91,97.21,0.9614,24.1,113.44,164.2,11.21,12.02,6.24,8.67,66.54,19.82,42.46,0.4498,54.44,67.66,7.57,51.06,3.26,24.43,34.63,58.16,6.49,39.18,76.24,9.84,19.3,11.55,45.79,180.42,79.59,3.6,16.22,31.99,59.67,9.7,13.17,5.41,98.19,20.12,59.5,12.99,7.91,28.41,40.99,51.01,78.23,11.47,28.75,26.63,54.23,96.18,7.23,67.31,21.02,16.98,60.5,19.01,7.54,20.58,19.72,115.86,65.41,118.74,28.8,174.25,51.32,18.66,13.09,46.21,76.22,18.38,49.8,13.18,7.3,3.96,136.61,12.21,14.39,21.83,62.39,15.55,26.25,29.47,3.81,14.03,17.05,160.73,4.59,80.74,16.5811,1.2,29.52,35.43,8.3,10.85,16.2,11.87,13.3,71.67,27.33,60.52,9.84,16.14,12.8,1.8,2.42,33.76,47.19,12.62,36.74,13.25,16.07,28.24,15.96,9.15,4.05,8.62,0.5429,34.32,49.07,11.84,293.03,2.22,89.6,25.47,58.42,30.13,143.23,19.4,7.14,36.5,20.46,37.11,30.65,79.96,34.92,101.48,21.29,33.24,68.79,4.22,11.36,38.63,9.02,160.35,28.77,19.05,34.19,72.25,10.46,31.28,96.12,40.86,50.86,1.35,48.37,36.98,20.18,117.48,13.74,201.65,34.85,19.46,46.23,23.88,11.91,110,9.77,16.48,12.72,39.84,5.41,29.03,17.25,103.26,15.36,36.63,38.09,71.67,42.97,78.07,16.86,16.47,6.05,26.51,16.91,25.04,60.29,1.26,154.1,27.67,69.33,29.82,27.84,4.73,10.71,26.4,114.37,34.76,14.74,74.67,10.56,6.56,2.28,88.4,29.03,71.35,31.41,115.33,81.56,9.2,49,15.67,40.09,7.22,5.51,14.13,103.38,19.91,13.95,11.92,70.67,59.99,5.29,44.99,24.99,28.64,25.61,17.31,266.23,26.72,25.77,28.55,5.55,100.29,27.69,10.17,10.83,62.27,55.45,19.42,46.89,20.23,21.13,172.71,16.91,12.11,39.7,13.35,6.37,67.69,29.95,85.31,67.72,16.7,20.74,97.86,9.72,62.52,77.31,9.67,9.29,8.64,34.09,10.59,57.33,39.4,104.5,44.5,60.64,154.15,54.89,105.49,3.66,6.18,29.43,5.93,48.71,22,12.28,169.55,60.98,19.52,39.54,24.25,42.45,18.07,18.85,43.41,2.95,6.6,114.22,28.92,48.51,5.29,7.47,26.23,61.28,78.75,9.14,53.37,10.18,89.37,7.71,6.44,35.18,18.74,32.1,58.58,25.94,25.57,25.65,7.21,8.59,75.01,19.97,39.37,20.79,20.84,43.69,30.96,160.22,38.66,46.7,8.36,9.07,23.13,14.75,7.49,33.71,28.42,26.87,38.33,43.94,19.64,138,56.34,8.25,46.6,2.01,10.5,46.07,117.09,140.68,35.69,89.39,16.82,104.97,74.11,5.09,42.54,28.31,37.91,21.2,3.26,41.98,102.32,19.23,55.7,25.7,25.93,81.26,61.1,108.46,108.45,85.57,18.9,74.53,14,25.85,9.88,35.01,63.16,4.78,15.06,14.49,22.2,2.11,12.35,13.79,53.81,44.61,14.18,9.42,116.83,137,14.42,64,97.18,3.42,13.6,14.05,48.17,25.48,13.49,30.72,11.2,15.78,68.42,51.63,32.15,4.22,51.95,27.55,40.18,77.04,4.18,36.79,140.05,108.68,38.41,59.96,105.5,72.79,25.02,20.17,13.44,62.32,50.91,95.22,26.076,49.14,6.76,3.02,68.92,21.8,42.62,57.25,179.69,11.49,12.14,11.1,13.73,17.08,45.06,13.35,62.62,23.28,9.73,52.25,27.11,71.09,14.52,21.59,30.65,40.03,67.61,11.41,15.19,10.64,34.29,56.11,56.44,30.77,39.05,52.14,136.16,14.58,77.42,2.14,813.78,33.86,59.02,14.77,19.64,17.91,20.05,30.37,68.38,17.44,1.21,120.33,42.7,5.03,33.58,90.67,25.13,29.96,7.31,53.42,10.12,46.3,540.23,4.38,21.64,83.73,5.45,119.4,26.99,12.77,14.28,29.64,11.94,48.14,5.03,5.87,6.16,12.26,8.63,5.89,39.26,2.1,1.36,1.26,2.5,0.62,4.12,2.93,2.18,2.12,5.25,19.03,0.3024,34.93,2.77,16.49,16.56,16.87,12.31,1.72,124.2,0.8722,3.26,0.4964,13.2,20.21,3.47,60.5,20.56,30.64,36.55,11.96,10.97,9.59,48.12,12.96,12.91,16.4,16.5,3,2.515,15.38,15.29,0.58,0.2101,9.79,0.135,13.0101,13.5,13.93,13.47,13.81,13.83,13.78,14.42,14.66,7.91,0.6724,0.3199,2.78,5.77,0.8812,0.71,25.94,1.56,11.79,12.23,6.51,0.9442,4.15,0.4398,2.62,3.67,1.76,0.6489,3.1,30.31,3.48,1.73,8.7,2.19,0.1244,0.1183,4.7,14.19,31.89,7.79,0.4569,1.94,0.5899,12.98,0.8,0.91,2.5,17.57,4.05,0.2399,14,13.18,2.36,0.289,1.03,10.97,63.6,16.08,11.12,16.46,8.52,14.47,5.3801,1.95,3.42,4.39,0.36,4.57,2.75,0.5131,6.37,0.134,16.28,15.34,9.93,15.37,14.36,11.3,33.75,15.68,7.03,0.4907,93.1,0.2569,2.17,15.75,2.84,0.8011,0.788,31.33,0.2799,9.13,8.6001,20.5,44.44,0.7498,2828.02,0.475,1.59,11.705,1.81,0.4725,0.5109,0.82,18.2368,0.331,66.17,0.9543,0.82,0.7501,0.611,1.01,1.3,5.05,0.87,8.01,12.5901,13.04,10.45; 42.73,24.76,150,5.77,20.71,74.78,55.37,7.12,38,18.12,47.27,7.73,116.36,1.53,42.68,24.53,48.12,4.92,26.23,13.99,13.93,36.84,108.6,15.1,26.12,25.67,69.24,15.64,10.78,6.72,216.02,25.4,33.36,5.02,47.32,37.32,12.5,25.62,14.16,38.27,23.77,43.8,42.5,26.17,87.29,15.15,3.46,29.67,8.65,65.57,67.54,40,123.54,24.18,29.23,17.73,95.34,106.56,13,46.95,24.27,4.78,6.91,25.59,105.72,16.08,48.25,13.21,55.8,17.1,42.87,4.1,60.34,17.9,3.95,113.03,157.65,4.66,2.29,74.49,77.63,5.156,15.01,4.73,12.18,13.19,18.94,13.56,38.95,26.5,12.92,76.37,11.9,39.79,74.94,39.73,57.63,15.79,64.34,54.04,252.05,28.39,780.81,55.62,33.36,134.42,14.14,27.79,12.13,143.53,45.63,15.68,17.6099,23.63,33.73,13.27,31.25,47.78,22,46.55,62.6,219.45,9.45,62.11,7.53,15.51,167.15,9.37,35.84,15.15,20.25,57.05,16.73,14.03,15.15,21.68,13.5,12.95,5.77,361.44,20,45.47,13.1493,26.72,7.98,43.36,15.12,39.74,17.21,24.17,60.83,16.14,17.06,10.64,6.51,33.12,15.01,355.6,72.84,14.76,25.95,63.07,36.79,74.35,63.84,50.21,71.28,16.45,22.54,49.04,16.21,11.7,68.62,14.79,11.67,31.86,8.52,15.2,69.08,13.17,24.42,15.695,61.23,26.99,35.29,16.31,7.2,26.24,15.54,14.676,15.41,15.91,22.05,32.12,12.48,4.69,24.92,23.65,12.62,126.5,18.46,9.84,33.75,12.42,16.2,26.35,0.73,1,131.38,22.35,19.1,15.76,7.32,44,33.77,48.56,12.2,17,46.43,77,7.5,28.09,23.02,5,71.7,125.38,8.03,3.7,13.12,28.81,37.62,2.38,11.34,47.1,1.19,54.28,44.8,39.22,10.22,15.06,38.667136,90.46,11.72,55.05,51.05,3.75,6.09,17.3,21.41,26.98,8.02,5.86,6.96,70.73,28.08,59.95,18.42,7.47,14.85,10.75,116.53,27.85,21.76,26.66,61.4,25.71,1.36,0.7901,9.52,48.37,97.37,126.08,47,4.17,55.61,13.77,11.52,34.34,11.3,128.06,7.5,36.89,2.53,2.45,1.92,13.22,15.09,11.57,55.3,32.07,54.38,72.65,25.5,118.7,57.38,1.96,3.02,47.95,26.1,40.55,9.85,59.99,132.88,78.76,42.69,16.91,457.01,114.39,65.76,9.91,77.3,42.09,4.91,32.44,56.43,59.69,35.57,19.17,22.11,28.78506072,38.22,14.41,6.21,69.36,24.4,38.01,89.43,155.41,43.73,76.84,14.55,137.21,53.86,66.5,11.06,22.88,16.63,61.17,11.56,15.03,84.79,86.63,55.48,15.4,29.57,101.26,0.69,81.42,75.89,11.31,30.84,16.28,11.87,13.14,33.72,38.18,19.32,101.5,27.42,36.97,18.89,23.56,32.16,28.46,25.64,12.09,17.48,5.45,72.28,3.18,41.07,12,14.5,16.6,23.78,26.7,4.3,32.7,25.86,16.92,42.77,64.28,11.01,12.18,1.06,9.31,17.79,60.64,26.7,84.15,18.77,26.31,33.03,63.4,109.41,9.44,18.65,23.97,55.3,42.46,83.69,21.05,75.72,3.19,13.49,29.15,98.12,5.61,6.39,82.82,101.71,13.97,28.88,42.06,26.5,13.77,45.51,94.67,62.82,28.92,13.57,5.54,23.2,118.34,11.24,10.36,4.1,24.18,19.05,66.13,51.54,16.37,34.38,95.56,121.46,26.03,16.23,5.81,23.39,9.34,63.81,59.11,7.04,17.84,8.74,120.32,3.48,22.69,2.81,91.33,16.38,26.21,25.87,26.36,9.7,81.08,19.47,75.59,32.71,73.05,6.74,24.96,70.88,18.67,30.92,25.41,25.77,25.54,45.03,18.18,38.88,9.6,6.83,18.73,117.13,2.67,1.63,74.96,7.6,33.59,13.41,12.48,14.62,42.32,40.29,45.55,7.97,21.31,21.03,17.52,15.05,12.98,13.15,122.65,13.51,23.8,43.08,4.73,63.4,1.11,9.25,29.82,71.94,6.25,94.61,25.47,25.5,25.6,8.26,73.64,25.3472,9.25,27.04,10.43,47.6,10.72,3.73,74.8,12.43,52.91,25.95,40.24,65.54,57.95,31.03,6.08,6.01,80.5,12.78,12.84,22.79,78.1,26.27,5.45,71.78,28.39,75.98,69.88,70.64,29.83,9.78,5.24,21.61,57.63,14.74,39.23,2.22,67.5,229.93,10.46,16.22,12.97,14.48,32.22,9.72,12.03,62.07,21.32,35.18,77.07,14.77,10.36,20.38,10.38,34.93,7.27,5.86,13.4,14.27,49.49,19.9,105.59,35.46,12.1,8.76,3.71,76.56,93.4,26.7508,13.35,1.5616,37.34,11.09,8.77,23.13,3.69,8.75,15.99,33.96,8.88,7.2,40.96,12.63,11.03,50.51,151.66,159.37,33.97,14.26,14.12,15.95,10.97,12.71,20.91,60.72,6.19,24.8,19.69,13.76,11.67,15.79,16.34,5.08,31.41,71.77,31.24,59.04,21.3,19.4,51.33,45.98,150.93,11.97,46.99,3.62,15.94,12.44,42.09,92.59,34.16,12.75,33.76,68.09,13.78,11.29,11.59,22.82,11.32,12.13,8.73,14.23,24.79,13.18,67.95,7.87,156.56,14.55,6.32,12.9,6.42,28.01,11.1,44.19,58.5,10.19,2.02,27.25,5.52,30.68,65.85,31.31,36.24,26.55,8.14,63.6,4.5399,145.31,8.23,10.05,16.99,18.57,30.09,26.28,35.4,26.45,26.04,34.48,2,32.72,15.9,6.69,16.28,14.04,1.23,16.22,4.14,15.55,18.02,2,15.0649,81.11,19.31,28.13,7.19,10.3699,486,19.44,17.82,14.61,45.82,29.75,6.4,64.72,4.8,13.02,19.25,18.88,31.18,29.1,44.92,8.97,25.04,7.69,7.13,3.66,17.73,8.91,19.92,97.11,61.24,13.39,7.44,74.37,17.86,8.96,23.74,77.81,0.94,10.21,158.07,23.8101,1.13,25.82,42.07,1.68,11.36,21.6,16.51,20.69,6.53,43.16,31.11,58.84,227.76,31.76,48.71,26.58,40.05,73.47,26.96,3.89,77.69,31.86,72.09,34.24,134.19,64.38,33.98,61.59,32.8,14.86,58.14,30.69,28,1.68,31.8,1.57,103.61,1.45,7.19,30.83,11.66,44.5,153.62,4.15,4.72,48.09,6.87,1.03,4.28,61.99,40.15,21.93,7.15,27.61,33.08,3.2,42.95,28,0.47,45.62,114.19,9.93,74.87000442,1.58,58.65,22.12,22.22,28.77,11.56,19.07,26.41,23.68,17.34,31,19.52,47.73,13.99,40.21,76.11,6.56,15.89,93.16,18.9,30.46,24.89,19.17,23.46,16.61,10.23,8.06,165.81,13.99,18.01,18.9,44.81,60.52,8.49,1.86,27.86,14.41,10.21,18.12,3.83,8.58,3.8,50.25,148.84,6.99,31.25,0.71,18.33,4.1,262.14,4.42,73.91,12.1,13.20970759,82.33,126.16,23.09,9.72,6.89,21.29,8.03,15.6,7.22,38.87,120.44,6.3,25.71,17.36,7.56,33.25,1.57,24.45,8.72,26.49,12.04,11.84,12.28,45.4,26.41,7.56,31.41,42.6,23.86,1.34,13.53,65.92,13.47,38.08,5.74,15.31,15.41,26.02,25.712,25.8,98.08,44.54,18.55,34.85,104.53,10.66,12.91,13.9,29.06,71.17,17.01,57.81,13.55,41.32,7.8,11.16,6.27,49.81,7.64,10.32,13.94,10.81,0.7498,15.52,14.21,20.62,109.04,23.01,11.65,7.36,5.17,21.85,114.23,14.3,16.77,9.77,16.49,10.03,23.69,61.77,9.17,15.36,7.87,15.77,10.07,15.29,11.25,14.1294,45.23,77.31,47.89,39.54,28.44,12.82,14.53,13,17.54,0.3641,2.55,26.31,63.92,11.88,31.72,25.69,26.505,4.6,27.83,5.09,8.76,29.12,14.53,21.01,12.98,130.06,12.4829,22.18,17.03,8.19,23.47,51,12.27,12.14,23.41,26.5,45.83,82.63,25.37,4.86,34.89,26.94,64.14,27.98,5.83,14.68,14.03,35.15,11.42,90.18,90.19,14.23,14.2,17.96,20.06,84.41,9.49,17.12,49.09,39.65,78.86,33.65,67.42,8.02,3.76,18.32,8.16,8.72,37.4,23.33,38.52,114.6,1.95,49.43,43.7,9.16,10.56,6.65,20.49,12.67,125.76,23.95,138.26,11.58,135.78,75.96,31.26,244.94,43.04,69.13,37.05,9,75.4,9.47,11.17,10.59,37.12,18.04,11.86,11.52,48.01,5.05,17.57,41.22,53.75,45.64,37.03,9.42,14.2,81.25,3.82,23.04,25.5,31.21,95.85,101.87,76.99,77.16,31.35,14.14,7.16,9.19,26.91,16.35,6.66,130.12,17.82,167.3,7.4,94.74,8.25,19.75,52.51,68.69,22.35,50.39,4.59,81.34,21.87,34.95,17.28,28.62,12.61,7.18,43.51,7.12,14.2,12.4392,3.04,15.53,7.53,14.95,5.39,23.75,40.57,5.33,21.94,26.2,20.78,18.4,8.35,16.08,14.67,194.42,15.58,18.03,4.52,69.18,9.39,4.635,15.05,83.71,98.34,958,29.91,184.75,19.84,63.95,20.38,25.15,168.32,69.68,53.16,5.85,15.33,9.14,15.6346,10.9,58.4,17.21,11.96,64.58,10.38,46.74,99.94,24.84,24.63,15.7076,35.49,15.5,14.35,14.32,8.48,14.67,16.79,13.64,53.88,12.3,25.89,47.77,8.74,9.03,12.7,71.11,25.61,10.45,73.73,11.24,4.59,113.23,113.21,367.93,6.47,33.67,1.78,1.76,127.6,8.66,24.38,4.51,5.89,57.46,21.28,14.94,16.22,14.53,16.88,15.18,16.01,29.14,14.41,2.56,7.14,8.3,52.61,11.04,15.56,28.02,14.96,4.48,5.8,5.07,13.27,9.06,14.79,121.15,33.21,29.22,405.47,16.15,54.74,12.36,38.63,73.47,13.33,22.42,19.85,69.31,23.96,13.83,17.87,10.84,13.66,36.3,3.9,57.99,14.29,16.38,2.68,19.74,11.1,0.7972,12.62,1.32,14.84,4.16,14.99,13.65,1.67,14.2,36.65,47.36,10.19,2.39,217.98,5.18,67.47,67.38,2.49,14.9101,14.98,87.78,15.16,48.04,16.4,14.66,4.01,16.26,14.9,15.22,4.52,13.1,15.63,13.92,11.83,8.35,13.37,47.97,88.18,24.73,5.89,11.49,66.05,23.18,24.66,13.64,18.25,21.08,5.83,46.78,15.31,47.44,14.8,16.29,39.8,10.56,18.1769,54.18,1652,74.88,58.43,48.54,57.45,3.29,18.28,16.68,14.73,14.4,15.09,14.6,15.3,12.17,64.14,88.71,45.68,9.54,13.13,49.41,1.99,33.03,14.84,26.89,8.23,30.99,60.32,32.83,18.97,7.94,32.53,31.42,40.29,37.61,21.53,23.34,84.26,19.57,30.51,36.94,6.75,3.93,9.55,13.82,42.61,10.4,19.09,5.1,45.36,22.22,60.82,76.48,3.83,10.09,23.76,100.8,35.59,9.68,13.8299,22.69,5.01,26.18,49.77,28.86,28.22,57.08,18.14,7.35,6.69,7.52,59.24,18.25,10.96,9.71,14.5,17.13,27.61,20.4,4.24,15.48,25.48,26.1,46.36,22.88,18.66,106.1,2.5,2.82,43.08,15.28,33.19,42.45,9.82,8.87,12.2,19.49,19.48,82.41,1.61,20.31,33.56,16.83,112.54,10.97,26.26,12.72,36.72,9.54,18.19,9.94,16.9,85.1,4.36,5.52,40.98,26.65,2.41,15.37,66.33,54.48,22.33,44.32,17.24,47.45,42.5,102.18,26.27,16.62,13.59,7.88,13.21,14.92,12.5,7.83,86.18,13.63,13.8001,33.27,59.22,74.44,36.93,59.94,36.56,41.61,15.84,107.49,38.91,1.86,4.97,60.77,4.7,26.82,3.16,49.16,26.76,51.04,11.7,76.43,264.69,99.79,25.91,3.47,11.67,78.25,68.79,13.99,37.08,37.07,8.25,84.16,0.8522,23.41,113.6,167.03,10.9224,12.15,6.25,9.13,65.31,17.91,40.79,0.4101,50.8,68.44,8.01,51.51,2.98,22.82,31.82,57.82,6.13,38.62,77.02,9.3,17.3,12.02,42.86,184.08,78.69,2.9075,13.89,33.35,52.31,9.13,13.23,5.42,95.63,20.99,65.95,13.55,7.5501,35.04,38.6,51.56,71.93,9.68,28.7,26.5,51.29,86.88,7.5,66.85,22.13,15.83,57.81,17.05,7.46,18.45,19.37,113.51,61.99,114.01,27.66,177.15,50.49,18.28,12.85,42.26,72.17,18.31,48.84,12.4,7.49,4.39,131.91,11.75,13.13,20.8,59.97,10.37,25.38,30.05,3.49,13.34,17.08,155.45,4.6,77.25,16.2999,0.934,29.3,33.23,7.84,10.45,14.55,12,12.59,70.57,27.04,58.35,10,14.88,10.28,2.09,2.53,31,47.8,12.04,33.69,13.39,15.4,25.44,15.51,8.74,3.94,8.82,0.6112,33.74,46.08,12.7,296.18,2.76,104.1,24.25,58.36,26.73,131.51,19.1,6.2,34,20.25,35.46,31.44,74.09,33.9,104.1,18.92,29.05,66.83,4.22,10.1999,37.26,8.34,163.29,25.84,18.34,33.91,66.17,11.16,30.41,79.67,39.24,50.16,1.22,47.28,36.96,19.69,117.73,13.52,204.04,31.79,15.78,47.26,21.29,11.96,105.37,9.1,14.79,11.24,37.97,6.23,29.64,15.22,114.7,15.85,34.11,36.47,71.76,41.35,75.83,16.03,15.7,5.34,24.84,16.77,25.04,60.03,1.32,161.9,26.57,71.37,33.6,26.29,4.48,9.71,26.4199,113.65,34.67,11.86,69.19,10.2832,5.64,2.15,81.34,27.63,66.56,29.42,111.82,80.81,8.96,50.18,14.9101,39.55,7.04,5.41,14.51,99.73,18.54,13.26,9.71,71.51,52.38,5.77,43.79,25.21,27.31,25.3892,16.5384,247,25.9599,25.5,28.13,7.35,93.9,27.53,10.79,10.36,58.72,51.73,19.11,45,18.5,23.04,161.97,15.93,12.89,36.18,14,6.34,75.6,28.41,86.79,64.86,15.994,18.83,93.48,9.51,66.37,74.28,9.84,9.24,8.43,33.47,9.95,56.57,40.87,101.38,44.3,58.55,148.92,53.07,105.69,3.96,5.77,26.71,5.95,49.04,20.45,10.88,162.1,58.01,18.81,35.46,20.01,41.42,17.28,17.29,40.33,2.82,6.3,112.64,26.1,44.12,5.27,8.25,22.99,68.65,75.61,9.07,53.28,10.89,87.69,7.53,5.49,28.77,16.93,31.76,56.22,26.5,25.85,25.7401,6.03,8.52,73.39,17.69,37.86,20.07,20.07,38.7,29.07,148.17,37.66,43.78,7.71,8.06,21.2,15.01,7.65,36.81,27.64,25.07,37.7,41.81,20.87,133.29,56.32,7.4,45.66,1.77,9.84,44.71,111.33,129.74,33.85,83.84,12.96,101.79,60.87,4.87,41.94,27.98,38.29,20.16,3.28,39.63,101.46,18.22,55.85,25.52,26.35,77.89,58.71,106.91,106.8875,82.26,18.76,74.11,14.07,22.74,9.85,32.93,60.84,3.9,14.49,14.35,21.27,1.82,12.3,13.61,54.78,46.51,13.6,9.1,116.12,134.97,14.18,56.62,96.69,3.28,13.6,14.19,47.52,24.93,12.36,27.7,10.77,15.9,66.72,50.39,30.56,4.05,51.47,27.54,37.84,79.17,4.41,34.21,134.96,105.34,35.36,55.08,92.92,69.47039835,24.05,19.76,12.71,60.18,48.91,90.11,26.55,49.2,5.82,2.9,68.28,20.82,40.56,56.32,171.54,11.4,11.94,10.99,11.91,13.69,45.01,10.93,61.78,19.05,9.11,46.67,23.65,66.85,14.14,21.6,30.4,35.89,63.93,10.56,14.96,9.02,29.15,52.18,56.31,29.22,38.54,54.51,133.93,13.7,71.27,2.27,825.61,32.84,56.14,13.07,19.25,17.79,17.89,31.23,69.13,14.81,1.75,115.34,41.85,4.97,33.71,89.67,24.5,27.18,5.98,53.57,9.24,43.24,526.42,3.48,20.92,81.57,5.13,117.44,23.66,12.76,14.3,32.23,12.09,46.25,4.9,5.77,6.24,11.94,8.45,6,38.21,2.05,1.18,1.19,2.17,0.73,3.23,3.05,1.46,2.06,4.68,18.27,0.3199,31.48,2.59,16.44,16.85,16.9,11.91,1.74,122.34,0.82,3.33,0.55,13.02,19.46,2.6,55.43,18.83,28.86,34.75,11.76,10.8,9.5,48.62,12.54,12.8,16.6,16.22,2.9,2.33,16.09,14.56,0.5024,0.24,9.84,0.15,13.19,13.82,14.1466,13.15,13.88,14.07,13.68,14.38,14.85,7.69,0.7663,0.324,1.89,5.35,0.9,0.77,23.7,1.2863,11.39,11.24,6.43,0.97,4.2127,0.3302,2.21,3.5,1.54,0.6,2.65,27.28,2.75,1.78,9.52,2.35,0.11,0.1212,5.68,13.86,31.34,7.6,0.42,1.56,0.6299,12.7,1.05,0.94,2.37,17.12,3.66,0.21,12.61,13.18,1.52,0.3221,0.6634,10.9,63.43,16.37,10.54,16.52,7.5,14.62,5.24,2,3.53,4.53,0.3581,4.75,2.59,0.7299,6.13,0.165,16.42,15.63,9.37,15.04,14.15,10.95,31.46,17.57,6.71,0.525,89.54,0.225,2.27,15.71,2.89,0.84,0.79,30.57,0.2603,8.49,7.26,19.76,44.32,0.6701,2930.11,0.51,2.29,12.6,2.07,0.49,0.5058,1.06,17.75,0.3579,63.51,0.8301,0.75,0.78,0.4875,0.7495,2.52,4.35,0.82,7.66,12.36,12.96,10.735];
I compared matrix x with the 1st and 20th row of c and they are equal:
isequal(x,c([1 20],:))
ans =
1
importdata seems to be working as expected.

MATLAB: How can I efficiently read in these data files?

I have 100 data files in a folder called "Experiment1", and I need to take all of the data from them and put them into a single matrix. Each data file contains 15 columns and 40 rows of data.
The order in which the files are in the folder is arbitrary. It doesn't matter in what order they get put into the combined matrix.
I've written some code using dlmread that will do the job:
for i = 1:100
%% Read in the relevant file.
filename = ['File_' int2str(i) '.dat']
Data = dlmread(fullfile(pwd, 'Experiment1',filename));
%% Put all the data I need in a separate matrix
NeededData(1+((i-1)*40):i+((i-1)*40)-i+40,1:15) = Data(:,1:15);
end
However, there are two things I don't like about my code.
The files have random names at present, and I'd need to manually change all their names to "File_1.dat", "File_2.dat", etc.
The code is cumbersome and hard to read.
How could I do things better?
Since you've fixed the problem of defining the name of the files to be read with dir, you can improve the way you add the read data (Data) to the output matrix (NeededData).
You can sumultaneously read the input files and add the data to the output matrix by inserting the call to dlmread directly in the assignment statement:
files=dir('*.dat');
n_files=length(files)
% Initialize the output matrix as empty
NeededData_0=[]
for i=1:n_files
% Simultaneously read input file and assign data to the output matrinx
NeededData_0=[NeededData_0;dlmread(files(i).name)]
end
In case you prefer working with the inides (as in your origina approach), since you know in advance that all the files have the same (40) number of rows) you can simplify the notation as follows:
files=dir('*.dat');
n_files=length(files)
% Define the number of rows in each inout file
n_rows=40;
% Define the number of colums in each inout file
n_col=15;
NeededData_2=nan(n_rows*n_files,n_col)
% Define the sequence of rows
r_list=1:n_rows:n_rows*n_files
for i=1:3
Data=dlmread(files(i).name)
NeededData_2(r_list(i):r_list(i)+n_rows-1,:)=Data
end
Hope this helps.
Using the suggestion to use dir present in the answers I have made the following code, which is clearly an improvement on my earlier effort. I would welcome further improvements, or suggestions for alternative approaches.
files = dir('*.dat');
for i = 1:length({files.name})
%% Read in the relevant file.
Data = dlmread(files(i).name);
%% Put all the data I need in a separate matrix
NeededData(1+((i-1)*40):i+((i-1)*40)-i+40,1:15) = Data(:,1:15);
end

Simplest way to read space delimited text file matlab

Ok, so I'm struggling with the most mundane of things I have a space delimited text file with a header in the first row and a row per observation and I'd like to open that file in matlab. If I do this in R I have no problem at all, it'll create the most basic matrix and voila!
But MATLAB seems to be annoying with this...
Example of the text file:
"picFile" "subjCode" "gender"
"train_1" 504 "m"
etc.
Can I get something like a matrix at all? I would then like to have MATLAB pull out some data by doing data(1,2) for example.
What would be the simplest way to do this?
It seems like having to write a loop using f-type functions is just a waste of time...
If you have a sufficiently new version of Matlab (R2013b+, I believe), you can use readtable, which is very much like how R does it:
T = readtable('data.txt','Delimiter',' ')
There are many functions for manipulating tables and converting back and forth between them and other data types such as cell arrays.
There are some other options in the data import and export section of the Statistics toolbox that should work in older versions of Matlab:
tblread: output in terms of separate variables for strings and numbers
caseread: output in terms of a char array
tdfread: output in terms of a struct
Alternatively, textscan should be able to accomplish what you need and probably will be the fastest:
fid = fopen('data.txt');
header = textscan(fid,'%s',3); % Optionally save header names
C = textscan(fid,'%s%d%s','HeaderLines',1); % Read data skipping header
fclose(fid); % Don't forget to close file
C{:}
Found a way to solve my problem.
Because I don't have the latest version of MATLAB and cannot use readable which would be the preferred option I ended up doing using textread and specifying the format of each column.
Tedious but maybe the "simplest" way I could find:
[picFile subCode gender]=textread('data.txt', '%s %f %s', 'headerlines',1);
T=[picFile(:) subCode(:) gender(:)]
The textscan solution by #horchler seems pretty similar. Thanks!

Reading complicated format CSV fileinto Matlab

My raw CSV file looks like the 1st pic. And I wants to use Matlab read it into the format as the 2rd pic. I have over 1000 the same kind of CSV files, it will be painful if I do it by copy/paste. How can I do this? Any examples?
raw data:
output data:
First thing to realize is that a .csv file has a very simple format. Your above file is actually a plain text file with the following text on each line:
id,A001
height
a1,a2,a3
3,4,5
3,4,5
6,7,5
weight
a1,a2,a3
4,4,5
5,4,6
i6,7,5
So it is not all that hard for you to write your own parser in Matlab. You want to use commands like
fid = fopen('filename.csv','r');
L = fgetl(fid); % get a text line from the file
commas = find(L==','); % find where the commas are in the line
n1 = str2num(L(1:commas(1)-1); % convert the first comma-delimited number on line L
fidout - fopen('myfile.csv','w');
Lout = [ L(commas(2)+1:commas(3)-1) ', a1, a1'];
fwrite(fidout,Lout); % write a line out to the output file
fclose all; % close all open files.
It will seem slow at first reading the various values in to various variables, and then arranging them to write out the way you want them written out to your output file. But once you get rolling it will go pretty fast and you will find yourself with a pretty good understanding of what is in files, and you will know first hand what is involved in writing something like texscan.m or csvwrite.m and so on.
Good luck!

Fastest way to import CSV files in MATLAB

I've written a script that saves its output to a CSV file for later reference, but the second script for importing the data takes an ungainly amount of time to read it back in.
The data is in the following format:
Item1,val1,val2,val3
Item2,val4,val5,val6,val7
Item3,val8,val9
where the headers are on the left-most column, and the data values take up the remainder of the row. One major difficulty is that the arrays of data values can be different lengths for each test item. I'd save it as a structure, but I need to be able to edit it outside the MATLAB environment, since sometimes I have to delete rows of bad data on a computer that doesn't have MATLAB installed. So really, part one of my question is: Should I save the data in a different format?
Second part of the question:
I've tried importdata, csvread, and dlmread, but I'm not sure which is best, or if there's a better solution. Right now I'm using my own script using a loop and fgetl, which is horribly slow for large files. Any suggestions?
function [data,headers]=csvreader(filename); %V1_1
fid=fopen(filename,'r');
data={};
headers={};
count=1;
while 1
textline=fgetl(fid);
if ~ischar(textline), break, end
nextchar=textline(1);
idx=1;
while nextchar~=','
headers{count}(idx)=textline(1);
idx=idx+1;
textline(1)=[];
nextchar=textline(1);
end
textline(1)=[];
data{count}=str2num(textline);
count=count+1;
end
fclose(fid);
(I know this is probably terribly written code - I'm an engineer, not a programmer, please don't yell at me - any suggestions for improvement would be welcome, though.)
It would probably make the data easier to read if you could pad the file with NaN values when your first script creates it:
Item1,1,2,3,NaN
Item2,4,5,6,7
Item3,8,9,NaN,NaN
or you could even just print empty fields:
Item1,1,2,3,
Item2,4,5,6,7
Item3,8,9,,
Of course, in order to pad properly you would need to know what the maximum number of values across all the items is before hand. With either format above, you could then use one of the standard file reading functions, like TEXTSCAN for example:
>> fid = fopen('uneven_data.txt','rt');
>> C = textscan(fid,'%s %f %f %f %f','Delimiter',',','CollectOutput',1);
>> fclose(fid);
>> C{1}
ans =
'Item1'
'Item2'
'Item3'
>> C{2}
ans =
1 2 3 NaN %# TEXTSCAN sets empty fields to NaN anyway
4 5 6 7
8 9 NaN NaN
Instead of parsing the string textline one character at a time. You could use strtok to break the string up for example
stringParts = {};
tline = fgetl(fid);
if ~ischar(tline), break, end
i=1;
while 1
[stringParts{i},r]=strtok(tline,',');
tline=r;
i=i+1;
if isempty(r), break; end
end
% store the header
headers{count} = stringParts{1};
% convert the data into numbers
for j=2:length(stringParts)
data{count}(j-1) = str2double(stringParts{j});
end
count=count+1;
I've had the same problem with reading csv data in Matlab, and I was surprised by how little support there is for this, but then I just found the import data tool. I'm in r2015b.
On the top bar in the "Home" tab, click on "Import Data" and choose the file you'd like to read. An app window will come up like this:
Import Data tool screenshot
Under "Import Selection" you have the option to "generate function", which gives you quite a bit of customization options, including how to fill empty cells, and what you'd like the output data structure to be. Plus it's written by MathWorks, so it's probably utilizing the fastest available method to read csv files. It was almost instantaneous on my file.
Q1) If you know the max number of columns you can fill empty entries with NaN
Also, if all values are numerical, do you really need "Item#" column? If yes, you can use only "#", so all data is numerical.
Q2) The fastest way to read num. data from a file without mex-files is csvread.
I try to avoid using strings in csv files, but if I have to, I use my csv2cell function:
http://www.mathworks.com/matlabcentral/fileexchange/20135-csv2cell