[Edited:] I have a file data2007a.csv and I copied and pasted (using TextEdit in MacBook) the first consecutive few lines to a new file datatest1.csv for testing:
Nomenclature,ReporterISO3,ProductCode,ReporterName,PartnerISO3,PartnerName,Year,TradeFlowName,TradeFlowCode,TradeValue in 1000 USD
S3,ABW,0,Aruba,ANT,Netherlands Antilles,2007,Export,6,448.91
S3,ABW,0,Aruba,ATG,Antigua and Barbuda,2007,Export,6,0.312
S3,ABW,0,Aruba,CHN,China,2007,Export,6,24.715
S3,ABW,0,Aruba,COL,Colombia,2007,Export,6,95.885
S3,ABW,0,Aruba,DOM,Dominican Republic,2007,Export,6,11.432
I wanted to use textscan to read it into MATLAB with only columns 2,3,5 (starting from the second row) and I wrote the following code
clc,clear all
fid = fopen('datatest1.csv');
data = textscan(fid,'%*s %s %d %*s %s %*[^\n]',...
'Delimiter',',',...
'HeaderLines',1);
fclose(fid);
But I ended up with only the second row of columns 2,3 and 5:
I then keep the first row in data2007a.csv and selected several others to saved as datatest2.csv:
Nomenclature,ReporterISO3,ProductCode,ReporterName,PartnerISO3,PartnerName,Year,TradeFlowName,TradeFlowCode,TradeValue in 1000 USD
S3,ABW,1,Aruba,USA,United States,2007,Export,6,1.392
S3,ABW,1,Aruba,VEN,Venezuela,2007,Export,6,5633.157
S3,ABW,2,Aruba,ANT,Netherlands Antilles,2007,Export,6,310.734
S3,ABW,2,Aruba,USA,United States,2007,Export,6,342.42
S3,ABW,2,Aruba,VEN,Venezuela,2007,Export,6,63.722
S3,AGO,0,Angola,DEU,Germany,2007,Export,6,105.334
S3,AGO,0,Angola,ESP,Spain,2007,Export,6,8533.125
And I wrote:
clc,clear all
fid = fopen('datatest2.csv');
data = textscan(fid,'%*s %s %d %*s %s %*[^\n]',...
'Delimiter',',',...
'HeaderLines',1);
fclose(fid);
data{1}
It gives exactly what I wanted:
When I use the same code for my original data file data2007a.csv, it goes as in the first case.
What is going wrong and how can I fix it?
[Added:] If one replicates my experiments1, one can find that both cases work and the problem does not exist! I really don't know what is going on.
1 For "replicate" I mean copy-and-paste the data given above and save it as two new files, say, datatest4a.csv and datatest4b.csv. I used visdiff('datatest1.csv', 'datatest4a.csv') to compare two files and it returned:
Given how you fixed it, I think this is an end-of-line character issue. This sometimes comes up when moving text files between Windows and Unix based systems, as they use different conventions.
When you add %*[^\n] to the end of a textscan format, as you have here. it means to skip everything to the end of line. But if it expects a specific end of line character, and can't find one, it will skip everything to the end of the file. This would explain why you get one row correctly read and then nothing else.
If you don't specify what the end of line character is, Matlab appears to default to... something... in this not very clear specification in the help:
The default end-of-line sequence is \n, \r, or \r\n, depending on the contents of your file.
One way to try and cure this without having to create a new file would be to add this 'EndOfLine', '\r\n' to your textscan call:
If you specify '\r\n', then textscan treats any of \r, \n, and the
combination of the two (\r\n) as end-of-line characters.
This will hopefully handle most standard(ish) EOL conventions. It is likely that copy-pasting and saving with a different bit of software than was originally used to create the file changed the end of line characters such that Matlab was able to recognise them.
I got a critical problem. I am working on a code which should analyze data from an experiment. I got the data in a .csv file and tried import these data to work with them in MATLAB.
My problem is that my code imported the .csv data and overwrote my .csv file.
Can you help to find a way how to import the files without overwriting/erasing the original .csv file?
My code is here. Is this okay or shall I try something else?
fid = fopen('data.csv');
data = textscan(fid, '%*s %*f %f', 'delimiter', ';', 'headerlines', 5);
fclose(fid);
I don't see a problem with your posted code snippet, but the importdata function may be more convenient to use: http://www.mathworks.com/help/matlab/ref/importdata.html
I am trying to read in a .csv file with MATLAB. Here is my code:
csvread('out2.csv')
This is what out2.csv looks like:
03/09/2013 23:55:12,129.32,129.33
03/09/2013 23:55:52,129.32,129.33
03/09/2013 23:56:02,129.32,129.33
On windows I am able to read this exact same file with the xlsread function with no problems. I am currently on a linux machine. When I first used xlsread to read the file I was told "File is not in recognized format" so I switched to using csvread. However, using csvread, I get the following error message:
Error using dlmread (line 139)
Mismatch between file and format string.
Trouble reading number from file (row 1u, field 2u) ==> /09/2013
23:55:12,129.32,129.33\n
Error in csvread (line 48)
m=dlmread(filename, ',', r, c)
I think the '/' in the date is causing the issue. On windows, the 1st column is interpreted as a string. On linux it seems to be interpreted as a number, so it tries to read the number and fails at the backslash. This is what I think is going on at least. Any help would be really appreciated.
csvread can only read doubles, so it's choking on the date field. Use textscan.
fid = fopen('out2.csv');
out = textscan(fid,'%s%f%f','delimiter',',');
fclose(fid);
date = datevec(out{1});
col1 = out{2};
col2 = out{3};
Update (8/31/2017)
Since this was written back in 2013, MATLAB's textscan function has been updated to directly read dates and times. Now the code would look like this:
fid = fopen('out2.csv');
out = textscan(fid, '%{MM/dd/uu HH:mm:ss}D%f%f', 'delimiter', ',');
fclose(fid)
[date, col1, col2] = deal(out{:});
An alternative as mentioned by #Victor Hugo below (and currently my personal go-to for this type of situation) would be to use readtable which will accept the same formatting string as textscan but assemble the results directly into a table object:
dataTable = readtable('out2.csv', 'Format', '%{MM/dd/uu HH:mm:ss}D%f%f')
dataTable.Properties.VariableNames = {'date', 'col1', 'col2'};
dataTable =
3×3 table
date col1 col2
___________________ ______ ______
03/09/2013 23:55:12 129.32 129.33
03/09/2013 23:55:52 129.32 129.33
03/09/2013 23:56:02 129.32 129.33
Unfortunately, the documentation for csvread clearly states:
M = csvread(filename) reads a comma-separated value formatted file, filename. The file can only contain numeric values.
Since / is neither a comma, nor a numeric value, it produces an error.
You can use readtable, as it will accept any input.
https://www.mathworks.com/help/matlab/ref/readtable.html
Yeah xlsread requires Microsoft Excel to be installed, unless it is run in 'basic' mode and 'basic' mode only reads .xls .xlsx and .xlsm files.
Another alternative are a number of user-written functions posted on MATLAB's file exchange that will work on linux and are more flexible at reading non formatted content.
One example:
https://www.mathworks.com/matlabcentral/fileexchange/75219-csv2cellfast-import-csv-files-on-machines-without-excel
As a part of my assignment, I have to read a .csv file. The file contains a mixture of text, numeric data and missing data under the columns:
Number, Title, Description (>100 words, variable length), Location, Time, Term, Company, Category, Source.
There are more than 0.5 million rows.
Suggest me a command to read this file into MATLAB.
I have already tried the following:
uiopen('filename.csv',1)
It gives error: Use textscan to read more complex formats. Then I tried:
data =textscan('filename.csv','%f %s %s %s %s %s %s %s %s %f','HeaderLines', 1, 'Delimiter', ',');
This command runs to completion, but it only gives an array (1X10) of cells (which are empty). Hence, I am not getting what I want.
I also tried textread command but it gives error.
textscan is what you want to use but according to the matlab documentation page for textscan the first argument is supposed to be a file id. Right now you are passing in a string.
You may want to try using readtable:
t = readtable('filename.csv');
This will create a table in Matlab, which can contain the strings and numeric data.
Alternatively, you can use the Import Tool (accessible from the Import Data button on the Matlab UI), or you can use uiimport to open it:
uiimport('filename.csv')
This will present a graphical presentation of your data, and can generate code for you to do the import as well.
The difficulty you may face with textscan is that you need to get the formats (%f, %s, etc...) correct, and any variations in this may cause failures. For example, if you have strings in a numeric field because of some missing/bad data, it may fail. If you choose to use textscan and don't get the results you're expecting, you may want to try with all '%s' format specifications.
textscan(f,'%s %s %s %s %s %s %s %s %s %s','HeaderLines', 1, 'Delimiter', ',');
I am having a .csv file which contains let's say 50 rows.
At the beginning of each row I have a file name in the following format 001_02_03.bmp followed by values separated by commas. Something like this :
001_02_03.bmp,20,30,45,10,40,20
Can someone tell me how can I read the first column from the data?
I know how to obtain the data from the second column onward. I am using the csvread function like this X = csvread('filename.csv', 0, 1);. If I try to read the first column in the same manner it outputs an error, saying the csvread does not support string format.
Use textscan, ie:
fid1 = fopen(csvFileName);
X = textscan(fid1, '%s%f%f%f%f%f%f', 'Delimiter', ',');
fclose(fid1);
FirstCol = X{1, 1};
A little more detail? csvread only works with purely numeric data, so you can't use it to get in data with a .bmp, or underscores for that matter. Thus we use textscan. The funny looking format string input to textscan is just saying that the columns are, in order, of type string %s, then the next 6 columns are of type double %f%f%f%f%f%f (or you might choose to alter this to reflect an integer datatype - I personally rarely bother unless the quantity of data is huge or floating point precision is a problem).
Note, if you just wanted to get the first column and ignore the rest, you can replace the format string with %s% %*[^\n]. A final point, if your csv file has a header line, you can skip it using the HeaderLines optional input to textscan.