I´m trying to import some csv files in matlab, but csvread is too slow.
I´m using txt2mat, but i don´t know how to skip the first column in the import.
This is the way im trying
myimportedfile = txt2mat(myfile,'ReadMode','block',1) %im skipping the headers too.
The reason i need to skip is beacause the first column is non-numerical data.
Is there a way to do this with txt2mat or is there a better way?
Tks in advance.
textscan gives you the ability to skip columns. It reads in data using an fprintf-like format string.
Example file:
Val1 Val2 Val3
1 2 3
4 5 6
7 8 9
Code:
tmp = textscan('example.txt', '%i %*i %i') % the * indicates fields to ignore
tmp{:}
Related
I have a .csv file with the first column containing dates, a snippet of which looks like the following:
date,values
03/11/2020,1
03/12/2020,2
3/14/20,3
3/15/20,4
3/16/20,5
04/01/2020,6
I would like to import this data into Matlab (I think the best way would probably be using the readtable() function, see here). My goal is to bring the dates into Matlab as a datetime array. As you can see above, the problem is that the dates in the original .csv file are not consistently formatted. Some of them are in the format mm/dd/yyyy and some of them are mm/dd/yy.
Simply calling data = readtable('myfile.csv') on the .csv file results in the following, which is not correct:
'03/11/2020' 1
'03/12/2020' 2
'03/14/0020' 3
'03/15/0020' 4
'03/16/0020' 5
'04/01/2020' 6
Does anyone know a way to automatically account for this type of data in the import?
Thank you!
My version: Matlab R2017a
EDIT ---------------------------------------
Following the suggestion of Max, I have tried specifiying some of the input options for the read command using the following:
T = readtable('example.csv',...
'Format','%{dd/MM/yyyy}D %d',...
'Delimiter', ',',...
'HeaderLines', 0,...
'ReadVariableNames', true)
which results in:
date values
__________ ______
03/11/2020 1
03/12/2020 2
NaT 3
NaT 4
NaT 5
04/01/2020 6
and you can see that this is not working either.
If you are sure all the dates involved do not go back more than 100 years, you can easily apply the pivot method which was in use in the last century (before th 2K bug warned the world of the danger of the method).
They used to code dates in 2 digits only, knowing that 87 actually meant 1987. A user (or a computer) would add the missing years automatically.
In your case, you can read the full table, parse the dates, then it is easy to detect which dates are inconsistent. Identify them, correct them, and you are good to go.
With your example:
a = readtable(tfile) ; % read the file
dates = datetime(a.date) ; % extract first column and convert to [datetime]
idx2change = dates.Year < 2000 ; % Find which dates where on short format
dates.Year(idx2change) = dates.Year(idx2change) + 2000 ; % Correct truncated years
a.date = dates % reinject corrected [datetime] array into the table
yields:
a =
date values
___________ ______
11-Mar-2020 1
12-Mar-2020 2
14-Mar-2020 3
15-Mar-2020 4
16-Mar-2020 5
01-Apr-2020 6
Instead of specifying the format explicitly (as I also suggested before), one should use the delimiterImportoptions and in the case of a csv-file, use the delimitedTextImportOptions
opts = delimitedTextImportOptions('NumVariables',2,...% how many variables per row?
'VariableNamesLine',1,... % is there a header? If yes, in which line are the variable names?
'DataLines',2,... % in which line does the actual data starts?
'VariableTypes',{'datetime','double'})% as what data types should the variables be read
readtable('myfile.csv',opts)
because the neat little feature recognizes the format of the datetime automatically, as it knows that it must be a datetime-object =)
Just starting with Neuroph NN GUI. Trying to create a dataset by importing a .csv file. What's the file format supposed to be?
I have 3 inputs and 1 output so I assumed the format of the import file would be ..
1,2,3,4
6,7,8,9
But I get error 9, or 4 or 10 depending on what combination I try of newlines, commas etc.
Any help out there ?
many thanks,
john.
That's because you aren't counting with the output column. The lastest columns are for the output.
So, for example, if you have 10 inputs and 1 output, your file will need to have 11 columns.
I came here, because the Neurophy can't import CSVs with title line. Example of a data file that works for me:
1.0,1.0,1.0
1.0,2.0,2.0
1.0,3.0,3.0
1.0,4.0,4.0
1.0,5.0,5.0
1.0,6.0,6.0
1.0,7.0,7.0
1.0,8.0,8.0
1.0,9.0,9.0
1.0,10.0,10.0
2.0,1.0,2.0
2.0,2.0,4.0
2.0,3.0,6.0
2.0,4.0,8.0
2.0,5.0,10.0
2.0,6.0,12.0
2.0,7.0,14.0
2.0,8.0,16.0
2.0,9.0,18.0
2.0,10.0,20.0
I want to import the large data set (multiple column) by using the following code. I want to get all in a single column instead only one row (multi column). So i did transpose operation but it still doesn't work appropriately.
clc
clear all
close all
dataX_Real = fopen('dataX_Real_in.txt');dataX_Real=dataX_Real';
I will really appreciate your support and suggestions. Thank You
The sample files can be found using the following link.
When using fopen, all you are doing is opening up the file. You aren't reading in the data. What is returned from fopen is actually a file pointer that gives you access to the contents of the file. It doesn't actually read in the contents itself. You would need to use things like fread or fscanf to read in the content from the text data.
However, I would recommend you use dlmread instead, as this doesn't require a fopen call to open your file. This will open up the file, read the contents and store it into a variable in one function call:
dataX_Real = dlmread('dataX_Real_in.txt');
By doing the above and using your text file, I get 44825 elements. Here are the first 10 entries of your data:
>> format long;
>> dataX_Real(1:10)
ans =
Columns 1 through 4
-0.307224970000000 0.135961950000000 -1.072544100000000 0.114566020000000
Columns 5 through 8
0.499754310000000 -0.340369000000000 0.470609910000000 1.107567700000000
Columns 9 through 10
-0.295783020000000 -0.089266816000000
Seems to match up with what we see in your text file! However, you said you wanted it as a single column. This by default reads the values in on a row basis, so here you can certainly transpose:
dataX_Real = dataX_Real.';
Displaying the first 10 elements, we get:
>> dataX_Real = dataX_Real.';
>> dataX_Real(1:10)
ans =
-0.307224970000000
0.135961950000000
-1.072544100000000
0.114566020000000
0.499754310000000
-0.340369000000000
0.470609910000000
1.107567700000000
-0.295783020000000
-0.089266816000000
In order to debug a very complex set of functions, I want to isolate a subfunction from the workspace in order to make different test. Therefore a need selected values from the function workspace to be defined already. By setting a break point at the specific position I can "look" into the current workspace by displaying the values in the console, like the variable HF33
HF33 =
1.0777 0.0865 0.0955
-0.1891 0.8110 -0.1889
0.0935 0.0846 1.0755
Is there some function / script that could convert this result to a new Matlab expression that can be pasted somewhere else (for example at the head of a new script), e.g.:
HF33 = [ 1.0777, 0.0865, 0.0955;
-0.1891, 0.8110, -0.1889;
0.0935, 0.0846, 1.0755 ];
With that I could test the subfunction and its behavior by easily changing the given values and see whats happening without having the huge debug workspace running.
Is there some easy function like res2exp(HF33)?
First: Create this function to get the variable name
function out = varname(var)
out = inputname(1);
end
you can print it direct to console:
fprintf('%s =%s\n',varname(varToSave),mat2str(varToSave));
Or use fopen and fprint to write it in a file
fop = fopen('filename','w');
fprint(fop,'%s = %s' ,varname(varToSave),mat2str(varToSave));
fclose(fop);
I think this will help you
It might be a function like mat2str() you are looking for but it will not give exactly the printout you are asking for. Here is an example of how it could be used:
>> A = magic(4)
A =
16 2 3 13
5 11 10 8
9 7 6 12
4 14 15 1
>> B = mat2str(A)
B =
[16 2 3 13;5 11 10 8;9 7 6 12;4 14 15 1]
And if you want the output to be totally copy/paste-able you could use:
disp(['C = ',mat2str(A)])
C = [16 2 3 13;5 11 10 8;9 7 6 12;4 14 15 1]
I made this up just now. It is not formatted beautifully, but it achieves what you are trying to do - if I understand you correctly.
a = [ 2 3 4 5
4 5 5 6
3 4 5 6];
fprintf('\nb = [\n\n');
disp(a);
fprintf(']\n\n');
Copy and paste this and see if it does what you want. It's also very simple code, so you could modify it if the spacing and newline characters aren't where you want them.
You could also make a small function out of this if you wanted to.
If you want me to make a function of it, let me know... I can do it tomorrow. But you can probably figure it out.
Ehh, I just made the function. It didn't take long.
function reprint_matrix(matrix)
var_name = inputname(1);
fprintf('\n%s = [\n\n', var_name);
disp(matrix);
fprintf(']\n\n');
end
I'm not sure what you are looking for, but I think this will help you:
http://www.mathworks.com/matlabcentral/fileexchange/24447-generate-m-file-code-for-any-matlab-variable/content/examples/html/gencode_example.html
Did not use it because I use mat-files to transfer data.
You can combine it with the clipboard function:
clipboard('copy',gencode(ans))
Though there are several ways to write variables to text, saving variables as text is definitely bad practice if it can be avoided. Hence, the best advice I can give you is to solve your problem in a different way.
Suppose you want to use HF33 in your subfunction, then here is what I would recommend:
First of all, save your variable of interest:
save HF33 HF33
Then when you are in the function where you want to use this variable:
load HF33
This assumes that your working directory (not workspace) is the same in both cases, but otherwise you can simply add the path in your save or load command. If you want to display it you can now simply call the variable HF33 without a semicolon (this is probably the only safe way to display it exactly the way you expect in all cases).
Note that this method can easily be adapted to transfer multiple variables at once.
I'm trying to download this comma separated info and save it so that it can be stored as a matrix which can then be accessed. So far I have code which I think should store the info in a file called test.csv but im not sure:
>> urlwrite('http://xweb.geos.ed.ac.uk/~weather/jcmb_ws/JCMB_2013_Mar.csv','test.csv');
d = csvread('test.csv');
??? Error using ==> dlmread at 145
Mismatch between file and format string.
Trouble reading number from file (row 1, field 1) ==> date-
Error in ==> csvread at 50
m=dlmread(filename, ',', r, c);
I am getting the above error. It reads the data fine using urlread. Does anybody know what the correct syntax should be and how I can get it stored as a matrix? Thanks in advance.
You can get the data directly from web without saving to file with URLREAD:
webdata = urlread('http://xweb.geos.ed.ac.uk/~weather/jcmb_ws/JCMB_2013_Mar.csv');
This will give you the whole file as one string with lines delimited by '\n'. You can process it in multiple ways. For example:
tmp = textscan(webdata,['%s',repmat('%f',1,8)],'delimiter',',','headerlines',1);
date = tmp{1};
data = horzcat(tmp{2:end});
To get column headers you can do, for example:
colheader = textscan(webdata,'%s',1,'delimiter','\n');
colheader = regexp(colheader{:},',','split');
colheader = colheader{:};
You can also convert the data to a structure:
Data = cell2struct(tmp, genvarname(colheader),2);
Try using readtext.m. That is a program which can read almost any text file. The problem in your data could be: they don't have uniform delimiters i.e. somewhere two columns are separated by tab, somewhere by comma.
The operation can be performed like this:
urlwrite('http://xweb.geos.ed.ac.uk/~weather/jcmb_ws/JCMB_2013_Mar.csv','test.csv');
data= readtext('test.csv');
It should work.
Your problem lies right here:
Trouble reading number from file (row 1, field 1) ==> date-
Matlab says it encountered "date-" in the first cell. I guess you have a header row or two. You can check in the file and call
d = csvread('test.csv',ROW);
Where ROW is the number of the row where actual data starts (number of header rows + 1).