I need to read data from a .dat file. There are no strings in the file, so using fscanf is valid. Below is the entirety of my code.
clear; clc;
file = 'OIL_SAMPLE1.dat';
water = fopen(file,'rt');
waterDataSize = [3 Inf];
format = '%i %d %f\n';
waterData = fscanf(water,format,waterDataSize)
fclose(water);
The file is three columns of numbers in the following format. There are 2151 lines in the .dat file.
1 300 .3
Any ideas? Thanks.
Related
I have a text file whose size is 40000 x 40000, called for example "Name" the first line and first column are the labels and all other number are decimal numbers.
I need to save it into mat file in matlab without the labels, it means without the first line and first columns.
I have tried the following method:
data = importdata('Name.txt') ; %
save data.mat -v7.3
But, I get the data cut into only 590 x 590 , it means it's only a part of the data in the file.
How can I save the the whole data into mat file in matlab ?
EDIT
I also tried this way:
M = readmatrix('Name.txt');
M(:,1) = [];
It read all the rows number of 40000, but the columns are read till 587 !!
At first you should find out where the error happens. Is your data read correctly?
The following solution should work in any case:
fID = fopen('Name.txt');
headerline = fgetl(fID);
C = textscan(fID,['%s' repmat('%f', 1, 40000)], 'Delimiter', '\r');
dMat = [C{2:end}];
save('Data.mat', 'dMat' , '-v7.3')
If this doesn't work you can always try to read the data via:
T = readtable('Name.txt')
and then have a look at the resulting table.
I have a CSV file with the following content:
Header line1
Space
Space
Space
,1,2,3,
1,81,82,83
And I am trying to read the data portion into a numeric matrix.
Here is the code I have implemented, however I am having issues.
%To get the number of rows in the file
for i = 1:9
headerline = fgetl(fid);
headerline = strsplit(headerline,',')
end
fclose(fid);
fopen(fid);
% to get the data
C = textscan(fid,'%s','headerline',4,'EmptyValue',=Inf)
rowsize = size(C{1});
data = []
% to store data in matrix
for i = 1:rowsize
data = [data, strsplit(C{1}{i},',')];
end
Can anybody recommend a better way to just read the whole file into a numeric matrix? Thanks!
All you really need is this;
fid = fopen('your.csv');
data = cell2mat(textscan(fid, '%f%f%f%f', 'Delimiter', ',', 'HeaderLines', 4));
You could also use csvread (https://www.mathworks.com/help/matlab/ref/csvread.html) if your csv just contains numeric values.
M = csvread(filename,R1,C1) reads data from the file starting at row offset R1 and column offset C1. For example, the offsets R1=0, C1=0 specify the first value in the file.
So in this case:
data = csvread('filename.csv', 4, 0)
I need to read the following csv file in MATLAB:
2009-04-29 01:01:42.000;16271.1;16271.1
2009-04-29 02:01:42.000;2.5;16273.6
2009-04-29 03:01:42.000;2.599609;16276.2
2009-04-29 04:01:42.000;2.5;16278.7
...
I'd like to have three columns:
timestamp;value1;value2
I tried the approaches described here:
Reading date and time from CSV file in MATLAB
modified as:
filename = 'prova.csv';
fid = fopen(filename, 'rt');
a = textscan(fid, '%s %f %f', ...
'Delimiter',';', 'CollectOutput',1);
fclose(fid);
But it returs a 1x2 cell, whose first element is a{1}='ÿþ2', the other are empty.
I had also tried to adapt to my case the answers to these questions:
importing data with time in MATLAB
Read data files with specific format in matlab and convert date to matal serial time
but I didn't succeed.
How can I import that csv file?
EDIT After the answer of #macduff i try to copy-paste in a new file the data reported above and use:
a = textscan(fid, '%s %f %f','Delimiter',';');
and it works.
Unfortunately that didn't solve the problem because I have to process csv files generated automatically, which seems to be the cause of the strange MATLAB behavior.
What about trying:
a = textscan(fid, '%s %f %f','Delimiter',';');
For me I get:
a =
{4x1 cell} [4x1 double] [4x1 double]
So each element of a corresponds to a column in your csv file. Is this what you need?
Thanks!
Seems you're going about it the right way. The example you provide poses no problems here, I get the output you desire. What's in the 1x2 cell?
If I were you I'd try again with a smaller subset of the file, say 10 lines, and see if the output changes. If yes, then try 100 lines, etc., until you find where the 4x1 cell + 4x2 array breaks down into the 1x2 cell. It might be that there's an empty line or a single empty field or whatever, which forces textscan to collect data in an additional level of cells.
Note that 'CollectOutput',1 will collect the last two columns into a single array, so you'll end up with 1 cell array of 4x1 containing strings, and 1 array of 4x2 containing doubles. Is that indeed what you want? Otherwise, see #macduff's post.
I've had to parse large files like this, and I found I didn't like textscan for this job. I just use a basic while loop to parse the file, and I use datevec to extract the timestamp components into a 6-element time vector.
%% Optional: initialize for speed if you have large files
n = 1000 %% <# of rows in file - if known>
timestamp = zeros(n,6);
value1 = zeros(n,1);
value2 = zeros(n,1);
fid = fopen(fname, 'rt');
if fid < 0
error('Error opening file %s\n', fname); % exit point
end
cntr = 0
while true
tline = fgetl(fid); %% get one line
if ~ischar(tline), break; end; % break out of loop at end of file
cntr = cntr + 1;
splitLine = strsplit(tline, ';'); %% split the line on ; delimiters
timestamp(cntr,:) = datevec(splitLine{1}, 'yyyy-mm-dd HH:MM:SS.FFF'); %% using datevec to parse time gives you a standard timestamp vector
value1(cntr) = splitLine{2};
value2(cntr) = splitLine{3};
end
%% Concatenate at the end if you like
result = [timestamp value1 value2];
I am working with Matlab code that writes coordinates to a text file as follows:
838959.41800,4713239.59000,1.00000,841305.41800,4708452.59000,1.00000,839175.41800,4712582.59000,1.00000
In the past, I've had to open each file individually and add the following to the first line of the .txt document:
x,y,z,
838959.41800,4713239.59000,1.00000,841305.41800,4708452.59000,1.00000,839175.41800,4712582.59000,1.00000
Any ideas on how to automate this process by embedding code which will write "x,y,z," to the first line of the Matlab output?
Thanks much,
-Aaron
Here is an example:
x = rand(1,3);
y = rand(1,3);
z = rand(1,3);
fid = fopen('data.txt','wt');
fprintf(fid, 'x,y,z,\n');
fprintf([repmat('%.6f,',[1 8]) '%.6f\n'], x,y,z)
fclose(fid);
The output file:
data.txt
x,y,z,
0.694829,0.317099,0.950222,0.034446,0.438744,0.381558,0.765517,0.795200,0.186873
I have a csv file which contains 2d arrays of 4 columns but a varying number of rows. Eg:
2, 354, 23, 101
3, 1023, 43, 454
1, 5463, 45, 7657
4, 543, 543, 654
3, 56, 7654, 344
...
I need to be able to import the data such that I can run operations on each block of data, however csvread, dlmread and textscan all ignore the blank lines.
I can't seem to find a solution anywhere, how can this be done?
PS:
It may be worth noting that the files of the format above are actually the concatenation of many files containing only one block of data (I don't want to have to read from thousands of files every time) therefore the blank line between blocks can be changed to any other delimiter / marker. This is just done with a python script.
EDIT: My Solution - based upon / inspired by petrichor below
I replaced the csvread with textscan which is faster. Then I realised that if I replaced the blank lines with lines of nan instead (modifying my python script) I could remove the need for a second textscan the slow point. My code is:
filename = 'data.csv';
fid = fopen(filename);
allData = cell2mat(textscan(fid,'%f %f %f %f','delimiter',','));
fclose(fid);
nanLines = find(isnan(allData(:,1)))';
iEnd = (nanLines - (1:length(nanLines)));
iStart = [1 (nanLines(1:end-1) - (0:length(nanLines)-2))];
nRows = iEnd - iStart + 1;
allData(nanLines,:)=[];
data = mat2cell(allData, nRows);
Which evaluates in 0.28s (a file of just of 103000 lines). I've accepted petrichor's solution as it indeed best solves my initial problem.
filename = 'data.txt';
%# Read all the data
allData = csvread(filename);
%# Compute the empty line indices
fid = fopen(filename);
lines = textscan(fid, '%s', 'Delimiter', '\n');
fclose(fid);
blankLines = find(cellfun('isempty', lines{1}))';
%# Find the indices to separate data into cells from the whole matrix
iEnd = [blankLines - (1:length(blankLines)) size(allData,1)];
iStart = [1 (blankLines - (0:length(blankLines)-1))];
nRows = iEnd - iStart + 1;
%# Put the data into cells
data = mat2cell(allData, nRows)
That gives the following for your data:
data =
[3x4 double]
[2x4 double]