MATLAB - Read multiple *.log files, calculate and save as .*txt in new folder - matlab

I make different measures and save it like *.log files, calculate it and save as .*txt.
%Filtering log files
l = dir('*.log');
%Array size detection
[rows cols] = size(l);
%Choose a last file
file_name = strcat(strcat(l(rows).folder,'/'),l(rows).name)
%Reading log last file
fileread = fopen(file_name);
%Convert to float
times = fread(fileread,'float32');
%Filtering times and set to 0 small values
times(times<1e-8)=0;
%Set right times values
times_s = times * 1.0e-06;
%Solve full rotation speed (Hz)
motorspeed_full = 1./(2.*times_s)
%Filtering inf values and set to 0
motorspeed_full(motorspeed_full>1e+10)=0;
%Solve half rotation speed (Hz)
motorspeed_half = 1./(times_s);
A = '.txt';
[filepath,name,ext] = fileparts(file_name);
Xfilename = cat(2,name,A);
dlmwrite(Xfilename,motorspeed_full,'precision','%.3f');
So, it's possible to choose a last file, calculating it and convert it to .*txt. So now, I have to make a calculation after every measure.
My aim is:
Making first 1...n measures (1...n - *.log's and wav's)
Calculating and saving 1...n *.log's to *.txt's (see picture)
Create folder with file_name (ex. 20181120_125713) and insert file_name.txt and file_name.wav into this folder structure
Questions:
How can I converting all *.log files to *.txt files using dlmwrite?
How can I create a new folder with file_name
mkdir(name);
for all files?
How to move the files with same names to folder with same name? Name of folder changes every time, so i can't work with
movefile source destination
Thank you very much for any help :*)

Here is my solution. Maybe it can help somebody:
clear all;
clc;
%Filtering log files
l = dir('*.log');
for k = 1:length(l)
next_name = l(k).name
%Array size detection
[rows cols] = size(l);
%Choose a last file
file_name = next_name;
%Reading log last file
fileread = fopen(file_name);
%Convert to float
times = fread(fileread,'float32');
%Filtering times and set to 0 small values
times(times<1e-8)=0;
%Set right times values
times_s = times * 1.0e-06;
%Solve full rotation speed (Hz)
motorspeed_full = 1./(2.*times_s);
%Filtering inf values and set to 0
motorspeed_full(motorspeed_full>1e+10)=0;
%Solve half rotation speed (Hz)
motorspeed_half = 1./(times_s);
A = '.txt';
[filepath,name,ext] = fileparts(file_name);
Xfilename = [name,A];
mkdir(name)
dlmwrite([name,filesep,Xfilename],motorspeed_full,'precision','%.3f');
% [name,'.wav'],[name, file, name, '.wav']
movefile([name,'.wav'],[name, filesep, name, '.wav']);
end

Related

Open multiple folders which have about 500 files under them and extract vtk files under them

I am trying to open multiple folders which have about 500 files under them and then use a function called vtkread to read the files in those folders. I am not sure how to set that up.
So here is my function but I am stuggling with setting up the mainscript to select files from a folder
function [Z_displacement,Pressure] = Processing_Code2_Results(filename, reduce_time, timestep_total)
fid = fopen(filename,'r');
Post_all = [];
vv=[1:500];
DANA0 = vtkRead('0_output_000000.vtk'); %extract all data from the vtk file including disp, pressure, points, times
C = [DANA0.points,reshape(DANA0.pointData.displacements,size(DANA0.points)),reshape(DANA0.pointData.pressure,[length(DANA0.points),1])];
disp0 = reshape(DANA0.pointData.displacements,[1,size(C,1),3]);
points = DANA0.points; % This is a matrix of the xyz points
for i = 1:reduce_time:timestep_total %34
DANA = vtkRead(sprintf('0_output_%06d.vtk',i)); % read in each successive timestep
disp(i,:,:) = DANA.pointData.displacements; % store displacement for multiple timesteps
pressure(i,:) = DANA.pointData.pressure; % store pressure for multiple timesteps
% press = pressure';
end
...
I have tried something like this:
clc; clear;
timestep_total = 500;
reduce_time = 100;
cd 'C:\Users\Admin\OneDrive - Kansas State University\PhD\Project\Modeling\SSGF_Model\New_Model_output'
for i = 1:3
filename = sprintf("Gotherm_%d",i)
[Z_displacement_{i},Pressure_{i}] = Processing_Code2_Results(filename, reduce_time, timestep_total);
end

How to add standrad deviation and moving average

What I want to is:
I got folder with 32 txt files and 1 excle file, each file contain some data in two columns: time, level.
I already managed to pull the data from the folder and open each file in Matlab and get the data from it. What I need to do is create plot for each data file.
each of the 32 plots should have:
Change in average over time
Standard deviation
With both of this things I am straggling can't make it work.
also I need to make another plot this time the plot should have the average over each minute from all the 32 files.
here is my code until now:
clc,clear;
myDir = 'my path';
dirInfo = dir([myDir,'*.txt']);
filenames = {dirInfo.name};
N = numel(filenames);
data=cell(N,1);
for i=1:N
fid = fopen([myDir,filenames{i}] );
data{i} = textscan(fid,'%f %f','headerlines',2);
fclose(fid);
temp1=data{i,1};
time=temp1{1};
level=temp1{2};
Average(i)=mean(level(1:find(time>60)));
AverageVec=ones(length(time),1).*Average(i);
Standard=std(level);
figure(i);
plot(time,level);
xlim([0 60]);
hold on
plot(time, AverageVec);
hold on
plot(time, Standard);
legend('Level','Average','Standard Deviation')
end
the main problam with this code is that i get only average over all the 60 sec not moving average, and the standard deviation returns nothing.
few things you need to know:
*temp1 is 1x2 cell
*time and level are 22973x1 double.
Apperently you need an alternative to movmean and movstd since they where introduced in 2016a. I combined the suggestion from #bla with two loops that correct for the edge effects.
function [movmean,movstd] = moving_ms(vec,k)
if mod(k,2)==0,k=k+1;end
L = length(vec);
movmean=conv(vec,ones(k,1)./k,'same');
% correct edges
n=(k-1)/2;
movmean(1) = mean(vec(1:n+1));
N=n;
for ct = 2:n
movmean(ct) = movmean(ct-1) + (vec(ct+n) - movmean(ct-1))/N;
N=N+1;
end
movmean(L) = mean(vec((L-n):L));
N=n;
for ct = (L-1):-1:(L-n)
movmean(ct) = movmean(ct+1) + (vec(ct-n) - movmean(ct+1))/N;
N=N+1;
end
%mov variance
movstd = nan(size(vec));
for ct = 1:n
movstd(ct) = sum((vec(1:n+ct)-movmean(ct)).^2);
movstd(ct) = movstd(ct)/(n+ct-1);
end
for ct = n+1:(L-n)
movstd(ct) = sum((vec((ct-n):(ct+n))-movmean(ct)).^2);
movstd(ct) = movstd(ct)/(k-1);
end
for ct = (L-n):L
movstd(ct) = sum((vec((ct-n):L)-movmean(ct)).^2);
movstd(ct) = movstd(ct)/(L-ct+n);
end
movstd=sqrt(movstd);
Someone with matlab >=2016a can compare them using:
v=rand(1,1E3);m1 = movmean(v,101);s1=movstd(v,101);
[m2,s2] = moving_ms(v,101);
x=1:1E3;figure(1);clf;
subplot(1,2,1);plot(x,m1,x,m2);
subplot(1,2,2);plot(x,s1,x,s2);
It should show a single red line since the blue line is overlapped.

Read all .csv-files in folder and plot their content

By an old post (https://stackoverflow.com/a/13744310/3900582) I have been able to read all the .csv-files in my folder into a cell array. Each .csv-file has the following structure:
0,1024
1,427
2,313
3,492
4,871
5,1376
6,1896
7,2408
8,2851
9,3191
Where the left column is the x-value and the right column is the y-value.
In total, there are almost 200 files and they are each up to 100 000 lines long. I would like to plot the contents of the files in one figure, to allow the data to be more closely inspected.
I was able to use the following code to solve my problem:
dd = dir('*.csv');
fileNames = {dd.name};
data = cell(numel(fileNames),2);
data(:,1) = regexprep(fileNames, '.csv','');
for i = 1:numel(fileNames)
data{i,2} = dlmread(fileNames{i});
end
fig=figure();
hold on;
for j = 1:numel(fileNames)
XY = data{j,2};
X = XY(:,1);
Y = XY(:,2);
plot(X,Y);
end

Processing a large dataset

My MATLAB program generates N=100 trajectories with T=10^8 time steps in each, i.e.
x = randn(10^8,100);
Ultimately, I want to process this data set and obtain an average autocorrelation of all trajectories:
y = mean(fft(x),2); % output size (10^8, 1)
Now since x is too big to store, my only viable option is to save it on the hard drive in small chunks of 10^6
x1 = randn(10^6, 100);
x2 = randn(10^6, 100);
etc
and then obtain y by processing each trajectory n=1:100 individually and accumulating the result:
for n=1:100
y = y + fft([x1(:,n); x2(:,n); ...; x100(:,n)]);
end
Is there a more elegant way of doing this? I have 100GB of RAM and a pool of 12 workers.
An easier way would be to generate your data once and then chop it into small pieces you save on disk, or, if possible, create the data on the workers themselves.
x = randn(10^8,100);
for ii=1:100
if ii ~=100
tmp = x(ii:ii+1e6)
else
tmp = x(ii:end); %ii+1e6 would result in end+1
end
filename = sprintf('Dataset%i',ii); %create filename
save(filename,tmp,'-v7.3'); %save file to disk in -v7.3 format
end
y = cell(100,1) %initialise output
parfor ii = 1:100
filename = sprintf('Dataset%i',ii); %get the filenames back
load(filename); %load the file
y{ii} = mean(fft(tmp),2); % I called it tmp when saving, so it's called tmp here
end
Now you can accumulate the results from the cell y in the desired manner. You can of course play around with the amount of files you create, as less files will be processed faster due to the overhead of parfor.

frame to video conversion matlab

So I just started with image processing/computer vision in MATLAB.
So my first task is to convert a series of images(frames) into a video. So I went through online sources (MATLAB website more specifically) to get a way to do it.
So the one that I implemented is http://www.mathworks.com/help/matlab/examples/convert-between-image-sequences-and-video.html which solved the problem for me.
However, when I play it, the video seems jumpy in some places. Like it would bring a different frame in the middle and make the whole video jumpy for that split second. It happens a couple of places in the video.
Any anyone knows why this happens?
Thanks
PS below is the code I use:
myFolder = 'C:\Users\owner\Desktop\MATLAB GUI\Color\Color'; %Specify Directory
filePattern = fullfile(myFolder, '*.jpg') %identify jpg files
jpegFiles = dir(filePattern) %use dir to list jpg files
size = length(jpegFiles); % length of the size of the file
outputVideo = VideoWriter(fullfile(myFolder,'video1.avi'));
outputVideo.FrameRate = 30;
open(outputVideo);
for i = 1:length(jpegFiles) %load all the files in the directory
j = i; %%accumulating the number of jpegfiles into handles.j
baseFileName = jpegFiles(i).name;
fullFileName = fullfile(myFolder, baseFileName);
%fprintf(1, 'Now reading %s\n', fullFileName); %filename of image
imageArray = imread(fullFileName); %image being read
%imageArray = rgb2gray(imageArray);
imagecell{i} = imageArray; %storing the images in imagecells
writeVideo(outputVideo,imagecell{i});
end
close(outputVideo);
video1 = VideoReader(fullfile(myFolder,'video1.avi'));
mov(video1.NumberOfFrames) = struct('cdata',[],'colormap',[]);
for ii = 1:video1.NumberOfFrames
mov(ii) = im2frame(read(video1,ii));
end
set(gcf,'position', [150 150 video1.Width video1.Height])
set(gca,'units','pixels');
set(gca,'position',[0 0 video1.Width video1.Height])
image(mov(1).cdata,'Parent',gca);
axis off;
movie(mov,1,video1.FrameRate);
Given that there may be too many files to be renamed (padded with zeros) here is a quick function that will do it for you: you just need to provide the directory/folder where the images are stored, the padding (if less 100 files, then padding can be 2; if less than 1000 files, then padding can be 3; etc.), and a common pattern. The code assumes that there is a common pattern in each file (like 'frame' or 'image') that when removed, leaves just the number:
renameFiles(directory,padSize,fileNamePattern)
filePattern = fullfile(directory, '*.jpg') %identify jpg files
jpegFiles = dir(filePattern) %use dir to list jpg files
for k=1:size(jpegFiles)
% get the source file that will be moved/renamed
fileSrc = jpegFiles(k).name;
% get the parts of the file
[path,name,ext] = fileparts(fileSrc);
% where does the pattern fit in the name?
idx = strfind(name,fileNamePattern);
% remove the pattern from the name
if idx==0
% pattern does not exist in file so skip it
continue;
elseif idx==1
% pattern is at the beginning of name so remove it
frameNumberStr = name(length(fileNamePattern)+1:end);
else
% pattern is at the end of name so remove it
frameNumberStr = name(1:idx-1);
end
% get the number of digits
numDigits = length(frameNumberStr);
% create the new file name
paddedName = [fileNamePattern repmat('0',1,padSize-numDigits) frameNumberStr];
fprintf('%s\n',paddedName);
% only move if the destination is different
if strcmp(paddedName,name) ~= 1
% set the destination file
fileDest = fullfile(directory,[paddedName ext]);
% move the file
movefile(fileSrc, fileDest,'f');
end
end
end
An example - if all files have the common pattern of 'frame' and there are less than 1000 files, then run this function as
rename(pwd,3,'frame')
All files that were named as frame1.jpg or frame99.jpg are now named as frame001.jpg and frame099.jpg.
Hope this helps!
If you have the Computer Vision System Toolbox you can use the vision.VideoFileWriter object. You can simply feed images into its step() method one at a time, and they will be written to the video file as video frames. Note that the images must all be the same size and have the same data type.