I have a perl script that reads a .txt and a .bam file, and creates an output called output.txt.
I have a lot of files that are all in different folders, but are only slightly different in the filename and directory path.
All of my txt files are in different subfolders called PointMutation, with the full path being
/Volumes/Lab/Data/Darwin/Patient/[Plate 1/P1H10]/PointMutation
The text(s) in the bracket is the part that changes, But the Patient subfolder contains all of my txt files.
My .bam file is located in a subfolder named DNA with a full path of
/Volumes/Lab/Data/Darwin/Patient/[Plate 1/P1H10]/SequencingData/DNA
Currently how I run this script is go on the terminal
cd /Volumes/Lab/Data/Darwin/Patient/[Plate 1/P1H10]/PointMutation
perl ~/Desktop/Scripts/Perl.pl "/Volumes/Lab/Data/Darwin/Patient/[Plate
1/P1H10]/PointMutation/txtfile.txt" "/Volumes/Lab/Data/Darwin/Patient/[Plate
1/P1H10]/SequencingData/DNA/bamfile.bam"
With only 1 or two files, that is fairly easy, but I would like to automate it once the files get much larger. Also once I run these once, I don't want to do it again, but I will get more information from the same patient, is there a way to block a folder from being read?
I would do something like:
for my $dir (glob "/Volumes/Lab/Data/Darwin/Patient/*/"){
# skip if not a directory
if (! -d $dir) {
next;
}
my $txt = "$dir/PointMutation/txtfile.txt";
my $bam = "$dir/SequencingData/DNA/bamfile.bam";
# ... you magical stuff here
}
This is assuming that all directories under /Volumes/Lab/Data/Darwin/Patient/ follow the convention.
That said, more long term/robust way of organizing analyses with lots of different files all over the place is either 1) organize all files necessary for each analysis under one directory, or 2) to create meta files (i'd use JSON/yaml) which contain the necessary file names.
Related
I want to iterate over all json files in a specific subdirectory.
#!/bin/sh
source_dir="nodes"
for file_path in $source_dir/*.json;
do
file_name=$(basename $file_path .${file_path##*.})
echo $file_name
done
My code is working as expected if there is at least one json file in the directory.
If there is no json file in the directory, the loop will still be executed. The file_name is then "*".
How do I have to change the for loop so that it is only executed if there is at least one json file in the directory?
you can wrap you loop in an if clause to check if the pattern matches anything.
Check this SO question on how to do this: Check if a file exists with wildcard in shell script
I'd like to remove '-2' from the filenames looking like this:
EID-NFBSS-2FE454B7-2_TD.eeg
EID-NFBSS-2FE454B7-2_TD.vhdr
EID-NFBSS-2FE454B7-2_TD.vmrk
EID-NFBSS-3B3BF9FA-2_BU.eeg
EID-NFBSS-2FE454B7-2_PO.txt
So as you may see the names of the files are different and there are different kind of extensions as well. All what I want to do is remove '-2' from all of the filenames. I was trying use this:
pattern = '-2';
replacement = '';
regexprep(filename,pattern,replacement)
and I got the results in the console, but after many attempts I have no idea how to 'say' to MATLAB switch the filnames in the same location.
#excaza hit it right on the money. You'll have to probe your desired directory for a list of files via dir, then loop through each filename and remove any occurrences of -2, then use movefile to rename the file, and delete to delete the old file.
Something like this comes to mind:
%// Get all files in this directory
d = fullfile('path', 'to', 'folder', 'here');
directory = dir(d);
%// For each file in this directory...
for ii = 1 : numel(directory)
%// Get the relative filename
name = directory(ii).name;
%// Replace any instances of -2 with nothing
name_after = regexprep(name, '-2', '');
%// If the string has changed after this...
if ~strcmpi(name, name_after)
%// Get the absolute path to both the original file and
%// the new file name
fullname = fullfile(directory, name);
fullname_after = fullfile(directory, name_after);
%// Create the new file
movefile(fullname, fullname_after);
%// Delete the old file
delete(fullname);
end
end
The logic behind this is quite simple. First, the string d determines the directory where you want to search for files. fullfile is used to construct your path by parts. The reason why this is important is because this allows the code to be platform agnostic. The delineation to separate between directories is different between operating systems. For example, in Windows the character is \ while on Mac OS and Linux, it's /. I don't know which platform you're running so fullfile is going to be useful here. Simply take each part of your directory and put them in as separate strings into fullfile.
Now, use dir to find all files in this directory of your choice. Replace the /path/to/folder/here with your desired path. Next, we iterate over all of the files. For each file, we get the relative filename. dir contains information about each file, and the field you want that is most important is the name attribute. However, this attribute is relative, which means that only the filename itself, without the full path to where this file is stored is given. After, we use regexprep as you have already done to replace any instances of -2 with nothing.
The next point is important. After we try and change the filename, if the file isn't the same, we need to create a new file by simply copying the old file to a new file of the changed name and we delete the old file. The function fullfile here helps establish absolute paths to where your file is located in the off-chance that are you running this script in a directory that doesn't include the files you're looking for.
We use fullfile to find the absolute paths to both the old and new file, use movefile to create the new file and delete to delete the old file.
So I know I can list all the files and directories in my current folder using functions like dir() or ls(), and I know once listed, I can tell them from each other with the field isdir.
But is there a way to exclude the directories from the very beggining and list the files alone?
Even better, is there a way to exclude the current . and parent .. directories -that will (of course) show everytime- and list every other file and directory? Seriously, who uses ls() wondering if . is there?
The output of dir is whatever the operating system is feeding it. So it might be different depending on what system you're running. Here is my approach to that:
list=dir();
CleanList=setdiff({list.name},{'.','..'})';
I'm not sure if there is a built in method for this, but why not write a custom function to do what you want?
Such as:
function list = files_dir(varargin)
% Similar functionality to 'dir', but only returns files (no folders)
list = dir(varargin{:});
list([list.isdir]) = [];
You can then customise this to perform other functionality, such as excluding hidden files.
And for your second request, where directories '.' and '..' are excluded:
function list = dir_exclude_self(varargin)
% same as 'dir', but doesn't return '.' or '..'
list = dir(varargin{:});
self_indices = ismember({list.name}, {'.', '..'});
list(self_indices) = [];
If you put functions like this in a specific place on your computer you can ensure they are always available to use by adding them to the MATLAB path in your startup.m file.
I have one dir with 50 folders, and each folder has 50 files. I have a script to read all files in each folder and save the results, but I need to type the folder name every time. Is there any loop or batch tools I can use? Any suggestions or code greatly appreciated.
There may be a cleaner way to do it, but the output of the dir command can be assigned to a variable. This gives you a struct, with the pertinent fields being name and isdir. For instance, assuming that the top-level directory (the one with 50 files) only has folders in it, the following will give you the first folder's name:
folderList = dir();
folderList(3).name
(Note that the first two entries in the folderList struct will be for "." (the current directory) and ".." (the parent directory), so if you want the first directory with files in it you have to go to the third entry). If you wish to go through the folders one by one, you can do something like the following:
folderList = dir();
for i = 3:length(folderList)
curr_directory = pwd;
cd(folderList(i).name); % changes directory to the next working directory
% operate with files as if you were in that directory
cd(curr_directory); % return to the top-level directory
end
If the top-level directory contains files as well as folders, then you need to check the isdir of each entry in the folderList struct--if it is "1", it's a directory, if it is "0", it's a file.
I have 4 folders in the same directory where each folder contains ~19 .xls files. I have written the code below to obtain the name of each of the folders and the name of each .xls file within the folders.
path='E:\Practice';
folder = path;
dirListing = dir(folder);
dirListing=dirListing(3:end);%first 2 are just pointers
for i=1:length(dirListing);
f{i} = fullfile(path, dirListing(i,1).name);%obtain the name of each folder
files{i}=dir(fullfile(f{i},'*.xls'));%find the .xls files
for j=1:length(files{1,i});
File_Name{1,i}{j,1}=files{1,i}(j,1).name;%find the name of each .xls file
end
end
Now I'm trying to import the data from excel into matlab by using xlsread. What I'm struggling with is knowing how to load the data into matlab within a loop where the excel files are in different directories (different folders).
This leaves me with a 1x4 cell named File_Name where each cell refers to a different folder located under 'path', and within each cell is then the name of the spreadsheets wanting to be imported. The size of the cells vary as the number of spreadsheets in each folder varies.
Any ideas?
thanks in advance
I'm not sure if I'm understanding your problem, but all you have to do is concatenate the strings that contain directory (f{}) and the file name. Modifying your code:
for i=1:length(dirListing);
f{i} = fullfile(path, dirListing(i,1).name);%obtain the name of each folder
files{i}=dir(fullfile(f{i},'*.xls'));%find the .xls files
for j=1:length(files{1,i});
File_Name{1,i}{j,1}=files{1,i}(j,1).name;%find the name of each .xls file
fullpath = [f{i} '/' File_Name{1,i}{j,1}];
disp(['Reading file: ' fullpath])
x = xlsread(fullpath);
end
end
This works on *nix systems. You may have to join the filenames with a '\' on Windows. I'll find a more elegant way and update this posting.
Edit: The command filesep gives the forward or backward slash, depending on your system. The following should give you the full path:
fullpath = [f{i} filesep File_Name{1,i}{j,1}];
Take a look at this helper function, written by a member of the matlab community.
It allows you to recursively search through directories to find files that match a certain pattern. This is a super handy function to use when looking to match files.
You should be able to find all your files in a single call to this function. Then you can loop through the results of the rdir function, loading the files one at a time into whatever data structure you want.