Using Fish, how can I delete the contents of a directory except for certain files (or directories). Something like rm !(file1|file2) from bash, but fishier.
There is no such feature in fish - that's issue #1444.
You can do something like
rm (string match -rv '^file1$|^file2$' -- *)
Note that this will fail on filenames with newlines in them.
Or you can do the uglier:
set -l files *
for file in file1 file2
if set -l index (contains -i -- $file $files)
set -e files[$index]
end
end
rm $files
which should work no matter what the filenames contain.
Or, as mentioned in that issue, you can use find, e.g.
find . -mindepth 1 -maxdepth 1 -type f -a ! \( -name 'file1' -o -name 'file2' \)
Related
source="/somedir/dir-a"
dest="/somedir2/dir-z"
I need to find all files recursively within the $source directory which contain the string 720p and move them to $dest
Just 2 things to take care of -
For all such files which are to be moved , first create that file's outer 2 directories in $dest and then move this matched file inside that
i have to do this for lakhs of files so a bit of parallelization would be helpful
Example
For a file like - "$source/dir-b/dir-c/file-720p.mp4" , it should do as follows :
mkdir -p "$dest/dir-b/dir-c"
mv "$source/dir-b/dir-c/file-720p.mp4" "$dest/dir-b/dir-c/file-720p.mp4"
You're looking for something like this:
src=foo
dst=bar
export dst
find "${src}" -name '*720p*' -type f -exec sh -c '
for p; do
np=${dst}${p#"${p%/*/*/*}"}
echo mkdir -p "${np%/*}" &&
echo mv "$p" "$np"
done' sh {} +
This can be parallelized using GNU find's -print0 primary in conjunction with GNU xargs, but I don't think that'd make much of a difference performance-wise, as this is rather an IO-intensive task.
Remove echos if the output is satisfactory.
I used this line to try and split all of the files in a directory into smaller files and put them in a new directory.
cd /path/to/files/ && find . -maxdepth 1 -type f -exec split -l 1000 '{}' "path/to/files/1/prefix {}" \;
The result was 'no file found', so how do I make this work so that I split all of the files in a directory into smaller 1000-line files and place them in a new directory?
Later on...
I tried many variations and this is not working. I read another article that split cannot operate on multiple files. Do I need to make a shell script, or how do I do this?
I had a bright idea to use a loop. So, I researched the 'for' loop and the following worked:
for f in *.txt.*; do echo "Professing $f file..."; split -l 1000 $f 1split.ALLEMAILS.txt. ; done
.txt. is in all of the files in the working directory. the 'echo' command was optional. for the 'split' command, instead of naming one file, I replaced that with $f as defined by the 'for' line.
The only thing I would like to have been able to do is move all of these to another directory in the command.
Right now, I am stuck on the find command for moving all matching files. This is what I have done so far that is not working:
find . -type f -name '1split.*' -exec mv {} new/directory/{} \;
I get the error ' not a directory ' ; or I tried:
find . -type f -name '1split.*' -exec mv * . 1/ \;
and I get ' no such file or directory '
Any ideas?
I found that this command moved ALL of the files to the new directory instead of the ones specifically meeting the criteria '1split.*'
So, the answers to my questions are:
for f in *.txt.*; do echo "Professing $f file..."; split -l 1000 $f 1split.ALLEMAILS.txt. ; done
and
mv *searchcriteria /new/directory/path/
I did not need a find command for this after all. So, combining both of these would have done the trick:
for f in *.txt.*; do echo "Professing $f file..."; split -l 1000 $f 1split.ALLEMAILS.txt. ; done
mv *searchcriteria /new/directory/path/ | echo "done."
---later on...
I found that this basically took 1 file and processed it.
I fixed that with a small shell script:
#!/bin/sh
for f in /file/path/*searchcriteria ; ## this was 'split.*' in my case
do echo "Processing $f in /file/path/..." ;
perl script.pl --script=options $f > output.file ;
done ;
echo "done."
I want to remove millions of files in a directory and pages mentioned that the following Perl code is the fastest:
perl -e 'chdir "BADnew" or die; opendir D, "."; while ($n = readdir D) { unlink $n }`
However, is it also possible to do this on only files containing the word 'sorted'? Does anyone know how to rewrite this?
It can be done using find and grep combination:
find BADnew -type f -exec grep -q sorted {} \; -exec rm {} \;
Second -exec command will be executed only if return code for first one is zero.
You can do dry run:
find BADnew -type f -exec grep -q sorted {} \; -exec echo {} \;
the core module File::Find will recursively traverse all the subdirectories and perform a subroutine on all files found
perl -MFile::Find -e 'find( sub { open $f,"<",$_; unlink if grep /sorted/, <$f> }, "BADnew")'
Try:
find /where -type f -name \* -print0 | xargs -0 grep -lZ sorted | xargs -0 echo rm
#can search for specific ^^^ names ^^^^^^ ^^^^
# what should contain the file |
# remove the echo if satisfied with the result +
The above:
the find searches for files with a specified name (* - any)
the xargs ... grep list files what are contains the string
the xargs rm - removes the files
don't dies on "arg count too long"
the files could have whitespaces in their names
needs grep what knows the -Z
Also a variant:
find /where -type f -name \* -print0 | xargs -0 grep -lZ sorted | perl -0 -nle unlink
You haven't made it clear, despite specific questions, whether you require the file name or the file contents to contain sorted. Here are both solutions
First, chdir to the directory you're interested in. If you really need a one-liner for whatever reason then it is pointless to put the chdir inside the program.
cd BADnew
Then you can either unlink all nodes that are files and whose name contains sorted
perl -e'opendir $dh, "."; while(readdir $dh){-f and /sorted/ and unlink}'
or you can open each file and read it to see if its contents contain sorted. I hope it's clear that this method will be far slower, not least because you have to read the entire file to establish a negative. Note that this solution relies on the
perl -e'opendir $dh, "."; while(readdir $dh){-f or next; #ARGV=$f=$_; /sorted/ and unlink($f),last while <>}'
How to copy a file to several directories of the form *Co*? or *52?
Apparently, just typing
cp fileA *Co*
won't work.
My other concern is that if a directory already contains fileA, I don't want it to be overwritten. That is, if the directory *Co* contains fileA, do NOT copy. Is there a one line solution for this, since I think writing a script with if-else is an overkill.
Thanks!
If your version of cp supports -n, you can do:
find . -name '*Co*' -exec cp -n fileA {} \;
If not:
find . -name '*Co*' -exec sh -c 'test -f $0/fileA || cp fileA $0' {} \;
Note that these will each descend recursively: if you don't want that you can limit the scope of find. To find either Co or *52, you can do:
find . \( -name '*Co*' -o -name '*52' \) -exec ...
I am interested into getting into bash scripting and would like to know how you can traverse a unix directory and log the path to the file you are currently looking at if it matches a regex criteria.
It would go like this:
Traverse a large unix directory path file/folder structure.
If the current file's contents contained a string that matched one or more regex expressions,
Then append the file's full path to a results text file.
Bash or Perl scripts are fine, although I would prefer how you would do this using a bash script with grep, awk, etc commands.
find . -type f -print0 | xargs -0 grep -l -E 'some_regexp' > /tmp/list.of.files
Important parts:
-type f makes the find list only files
-print0 prints the files separated not by \n but by \0 - it is here to make sure it will work in case you have files with spaces in their names
xargs -0 - splits input on \0, and passes each element as argument to the command you provided (grep in this example)
The cool thing with using xargs is, that if your directory contains really a lot of files, you can speed up the process by paralleling it:
find . -type f -print0 | xargs -0 -P 5 -L 100 grep -l -E 'some_regexp' > /tmp/list.of.files
This will run the grep command in 5 separate copies, each scanning another set of up to 100 files
use find and grep
find . -exec grep -l -e 'myregex' {} \; >> outfile.txt
-l on the grep gets just the file name
-e on the grep specifies a regex
{} places each file found by the find command on the end of the grep command
>> outfile.txt appends to the text file
grep -l -R <regex> <location> should do the job.
If you wanted to do this from within Perl, you can take the find commands that people suggested and turn them into a Perl script with find2perl:
If you have:
$ find ...
make that
$ find2perl ...
That outputs a Perl program that does the same thing. From there, if you need to do something that easy in Perl but hard in shell, you just extend the Perl program.
find /path -type f -name "*.txt" | awk '
{
while((getline line<$0)>0){
if(line ~ /pattern/){
print $0":"line
#do some other things here
}
}
}'
similar thread
find /path -type f -name "outfile.txt" | awk '
{
while((getline line<$0)>0){
if(line ~ /pattern/){
print $0":"line
}
}
}'