Move folders that have more than one file into another directory - sh

POWER SHELL ERROR Picture of code and directories
I would like to create a batch file that moves all folders that contain more than one file to another directory.
I tried the code below
mkdir "OOOO3_MORE_THAN_ONE"
for dir in *; do
# if the file is a directory
if [ -d "$dir" ]; then
# count number of files
count=$(find "$dir" -type f | wc -l)
#i f count=2 then move
if [ "$count" -le 1 ]; then
# move dir
mv -v "$dir" /completepath/"OOOO3_MORE_THAN_ONE"
fi
fi
done
I just get a new folder without any folders inside. The folders with multiple files did not move to the new directory
I also tried the below code, it's a little different, but still resulted in an empty folder
#! /bin/bash -p
shopt -s nullglob # glob patterns that match nothing expand to nothing
shopt -s dotglob # glob patterns expand names that start with '.'
destdir='subset'
[[ -d $destdir ]] || mkdir -- "$destdir"
for dir in * ; do
[[ -L $dir ]] && continue # Skip symbolic links
[[ -d $dir ]] || continue # Skip non-directories
[[ $dir -ef $destdir ]] && continue # Skip the destination dir.
numfiles=$(find "./$dir//." -type f -print | grep -c //)
(( numfiles > 1 )) && mv -v -- "$dir" "$destdir"
done

Alright, you have two problems (as originally posted).
(1) you are attempting to move to /completepath/OOOO3_MORE_THAN_ONE after creating "OOOO3_MORE_THAN_ONE" in the current working directory. Unless you are executing the script in /completepath when the directory ./OOOO3_MORE_THAN_ONE is created, your calls to mv -v "$dir" /completepath/"OOOO3_MORE_THAN_ONE"will fail. (the double-quotes are superfluous here)
(2) you have a "chicken-and-the-egg" problem because you:
mkdir "OOOO3_MORE_THAN_ONE"
before you call:
for dir in *; do
(OOOO3_MORE_THAN_ONE is included in '*', but not excluded before your calls to find and mv)
So you will effectively try and move OOOO3_MORE_THAN_ONE below itself when its name is reached in your list.
So How To Correct the Problems?
Rearrange your code. Since your Question is tagged [sh] (POSIX shell), you do not have the benefit of arrays available to pre-store the count and dir names, but you can always use a temporary file created with mktemp. You will want to read through each directory entry identified with for dir in * and write all count and dir information out to your temporary file before you start changing the directory structure. Then you can simply loop over the entries in your temporary file, checking if $count -gt 1 and moving to your new $dir if so, e.g.
#!/bin/sh
## initialize newdir from 1st argument (or default: OOOO3_MORE_THAN_ONE)
newdir="${1:-OOOO3_MORE_THAN_ONE}"
## set your complete path from 2nd arg (or '.' by default)
cmpltpath="${2:-.}"
## now create a temporary file to hold count dirname pairs
tmpfile=$(mktemp)
for dir in *; do ## write count and dirname pairs to temporary file
[ -d "$dir" ] && echo "$(find "$dir" -type f | wc -l) $dir" >> "$tmpfile"
done
## now create the directory to move to using cmpltpath, exit on failure
mkdir -p "$cmpltpath/$newdir" || exit 1
## read count and dirname from tmpfile & move if count > 1
while read -r count dir || [ -n "$dir" ]; do
## if count=2 then move
if [ "$count" -gt 1 ]; then
## move to dir
mv -v "$dir" "$cmpltpath/$newdir"
fi
done < "$tmpfile"
rm "$tmpfile" ## tidy up and remove tmpfile (or set trap after mktemp)
(note: the script takes the directory name to create and move files below as the first argument (positional parameter) for the script, and the complete path (absolute or relative) to precede your new directory)
(also note: if you have bash (or another advanced shell that supports associative arrays), you can simply save the directory name and count within an associative array keyed on directory name and avoid using a temporary file altogether)
Original directory
Using a directory tree where each subdirectory d1, d2, d3 has 1, 2 or 3 files below them, e.g.:
$ tree
.
├── d1
│   └── file1
├── d2
│   ├── file1
│   └── file2
├── d3
│   ├── file1
│   ├── file2
│   └── file3
└── mvscript.sh
Example Use/Resulting Directory Structure
Now running the script will move all directories with greater than 1 file below into your new directory:
$ sh mvscript.sh
'd2' -> './OOOO3_MORE_THAN_ONE/d2'
'd3' -> './OOOO3_MORE_THAN_ONE/d3'
$ tree
.
├── OOOO3_MORE_THAN_ONE
│   ├── d2
│   │   ├── file1
│   │   └── file2
│   └── d3
│   ├── file1
│   ├── file2
│   └── file3
├── d1
│   └── file1
└── mvscript.sh
Your Second Approach
Your second approach is not bad, but unless you have some special requirements and need to match dot-files, you may want to adjust GLOBIGNORE instead of setting dotglob as specified in man bash under the Pathname Expansion section. Also note there is no space between the #! and /bin/bash on the first line.
A basic tweak of your second attempt could be:
#!/bin/bash
destdir='subset'
mkdir -p -- "$destdir" || exit 1
for dir in * ; do
[[ -L $dir ]] && continue # Skip symbolic links
[[ -d $dir ]] || continue # Skip non-directories
[[ $dir -ef $destdir ]] && continue # Skip the destination dir.
numfiles=$(find "$dir" -type f -printf ".\n" | wc -l)
(( numfiles > 1 )) && mv -v -- "$dir" "$destdir"
done
Example Use/Output
A similar test would result in:
$ bash mvscript2.sh
'd2' -> 'subset/d2'
'd3' -> 'subset/d3'
$ tree
.
├── d1
│   └── file1
├── mvscript2.sh
└── subset
├── d2
│   ├── file1
│   └── file2
└── d3
├── file1
├── file2
└── file3

Related

Complete files in a different directory in zsh

Does the compadd command for ZSH not support completion when some characters are entered?
I have a executable file called 'index_for_test.js',and i add a shell script to .zshrc.
$PATH:
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Users/hanqing/Development/compadd-test
the index_for_test.js at the root of /Users/hanqing/Development/compadd-test
./
├── dir1
│ ├── a.js
│ └── b.js
├── dir2
│ ├── a.ts
│ └── b.ts
└── index_for_test.js
total 24
drwxr-xr-x 6 hanqing staff 192 11 4 13:45 .
drwxr-xr-x 10 hanqing staff 320 11 4 13:42 ..
-rw-r--r--# 1 hanqing staff 6148 11 4 14:04 .DS_Store
drwxr-xr-x 4 hanqing staff 128 11 4 13:43 dir1
drwxr-xr-x 4 hanqing staff 128 11 4 13:43 dir2
-rwxr-xr-x 1 hanqing staff 155 11 4 13:50 index_for_test.js
// index_for_test.js
#! /usr/bin/env node
const fs =require('fs')
const path=require('path')
const files=fs.readdirSync(path.join(process.cwd()))
console.log(files.join('\n'))
the script be added to .zshrc :
_index_for_test_completion() {
local abc=(`index_for_test.js`)
echo '\nabc:\n'
echo $abc'\n'
compadd -- $abc
}
compdef _index_for_test_completion index_for_test.js
When input index_for_test.js followed by input a space then press tab, it work find.But when input index_for_test.js ../ then press tab,it does not show completion list, even if compadd accept the arguments.
Image:
normal: index_for_test.js
error: index_for_test.js ../
Expect
If this is my mistake,please let me know the reason,thanks.
In addition,if the behavior is right,I wonder that how to achieve completion like cd command;
behavior of cd
index_for_test.js only prints names of files in the current directory. A file name in the current directory cannot start with ../, so there is no completion starting with ../.
If you want to complete files in a directory, you need to pass this directory to your completion script, and have it work in that directory.
In addition, you should complete directory names inside zsh, if all directories may potentially contain interesting files. If you want to allow only certain directories, have your script complete directories.
In addition, your script is broken when file names contain whitespace. Use a null byte as the separator: file names can't contain null bytes.
Untested code. May need some tweaking.
#!/usr/bin/env node
const fs = require('fs');
const path = require('path');
const files = fs.readdirSync(process.argv[2]);
console.log(files.join('\0'))
_index_for_test_completion() {
local dir=$words[CURRENT] files
if [[ ! -d $dir ]]; then dir=$dir:h; fi
files=(${(ps:\0:)$(index_for_test.js $dir))
print -lr '' abc: $files
if [[ $dir != $words[CURRENT] ]]; then dir+="/"; fi
compadd -- $^dir$files
}
compdef _index_for_test_completion index_for_test.js

Linux: Recursively find all .txt files that don't have a matching .tif

I am using Debian Linux. I'm a newbie. I'll do my best to ask in the simplest way I know.
I have a pretty deep tree of directories on a drive that contain thousands of .tif files and .txt files. I'd like to recursively find (list) all .txt files that do not have a matching .tif file (basename). The .tif files and .txt files are also located in separate directories throughout the tree.
In simple form it could look like this...
directory1: hf-770.tif, hf-771.tif, hf-772.tif
directory2: hf-770.txt, hf-771.txt, hf-771.txt, hr-001.txt, tb-789.txt
I need to find (list) hr-001.txt and tb-789.txt as they do not have a matching .tif file. Again the directory tree is quite deep with multiple sub-directories throughout.
I researched and experimented with variations of the following commands but cannot seem to make it work. Thank you so much.
find -name "*.tif" -name "*.txt" | ls -1 | sed 's/\([^.]*\).*/\1/' | uniq
You can write a shell script for this:
#!/bin/bash
set -ue
while IFS= read -r -d '' txt
do
tif=$(basename "$txt" | sed s/\.txt$/.tif/)
found=$(find . -name "$tif")
if [ -z "$found" ]
then
echo "$txt has no tif"
fi
done < <(find . -name \*.txt -print0)
This has a loop over all .txt files it finds in the current directory or below. For each found file, it replaces the .txt extension with .tif, then tries to find that file. If it cannot find it (returned text is empty), it prints the .txt file name.
robert#saaz:$ tree
.
├── bar
│   └── a.txt
├── foo
│   ├── a.tif
│   ├── b.tif
│   ├── c.tif
│   └── d.txt
└── txt-without-tif
2 directories, 6 files
robert#saaz:$ bash txt-without-tif
./foo/d.txt has no tif

Use diff and ignore empty directories

This is my tree
├── test
│   ├── dir1
│   └── dir2
│   ├── file
│   └── file2
└── test2
└── dir2
├── file
└── file2
I use diff: diff -r test/ test2/
Only in test: dir1
So the only difference is that there is an empty directory (dir1) in in test/ which does not exist in test2/.
I want to ignore empty directories as a difference. So I want in this case that diff tells me that the content of test/ is the same as the content of test2/
How can I achieve this?
I found an way of doing it but i'm not really happy with it.
diff can be told to exclude files matching a pattern. Sadly the pattern only works on filename, so my solution may exclude more directories (and files too) than expected.
Here it is, fwiw :
diff $(find test -empty -type d -exec sh -c 'echo -n "-x $(basename $1) "' _ {} \;) -r test/ test2/
I added the following command
find test -empty -type d -exec sh -c 'echo -n "-x $(basename $1) "' _ {} \;)
which outputs the basename of every empty dir preceding by diff's exclude option -x. With your example tree, it would output : -x dir1

How can I tell Perl's prove utility to ignore certain tests?

Currently I run prove like this:
prove -v -r .
Is there a way to exclude files using regex? I could not see that in perldoc prove or the output of prove -H.
I usually do something like this:
$ find t/ -name '*.t' ! -name '*timeout*' | xargs prove -l
(skips slow timeout tests)
I managed to do this by using a sub-directory :
$ tree t/
t/
├── 01_tests.t
├── 02_tests.t
├── 03_tests.t
└── ignored
└── 01_ignored.t
Then I do this to execute the normal tests:
$ prove -v t/*.t
t/01_tests.t ........ ok
t/02_tests.t ........ ok
t/03_tests.t ........ ok
Result: PASS
And this to execute the ignored tests in another context:
$ prove -v t/ignored/
t/ignored/01_ignored.t ........ ok
Result: PASS
Simple and efficient solution if your tests are not already stored in different sub-directories.
For making it easier to use, and document how to run the tests, I usually put those in a Makefile.

How to compare the content of a tarball with a folder

How can I compare a tar file (already compressed) of the original folder with the original folder?
First I created archive file using
tar -kzcvf directory_name.zip directory_name
Then I tried to compare using
tar -diff -vf directory_name.zip directory_name
But it didn't work.
--compare (-d) is more handy for that.
tar --compare --file=archive-file.tar
works if archive-file.tar is in the directory it was created. To compare archive-file.tar against a remote target (eg if you have moved archive-file.tar to /some/where/) use the -C parameter:
tar --compare --file=archive-file.tar -C /some/where/
If you want to see tar working, use -v without -v only errors (missing files/folders) are reported.
Tipp: This works with compressed tar.bz/ tar.gz archives, too.
It should be --diff
Try this (without the last directory_name):
tar --diff -vf directory_name.zip
The problem is that the --diff command only looks for differences on the existing files among the tar file and the folder. So, if a new file is added to the folder, the diff command does not report this.
The method of pix is way slow for large compressed tar files, because it extracts each file individually. I use the tar --diff method loking for files with different modification time and extract and diff only these. The files are extracted into a folder base.orig where base is either the top level folder of the tar file or teh given comparison folder. This results in diffs including the date of the original file.
Here is the script:
#!/bin/bash
set -o nounset
# Print usage
if [ "$#" -lt 1 ] ; then
echo 'Diff a tar (or compressed tar) file with a folder'
echo 'difftar-folder.sh <tarfile> [<folder>] [strip]'
echo default for folder is .
echo default for strip is 0.
echo 'strip must be 0 or 1.'
exit 1
fi
# Parse parameters
tarfile=$1
if [ "$#" -ge 2 ] ; then
folder=$2
else
folder=.
fi
if [ "$#" -ge 3 ] ; then
strip=$3
else
strip=0
fi
# Get path prefix if --strip is used
if [ "$strip" -gt 0 ] ; then
prefix=`tar -t -f $tarfile | head -1`
else
prefix=
fi
# Original folder
if [ "$strip" -gt 0 ] ; then
orig=${prefix%/}.orig
elif [ "$folder" = "." ] ; then
orig=${tarfile##*/}
orig=./${orig%%.tar*}.orig
elif [ "$folder" = "" ] ; then
orig=${tarfile##*/}
orig=${orig%%.tar*}.orig
else
orig=$folder.orig
fi
echo $orig
mkdir -p "$orig"
# Make sure tar uses english output (for Mod time differs)
export LC_ALL=C
# Search all files with a deviating modification time using tar --diff
tar --diff -a -f "$tarfile" --strip $strip --directory "$folder" | grep "Mod time differs" | while read -r file ; do
# Substitute ': Mod time differs' with nothing
file=${file/: Mod time differs/}
# Check if file exists
if [ -f "$folder/$file" ] ; then
# Extract original file
tar -x -a -f "$tarfile" --strip $strip --directory "$orig" "$prefix$file"
# Compute diff
diff -u "$orig/$file" "$folder/$file"
fi
done
To ignore differences in some or all of the metadata (user, time, permissions), you can pipe the result to awk:
tar --compare --file=archive-file.tar -C /some/where/ | awk '!/Mode/ && !/Uid/ && !/Gid/ && !/time/'
That should output only the true differences between the tar and the directory /some/where/
I recently needed a better compare than what "tar --diff" produced so I made this short script:
#!/bin/bash
tar tf "$1" | while read ; do
if [ "${REPLY%/}" = "$REPLY" ] ; then
tar xOf "$1" "$REPLY" | diff -u - "$REPLY"
fi
done
The easy way is to write:
tar df file This compares the file with the current working directory, and tell us about if any of the files has been removed.
tar df file -C path/folder This compares the file with the folder.