We have a lot of files from 1992 through 2015 dumped into one huge folder which we need to zip up and delete the original files.
The files have yyyymm in their name.
Here is the script I wrote
#!/usr/bin/ksh
MYHOME=/home/myhome
MYARCHIVE=$MYHOME/data/ARCHIVE
MOVED=$MYARCHIVE/MONTHLY_MOVED
TEMPFOLDER=$MYARCHIVE/TempFolder
. $MYHOME/bin/FILEWATCHERS/Profile.ksh
mkdir -p $MOVED
for y in 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015
do
for m in 01 02 03 04 05 06 07 08 09 10 11 12
do
echo "Year $y month $m"
mkdir -p $TEMPFOLDER
find $MYARCHIVE/. -name . -o -type d -prune -o -name '*$y$m*' -print | xargs -i mv {} $TEMPFOLDER/.
cd $TEMPFOLDER
count=`ls -1 | wc -l`
echo "Count is $count"
if [ "$count" -ne 0 ]
then
tar cvf $MOVED/Monthly_Files_$y$m.tar .
fi
cd ..
rm -rf $TEMPFOLDER
done
done
The script is not working even then there are 20000 files with file pattern 201512.
Separately on the shell (ksh and csh) this command works
find /home/myhome/data/ARCHIVE/. -name . -o -type d -prune -o -name '*201512*' -print | xargs -i mv {} /home/myhome/ARCHIVE/TempFolder/.
what is missing in the script?
I think the problem lie in this line
find $MYARCHIVE/. -name . -o -type d -prune -o -name '*$y$m*' -print | xargs -i mv {} $TEMPFOLDER/.
the shell variable is not getting substituted with the values '*$y$m*' instead of single quotes use double quotes "*$y$m*" then it should work.
Single Quote - Enclosing characters in single quotes (‘'’) preserves the literal value of each character within the quotes. A single quote may not occur between single quotes, even when preceded by a backslash.
Double Quotes - Enclosing characters in double quotes (‘"’) preserves the literal value of all characters within the quotes, with the exception of ‘$’, ‘’, ‘\’, and, when history expansion is enabled, ‘!’. The characters ‘$’ and ‘’ retain their special meaning within double quotes. The backslash retains its special meaning only when followed by one of the following characters: ‘$’, ‘`’, ‘"’, ‘\’, or newline. Within double quotes, backslashes that are followed by one of these characters are removed. Backslashes preceding characters without a special meaning are left unmodified. A double quote may be quoted within double quotes by preceding it with a backslash. If enabled, history expansion will be performed unless an ‘!’ appearing in double quotes is escaped using a backslash. The backslash preceding the ‘!’ is not removed.
The special parameters ‘*’ and ‘#’ have special meaning when in double quotes.
The statement should be as below
find $MYARCHIVE/. -name . -o -type d -prune -o -name "*$y$m*" -print | xargs -i mv {} $TEMPFOLDER/.
Related
Can someone please give me the exact syntax for removing ^# from thousands of html files in nested directories using sed? The ^# is a control character inserted by a windows program that generated these files. I cannot seem to get the syntax right.
I tried this (but it did not work) using a file since I could not enter the control-character at the command prompt:
find ./ *.html -type f -exec sed -i 's/^#//g' {} ;
POSIX sed doesn't handle NUL in input but GNU sed can with hex escape:
find . -name '*.html' -type f -exec sed -i 's/\x0//g' '{}' +
I am now working with "Jenkinsfile".
I need to do a "find" by type of the file extension, to do a "sed -i", ignoring some hidden directories and other folders.
I don't know the correct syntax.
Example:
def replacePath() {
sh 'sed -i "s/A\\/B/C\\/D\\/E\\/F\\/G\\/A\\/B\\/opt\\/C/g" \$(find . -type f -name "*.json" not path ..... -print0) '
Try using xargs, like so:
find . -type f -name '*.json' ... -print0 | xargs -0 sed -i 's/pattern/replacement/g'
Using xargs has fewer problems than passing argument on the command line with $(...), particularly when used with -print0, as xargs can cope with filenames containing shell metacharacters.
I have a copyright clause I want to append to the top of a bunch of files in a directory as a comment (C#). It is to look like this:
/*COPYRIGHTCOPYRIGHTCOPYRIGHTCOPYRIGHTCOPYRIGHTCOPYRIGHTCOPYRIGHT*/
/*C Use, duplication, or disclosure of this software and T*/
/*C related documentation is blah blah blah blah T*/
/*C Copyright 2004 - 2018 COMPANY T*/
/*C All rights reserved. T*/
/*COPYRIGHTCOPYRIGHTCOPYRIGHTCOPYRIGHTCOPYRIGHTCOPYRIGHTCOPYRIGHT*/
Some of the files contain a clause already, but with the 2017 or 2016 date, I want to make sure the date in the clause is set to 2018. And if there isn't any clause, I want to insert one.
So far I have implemented find to only modify the files I need in the directory:
find . -type d \( -name ThirdParty -o -name 3rdParty -o -name 3rd_party \) -prune -o -type f \( -name "*.java" -o -name "*.cs" -o -name "*.cpp" -o -name "*.cxx" -o -name "*.cc" -o -name "*.c" -o -name "*.h" -o -name "*.scala" -o -name "*.css" -o -name "*.js" \) -print0
Because this clause is to be a comment at the top of each file, it starts and ends with /*...*/. I tried using sed but it is tricky since the slashes and asterisks are used for other things in sed.
Here is my find and sed combined. This replaces all instances of 2017 with 2018 and appends the clause but it appends it even if its already there. I only need it to append if it's not there.
find ....... -print0 | xargs -0 sed -i 's/2004 - 2017/2004 - 2018/g; 1s/^/\/*COPYRIGHTCOPYRIGHTCOPYRIGHTCOPYRIGHTCOPYRIGHTCOPYRIGHTCOPYRIGHT*\/\n\/*C Use, duplication, or disclosure of this software and.....etc'
Is there a better way of doing this in grep or sed? Thanks
EDIT: I AM USING CYGWIN
Please try:
find ... -print0 | while read -r -d $'\0' f; do
if [[ $(head -1 "$f") = "/*COPYRIGHTCOPYRIGHTCOPYRIGHTCOPYRIGHTCOPYRIGHTCOPYRIGHTCOPYRIGHT*/" ]]; then
sed -i 's/2004 - 2017/2004 - 2018/g' "$f"
else
sed -i '1s#^#/*COPYRIGHTCOPYRIGHTCOPYRIGHTCOPYRIGHTCOPYRIGHTCOPYRIGHTCOPYRIGHT*/\n/*C Use, duplication, or disclosure of this software and.....etc */\n#' "$f"
fi
done
It passes the result of find not to xargs but to the while loop and switches the processing depending of the first line of the file.
grep is for g/re/p and sed is for s/old/new on individual lines so neither tool is appropriate for your problem.
This should do what you want using GNU awk (which you have on cygwin) for inplace editing:
find ... -exec awk -i inplace '
FNR==1 {
print "/*COPYRIGHTCOPYRIGHTCOPYRIGHTCOPYRIGHTCOPYRIGHTCOPYRIGHTCOPYRIGHT*/"
print "/*C Use, duplication, or disclosure of this software and T*/"
print "/*C related documentation is blah blah blah blah T*/"
print "/*C Copyright 2004 - 2018 COMPANY T*/"
print "/*C All rights reserved. T*/"
print "/*COPYRIGHTCOPYRIGHTCOPYRIGHTCOPYRIGHTCOPYRIGHTCOPYRIGHTCOPYRIGHT*/"
if ( /COPYRIGHT/ ) { inCopy=1 }
next
}
inCopy && /COPYRIGHT/ { inCopy=0 }
!inCopy
' {} +
If an old copyright block is already present at the start of the file it just replaces the whole block with the new one. You can change each /COPYRIGHT/ to index($0,"/*COPYRIGHTCOPYRIGHTCOPYRIGHTCOPYRIGHTCOPYRIGHTCOPYRIGHTCOPYRIGHT*/") if you want more precise matching on that text to indicate the start and end of the comment block.
I want to view for the files in /var/www the following data:
fullpath size mtime ctime md5
I ran the following command:
find /var/www/ -maxdepth 1 ! -type d -printf '%p\t%s\t%t\t%c\t' -exec md5sum {} \;
which gives me:
(fullpath size mtime ctime md5 fullpath)
/var/www/intranet/admin/tpl/view.tpl.php 1448 Wed Dec 16 18:51:06.0000000000 2015 Fri Sep 15 09:08:36.0805775786 2017 e0b7dacaf7c90fb0fbe7a69c331e36aa /var/www/intranet/admin/tpl/view.tpl.php
How can I filter the last fullpath?????? I do not want to show it. All fields are TAB separated.
I tried:
find /var/www/ -maxdepth 1 ! -type d -printf '%p\t%s\t%t\t%c\t'
-exec md5sum {} | awk '{print $1}'\;
for which I received the error: "find: missing argument to `-exec'"
find /var/www/ -maxdepth 1 ! -type d -printf '%p\t%s\t%t\t%c\t' -exec md5sum {} + | awk '{print $1}'
for which I got only the md5sum.
Thanks in advance!
Pipelines (|) are a shell-feature. To get shell features, one needs to invoke a shell:
find /var/www -maxdepth 1 ! -type d -printf '%p\t%s\t%t\t%c\t' -exec sh -c 'md5sum "$1" | awk '\''{print $1}'\' MD5 {} \;
Or, if you prefer commands spread over multiple lines:
find /var/www \
-maxdepth 1 \
! -type d \
-printf '%p\t%s\t%t\t%c\t' \
-exec sh -c 'md5sum "$1" | awk '\''{print $1}'\' MD5 {} \;
Notes
sh -c somestring invokes a shell and instructs it to execute whatever commands are in somestring.
sh -c somestring MD5 {} invokes the shell and executes somestring and assigns $0 to MD5 and $1 to whatever find substitutes for {}.
$0 is only used by the shell when it creates error messages and otherwise unimportant.
A complication is that our command, somestring, must contain both single quotes and double-quotes which is why we need escaped single-quotes.
In our case, we want somestring to be:
md5sum "$1" | awk '{print $1}'
To prevent the main shell from substituting in for $1, we need to put this inside single-quotes. However, we can't put single-quotes inside single-quotes. The workaround is to use this for our single-quoted string:
'md5sum "$1" | awk '\''{print $1}'\'
Command wc has an option --files0-from=F. According to the manual, it reads input from the files specified by NUL-terminated names in file F; If F is - then read names from standard input. Why NUL-terminated names? Isn't it more convenient just separating the names with newline or space?
It's more convenient if you have filenames with spaces (or new-lines, or tabs) in them.
This is sometimes used with find -print0 that outputs its list of files with \0 as a separator instead of spaces.
$ find . -type f -print0 | wc -c --files0-from=-
15 ./c d
12 ./a b
27 total
xargs has a -0 option for similar reasons.