I want to copy all the files and sub directories from a directory to a different directory. However I only want to copy them if they are not already in the destination directory or if the timestamp on the source directory file is newer than the timestamp on the destination directory. I am having troubles getting into all of the sub directories. I am able to get down one level but not the next. For example, with directory /a/b/c, I am able to get to sub directory b but not to c. The only way I could see doing this was with a recursive function. My code is below.
#!/bin/bash
SOURCEDIR=/home/kyle/Smaug/csis252
DESTDIR=/home/kyle/Desktop/csis252
copy() {
local DIRECTORY=$1
for FILE in `ls $DIRECTORY`
do
if [ -f $DIRECTORY/$FILE ]
then
echo $FILE file
cp $DIRECTORY/$FILE $DESTDIR/$DIRECTORY/$FILE
fi
if [ -d $FILE ]
then
echo $FILE directory
mkdir $DESTDIR/$DIRECTORY/$FILE
copy $DIRECTORY/$FILE
fi
done
}
cd $SOURCEDIR
copy .
I forgot the $DIRECTORY in the second if statement. I feel stupid now but sometimes it just takes someone else to read through to find stuff like this. I did not use the cp -r $SOURCEDIR $DESTDIR because this would copy everything and sometimes I do not want that. I tried the cp -ur $SOURCEDIR $DESTDIR but it would only copy new stuff over, not update the existing stuff. The final version of my code is below.
#!/bin/bash
SOURCEDIR=/home/kyle/Smaug/csis252
DESTDIR=/home/kyle/Desktop/csis252
copy() {
local DIRECTORY=$1
for FILE in `ls $DIRECTORY`
do
if [ ! -f $DESTDIR/$DIRECTORY/$FILE ] && [ ! -d $DESTDIR/$DIRECTORY/$FILE ]
then
if [ -f $DIRECTORY/$FILE ]
then
echo "$DIRECTORY/$FILE copied"
cp $DIRECTORY/$FILE $DESTDIR/$DIRECTORY/$FILE
fi
if [ -d $DIRECTORY/$FILE ]
then
echo "$DIRECTORY/$FILE directory made"
mkdir $DESTDIR/$DIRECTORY/$FILE
copy $DIRECTORY/$FILE
fi
else
if [ $DESTDIR/$DIRECTORY/$FILE -nt $DIRECTORY/$FILE ] && [ ! -d $DIRECTORY/$FILE ]
then
cp $DIRECTORY/$FILE $DESTDIR/$DIRECTORY/$FILE
echo "$DIRECTORY/$FILE updated"
fi
fi
done
}
cd $SOURCEDIR
copy .
Simpler would be
cp -ur $SOURCEDIR $DESTDIR
-r recursivly copies folders and subfolders
-u updates, copies only when the source is newer
if [ -f $DIRECTORY/$FILE ]
...
if [ -d $FILE ]
You forgot $DIRECTORY/ in your -d check. This isn't a problem for the top-level directories, because when DIRECTORY is ., [ -d dir ] and [ -d ./dir ] will always give the same result, but for subdirectories it does matter.
Note: you may want to look at pre-written programs that do this. cp (at least the GNU version) or rsync can probably avoid the need for any custom script, and also handle special files (special file name characters, or special file types) better than any script will.
Related
We are using open source Talend studio and we have more then 50 jobs.
Each build generate zip file contains all it's artifacts ( .bat .sh context, jar files)
Is there a way to generate multiple build process from the studio or command line ( Talend open source tool )
In the "build job" window, there is a double arrow in the left,
Click on it, and you get the job tree, select all jobs or what you want, and you will get a single zip file containing all your jobs each one in a separate folder.
Not an ideal solution but you can use a small script to split the whole zip into separate job zips:
ZIP=test.zip # path to your all-in-one zip file
ROOT=$(basename $ZIP .zip)
DEST=./dest
rm -rf $DEST # be careful with this one!
mkdir -p $DEST
unzip $ZIP
find $ROOT -mindepth 1 -maxdepth 1 -type d ! -name lib|while read JOBPATH
do
JOB=$(basename $JOBPATH)
echo "job: $JOB"
DJOB="$DEST/$JOB"
mkdir -p "$DJOB"
cp -R "$JOBPATH" "$DJOB/$JOB"
cp $ROOT/jobInfo.properties $DJOB # here you should replace job=<proper job name> and jobId, but not sure you really need it
mkdir -p "$DJOB/lib"
RUNFILE="${JOBPATH}/${JOB}_run.sh"
LIBS=$(grep "^java" "$RUNFILE"|cut -d' ' -f 5)
IFS=':' read -ra ALIB <<< "$LIBS"
for LIB in "${ALIB[#]}"; do
if [ "$LIB" = "." -o "$LIB" = "\$ROOT_PATH" ]; then continue; fi
echo "$LIB"
done|grep "\$ROOT_PATH/../lib"|cut -b 19-|while read DEP
do
cp "$ROOT/lib/$DEP" "$DJOB/lib/"
done
(cd $DJOB ; zip -r -m ../$JOB.zip .)
rmdir $DJOB
done
I want to do the following in Shell Script:
I have two directories and I want to copy over only these files from the source directory that have changed or are missing in the destination directory.
Note 0: (in case the above is clear): I want to replace destination folder with source directory. Changed files in destination should be restored to the original state from the source directory. Unchanged files should not be touched (to save time).
Note 1: I looked up rsynch but I cannot figure out if it can do what I want. There is only -u but that doesn't seem to do what I want.
EDIT: This is my script: (fixed)
#!/bin/sh
################################################################################
# Copies resources to build output directory (Ubuntu)
################################################################################
directoryWtSource="/usr/share/Wt"
directoryBuildOutput="bin__output__"
# Get the absolute path to the script
scriptPath=`readlink -e $0`
directoryDestination=`dirname $scriptPath`
################################################################################
# Start
################################################################################
cd ..
directoryWtSourceResources=$directoryWtSource/resources
directoryBuildOutputResources=${PWD}/$directoryBuildOutput
if [ ! -d ${PWD}/$directoryBuildOutput ]
then
mkdir ${PWD}/$directoryBuildOutput
fi
#echo $directoryWtSourceResources
#echo $directoryBuildOutputResources
# BEGIN - REMOVE THIS: No need for cp
#if [ ! -d $directoryBuildOutputResources ]
#then
# echo " --> No destination directory found: copying resources..."
# cp -Rp $directoryWtSourceResources $directoryBuildOutputResources
#else
# echo " --> Destination directory found: won't copy."
#fi
# END - REMOVE THIS
rsync -arv $directoryWtSourceResources $directoryBuildOutputResources
################################################################################
# End
################################################################################
rsync can do this:
rsync -av source/ destination/
This command will also print out the list of files that have been replaced.
I'm attempting to write a script in the Bourne shell that will do the following:
Read in a filename
If the file does not exist in the target directory, it will display a message to the user stating such
If the file exists in the target directory, it will be moved to a /trash folder
If the file exists in the target directory, but a file of the same name is in the /trash folder, it will still move the file to the /trash directory, but will attach a _bak extention to the file.
My use of the Bourne shell is minimal, so here's what I have so far. Any pointers or tips would be greatly appreciated, thanks!
#!/bin/sh
#Scriptname: Trash Utility
source_dir=~/p6_tmp
target_dir=~/trash
echo "Please enter the filename you wish to trash:"
read filename
if [ -f $source_dir $filename]
then mv "$filename" "$target_dir"
else
echo "$filename does not exist"
fi
You cannot use ~ to refer to $HOME in a sh script. Switch to $HOME (or change the shebang to a shell which supports this, such as #!/bin/bash).
To refer to a file in a directory, join them with a slash:
if [ -f "$source_dir/$filename" ]
Notice also the required space before the terminating ] token.
To actually move the file you tested for, use the same expression for the source argument to mv:
mv "$source_dir/$filename" "$target_dir"
As a general design, a script which takes a command-line parameter is much easier to integrate into future scripts than one wich does interactive prompting. Most modern shells offer file name completion and history mechanisms, so a noninteractive script also tends to be more usable (you practically never need to transcribe a file name manually).
A Bash Solution:
#!/bin/bash
source_dir="~/p6_tmp"
target_dir="~/trash"
echo "Please enter the filename you wish to trash:"
read filename
if [ -f ${source_dir}/${filename} ]
then
if [ -f ${target_dir}/${filename} ]
then
mv "${source_dir}/${filename}" "${target_dir}/${filename}_bak"
else
mv "${source_dir}/${filename}" "$target_dir"
fi
else
echo "The file ${source_dir}/${filename} does not exist"
fi
Here's the completed script. Thanks again to all who helped!
#!/bin/sh
#Scriptname: Trash Utility
#Description: This script will allow the user to enter a filename they wish to send to the trash folder.
source_dir=~/p6_tmp
target_dir=~/trash
echo "Please enter the file you wish to trash:"
read filename
if [ -f "$source_dir/$filename" ]
then
if [ -f "$target_dir/$filename" ]
then mv "$source_dir/$filename" "$target_dir/$(basename "$filename")_bak"
date "+%Y-%m-%d %T - Trash renamed ~/$(basename "$source_dir")/$filename to ~/$(basename "/$target_dir")/$(basename "$filename")_bak" >> .trashlog
else mv "$source_dir/$filename" "$target_dir"
date "+%Y-%m-%d %T - Trash moved ~/$(basename "/$source_dir")/$filename to ~/$(basename "/$target_dir")/$filename" >> .trashlog
fi
else
date "+%Y-%m-%d %T - Trash of ~/$(basename "/$source_dir")/$filename does not exist" >> .trashlog
fi
this is my script where it is searching all the log files and zipping it alongwith deleting the older archive.
However, when i run this script i get the following error :
./file.sh: test: unknown operator .
Code:
#Directory of archives
archive_dr="/u01/apps/weblogic/weblogic10/user_projects/archive"
#Directory of log files
logdir="/u01/apps/weblogic/weblogic10/user_projects/domains/BPM/servers/BPM_MS1/logs"
cd $archive_dr
#Removing older archived files
if [ find . \( -name '*.log0*.gz' -o \
-name '*.out0*.gz' \) ]
then
rm *.out00*.gz *.log00*.gz
fi
cd $logdir
#Search,zip and move the new archive files
if [ find . \( -name '*.log0*' -o -name '*.out0*' \) \
-atime +30 ]
then
for log_files in `find . \( \
-name '*.log0*' -o -name '*.out0*' \
\) -atime +30`
do
gzip $log_files
mv $log_files.gz /u01/a*/w*/w*/us*/archive
done
if [$? = 0]; then
echo "Logs Archieved Successfully"|
mailx -s " Logs Archieved Successfully" \
-c 'x#abc.com' y#abc.com'
fi
Please suggest where i am going wrong ?
Change:
if [ find . \( -name '*.log0*.gz' -o -name '*.out0*.gz' \) ]; then
to:
if [ "$(find . \( -name '*.log0*.gz' -o -name '*.out0*.gz' \))" ]; then
You want to run the find command and test whether it returns any output. The test command (which is what [ is an abbreviation for) doesn't execute its contents, it expects it to be an expression to test, as in if [ "$foo" = 3 ].
Note also that find recurses into subdirectories, but you rm only in the current directory. If you don't want to recurse, add the -maxdepth 1 option.
There's no need for the second if. If that find doesn't find any files, the for loop will have nothing to operate on and will just terminate immediately.
Sorry, not able to edit the post properly. But, got it running , :)
CODE
#Directory of archives
archive_dr="/u01/apps/weblogic/weblogic10/user_projects/archive"
#Directory of log files
logdir="/u01/apps/weblogic/weblogic10/user_projects/domains"
cd $archive_dr
#Removing older archived files
find . \( -name '*.log00*.gz' -o -name '*.out00*.gz' \) -exec rm {} \;
cd $logdir
#Search,zip and move the new archive files
for log_files in `find . \( -name '*.log0*' -o -name '*.out0*' \) -ctime +5`
do
gzip $log_files
mv $log_files.gz /u01/a*/w*/w*/us*/archive
done
if [ $? = 0 ]; then
echo "text"|mailx -s "test" -c abc#def.com' mno#pqr.com'
fi
How can I compare a tar file (already compressed) of the original folder with the original folder?
First I created archive file using
tar -kzcvf directory_name.zip directory_name
Then I tried to compare using
tar -diff -vf directory_name.zip directory_name
But it didn't work.
--compare (-d) is more handy for that.
tar --compare --file=archive-file.tar
works if archive-file.tar is in the directory it was created. To compare archive-file.tar against a remote target (eg if you have moved archive-file.tar to /some/where/) use the -C parameter:
tar --compare --file=archive-file.tar -C /some/where/
If you want to see tar working, use -v without -v only errors (missing files/folders) are reported.
Tipp: This works with compressed tar.bz/ tar.gz archives, too.
It should be --diff
Try this (without the last directory_name):
tar --diff -vf directory_name.zip
The problem is that the --diff command only looks for differences on the existing files among the tar file and the folder. So, if a new file is added to the folder, the diff command does not report this.
The method of pix is way slow for large compressed tar files, because it extracts each file individually. I use the tar --diff method loking for files with different modification time and extract and diff only these. The files are extracted into a folder base.orig where base is either the top level folder of the tar file or teh given comparison folder. This results in diffs including the date of the original file.
Here is the script:
#!/bin/bash
set -o nounset
# Print usage
if [ "$#" -lt 1 ] ; then
echo 'Diff a tar (or compressed tar) file with a folder'
echo 'difftar-folder.sh <tarfile> [<folder>] [strip]'
echo default for folder is .
echo default for strip is 0.
echo 'strip must be 0 or 1.'
exit 1
fi
# Parse parameters
tarfile=$1
if [ "$#" -ge 2 ] ; then
folder=$2
else
folder=.
fi
if [ "$#" -ge 3 ] ; then
strip=$3
else
strip=0
fi
# Get path prefix if --strip is used
if [ "$strip" -gt 0 ] ; then
prefix=`tar -t -f $tarfile | head -1`
else
prefix=
fi
# Original folder
if [ "$strip" -gt 0 ] ; then
orig=${prefix%/}.orig
elif [ "$folder" = "." ] ; then
orig=${tarfile##*/}
orig=./${orig%%.tar*}.orig
elif [ "$folder" = "" ] ; then
orig=${tarfile##*/}
orig=${orig%%.tar*}.orig
else
orig=$folder.orig
fi
echo $orig
mkdir -p "$orig"
# Make sure tar uses english output (for Mod time differs)
export LC_ALL=C
# Search all files with a deviating modification time using tar --diff
tar --diff -a -f "$tarfile" --strip $strip --directory "$folder" | grep "Mod time differs" | while read -r file ; do
# Substitute ': Mod time differs' with nothing
file=${file/: Mod time differs/}
# Check if file exists
if [ -f "$folder/$file" ] ; then
# Extract original file
tar -x -a -f "$tarfile" --strip $strip --directory "$orig" "$prefix$file"
# Compute diff
diff -u "$orig/$file" "$folder/$file"
fi
done
To ignore differences in some or all of the metadata (user, time, permissions), you can pipe the result to awk:
tar --compare --file=archive-file.tar -C /some/where/ | awk '!/Mode/ && !/Uid/ && !/Gid/ && !/time/'
That should output only the true differences between the tar and the directory /some/where/
I recently needed a better compare than what "tar --diff" produced so I made this short script:
#!/bin/bash
tar tf "$1" | while read ; do
if [ "${REPLY%/}" = "$REPLY" ] ; then
tar xOf "$1" "$REPLY" | diff -u - "$REPLY"
fi
done
The easy way is to write:
tar df file This compares the file with the current working directory, and tell us about if any of the files has been removed.
tar df file -C path/folder This compares the file with the folder.