I would like to delete all files in a specific directory according to the creation (or modification) date. I know that to accomplish this task there is the command:
find /tmp/log -maxdepth 1 -name 'file*' -mmin +60 -type f -exec rm {} \;
The problem is that I'm using an OpenWrt system versione that doesn't support the flag -mtime (either -mmin). So is there any alternative to delete files according to the date?
OpenWrt seems to replace most of the Linux commands with BusyBox versions (ouch) (see https://openwrt.org/docs/techref/busybox)
According to BusyBox documentation, the current version does support -mtime, but seemingly your version does not? (I'd try it again, to make sure you didn't goof up the command line)
Anyway, how to get this done using BusyBox commands?
Install a new version of BusyBox?
or Install the "real" version of the find command
or if this command date -d '#946710000' prints a date ~ 1/1/2000, try the following (not tested on BusyBox !!):
cur=`date "+%s"` # current timestamp
old=`echo $cur|awk '{print $1 - (3600*10)}'` # 10 days ago
for f in *;do
date -r "$f" "+%s^$f" # timestamp of named file
done |
awk -F'^' -v "old=$old" '$1<old{print $2}' # older?
(assumes no ^ in filenames)
Related
I am very new to Clearcase and one of the task that I have got on my hand is to find frequently modified files in ClearCase, Suppose we have an integration stream and there are numerous files in our stream, need to know of certain files which are modified frequently like a certain file is modified 5 times in last two months.
I have access to ClearCase commands as well as GUI
Is there a way we can have solution to this problem.
Thanks
You can do, following find examples, a search between two dates:
cleartool find . -version "{created_since(date1) &&
!created_since(date2) &&
brtype(myIntStream)" -exec "cleartool descr -fmt "%En"\
|sort| uniq -c | sort -n
(This is the Windows syntax, which means you need GoW (Gnu On Windows) installed for the v and uniq commands.
As Brian Cowan adds in the comments, the command would be:
cleartool find -all -version "{created_since(date1) &&
!created_since(date2) &&
brtype(myIntStream)" -exec "cleartool desc -fmt \"%En\n\" \"%CLEARCASE_XPN\"" \
|sort| uniq -c | sort -n
On Unix:
cleartool find -all -version "{created_since(date1) &&
!created_since(date2) &&
brtype(myIntStream)" -exec 'cleartool desc -fmt "%En\n" "$CLEARCASE_XPN"' \
|sort| uniq -c | sort -n
-all instead of the current directory format, to avoid issues if the command isn't run at the VOB root.
If you don't care about the interval, but only want the last 2 months, drop the !created_since line.
Alternatively, use "today" as the second date, though that would everything modified since midnight your local on the day you run the command.
I have a bucket in google cloud storage. I have a tmp folder in bucket. Thousands of files are being created each day in this directory. I want to delete files that are older than 1 day every night. I could not find an argument on gsutil for this job. I had to use a classic and simple shell script to do this. But the files are deleting very slowly.
I have 650K files accumulated in the folder. 540K of them must be deleted. But my own shell script worked for 1 day and only 34K files could be deleted.
The gsutil lifecycle feature is not able to do exactly what I want. He's cleaning the whole bucket. I just want to delete the files regularly at the bottom of certain folder.. At the same time I want to do deletion faster.
I'm open to your suggestions and your help. Can I do this with a single gsutil command? or a different method?
simple script I created for testing (I prepared to delete bulk files temporarily.)
## step 1 - I pull the files together with the date format and save them to the file list1.txt.
gsutil -m ls -la gs://mygooglecloudstorage/tmp/ | awk '{print $2,$3}' > /tmp/gsutil-tmp-files/list1.txt
## step 2 - I filter the information saved in the file list1.txt. Based on the current date, I save the old dated files to file list2.txt.
cat /tmp/gsutil-tmp-files/list1.txt | awk -F "T" '{print $1,$2,$3}' | awk '{print $1,$3}' | awk -F "#" '{print $1}' |grep -v `date +%F` |sort -bnr > /tmp/gsutil-tmp-files/list2.txt
## step 3 - After the above process, I add the gsutil delete command to the first line and convert it into a shell script.
cat /tmp/gsutil-tmp-files/list2.txt | awk '{$1 = "/root/google-cloud-sdk/bin/gsutil -m rm -r "; print}' > /tmp/gsutil-tmp-files/remove-old-files.sh
## step 4 - I'm set the script permissions and delete old lists.
chmod 755 /tmp/gsutil-tmp-files/remove-old-files.sh
rm -rf /tmp/gsutil-tmp-files/list1.txt /tmp/gsutil-tmp-files/list2.txt
## step 5 - I run the shell script and I destroy it after it is done.
/bin/sh /tmp/gsutil-tmp-files/remove-old-files.sh
rm -rf /tmp/gsutil-tmp-files/remove-old-files.sh
There is a very simple way to do this, for example:
gsutil -m ls -l gs://bucket-name/ | grep 2017-06-23 | grep .jpg | awk '{print $3}' | gsutil -m rm -I
There isn't a simple way to do this with gsutil or object lifecycle management as of today.
That being said, would it be feasible for you to change the naming format for the objects in your bucket? That is, instead of uploading them all under "gs://mybucket/tmp/", you could append the current date to that prefix, resulting in something like "gs://mybucket/tmp/2017-12-27/". The main advantages to this would be:
Not having to do a date comparison for every object; you could run gsutil ls "gs://mybucket/tmp/" | grep "gs://[^/]\+/tmp/[0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}/$" to find those prefixes, then do date comparisons on the last portion of of those paths.
Being able to supply a smaller number of arguments on the command line (prefixes, rather than the name of each individual file) to gsutil -m rm -r, thus being less likely to pass in more arguments than your shell can handle.
I need to rename all files whose name contains the substring 200at and substitute it with 200_at.
In Ubuntu, I would do:
find . -type f -name '*200at*' -exec rename -n 's/200at/200_at/' {} \;
In CentOS (version 5.9) it doesn't work. The command rename doesn't seem to support perl expressions and the above command does nothing at all.
Any ideas for an alternative?
In my experience with CentOS, we always used the mv (move) command for renaming things. ie:
mv 200at ./200_at
hope that works as your alternative.
How can i delete all the file that are ending with *0x0.jpg in CENTOS ? I need to delete multiple files nested into folders and subfolders
I assume you have a shell - try
find /mydirectory -type f -print | grep '0x0.jpg$' | xargs -n1 rm -f
There is probable a more elegant solution but that should work
However I would put an echo in before rm on the first run to ensure that the right files are going to be removed.
Ed Heal's answer works just fine but neither the grep nor xargs calls are necessary. The following should work just as well and be a good bit more efficient for large amounts of files.
find /mydirectory -name '*0x0.jpg' -type f -exec rm -rf () \+
I suppose I could compare the number of files in the source directory to the number of files in the target directory as cp progresses, or perhaps do it with folder size instead? I tried to find examples, but all bash progress bars seem to be written for copying single files. I want to copy a bunch of files (or a directory, if the former is not possible).
You can also use rsync instead of cp like this:
rsync -Pa source destination
Which will give you a progress bar and estimated time of completion. Very handy.
To show a progress bar while doing a recursive copy of files & folders & subfolders (including links and file attributes), you can use gcp (easily installed in Ubuntu and Debian by running "sudo apt-get install gcp"):
gcp -rf SRC DEST
Here is the typical output while copying a large folder of files:
Copying 1.33 GiB 73% |##################### | 230.19 M/s ETA: 00:00:07
Notice that it shows just one progress bar for the whole operation, whereas if you want a single progress bar per file, you can use rsync:
rsync -ah --progress SRC DEST
You may have a look at the tool vcp. Thats a simple copy tool with two progress bars: One for the current file, and one for overall.
EDIT
Here is the link to the sources: http://members.iinet.net.au/~lynx/vcp/
Manpage can be found here: http://linux.die.net/man/1/vcp
Most distributions have a package for it.
Here another solution: Use the tool bar
You could invoke it like this:
#!/bin/bash
filesize=$(du -sb ${1} | awk '{ print $1 }')
tar -cf - -C ${1} ./ | bar --size ${filesize} | tar -xf - -C ${2}
You have to go the way over tar, and it will be inaccurate on small files. Also you must take care that the target directory exists. But it is a way.
My preferred option is Advanced Copy, as it uses the original cp source files.
$ wget http://ftp.gnu.org/gnu/coreutils/coreutils-8.21.tar.xz
$ tar xvJf coreutils-8.21.tar.xz
$ cd coreutils-8.21/
$ wget --no-check-certificate wget https://raw.githubusercontent.com/jarun/advcpmv/master/advcpmv-0.8-8.32.patch
$ patch -p1 -i advcpmv-0.8-8.32.patch
$ ./configure
$ make
The new programs are now located in src/cp and src/mv. You may choose to replace your existing commands:
$ sudo cp src/cp /usr/local/bin/cp
$ sudo cp src/mv /usr/local/bin/mv
Then you can use cp as usual, or specify -g to show the progress bar:
$ cp -g src dest
A simple unix way is to go to the destination directory and do watch -n 5 du -s . Perhaps make it more pretty by showing as a bar . This can help in environments where you have just the standard unix utils and no scope of installing additional files . du-sh is the key , watch is to just do every 5 seconds.
Pros : Works on any unix system Cons : No Progress Bar
To add another option, you can use cpv. It uses pv to imitate the usage of cp.
It works like pv but you can use it to recursively copy directories
You can get it here
There's a tool pv to do this exact thing: http://www.ivarch.com/programs/pv.shtml
There's a ubuntu version in apt
How about something like
find . -type f | pv -s $(find . -type f | wc -c) | xargs -i cp {} --parents /DEST/$(dirname {})
It finds all the files in the current directory, pipes that through PV while giving PV an estimated size so the progress meter works and then piping that to a CP command with the --parents flag so the DEST path matches the SRC path.
One problem I have yet to overcome is that if you issue this command
find /home/user/test -type f | pv -s $(find . -type f | wc -c) | xargs -i cp {} --parents /www/test/$(dirname {})
the destination path becomes /www/test/home/user/test/....FILES... and I am unsure how to tell the command to get rid of the '/home/user/test' part. That why I have to run it from inside the SRC directory.
Check the source code for progress_bar in the below git repository of mine
https://github.com/Kiran-Bose/supreme
Also try custom bash script package supreme to verify how progress bar work with cp and mv comands
Functionality overview
(1)Open Apps
----Firefox
----Calculator
----Settings
(2)Manage Files
----Search
----Navigate
----Quick access
|----Select File(s)
|----Inverse Selection
|----Make directory
|----Make file
|----Open
|----Copy
|----Move
|----Delete
|----Rename
|----Send to Device
|----Properties
(3)Manage Phone
----Move/Copy from phone
----Move/Copy to phone
----Sync folders
(4)Manage USB
----Move/Copy from USB
----Move/Copy to USB
There is command progress, https://github.com/Xfennec/progress, coreutils progress viewer.
Just run progress in another terminal to see the copy/move progress. For continuous monitoring use -M flag.