Perl script for housekeeping - perl

im trying to build a perl script for housekeenping a windows server but it's getting really dificult. so if you guys could help i apreciate.
so...this program shoul find PDF files, zip them and then delete files (outside the zip file) which are bigger than 1mb .
and i think the problem is in the IF filesize condition. so when i put 2 PDF files into a directory, (one smaller than 1mb and another larger than 1mb) , no matter how many times i changed the code this is result:
zip all files, delete the pdf files (not those in the zip file).
zip all files, delete the bigger file (not those in the zip file).
zip the bigger file, delete the pdf files (not those in the zip file).
this is my code:
#!/usr/bin/perl
#1 megabyte = 1000000 bytes
use File::Find;
use lib qw(/st/APPL/PORTABLE/Perl/5.8.8);
use MIME::Lite;
use Strict;
use warnings;
use Win32::DriveInfo;
use Archive::Zip;
use Switch;
use IO::Compress::Zip qw(zip $ZipError);
use File::stat;
#my $backup_root = "/path/to/folder"
my $backup_root = "D:/st/APPL/PORTABLE/Perl/bin/teste";
# purge backups older than AGE in days
my #file_list;
my #find_dirs = ($backup_root); # directories to search
my $now = time(); # get current time
my $days = 31; # how many days old
my $seconds_per_day = 60 * 60 * 24; # seconds in a day
my $AGE = $days * $seconds_per_day; # age in seconds
find(
sub {
my $file = $File::Find::name;
my $filesize = stat($file)->size;
if ( -f $file ) {
push( #file_list, $file );
print "Ficheiro $file encontrado!\n";
print "Size: $filesize\n";
}
if ( $filesize >= 1105593 ) {
#my #files = <*20131221*.pdf>;
my #files = <*.pdf>;
zip \#files => 'output.zip'
or die "zip failed: $ZipError\n";
for my $file (#file_list) {
my #stats = stat($file);
if ( $now - $stats[9] > $AGE ) {
unlink $file;
}
}
print "Deleted files older than $days days.\n";
} elsif ( $filesize <= 1105593 ) {
print "O ficheiro e mais pequeno que 1 mb !";
}
},
#find_dirs
);

It's not clear how output.zip connected with files in $backup_root, but I think that conceptual mistake is recreating zip file inside find callback(=inside loop).
Try following code:
#!/usr/bin/perl
# purge backups older than AGE in days
use strict;
use warnings;
use File::Find;
use lib qw(/st/APPL/PORTABLE/Perl/5.8.8);
use IO::Compress::Zip qw(zip $ZipError);
use File::stat;
my $backup_root = "D:/st/APPL/PORTABLE/Perl/bin/teste";
my #file_list;
my #find_dirs = ($backup_root); # directories to search
my $now = time(); # get current time
my $days = 31; # how many days old
my $seconds_per_day = 60 * 60 * 24; # seconds in a day
my $AGE = $days * $seconds_per_day; # age in seconds
# get list of all files and directories in #find_dirs
find( sub { push #file_list, $File::Find::name; }, #find_dirs );
# zip all pdfs found in $backup_root directory
my #found_pdfs = grep{/\.pdf$/} grep {-f} #file_list;
zip \#found_pdfs => 'output.zip' or die "zip failed: $ZipError\n";
# unlink old big files in $backup_root directory
unlink $_ for grep {$now - stat($_)->mtime > $AGE } grep {-s >= 1105593} grep {-f} #file_list;

Related

Delete old report files from PATH for x days in Perl

I have a script which should create a report on daily basis on some alarm data. These report files(.csv) will be stored in the perticular path.
Before generating the report, I need to look whether there is existance of any old files which are older than x days (passed as an argument to the script based on requirement).
Below is script to delete the old logs/reports.
...
use File::Find;
my $days = $ARGV[0]; #pass as a parameter, i.e., number of days
my $PATH = "/path/to/the/logfiles/";
print "backup path -> $PATH\n";
my #file_list;
my #find_dirs = ($PATH); # directories to search
my $now = time(); # get current time
my $seconds_per_day = 60*60*24; # seconds in a day
my $AGE = $days*$seconds_per_day; # age in seconds
find ( sub {
my $file = $File::Find::name;
if ( -f $file ) {
push (#file_list, $file)
}
}, #find_dirs);
for my $file (#file_list) {
my #stats = stat($file);
if ($now-$stats[9] > $AGE) {
unlink $file;
}
}
print "Deleted files older than $days days.\n";
#
# GENERATE REPORT CODE
#
...
...
This script works fine for me. I want to know whether we have any Standard Perl Module which will do the same operation which can act as logrotation.

Move the files older than three hours to archive folder without overwriting existing files

The below script will move the files having modification time older than three hours from /tmp/test1 folder to /data/ARCHIVE folder.
The issue is that if the ARCHIVE folder already has a file with same name as in the test1 folder it will overwrite them, which is unwanted.
#!/usr/bin/perl
use strict;
use warnings;
use File::Find;
use File::Copy;
my $dstdir = '/data/ARCHIVE/';
#ARGV = ("/tmp/test1") unless #ARGV;
print STDERR "Begin # ", scalar localtime, "\n";
find(
sub {
if ( -f $_ && -M _ >= 3 / 24 ) {
print STDERR "Moving '$_'\n";
move( $File::Find::name, $dstdir ) or die "$!\n";
}
},
#ARGV
);
print STDERR "Ended # ", scalar localtime, "\n";
1;
Add the epoch to the file name. But don't run your code more than once a second.
my $now = strftime( '%Y%m%d%H%M%S', localtime );
find(
sub {
if ( -f $_ && -M _ >= 3 / 24 ) {
print STDERR "Moving '$_'\n";
my $dst = "$dstdir/$_.$now";
move( $File::Find::name, $dst ) or die "$!\n";
}
},
#ARGV
);

Perl: Fastest way to find files older than X number of minutes, sorted oldest to newest

im trying to check if there is a file (i dont care about folders) that older than X minuts. unfortunatly i can;t tell where is my bug on this code.
i will appriciate any help :)
1. Find the files older than X number of minute
#!/usr/bin/perl
my $maindir = "C:\\Users\\Dor\\Desktop\\aba";
my $minutesold = 60;
my $now = time;
my $filedisc;
# Declare arrays
my #xmlfiles;
my #qulfiedfiles;
# Declare a Dictionary
my %filedisc;
opendir(my $dh, $maindir) or die "opendir($maindir): $!";
# Read all the files
while (my $de = readdir($dh))
{
# get the Full path of the file
my $f = $maindir . $de;
if ( -f $f )
{
push (#xmlfiles, $f);
}
}
closedir($dh);
# For every file in directory
for my $file (#xmlfiles) {
# Get stats about a file
my #stats = stat($file);
# If time stamp is older than minutes provided
if ($stats[9] >= ($now - (( $minutesold * 60) ))){
# Put the File and Time stamp in the dictionary
print($stats[9] ." .| " .$file ."\n\n");
}
#print($now ."\n")
#print($now - ( $minutesold * 60) ."\n");
}
It's usually best to use glob rather than opendir/readdir, so as to avoid having to “rebuild” the full path to the file for every result
You will probably want to enable the :bsd_glob option on Windows so that paths with spaces, such as C:\Program Files, are handled correctly
use strict;
use warnings 'all';
use File::Glob ':bsd_glob'; # Provide for spaces in path
my $root = 'C:\Users\Dor\Desktop\aba';
my $minutesold = 60;
my #old_files = grep { -f and -M * 24 * 60 > $minutes_old } glob "$root\\*.*";
The path and file isn't correct.
my $f = $maindir . $de;
Should be (add slash between path and file)
my $f = "$maindir/$de";
Solving this in kind of a functional programming style is the way to go here I think:
my $dir = shift() || $ENV{HOME}; #command line arg or else home dir
my $minutesold = 60; #1h
opendir my $dh, $dir or die "ERR: opendir($dir) $!\n";
print
map "$$_{timestamp} .| $$_{file}\n",
#sort { $$a{timestamp} <=> $$b{timestamp} } # sort by age
#sort { $$a{file} cmp $$b{file} } # sort by name
grep $^T-$$_{timestamp} >= 60*$minutesold, # $^T is program startup time()
map {{timestamp=>(stat($_))[9], file=>$_}}
grep -f $_,
map "$dir/$_",
readdir $dh;
closedir $dh;
You have missed one simple way to get the modification time of file in perl: the -M switch.
my $modifiedTimeinDays = -M "$file";
my $modifiedTimeinSec = $modifiedTimeinDays*60*60*24;
if($modifiedTimeinSec > 60)
{
# file older than 60 sec
}
As simple as that.
See perldoc -f -X to learn about all of the file tests.

Zipping files with perl, last line is cut off

I tried to zip some CSV files which are about 5 MB with Perl. Below is my zip code.
The files are zipped but when I open them with the Windows unzip utility I found that the last lines of the CSV files are missing. What could be the problem here? I tried to change the chuncksize and the desiredCompressionLevel but this didn't help.
sub zip_util{
my $directory = shift;
$zip = Archive::Zip->new();
$zip->setChunkSize(65536);
# Add a file from disk
my $file1=File::Spec->catfile($directory, 'file.csv');
my $file2=File::Spec->catfile($directory, 'file2.csv');
my $file3=File::Spec->catfile($directory, 'fil3.csv');
$zip->addFile($file1,'file1.csv')->desiredCompressionLevel( 6 );
$zip->addFile($file2,'file2.csv')->desiredCompressionLevel( 6 );
$zip->addFile($fil3,'file3.csv')->desiredCompressionLevel( 6 );
# Save the Zip file
my $zipped_file=File::Spec->catfile($directory,'files.zip');
unless ( $zip->writeToFileNamed($zipped_file) == AZ_OK ) {
print LOG ": Zip Creation error\n";
}
I've checked it with warnings and strictures and I've found certain problems.
$zip doesn't use my (I don't know if it's intentional, but use strict really helps with globals like this).
You're running $zip->addFile($fil3,'file3.csv'). $fil3 variable definitely doesn't exist. If anything, this variable is $file3.
I guess it's issue during Copy and Paste, but subroutine doesn't have matching brace.
I've done quick script which I've used during testing.
use strict;
use warnings;
use Archive::Zip qw( :ERROR_CODES );
use File::Spec;
sub zip_util {
my ($directory) = #_;
my $zip = Archive::Zip->new();
$zip->setChunkSize(65536);
# Add a file from disk
my $file1 = File::Spec->catfile( $directory, 'file.csv' );
my $file2 = File::Spec->catfile( $directory, 'file2.csv' );
my $file3 = File::Spec->catfile( $directory, 'file3.csv' );
$zip->addFile( $file1, 'file1.csv' )->desiredCompressionLevel(6);
$zip->addFile( $file2, 'file2.csv' )->desiredCompressionLevel(6);
$zip->addFile( $file3, 'file3.csv' )->desiredCompressionLevel(6);
# Save the Zip file
my $zipped_file = File::Spec->catfile( $directory, 'files.zip' );
if ( $zip->writeToFileNamed($zipped_file) != AZ_OK ) {
print STDERR "Zip Creation error\n";
}
}
zip_util '.';
The problem is that it was working. So, I've done this script to make some sort of 5MB files:
use strict;
use warnings;
use 5.010;
for ( 1 .. 524_288 ) {
my $number = $_ % 10;
say "$number-------";
}
Both files in ZIP and original "CSV" had this same size and content. So, it's probably the second issue which I've mentioned - use of $fil3 variable or something with your files (which you sadly didn't uploaded, so I cannot look into those).

Zipping files by group size in a directory

I have 2 directories that I have to open and zip all files in there. My issue is that my zip files can only be 5MB each, and these files are big. I have to some how group these files by size before zipping them. I would like to do this using Perl's Archive::Zip module. I have some code but I was wondering if someone here would know a way to do this, well here is the code:
#!/perl/bin/perl -w
use strict;
use warnings;
use Archive::Zip qw/AZ_OK/;
use File::Temp qw/tempfile/;
use constant MB => 1024 * 1024;
#my #dir = '/dir1 dir2/';
my $dir = qw( dir1/);
my #files = do {
opendir my $fd, "$dir" or die $! or die $!;
grep -f, map "$dir$_", readdir $fd;
};
my $zip = Archive::Zip->new;
my $total;
my $limit = 5*MB;
foreach my $file (#files) {
my $temp = Archive::Zip->new;
my $member = $temp->addFile($file);
next unless $member->compressedSize;
my $fh = tempfile();
$temp->writeToFileHandle($fh) == AZ_OK or die $!;
$zip->addMember($member);
$total += $member->compressedSize;
die "$total bytes exceeds archive size limit" if $total > $limit;
}
print "Total archive size: $total bytes\n\n";
$zip->writeToFileNamed('zipped.zip') == AZ_OK or die $!;
Thanks!
IMHO you are solving the wrong problem. If you need to zip the files into 5M chunks, why not zip them all together and then split into 5M chunks?
You can do the splitting internally to perl (a good example is here); or for less portable solution use a system command split (available on Unix/Linux; there's a DOS port as well)