Rolling archival file containing last 1 year's data - perl

Question

A script needs to be developed to create a archive file containing only last 1 year's data. Script will copy the content of main data file (CEMI.log) and append the same to archived log file (CEMI.Archive.log). This will happen once in a week (every monday at 3 am). After one year, the script should be able to remove the data from the archive file which is older than a year.
I am stuck at a point where I need to remove last one year's data. How can this be done using shell or perl script?
sample file:
-bash-3.2# more test.txt
2015-01-15,09:17:10,101,20a6475d-4d0c-4fe4-8765-35065ddfe887,_1.1,L,
2015-01-15,09:18:57,70,al Test,20a6475d-4d0c-4fe4-8765-35065ddfe887,1,L,
2015-01-15,10:59:28,1,,,,Best Practice,9f02745244d6440584b24012d882f935,,L,
2015-01-15,11:00:52,2,,,,Best Practice,9f02745244d6440584b24012d882f935,,L,
2015-01-15,11:00:56,1,,,,Best Practice,9f02745244d6440584b24012d882f935,,L,
2015-01-15,11:03:14,1,,,,Best Practice,9f02745244d6440584b24012d882f935,,L,
2015-01-15,11:03:38,1,,,,Best Practice,9f02745244d6440584b24012d882f935,,L,
2015-01-15,11:04:01,1,,,,Best Practice,9f02745244d6440584b24012d882f935,,L,
2015-01-15,11:05:07,2,,,,Best Practice,9f02745244d6440584b24012d882f935,,L,
2015-01-15,11:06:45,32,,,,Best Practice,9f02745244d6440584b24012d882f935,,L,
2015-01-15,12:57:13,36,,,560909,Best Practice,e8418950-6561-4465-b16b-30e118e826b7,,L,
2015-01-15,13:37:56,1032,,xml-data1,Test,20a6475d-4d0c-4fe4-8765-35065ddfe887,Request_1.1,L,Success
2015-01-15,13:39:01,38,,,Internal Test,20a6475d-4d0c-4fe4-8765-35065ddfe887,Request_1.1,L,
2015-01-15,13:39:50,113,,xml-dat1,al Test,20a6475d-4d0c-4fe4-8765-35065ddfe887,Request_1.1,L,
2015-01-15,13:40:17,74,,, Test,20a6475d-4d0c-4fe4-8765-35065ddfe887,Request_1.1,L,

Answer 1

This will stop at the first line not matching your date format.
#!/usr/bin/env perl
use strict;
use warnings;
use DateTime;
use File::Copy qw/move/;
use File::Temp qw/tempfile/;
# filenames
my $log = "test.txt";
my ( undef, $tmp ) = tempfile( undef, OPEN => 0 ); # get safe temporary filename, but do not open file
# move old log file to temporary location
move $log => $tmp
or die "Cannot rename '$log' to '$tmp': $!";
# open temporary file (contains old log) and new log
open( my $in, "<", $tmp )
or die "Cannot open '$tmp': $!";
open( my $out, ">", $log )
or die "Cannot open '$log': $!";
# calculate a DateTime value one year in the past
my $limit = DateTime->today->subtract( years => 1 );
# skip lines with date that is too old
while (<$in>) {
if (m/^(\d\d\d\d)-(\d\d)-(\d\d)/) {
# get DateTime object from matched date parts
my $dt = DateTime->new(
year => $1,
month => $2,
day => $3,
);
# keep on skipping lines while dates are too old
if ( $dt < $limit ) {
next;
}
}
# if no date was found or date too young, end skipping
last;
}
# copy all remaining lines from temporary file to new log
while (<$in>) {
print $out $_;
}
close $in or warn "Error closing '$in': $!";
close $out or warn "Error closing '$out': $!";
unlink $tmp or die "Cannot delete '$tmp': $!";
How to execute this:
$ cat > test.txt # press Ctrl+D to end text input
2014-01-01,will be removed
stops here
2014-01-01,will be kept because stopped
2015-01-01,would be kept anyway
$ perl rotate_log.pl
$ cat test.txt
2014-01-01,will be kept because stopped
2015-01-01,would be kept anyway