Increment date in shell script on AIX [duplicate] - perl

This question already has answers here:
What is the optimal way to loop between two dates in Perl?
(5 answers)
Closed 6 years ago.
I have to create a shell script file that accepts a string, start date and end date and generates the output file in the below format.
Note : Dates are in MM/DD/YYYY
string,start date
string,start date + 1
...
string,end date
I am not able to increment the date. I tried using date -d option but it is not available on AIX.
Could someone please help with a built in shell script command or perl command to increment the date ?.

Given a constraint of 'uses core':
#!/usr/bin/env perl
use strict;
use warnings;
use Time::Piece;
use Time::Seconds;
my $FORMAT = '%m/%d/%Y';
my $start = '01/22/2016';
my $end = '01/31/2016';
my $start_t = Time::Piece->strptime( $start, $FORMAT );
my $end_t = Time::Piece->strptime( $end, $FORMAT );
while ( $start_t <= $end_t ) {
print $start_t ->strftime($FORMAT), "\n";
$start_t += ONE_DAY;
}
Both Time::Piece and Time::Seconds are core as of perl 5.9.5. The latter is only needed for ONE_DAY - otherwise you can just add 60 * 60 * 24 instead.
I'm sure you can figure out how to print the strings etc. yourself.
(I'd also note - this is a horrible time format, and should be avoided)

You can make a shellscript using awk like this:
str="some string"
startd="12/25/2016"
endd="1/17/2017"
maxloop=10
echo "${maxloop}" |
awk -v string="${str}" -v startdate="${startd}" -v enddate="${endd}" '{
split(startdate,A,"[/]");
T1=mktime(A[3] " " A[1] " " A[2] " 0 0 0");
split(enddate,B,"[/]");
T2=mktime(B[3] " " B[1] " " B[2] " 23 59 59");
linenr=1;
while (T1 < T2) {
printf("%s,%s\n",string,
strftime("%m-%d-%Y",T1));
T1+=3600*24;
if (linenr++ > $1) break;
}
}'
The linenr/maxloop is not adding something to the solution but will help against
mistakes in the input values or maintenance of the script.

Related

Why the numbers are not resolved in IF condition in Perl without quotes

I have a script which will do certain set of operation based on the time in IF condition. So basically IF will look for the Current Time which is Hour and Minutes.
Here is the script:
use strict;
use warnings;
my $TIME = `date '+%H%M'`;
chomp $TIME;
print $TIME."\n";
if (( $TIME > 0100) && ($TIME < 0300)){
print "1st IF\n";
} elsif (( $TIME > 0300) && ($TIME < 0500)){
print "2nd IF\n";
} else {
print "Default IF\n";
}
Lets say now the time is 0130 and it suppose to print 1st IF. But its printing Default IF.
Similarly when the time is 0430 and it suppose to print 2nd IF. But its printing Default IF.
But when I enclose the time value in double quotes "" it works. Like below -
if (( $TIME > "0100") && ($TIME < "0300")){
Do the $TIME is resolving as a string ? I was thinking the $TIME would have numbers and it doesn't require quotes.
As you can find in perlnumber, literal numbers starting with zeros are interpreted as octal numbers. 0100 is therefore 64 and 0300 is 192.
When numifying a string, leading zeros are ignored.
See also Time::Piece on how to handle time in Perl without the need to shell out.
use Time::Piece;
my $time = localtime->strftime('%H%M');
Time::Piece has been a standard part of the Perl distribution since Perl 5.10 (in 2002). If you use the right tool for the job, then things get a lot easier :-)
use Time::Piece;
my $hour = localtime->hour;
if ($hour > 1 and $hour < 3) {
# do something
} elsif ($hour > 3 and $hour < 5) {
# do something else
} else {
# do the default thing
}
Update: You don't even need to use Time::Piece. You can get the hour from localtime() easily enough.
my $hour = (localtime)[2];
# The rest of the code is the same as my previous example

Replace date in a txt file having xml

I have a text file in which i have a xml data. Now i want to replace the date from current date plus 2 days in dd/mm/yyyy format . I tried it will sed command in Unix but i am unable to do that .Below command i tried to change the date .But it is not changing the text.
sed -i 's/\<Date\>*\<Date\>/\<Date\>date +"%m/%d/%y"<Date\>/g' avltest.xml
<Args>
<Date>01/10/2017</Date>
<\Args>
In the Date field i want to change the date whenever i run my command or i can use that command from a script.
Here's a perl version:
perl -MPOSIX=strftime -i -pe 's{<Date>.*?</Date>}{"<Date>" . strftime("%m/%d/%Y", localtime(time + 2 * 24 * 60 * 60)) . "</Date>"}e' avltest.xml
I removed the redundant backslashes before < and >, added the missing / in </Date>, and fixed the * part.
Don't use regular expressions to modify XML. It's dirty, hacky, brittle and unnecessary:
#!/usr/bin/perl
use warnings;
use strict;
use XML::Twig;
use Time::Piece;
#load your file
my $twig = XML::Twig->new->parsefile('avltest.xml');
#iterate all <Date> elements.
foreach my $date_elt ( $twig->get_xpath('//Date') ) {
my $date = localtime;
#increment it by two days.
$date += 2 * 60 * 60 * 24; #two days.
#replace it.
$date_elt -> set_text($date -> strftime("%m/%d/%Y"));
}
#print XML to STDOUT.
$twig -> set_pretty_print('indented_a');
$twig -> print;
This will do what you want as described, and not be tripped up by 'end of year/end of month'.
You can use parsefile_inplace in XML twig if you prefer - that requires a slightly different code setup.

insert null values in missing rows of file

I have a text file which consists some data of 24 hours time stamp segregated in 10 minutes interval.
2016-02-06,00:00:00,ujjawal,36072-2,MT,37,0,1
2016-02-06,00:10:00,ujjawal,36072-2,MT,37,0,1
2016-02-06,00:20:00,ujjawal,36072-2,MT,37,0,1
2016-02-06,00:40:00,ujjawal,36072-2,MT,37,0,1
2016-02-06,00:50:00,ujjawal,36072-2,MT,42,0,2
2016-02-06,01:00:00,ujjawal,36072-2,MT,55,0,2
2016-02-06,01:10:00,ujjawal,36072-2,MT,41,0,2
2016-02-06,01:20:00,ujjawal,36072-2,MT,46,0,2
2016-02-06,01:30:00,ujjawal,36072-2,MT,56,0,3
2016-02-06,01:40:00,ujjawal,36072-2,MT,38,0,2
2016-02-06,01:50:00,ujjawal,36072-2,MT,49,0,2
2016-02-06,02:00:00,ujjawal,36072-2,MT,58,0,4
2016-02-06,02:10:00,ujjawal,36072-2,MT,43,0,2
2016-02-06,02:20:00,ujjawal,36072-2,MT,46,0,2
2016-02-06,02:30:00,ujjawal,36072-2,MT,61,0,2
2016-02-06,02:40:00,ujjawal,36072-2,MT,57,0,3
2016-02-06,02:50:00,ujjawal,36072-2,MT,45,0,2
2016-02-06,03:00:00,ujjawal,36072-2,MT,45,0,3
2016-02-06,03:10:00,ujjawal,36072-2,MT,51,0,2
2016-02-06,03:20:00,ujjawal,36072-2,MT,68,0,3
2016-02-06,03:30:00,ujjawal,36072-2,MT,51,0,2
2016-02-06,03:40:00,ujjawal,36072-2,MT,68,0,3
2016-02-06,03:50:00,ujjawal,36072-2,MT,67,0,3
2016-02-06,04:00:00,ujjawal,36072-2,MT,82,0,8
2016-02-06,04:10:00,ujjawal,36072-2,MT,82,0,5
2016-02-06,04:20:00,ujjawal,36072-2,MT,122,0,4
2016-02-06,04:30:00,ujjawal,36072-2,MT,133,0,3
2016-02-06,04:40:00,ujjawal,36072-2,MT,142,0,3
2016-02-06,04:50:00,ujjawal,36072-2,MT,202,0,1
2016-02-06,05:00:00,ujjawal,36072-2,MT,731,1,3
2016-02-06,05:10:00,ujjawal,36072-2,MT,372,0,7
2016-02-06,05:20:00,ujjawal,36072-2,MT,303,0,2
2016-02-06,05:30:00,ujjawal,36072-2,MT,389,0,3
2016-02-06,05:40:00,ujjawal,36072-2,MT,454,0,1
2016-02-06,05:50:00,ujjawal,36072-2,MT,406,0,6
2016-02-06,06:00:00,ujjawal,36072-2,MT,377,0,1
2016-02-06,06:10:00,ujjawal,36072-2,MT,343,0,5
2016-02-06,06:20:00,ujjawal,36072-2,MT,370,0,2
2016-02-06,06:30:00,ujjawal,36072-2,MT,343,0,9
2016-02-06,06:40:00,ujjawal,36072-2,MT,315,0,8
2016-02-06,06:50:00,ujjawal,36072-2,MT,458,0,3
2016-02-06,07:00:00,ujjawal,36072-2,MT,756,1,3
2016-02-06,07:10:00,ujjawal,36072-2,MT,913,1,3
2016-02-06,07:20:00,ujjawal,36072-2,MT,522,0,3
2016-02-06,07:30:00,ujjawal,36072-2,MT,350,0,7
2016-02-06,07:40:00,ujjawal,36072-2,MT,328,0,6
2016-02-06,07:50:00,ujjawal,36072-2,MT,775,1,3
2016-02-06,08:00:00,ujjawal,36072-2,MT,310,0,9
2016-02-06,08:10:00,ujjawal,36072-2,MT,308,0,6
2016-02-06,08:20:00,ujjawal,36072-2,MT,738,1,3
2016-02-06,08:30:00,ujjawal,36072-2,MT,294,0,6
2016-02-06,08:40:00,ujjawal,36072-2,MT,345,0,1
2016-02-06,08:50:00,ujjawal,36072-2,MT,367,0,6
2016-02-06,09:00:00,ujjawal,36072-2,MT,480,0,3
2016-02-06,09:10:00,ujjawal,36072-2,MT,390,0,3
2016-02-06,09:20:00,ujjawal,36072-2,MT,436,0,3
2016-02-06,09:30:00,ujjawal,36072-2,MT,1404,2,3
2016-02-06,09:40:00,ujjawal,36072-2,MT,346,0,3
2016-02-06,09:50:00,ujjawal,36072-2,MT,388,0,3
2016-02-06,10:00:00,ujjawal,36072-2,MT,456,0,2
2016-02-06,10:10:00,ujjawal,36072-2,MT,273,0,7
2016-02-06,10:20:00,ujjawal,36072-2,MT,310,0,3
2016-02-06,10:30:00,ujjawal,36072-2,MT,256,0,7
2016-02-06,10:40:00,ujjawal,36072-2,MT,283,0,3
2016-02-06,10:50:00,ujjawal,36072-2,MT,276,0,3
2016-02-06,11:00:00,ujjawal,36072-2,MT,305,0,1
2016-02-06,11:10:00,ujjawal,36072-2,MT,310,0,9
2016-02-06,11:20:00,ujjawal,36072-2,MT,286,0,3
2016-02-06,11:30:00,ujjawal,36072-2,MT,286,0,3
2016-02-06,11:40:00,ujjawal,36072-2,MT,247,0,7
2016-02-06,11:50:00,ujjawal,36072-2,MT,366,0,2
2016-02-06,12:00:00,ujjawal,36072-2,MT,294,0,2
2016-02-06,12:10:00,ujjawal,36072-2,MT,216,0,5
2016-02-06,12:20:00,ujjawal,36072-2,MT,233,0,1
2016-02-06,12:30:00,ujjawal,36072-2,MT,785,1,2
2016-02-06,12:40:00,ujjawal,36072-2,MT,466,0,1
2016-02-06,12:50:00,ujjawal,36072-2,MT,219,0,9
2016-02-06,13:00:00,ujjawal,36072-2,MT,248,0,6
2016-02-06,13:10:00,ujjawal,36072-2,MT,223,0,7
2016-02-06,13:20:00,ujjawal,36072-2,MT,276,0,8
2016-02-06,13:30:00,ujjawal,36072-2,MT,219,0,6
2016-02-06,13:40:00,ujjawal,36072-2,MT,699,1,2
2016-02-06,13:50:00,ujjawal,36072-2,MT,439,0,2
2016-02-06,14:00:00,ujjawal,36072-2,MT,1752,2,3
2016-02-06,14:10:00,ujjawal,36072-2,MT,203,0,5
2016-02-06,14:20:00,ujjawal,36072-2,MT,230,0,7
2016-02-06,14:30:00,ujjawal,36072-2,MT,226,0,1
2016-02-06,14:40:00,ujjawal,36072-2,MT,195,0,6
2016-02-06,14:50:00,ujjawal,36072-2,MT,314,0,1
2016-02-06,15:00:00,ujjawal,36072-2,MT,357,0,2
2016-02-06,15:10:00,ujjawal,36072-2,MT,387,0,9
2016-02-06,15:20:00,ujjawal,36072-2,MT,1084,1,3
2016-02-06,15:30:00,ujjawal,36072-2,MT,1295,2,3
2016-02-06,15:40:00,ujjawal,36072-2,MT,223,0,8
2016-02-06,15:50:00,ujjawal,36072-2,MT,254,0,1
2016-02-06,16:00:00,ujjawal,36072-2,MT,252,0,7
2016-02-06,16:10:00,ujjawal,36072-2,MT,268,0,1
2016-02-06,16:20:00,ujjawal,36072-2,MT,242,0,1
2016-02-06,16:30:00,ujjawal,36072-2,MT,254,0,9
2016-02-06,16:40:00,ujjawal,36072-2,MT,271,0,3
2016-02-06,16:50:00,ujjawal,36072-2,MT,244,0,7
2016-02-06,17:00:00,ujjawal,36072-2,MT,281,0,1
2016-02-06,17:10:00,ujjawal,36072-2,MT,190,0,8
2016-02-06,17:20:00,ujjawal,36072-2,MT,187,0,1
2016-02-06,17:30:00,ujjawal,36072-2,MT,173,0,9
2016-02-06,17:40:00,ujjawal,36072-2,MT,140,0,5
2016-02-06,17:50:00,ujjawal,36072-2,MT,147,0,6
2016-02-06,18:00:00,ujjawal,36072-2,MT,109,0,4
2016-02-06,18:10:00,ujjawal,36072-2,MT,99,0,1
2016-02-06,18:20:00,ujjawal,36072-2,MT,66,0,6
2016-02-06,18:30:00,ujjawal,36072-2,MT,67,0,4
2016-02-06,18:40:00,ujjawal,36072-2,MT,40,0,2
2016-02-06,18:50:00,ujjawal,36072-2,MT,52,0,3
2016-02-06,19:00:00,ujjawal,36072-2,MT,40,0,3
2016-02-06,19:10:00,ujjawal,36072-2,MT,30,0,2
2016-02-06,19:20:00,ujjawal,36072-2,MT,25,0,3
2016-02-06,19:30:00,ujjawal,36072-2,MT,35,0,4
2016-02-06,19:40:00,ujjawal,36072-2,MT,14,0,1
2016-02-06,19:50:00,ujjawal,36072-2,MT,97,0,7
2016-02-06,20:00:00,ujjawal,36072-2,MT,14,0,1
2016-02-06,20:10:00,ujjawal,36072-2,MT,12,0,4
2016-02-06,20:20:00,ujjawal,36072-2,MT,11,0,2
2016-02-06,20:30:00,ujjawal,36072-2,MT,12,0,1
2016-02-06,20:40:00,ujjawal,36072-2,MT,6,0,1
2016-02-06,20:50:00,ujjawal,36072-2,MT,13,0,2
2016-02-06,21:00:00,ujjawal,36072-2,MT,5,0,1
2016-02-06,21:10:00,ujjawal,36072-2,MT,12,0,2
2016-02-06,21:20:00,ujjawal,36072-2,MT,1,0,1
2016-02-06,21:30:00,ujjawal,36072-2,MT,21,0,2
2016-02-06,21:50:00,ujjawal,36072-2,MT,9,0,3
2016-02-06,22:00:00,ujjawal,36072-2,MT,2,0,1
2016-02-06,22:10:00,ujjawal,36072-2,MT,12,0,5
2016-02-06,22:20:00,ujjawal,36072-2,MT,1,0,1
2016-02-06,22:30:00,ujjawal,36072-2,MT,9,0,1
2016-02-06,22:40:00,ujjawal,36072-2,MT,13,0,1
2016-02-06,23:00:00,ujjawal,36072-2,MT,20,0,2
2016-02-06,23:10:00,ujjawal,36072-2,MT,10,0,3
2016-02-06,23:20:00,ujjawal,36072-2,MT,10,0,1
2016-02-06,23:30:00,ujjawal,36072-2,MT,6,0,1
2016-02-06,23:40:00,ujjawal,36072-2,MT,12,0,1
if you see above sample as per 10 minutes interval there should be total 143 rows in 24 hours in this file but after second last line which has time 2016-02-06,23:40:00 data for date, time 2016-02-06,23:50:00 is missing.
similarly after 2016-02-06,22:40:00 data for date, time 2016-02-06,22:50:00 is missing.
can we insert missing date,time followed by 6 null separated by commas e.g. 2016-02-06,22:50:00,null,null,null,null,null,null where ever any data missing in rows of this file based on count no 143 rows and time stamp comparison in rows 2016-02-06,00:00:00 to 2016-02-06,23:50:00 which is also 143 in count ?
here is what i have tried
created a file with 143 entries of date and time as 2.csv and used below command
join -j 2 -o 1.1,1.2,1.3,1.4,1.5,1.6,1.7,1.8,2.1,2.1,2.2 <(sort -k2 1.csv) <(sort -k2 2.csv)|grep "2016-02-06,21:30:00"| sort -u|sed "s/\t//g"> 3.txt
part of output is repetitive like this :
2016-02-06,21:30:00
2016-02-06,21:30:00
2016-02-06,00:00:00,ujjawal,36072-2,MT,37,0,1
2016-02-06,21:30:00
2016-02-06,21:30:00
2016-02-06,00:10:00,ujjawal,36072-2,MT,37,0,1
2016-02-06,21:30:00
2016-02-06,21:30:00
2016-02-06,00:20:00,ujjawal,36072-2,MT,37,0,1
2016-02-06,21:30:00
2016-02-06,21:30:00
2016-02-06,00:40:00,ujjawal,36072-2,MT,37,0,1
2016-02-06,21:30:00
2016-02-06,21:30:00
2016-02-06,00:50:00,ujjawal,36072-2,MT,42,0,2
2016-02-06,21:30:00
any suggestions ?
I'd actually not cross reference a new csv file, and instead do it like this:
#!/usr/bin/env perl
use strict;
use warnings;
use Time::Piece;
my $last_timestamp;
my $interval = 600;
#read stdin line by line
while ( <> ) {
#extract date and time from this line.
my ( $date, $time, #fields ) = split /,/;
#parse the timestamp
my $timestamp = Time::Piece -> strptime ( $date . $time, "%Y-%m-%d%H:%M:%S" );
#set last if undefined.
$last_timestamp //= $timestamp;
#if there's a gap... :
if ( $last_timestamp + $interval < $timestamp ) {
#print "GAP detected at $timestamp: ",$timestamp - $last_timestamp,"\n";
#print lines to fill in the gap
for ( ($timestamp - $last_timestamp) % 600 ) {
$last_timestamp += 600;
print join ( ",", $last_timestamp -> strftime("%Y-%m-%d,%H:%M:%S"), ("null")x6),"\n";
}
}
$last_timestamp = $timestamp;
print;
}
Which for your sample gives me lines (snipped for brevity):
2016-02-06,22:40:00,ujjawal,36072-2,MT,13,0,1
2016-02-06,22:50:00,null,null,null,null,null,null
2016-02-06,23:00:00,ujjawal,36072-2,MT,20,0,2
Note - this is assuming the timestamps are exactly 600s apart. You can adjust the logic a little if that isn't a valid assumption, but it depends exactly what you're trying to get at that point.
Here's another Perl solution
It initialises $date to the date contained in the first line of the file, and a time of 00:00:00
It then fills the %values hash with records using the value of $date as a key, incrementing the value by ten minutes until the day of month changes. These form the "default" values
Then the contents of the file are used to overwrite all elements of %values for which we have an actual value. Any gaps will remain set to their default from the previous step
Then the hash is simply printed in sorted order, resulting in a full set of data with defaults inserted as necessary
use strict;
use warnings 'all';
use Time::Piece;
use Time::Seconds 'ONE_MINUTE';
use Fcntl ':seek';
my $delta = 10 * ONE_MINUTE;
my $date = Time::Piece->strptime(<ARGV> =~ /^([\d-]+)/, '%Y-%m-%d');
my %values;
for ( my $day = $date->mday; $date->mday == $day; $date += $delta ) {
my $ds = $date->strftime('%Y-%m-%d,%H:%M:%S');
$values{$ds} = $ds. ',null' x 6 . "\n";
}
seek ARGV, 0, SEEK_SET;
while ( <ARGV> ) {
my ($ds) = /^([\d-]+,[\d:]+)/;
$values{$ds} = $_;
}
print $values{$_} for sort keys %values;
here is the answer..
cat 1.csv 2.csv|sort -u -t, -k2,2
...or a shell script:
#! /bin/bash
set -e
file=$1
today=$(head -1 $file | cut -d, -f1)
line=0
for (( h = 0 ; h < 24 ; h++ ))
do
for (( m = 0 ; m < 60 ; m += 10 ))
do
stamp=$(printf "%02d:%02d:00" $h $m)
if [ $line -eq 0 ]; then IFS=',' read date time data; fi
if [ "$time" = "$stamp" ]; then
echo $date,$time,$data
line=0
else
echo $today,$stamp,null,null,null,null,null,null
line=1
fi
done
done <$file
I would write it like this in Perl
This program expects the name of the input file as a parameter on the command line, and prints its output to STDOUT, which may be redirected as normal
use strict;
use warnings 'all';
use feature 'say';
use Time::Piece;
use Time::Seconds 'ONE_MINUTE';
my $format = '%Y-%m-%d,%H:%M:%S';
my $delta = 10 * ONE_MINUTE;
my $next;
our #ARGV = 'mydates.txt';
while ( <> ) {
my $new = Time::Piece->strptime(/^([\d-]+,[\d:]+)/, $format);
while ( $next and $next < $new ) {
say $next->strftime($format) . ',null' x 6;
$next += $delta;
}
print;
$next = $new + $delta;
}
while ( $next and $next->hms('') > 0 ) {
say $next->strftime($format) . ',null' x 6;
$next += $delta;
}
output
2016-02-06,00:00:00,ujjawal,36072-2,MT,37,0,1
2016-02-06,00:10:00,ujjawal,36072-2,MT,37,0,1
2016-02-06,00:20:00,ujjawal,36072-2,MT,37,0,1
2016-02-06,00:30:00,null,null,null,null,null,null
2016-02-06,00:40:00,ujjawal,36072-2,MT,37,0,1
2016-02-06,00:50:00,ujjawal,36072-2,MT,42,0,2
2016-02-06,01:00:00,ujjawal,36072-2,MT,55,0,2
2016-02-06,01:10:00,ujjawal,36072-2,MT,41,0,2
2016-02-06,01:20:00,ujjawal,36072-2,MT,46,0,2
2016-02-06,01:30:00,ujjawal,36072-2,MT,56,0,3
2016-02-06,01:40:00,ujjawal,36072-2,MT,38,0,2
2016-02-06,01:50:00,ujjawal,36072-2,MT,49,0,2
2016-02-06,02:00:00,ujjawal,36072-2,MT,58,0,4
2016-02-06,02:10:00,ujjawal,36072-2,MT,43,0,2
2016-02-06,02:20:00,ujjawal,36072-2,MT,46,0,2
2016-02-06,02:30:00,ujjawal,36072-2,MT,61,0,2
2016-02-06,02:40:00,ujjawal,36072-2,MT,57,0,3
2016-02-06,02:50:00,ujjawal,36072-2,MT,45,0,2
2016-02-06,03:00:00,ujjawal,36072-2,MT,45,0,3
2016-02-06,03:10:00,ujjawal,36072-2,MT,51,0,2
2016-02-06,03:20:00,ujjawal,36072-2,MT,68,0,3
2016-02-06,03:30:00,ujjawal,36072-2,MT,51,0,2
2016-02-06,03:40:00,ujjawal,36072-2,MT,68,0,3
2016-02-06,03:50:00,ujjawal,36072-2,MT,67,0,3
2016-02-06,04:00:00,ujjawal,36072-2,MT,82,0,8
2016-02-06,04:10:00,ujjawal,36072-2,MT,82,0,5
2016-02-06,04:20:00,ujjawal,36072-2,MT,122,0,4
2016-02-06,04:30:00,ujjawal,36072-2,MT,133,0,3
2016-02-06,04:40:00,ujjawal,36072-2,MT,142,0,3
2016-02-06,04:50:00,ujjawal,36072-2,MT,202,0,1
2016-02-06,05:00:00,ujjawal,36072-2,MT,731,1,3
2016-02-06,05:10:00,ujjawal,36072-2,MT,372,0,7
2016-02-06,05:20:00,ujjawal,36072-2,MT,303,0,2
2016-02-06,05:30:00,ujjawal,36072-2,MT,389,0,3
2016-02-06,05:40:00,ujjawal,36072-2,MT,454,0,1
2016-02-06,05:50:00,ujjawal,36072-2,MT,406,0,6
2016-02-06,06:00:00,ujjawal,36072-2,MT,377,0,1
2016-02-06,06:10:00,ujjawal,36072-2,MT,343,0,5
2016-02-06,06:20:00,ujjawal,36072-2,MT,370,0,2
2016-02-06,06:30:00,ujjawal,36072-2,MT,343,0,9
2016-02-06,06:40:00,ujjawal,36072-2,MT,315,0,8
2016-02-06,06:50:00,ujjawal,36072-2,MT,458,0,3
2016-02-06,07:00:00,ujjawal,36072-2,MT,756,1,3
2016-02-06,07:10:00,ujjawal,36072-2,MT,913,1,3
2016-02-06,07:20:00,ujjawal,36072-2,MT,522,0,3
2016-02-06,07:30:00,ujjawal,36072-2,MT,350,0,7
2016-02-06,07:40:00,ujjawal,36072-2,MT,328,0,6
2016-02-06,07:50:00,ujjawal,36072-2,MT,775,1,3
2016-02-06,08:00:00,ujjawal,36072-2,MT,310,0,9
2016-02-06,08:10:00,ujjawal,36072-2,MT,308,0,6
2016-02-06,08:20:00,ujjawal,36072-2,MT,738,1,3
2016-02-06,08:30:00,ujjawal,36072-2,MT,294,0,6
2016-02-06,08:40:00,ujjawal,36072-2,MT,345,0,1
2016-02-06,08:50:00,ujjawal,36072-2,MT,367,0,6
2016-02-06,09:00:00,ujjawal,36072-2,MT,480,0,3
2016-02-06,09:10:00,ujjawal,36072-2,MT,390,0,3
2016-02-06,09:20:00,ujjawal,36072-2,MT,436,0,3
2016-02-06,09:30:00,ujjawal,36072-2,MT,1404,2,3
2016-02-06,09:40:00,ujjawal,36072-2,MT,346,0,3
2016-02-06,09:50:00,ujjawal,36072-2,MT,388,0,3
2016-02-06,10:00:00,ujjawal,36072-2,MT,456,0,2
2016-02-06,10:10:00,ujjawal,36072-2,MT,273,0,7
2016-02-06,10:20:00,ujjawal,36072-2,MT,310,0,3
2016-02-06,10:30:00,ujjawal,36072-2,MT,256,0,7
2016-02-06,10:40:00,ujjawal,36072-2,MT,283,0,3
2016-02-06,10:50:00,ujjawal,36072-2,MT,276,0,3
2016-02-06,11:00:00,ujjawal,36072-2,MT,305,0,1
2016-02-06,11:10:00,ujjawal,36072-2,MT,310,0,9
2016-02-06,11:20:00,ujjawal,36072-2,MT,286,0,3
2016-02-06,11:30:00,ujjawal,36072-2,MT,286,0,3
2016-02-06,11:40:00,ujjawal,36072-2,MT,247,0,7
2016-02-06,11:50:00,ujjawal,36072-2,MT,366,0,2
2016-02-06,12:00:00,ujjawal,36072-2,MT,294,0,2
2016-02-06,12:10:00,ujjawal,36072-2,MT,216,0,5
2016-02-06,12:20:00,ujjawal,36072-2,MT,233,0,1
2016-02-06,12:30:00,ujjawal,36072-2,MT,785,1,2
2016-02-06,12:40:00,ujjawal,36072-2,MT,466,0,1
2016-02-06,12:50:00,ujjawal,36072-2,MT,219,0,9
2016-02-06,13:00:00,ujjawal,36072-2,MT,248,0,6
2016-02-06,13:10:00,ujjawal,36072-2,MT,223,0,7
2016-02-06,13:20:00,ujjawal,36072-2,MT,276,0,8
2016-02-06,13:30:00,ujjawal,36072-2,MT,219,0,6
2016-02-06,13:40:00,ujjawal,36072-2,MT,699,1,2
2016-02-06,13:50:00,ujjawal,36072-2,MT,439,0,2
2016-02-06,14:00:00,ujjawal,36072-2,MT,1752,2,3
2016-02-06,14:10:00,ujjawal,36072-2,MT,203,0,5
2016-02-06,14:20:00,ujjawal,36072-2,MT,230,0,7
2016-02-06,14:30:00,ujjawal,36072-2,MT,226,0,1
2016-02-06,14:40:00,ujjawal,36072-2,MT,195,0,6
2016-02-06,14:50:00,ujjawal,36072-2,MT,314,0,1
2016-02-06,15:00:00,ujjawal,36072-2,MT,357,0,2
2016-02-06,15:10:00,ujjawal,36072-2,MT,387,0,9
2016-02-06,15:20:00,ujjawal,36072-2,MT,1084,1,3
2016-02-06,15:30:00,ujjawal,36072-2,MT,1295,2,3
2016-02-06,15:40:00,ujjawal,36072-2,MT,223,0,8
2016-02-06,15:50:00,ujjawal,36072-2,MT,254,0,1
2016-02-06,16:00:00,ujjawal,36072-2,MT,252,0,7
2016-02-06,16:10:00,ujjawal,36072-2,MT,268,0,1
2016-02-06,16:20:00,ujjawal,36072-2,MT,242,0,1
2016-02-06,16:30:00,ujjawal,36072-2,MT,254,0,9
2016-02-06,16:40:00,ujjawal,36072-2,MT,271,0,3
2016-02-06,16:50:00,ujjawal,36072-2,MT,244,0,7
2016-02-06,17:00:00,ujjawal,36072-2,MT,281,0,1
2016-02-06,17:10:00,ujjawal,36072-2,MT,190,0,8
2016-02-06,17:20:00,ujjawal,36072-2,MT,187,0,1
2016-02-06,17:30:00,ujjawal,36072-2,MT,173,0,9
2016-02-06,17:40:00,ujjawal,36072-2,MT,140,0,5
2016-02-06,17:50:00,ujjawal,36072-2,MT,147,0,6
2016-02-06,18:00:00,ujjawal,36072-2,MT,109,0,4
2016-02-06,18:10:00,ujjawal,36072-2,MT,99,0,1
2016-02-06,18:20:00,ujjawal,36072-2,MT,66,0,6
2016-02-06,18:30:00,ujjawal,36072-2,MT,67,0,4
2016-02-06,18:40:00,ujjawal,36072-2,MT,40,0,2
2016-02-06,18:50:00,ujjawal,36072-2,MT,52,0,3
2016-02-06,19:00:00,ujjawal,36072-2,MT,40,0,3
2016-02-06,19:10:00,ujjawal,36072-2,MT,30,0,2
2016-02-06,19:20:00,ujjawal,36072-2,MT,25,0,3
2016-02-06,19:30:00,ujjawal,36072-2,MT,35,0,4
2016-02-06,19:40:00,ujjawal,36072-2,MT,14,0,1
2016-02-06,19:50:00,ujjawal,36072-2,MT,97,0,7
2016-02-06,20:00:00,ujjawal,36072-2,MT,14,0,1
2016-02-06,20:10:00,ujjawal,36072-2,MT,12,0,4
2016-02-06,20:20:00,ujjawal,36072-2,MT,11,0,2
2016-02-06,20:30:00,ujjawal,36072-2,MT,12,0,1
2016-02-06,20:40:00,ujjawal,36072-2,MT,6,0,1
2016-02-06,20:50:00,ujjawal,36072-2,MT,13,0,2
2016-02-06,21:00:00,ujjawal,36072-2,MT,5,0,1
2016-02-06,21:10:00,ujjawal,36072-2,MT,12,0,2
2016-02-06,21:20:00,ujjawal,36072-2,MT,1,0,1
2016-02-06,21:30:00,ujjawal,36072-2,MT,21,0,2
2016-02-06,21:40:00,null,null,null,null,null,null
2016-02-06,21:50:00,ujjawal,36072-2,MT,9,0,3
2016-02-06,22:00:00,ujjawal,36072-2,MT,2,0,1
2016-02-06,22:10:00,ujjawal,36072-2,MT,12,0,5
2016-02-06,22:20:00,ujjawal,36072-2,MT,1,0,1
2016-02-06,22:30:00,ujjawal,36072-2,MT,9,0,1
2016-02-06,22:40:00,ujjawal,36072-2,MT,13,0,1
2016-02-06,22:50:00,null,null,null,null,null,null
2016-02-06,23:00:00,ujjawal,36072-2,MT,20,0,2
2016-02-06,23:10:00,ujjawal,36072-2,MT,10,0,3
2016-02-06,23:20:00,ujjawal,36072-2,MT,10,0,1
2016-02-06,23:30:00,ujjawal,36072-2,MT,6,0,1
2016-02-06,23:40:00,ujjawal,36072-2,MT,12,0,1
2016-02-06,23:50:00,null,null,null,null,null,null

Optimize perl script to filter rows based on date in the file

I am a beginner with programming not just perl !
Please let me know what needs to change or how else this can be done.
Need to optimize the perl code to run faster.
For a test run, with around a 500MB file with 3 million rows in it, runtime is 28 minutes.
I know a tool which processes the 39 million rows in 15 mins, but i want to acheive this running on the command prompt without resorting to the tool.
Earlier I used Date::Manip and Date::Parse and moved on to DateTime, thinking it should be faster.
My approach was If the dates are ISO-8601 (ie, YYYY-MM-DD) and we do not need to validate them,
we can compare lexicographically (ie, the lt and gt operators.)
Input File Date Format is 07/18/2013 13:45:49
Input File Size 42GB.
Number of Rows 39 Million.
Column Delimiter : |~|
Platform : GNU/Linux
I have tried ">" and "gt" and did not find any difference in runtime.
Code snippet:
use DateTime::Format::Strptime;
my $idate = "07/17/2013 00:00:00";
my $Strp = DateTime::Format::Strptime->new(
pattern => '%m/%d/%Y %H:%M:%S',
);
my $inputdt = $Strp->parse_datetime($idate);
open (FILE,"myinputfile.dat") or die "could not input File\n";
while (defined(my $line = <FILE>)) {
my #chunks = split '[|]~[|]', $line;
my $fdate = $Strp->parse_datetime($chunks[6]);
if ( $fdate > $inputdt) {
open(FILEOUT, ">>myoutputfile.dat") or die "Could not write\n";
print FILEOUT "$line";
}
}
close(FILE);
close (FILEOUT);
There are two and a half big performance problems here:
You open the output file in every iteration. Just open it once, before the loop.
The parse_datetime returns a DateTime object. Object orientation with Perl implies a significant overhead. Because your pattern is well defined, we can do the parsing ourself, and remove all object orientation.
Reading a file in the GB range just takes some time. To speed this up, upgrade your hardware (e.g. to a SSD).
To parse the date string into a sortable representation, we just reorder the various parts to a string:
# %m/%d/%Y %H:%M:%S → %Y/%m/%d %H:%M:%S
$fdate =~ s{^ ([0-9]{2} / [0-9]{2}) / ([0-9]{4}) }{$2/$1}x;
if ($fdate gt $inputdate) { ... }
This would lead to the code
use strict; use warnings;
use constant DATE_FIELD => shift #ARGV;
my $inputdate = shift #ARGV;
$inputdate =~ s{^ ([0-9]{2} / [0-9]{2}) / ([0-9]{4}) }{$2/$1}x;
<>; # remove the header line
while (<>) {
my $filedate = (split /\|~\|/, $_, DATE_FIELD + 2)[DATE_FIELD];
$filedate =~ s{^ ([0-9]{2} / [0-9]{2}) / ([0-9]{4}) }{$2/$1}x;
print if $filedate gt $inputdate;
}
The in- and output, as well as the start date, are specified on the command line, e.g.
./script 6 '07/17/2013 00:00:00' myinputfile.dat >>myoutputfile.dat

Use Perl to Add 2 days to Today

I want to use perl and add two days from today and output this as a unix time. I have found lots of information on how to convert Unix time to readable time but I need the ouput to be a unix time. I found this
my $time = time; # or any other epoch timestamp
my #months = ("Jan","Feb","Mar","Apr","May","Jun","Jul",
"Aug","Sep","Oct","Nov","Dec");
my ($sec, $min, $hour, $day,$month,$year) = (localtime($time))[0,1,2,3,4,5];
# You can use 'gmtime' for GMT/UTC dates instead of 'localtime'
print "Unix time ".$time." converts to ".$months[$month].
" ".$day.", ".($year+1900);
How do I take current time and add 2 days and output as Unix time.
Have you considered using the DateTime package? It contains a lot of date manipulation and computation routines, including the ability to add dates.
There is an FAQ, with examples, here (in particular, see the section on sample calculations and DateTime formats).
Here is a snippet:
use strict;
use warnings;
use DateTime;
my $dt = DateTime->now();
print "Now: " . $dt->datetime() . "\n";
print "Now (epoch): " . $dt->epoch() . "\n";
my $two_days_from_now = $dt->add(days => 2);
print "Two days from now: " . $two_days_from_now->datetime() . "\n";
print "Two days from now (epoch): " . $two_days_from_now->epoch() . "\n";
Which produces the following output:
Now: 2013-02-23T18:30:58
Now (epoch): 1361644258
Two days from now: 2013-02-25T18:30:58
Two days from now (epoch): 1361817058
yuu can change the timestamp, which is seconds from epoc
change
my $time = time;
to
my $time = time + 2 * 24 * 60 * 60 ; # 60 seconds 60 minutes 24 hours times 2
Now is time(). Two days from now is time() + 2 * 86400. Today (midnight at the beginning of the current day) is int(time() / 86400) * 86400. Two days from today is today plus 2 * 86400. localtime in scalar context will print any of them out as a readable date, unless you really want gmtime or strftime (from the POSIX module).
The time function returns the current epoch seconds, and core module Time::Seconds provides useful constants for time periods.
use strict;
use warnings;
use Time::Seconds 'ONE_DAY';
print time + ONE_DAY * 2;
output
1361872751