Undefined subroutine &package::subroutine called at line <of script> - perl

I am debugging this script at work - the boss says that is used to work on Solaris, but since they switched over to linux, it stopped working. I had to rewrite it with strict and warnings .
When I run it I get error:
Undefined subroutine &Logging::openLog called at /path/to/script line 27
here is script (well part of it)
1 #!/usr/local/bin/perl
2
3 unshift #INC, "/production/fo/lib";
4 use strict;
5 use warnings;
6 use Sys::Hostname;
7 use Getopt::Long qw(:config bundling auto_version);
8 use File::Path;
9
10 require "dbconfig2.pl";
11 require "logging2.pl";
12 require "hpov.pl";
13
14 # global variables
15 my $parseDate = "";
16 my #fileList = "";
17 my #transList = "";
18 my $mLogDate = "";
19 my $lHost = hostname;
20 my $corefiles_dir="/production/log/corefiles";
21 my $default_Threshold=90;
22
23 # do stuff
24
25 parseOptions();
26 Dbconfig::readconfigFile("$config");
27 Logging::openLog("$Dbconfig::prefs{logFile}","overwrite");
28 # msglog actions TODO logs, compress only, data files
29 my $check_shdw=`ls -l /etc/motd | awk '{print \$11}' | grep 'motd.shdw'`; #Check if hostname is shadow
30 $check_shdw =~ y/\n//d; #remove new line if any
31 if ( $check_shdw eq "motd.shdw" )
32 {
33 Logging::printLog("INFO","Enviroment is Shadow, triggering core files compressing");
34 if (is_folder_empty($corefiles_dir)) {
35 print "Corefile Directory is EMPTY......! \n";
36 }
37 else {
38 gzip_corefiles() ; #Execute compress core files
39 }
40 }
41
The script uses require statements to I guess call upon the routines that the script creator built.
For the purpsoe of this script - the dbconfig just slurps in a config file and breaks them down into values.
like the "$Dbconfig::prefs{logFile}" equals a logfile location /prod/logs/script.log - that's it.
#!/usr/local/bin/perl
package Dbconfig;
#use warnings;
use DBI;
use DBD::Oracle;
%prefs = "";
#$dbPrefs = "";
$raiseError = 0;
%startupItem = "";
# readconfigFile(file) - read in a configuration file.
sub readconfigFile {
my $file = shift;
if ( ! -e $file ) {
$errorTxt = "Error: $file does not exist.\n";
$raiseError = 1;
}
# read in the cfg variables
open(CFGFILE,"<","$file") or die "Cannot open $file for reading: $!\n";
while(<CFGFILE>) {
chomp; # kill newlines
s/#.*//; # ignore comments
s/^\s+//; # ignore leading whitespace
s/\s+$//; # ignore trailing whitespace
next unless length;
my($var,$value) = split(/\s*=\s*/, $_, 2);
$prefs{$var} = $value;
}
close(CFGFILE);
}
Then there is this logging package. In line 27 of the script (where the error comes in) i see an "overwrite" invocation, but don't see anything referenceing overwrite in the logging.pl package - but not really sure if it matters. the parent script does not seem to write to any log file. I am not even sure if the filehandle LOGFILE is gtting created.
#!/usr/local/bin/perl
package Logging;
use File::Copy;
use warnings;
use strict;
my $timestamp = "";
my $filestamp = "";
# openLog(logfile name) - opens a log file
sub openLog {
my $file = shift;
my $rotate = shift;
# force a rotation if it exists.
if ( -e $file && $rotate eq "rotate" ) {
print "Warning: $file exists. Rotating.\n";
rotateLog($file);
}
getTime();
open(LOGFILE,">","$file") or warn "Error: Cannot open $file for writing: $!\n";
print LOGFILE "[$timestamp] - Normal - Opening log for $file.\n";
}
# rotateLog(log file) - rotate a log.
sub rotateLog {
my $file = shift;
getTime();
openLog("$file");
print LOGFILE "[$timestamp] - Warning - Rotating $file to $file.$filestamp.log";
closeLog($file);
move($file,$file-"$filestamp.log");
openLog($file);
}
time() - grab timestamp for the log.
sub getTime {
undef $timestamp;
undef $filestamp;
($sec,$min,$hour,$mday,$mon,$year) = (localtime(time))[0,1,2,3,4,5];
$sec = sprintf("%02d",$sec);
$min = sprintf("%02d",$min);
$hour = sprintf("%02d",$hour);
$mday = sprintf("%02d",$mday);
$year = sprintf("%04d",$year +1900);
$mon = sprintf("%02d",$mon +1);
$timestamp = "$mon-$mday-$year $hour:$min:$sec";
$filestamp = "$year$mon$mday$hour$min$sec";
}
just wondering - is there a problem with logging.pl calling something from dbconfig.pl in line 27? Like can one module call a value fron another module? besides using strict and warnings, and alot of print statements I do not know what my next debugging
step is. I have not idea how to check and see that the LOGFILE filehandle is getting created - if it does not error out, I can only suppose that it is. Like is there something extra I have to do to get the modules talking to each other?
I am not a scripting king - just the only guy in my row who can even begin to understand this stuff.

Not sure if this will effect things but ....
1) Packages need to return true, normal procedure is to end the file with the line:
1;
to ensure that.
2) Theres a comment in the logger package without the leading # which would cause compilation failure:
time() - grab timestamp for the log.
3) This line:
unshift #INC, "/production/fo/lib";
is adding the directory to search path for modules, make sure your logging2.pl file is actually in that location (it propably is otherwise you would get different errors, but worth a double check)

That looks all OK then.
For some reason although require "logging2.pl" works (there'd be an error if not) the sub-routines in it aren't loaded and available. Unlike the load of DBconfig2.pl which works OK (otherwise the call to Dbconfig::readconfigFile() would fail first).
Only difference I can see is the leading space on the package command in Logging2.pl, don't think that would matter though.
Could try calling openLog without the package prefix (Logging::) to see if its been loading into main from some reason and print the contents of %INC after the require statements to make sure its been loaded correctly?

Related

Use of uninitialized value $e2 in string eq & find: warning:

Hopefully you can help a scientist to decipher whats wrong with the code I'm trying to run to clean up some NGS results. The Perl file itself comes from https://github.com/mtokuyama/ERVmap, though I am posting the code below for reference. The other Perl files in the package work just fine and, while I have built a passing ability to use the linux terminal, Perl is a little beyond me.
The linux terminal I'm using is currently running: Ubuntu 16.04.6 LTS
This is the Perl code I'm trying to run using the following command line on linux as instructed by their GitHub page:
perl clean_htseq.pl ./ c c2 __
#!/usr/bin/env perl
#$Id: run_clean_htseq.pl,v 1.2 2015/03/02 17:24:35 yk336 Exp $
#
# create pbs file
#
use warnings;
use strict;
use File::Basename;
use POSIX;
my $dir = shift;
my $e1 = shift;
my $e2 = shift;
my $stop = shift;
die "$e1 eq $e2" if ($e1 eq $e2);
my $find = "find $dir -name \"*${e1}\"";
my $out = `$find`;
my #files = split(/\n/, $out);
for my $f (#files) {
my $o = $f;
$o =~ s/${e1}$/$e2/;
my $cmd = "./clean_htseq.pl $stop $f > $o";
print "$cmd\n";
system($cmd);
}
The first error that I had was that the _clean_htseq.pl_ wasn't found (line 30, already altered to solution) which i solved by adding the ./ in front of it and giving the software permission to use the script file.
My current issue with the code/command line is the following error:
Use of uninitialized value $e2 in string eq at ./clean_htseq.pl line 18.
find: warning: Unix filenames usually don't contain slashes (though pathnames do). That means that '-name ‘*./SRR7251667.c’' will probably evaluate to false all the time on this system. You might find the '-wholename' test more useful, or perhaps '-samefile'. Alternatively, if you are using GNU grep, you could use 'find ... -print0 | grep -FzZ ‘*./SRR7251667.c’'.
This has been tracked down to the "__" at the end of the command line, while i'm sure this is supposed to mean something to the script I removed it and resulted in the following error:
Use of uninitialized value $stop in concatenation (.) or string at clean_htseq.pl line 30.
./clean_htseq.pl ./SRR7251667.c > ./SRR7251667.c2
Use of uninitialized value $e1 in string eq at ./clean_htseq.pl line 18.
Use of uninitialized value $e2 in string eq at ./clean_htseq.pl line 18.
Use of uninitialized value $e1 in concatenation (.) or string at ./clean_htseq.pl line 18.
Use of uninitialized value $e2 in concatenation (.) or string at ./clean_htseq.pl line 18.
eq at ./clean_htseq.pl line 18.
An error occurs too when I remove the "." from "./" but it comes back with an error about not finding the _clean_htseq.pl_ file which is in the working directory.
Your problem seems to be here:
my $dir = shift;
my $e1 = shift;
my $e2 = shift;
my $stop = shift;
Outside of a subroutine, shift works on #ARGV—the array that holds the command line arguments. You shift four times, so you need four arguments:
perl clean_htseq.pl ./ c c2 __
You only seem to give it two, and $stop has no value (so you are giving it less than two):
./clean_htseq.pl $stop $f
You can't just remove arguments and hope things still work out. Likely you're going to have to look at the source to see what those things mean (which should motivate you as a scientist to use good variable names and document code—Best Practices for Scientific Computing).
A first step may be to set defaults. The defined-or operator does well here:
use v5.10;
my $dir = shift // 'default_dir';
my $e1 = shift // 'default_value';
my $e2 = shift // 'default_value';
my $stop = shift // 'default_value';
Or, you could just give up if there aren't enough arguments. An array in scalar context gives you the number of elements in the array (although it doesn't guarantee anything about their values):
die "Need four arguments!\n" unless #ARGV == 4;
There are various other improvements which would help this script, some of which I go through in the "Secure Programming Techniques" chapter in Mastering Perl. Taking unchecked user input and passing it to another program is generally not a good idea.

Error 500 when with perl cgi - but not any of the common pitfalls

I have a very tricky to diagnose perl problem, that has been seriously hampering my ability to maintain a perl/cgi website. It usually occurs when editing a script - after a change I get error 500, and then after I revert it it wont work again unless I delete the file and start from scratch, however I currently have a state which it can be reproduced by the following simple two scripts which show just how crazy this bug is:
file1.pl
#! /usr/bin/perl
use CGI::Carp qw(fatalsToBrowser warningsToBrowser);
print "content-type: text/html\n\nIt works";
file2.pl
#! /usr/bin/perl
use CGI::Carp qw(fatalsToBrowser warningsToBrowser);
print "content-type: text/html\n\nIt works";
(Ie... they're identical)
server.com/cgi-bin/file1.pl works
server.com/cgi-bin/file2.pl results in error 500
Both files have the same size and md5 hash.
Both have the same permissions (755) and the same owner and group.
Both are in the correct folder (hosting supplied cgi-bin).
Both were uploaded in text mode.
Both work with local perl interpreters.
If i rename file1 -> file3, file2 -> file1, and file3->file2, (ie swapping both files), now file2.pl works and file1.pl doesn't. So my guess is some state is attached to the files themselves.
If i edit the files in filezilla and re-upload (eg add some whitespace after a semicolon), same behaviour occurs with the re-uploaded files.
My error 500 page is set to auto-retry using a meta refresh (in case of out memory errors, etc), and it doesn't go away after countless refreshes. It doesn't seem to matter which ones is accessed first.
I do not have access to the http error_log on this hosting so do not know the reason for the failure. The error also occurs without the "use error messages to browser" diagnostic line.
Can anyone give me a hint as to what this could be and help me fix it?
What you describe can be either caused by some problem on your hosting provider side (some bad caching, or transparent proxies, or any other magic), or—and that is what I think it is—still caused by wrong file permissions or line breaks, even if your file manager reports that everything is good.
If I'm reading your description correctly you basically
can put a script and it will work, but
cannot edit it as it will stop working after that.
As you don't have shell access, just put the following small script to the same directory and run it (hope it will run as you are not going to edit it):
#!/usr/bin/perl
use strict;
use warnings;
print "Content-Type: text/plain\n\n";
opendir( my $dirh, "." );
my #files = grep { -f $_; } readdir $dirh;
closedir $dirh;
foreach my $file (#files) {
my #stat = stat $file;
my ($dev, $ino, $mode, $nlink, $uid, $gid, $rdev,
$size, $atime, $mtime, $ctime, $blksize, $blocks
) = stat($file);
my $octmode = sprintf "%04o", $mode & 07777;
print "$file\tmode=$octmode\tuid=$uid\tgid=$gid\tsize=$size\t";
if ( -r $file ) {
open( my $fh, $file );
my $firstline = <$fh>;
print $firstline =~ /\r\n/ ? "crlf\n" : "lf\n";
close $fh;
} else {
print "can't read\n";
}
}
It will show the real permissions, linebreaks, and size of the files—those taken from the server's filesystem, not which your FTP client shows.
Maybe it's worth adding MD5 or SHA1 hash calculation to this script but not sure if you have Digest::MD5 or Digest::SHA1 available.
If you see the same output for test1.pl and test2.pl, just go ahead and contact your hosting provider's support.
My guess: the files don't use the same newline convention.
You can check this (in a Unix shell) using the file command.
Not being able to inspect the errorlog is a big headache.
Nevertheless, I suspect that the cause is still most likely line endings. I would upload a script to examine all of your files like the following:
#!/usr/bin/env perl
use strict;
use warnings;
use autodie;
use CGI qw(header);
use CGI::Carp qw(fatalsToBrowser warningsToBrowser);
use File::stat;
print header('text/plain');
my $fmt = "%-15s %4s %4s %4s %7s %4s %4s\n";
printf $fmt, qw(File mode uid gid size lf crlf);
printf $fmt, map { '-' x $_ } $fmt =~ /(\d+)/g;
opendir my $dh, '.';
while ( my $file = readdir $dh ) {
next unless -f $file;
my $stat = stat $file;
my %chars;
my $data = do { local ( #ARGV, $/ ) = $file; <> };
$chars{$1}++ while $data =~ /(\R)/g;
printf $fmt, $file, sprintf( "%04o", $stat->mode & 07777 ), $stat->uid,
$stat->gid, $stat->size, map { $_ // 0 } #chars{ "\n", "\r\n" };
}
Outputs:
Content-Type: text/plain; charset=ISO-8859-1
File mode uid gid size lf crlf
--------------- ---- ---- ---- ------- ---- ----
env.cgi 0775 0 0 266 25 0
le.pl 0775 501 0 696 28 0
lineendings.pl 0755 501 0 516 30 0
mywiki.pl 0755 501 0 226947 0 6666
test.cgi 0755 0 0 2071 65 0
wiki.pl 0755 0 0 219231 6494 0
For additional testing, I would recommend executing each of the scripts using system and inspecting the error conditions if there are any.
I have had the same problem, got help from user BOC as below:
"You may have problem with encoding of characters. Some editors replace some characters by very close characters when you save files (for example " by “). Try changing editor (notepad++ works well on windows). – BOC"
I downloaded and used Notepad++ instead of Notepad and Winword; It works now for me.

perl File::Find - delete files with certain conditions, then delete parent folder if empty

I am attempting to use File::Find to 1) go thru a given folder and subfolders, deleting any files that are older than 30 days, and b) if the parent folder is empty after all the deletions, also delete it.
Here is my code:
use strict;
use warnings;
no warnings 'uninitialized';
use File::Find;
use File::Basename;
use File::Spec::Functions;
# excluding some home brew imports
# go into given folder, delete anything older than 30 days, and if folder is then empty, delete it
my $testdir = 'C:/jason/temp/test';
$testdir =~ s#\\#/#g;
open(LOG, ">c:/jason/temp/delete.log");
finddepth({ wanted => \&myWanted, postprocess => \&cleanupDir }, $testdir);
sub myWanted {
if ($_ !~ m/\.pdf$/i &&
int(-M $_) > 30
)
{
my $age = int(-M $_);
my $path = $File::Find::name;
print LOG "age : $age days - $path\n";
unlink($path);
}
}
sub cleanupDir {
my $path = $File::Find::dir;
if ( &folderIsEmpty($path) ) {
print LOG "deleting : $path\n";
unlink($path);
} else {
print LOG "$path not empty\n";
my #files = glob("$path/*");
foreach my $file(#files){
print LOG "\t$file\n";
}
}
}
I had thought that finddepth() would go to the bottom of the tree and work its way up, but that didn't happen. The script, run on an unzip of some ebook contents, did not delete directories that had subfolders, even though all the files were deleted.
age : 54 days - C:/jason/temp/test/mimetype
age : 54 days - C:/jason/temp/test/META-INF/container.xml
age : 54 days - C:/jason/temp/test/META-INF/ncx.xml.kindle
deleting : C:/jason/temp/test/META-INF
age : 54 days - C:/jason/temp/test/OEBPS/content.opf
age : 54 days - C:/jason/temp/test/OEBPS/cover.html
age : 54 days - C:/jason/temp/test/OEBPS/ncx.xml
age : 54 days - C:/jason/temp/test/OEBPS/pagemap.xml
age : 54 days - C:/jason/temp/test/OEBPS/t01_00_text.html
age : 54 days - C:/jason/temp/test/OEBPS/t02_00_text.html
age : 54 days - C:/jason/temp/test/OEBPS/t03_00_text.html
age : 54 days - C:/jason/temp/test/OEBPS/t04_00_text.html
age : 54 days - C:/jason/temp/test/OEBPS/t05_00_text.html
age : 54 days - C:/jason/temp/test/OEBPS/t06_00_text.html
age : 54 days - C:/jason/temp/test/OEBPS/t07_00_text.html
age : 54 days - C:/jason/temp/test/OEBPS/t08_00_text.html
age : 54 days - C:/jason/temp/test/OEBPS/t08_01_text.html
age : 54 days - C:/jason/temp/test/OEBPS/media/cover.jpg
age : 54 days - C:/jason/temp/test/OEBPS/media/flamlogo.gif
age : 54 days - C:/jason/temp/test/OEBPS/media/logolnmb.jpg
age : 54 days - C:/jason/temp/test/OEBPS/media/stylesheet.css
deleting : C:/jason/temp/test/OEBPS/media
C:/jason/temp/test/OEBPS not empty
C:/jason/temp/test/OEBPS/media
C:/jason/temp/test not empty
C:/jason/temp/test/META-INF
C:/jason/temp/test/OEBPS
looks like the C:/jason/temp/test/OEBPS/media/ was deleted, but that deletion was not registered by the time the preprocess func was called. Any ideas as to how to get this to work? thanks!
thanks,
bp
As Miller has commented, you can't unlink a directory. Also, File::Find does a chdir into a node's containing directory before it calls wanted. That means that, in the postprocess subroutine, you are trying to remove your currently working directory. Windows won't like that.
I would write it like this. I have tested it, but you should obviously be very careful with anything that deletes the contents of your disk storage.
use strict;
use warnings;
use autodie;
use File::Find;
use File::Spec::Functions;
my $testdir = 'C:\jason\temp\test';
open my $log, '>', 'C:\jason\temp\delete.log';
finddepth(\&wanted, $testdir);
sub wanted {
my $full_name = canonpath $File::Find::name;
if (-f) {
my $age = int(-M);
unless ( /\.pdf\z/ or $age <= 30) {
print $log "Age: $age days - $full_name\n";
unlink;
}
}
elsif (-d) {
my #contents = do {
opendir my ($dh), $_;
grep { not /\A\.\.?\z/ } readdir $dh;
};
rmdir unless #contents;
}
}
I suspect you aren't actually deleting the directory. From the documentation for unlink:
Note: unlink will not attempt to delete directories unless you are superuser and the -U flag is supplied to Perl. Even if these conditions are met, be warned that unlinking a directory can inflict damage on your filesystem. Finally, using unlink on directories is not supported on many operating systems. Use rmdir instead.
I never liked File::Find because it just is a mess. It swallows up your entire program because it wants everything to be in your wanted subroutine. Plus, I don't like the fact that half of my code is scattered all over the place. However, what other tools come standard with every installation of Perl. I have to make do.
I prefer to toss all of my files into an array. It keeps the code clean. My find just finds. I do the rest of my processing elsewhere. I also embed my wanted subroutine embedded in my find command. It keeps everything in one place.
Also, you can't use unlink to remove a directory. Use remove_tree from File::Path. That's a standard module. You can also use readdir to see how many subdirectories a directory has. That's a good way to check to see if it's empty:
use strict;
use warnings;
use feature qw(say);
use File::Find;
use File::Path qw(make_path remove_tree);
my $testdir = 'C:/jason/temp/test';
my $mdate_limit = 30;
my #files; # We'll store the files here
my %dirs; # And we'll track the directories that my be empty
#
# First find the files
#
find ( sub {
return unless -f; # We want just files.
return if -M < $mdate_limit; # Skip if we've modified since $mdate_limit days
push #files, $File::Find::name; # We're interested in this file,
$dirs{$File::Find::dir} = 1; # and the directory that file is in
}, $testdir );
#
# Delete the files that you've found
#
unlink #files;
#
# Go through the directories and see which are empty
#
for my $dir ( sort keys %dirs ) {
opendir my $dir_fh, $dir or next; # We'll skip bad reads
my #dir_files = readdir $dir_fh;
close $dir_fh;
if ( #dir_files <= 2 ) { # Directory is empty if there's only "." and ".." in it
remove_tree( $dir )
or warn qq(Can't remove directory "$dir"\n);
}
}
Notice that I've embedded my wanted routine:
find ( sub {
return unless -d; # We want just files.
return if -M < $mdate_limit; # File hast been modified in the $mdate_limit days
push #files, $Find::File::name; # We're interested in this file
$dirs{$Find::File::dir} = 1; # The directory that file is in
}, $testdir );
The alternative is this:
file (\&wanted, $testdir);
sub wanted {
return unless -d; # Okay...
return if -M < $mdate_limit; # Um... Where's $mdate_limit defined?
push #files, $Find::File::name; # And #files?
$dirs{$Find::File::dir} = 1; # And %dirs?
}
The problem is that my wanted subroutine contains three global variables. And, it's possible for my find command to get separated from my wanted subroutine. In 3 months time, you'll have to search all over your code to find that wanted routine.
And, when you do see that wanted subroutine, there are those three mysterious global variables. Where are they defined? Is that a bug?
By combining the subroutine with my find, I guarantee that the subroutine the find command needs won't drift away from my find. Plus, it hides the globalness of those three variables embedded in my subroutine.
There is nothing preventing me from deleting the files inside the find command. It's usually not a good idea to change the directory structure while searching it, but this should be fine.
However, I like my find command to just find the files I'm interested in. I don't want 1/2 of my program stuffed in there. It becomes a maintenance nightmare. I'll put up with a bit of inefficiency. It might take a full second or two to load my #files array with a million of files, but I'll spend a lot longer than that as soon as I have to debug my program.

Open the latest log file and print lines later than a certain timestamp

I'm writing a Perl script and I need to capture some lines from a garbage collection log and write them to a file.
The log is located on a remote host and I'm connecting using the Net::OpenSSH module.
I need to read the latest log file available.
In the shell I can locate the latest log with the following commands:
cd builds/5.7.1/5.7.1.126WRF_B/jboss-4.2.3/bin
ls -lat | grep '.log$' | tail -1
Which will return the latest log:
-rw-r--r-- 1 load other 2406173 Jul 11 11:53 18156.stdout.log
So in Perl I'd like to be able write something that locates and opens that log for reading.
When I have that log file, I want to print all lines that have a timestamp greater than a specified time. The specified timestamp is a $Runtime variable subtracted from the latest log message time.
Here are the last messages of the garbage collection log:
...
73868.629: [GC [PSYoungGen: 941984K->14720K(985216K)] 2118109K->1191269K(3065984K), 0.2593295 secs] [Times: user=0.62 sys=0.00, real=0.26 secs]
73873.053: [GC [PSYoungGen: 945582K->12162K(989248K)] 2122231K->1189934K(3070016K), 0.2329005 secs] [Times: user=0.60 sys=0.01, real=0.23 secs]
So if $Runtime had a value of 120 seconds, I would need to print all the lines from timestamp (73873.053 - 120) seconds up.
In the end my script would look something like this...
open GARB, ">", "./report/archive/test-$now/GC.txt" or die "Unable to create file: $!";
my $ssh2 = Net::OpenSSH->(
$pathHost,
user => $pathUser,
password => $pathPassword
);
$ssh2->error and die "Couldn't establish SSH connection: ". $ssh2->error;
# Something to find and open the log file.
print GARB #Something to return certain lines.
close GARB;
I realize this is somewhat similar to this question, but I can't think of a way to tailor it to what I'm looking for. Any help is greatly appreciated!
Find the latest file and feed it to perl:
LOGFILE=`ls -t1 $DIR | grep '.log$' | head -1`
if [ -z $LOGFILE ]; then
echo "$0: No log file found - exiting"
exit 1;
fi
perl myscript.pl $LOGFILE
The pipe in the first line lists the file in the directory, name-only, in one column, most recent first; filters for log files, and then only returns the first one.
I have no idea how to translate your timestamps into something I can understand and do math and comparisons upon. but in general:
$threshold_ts = $time_specified - $offset;
while (<>) {
my ($line_ts) = split(/\s/, $_, 2);
print if compare_time_stamps($line_ts, $threshold_ts);
}
Writing the threshold manipulation and comparison is left as an exercise for the reader.
I think that the page for Net::OpenSSH gives a pretty good baseline for this:
my ($rout, $pid) = $ssh->pipe_out("cat /tmp/foo") or
die "pipe_out method failed: " . $ssh->error;
while (<$rout>) { print }
close $rout;
But instead, you want to do some discarding work:
my ($rout, $pid) = $ssh->pipe_out("cat /tmp/foo") or
die "pipe_out method failed: " . $ssh->error;
my $line;
while ( $line = <$rout>
and substr( $line, 0, index( $line, ':' )) < $start
) {}
while ( $line = <$rout>
and substr( $line, 0, index( $line, ':' )) <= $start + $duration
) {
print $line;
}
close $rout;
Here's an untested approach. I've not used Net::OpenSSH so there might be better ways to do it. I'm not even sure it works. What does work is the parsing part which I have tested.
use strict; use warnings;
use Net::OpenSSH;
my $Runtime = 120;
my $now = time;
open my $garb, '>',
"./report/archive/test-$now/GC.txt" or die "Unable to create file: $!";
my $ssh2 = Net::OpenSSH->(
$pathHost,
user => $pathUser,
password => $pathPassword
);
$ssh2->error and die "Couldn't establish SSH connection: ". $ssh2->error;
# Something to find and open the log file.
my $fileCapture = $ssh2->capture(
q~ls -lat builds/5.7.1/5.7.1.126WRF_B/jboss-4.2.3/bin |grep '.log$' |tail -1~
);
$fileCapture =~ m/\s(.+?)$/; # Look for the file name
my $filename = $1; # And save it in $filename
# Find the time of the last log line
my $latestTimeCapture = $ssh2->capture(
"tail -n 1 builds/5.7.1/5.7.1.126WRF_B/jboss-4.2.3/bin/$filename");
$latestTimeCapture =~ m/^([\d\.]+):/;
my $logTime = $1 - $Runtime;
my ($in, $out, $pid) = $ssh2->open2(
"builds/5.7.1/5.7.1.126WRF_B/jboss-4.2.3/bin/$filename");
while (<$in>) {
# Something to return certain lines.
if (m/^([\d\.]+):/ && $1 > $logTime) {
print $garb $_; # Assume the \n is still in there
}
}
waitpid($pid);
print $garb;
close $garb;
It uses your ls line to look up the file with the capture method. It then opens a pipe through the SSH tunnel to read that file. $in is a filehandle to that pipe which we can read.
Since we are going to process the file line by line, starting at the top, we need to first grab the last line to get the last timestamp. That is done with tail and, again, the capture method.
Once we have that, we read from the pipe line by line. This now is a simple regex (the same used above). Grab the timestamp and compare it to the time we have set earlier (minus the 120 seconds). If it is higher, print the line to the output filehandle.
The docs say we have to use waitpid on the $pid returned from $ssh2->open2 so it reaps the subprocess, so we do that before closing our output file.
You will need to either keep an accumulator containing all the lines (more memory) or iterate through the log more than once (more time).
With an accumulator:
my #accumulated_lines;
while (<$log_fh>) {
push #accumulated_lines, $_;
# Your processing to get $Runtime goes here...
if ($Runtime > $TOO_BIG) {
my ($current_timestamp) = /^(\d+(?:\.\d*))/;
my $start_timestamp = $current_timestamp - $Runtime;
for my $previous_line (#accumulated_lines) {
my ($previous_timestamp) = /^(\d+(?:\.\d*))/;
next unless $previous_timestamp <= $current_timestamp;
next unless $previous_timestamp >= $start_timestamp;
print $previous_line;
}
}
}
Or you can iterate through the log twice, which is similar, but without the nested loop. I've assumed you might have more than one of these spans in your log.
my #report_spans;
while (<$log_fh>) {
push #accumulated_lines, $_;
# Your processing to get $Runtime goes here...
if ($Runtime > $TOO_BIG) {
my ($current_timestamp) = /^(\d+(?:\.\d*))/;
my $start_timestamp = $current_timestamp - $Runtime;
push #report_spans, [ $start_timestamp, $current_timestamp ];
}
}
# Don't bother continuing if there's nothing to report
exit 0 unless #report_spans;
# Start over
seek $log_fh, 0, 0;
while (<$log_fh>) {
my ($previous_timestamp) = /^(\d+(?:\.\d*))/;
SPAN: for my $span (#report_spans) {
my ($start_timestamp, $current_timestamp) = #$span;
next unless $previous_timestamp <= $current_timestamp;
next unless $previous_timestamp >= $start_timestamp;
print; # same as print $_;
last SPAN; # don't print out the line more than once, if that's even possible
}
}
If you might have overlapping spans, the latter has the advantage of not showing the same log lines twice. If you don't have overlapping spans, you could optimize the top one by resetting the accumulator every time you output:
my #accumulator = ();
which would save memory.
Use SFTP to access the remote filesystem. You can use Net::SFTP::Foreign (alone or via Net::OpenSSH).
It will allow you to list the contents of the remote filesystem, pick the file you want to process, open it and manipulate it as a local file.
The only tricky thing you would need to do is to read lines backward, for instance reading chunks of the file starting from the end and breaking them in lines.

How can I pipe input into a Java command from Perl?

I need to run a string through a Java program and then retrieve the output. The Java program accepts the string through standard input. The following works:
my $output = `echo $string | java -jar java_program.jar`;
There is one problem: $string could be just about anything. Any thoughts on a good solution to this problem?
I suggest you to look at IPC::Run3 module. It uses very simple interface and allow to get STDERR and STDOUT. Here is small example:
use IPC::Run3;
## store command output here
my ($cmd_out, $cmd_err);
my $cmd_input = "put your input string here";
run3([ 'java', '-jar', 'java_program.jar'], \$cmd_input, \$cmd_out, \$cmd_err);
print "command output [$cmd_out] error [$cmd_err]\n";
See IPC::Run3 comparation with other modules.
If you can use CPAN modules (and I'm assuming most people can), look at Ivan's answer on using IPC::Run3. It should handle everything you need.
If you can't use modules, here's how to do things the plain vanilla way.
You can use a pipe to do your input, and it will avoid all those command line quoting issues:
open PIPE, "| java -jar java_program.jar";
print PIPE "$string";
close(PIPE);
It looks like you actually need the output of the command, though. You could open two pipes with something like IPC::Open2 (to and from the java process) but you risk putting yourself in deadlock trying to deal with both pipes at the same time.
You can avoid that by having java output to a file, then reading from that file:
open PIPE, "| java -jar java_program.jar > output.txt";
print PIPE "$string";
close(PIPE);
open OUTPUT, "output.txt";
while (my $line = <OUTPUT>) {
# do something with $line
}
close(OUTPUT);
The other option is to do things the other way around. Put $string in a temporary file, then use it as input to java:
open INPUT, "input.txt";
print INPUT "$string";
close(INPUT);
open OUTPUT, "java -jar java_program.jar < input.txt |";
while (my $line = <OUTPUT>) {
# do something with the output
}
close(OUTPUT);
Note that this isn't the greatest way to do temporary files; I've just used output.txt and input.txt for simplicity. Look at the File::Temp docs for various cleaner ways to create temporary files more cleanly.
Have you looked into IPC::Run?
Syntax similar to this might be what you are looking for:
use IPC::Run qw( run );
my $input = $string;
my ($out, $err);
run ["java -jar java_program.jar"], \$input, \$out, \$err;
Create a pipeline just like your shell would.
Here's our scary string:
my $str = "foo * ~ bar \0 baz *";
We'll build our pipeline backwards, so first we gather the output from the Java program:
my $pid1 = open my $fh1, "-|";
die "$0: fork: $!" unless defined $pid1;
if ($pid1) {
# grab output from Java program
while (<$fh1>) {
chomp;
my #c = unpack "C*" => $_;
print "$_\n => #c\n";
}
}
Note the special "-|" argument to Perl's open operator.
If you open a pipe on the command '-' , i.e., either '|-' or '-|' with 2-arguments (or 1-argument) form of open(), then there is an implicit fork done, and the return value of open is the pid of the child within the parent process, and 0 within the child process … The filehandle behaves normally for the parent, but i/o to that filehandle is piped from/to the STDOUT/STDIN of the child process.
The unpack is there to peek into the contents of the data read from the pipe.
In your program, you'll want to run the Java program, but the code below uses a reasonable facsimile:
else {
my $pid2 = open my $fh2, "-|";
die "$0: fork: $!" unless defined $pid2;
if ($pid2) {
$| = 1;
open STDIN, "<&=" . fileno($fh2)
or die "$0: dup: $!";
# exec "java", "-jar", "java_program.jar";
# simulate Java program
exec "perl", "-pe", q(
BEGIN { $" = "][" }
my #a = split " ", scalar reverse $_;
$_ = "[#a]\n";
);
die "$0: exec failed";
}
Finally, the humble grandchild simply prints the scary string (which arrives on the standard input of the Java program) and exits. Setting $| to a true value flushes the currently selected filehandle and puts it in unbuffered mode.
else {
print $str;
$| = 1;
exit 0;
}
}
Its output:
$ ./try
[*][zab][][rab][~][*][oof]
=> 91 42 93 91 122 97 98 93 91 0 93 91 114 97 98 93 91 126 93 91 42 93 91 111 111 102 93
Note that the NUL survives the trip.
The builtin IPC::Open2 module provides a function to handle bidirectional-piping without an external file.