open3-error when trying to run a script using PBS::Client - perl

I'm trying to get the following perl script to work. First, a fastq-file is read, this file is then used to be analysed by a number of programs.
Code:
use warnings;
use PBS::Client;
$directory = $ARGV[0];
opendir(DIR, $directory);
#files=();
while ($file = readdir(DIR)) {
push(#files, $file);
}
closedir(DIR);
#fastq_files = grep(/fastq/, #files);
$client = PBS::Client->new();
foreach $fastq (#fastq_files){
#commands = ();
$wd = "/store/www/labresults_QC/snRNA_sequence_analyser/".$ARGV[0];
$name = $fastq."_process_map";
$queue = "system";
$wallt = "72:00:00";
chomp($fastq);
$fastq =~ /.+[^\.fastq]/;
push (#commands, "/opt/fastx_toolkit-0.0.13.2/bin/fastq_quality_filter -q 30 -p 80 -i " . $fastq . " -o ";
push (#commands, "/opt/fastx_toolkit-0.0.13.2/bin/fastx_clipper -i " . $& . "_qc.fastq -o " . $& . "_qc_clipped.fastq -v -l 15 -a TGGAATTCTCGGGTGCCAAGG -Q33\n");
push (#commands, "/opt/fastx_toolkit-0.0.13.2/bin/fastx_collapser -i " . $& . "_qc_clipped.fastq -o " . $& . "_qc_clipped_collapse.fa -v -Q33\n");
push (#commands, "/opt/bowtie-1.0.0/bowtie -f /opt/genomes/9606/GRCh37/bowtie/GRCh37 " . $& . "_qc_clipped_collapse.fa " . $& . "_mapped.sam -k 100 -n 0 -l 25 --best");
$job = PBS::Client::Job -> new(
wd => $wd,
queue => $queue,
name => $name,
wallt => $wallt,
cmd => [[#commands]]);
$client -> qsub($job);
}
However, when trying to execute through a Linux commandline, it gives this error message:
open3: exec of /store/www/labresults_QC/snRNA_sequence_analyser/data/data_raw/test_run/n8XyeYIkfv failed at /store/bin/perl_libs/lib/perl5//PBS/Client.pm line 150
The error message points to this piece of code in the PBS Client module:
#-------------------------------------------------------------------
# Thanks to Sander Hulst
sub call_qsub
{
my #args = #_;
# If the qsub command fails, for instance, pbs_server is not running,
# PBS::Client's qsub should not silently ignore. Disable any reaper
# functions so the exit code can be captured
use Symbol qw(gensym);
use IPC::Open3;
my $stdout = gensym();
my $stderr = gensym();
{
local $SIG{CHLD} = sub{};
my $pid = open3(gensym, $stdout, $stderr, #args); # This is line 150
waitpid($pid,0);
}
confess <$stderr> if ($?);
return <$stdout>;
}
#-------------------------------------------------------------------
Anyone got a clue what this means?
EDIT
After some investigation it seems that this line is failing: $client -> qsub($job);
but I don't know why. Any ideas what I'm doing wrong?
FINAL EDIT:
So, we finally found the real cause of the problem. It turned out something went wrong in the latest installation of PBS::Client we did. So we reverted to an older version, and the problem was gone!

The module generates a script then tries to execute it without having made it executable. Workaround:
use PBS::Client qw( );
BEGIN {
my $orig_genScript = \&PBS::Client::genScript;
my $new_genScript = sub {
my $script_qfn = $orig_genScript->(#_);
chmod(0700, $script_qfn) or die $!;
return $script_qfn;
};
no warnings 'redefine';
*PBS::Client::genScript = $new_genScript;
}

Related

Running sftp commands using Expect.pm

I am trying to download few files from specific folders using sftp utility and am doing it through Expect.pm . Below is the code for that :
use strict;
use Expect;
my $userid = `whoami`;
chomp($userid);
my $Password = "<password>";
my $command = "sftp " . "$userid" . "\#<server-name>";
my $spawn_ok = 0;
my $timeout = 10;
print "$command \n";
my $exp = new Expect();
$exp->log_file("FTPLOGFILE.txt");
$exp->spawn("$command") or die "Cannot spawn $command: $!\n";
$exp->log_stdout(0);
$exp->expect($timeout,
[ 'Password:',
sub {
$spawn_ok = 1;
my $fh = shift;
print "Sending password \n";
$fh->send("$Password\r");
exp_continue;
}
],
[ 'sftp> ',
sub {
my $fh = shift;
$spawn_ok = 3;
print "Downloading cfg files \n";
$fh->send("get /home/cfg/*.cfg /tmp/ACC_CCM_CFG","\n");
$fh->send("bye","\n");
exp_continue;
}
]
The problem is apart from downloading files into the above folder : /tmp/ACC_CCM_CFG i also want to run the below command inside the same sftp session :
get /home/appl/*.pl /tmp/ACC_CCM_APPL
But i can't do that since the regular expression for the expect function would be same (sftp>) . How do i run a series of commands inside the same sftp session using expect if the regex of the prompt does not change .
Please throw some light on the above since i can't find any solution .
Any reason for using Expect and system sftp command for this? Using Net::SFTP module would make the job much more easier and clean.

How to make Apache "sleep" for a few seconds

I want my server to stop for a few seconds and then start again.
I am trying to use sleep(5).
Will this help?
I've tried with a Perl script containing:
if($mech=~ m/(Connection refused)/) {print "SLEEPING\n";sleep(5);redo;}
A server hosts web pages. A client connects to a server. A Perl script using WWW::Mechanize is a client.
I suspect you're trying to do the following:
my $retries = 3;
my $resp; # response goes here
while ( $retries-- > 0 ) {
$resp = $ua->get( "http://www.google.com/" );
if ( ! $resp->is_success ) {
warn( "Failed to get webpage: " . $resp->status_line );
sleep( 5 );
next; # continue the while loop
}
last; # success!
}
if ( $retries == 0 ) {
die( "Too many retries!" );
}
# old versions of LWP::UserAgent didn't have decoded_content()
my $content = $resp->can('decoded_content') ?
$resp->decoded_content : $resp->content;
print( $content );
UPDATE
Here is the code you supplied in a comment:
use WWW::Mechanize;
my $mech = new WWW::Mechanize;
eval {
$mech->get($url);
};
if($#) {
print STDERR "Burst\n";
print STDERR Dumper($mech);
# my $var=$mech;
# print STDERR "var $var\n";
# if($mech=~ m/(Connection refused)/)
# {
# print "SLEEPING\n";
# sleep(5);
# redo;
# }
From perdoc -f redo:
The "redo" command restarts the loop block without evaluating the conditional again. The "continue" block, if any, is not executed. If the LABEL is omitted, the command refers to the innermost enclosing loop.
Seeing as you haven't put your code in a loop the call to redo doesn't have any effect.

Perl sftp downloads with Net::SFTP::Foreign

Im a beginner. I have written a perl script which does the following
-Create a directory under “/x01/abcd/abc_logs/abcd_Logs” by the current date, in the format of “YYYYMMDD” if it has not already been created.
i.e: if the script is run on “01st of jan 2013”, the directory “20130101” will be created under the said path. So whenever there is a need to inspect the logs always look for a directory by the current date.
-Check if the log file(s) have already been downloaded earlier within the same day, and if not log(s) will be downloaded to the TODAY’s directory.
Im having a hard time, coming up with a solution to print a message when there are no files in the share. This is of course when the user specify 2 or more files that are not there in the share. I know that this happens because there is a "die" statement in the "sub get_LOGS". I just cannot seem to understand how to return a message when all the files I specify do not happen to be in the share.
usage of this script is as follows
./abc_logs ....<file(n)>
following is the script.
my $LOGS_LOCAL_PATH = "/x02/abc/abcba2/";
chomp $LOGS_LOCAL_PATH;
my $LOGS_REM_PATH = "/x01/INT/abc/vabc2/";
chomp $LOGS_REM_PATH;
my $TODAY = `date +%Y%m%d`;
chomp $TODAY;
my #GETLOOP = #ARGV;
unless ($#ARGV >= 0) {
print "\nUsage: gtp_logs.pl <file1> <file2> <file3>.....<file(n)>\n\n";
exit;
}
system("clear");
unless ( -d "$LOGS_LOCAL_PATH"."$TODAY") {
print "Directory \"$TODAY\" doesn't exist. So creating the directory..!\n";
print "OK..Done.....!\n\n";
system("mkdir $LOGS_LOCAL_PATH/$TODAY");
}
else {
print "Directory already exists. Logs will be downloaded to ==> \"$LOGS_LOCAL_PATH$TODAY\".....!\n\n";
}
# if_DOWNLOADED($LOGS_LOCAL_PATH,$TODAY,#GETLOOP);
chdir("$LOGS_LOCAL_PATH"."$TODAY") || die "cannot cd to ($!)";
foreach my $GETL (#GETLOOP) {
my $is_downloaded = if_DOWNLOADED($LOGS_LOCAL_PATH,$TODAY,$GETL);
if(!$is_downloaded)
{
get_LOGS("172.25.70.221","abc","abc2","/x01/INT/abc",$GETL);
print "File \"$GETL\" downloaded to ==> \"$LOGS_LOCAL_PATH$TODAY\"\n\n";
}
else
{
print "File \"$GETL\" has already been Downloaded to ==> \"$LOGS_LOCAL_PATH$TODAY\"\n\n";
}
}
sub get_LOGS {
my $LOG_HOST = shift;
my $REM_USER = shift;
my $REM_PASSW = shift;
my $REM_PATH = shift;
my $REM_FILE = shift;
print "Connecting to the sftp share! Please wait....!\n";
my $sftp = Net::SFTP::Foreign->new($LOG_HOST, user => $REM_USER, password => $REM_PASSW);
$sftp->setcwd($REM_PATH) or die "unable to change cwd: " . $sftp->error;
print "OK. On the share! Downloading the file \"$REM_FILE\"...................!\n\n\n\n";
$sftp->error and die "Problem connecting to the share...!!!! " . $sftp->error;
$sftp->get($REM_FILE) or die "File does not seem to be present on the remote share. Please re-request..!!!" . $sftp->error;
return $REM_FILE;
}
sub if_DOWNLOADED {
my $DWD_FILE_PATH = shift;
my $DWD_DIR = shift;
my $DWD_FILE = shift;
if (-e "$DWD_FILE_PATH/$DWD_DIR/$DWD_FILE")
{
return 1;
}
else
{
return 0;
}
}
Please can someone help me finding a solution to this matter? Please try to use the same script and modify.
/V
Some comments to your code:
Use strict and warnings in order to catch lots of errors early.
Read some book on style (i.e. Damian Conway's Perl Best Practices). But in any case try to be consistent when naming variables, subroutines, and everything and also with their case.
When you have to use some calculated value in several places, try to calculate it once and save it in a variable.
Don't use subroutines for trivial things.
You don't need to call chomp on variables you have defined and that don't have a "\n" character at the end.
Opening a new SFTP connection for every file transfer is very inefficient. You can open just one at the beginning and use it for all the transfers.
And now, a simplified version of your script:
#!/usr/bin/perl
use strict;
use warnings;
my $host = "172.25.70.221";
my $user = "abc";
my $password = "abc1234321";
my $LOGS_LOCAL_PATH = "/x02/ABC/abc2";
my $LOGS_REM_PATH = "/x01/INT/abc/vim";
my $TODAY = `date +%Y%m%d`;
chomp $TODAY;
my $TODAY_LOCAL_PATH = "$LOGS_LOCAL_PATH/$TODAY";
my #files = #ARGV;
#files or die "\nUsage: gtp_logs.pl <file1> <file2> <file3>.....<file(n)>\n\n";
system("clear");
if ( -d $TODAY_LOCAL_PATH) {
print "Directory already exists. Logs will be downloaded to ==> \"$TODAY_LOCAL_PATH\".....!\n\n";
}
else {
print "Directory \"$TODAY\" doesn't exist. So creating the directory..!\n";
mkdir "$TODAY_LOCAL_PATH" or die "unable to create directory: $!\n";
print "OK..Done.....!\n\n";
}
chdir $TODAY_LOCAL_PATH or die "cannot cd to ($!)\n";
my $sftp = Net::SFTP::Foreign->new($host, user => $user, password => $password);
$sftp->error
and die "Problem connecting to the share...!!!! " . $sftp->error;
my $ok = 0;
my $failed = 0;
foreach my $file (#files) {
if (-e "$TODAY_LOCAL_PATH/$file") {
print "File \"$file\" has already been Downloaded to ==> \"$TODAY_LOCAL_PATH\"\n";
}
else {
if ($sftp->get("$LOGS_REM_PATH/$file")) {
print "File \"$file\" downloaded to ==> \"$TODAY_LOCAL_PATH\"\n";
$ok++;
}
else {
print "Unable to download file \"$file\" : " . $sftp->error . "\n";
$failed++;
}
}
}
print "$ok files have been downloaded, $failed files failed!\n\n";

Perl SNMP trap generator for scale testing?

I've hacked the script below together to let me generate traps to a test server. What I really need is something that will generate traps at a large scale so that I can check my tools on the receiving end to find out where the bottleneck is, such as UDP, Net::SNMP, Perl, etc.
I had hoped this script would let me generate something like 10k events/second but I am sadly mistaken.
Does anyone know if I can do this in Perl or have a suggestion of another way to do it?
#! /usr/bin/perl
use strict;
use warnings;
use Log::Fast;
use FindBin;
use Getopt::Long;
use File::Basename;
use Cwd qw(abs_path);
my $ROOT_DIR = abs_path("$FindBin::Bin/..");
use POSIX qw/strftime/;
use Net::SNMP qw(:ALL);
use Time::HiRes qw( time sleep );
#FIXME - I had to add below for Perl 5.10 users.
# on Perl 5.10, I would get the following when running:
# perl -e"autoflush STDOUT, 1;"
# Can't locate object method "autoflush" via package "IO::Handle" at -e line 1.
use FileHandle;
# Create default logger, will reconfigure it as soon as we read configuration from database
my $log = Log::Fast->global();
my $myname = $0;
$myname =~ s{.*/}{}; # leave just program name without path
# Command line options
my $options = {
debug => 0,
verbose => 0,
logfile => "./$myname.log",
help => 0,
community => "public",
trapsource => "127.0.0.1",
timelimit => 1,
};
sub usage_and_exit {
my ($exit_code) = #_;
print STDERR qq{
This program is used to generate SNMP traps to a specified host at a specified rate
Usage: $myname [-o --option]
-h : this (help) message
-d : debug level (0-5) (0 = disabled [default])
-v : Also print results to STDERR
-l : log file (defaults to local dir
-r : Rate (events/sec)
-ts : host to generate messages FROM
-td : host to generate messages TO
-tl : Run for this many seconds (default 1)
-c : community
Example: $myname -td 192.168.28.29 -r 1 -tl 5 -v
};
exit($exit_code);
}
GetOptions(
'debug|d=i' => \$options->{debug},
'help|h!' => \$options->{help},
'verbose|v!' => \$options->{verbose},
'logfile|l=s' => \$options->{logfile},
'rate|r=i' => \$options->{rate},
'trapsource|ts=s' => \$options->{trapsource},
'trapdest|td=s' => \$options->{trapdest},
'community|c=s' => \$options->{community},
'timelimit|tl=i' => \$options->{timelimit},
) or usage_and_exit(1); # got some invalid options
if ( $options->{help} ) {
usage_and_exit(0);
}
# Reconfigure log to use logfile (as we finally got it from $settings), also
# set proper level and output based on $options{verbose} and $options{debug}
setup_log();
# Finally we are initialized, announce this to the world :-)
$log->INFO("Program initialized successfully");
my $date = strftime "%Y-%m-%d %H:%M:%S", localtime;
# start func
my $period = 1 / $options->{rate};
my $start = time();
my $limit = time() + $options->{timelimit};
my $total = $options->{rate} * $options->{timelimit};
$log->INFO("Generating $options->{rate} trap(s) every second for $options->{timelimit} seconds (1 every $period seconds, $total total events)");
while($start < $limit) {
my $elapsed = time() - $start;
if ($elapsed < $period) {
sleep($period - $elapsed);
my ($session, $error) = Net::SNMP->session(
-hostname => $options->{trapdest},
-community => $options->{community},
-port => SNMP_TRAP_PORT, # Need to use port 162
-version => 'snmpv2c'
);
if (!defined($session)) {
$log->INFO("ERROR: %s.", $error);
exit 1;
}
my $result = $session->snmpv2_trap(
-varbindlist => [
'1.3.6.1.2.1.1.3.0', TIMETICKS, 600,
'1.3.6.1.6.3.1.1.4.1.0', OBJECT_IDENTIFIER, '1.3.6.1.4.1.326',
'1.3.6.1.6.3.18.1.3.0', IPADDRESS, $options->{trapsource}
]
);
if (!defined($result)) {
$log->INFO("ERROR: %s.", $session->error());
} else {
$log->INFO("SNMPv2-Trap-PDU sent from $options->{trapsource} to $options->{trapdest}.");
}
} else {
$start = time();
}
}
#-------------------------------------------
# There should only be subs from here down
#-------------------------------------------
# =================================================================================================
# Helper functions
# =================================================================================================
# commify not used yet
sub commify {
my $text = reverse $_[0];
$text =~ s/(\d\d\d)(?=\d)(?!\d*\.)/$1,/g;
return scalar reverse $text;
}
sub setup_log {
my $log_dir = dirname($options->{logfile});
# Create log dir, and build log path if not provided by command line option
if ( !-d $log_dir ) {
mkdir( $log_dir, 0755 ) or die("mkdir $log_dir: $!");
}
if ( !$options->{logfile} ) {
$options->{logfile} = $log_dir . "/" . basename( $0, '.pl' ) . '.log';
}
my $log_options = {};
# Set up output to file or both file and stderr
if ( $options->{verbose} ) {
# make multiplexer FH sending data both to file and STDERR
open( my $fh, '>>:tee', $options->{logfile}, \*STDERR )
or die("$options->{logfile}: $!");
$fh->autoflush(1);
$log_options->{fh} = $fh;
}
else {
open( my $fh, '>>', $options->{logfile} ) or die("$options->{logfile}: $!");
$log_options->{fh} = $fh;
}
# Setup extra information to put in every log line, depending on debug level
if ( $options->{debug} > 1 ) {
$log_options->{prefix} = "%D %T %S [%L] ";
}
else {
$log_options->{prefix} = "%D %T [%L] ";
}
$log_options->{level} = $options->{debug} > 0 ? 'DEBUG' : 'INFO';
$log->config($log_options);
$SIG{__WARN__} = sub {
my $msg = shift;
$msg =~ s/\n//;
$log->WARN($msg);
};
$log->INFO("Starting logging to $options->{logfile} with pid $$");
}
sub DEBUG {
my ( $level, #log_args ) = #_;
if ( $options->{debug} >= $level ) {
$log->DEBUG(#log_args);
}
}
Perhaps use something like Parallel::ForkManager ? In addition, with specific regard to testing scalability of your SNMP collector, you'll probably be interested in the use case of receiving the traps from many HOSTS, not just a single host sending traps at a high rate. For that, you might want to look at using pssh.
One problem might be the slowness of Net::SNMP in pure-perl - perhaps exectuting snmptest or snmptrap via the shell might be faster ? Worth a try.

Perl find file on remote machine

I have a scenario where i need to list all the directories under msystem on a remote machine which contains a log.txt.If it is found then get the list using ll command from the msystem directory file.How can achieve this
this is the directory structure
msystem
dir1 dir2/info/log.txt dir3/ dir4/info/log.txt
my $ssh = Net::SSH::Perl->new($hostname, protocol => '1,2', debug => 0, interactive => 1);
$ssh->login($username, $password);
($stdout,$stderr,$exit) = $ssh->cmd("$check_lock_file");
if((defined $stderr) && ($stderr =~ /No such file or directory/))
{
($stdout,$stderr,$exit) = $ssh->cmd("What command to be used and get the ouput");
if((defined $stderr) && ($stderr =~ /No such file or directory/))
{
print ""Error;
print "$stderr";
exit;
}
elsif($exit eq '0')
{
print "dir2 dir4";
}
}
You can also do that using SFTP:
use Net::SFTP::Foreign;
my $sftp = Net::SFTP::Foreign->new($hostname,
user => $user, password => $password);
my #files = $sftp->find('/path/to/mysystem',
wanted => qr{^(?:.*/)?log\.txt$});
print "$_->{longname}\n" for #files;
Though, running find in the remote host is going to be faster.
Use find with exec.
Simply:
...$ssh->cmd("find mysystem/ -name "log.txt" -exec ls -la {} \\;");
elsif($exit eq '0')
{
foreach my $line (split(/\n/,$stdout)){
print $line."\n";
}
}