I wanted to make a program that searches all the lines that contains all the factors given, from a file mydata. I tried to egrep first factor from mydata and save it in a variable a. Then, I tried to egrep the next factor from a and save the result to a again until I egrep all the factors. But when I executed the program, it said
"command not found" in line 14.
if [ $# -eq 0 ]
then
echo -e "Usage: phoneA searchfor [...searchfor]\n(You didn't tell me what you want to search for.)"
else
a=""
for i in $*
do
if [ -z "$a" ]
then
a=$(egrep "$i" mydata)
else
a=$("$a" | egrep "$i")
fi
done
awk -f display.awk "$a"
fi
I expected all the lines including all the factors outputted on the screen in the pattern that I made in display.awk.
$a contains data, not a command. You need to write that data to the pipe.
if [ $# -eq 0 ]
then
printf '%s\n' "Usage: phoneA searchfor [...searchfor]" "(You didn't tell me what you want to search for.)" >&2
exit 1
fi
a=""
for i in "$#"; do
if [ -z "$a" ]; then
a=$(egrep "$i" mydata)
else
a=$(printf '%s' "$a" | egrep "$i")
fi
done
awk -f display.awk "$a"
I have an assignment to parse out AutoSys JIL files. This is a JIL job definition, it is a config file that the AUTOSYS scheduler reads in and runs. , Imagine a file formatted like this, with thousands of job definitions like the one below, stacked on top of each other in the exact same format. All beginning with the header and ending with the timezone.
/* ----------------- COME_AND_PLAY_WITH_US_DANNY ----------------- */
insert_job: COME_AND_PLAY_WITH_US_DANNY job_type: CMD
command: /bin/bash -ls
machine: capser.com
owner: twins
permission: foo,foo
date_conditions: 1
days_of_week: mo,tu,we,th,fr
start_times: "04:00"
description: "Forever, and ever and ever"
std_in_file: "/home/room217"
std_out_file: "${CASPERSYSLOG}/room217.out"
std_err_file: "${CASPERSYSLOG}/room217.err
alarm_if_fail: 1
profile: "/autosys_profile"
timezone: US/Eastern
This is the script. I need to extract the job, machine and command from the job definition above. It works fine, but eventually I am going to want to store the information in some kind of container and send it, while this script writes out the results line by line in the terminal. Right now I am redirecting the results to a temporary file.
#!/foo/bar/perl5/core/5.10/exec/bin/perl
use strict;
use warnings;
use File::Basename ;
my($job, $machine, $command) ;
my $filename = '/tmp/autosys.jil_output.padc';
open(my $fh, '<:encoding(UTF-8)', $filename)
or die "Could not open file '$filename' $!";
my $count = 0;
while (my $line = <$fh>) {
#chomp $line;
if($line =~ /\/\* -{17} \w+ -{17} \*\//) {
$count = 1; }
elsif($line =~ /(alarm_if_fail:)/) {
$count = 0 ; }
elsif ($count) {
if ($line =~ m/insert_job: (\w+).*job_type: CMD/) {
$job = $1 ;
}
elsif($line =~ m/command:(.*)/) {
$command = $1 ;
}
elsif($line =~ m/machine:(.*)/) {
$machine = $1 ;
print "$job\t $machine\t $command \n ";
}
}
#sleep 1 ;
}
My question is When I place the print $job, $machine $command statement within the last elsif statement, it works fine. However when I place it out side of the last elsif statement, like the example below the output is duplicated over and over again - each line is duplicated like four to five times in the output. I do not understand that. How come I have to put the print statement within the last elsif statement to get the script to print out one line at a time, correctly.
elsif ( $line =~ m/machine:(.*)/ ) {
$machine = $1;
}
print "$job\t $machine\t $command \n ";
Reformat of above code for readability
#!/foo/bar/perl5/core/5.10/exec/bin/perl
use strict;
use warnings;
use File::Basename;
my ( $job, $machine, $command );
my $filename = '/tmp/autosys.jil_output.padc';
open( my $fh, '<:encoding(UTF-8)', $filename )
or die "Could not open file '$filename' $!";
my $count = 0;
while ( my $line = <$fh> ) {
#chomp $line;
if ( $line =~ /\/\* -{17} \w+ -{17} \*\// ) {
$count = 1;
}
elsif ( $line =~ /(alarm_if_fail:)/ ) {
$count = 0;
}
elsif ( $count ) {
if ( $line =~ m/insert_job: (\w+).*job_type: CMD/ ) {
$job = $1;
}
elsif ( $line =~ m/command:(.*)/ ) {
$command = $1;
}
elsif ( $line =~ m/machine:(.*)/ ) {
$machine = $1;
print "$job\t $machine\t $command \n ";
}
}
# sleep 1;
}
As I've said in my comment, please format your code sensibly. Without doing so you will get people either ignoring your question, or being grumpy about answering like me
Let's assume that the unidentified text block is just a sample of your input
Let's also assume that, even though your code works fine with your sample data, there are some data blocks in the real data that don't work
On top of that, I'm assuming that any data field value that contains spaces requires enclosing quotes, which makes your example command: /bin/bash -ls incorrect, and invalid syntax
Please also make sure that you have given a proper example of your problem with runnable code and data. If I execute the code that you show against your sample data then everything works fine, so what problem do you have?
As far as I can tell, you want to display the insert_job, machine, and command fields from every JIL data block whose job_type field is CMD. Is that right?
Here's my best guess: xxfelixxx's comment is correct, and you are simply printing all the fields that you have collected every time you read a line from the data file
My solution is to transform each data block into a hash.
It is dangerous to use comments to delineate the blocks, and you have given no information about the ordering of the fields, so I have to assume that the insert_job field comes first. That makes sense if the file is to be used as a list of imperatives, but the additional job_type field on the same line is weird. Is that a genuine sample of your data, or another problem with your example?
Here's a working solution to my imagination of your problem.
#!/foo/bar/perl5/core/5.10/exec/bin/perl
use strict;
use warnings 'all';
my $data = do {
local $/;
<DATA>;
};
my #data = grep /:/, split /^(?=insert_job)/m, $data;
for ( #data ) {
my %data = /(\w+) \s* : \s* (?| " ( [^""]+ ) " | (\S+) )/gx;
next unless $data{job_type} eq 'CMD';
print "#data{qw/ insert_job machine command /}\n";
}
__DATA__
/* ----------------- COME_AND_PLAY_WITH_US_DANNY ----------------- */
insert_job: COME_AND_PLAY_WITH_US_DANNY job_type: CMD
command: /bin/bash -ls
machine: capser.com
owner: twins
permission: foo,foo
date_conditions: 1
days_of_week: mo,tu,we,th,fr
start_times: "04:00"
description: "Forever, and ever and ever"
std_in_file: "/home/room217"
std_out_file: "${CASPERSYSLOG}/room217.out"
std_err_file: "${CASPERSYSLOG}/room217.err
alarm_if_fail: 1
profile: "/autosys_profile"
timezone: US/Eastern
/* ----------------- COME_AND_PLAY_WITH_US_AGAIN_DANNY ----------------- */
insert_job: COME_AND_PLAY_WITH_US_AGAIN_DANNY job_type: CMD
command: /bin/bash -ls
machine: capser.com
owner: twins
permission: foo,foo
date_conditions: 1
days_of_week: mo,tu,we,th,fr
start_times: "04:00"
description: "Forever, and ever and ever"
std_in_file: "/home/room217"
std_out_file: "${CASPERSYSLOG}/room217.out"
std_err_file: "${CASPERSYSLOG}/room217.err
alarm_if_fail: 1
profile: "/autosys_profile"
timezone: US/Eastern
/* ----------------- NEVER_PLAY_WITH_US_AGAIN_DANNY ----------------- */
insert_job: NEVER_PLAY_WITH_US_AGAIN_DANNY job_type: CMD
command: /bin/bash -rm *
machine: capser.com
owner: twins
permission: foo,foo
date_conditions: 1
days_of_week: mo,tu,we,th,fr
start_times: "04:00"
description: "Forever, and ever and ever"
std_in_file: "/home/room217"
std_out_file: "${CASPERSYSLOG}/room217.out"
std_err_file: "${CASPERSYSLOG}/room217.err
alarm_if_fail: 1
profile: "/autosys_profile"
timezone: US/Eastern
output
COME_AND_PLAY_WITH_US_DANNY capser.com /bin/bash
COME_AND_PLAY_WITH_US_AGAIN_DANNY capser.com /bin/bash
NEVER_PLAY_WITH_US_AGAIN_DANNY capser.com /bin/bash
This is a ksh solution to turn a JIL file into comma separated file you can open in excel
#!/usr/bin/ksh
# unix scprit to flatten autorep -q
resetVar()
{
AIF=""
AD=""
AH=""
BF=""
BN=""
BS=""
BT=""
COM=""
COD=""
DC=""
DOW=""
DES=""
EC=""
IJ=""
JL=""
JT=""
MAC=""
MES=""
MRA=""
NR=""
OWN=""
PER=""
PRI=""
PRO=""
RC=""
RW=""
SM=""
ST=""
SEF=""
SOF=""
TRT=""
WF=""
WFMS=""
WI=""
LSD=""
LST=""
LED=""
LET=""
STA=""
RUN=""
}
writePartToFile()
{
echo "$AIF;$AD;$AH;$BF;$BN;$BS;$BT;$COM;$COD;$DC;$DOW;$DES;$EC;$IJ;$JL;$JT;$MAC;$MES;$MRA;$NR;$OWN;$PER;$PRI;$PRO;$RC;$RW;$SM;$ST;$SEF;$SOF;$TRT;$WF;$WFMS;$WI" >> $TO_TPM
#echo "$AIF;$AD;$AH;$BF;$BN;$BS;$BT;$COM;$COD;$DC;$DOW;$DES;$EC;$IJ;$JL;$JT;$MAC;$MES;$MRA;$NR;$OWN;$PER;$PRI;$PRO;$RC;$RW;$SM;$ST;$SEF;$SOF;$TRT;$WF;$WFMS;$WI"
resetVar
}
JOB_NAME="flatten JIL"
part1=""
part2=""
#---------------------------------
if test "$1." = "."
then
echo "Missing first parameter (jil file to flatten)";
exit 1;
fi
if test "$2." = "."
then
echo "Missing second parameter (resulting flat file)";
exit 1;
fi
TO_FLATTEN=$1
TO_RESULT=$2
CLE_FILE="lesCles"
CLE_TMP="lesClesTmp"
TO_TPM="tempFichier"
TO_STATUS="statusFichier"
rm $TO_RESULT
rm $CLE_TMP
rm $CLE_FILE
rm $TO_TPM
rm $TO_STATUS
echo 'alarm_if_fail;auto_delete;auto_hold;box_failure;box_name;box_success;box_terminator;command;condition;date_conditions;days_of_week;description;exclude_calendar;insert_job;job_load;job_terminator;machine;max_exit_success;max_run_alarm;n_retrys;owner;permission;priority;profile;run_calendar;run_window;start_mins;start_times;std_err_file;std_out_file;term_run_time;watch_file;watch_file_min_size;watch_interval;last_start_date;last_start_time;last_end_date;last_end_time;status;run' >> $TO_RESULT;
while read line; do
if test "${line#*:}" != "$line"
then
cle="$(echo "$line" | cut -d":" -f 1)"
#echo "cle = $cle"
part2="$(echo "$line" | cut -d":" -f 2)"
#echo "part2 = $part2"
val="$(echo "$part2" | cut -d" " -f 2)"
#echo "val = $val"
fi
if test "$cle" = "insert_job"
then
#on n'est sur la premiere ligne
if test "$IJ." = "."
then
;
else
if test "$BN." = "."
then
echo $IJ >> $CLE_TMP
else
echo $BN >> $CLE_TMP
fi
writePartToFile
fi
IJ=$val
JT="$(echo "$line" | cut -d":" -f 3)"
else
#on n est pas sur le premiere ligne
val=$part2
case $cle in
alarm_if_fail) AIF=$val;;
auto_delete) AD=$val;;
auto_hold) AH=$val;;
box_failure) BF=$val;;
box_name) BN=$val;;
box_success) BS=$val;;
box_terminator) BT=$val;;
command) COM=$val;;
condition) COD=$val;;
date_conditions) DC=$val;;
days_of_week) DOW=$val;;
description) DES=$val;;
exclude_calendar) EC=$val;;
insert_job) IJ=$val;;
job_load) JL=$val;;
job_terminator) JT=$val;;
machine) MAC=$val;;
max_exit_success) MES=$val;;
max_run_alarm) MRA==$val;;
n_retrys) NR=$val;;
'#owner') OWN=$val;;
permission) PER=$val;;
priority) PRI=$val;;
profile) PRO=$val;;
run_calendar) RC=$val;;
run_window) RW=$val;;
start_mins) SM=$val;;
start_times) ST=$val;;
std_err_file) SEF=$val;;
std_out_file) SOF=$val;;
term_run_time) TRT=$val;;
watch_file) WF=$val;;
watch_file_min_size) WFMS=$val;;
watch_interval) WI=$val;;
esac
fi
done < $TO_FLATTEN;
#Traiter derniere occurence
if test "$BN." = "."
then
echo $IJ >> $CLE_TMP
else
echo $BN >> $CLE_TMP
fi
writePartToFile
echo "Les cles"
cat $CLE_TMP | sort | uniq > $CLE_FILE
cat $CLE_FILE
rm $CLE_TMP
#------------------------------
while read line; do
autorep -J ${line} -w >> $TO_STATUS;
done < $CLE_FILE;
#----------------------------------------
echo " Resultats"
while read line; do
unJob="$(echo "$line" | cut -d";" -f 14)"
details="$(grep -w "$unJob" "$TO_STATUS" | head -n 1)"
LSD="$(echo "$details" | awk '{print $2}')"
if test "$LSD" = "-----"
then
LST=""
LED="$(echo "$details" | awk '{print $3}')"
if test "$LED" = "-----"
then
LET=""
STA="$(echo "$details" | awk '{print $4}')"
RUN="$(echo "$details" | awk '{print $5}')"
else
LET="$(echo "$details" | awk '{print $4}')"
STA="$(echo "$details" | awk '{print $5}')"
RUN="$(echo "$details" | awk '{print $6}')"
fi
else
LST="$(echo "$details" | awk '{print $3}')"
LED="$(echo "$details" | awk '{print $4}')"
if test "$LED" = "-----"
then
LET=""
STA="$(echo "$details" | awk '{print $5}')"
RUN="$(echo "$details" | awk '{print $6}')"
else
LET="$(echo "$details" | awk '{print $5}')"
STA="$(echo "$details" | awk '{print $6}')"
RUN="$(echo "$details" | awk '{print $7}')"
fi
fi
echo " ligne= ${line};${LSD};${LST};${LED};${LET};${STA};${RUN}"
echo "${line};${LSD};${LST};${LED};${LET};${STA};${RUN}" >> $TO_RESULT
resetVar
done < $TO_TPM;
How to run a if block in unix from perl.
Eg:
location="/home/shon";
if [[ -f $location/sample.txt ]]
then
echo "file found...."
else
echo "Error in getting file"
exit 255
fi
from perl.
As suggested, this code is simple to translate into perl. Assuming you have something more complex, you can spawn a shell to run it: put the shell code in a quoted heredoc so that perl does not substitute the shell variables.
system 'bash', '-c', <<'END_SHELL_CODE';
location="/home/shon"
if [[ -f $location/sample.txt ]]; then
echo "file found...."
else
echo "Error in getting file"
exit 255
fi
END_SHELL_CODE
Try this
#! /usr/bin/perl
my $location ="/home/shon/sample.txt";
if (-f $location)
{
print "file found....";
}
else
{
print "Error in getting file";
}
Here is the loop in bash:
while [ $# -ge 1 ]; do
case $1 in
-a)
shift
NUM_AGENTS=$1
;;
-h)
shift
HOST_NAME=$1
;;
-t)
shift
TIME_STAGGER=$1
;;
-un)
shift
USER_NAME=$1
;;
-pw)
shift
USER_PASS=$1
;;
-p)
shift
TARGET_PAGE=$1
;;
-s)
shift
COMMON_SID=$1
;;
esac
shift
done
How can i convert this in perl so that the argument would populate the values in the command line
php loadAgent_curl.php $NUM_AGENTS $HOST_NAME $procStartTime $i $TARGET_PAGE $reqlogfile $resplogfile $USER_NAME $USER_PASS $execDelay $COMMON_SID &
------- appended to question:
this certainly helps, and i really appreciate it, is there any way to access these parameters outside the getOptions ? here is rest of the bash script: my $i="0";
my $startTime=date +%s;
startTime=$[$startTime+$NUM_AGENTS+10]
my $PWD=pwd;
my $logdir="\$PWD/load-logs";
system(mkdir $logdir/$startTime);
my $reqlogfile="$logdir/$startTime/req.log";
my $resplogfile="$logdir/$startTime/resp.log";
print "\n";
print "##################\n";
print "LAUNCHING REQUESTS\n";
print " HOST NAME : \$HOST_NAME\n ";
print " TARGET PAGE : \$TARGET_PAGE\n ";
print " # AGENTS : \$NUM_AGENTS\n ";
print " EXECUTION TIME : \$startTime (with random stagger between 0 and \$TIME_STAGGER seconds)\n ";
print " REQ LOG FILE : $reqlogfile\n ";
print " RESP LOG FILE : $resplogfile\n ";
print "##################\n";
print "\n";
#
#
highestStart=$startTime
$startTime += $ARGV[0] + 5;
my $dTime = localtime( $startTime );
print "\n##################\nLAUNCHING REQUESTS\n
COUNT: $ARGV[0]\n
DELAY: | 1 \n
The scripts will fire at : $dTime\n##################\n\n";
while ( $ARGV[0] > $i )
{
$i++;
system("php avtestTimed.php $ARGV[0] $ARGV[2] $startTime");
print "RUN system('php avtestTimed.php $ARGV[0] $ARGV[2] $startTime'); \n";
sleep 1;
}
#
#
while [ $NUM_AGENTS -gt "$i" ]
do
i=$[$i+1]
execDelay=$((RANDOM % $TIME_STAGGER))"."$((RANDOM % 100))
procStartTime=$[$startTime]
procStartTime=$[$startTime+$execDelay]
if [ $procStartTime -gt $highestStart ]
then
highestStart=$procStartTime
fi
echo "STATUS: Queueing request $i with a delay of $execDelay seconds"
echo " '--> COMMAND: php loadAgent_curl.php $NUM_AGENTS $HOST_NAME $procStartTime $i $TARGET_PAGE $reqlogfile $resplogfile $USER_NAME $USER_PASS $execDelay $COMMON_SID"
php loadAgent_curl.php $NUM_AGENTS $HOST_NAME $procStartTime $i $TARGET_PAGE $reqlogfile $resplogfile $USER_NAME $USER_PASS $execDelay $COMMON_SID &
sleep 1
done
echo "STATUS: Waiting for queued requests to be ready"
while [ date +%s -lt $startTime ]
do
sleep 1
done
#
echo "STATUS: Waiting for last request to issue"
while [ date +%s -lt $highestStart ]
do
sleep 1
done
#
echo "STATUS: Last response issued"
#
echo "STATUS: Waiting for response log file to be created"
while [ ! -e "$resplogfile" ]
do
sleep 1
done
#
while [ wc -l "$resplogfile"| awk '{print $1'} -lt $NUM_AGENTS ]
do
#echo "(wc -l "$resplogfile"| awk '{print $1'} of $NUM_AGENTS responses recorded)"
sleep 1
done
echo "STATUS: FINISHED"
while true; do
read -p "Do you wish to view the request log? [y/n]" yn
case $yn in
[Yy]* ) cat $reqlogfile; break;;
[Nn]* ) exit;;
* ) echo "Please answer yes or no.";;
esac
done
while true; do
read -p "Do you wish to view the response log? [y/n]" yn
case $yn in
[Yy]* ) cat $resplogfile; break;;
[Nn]* ) exit;;
* ) echo "Please answer yes or no.";;
esac
done
Getopt::Long library is a standard Perl way to process command line options.
Something like this will work. Not tested - caveat emptor!
Please note that since your PHP parameters are a mix between command line options AND some unidentified variables, I have designed the first example so that ALL the possible options should be stored in %args hash (e.g. your program should use $args{procStartTime} instead of $procStartTime). This allowed me to make it very short and generic.
If this is hard to read/understand, I also have a second example that's more straightforward but less generic
use Getopt::Long;
my #php_arg_order = qw(a h procStartTime i p reqlogfile
resplogfile un pw execDelay s);
my %args = map {$_ => ""} #php_arg_order;
$args{procStartTime} = "something";
$args{reqlogfile} = "a.log";
# More defaults for variables NOT passed in via command line.
# Populate them all in %args as above.
# Now load actual command line parameters.
GetOptions(\%args, map { "$_=s" } #php_arg_order) or die "Unknown parameter!\n";
system(join(" ",
"php", "loadAgent_curl.php",map { $args{$_} } #php_arg_order}, "&"));
A second, less advanced but more direct option is:
use Getopt::Long;
my %args = ();
# Now load actual command line parameters.
GetOptions(\%args,
"NUM_AGENTS|a=s"
,"HOST_NAME|h=s"
,"USER_NAME|un=s"
# ... the rest of options
# The "XXX|x" notaion allows using alias "-x" parameter
# but stores in $args{XXX} instead for better readability
) or die "Unknown parameter!\n";
system("php loadAgent_curl.php $args{NUM_AGENTS} $args{HOST_NAME} $procStartTime $i $args{TARGET_PAGE} $reqlogfile $resplogfile $args{USER_NAME} $args{USER_PASS} $execDelay $args{COMMON_SID} &");
#!/bin/bash
i="0"
echo ""
echo "##################"
echo "LAUNCHING REQUESTS"
echo " COUNT: $2 "
echo " DELAY: $3 "
echo " SESSID: $1"
echo "##################"
echo ""
while [ $2 -gt "$i" ]
do
i=$[$i+1]
php avtest.php $1 $4 &
echo "EXECUTING REQUEST $i"
sleep $3
done
here is a better/modified script in bash
#!/bin/bash
i="0"
#startTime=`date +%s`
startTime=$(date -u +%s)
startTime=$[$startTime+$1+5]
#startTime=$($startTime+$1+5)
dTime=`date -d #$startTime`
echo ""
echo "##################"
echo "LAUNCHING REQUESTS"
echo " COUNT: $1 "
echo " DELAY: 1 "
#echo " EXECUTION: $startTime "
echo " The scripts will fire at : $dTime "
echo "##################"
echo ""
while [ $1 -gt "$i" ]
do
i=$[$i+1]
php avtestTimed.php $1 $3 $startTime &
echo "QUEUEING REQUEST $i"
sleep 1
done
Here's a direct translation
#!/usr/bin/env perl
use strict;
use warnings;
print <<HERE;
##################
LAUNCHING REQUESTS
COUNT: $ARGV[1]
DELAY: $ARGV[2]
SESSID: $ARGV[0]
##################
HERE
my $i = 0;
while($ARGV[1] > $i){
$i += 1;
system("php avtest.php $ARGV[0] $ARGV[3] &");
print "EXECUTING REQUEST $i\n";
sleep $ARGV[2];
}
But it would make more sense to read the command line parameters into variables named after what they're for and not rely on remembering argument ordering.
A brief errata in the conversion:
I use a here string to represent multiline text. I could also have put in multiple print statements to more closely mimic the bash version
In bash arguments are accessed as numbered variables, starting with $1 and going up. In Perl the argument list is represented by the array #ARGV, which is numbered starting at zero (like arrays in most languages). In both bash and Perl the name of the script can be found in the variable $0.
In Perl arrays are written as #arrayname when refering to the entire array, but they use $arrayname[index] when accessing array members. So the Perl $list[0] is like the bash ${list[0]} and the Perl #list is like the bash ${list[#]}.
In Perl variables are declared with the my keyword; the equivalent in bash would be declare.
I've used the system function for spawning background processes. Its argument can be simply the command line as you might use it in bash.
Unlike echo, print requires to be told if there should be a newline at the end of the line. For recent versions of Perl the say function exists which will append a newline for you.
The Perl sleep function is pretty self-explanatory.
EDIT: Due to a typo $i in the print statement had been represented as $ni leading to runtime errors. This has been corrected.