Parsing AutoSys JIL with perl - perl

Question

I have an assignment to parse out AutoSys JIL files. This is a JIL job definition, it is a config file that the AUTOSYS scheduler reads in and runs. , Imagine a file formatted like this, with thousands of job definitions like the one below, stacked on top of each other in the exact same format. All beginning with the header and ending with the timezone.
/* ----------------- COME_AND_PLAY_WITH_US_DANNY ----------------- */
insert_job: COME_AND_PLAY_WITH_US_DANNY job_type: CMD
command: /bin/bash -ls
machine: capser.com
owner: twins
permission: foo,foo
date_conditions: 1
days_of_week: mo,tu,we,th,fr
start_times: "04:00"
description: "Forever, and ever and ever"
std_in_file: "/home/room217"
std_out_file: "${CASPERSYSLOG}/room217.out"
std_err_file: "${CASPERSYSLOG}/room217.err
alarm_if_fail: 1
profile: "/autosys_profile"
timezone: US/Eastern
This is the script. I need to extract the job, machine and command from the job definition above. It works fine, but eventually I am going to want to store the information in some kind of container and send it, while this script writes out the results line by line in the terminal. Right now I am redirecting the results to a temporary file.
#!/foo/bar/perl5/core/5.10/exec/bin/perl
use strict;
use warnings;
use File::Basename ;
my($job, $machine, $command) ;
my $filename = '/tmp/autosys.jil_output.padc';
open(my $fh, '<:encoding(UTF-8)', $filename)
or die "Could not open file '$filename' $!";
my $count = 0;
while (my $line = <$fh>) {
#chomp $line;
if($line =~ /\/\* -{17} \w+ -{17} \*\//) {
$count = 1; }
elsif($line =~ /(alarm_if_fail:)/) {
$count = 0 ; }
elsif ($count) {
if ($line =~ m/insert_job: (\w+).*job_type: CMD/) {
$job = $1 ;
}
elsif($line =~ m/command:(.*)/) {
$command = $1 ;
}
elsif($line =~ m/machine:(.*)/) {
$machine = $1 ;
print "$job\t $machine\t $command \n ";
}
}
#sleep 1 ;
}
My question is When I place the print $job, $machine $command statement within the last elsif statement, it works fine. However when I place it out side of the last elsif statement, like the example below the output is duplicated over and over again - each line is duplicated like four to five times in the output. I do not understand that. How come I have to put the print statement within the last elsif statement to get the script to print out one line at a time, correctly.
elsif ( $line =~ m/machine:(.*)/ ) {
$machine = $1;
}
print "$job\t $machine\t $command \n ";
Reformat of above code for readability
#!/foo/bar/perl5/core/5.10/exec/bin/perl
use strict;
use warnings;
use File::Basename;
my ( $job, $machine, $command );
my $filename = '/tmp/autosys.jil_output.padc';
open( my $fh, '<:encoding(UTF-8)', $filename )
or die "Could not open file '$filename' $!";
my $count = 0;
while ( my $line = <$fh> ) {
#chomp $line;
if ( $line =~ /\/\* -{17} \w+ -{17} \*\// ) {
$count = 1;
}
elsif ( $line =~ /(alarm_if_fail:)/ ) {
$count = 0;
}
elsif ( $count ) {
if ( $line =~ m/insert_job: (\w+).*job_type: CMD/ ) {
$job = $1;
}
elsif ( $line =~ m/command:(.*)/ ) {
$command = $1;
}
elsif ( $line =~ m/machine:(.*)/ ) {
$machine = $1;
print "$job\t $machine\t $command \n ";
}
}
# sleep 1;
}

Answer 1

As I've said in my comment, please format your code sensibly. Without doing so you will get people either ignoring your question, or being grumpy about answering like me
Let's assume that the unidentified text block is just a sample of your input
Let's also assume that, even though your code works fine with your sample data, there are some data blocks in the real data that don't work
On top of that, I'm assuming that any data field value that contains spaces requires enclosing quotes, which makes your example command: /bin/bash -ls incorrect, and invalid syntax
Please also make sure that you have given a proper example of your problem with runnable code and data. If I execute the code that you show against your sample data then everything works fine, so what problem do you have?
As far as I can tell, you want to display the insert_job, machine, and command fields from every JIL data block whose job_type field is CMD. Is that right?
Here's my best guess: xxfelixxx's comment is correct, and you are simply printing all the fields that you have collected every time you read a line from the data file
My solution is to transform each data block into a hash.
It is dangerous to use comments to delineate the blocks, and you have given no information about the ordering of the fields, so I have to assume that the insert_job field comes first. That makes sense if the file is to be used as a list of imperatives, but the additional job_type field on the same line is weird. Is that a genuine sample of your data, or another problem with your example?
Here's a working solution to my imagination of your problem.
#!/foo/bar/perl5/core/5.10/exec/bin/perl
use strict;
use warnings 'all';
my $data = do {
local $/;
<DATA>;
};
my #data = grep /:/, split /^(?=insert_job)/m, $data;
for ( #data ) {
my %data = /(\w+) \s* : \s* (?| " ( [^""]+ ) " | (\S+) )/gx;
next unless $data{job_type} eq 'CMD';
print "#data{qw/ insert_job machine command /}\n";
}
__DATA__
/* ----------------- COME_AND_PLAY_WITH_US_DANNY ----------------- */
insert_job: COME_AND_PLAY_WITH_US_DANNY job_type: CMD
command: /bin/bash -ls
machine: capser.com
owner: twins
permission: foo,foo
date_conditions: 1
days_of_week: mo,tu,we,th,fr
start_times: "04:00"
description: "Forever, and ever and ever"
std_in_file: "/home/room217"
std_out_file: "${CASPERSYSLOG}/room217.out"
std_err_file: "${CASPERSYSLOG}/room217.err
alarm_if_fail: 1
profile: "/autosys_profile"
timezone: US/Eastern
/* ----------------- COME_AND_PLAY_WITH_US_AGAIN_DANNY ----------------- */
insert_job: COME_AND_PLAY_WITH_US_AGAIN_DANNY job_type: CMD
command: /bin/bash -ls
machine: capser.com
owner: twins
permission: foo,foo
date_conditions: 1
days_of_week: mo,tu,we,th,fr
start_times: "04:00"
description: "Forever, and ever and ever"
std_in_file: "/home/room217"
std_out_file: "${CASPERSYSLOG}/room217.out"
std_err_file: "${CASPERSYSLOG}/room217.err
alarm_if_fail: 1
profile: "/autosys_profile"
timezone: US/Eastern
/* ----------------- NEVER_PLAY_WITH_US_AGAIN_DANNY ----------------- */
insert_job: NEVER_PLAY_WITH_US_AGAIN_DANNY job_type: CMD
command: /bin/bash -rm *
machine: capser.com
owner: twins
permission: foo,foo
date_conditions: 1
days_of_week: mo,tu,we,th,fr
start_times: "04:00"
description: "Forever, and ever and ever"
std_in_file: "/home/room217"
std_out_file: "${CASPERSYSLOG}/room217.out"
std_err_file: "${CASPERSYSLOG}/room217.err
alarm_if_fail: 1
profile: "/autosys_profile"
timezone: US/Eastern
output
COME_AND_PLAY_WITH_US_DANNY capser.com /bin/bash
COME_AND_PLAY_WITH_US_AGAIN_DANNY capser.com /bin/bash
NEVER_PLAY_WITH_US_AGAIN_DANNY capser.com /bin/bash

Answer 2

This is a ksh solution to turn a JIL file into comma separated file you can open in excel
#!/usr/bin/ksh
# unix scprit to flatten autorep -q
resetVar()
{
AIF=""
AD=""
AH=""
BF=""
BN=""
BS=""
BT=""
COM=""
COD=""
DC=""
DOW=""
DES=""
EC=""
IJ=""
JL=""
JT=""
MAC=""
MES=""
MRA=""
NR=""
OWN=""
PER=""
PRI=""
PRO=""
RC=""
RW=""
SM=""
ST=""
SEF=""
SOF=""
TRT=""
WF=""
WFMS=""
WI=""
LSD=""
LST=""
LED=""
LET=""
STA=""
RUN=""
}
writePartToFile()
{
echo "$AIF;$AD;$AH;$BF;$BN;$BS;$BT;$COM;$COD;$DC;$DOW;$DES;$EC;$IJ;$JL;$JT;$MAC;$MES;$MRA;$NR;$OWN;$PER;$PRI;$PRO;$RC;$RW;$SM;$ST;$SEF;$SOF;$TRT;$WF;$WFMS;$WI" >> $TO_TPM
#echo "$AIF;$AD;$AH;$BF;$BN;$BS;$BT;$COM;$COD;$DC;$DOW;$DES;$EC;$IJ;$JL;$JT;$MAC;$MES;$MRA;$NR;$OWN;$PER;$PRI;$PRO;$RC;$RW;$SM;$ST;$SEF;$SOF;$TRT;$WF;$WFMS;$WI"
resetVar
}
JOB_NAME="flatten JIL"
part1=""
part2=""
#---------------------------------
if test "$1." = "."
then
echo "Missing first parameter (jil file to flatten)";
exit 1;
fi
if test "$2." = "."
then
echo "Missing second parameter (resulting flat file)";
exit 1;
fi
TO_FLATTEN=$1
TO_RESULT=$2
CLE_FILE="lesCles"
CLE_TMP="lesClesTmp"
TO_TPM="tempFichier"
TO_STATUS="statusFichier"
rm $TO_RESULT
rm $CLE_TMP
rm $CLE_FILE
rm $TO_TPM
rm $TO_STATUS
echo 'alarm_if_fail;auto_delete;auto_hold;box_failure;box_name;box_success;box_terminator;command;condition;date_conditions;days_of_week;description;exclude_calendar;insert_job;job_load;job_terminator;machine;max_exit_success;max_run_alarm;n_retrys;owner;permission;priority;profile;run_calendar;run_window;start_mins;start_times;std_err_file;std_out_file;term_run_time;watch_file;watch_file_min_size;watch_interval;last_start_date;last_start_time;last_end_date;last_end_time;status;run' >> $TO_RESULT;
while read line; do
if test "${line#*:}" != "$line"
then
cle="$(echo "$line" | cut -d":" -f 1)"
#echo "cle = $cle"
part2="$(echo "$line" | cut -d":" -f 2)"
#echo "part2 = $part2"
val="$(echo "$part2" | cut -d" " -f 2)"
#echo "val = $val"
fi
if test "$cle" = "insert_job"
then
#on n'est sur la premiere ligne
if test "$IJ." = "."
then
;
else
if test "$BN." = "."
then
echo $IJ >> $CLE_TMP
else
echo $BN >> $CLE_TMP
fi
writePartToFile
fi
IJ=$val
JT="$(echo "$line" | cut -d":" -f 3)"
else
#on n est pas sur le premiere ligne
val=$part2
case $cle in
alarm_if_fail) AIF=$val;;
auto_delete) AD=$val;;
auto_hold) AH=$val;;
box_failure) BF=$val;;
box_name) BN=$val;;
box_success) BS=$val;;
box_terminator) BT=$val;;
command) COM=$val;;
condition) COD=$val;;
date_conditions) DC=$val;;
days_of_week) DOW=$val;;
description) DES=$val;;
exclude_calendar) EC=$val;;
insert_job) IJ=$val;;
job_load) JL=$val;;
job_terminator) JT=$val;;
machine) MAC=$val;;
max_exit_success) MES=$val;;
max_run_alarm) MRA==$val;;
n_retrys) NR=$val;;
'#owner') OWN=$val;;
permission) PER=$val;;
priority) PRI=$val;;
profile) PRO=$val;;
run_calendar) RC=$val;;
run_window) RW=$val;;
start_mins) SM=$val;;
start_times) ST=$val;;
std_err_file) SEF=$val;;
std_out_file) SOF=$val;;
term_run_time) TRT=$val;;
watch_file) WF=$val;;
watch_file_min_size) WFMS=$val;;
watch_interval) WI=$val;;
esac
fi
done < $TO_FLATTEN;
#Traiter derniere occurence
if test "$BN." = "."
then
echo $IJ >> $CLE_TMP
else
echo $BN >> $CLE_TMP
fi
writePartToFile
echo "Les cles"
cat $CLE_TMP | sort | uniq > $CLE_FILE
cat $CLE_FILE
rm $CLE_TMP
#------------------------------
while read line; do
autorep -J ${line} -w >> $TO_STATUS;
done < $CLE_FILE;
#----------------------------------------
echo " Resultats"
while read line; do
unJob="$(echo "$line" | cut -d";" -f 14)"
details="$(grep -w "$unJob" "$TO_STATUS" | head -n 1)"
LSD="$(echo "$details" | awk '{print $2}')"
if test "$LSD" = "-----"
then
LST=""
LED="$(echo "$details" | awk '{print $3}')"
if test "$LED" = "-----"
then
LET=""
STA="$(echo "$details" | awk '{print $4}')"
RUN="$(echo "$details" | awk '{print $5}')"
else
LET="$(echo "$details" | awk '{print $4}')"
STA="$(echo "$details" | awk '{print $5}')"
RUN="$(echo "$details" | awk '{print $6}')"
fi
else
LST="$(echo "$details" | awk '{print $3}')"
LED="$(echo "$details" | awk '{print $4}')"
if test "$LED" = "-----"
then
LET=""
STA="$(echo "$details" | awk '{print $5}')"
RUN="$(echo "$details" | awk '{print $6}')"
else
LET="$(echo "$details" | awk '{print $5}')"
STA="$(echo "$details" | awk '{print $6}')"
RUN="$(echo "$details" | awk '{print $7}')"
fi
fi
echo " ligne= ${line};${LSD};${LST};${LED};${LET};${STA};${RUN}"
echo "${line};${LSD};${LST};${LED};${LET};${STA};${RUN}" >> $TO_RESULT
resetVar
done < $TO_TPM;