I want to increment snapshots numbering of all snapshots starting from certain snapshot (by using sed). How to increment snapshot numbering starting for example from snapshot=22?
//
I have already found the command which increment whole snapshot numbering
sed -r 's/(snapshot=)([0-9]+)/echo \1$((\2+1))/ge' myInputFile
I have file which looks like that:
#-----------
snapshot=4
#-----------
time=1449929
mem_heap_B=110
mem_heap_extra_B=40
mem_stacks_B=0
heap_tree=peak
#-----------
snapshot=5
#-----------
time=1449929
mem_heap_B=110
mem_heap_extra_B=40
mem_stacks_B=0
heap_tree=peak
.
.
.
#-----------
snapshot=22
#-----------
time=1448920
mem_heap_B=10
mem_heap_extra_B=24
mem_stacks_B=0
heap_tree=detailed
.
.
.
#-----------
snapshot=46
#-----------
time=1449964
mem_heap_B=110
mem_heap_extra_B=24
mem_stacks_B=0
heap_tree=detailed
.
.
.
#-----------
snapshot=172
#-----------
time=1448920
mem_heap_B=10
I want to get following output:
#-----------
snapshot=4
#-----------
time=1449929
mem_heap_B=110
mem_heap_extra_B=40
mem_stacks_B=0
heap_tree=peak
#-----------
snapshot=5
#-----------
time=1449929
mem_heap_B=110
mem_heap_extra_B=40
mem_stacks_B=0
heap_tree=peak
.
.
.
#-----------
snapshot=23
#-----------
time=1448920
mem_heap_B=10
mem_heap_extra_B=24
mem_stacks_B=0
heap_tree=detailed
.
.
.
#-----------
snapshot=47
#-----------
time=1449964
mem_heap_B=110
mem_heap_extra_B=24
mem_stacks_B=0
heap_tree=detailed
.
.
.
#-----------
snapshot=173
#-----------
time=1448920
mem_heap_B=10
You will find it difficult to do that with sed because sed doesn't have built-in arithmetic functionality (and implementing it in pure sed is not a sane idea).
However, it is quite simple with awk:
awk -F = 'BEGIN { OFS = FS } $1 == "snapshot" && $2 >= 22 { ++$2 } 1' filename
This works by splitting lines into fields at = separators, and then:
BEGIN {
OFS = FS # use the same separator in the output
} # as for the input
$1 == "snapshot" && $2 >= 22 { # if the first field in a line is "snapshot"
# and the second larger than or equal to 22:
++$2 # increment the second field
}
1 # print the resulting line (unchanged if the
# condition was not met)
Related
I have an assignment to parse out AutoSys JIL files. This is a JIL job definition, it is a config file that the AUTOSYS scheduler reads in and runs. , Imagine a file formatted like this, with thousands of job definitions like the one below, stacked on top of each other in the exact same format. All beginning with the header and ending with the timezone.
/* ----------------- COME_AND_PLAY_WITH_US_DANNY ----------------- */
insert_job: COME_AND_PLAY_WITH_US_DANNY job_type: CMD
command: /bin/bash -ls
machine: capser.com
owner: twins
permission: foo,foo
date_conditions: 1
days_of_week: mo,tu,we,th,fr
start_times: "04:00"
description: "Forever, and ever and ever"
std_in_file: "/home/room217"
std_out_file: "${CASPERSYSLOG}/room217.out"
std_err_file: "${CASPERSYSLOG}/room217.err
alarm_if_fail: 1
profile: "/autosys_profile"
timezone: US/Eastern
This is the script. I need to extract the job, machine and command from the job definition above. It works fine, but eventually I am going to want to store the information in some kind of container and send it, while this script writes out the results line by line in the terminal. Right now I am redirecting the results to a temporary file.
#!/foo/bar/perl5/core/5.10/exec/bin/perl
use strict;
use warnings;
use File::Basename ;
my($job, $machine, $command) ;
my $filename = '/tmp/autosys.jil_output.padc';
open(my $fh, '<:encoding(UTF-8)', $filename)
or die "Could not open file '$filename' $!";
my $count = 0;
while (my $line = <$fh>) {
#chomp $line;
if($line =~ /\/\* -{17} \w+ -{17} \*\//) {
$count = 1; }
elsif($line =~ /(alarm_if_fail:)/) {
$count = 0 ; }
elsif ($count) {
if ($line =~ m/insert_job: (\w+).*job_type: CMD/) {
$job = $1 ;
}
elsif($line =~ m/command:(.*)/) {
$command = $1 ;
}
elsif($line =~ m/machine:(.*)/) {
$machine = $1 ;
print "$job\t $machine\t $command \n ";
}
}
#sleep 1 ;
}
My question is When I place the print $job, $machine $command statement within the last elsif statement, it works fine. However when I place it out side of the last elsif statement, like the example below the output is duplicated over and over again - each line is duplicated like four to five times in the output. I do not understand that. How come I have to put the print statement within the last elsif statement to get the script to print out one line at a time, correctly.
elsif ( $line =~ m/machine:(.*)/ ) {
$machine = $1;
}
print "$job\t $machine\t $command \n ";
Reformat of above code for readability
#!/foo/bar/perl5/core/5.10/exec/bin/perl
use strict;
use warnings;
use File::Basename;
my ( $job, $machine, $command );
my $filename = '/tmp/autosys.jil_output.padc';
open( my $fh, '<:encoding(UTF-8)', $filename )
or die "Could not open file '$filename' $!";
my $count = 0;
while ( my $line = <$fh> ) {
#chomp $line;
if ( $line =~ /\/\* -{17} \w+ -{17} \*\// ) {
$count = 1;
}
elsif ( $line =~ /(alarm_if_fail:)/ ) {
$count = 0;
}
elsif ( $count ) {
if ( $line =~ m/insert_job: (\w+).*job_type: CMD/ ) {
$job = $1;
}
elsif ( $line =~ m/command:(.*)/ ) {
$command = $1;
}
elsif ( $line =~ m/machine:(.*)/ ) {
$machine = $1;
print "$job\t $machine\t $command \n ";
}
}
# sleep 1;
}
As I've said in my comment, please format your code sensibly. Without doing so you will get people either ignoring your question, or being grumpy about answering like me
Let's assume that the unidentified text block is just a sample of your input
Let's also assume that, even though your code works fine with your sample data, there are some data blocks in the real data that don't work
On top of that, I'm assuming that any data field value that contains spaces requires enclosing quotes, which makes your example command: /bin/bash -ls incorrect, and invalid syntax
Please also make sure that you have given a proper example of your problem with runnable code and data. If I execute the code that you show against your sample data then everything works fine, so what problem do you have?
As far as I can tell, you want to display the insert_job, machine, and command fields from every JIL data block whose job_type field is CMD. Is that right?
Here's my best guess: xxfelixxx's comment is correct, and you are simply printing all the fields that you have collected every time you read a line from the data file
My solution is to transform each data block into a hash.
It is dangerous to use comments to delineate the blocks, and you have given no information about the ordering of the fields, so I have to assume that the insert_job field comes first. That makes sense if the file is to be used as a list of imperatives, but the additional job_type field on the same line is weird. Is that a genuine sample of your data, or another problem with your example?
Here's a working solution to my imagination of your problem.
#!/foo/bar/perl5/core/5.10/exec/bin/perl
use strict;
use warnings 'all';
my $data = do {
local $/;
<DATA>;
};
my #data = grep /:/, split /^(?=insert_job)/m, $data;
for ( #data ) {
my %data = /(\w+) \s* : \s* (?| " ( [^""]+ ) " | (\S+) )/gx;
next unless $data{job_type} eq 'CMD';
print "#data{qw/ insert_job machine command /}\n";
}
__DATA__
/* ----------------- COME_AND_PLAY_WITH_US_DANNY ----------------- */
insert_job: COME_AND_PLAY_WITH_US_DANNY job_type: CMD
command: /bin/bash -ls
machine: capser.com
owner: twins
permission: foo,foo
date_conditions: 1
days_of_week: mo,tu,we,th,fr
start_times: "04:00"
description: "Forever, and ever and ever"
std_in_file: "/home/room217"
std_out_file: "${CASPERSYSLOG}/room217.out"
std_err_file: "${CASPERSYSLOG}/room217.err
alarm_if_fail: 1
profile: "/autosys_profile"
timezone: US/Eastern
/* ----------------- COME_AND_PLAY_WITH_US_AGAIN_DANNY ----------------- */
insert_job: COME_AND_PLAY_WITH_US_AGAIN_DANNY job_type: CMD
command: /bin/bash -ls
machine: capser.com
owner: twins
permission: foo,foo
date_conditions: 1
days_of_week: mo,tu,we,th,fr
start_times: "04:00"
description: "Forever, and ever and ever"
std_in_file: "/home/room217"
std_out_file: "${CASPERSYSLOG}/room217.out"
std_err_file: "${CASPERSYSLOG}/room217.err
alarm_if_fail: 1
profile: "/autosys_profile"
timezone: US/Eastern
/* ----------------- NEVER_PLAY_WITH_US_AGAIN_DANNY ----------------- */
insert_job: NEVER_PLAY_WITH_US_AGAIN_DANNY job_type: CMD
command: /bin/bash -rm *
machine: capser.com
owner: twins
permission: foo,foo
date_conditions: 1
days_of_week: mo,tu,we,th,fr
start_times: "04:00"
description: "Forever, and ever and ever"
std_in_file: "/home/room217"
std_out_file: "${CASPERSYSLOG}/room217.out"
std_err_file: "${CASPERSYSLOG}/room217.err
alarm_if_fail: 1
profile: "/autosys_profile"
timezone: US/Eastern
output
COME_AND_PLAY_WITH_US_DANNY capser.com /bin/bash
COME_AND_PLAY_WITH_US_AGAIN_DANNY capser.com /bin/bash
NEVER_PLAY_WITH_US_AGAIN_DANNY capser.com /bin/bash
This is a ksh solution to turn a JIL file into comma separated file you can open in excel
#!/usr/bin/ksh
# unix scprit to flatten autorep -q
resetVar()
{
AIF=""
AD=""
AH=""
BF=""
BN=""
BS=""
BT=""
COM=""
COD=""
DC=""
DOW=""
DES=""
EC=""
IJ=""
JL=""
JT=""
MAC=""
MES=""
MRA=""
NR=""
OWN=""
PER=""
PRI=""
PRO=""
RC=""
RW=""
SM=""
ST=""
SEF=""
SOF=""
TRT=""
WF=""
WFMS=""
WI=""
LSD=""
LST=""
LED=""
LET=""
STA=""
RUN=""
}
writePartToFile()
{
echo "$AIF;$AD;$AH;$BF;$BN;$BS;$BT;$COM;$COD;$DC;$DOW;$DES;$EC;$IJ;$JL;$JT;$MAC;$MES;$MRA;$NR;$OWN;$PER;$PRI;$PRO;$RC;$RW;$SM;$ST;$SEF;$SOF;$TRT;$WF;$WFMS;$WI" >> $TO_TPM
#echo "$AIF;$AD;$AH;$BF;$BN;$BS;$BT;$COM;$COD;$DC;$DOW;$DES;$EC;$IJ;$JL;$JT;$MAC;$MES;$MRA;$NR;$OWN;$PER;$PRI;$PRO;$RC;$RW;$SM;$ST;$SEF;$SOF;$TRT;$WF;$WFMS;$WI"
resetVar
}
JOB_NAME="flatten JIL"
part1=""
part2=""
#---------------------------------
if test "$1." = "."
then
echo "Missing first parameter (jil file to flatten)";
exit 1;
fi
if test "$2." = "."
then
echo "Missing second parameter (resulting flat file)";
exit 1;
fi
TO_FLATTEN=$1
TO_RESULT=$2
CLE_FILE="lesCles"
CLE_TMP="lesClesTmp"
TO_TPM="tempFichier"
TO_STATUS="statusFichier"
rm $TO_RESULT
rm $CLE_TMP
rm $CLE_FILE
rm $TO_TPM
rm $TO_STATUS
echo 'alarm_if_fail;auto_delete;auto_hold;box_failure;box_name;box_success;box_terminator;command;condition;date_conditions;days_of_week;description;exclude_calendar;insert_job;job_load;job_terminator;machine;max_exit_success;max_run_alarm;n_retrys;owner;permission;priority;profile;run_calendar;run_window;start_mins;start_times;std_err_file;std_out_file;term_run_time;watch_file;watch_file_min_size;watch_interval;last_start_date;last_start_time;last_end_date;last_end_time;status;run' >> $TO_RESULT;
while read line; do
if test "${line#*:}" != "$line"
then
cle="$(echo "$line" | cut -d":" -f 1)"
#echo "cle = $cle"
part2="$(echo "$line" | cut -d":" -f 2)"
#echo "part2 = $part2"
val="$(echo "$part2" | cut -d" " -f 2)"
#echo "val = $val"
fi
if test "$cle" = "insert_job"
then
#on n'est sur la premiere ligne
if test "$IJ." = "."
then
;
else
if test "$BN." = "."
then
echo $IJ >> $CLE_TMP
else
echo $BN >> $CLE_TMP
fi
writePartToFile
fi
IJ=$val
JT="$(echo "$line" | cut -d":" -f 3)"
else
#on n est pas sur le premiere ligne
val=$part2
case $cle in
alarm_if_fail) AIF=$val;;
auto_delete) AD=$val;;
auto_hold) AH=$val;;
box_failure) BF=$val;;
box_name) BN=$val;;
box_success) BS=$val;;
box_terminator) BT=$val;;
command) COM=$val;;
condition) COD=$val;;
date_conditions) DC=$val;;
days_of_week) DOW=$val;;
description) DES=$val;;
exclude_calendar) EC=$val;;
insert_job) IJ=$val;;
job_load) JL=$val;;
job_terminator) JT=$val;;
machine) MAC=$val;;
max_exit_success) MES=$val;;
max_run_alarm) MRA==$val;;
n_retrys) NR=$val;;
'#owner') OWN=$val;;
permission) PER=$val;;
priority) PRI=$val;;
profile) PRO=$val;;
run_calendar) RC=$val;;
run_window) RW=$val;;
start_mins) SM=$val;;
start_times) ST=$val;;
std_err_file) SEF=$val;;
std_out_file) SOF=$val;;
term_run_time) TRT=$val;;
watch_file) WF=$val;;
watch_file_min_size) WFMS=$val;;
watch_interval) WI=$val;;
esac
fi
done < $TO_FLATTEN;
#Traiter derniere occurence
if test "$BN." = "."
then
echo $IJ >> $CLE_TMP
else
echo $BN >> $CLE_TMP
fi
writePartToFile
echo "Les cles"
cat $CLE_TMP | sort | uniq > $CLE_FILE
cat $CLE_FILE
rm $CLE_TMP
#------------------------------
while read line; do
autorep -J ${line} -w >> $TO_STATUS;
done < $CLE_FILE;
#----------------------------------------
echo " Resultats"
while read line; do
unJob="$(echo "$line" | cut -d";" -f 14)"
details="$(grep -w "$unJob" "$TO_STATUS" | head -n 1)"
LSD="$(echo "$details" | awk '{print $2}')"
if test "$LSD" = "-----"
then
LST=""
LED="$(echo "$details" | awk '{print $3}')"
if test "$LED" = "-----"
then
LET=""
STA="$(echo "$details" | awk '{print $4}')"
RUN="$(echo "$details" | awk '{print $5}')"
else
LET="$(echo "$details" | awk '{print $4}')"
STA="$(echo "$details" | awk '{print $5}')"
RUN="$(echo "$details" | awk '{print $6}')"
fi
else
LST="$(echo "$details" | awk '{print $3}')"
LED="$(echo "$details" | awk '{print $4}')"
if test "$LED" = "-----"
then
LET=""
STA="$(echo "$details" | awk '{print $5}')"
RUN="$(echo "$details" | awk '{print $6}')"
else
LET="$(echo "$details" | awk '{print $5}')"
STA="$(echo "$details" | awk '{print $6}')"
RUN="$(echo "$details" | awk '{print $7}')"
fi
fi
echo " ligne= ${line};${LSD};${LST};${LED};${LET};${STA};${RUN}"
echo "${line};${LSD};${LST};${LED};${LET};${STA};${RUN}" >> $TO_RESULT
resetVar
done < $TO_TPM;
Stuck trying to figure out how to single quotes withing single quotes within double quotes. Here's what I'm trying to do....
From perl, I want to run a system command that...
- does an ssh into a remote machine
- executes 'uptime' and then plucks the last field out of that (avg load last 15 min).
\#\!/usr/bin/env perl
my $cmd = "ssh othermachine 'uptime | awk '{print $NF}'' > local_file.dat";
system($cmd);
Of course this won't run ...
% ./try.pl
Missing }.
%
Missing "}" ??? Looks like it's interpreting the $NF} as a var? I tried escaping the {} chars with no luck. I tried escaping the $, no luck. I tried a space before the }, no luck but different msg (Undefined variable).
c-shell BTW and thanks in advance !
You want the following to be ssh's second argument:
uptime | awk '{print $NF}'
To do that, you simply placed single quotes around it. But that doesn't work because it contains single quotes.
You want to build a string that contains $NF, but you did it as follows:
"...$NF..."
That will place the value of (non-existent) Perl variable $NF in the string.
Do it step by step.
Static:
Remote command:
uptime | awk '{print $NF}'
Local command:
ssh othermachine 'uptime | awk '\''{print $NF}'\''' >local_file.dat
String literal:
my $local_cmd = q{ssh othermachine 'uptime | awk '\''{print $NF}'\''' >local_file.dat}
Dynamic:
use String::ShellQuote qw( shell_quote );
my $remote_cmd = q{uptime | awk '{print $NF}'};
my $local_cmd = shell_quote('ssh', 'othermachine', $remote_cmd) . ' >local_file.dat';
Use Net::OpenSSH and let it do the quoting for you:
use Net::OpenSSH;
my $ssh = Net::OpenSSH->new($othermachine,
remote_shell => 'tcsh');
$ssh->system({stdout_file => 'local_file.dat'},
'uptime', \\'|', 'awk', '{print $NF}')
or die "ssh command failed: " . $ssh->error;
What is the best way to break long lines having system commands which are being executed by " ` "
example:
my #data = `cat data.txt | perl -ne '/CTX:|ed-as-of time:\\s+(\\w+)\\s+[Feb|Mar|April|May]/ && print' | sed '\$!N;/Sync/P;D'|sed 'N;s/\\n/ /'`;
I tried to break it using " `." like below, but am getting errors (sh: 1: Syntax error: end of file unexpected:).
my #data = `cat data.txt | `.
`perl -ne '/CTX:|ed-as-of time:\\s+(\\w+)\\s+[Feb|Mar|April|May]/ && print' | sed '\$!N;/Sync/P;D'|sed 'N;s/\\n/ /'`;
The best ways are to assemble the command in a scalar variable
$cmd = "command1 --foo --bar | command2 2>log.err | "
. " command3 --are-we-done-yet | ...";
#data = `$cmd`; # or #data = qx($cmd)
or to assemble it as an argument to readpipe
#data = readpipe("command1 --foo --bar | command2 >log.err | "
. "command3 --are-we-done-yet | ...");
I have many files(.js files), under a directory and subdirectories. I want to replace the lines which ends in ';', with the original line plus print.f("filename" with linenumber");. The actual filename and linenumber where it was found.
example - line ending with ';'
var x = 5;
should be replaced with
var x = 5;print.f("test.js with line number 16");
test.js and 16 are the filename and linenumber where var x = 5; was found respectively.
Edit
The test.js and 16 are samples, in the output it should be the filename and the linemunber
If you are flexible in using awk:
awk '{printf $0} /.*\;$/{printf " print.f(\"" FILENAME " with line number " NR "\");"} {printf "\n"}' filename
This will add the print.f statement whenever the lines ends with ;.
Unless you have GNU awk 4.1.0 or later, you will have to write the output of the command temporarily to a file and then move it back.
For GNU awk 4.1.0 and above, you can use -i option for inplace file editing.
gawk -i '{printf $0} /.*\;$/{printf " print.f(\"" FILENAME " with line number " NR "\");"} {printf "\n"}' filename
To find all .js files and run the above gawk command on them:
find . -type f -name "*.js" | xargs gawk -i '{printf $0} /.*\;$/{printf " print.f(\"" FILENAME " with line number " NR "\");"} {printf "\n"}'
I want to change the second column to upper case and I want to do it in shell script only. (no one liners!)
#!/bin/sh
# read file line by line
file="/pdump/country.000000.txt"
while read line
do
mycol=`echo $line | awk -F"," '{print $2}'`
mycol_new=`echo $mycol | tr "[:lower:]" [:upper:]`
echo $line | awk -F"," '{print $1 "," $mycol_new "," $3 "," $4 "," $5 "," $6 "," $7 "," $8}'
done < $file
I am not able to replace the $2 with $mycol_new.
Any suggestion?
awk cannot see $mycol_new because it is a shell variable. Here is one way of passing a shell variable into awk using the -v flag:
echo $line | awk -v var="$mycol_new" -F"," '{print $1 "," var "," $3 "," $4 "," $5 "," $6 "," $7 "," $8}'
Here is an alternative method which lets the shell expand $mycol_new:
echo $line | awk -F"," '{print $1 ",'"$mycol_new"'," $3 "," $4 "," $5 "," $6 "," $7 "," $8}'
why no one liners? Doing homework?
$ cat file
one two three four
five six seven eight
$ awk '{$2=toupper($2)}1' file
one TWO three four
five SIX seven eight
If you want to do this all in the shell, then you don't need awk:
IFS=,
while read line; do
set -- $line
a="$1"
b="${2^^}" # assumes bash, use "tr" otherwise
shift 2
set -- "$a" "$b" "$#"
echo "$*"
done < "$file" > "$file.new"