I need to cache a string from a file to do a grep search later. That string has a slash before a parenthesis. After I grab the string and echo it in the shell, it behaves the way I expect; but in my script it drops the slash off.
There's a lot of text below, but the part of the string that matters is "(remainingTime)"
Line in file:
"session_will_expire_message" = "User session will expire in \(remainingTime). Please sign in again to avoid expiration in the middle of the transaction.";
Shell Command (working as expected):
% line="\"User session will expire in \(remainingTime). Please sign in again to avoid expiration in the middle of the transaction.\";"
% echo $line
"User session will expire in \(remainingTime). Please sign in again to avoid expiration in the middle of the transaction.";
% echo -E - $line
"User session will expire in \(remainingTime). Please sign in again to avoid expiration in the middle of the transaction.";
% val="$(echo -E - $line | cut -d";" -f1 | cut -d"=" -f2-)"
% echo $val
"User session will expire in \(remainingTime). Please sign in again to avoid expiration in the middle of the transaction."
% echo -E - "${val}"
"User session will expire in \(remainingTime). Please sign in again to avoid expiration in the middle of the transaction."
Script (NOT working as expected):
while read line; do
if [[ $(echo $line | grep -c "=") -gt 0 ]]; then
key="$(echo $line | cut -d";" -f1 | cut -d"=" -f1 | tr -d " ")"
val="$(echo -E - $line | cut -d";" -f1 | cut -d"=" -f2-)"
/* !!--> */ [[ $IS_VERBOSE == "TRUE" ]] && echo "key: $key" && echo -E - "value: ${val}"
/* !!--> */ [[ $key != "" && "${val}" != "" ]] && LOCALIZED_PAIR[$key]="${val}"
fi
done < $LOCALIZED_DICT # <-- This is an URI for .strings file
The lines marked with "!! -->" are outputting to shell and to a file without the slash...
key: "session_will_expire_message"
value: "User session will expire in (remainingTime). Please sign in again to avoid expiration in the middle of the transaction."
Why is my script working differently, and how can I fix it?
EDIT:
It seems the slash is getting stripped out by the while loop:
% while read l; do echo $l; done < ~/Desktop/tmpOutput
"User session will expire in (remainingTime). Please sign in again to avoid expiration in the middle of the transaction.";
If the slashes are being split when the while loop reads the line, is there a way to avoid that?
You have a lot of unnecessary extra processes. Aside from that, though, use the -r option to prevent read from processing any backslashes in the input.
while IFS== read -r key val; do
[[ -z $key || -z $val ]] && continue
if [[ $IS_VERBOSE == TRUE ]]; then
print -r "key: $key"
print -r "value: $val"
fi
LOCALIZED_PAIR[$key]=$val
done < "$LOCALIZED_DICT"
ELK is installed on docker . Due to old logs and indexes server hard disk capacity gets full resulting crash of ELK container.
Run below shell script on docker shell on which elk is installed
#!/bin/bash
DAYSAGO=date --date="200 days ago" +%Y%m%d
ALLLINES=/usr/bin/curl -s -XGET http://127.0.0.1:9200/_cat/indices?v | egrep logstash
echo
echo "THIS IS WHAT SHOULD BE DELETED FOR ELK:"
echo
echo "$ALLLINES" | while read LINE
do
FORMATEDLINE=echo $LINE | awk '{ print $3 }' | awk -F'-' '{ print $2 }' | sed 's/\.//g'
if [ "$FORMATEDLINE" -lt "$DAYSAGO" ]
then
TODELETE=echo $LINE | awk '{ print $3 }'
echo "http://127.0.0.1:9200/$TODELETE"
fi
done
echo
echo -n "if this make sence, Y to continue N to exit [Y/N]:"
read INPUT
if [ "$INPUT" == "Y" ] || [ "$INPUT" == "y" ] || [ "$INPUT" == "yes" ] || [ "$INPUT" == "YES" ]
then
echo "$ALLLINES" | while read LINE
do
FORMATEDLINE=echo $LINE | awk '{ print $3 }' | awk -F'-' '{ print $2 }' | sed 's/\.//g'
if [ "$FORMATEDLINE" -lt "$DAYSAGO" ]
then
TODELETE=echo $LINE | awk '{ print $3 }'
/usr/bin/curl -XDELETE http://127.0.0.1:9200/$TODELETE
sleep 1
fi
done
else
echo SCRIPT CLOSED BY USER, BYE ...
echo
exit
fi
I found the following script posted by Sirus_Black. But I have a question that I am hoping he, or anyone can enlighten me with.
on #*:text:*:#:linkpost $1-
on #*:action:*:#:linkpost $1-
on #*:notice:*:#:linkpost $1-
alias -l linkpost {
if ((!%p) && (!$hfind(permit,$nick))) { inc -u4 %p
var %purge /^!(link\so(n|ff)|(permit))\b/iS
var %domain com|edu|gov|mil|net|org|biz|info|name|museum|us|ca|uk
var %exception /(?:https?:\/\/)?w{3}\.(youtube|imgur)\.com/
var %link /(?<=^|\s)((?>\S{3,8}:\/\/|w{3}\56)\S+)|\56( $+ %domain $+ )\b/iS
if ($findtok(%chanon1,#,1,32)) && ($nick(#,$nick,vr)) && ($regex($1-,%link)) && (!$regex($1-,%exception)) {
timeout 30 # $nick | /mode # -b $nick
msg # $nick You did not have permission to post a link ask a mod to !permit you
msg # /timeout $nick 1
}
elseif (($regex($1-,%purge)) && ($regml(1) = permit) && ($nick isop #) && ($$2 ison #)) {
hadd -mz permit $v1 30 | notice $v1 You have 30 seconds to post a link. Starting now!
msg # You now have 30 seconds to post a link!
}
elseif (($regml(1) = link on) && ($nick isop #)) {
goto $iif(!$istok(%chanon1,#,32),a,b) | :a | set %chanon1 $addtok(%chanon,#,32)
.msg # Link Protection Is Now on in: $+($chr(2),#)
halt | :b | .msg # $nick $+ , my link protection is already on in $&
$+($chr(2),#,$chr(2)) !
}
elseif (($regml(1) = link off) && ($nick isop #)) {
goto $iif($istok(%chanon1,#,32),c,d) | :c | set %chanon1 $remtok(%chanon,#,1,32)
.msg # Link Protection Is Now off in: $+($chr(2),#)
halt | :d | .msg # $nick $+ , My link protection is already off . $&
!
}
}
}
The script works as intended, but I am struggling with my limited knowledge of how to add onto it.
Say I want to make exceptions for some other websites.
test1.tv test2.eu and ima.ninja
I would imagine, it would be as simple as changing the variable line to:
var %exception /(?:https?:\/\/)?w{3}\.(test1|test2|ima)\.com/|\.tv|\.eu|\.ninja
But when I tried this, even those links would result in a time out.
Update: okay, I tested it again
var %exception /(?:https?:\/\/)?w{3}\.(test1|test2|ima)\.com/|\.tv|\.eu|\.ninja
this allows me to use test1.tv as I want, but http://test1.tv results in a time out. I also noticed that using any .eu and .tv web site would be permitted as long as the http:// was not present.
/brainfart
try this
on #*:text:*:#:linkpost $1-
on #*:action:*:#:linkpost $1-
on #*:notice:*:#:linkpost $1-
alias -l linkpost {
if ((!%p) && (!$hfind(permit,$nick))) {
var %purge /^!(link\so(n|ff)|(permit))\b/iS
var %domain com|edu|gov|mil|net|org|biz|info|name|museum|us|ca|uk
var %exception /https?:\/\/(?:w{3})?\.(youtube|imgur|ima|test)\.(ninja|eu|com)/
var %link /(?<=^|\s)((?>\S{3,5}:\/\/|w{3}\56)\S+)|\56( $+ %domain $+ )\b/iS
if ($findtok(%chanon1,#,1,32)) && ($nick(#,$nick,vr)) && ($regex($1-,%link)) && (!$regex($1-,%exception)) {
timeout 30 # $nick | /mode # -b $nick
msg # $nick You did not have permission to post a link ask a mod to !permit you
msg # /timeout $nick 1
}
elseif (($regex($1-,%purge)) && ($regml(1) = permit) && ($nick isop #) && ($$2 ison #)) {
hadd -mz permit $v1 30 | notice $v1 You have 30 seconds to post a link. Starting now!
msg # You now have 30 seconds to post a link!
}
elseif (($regml(1) = link on) && ($nick isop #)) {
goto $iif(!$istok(%chanon1,#,32),a,b) | :a | set %chanon1 $addtok(%chanon,#,32)
.msg # Link Protection Is Now on in: $+($chr(2),#)
halt | :b | .msg # $nick $+ , my link protection is already on in $&
$+($chr(2),#,$chr(2)) !
}
elseif (($regml(1) = link off) && ($nick isop #)) {
goto $iif($istok(%chanon1,#,32),c,d) | :c | set %chanon1 $remtok(%chanon,#,1,32)
.msg # Link Protection Is Now off in: $+($chr(2),#)
halt | :d | .msg # $nick $+ , My link protection is already off . $&
!
}
}
}
I just need to get just the sender's email address and the email subject from a pop3 account, from the command line or a BASH script. I'm running on a MacOS X machines, but want to avoid AppleScript. I've looked at getmail_fetch, but this seems to pull down the entire emails - I was hoping for something more space/time efficient. Is there some obvious trick I'm missing?
Solution using legacy sh.
#!/bin/sh
#
#
STATE=1
USER=myname
PASS=mypass
HOST=mail.example.com
PORT=pop3
MSGFILE=/tmp/msgid
POPFILE=/tmp/popcmds
DONE=/tmp/popdone
doneproc()
{
cat $DONE.tail $DONE.telnet | while read p; do kill -9 $p; done
rm -f $POPFILE $MSGFILE $DONE.tail $DONE.telnet
exit
}
reqmsg()
{
MSGID=`head -1 $MSGFILE`
if [ -z "$MSGID" ]; then echo "QUIT" >>$POPFILE; doneproc; fi
tail -n +2 $MSGFILE > $MSGFILE-
mv $MSGFILE- $MSGFILE
echo "TOP $MSGID" >>$POPFILE
}
rm -f $MSGFILE $DONE.tail $DONE.telnet
> $MSGFILE
> $POPFILE
sh -c "echo \$\$ > $DONE.tail; exec tail -f $POPFILE" | sh -c "echo \$\$ > $DONE.telnet; exec telnet $HOST $PORT" | while read -t 10 f
do
case "$STATE" in
1) case "$f" in
+OK*) echo "USER $USER" >>$POPFILE; STATE=2;;
esac;;
2) case "$f" in
+OK*) echo "PASS $PASS" >>$POPFILE; STATE=3;;
*) echo "Bad response to user command ($f)" >&2; exit 1;;
esac;;
3) case "$f" in
+OK*) echo "LIST" >>$POPFILE; STATE=4;;
*) echo "Bad response to password command ($f)" >&2; exit 1;;
esac;;
4) case "$f" in
+OK*) STATE=5;;
*) echo "Bad response to LIST command ($f)" >&2; exit 1;;
esac;;
5) case "$f" in
.*) reqmsg; STATE=6;;
[0-9]*) msgid=`echo $f | awk '{print $1;}'`
echo $msgid >> $MSGFILE;;
*) echo "Bad response to LIST command ($f)" >&2; exit 1;;
esac;;
6) case "$f" in
+OK*) STATE=7;;
*) echo "Bad response to TOP command ($f)" >&2; exit 1;;
esac;;
7) case "$f" in
.*) echo "---"; reqmsg; STATE=6;;
From:*) echo $f;;
Subject:*) echo $f;;
esac;;
*) echo STATE $STATE and got $f;;
esac
done
Solution using modern bash.
#!/bin/bash
#
#
STATE=1
USER=myname
PASS=mypass
HOST=mail.example.com
PORT=pop3
declare -A MSGIDS
declare -i NUMID
NUMID=0
declare -i CURID
CURID=0
coproc telnet $HOST $PORT
reqmsg()
{
if [ $CURID = $NUMID ]; then echo "QUIT" >&${COPROC[1]}; fi
echo "TOP ${MSGIDS[$CURID]}" >&${COPROC[1]}
((CURID++))
}
while read -t 10 f <&${COPROC[0]}
do
case "$STATE" in
1) case "$f" in
+OK*) echo "USER $USER" >&${COPROC[1]}; STATE=2;;
esac;;
2) case "$f" in
+OK*) echo "PASS $PASS" >&${COPROC[1]}; STATE=3;;
*) echo "Bad response to user command ($f)" >&2; exit 1;;
esac;;
3) case "$f" in
+OK*) echo "LIST" >&${COPROC[1]}; STATE=4;;
*) echo "Bad response to password command ($f)" >&2; exit 1;;
esac;;
4) case "$f" in
+OK*) STATE=5;;
*) echo "Bad response to LIST command ($f)" >&2; exit 1;;
esac;;
5) case "$f" in
.*) reqmsg; STATE=6;;
[0-9]*) read msgid size < <(echo $f)
MSGIDS[$NUMID]=$msgid
((NUMID++));;
*) echo "Bad response to LIST command ($f)" >&2; exit 1;;
esac;;
6) case "$f" in
+OK*) STATE=7;;
*) echo "Bad response to TOP command ($f)" >&2; exit 1;;
esac;;
7) case "$f" in
.*) echo "---"; reqmsg; STATE=6;;
From:*) echo $f;;
Subject:*) echo $f;;
esac;;
*) echo STATE $STATE and got $f;;
esac
done
I just tested these scripts on Linux Mint Quiana and also Scientific Linux, both still work fine with a slight tweak. As in the earlier comment TOP on my server appears to need 2 arguments but as you are only looking at the mail headers just add a space and 0 to the end of the TOP command (0 lists only the mail headers) ie
(first script)
echo "TOP $MSGID 0" >>$POPFILE
or
(second script)
echo "TOP ${MSGIDS[$CURID]} 0" >&${COPROC[1]}
I also needed to change "read -t 10 f" to "read -t 50 f" as my POP3 server seems quite slow.
Scientific Linux was happy with the invocation of the "sh" shell however Mint needed the references to "sh" changed to "bash" then all was well
Finally to avoid leaving your POP3 password lying around in a script file it would be better to prompt for it on running the script:
ie
echo "enter password for this POP3 email connecton"
read password
PASS=$password
Here is the loop in bash:
while [ $# -ge 1 ]; do
case $1 in
-a)
shift
NUM_AGENTS=$1
;;
-h)
shift
HOST_NAME=$1
;;
-t)
shift
TIME_STAGGER=$1
;;
-un)
shift
USER_NAME=$1
;;
-pw)
shift
USER_PASS=$1
;;
-p)
shift
TARGET_PAGE=$1
;;
-s)
shift
COMMON_SID=$1
;;
esac
shift
done
How can i convert this in perl so that the argument would populate the values in the command line
php loadAgent_curl.php $NUM_AGENTS $HOST_NAME $procStartTime $i $TARGET_PAGE $reqlogfile $resplogfile $USER_NAME $USER_PASS $execDelay $COMMON_SID &
------- appended to question:
this certainly helps, and i really appreciate it, is there any way to access these parameters outside the getOptions ? here is rest of the bash script: my $i="0";
my $startTime=date +%s;
startTime=$[$startTime+$NUM_AGENTS+10]
my $PWD=pwd;
my $logdir="\$PWD/load-logs";
system(mkdir $logdir/$startTime);
my $reqlogfile="$logdir/$startTime/req.log";
my $resplogfile="$logdir/$startTime/resp.log";
print "\n";
print "##################\n";
print "LAUNCHING REQUESTS\n";
print " HOST NAME : \$HOST_NAME\n ";
print " TARGET PAGE : \$TARGET_PAGE\n ";
print " # AGENTS : \$NUM_AGENTS\n ";
print " EXECUTION TIME : \$startTime (with random stagger between 0 and \$TIME_STAGGER seconds)\n ";
print " REQ LOG FILE : $reqlogfile\n ";
print " RESP LOG FILE : $resplogfile\n ";
print "##################\n";
print "\n";
#
#
highestStart=$startTime
$startTime += $ARGV[0] + 5;
my $dTime = localtime( $startTime );
print "\n##################\nLAUNCHING REQUESTS\n
COUNT: $ARGV[0]\n
DELAY: | 1 \n
The scripts will fire at : $dTime\n##################\n\n";
while ( $ARGV[0] > $i )
{
$i++;
system("php avtestTimed.php $ARGV[0] $ARGV[2] $startTime");
print "RUN system('php avtestTimed.php $ARGV[0] $ARGV[2] $startTime'); \n";
sleep 1;
}
#
#
while [ $NUM_AGENTS -gt "$i" ]
do
i=$[$i+1]
execDelay=$((RANDOM % $TIME_STAGGER))"."$((RANDOM % 100))
procStartTime=$[$startTime]
procStartTime=$[$startTime+$execDelay]
if [ $procStartTime -gt $highestStart ]
then
highestStart=$procStartTime
fi
echo "STATUS: Queueing request $i with a delay of $execDelay seconds"
echo " '--> COMMAND: php loadAgent_curl.php $NUM_AGENTS $HOST_NAME $procStartTime $i $TARGET_PAGE $reqlogfile $resplogfile $USER_NAME $USER_PASS $execDelay $COMMON_SID"
php loadAgent_curl.php $NUM_AGENTS $HOST_NAME $procStartTime $i $TARGET_PAGE $reqlogfile $resplogfile $USER_NAME $USER_PASS $execDelay $COMMON_SID &
sleep 1
done
echo "STATUS: Waiting for queued requests to be ready"
while [ date +%s -lt $startTime ]
do
sleep 1
done
#
echo "STATUS: Waiting for last request to issue"
while [ date +%s -lt $highestStart ]
do
sleep 1
done
#
echo "STATUS: Last response issued"
#
echo "STATUS: Waiting for response log file to be created"
while [ ! -e "$resplogfile" ]
do
sleep 1
done
#
while [ wc -l "$resplogfile"| awk '{print $1'} -lt $NUM_AGENTS ]
do
#echo "(wc -l "$resplogfile"| awk '{print $1'} of $NUM_AGENTS responses recorded)"
sleep 1
done
echo "STATUS: FINISHED"
while true; do
read -p "Do you wish to view the request log? [y/n]" yn
case $yn in
[Yy]* ) cat $reqlogfile; break;;
[Nn]* ) exit;;
* ) echo "Please answer yes or no.";;
esac
done
while true; do
read -p "Do you wish to view the response log? [y/n]" yn
case $yn in
[Yy]* ) cat $resplogfile; break;;
[Nn]* ) exit;;
* ) echo "Please answer yes or no.";;
esac
done
Getopt::Long library is a standard Perl way to process command line options.
Something like this will work. Not tested - caveat emptor!
Please note that since your PHP parameters are a mix between command line options AND some unidentified variables, I have designed the first example so that ALL the possible options should be stored in %args hash (e.g. your program should use $args{procStartTime} instead of $procStartTime). This allowed me to make it very short and generic.
If this is hard to read/understand, I also have a second example that's more straightforward but less generic
use Getopt::Long;
my #php_arg_order = qw(a h procStartTime i p reqlogfile
resplogfile un pw execDelay s);
my %args = map {$_ => ""} #php_arg_order;
$args{procStartTime} = "something";
$args{reqlogfile} = "a.log";
# More defaults for variables NOT passed in via command line.
# Populate them all in %args as above.
# Now load actual command line parameters.
GetOptions(\%args, map { "$_=s" } #php_arg_order) or die "Unknown parameter!\n";
system(join(" ",
"php", "loadAgent_curl.php",map { $args{$_} } #php_arg_order}, "&"));
A second, less advanced but more direct option is:
use Getopt::Long;
my %args = ();
# Now load actual command line parameters.
GetOptions(\%args,
"NUM_AGENTS|a=s"
,"HOST_NAME|h=s"
,"USER_NAME|un=s"
# ... the rest of options
# The "XXX|x" notaion allows using alias "-x" parameter
# but stores in $args{XXX} instead for better readability
) or die "Unknown parameter!\n";
system("php loadAgent_curl.php $args{NUM_AGENTS} $args{HOST_NAME} $procStartTime $i $args{TARGET_PAGE} $reqlogfile $resplogfile $args{USER_NAME} $args{USER_PASS} $execDelay $args{COMMON_SID} &");