Simple newbie question that seems to be stumping me.
I've looked at other topics and resources and I'm not seeing the simple answer.
I'm running a very simple (perl -lne) command that helps me query specific data out of a pwebstats output , and output into a linear .csv file format for my reports.
The question is this:
Is there a quick, and clean way of assigning variables for the file name, Year , Month and Day ( i.e allowing the person running the command to simply enter the File Name ($FILENAME) , Year ($YEAR), Month ($MONTH) and Day ($DAY) and the execute without having to alter the perl -lne command each time?)
EXAMPLE OF COMMAND :
zcat /opt/log/file/$Year/$Month/$FILENAME/sm/$FILENAME.$Year-$Month-$Day \
| perl -lne '/.{0,0}FILENAME PAX Total HTTP IP Packet Count.{0,6}/ && print $&' \
> /tmp/jhoney/$FILENAME-IPcountTMP.csv
zcat /opt/log/file/$Year/$Month/$FILENAME/sm/$FILENAME.$Year-$Month-$Day \
| perl -lne '/.{0,0}PAX HTTP Average TCP RTT.{0,20}/ && print $&' \
> /tmp/jhoney/$FIELNAME-TCPrttTMP.csv
paste -d , /tmp/jhoney/$FILENAME-IPcountTMP.csv /tmp/jhoney/$FILENAME-TCPrttTMP.csv \
> /tmp/jhoney/$FILENAME-PAXdata.csv
rm -r /tmp/jhoney/$FILENAME-TCPrttTMP.csv /tmp/jhoney/$FILENAME-IPcountTMP.csv
I know the answer is looking right in my face, but, I'm learning perl and pyhton as I go.
Any help would be appreciated.
What you keep calling a Perl command are sh commands. Parsing command line options in sh is not trivial. If you don't mind switching to positional parameters, it would be a lot simpler.
Usage:
PAXStat 2015 09 06 FILENAME
Script:
#!/bin/sh
Year=$1
Month=$2
Day=$3
FILENAME=$4
zcat /opt/log/file/$Year/$Month/$FILENAME/sm/$FILENAME.$Year-$Month-$Day \
| perl -lne '/.{0,0}FILENAME PAX Total HTTP IP Packet Count.{0,6}/ && print $&' \
> /tmp/jhoney/$FILENAME-IPcountTMP.csv
zcat /opt/log/file/$Year/$Month/$FILENAME/sm/$FILENAME.$Year-$Month-$Day \
| perl -lne '/.{0,0}PAX HTTP Average TCP RTT.{0,20}/ && print $&' \
> /tmp/jhoney/$FIELNAME-TCPrttTMP.csv
paste -d , /tmp/jhoney/$FILENAME-IPcountTMP.csv /tmp/jhoney/$FILENAME-TCPrttTMP.csv \
> /tmp/jhoney/$FILENAME-PAXdata.csv
rm -r /tmp/jhoney/$FILENAME-TCPrttTMP.csv /tmp/jhoney/$FILENAME-IPcountTMP.csv
Related
Ok, so I have this command that turns off my touchscreen. It works when I execute it in a root shell.
So this works:
sudo su
/usr/bin/echo $(ls /sys/bus/hid/drivers/hid-multitouch | awk NR==1'{print $1}') > /sys/bus/hid/drivers/hid-multitouch/unbind
And then my touchscreen stops working, which is the result that I wanted.
Now I want to make a touchscreen.service file to execute this on every boot. So in the service file I include:
ExecStart=/usr/bin/echo $(ls /sys/bus/hid/drivers/hid-multitouch | awk NR==1'{print $1}') > /sys/bus/hid/drivers/hid-multitouch/unbind
However it isn't working > nor throwing any errors that I've been able to catch.
I do know from earlier fidlings with .service files that I might actually need to use /usr/bin/sh -c, so I have also tried:
ExecStart=/usr/bin/sh -c "/usr/bin/echo $(ls /sys/bus/hid/drivers/hid-multitouch | awk NR==1'{print $1}') > /sys/bus/hid/drivers/hid-multitouch/unbind"
Yet this also doesn't work.. maybe because of the awk NR==1'{print $1}'part? I have also tried replacing it with awk NR==1'\''{print $1}'\''but again it fails to work.
Does anyone have any ideas on how to get the command that is working in my root cli environment to also work as a systemd service?
To start with,
The syntax of the awk command is just wrong. The quotes are incorrectly placed. The part NR == 1 is part of the awk command to indicate the first line record in the file, i.e.
awk NR==1'{print $1}'
# ^^^^^^^ should be within quotes
awk 'NR == 1 { print $1 }'
Your sequence of echo, ls and the command substitution $(..) doesn't look right. You are effectively echo-ing the literal string /sys/bus/hid/drivers/hid-multitouch (if ls finds the file at that path) over to the pipe and awk just writes that to the /sys/bus/hid/drivers/hid-multitouch/unbind file which might not be your desired action. You just needed to do run the command on the file directly as
awk 'NR == 1 { print $1 }' /sys/bus/hid/drivers/hid-multitouch > /sys/bus/hid/drivers/hid-multitouch/unbind
Now that, that the awk command is fixed, you have two options to run the above command as part of systemd, either put your command in a script or run the command directly. For putting it in a script refer to the Unix.SE answer Where do I put scripts executed by systemd units?. As for running the command directly in ExecStart. Aside from using /bin/sh also use the path /bin/awk
So putting it together and using /bin/ over /usr/bin, you can do below. This command uses ".." over awk script and needs escape of $1
ExecStart=/bin/sh -c '/bin/awk "NR == 1 { print \$1 }" /sys/bus/hid/drivers/hid-multitouch > /sys/bus/hid/drivers/hid-multitouch/unbind'
I would like to execute this make command to first replace the first line of all csv files inside the directory and then replace the # for commas through the other lines.
The second command is working fine and does what it is supposed to do, but the first one only replaces the line on the first file.
Could anyone give me a help on that?
csv:
$(DOCKER_RUN) npm run csv-generator
make format-csv
format-csv:
#sed -i '' '1 s/^.*$$/"bar","repository"/g' $(CURDIR)/foo/npm/*.csv
#sed -i '' 's/\(.*\)#/\1","/g' $(CURDIR)/foo/npm/*.csv
The reason that the first sed command "fails" is that sed doesn't reset the line counter between input files (on your system, and neither on my Mac OS X machine, see comments):
$ cat test1
a
b
g
$ cat test2
aa
bb
cc
$ sed -n '=' test1 test2 # the '=' sed command outputs line numbers
1
2
3
4
5
6
This is why the first sed command isn't doing what you want it to do, it only affects the first file's first line.
The solution is to loop over the files and call sed for each of them (untested in Makefile):
#for f in $(CURDIR)/foo/npm/*.csv; do \
sed -i '' '1 s/^.*$$/"bar","repository"/g' $f; \
done
Using find and xargs will also work, just make sure that find isn't picking up files further down in the folders.
EDIT: In light of the comments on this answer, I would recommend avoiding the use of sed -i on multiple files altogether, and convert both statements into for-loops (in this case, they may be collapsed into one loop with two statements):
#for f in $(CURDIR)/foo/npm/*.csv; do \
sed -i '' '1 s/^.*$$/"bar","repository"/g' $f; \
sed -i '' 's/\(.*\)#/\1","/g' $f; \
done
In my experience, using for-loops in Makefiles seems to be far more common compared to using find and xargs. This is probably due to incompatibility between find and xargs versions between Unices. It also makes the Makefile a lot easier to read if one uses explicit loops.
I managed to solve with:
#find $(CURDIR)/foo/npm -name "*.csv" -type f | xargs -L 1 sed -i '' '1 s/^.*$$/"bar"/g'
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 9 years ago.
Improve this question
I want to use cron job, that once per three day will clean and sort maillog.
My job looks like
/bin/sed -i /status=/!d /var/log/maillog |
(/bin/grep "status=bounced" /var/log/maillog | /bin/grep -E -o --color "\b[a-zA-Z0-9.-]+#[a-zA-Z0-9.-]+\.[a-zA-Z0-9.-]+\b" | /bin/sort -u >> /root/unsent.log) |
(/bin/grep "status=deferred" /var/log/maillog | /bin/grep -E -o --color "\b[a-zA-Z0-9.-]+#[a-zA-Z0-9.-]+\.[a-zA-Z0-9.-]+\b" | /bin/sort -u >> /root/deferred.log) |
(/bin/grep "status=sent" /var/log/maillog | /bin/grep -E -o --color "\b[a-zA-Z0-9.-]+#[a-zA-Z0-9.-]+\.[a-zA-Z0-9.-]+\b" | /bin/sort -u >> /root/sent.log) |
/bin/sed -i "/status=/d" /var/log/maillog
Job works fine and do 3 step:
Delete from maillog all lines that don't contain "status="
Sort sent, bounced, deffered in different logs.
Delete from maillog all lines that contain "status"
After this job my maillog is fully clean and sorted to 3 logs.
But Postfix doesn't want to write next records to maillog.
I delete sed command, and Postfix writes next records fine.
Why sed command blocks maillog after execution cron job?
sed -i will unlink the file it modifies, so syslog/postfix will continue writing to a nonexistent file.
From http://en.wikipedia.org/wiki/Sed:
Note: "sed -i" overwrites the original file with a new one, breaking any links the original may have had
It is more common to process log files after rotating them out of place with a tool like logrotate or savelog, so that syslog can continue writing uninterrupted.
If you must edit /var/log/maillog in place, you can add a line to the end of your cron job to reload syslog when you are done. Note that you can lose log lines written to the file while your script is running if you do this. The command will depend on what distribution / operating system you are running. On ubuntu, which uses rsyslog, it would be reload rsyslog >/dev/null 2>&1.
I've reformatted your original code to highlight the pipe-lines you added
/bin/sed -i /status=/!d /var/log/maillog \
| (/bin/grep "status=bounced" /var/log/maillog \
| /bin/grep -E -o --color "\b[a-zA-Z0-9.-]+#[a-zA-Z0-9.-]+\.[a-zA-Z0-9.-]+\b" \
| /bin/sort -u >> /root/unsent.log\
) \
| (/bin/grep "status=deferred" /var/log/maillog \
| /bin/grep -E -o --color "\b[a-zA-Z0-9.-]+#[a-zA-Z0-9.-]+\.[a-zA-Z0-9.-]+\b" \
| /bin/sort -u >> /root/deferred.log\
) \
| (/bin/grep "status=sent" /var/log/maillog \
| /bin/grep -E -o --color "\b[a-zA-Z0-9.-]+#[a-zA-Z0-9.-]+\.[a-zA-Z0-9.-]+\b" \
| /bin/sort -u >> /root/sent.log \
) \
| /bin/sed -i "/status=/d" /var/log/maillog
As #alberge noted, you could very likely lose log messages with all of this sed -i processing on the same file.
I propose a different approach:
I would move the maillog to a dated filename, (the assumption here is that Postfix, will create a new file with the standard name that it 'likes' to use (/var/log/maillog).
Then your real goal seems to be to extract various categories of messages to separately named files, i.e. unsent.log, deferred.log, sent.log AND then you're discarding any lines that don't contain the string status= (although you do that first).
Here's my alternate (please read the whole message, don't copy/paste/excute right away!).
logDate=$(/bin/date +%Y%m%d.%H%M%S)
/bin/mv /var/log/maillog /var/log/maillog.${logDate}
/bin/grep "status=bounced" /var/log/maillog.${logDate} \
| /bin/grep -E -o --color "\b[a-zA-Z0-9.-]+#[a-zA-Z0-9.-]+\.[a-zA-Z0-9.-]+\b" \
| /bin/sort -u \
>> /root/unsent.log.${logDate}
/bin/grep "status=deferred" /var/log/maillog.${logDate} \
| /bin/grep -E -o --color "\b[a-zA-Z0-9.-]+#[a-zA-Z0-9.-]+\.[a-zA-Z0-9.-]+\b" \
| /bin/sort -u \
>> /root/deferred.log.${logDate}
/bin/grep "status=sent" \
| /bin/grep -E -o --color "\b[a-zA-Z0-9.-]+#[a-zA-Z0-9.-]+\.[a-zA-Z0-9.-]+\b" \
| /bin/sort -u \
>> /root/sent.log.${logDate}
To test that this code is working, replace the 2nd line ( /bin/mv .... ) with
/bin/cp /var/log/maillog /var/log/maillog.${logDate}
Copy/paste that into a terminal window, confirm that the /var/log/maillog.${logDate} was copied correctly, then copy/paste each section, 1 at a time and check that the expected output is created in each of the /root logfiles.
(If you get error messages for any of these blocks, make sure there are NO space/tab chars after the last '\' char on each of the continued lines. OR you can fold each of those 3 pipelines back into one line, removing the '\' chars as you go.
(Note that to create each of the /root logfiles, I don't use any connecting sections via pipes surrounded by sub-processes. But, in other situations, I do use this sort of technique for advanced problems, so don't throw the technique away, just use it when it is really required ;-)!
After you confirm that all of this is working as you needed, then you extend the script to do a final cleaning up :
/bin/rm /var/log/maillog.${logDate}
I've added ${logDate} to each of your output files, but as I see you're using sort -u >> you may want to remove that 'extension' to your sub-logfile names (unsent.log, deferred.log, sent.log) And just let those files get grow naturally. In either case, you'll have to comeback at some point and determine how far back you want to keep this data, and develop a plan and method for how you'll clean up these logfiles when they're not useful. I think someone mentioned logrotate package. You might want to look into that as your long-term solution.
This solution avoids a lot of extra processes being created, and it eliminates (mostly) the possibility of lost log records. I'm think you might lose all or part of a record if Postfix is writing to the logfile in the same split-second as you are moving the file. But your solution would have similar problems AND more opportunities for that to happen.
If I have misunderstood the intention of your design, using the nested ( .... ) | ( .... ) sub-processes, sorry! Consider updating your post to include why you are using that techinque.
I hope this helps.
There are often times I will grep -n whatever file to find what I am looking for. Say the output is:
1234: whatev 1
5555: whatev 2
6643: whatev 3
If I want to then just extract the lines between 1234 and 5555, is there a tool to do that? For static files I have a script that does wc -l of the file and then does the math to split it out with tail & head but that doesn't work out so well with log files that are constantly being written to.
Try using sed as mentioned on
http://linuxcommando.blogspot.com/2008/03/using-sed-to-extract-lines-in-text-file.html. For example use
sed '2,4!d' somefile.txt
to print from the second line to the fourth line of somefile.txt. (And don't forget to check http://www.grymoire.com/Unix/Sed.html, sed is a wonderful tool.)
The following command will do what you asked for "extract the lines between 1234 and 5555" in someFile.
sed -n '1234,5555p' someFile
If I understand correctly, you want to find a pattern between two line numbers. The awk one-liner could be
awk '/whatev/ && NR >= 1234 && NR <= 5555' file
You don't need to run grep followed by sed.
Perl one-liner:
perl -ne 'if (/whatev/ && $. >= 1234 && $. <= 5555) {print}' file
Line numbers are OK if you can guarantee the position of what you want. Over the years, my favorite flavor of this has been something like this:
sed "/First Line of Text/,/Last Line of Text/d" filename
which deletes all lines from the first matched line to the last match, including those lines.
Use sed -n with "p" instead of "d" to print those lines instead. Way more useful for me, as I usually don't know where those lines are.
Put this in a file and make it executable:
#!/usr/bin/env bash
start=`grep -n $1 < $3 | head -n1 | cut -d: -f1; exit ${PIPESTATUS[0]}`
if [ ${PIPESTATUS[0]} -ne 0 ]; then
echo "couldn't find start pattern!" 1>&2
exit 1
fi
stop=`tail -n +$start < $3 | grep -n $2 | head -n1 | cut -d: -f1; exit ${PIPESTATUS[1]}`
if [ ${PIPESTATUS[0]} -ne 0 ]; then
echo "couldn't find end pattern!" 1>&2
exit 1
fi
stop=$(( $stop + $start - 1))
sed "$start,$stop!d" < $3
Execute the file with arguments (NOTE that the script does not handle spaces in arguments!):
Starting grep pattern
Stopping grep pattern
File path
To use with your example, use arguments: 1234 5555 myfile.txt
Includes lines with starting and stopping pattern.
If I want to then just extract the lines between 1234 and 5555, is
there a tool to do that?
There is also ugrep, a GNU/BSD grep compatible tool but one that offers a -K option (or --range) with a range of line numbers to do just that:
ugrep -K1234,5555 -n '' somefile.log
You can use the usual GNU/BSD grep options and regex patterns (but it also offers a lot more such as -K.)
If you want lines instead of line ranges, you can do it with perl: eg. if you want to get line 1, 3 and 5 from a file, say /etc/passwd:
perl -e 'while(<>){if(++$l~~[1,3,5]){print}}' < /etc/passwd
I am trying to comprehend Perl following the way describe in the book "Minimal Perl".
I've uploaded all source txt files onto my own server : results folder
I got the output from using several bash commands in a "chain" like this:
cat run*.txt | grep '^Bank[[:space:]]Balance'|cut -d ':' -f2 | grep -E '\$[0-9]+'
I know this is far from the most concise and efficient, but at least it works...
As our uni subject now moves onto the Perl part, I'd like to know if there is a way to get the same results in one line?
I am trying something like the following code but stuck in the middle:
Chenxi Mao#chenxi-a6b123bb /cygdrive/c/eMarket/output
$ perl -wlne 'print; if $n=~/^Bank Balance/'
syntax error at -e line 1, near "if $n"
Execution of -e aborted due to compilation errors.
you shouldn't have a ; after the print. So
perl -wlne 'print $1 if $n=~/^Bank Balance\s*:\s*(\d+)/'
perl -F/\:/ -ane 'print $F[1]."\n" if /Bank Balance/ && $F[1]!~/\$-/' run*.txt
also here's a short version of your bash command, using just awk
awk -F": " '/Bank[ \t]*Balance/&& $2!~/\$-/{print $2}' run*.txt