I'm using feedgnuplot and gnuplot to show real-time data on my linux desktop.
My data is produced by command which outputs 12 space separated integer values on each line and up to four lines per second.
I like to add the time, so I put time in front of the line before feeding data to feedgnuplot.
If I specify the time as seconds, the data is properly plotted, but the x-axis is quite unreadable:
data-producer |
while read -a line
do
echo -n $(date +"%s")
for n in "${line[#]}"
do
echo -en "\t$n"
done
echo
done |
feedgnuplot --terminal 'qt' \
--domain \
--stream 1 \
--xlen 5 \
--with lines \
--set "xdata time" \
--set "timefmt '%s'"
So I tried to get human readable time on the horizontal scale:
data-producer |
while read -a line
do
echo -n $(date +"%H:%M:%S")
for n in "${line[#]}"
do
echo -en "\t$n"
done
echo
done |
feedgnuplot --terminal 'qt' \
--domain \
--stream 1 \
--xlen 5 \
--with lines \
--set "xdata time" \
--set "timefmt '%H:%M:%S'"
This line does not work because feedgnuplot complains about comparison operators not applied to numeric data:
Argument "09:45:58" isn't numeric in numeric lt (<) at /usr/bin/feedgnuplot line 694.
Argument "09:45:57" isn't numeric in numeric ge (>=) at /usr/bin/feedgnuplot line 797.
Looking into the feedgnuplot code (it is a perl script) I see that comparison is performed on x values to sort them and to assess whether the graph has to be plotted again or not.
Is it possible to have feedgnuplot handle times by using some command line switches? If not, is there any other option before resorting to patching the feedgnuplot source code? Thank you.
Gnuplot requires some special settings for datetime data (e.g. a using statement must be specified). Therefore, feedgnuplot provides an own option for time data, --timefmt <format>:
for i in `seq 0 100`; do echo $i; sleep 1; done |
while read -a line
do
echo -n $(date +"%Y-%m-%d %H:%M:%S")
for n in "${line[#]}"
do
echo -en "\t$n"
done
echo
done |
feedgnuplot --terminal 'qt' \
--domain \
--stream 1 \
--lines \
--timefmt "%Y-%m-%d %H:%M:%S" \
--set 'format x "%H:%M:%S"'
Note, that different versions of gnuplot use different reference points for time in seconds, so that versions 4.6 (reference 1st January 2000) and earlier give wrong results when using %s. So it is better to use a time format of the kind %H:%M:%S. In the code above I used a fully defined datetime to avoid possible problems with day-spanning plots.
Related
I have scripts like this
#!/bin/sh
SHELL_CMD=busybox
and try to substitute some pattern to bash shell using makefile,
#!/bin/bash
#SHELL_CMD=busybox
Followings are the procedures in Makefile
release:
#rm -rf my_temp/
#mkdir my_temp/
#cp dir1/burn_*.sh dir1/dump_*.sh my_temp/
#cd my_temp/; \
for f in $(shell ls); do \
sed 's:#!/bin/sh\nSHELL_CMD=busybox:#!/bin/bash\n#SHELL_CMD=busybox:1' $${f} > temp; mv temp $${f}; \
done; \
cd ..;
#cd my_temp/; tar -jcv -f bash.tar.bz2 *.sh; cd ..;
My questions are:
1 in the for loop, it didn't get the correct script names in for loop.
How to patch it ?
Any better patterns in this sed substitution?
You are much better off doing the substitution without trying to match the newline in the source string. Unless you are ready to do some complex sed-fu (How can I replace a newline (\n) using sed?) you can just apply the substitution on each of the line with a
You can do this to apply the action on both 1st and 2nd lines. Also the $(shell ls) part is not needed. You can just run a shell glob expression to get you the files ending with .sh
#for f in *.sh; do \
sed -i -e '1 s:#!/bin/sh:#!/bin/bash:1' -e '2 s:^:#:1' $${f} ;\
done
If you don't want the -i in-place substitution, use the tmp file approach as you had originally shown.
With Sed, I want to use "!" to exclude lines that matches "he". Here is an example.
echo "hello" |sed "/he/!s/hello/hi/"
To my surprise, my ubuntu 14 returns
"bash !s/hello/hi: event not found"
error. Any ideas? How could I exclude line ranges corresponding to a pattern with Sed?
You have history expansion enabled. You need to disable it with set +H.
Example
Let's enable history expansion and run your command:
$ set -H
$ echo "hello" |sed "/he/!s/hello/hi/"
bash: !s/hello/hi/: event not found
Now, let's disable it and observe that the command now runs correctly and without error:
$ set +H
$ echo "hello" |sed "/he/!s/hello/hi/"
hello
Alternative
If you want to keep history expansion enabled, then single-quote your string:
$ set -H
$ echo "hello" |sed '/he/!s/hello/hi/'
hello
I would like to execute this make command to first replace the first line of all csv files inside the directory and then replace the # for commas through the other lines.
The second command is working fine and does what it is supposed to do, but the first one only replaces the line on the first file.
Could anyone give me a help on that?
csv:
$(DOCKER_RUN) npm run csv-generator
make format-csv
format-csv:
#sed -i '' '1 s/^.*$$/"bar","repository"/g' $(CURDIR)/foo/npm/*.csv
#sed -i '' 's/\(.*\)#/\1","/g' $(CURDIR)/foo/npm/*.csv
The reason that the first sed command "fails" is that sed doesn't reset the line counter between input files (on your system, and neither on my Mac OS X machine, see comments):
$ cat test1
a
b
g
$ cat test2
aa
bb
cc
$ sed -n '=' test1 test2 # the '=' sed command outputs line numbers
1
2
3
4
5
6
This is why the first sed command isn't doing what you want it to do, it only affects the first file's first line.
The solution is to loop over the files and call sed for each of them (untested in Makefile):
#for f in $(CURDIR)/foo/npm/*.csv; do \
sed -i '' '1 s/^.*$$/"bar","repository"/g' $f; \
done
Using find and xargs will also work, just make sure that find isn't picking up files further down in the folders.
EDIT: In light of the comments on this answer, I would recommend avoiding the use of sed -i on multiple files altogether, and convert both statements into for-loops (in this case, they may be collapsed into one loop with two statements):
#for f in $(CURDIR)/foo/npm/*.csv; do \
sed -i '' '1 s/^.*$$/"bar","repository"/g' $f; \
sed -i '' 's/\(.*\)#/\1","/g' $f; \
done
In my experience, using for-loops in Makefiles seems to be far more common compared to using find and xargs. This is probably due to incompatibility between find and xargs versions between Unices. It also makes the Makefile a lot easier to read if one uses explicit loops.
I managed to solve with:
#find $(CURDIR)/foo/npm -name "*.csv" -type f | xargs -L 1 sed -i '' '1 s/^.*$$/"bar"/g'
Simple newbie question that seems to be stumping me.
I've looked at other topics and resources and I'm not seeing the simple answer.
I'm running a very simple (perl -lne) command that helps me query specific data out of a pwebstats output , and output into a linear .csv file format for my reports.
The question is this:
Is there a quick, and clean way of assigning variables for the file name, Year , Month and Day ( i.e allowing the person running the command to simply enter the File Name ($FILENAME) , Year ($YEAR), Month ($MONTH) and Day ($DAY) and the execute without having to alter the perl -lne command each time?)
EXAMPLE OF COMMAND :
zcat /opt/log/file/$Year/$Month/$FILENAME/sm/$FILENAME.$Year-$Month-$Day \
| perl -lne '/.{0,0}FILENAME PAX Total HTTP IP Packet Count.{0,6}/ && print $&' \
> /tmp/jhoney/$FILENAME-IPcountTMP.csv
zcat /opt/log/file/$Year/$Month/$FILENAME/sm/$FILENAME.$Year-$Month-$Day \
| perl -lne '/.{0,0}PAX HTTP Average TCP RTT.{0,20}/ && print $&' \
> /tmp/jhoney/$FIELNAME-TCPrttTMP.csv
paste -d , /tmp/jhoney/$FILENAME-IPcountTMP.csv /tmp/jhoney/$FILENAME-TCPrttTMP.csv \
> /tmp/jhoney/$FILENAME-PAXdata.csv
rm -r /tmp/jhoney/$FILENAME-TCPrttTMP.csv /tmp/jhoney/$FILENAME-IPcountTMP.csv
I know the answer is looking right in my face, but, I'm learning perl and pyhton as I go.
Any help would be appreciated.
What you keep calling a Perl command are sh commands. Parsing command line options in sh is not trivial. If you don't mind switching to positional parameters, it would be a lot simpler.
Usage:
PAXStat 2015 09 06 FILENAME
Script:
#!/bin/sh
Year=$1
Month=$2
Day=$3
FILENAME=$4
zcat /opt/log/file/$Year/$Month/$FILENAME/sm/$FILENAME.$Year-$Month-$Day \
| perl -lne '/.{0,0}FILENAME PAX Total HTTP IP Packet Count.{0,6}/ && print $&' \
> /tmp/jhoney/$FILENAME-IPcountTMP.csv
zcat /opt/log/file/$Year/$Month/$FILENAME/sm/$FILENAME.$Year-$Month-$Day \
| perl -lne '/.{0,0}PAX HTTP Average TCP RTT.{0,20}/ && print $&' \
> /tmp/jhoney/$FIELNAME-TCPrttTMP.csv
paste -d , /tmp/jhoney/$FILENAME-IPcountTMP.csv /tmp/jhoney/$FILENAME-TCPrttTMP.csv \
> /tmp/jhoney/$FILENAME-PAXdata.csv
rm -r /tmp/jhoney/$FILENAME-TCPrttTMP.csv /tmp/jhoney/$FILENAME-IPcountTMP.csv
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 9 years ago.
Improve this question
I want to use cron job, that once per three day will clean and sort maillog.
My job looks like
/bin/sed -i /status=/!d /var/log/maillog |
(/bin/grep "status=bounced" /var/log/maillog | /bin/grep -E -o --color "\b[a-zA-Z0-9.-]+#[a-zA-Z0-9.-]+\.[a-zA-Z0-9.-]+\b" | /bin/sort -u >> /root/unsent.log) |
(/bin/grep "status=deferred" /var/log/maillog | /bin/grep -E -o --color "\b[a-zA-Z0-9.-]+#[a-zA-Z0-9.-]+\.[a-zA-Z0-9.-]+\b" | /bin/sort -u >> /root/deferred.log) |
(/bin/grep "status=sent" /var/log/maillog | /bin/grep -E -o --color "\b[a-zA-Z0-9.-]+#[a-zA-Z0-9.-]+\.[a-zA-Z0-9.-]+\b" | /bin/sort -u >> /root/sent.log) |
/bin/sed -i "/status=/d" /var/log/maillog
Job works fine and do 3 step:
Delete from maillog all lines that don't contain "status="
Sort sent, bounced, deffered in different logs.
Delete from maillog all lines that contain "status"
After this job my maillog is fully clean and sorted to 3 logs.
But Postfix doesn't want to write next records to maillog.
I delete sed command, and Postfix writes next records fine.
Why sed command blocks maillog after execution cron job?
sed -i will unlink the file it modifies, so syslog/postfix will continue writing to a nonexistent file.
From http://en.wikipedia.org/wiki/Sed:
Note: "sed -i" overwrites the original file with a new one, breaking any links the original may have had
It is more common to process log files after rotating them out of place with a tool like logrotate or savelog, so that syslog can continue writing uninterrupted.
If you must edit /var/log/maillog in place, you can add a line to the end of your cron job to reload syslog when you are done. Note that you can lose log lines written to the file while your script is running if you do this. The command will depend on what distribution / operating system you are running. On ubuntu, which uses rsyslog, it would be reload rsyslog >/dev/null 2>&1.
I've reformatted your original code to highlight the pipe-lines you added
/bin/sed -i /status=/!d /var/log/maillog \
| (/bin/grep "status=bounced" /var/log/maillog \
| /bin/grep -E -o --color "\b[a-zA-Z0-9.-]+#[a-zA-Z0-9.-]+\.[a-zA-Z0-9.-]+\b" \
| /bin/sort -u >> /root/unsent.log\
) \
| (/bin/grep "status=deferred" /var/log/maillog \
| /bin/grep -E -o --color "\b[a-zA-Z0-9.-]+#[a-zA-Z0-9.-]+\.[a-zA-Z0-9.-]+\b" \
| /bin/sort -u >> /root/deferred.log\
) \
| (/bin/grep "status=sent" /var/log/maillog \
| /bin/grep -E -o --color "\b[a-zA-Z0-9.-]+#[a-zA-Z0-9.-]+\.[a-zA-Z0-9.-]+\b" \
| /bin/sort -u >> /root/sent.log \
) \
| /bin/sed -i "/status=/d" /var/log/maillog
As #alberge noted, you could very likely lose log messages with all of this sed -i processing on the same file.
I propose a different approach:
I would move the maillog to a dated filename, (the assumption here is that Postfix, will create a new file with the standard name that it 'likes' to use (/var/log/maillog).
Then your real goal seems to be to extract various categories of messages to separately named files, i.e. unsent.log, deferred.log, sent.log AND then you're discarding any lines that don't contain the string status= (although you do that first).
Here's my alternate (please read the whole message, don't copy/paste/excute right away!).
logDate=$(/bin/date +%Y%m%d.%H%M%S)
/bin/mv /var/log/maillog /var/log/maillog.${logDate}
/bin/grep "status=bounced" /var/log/maillog.${logDate} \
| /bin/grep -E -o --color "\b[a-zA-Z0-9.-]+#[a-zA-Z0-9.-]+\.[a-zA-Z0-9.-]+\b" \
| /bin/sort -u \
>> /root/unsent.log.${logDate}
/bin/grep "status=deferred" /var/log/maillog.${logDate} \
| /bin/grep -E -o --color "\b[a-zA-Z0-9.-]+#[a-zA-Z0-9.-]+\.[a-zA-Z0-9.-]+\b" \
| /bin/sort -u \
>> /root/deferred.log.${logDate}
/bin/grep "status=sent" \
| /bin/grep -E -o --color "\b[a-zA-Z0-9.-]+#[a-zA-Z0-9.-]+\.[a-zA-Z0-9.-]+\b" \
| /bin/sort -u \
>> /root/sent.log.${logDate}
To test that this code is working, replace the 2nd line ( /bin/mv .... ) with
/bin/cp /var/log/maillog /var/log/maillog.${logDate}
Copy/paste that into a terminal window, confirm that the /var/log/maillog.${logDate} was copied correctly, then copy/paste each section, 1 at a time and check that the expected output is created in each of the /root logfiles.
(If you get error messages for any of these blocks, make sure there are NO space/tab chars after the last '\' char on each of the continued lines. OR you can fold each of those 3 pipelines back into one line, removing the '\' chars as you go.
(Note that to create each of the /root logfiles, I don't use any connecting sections via pipes surrounded by sub-processes. But, in other situations, I do use this sort of technique for advanced problems, so don't throw the technique away, just use it when it is really required ;-)!
After you confirm that all of this is working as you needed, then you extend the script to do a final cleaning up :
/bin/rm /var/log/maillog.${logDate}
I've added ${logDate} to each of your output files, but as I see you're using sort -u >> you may want to remove that 'extension' to your sub-logfile names (unsent.log, deferred.log, sent.log) And just let those files get grow naturally. In either case, you'll have to comeback at some point and determine how far back you want to keep this data, and develop a plan and method for how you'll clean up these logfiles when they're not useful. I think someone mentioned logrotate package. You might want to look into that as your long-term solution.
This solution avoids a lot of extra processes being created, and it eliminates (mostly) the possibility of lost log records. I'm think you might lose all or part of a record if Postfix is writing to the logfile in the same split-second as you are moving the file. But your solution would have similar problems AND more opportunities for that to happen.
If I have misunderstood the intention of your design, using the nested ( .... ) | ( .... ) sub-processes, sorry! Consider updating your post to include why you are using that techinque.
I hope this helps.