I am trying to use Python's Popen() to retrieve graph data from a multiple rrd files. Due to complexity of app where the following piece of code is utilised, I rely on rrdtool graph parameter -Z for handling missing files for me:
#!/bin/python3
import subprocess
cmd = '/opt/rrdtool/bin/rrdtool graph - -a JSONTIME -Z --width 924 --start 1486428000 --end 1486471200 DEF:foo1=ch1.rrd:flows:MAX DEF:foo2=ch2.rrd:flows:MAX AREA:foo1#000:"ch1" AREA:foo2#606060:"ch2":STACK'
path = '/data/live/pokus/rrd/channels/'
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, cwd=path, shell=True)
p.wait()
if p.returncode is not 0:
print("Error")
else:
print(p.stdout.read().decode(encoding="utf-8"))
Following code works as expected when both files ch1.rrd and ch2.rrd are present. When one of them is missing, whole thing hangs indefinitely until I kill the rrdtool process manually from htop. Then python detects nonzero return code and reports error.
Using shell=False and shlex.split() on cmd does not help.
When I execute the same command from bash, even with the missing files rrdtool does the job as expected.
Unfortunately I can't use rrdtool bindings for python and also I am stuck on python 3.4.5. rrdtool version is 1.6.0.
I am glab for any idea how to overcome this. I would prefer solution that does not include testing whether files exist from python and keeps the -Z parameter in rrdtool command. Also using timeout on p.wait() isn't a viable solution.
Thanks in advance
Ok, I found the solution.
The reason why Python (namely p.wait) hanged was because rrdtool did not know the minimum step size (parameter -S) resulting in step size of two seconds. This way, output of the rrdtool was able to fill the OS buffers and that deadlocked p.wait. According to Python docs, Popen.communicate should be a way to go.
Related
I have a script that runs in the background which uses watch to monitor a directory for changes. This works just fine. However, I had a need for the script which runs the monitoring script to be run as daemon. The daemon is running as the same user, but now watch is returning "1 Various failures."
I suspect that there is some environment variable that is not set right, but there are too many to use trial and error to diagnose the issue. And unfortunately, "Various failures" is not very helpful. Any ideas how I might diagnose this?
The command is
watch -d -t -g ls -l
I think something like the following can be used as a work-around.
diff <(ls -l) <(sleep 1; ls -l)
I finally discovered the following on stderr.
Error opening terminal: unknown.
It was easily resolved by the answer to error opening terminal. So it did turn out to be an environment variable. I don't see this error message in the code. Perhaps it occurs when making a system call that needs the terminal.
I'm relatively new to NumPy/SciPy and IPython.
To execute a python script in the python interactive mode, we may use the following commands.
>>> import os
>>> os.system('executable.py')
Then the print outputs can be seen from the python prompt.
But the same idea doesn't work with IPython notebook.
In [64]:
import os
os.system('executable.py')
Out[64]:
0
In this case, I cannot see any print outputs. The notebook only tells weather execution was successful or not. Are there any ways to see the outputs when I use IPython notebook?
Use the magic function %run:
%run executable.py
This properly redirects stdout to the browser and you will see the output from the program in the notebook.
It gives you both, the typical features of running from command line plus Python tracebacks if there is exception.
Parameters after the filename are passed as command-line arguments to
the program (put in sys.argv). Then, control returns to IPython's
prompt.
This is similar to running at a system prompt python file args,
but with the advantage of giving you IPython's tracebacks, and of
loading all variables into your interactive namespace for further use
(unless -p is used, see below).
The option -t times your script. With -d it runs in the debugger pdb. More nice options to explore.
I'm trying to use named pipes in a project. I have two terminals open, Terminal A and Terminal B.
In terminal A, I issued this command:
mkfifo myFifo && tail -f myFifo | csh -s
It seems as if standard out is being redirected somewhere else, though, because my prompt disappears and some commands aren't reflected in terminal A.
For example, if in terminal B I begin a python session via issuing echo "python" > myFifo, then echo "print 'Hello, World'" > myFifo, I don't see Hello, World in terminal A.
However, if I issue echo ls > myFifo within terminal B, I see the correct output from ls in terminal A.
Does anyone know why sometimes the output appears and sometime it doesn't?
I'm running on CentOS 6.6
Thanks,
erip
You read from the FIFO with csh, if you start an interactive Python shell in csh, then it won't be reading from the FIFO because it's busy running python.
Python doesn't somehow automagically do a REPL on the FIFO. How should it even know about the FIFO? It has no knowledge of it.
You could, perhaps, tell Python to read commands from the FIFO with something like:
>>> import os, sys, time
>>> fifo = open(os.open('myFifo', os.O_NONBLOCK), 'r')
And then:
$ echo 'print(42+5)' > ! myFifo
Will give you:
>>> eval(fifo.read())
47
Perhaps there's also a way to tell Python to read commands from myFifo by overwriting sys.stdin, but I can't get that working in my testing.
It's a bit unclear to me what exactly you're trying to achieve here, though. I suspect there might be another solution which is much more appropriate to the problem you're having.
I'm running a script in solaris 11 with different results depending of the shell used.
The script has an echo redirecting to a file given by an environment value:
echo "STARTING EXEC" >> $FILE
ps. EXEC is just the message the script show, it's not using exec command.
If I execute the script without the variable FILE defined (using /usr/bin/ksh):
./start.sh[10]: : cannot open
and the script continue the execution.
The flags for ksh are:
echo $-
imsuBGEl
But if I change to /usr/xpg4/bin/sh, the script show me the echo in stdout and there is no error shown.
The flags for xpg4 sh are:
echo $-
imsu
I tried to change the flags with set +- (I can't remove El flags, but BG are removed ok), but can't get the same behavior.
Is there anything I can do to get the same result using ksh without cannot open error?
/usr/bin/ksh --version
version sh (AT&T Research) 93u 2011-02-08
I'll want the script keep going, showing the message in stdout, instead of showing the error just like it does now.
Like shellter said in the comments, the good thing to do is to check if the FILE variable is defined before doing anything. This is a script migration from an HPUX to a SOLARIS environment, and client think they must have the same result as before (we unset FILE variable before execution to test it).
You are likely running Solaris 11, not Solaris 64.
Should you want to have your scripts to work under Solaris 11 without having to search everywhere the bogus redirections, you can simply replace all shebangs (first line) by #!/usr/xpg4/bin/sh.
The final solution we are going to take is to install the ksh88 package and use it like default shell (/usr/sunos/bin/ksh). This shell have the same behavior the client had before, and we can let the scripts with no modifications.
The ksh used in solaris 11 is the 93 (http://docs.oracle.com/cd/E23824_01/html/E24456/userenv-1.html#shell-1)
Thanks #jlliagre and #shellter for your help.
I have some code in python that works but is unfortunately super slow. Someone at #python suggested that I can run the code through a profiler to see the lines and functions in which the code was taking the most amount of time.
The python source code that I want to profile reads from STDIN. But since the input is large I compiled the input as a file so that I can simply redirect it to the python code at the shell. So at the shell, I issue the command..
cat input | python pythonsource.py
The problem is, when I try to run the profiler in ipython I can't seem to find a way to redirect the input to the python code. At the ipython shell, I tried,
run -p -l 1.0 pythonsource.py input (didn't work. simply waits at STDIN for input)
run -p -l 1.0 pythonsource.py << input (didn't work)
run -p -l 1.0 cat input | python pythonsource.py (didn't work.)
I'm not sure how to do I can make the ipython profiler command redirect the input to STDIN for the pythonsource to read from. Could someone please tell me how to fix this? Or have I got it totally wrong? Maybe there is some other cleaner, more smarter way of profiling python code?
And maybe what I ask next should be a part of another question..but I was wondering what does ipython mean when it refers to "primitive calls" in some of the output of the ipython profiler?
Thank you.
The right way to redirect input on the shell commandline is as follows:
cat input | run -p -l 1.0 python pythonsource.py
run -p -l 1.0 python pythonsource.py < input
The syntax << string creates a "here document", it does not redirect input.
From the python prompt (and hopefully in ipython), you can redirect standard input like this:
import sys
save_stdin = sys.stdin
sys.stdin = open('input')
run -p <etc.>
sys.stdin = save_stdin # Restore the real stdin
Or (recommended) you could rewrite your source to work with a filename: If you call it as follows, "input" will be in sys.argv[1] and you can open it and read from it:
python pythonsource.py input