I am trying to use Python's Popen() to retrieve graph data from a multiple rrd files. Due to complexity of app where the following piece of code is utilised, I rely on rrdtool graph parameter -Z for handling missing files for me:
#!/bin/python3
import subprocess
cmd = '/opt/rrdtool/bin/rrdtool graph - -a JSONTIME -Z --width 924 --start 1486428000 --end 1486471200 DEF:foo1=ch1.rrd:flows:MAX DEF:foo2=ch2.rrd:flows:MAX AREA:foo1#000:"ch1" AREA:foo2#606060:"ch2":STACK'
path = '/data/live/pokus/rrd/channels/'
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, cwd=path, shell=True)
p.wait()
if p.returncode is not 0:
print("Error")
else:
print(p.stdout.read().decode(encoding="utf-8"))
Following code works as expected when both files ch1.rrd and ch2.rrd are present. When one of them is missing, whole thing hangs indefinitely until I kill the rrdtool process manually from htop. Then python detects nonzero return code and reports error.
Using shell=False and shlex.split() on cmd does not help.
When I execute the same command from bash, even with the missing files rrdtool does the job as expected.
Unfortunately I can't use rrdtool bindings for python and also I am stuck on python 3.4.5. rrdtool version is 1.6.0.
I am glab for any idea how to overcome this. I would prefer solution that does not include testing whether files exist from python and keeps the -Z parameter in rrdtool command. Also using timeout on p.wait() isn't a viable solution.
Thanks in advance
Ok, I found the solution.
The reason why Python (namely p.wait) hanged was because rrdtool did not know the minimum step size (parameter -S) resulting in step size of two seconds. This way, output of the rrdtool was able to fill the OS buffers and that deadlocked p.wait. According to Python docs, Popen.communicate should be a way to go.
I'm relatively new to NumPy/SciPy and IPython.
To execute a python script in the python interactive mode, we may use the following commands.
>>> import os
>>> os.system('executable.py')
Then the print outputs can be seen from the python prompt.
But the same idea doesn't work with IPython notebook.
In [64]:
import os
os.system('executable.py')
Out[64]:
0
In this case, I cannot see any print outputs. The notebook only tells weather execution was successful or not. Are there any ways to see the outputs when I use IPython notebook?
Use the magic function %run:
%run executable.py
This properly redirects stdout to the browser and you will see the output from the program in the notebook.
It gives you both, the typical features of running from command line plus Python tracebacks if there is exception.
Parameters after the filename are passed as command-line arguments to
the program (put in sys.argv). Then, control returns to IPython's
prompt.
This is similar to running at a system prompt python file args,
but with the advantage of giving you IPython's tracebacks, and of
loading all variables into your interactive namespace for further use
(unless -p is used, see below).
The option -t times your script. With -d it runs in the debugger pdb. More nice options to explore.
I have an IPython notebook where I've accidentally dumped a huge output (15 MB) that crashed the notebook. Now when I open the notebook and attempt to delete the troublesome cell, the notebook crashes again—thus preventing me from fixing the problem and restoring the notebook to stability.
The best fix I can think of is manually pasting the input cells to a new notebook, but is there a way to just open the notebook without any outputs?
you can use cli to clear outputs
jupyter nbconvert --ClearOutputPreprocessor.enabled=True --inplace Notebook.ipynb
There is this nice snippet (that I use as a git commit hook) to strip the output of an ipython notebook:
#!/usr/bin/env python
def strip_output(nb):
for ws in nb.worksheets:
for cell in ws.cells:
if hasattr(cell, "outputs"):
cell.outputs = []
if hasattr(cell, "prompt_number"):
del cell["prompt_number"]
if __name__ == "__main__":
from sys import stdin, stdout
from IPython.nbformat.current import read, write
nb = read(stdin, "ipynb")
strip_output(nb)
write(nb, stdout, "ipynb")
stdout.write("\n")
You can easily make it a bit nicer to use, currently you'd have to call it as
strip_output.py < my_notebook.ipynb > my_notebook_stripped.ipynb
If you are running jupyter 4.x, you will get some API deprecation warnings when running filmor's script. Although the script still works, I update the script a bit to remove the warnings.
#!/usr/bin/env python
def strip_output(nb):
for cell in nb.cells:
if hasattr(cell, "outputs"):
cell.outputs = []
if hasattr(cell, "prompt_number"):
del cell["prompt_number"]
if __name__ == "__main__":
from sys import stdin, stdout
from nbformat import read, write
nb = read(stdin, 4)
strip_output(nb)
write(nb, stdout, 4)
stdout.write("\n")
As for later versions of jupyter, there is a Restart Kernel and Clear All Outputs... option that clears the outputs but also removed the variables.
Here is a further modification from #Edward Fung's answer that will output the cleaned notebook to a new file rather than rely on stin and stout
from nbformat import read, write
def strip_output(nb):
for cell in nb.cells:
if hasattr(cell, "outputs"):
cell.outputs = []
if hasattr(cell, "prompt_number"):
del cell["prompt_number"]
nb = read(open("my_notebook.ipynb"), 4)
strip_output(nb)
write(nb, open("my_notebook_cleaned.ipynb", "w"), 4)
Using the --ClearOutputPreprocessor, you can reduce the size of your notebook file due to the outputs.
jupyter nbconvert --ClearOutputPreprocessor.enabled=True --inplace sample.ipynb
Note that --clear-output is the broken command like below:
jupyter nbconvert --clear-output --inplace sample.ipynb
In my case, I tried to see the answer to this question, but I found out that it is a command that cannot remove output.
I am not able to post a commenet, so feel free to edit/move to #Shumaila Ahmed answer.
I had to use quotes on the file path, as:
jupyter nbconvert --ClearOutputPreprocessor.enabled=True --inplace 'Notebook.ipynb'
Works like charm on Ubuntu 21.04, thanks!
I've build an environment with zc.buildout including IPython script.
My problem is simple:
if I launch IPython in console, everything is OK and I get all my eggs in sys.path
but if I launch IPython notebook, I only get default system path.
Is there any way to include all my eggs while starting notebook?
Regards,
Thierry
So, I guess somewhere in the notebook startup a process is forked, which means sys.path will get reset and buildout's tricks won't help.
I solved the problems as follows, although it's a bit dirty:
Create an entry point as follows:
setup(...
entry_points = {
'console_scripts': ['ipython = <yourpackage>.ipython:main']
})
Put the following in /ipython.py:
from IPython.frontend.terminal.ipapp import launch_new_instance
import os
import sys
def main():
os.environ['PYTHONPATH']=':'.join(sys.path)
sys.exit(launch_new_instance())
Now, running bin/ipython notebook will give you the sys.path you expect.
Using IPython .10-2, I can do the following from a python command line program:
from IPython.Shell import IPShellEmbed
IPShellEmbed(argv=[])()
This spawns an interactive IPython session at the point of invocation. I have been unable to find an equivalent for the .12 series of IPython.
Is there?
thanks, jim
The simplest way:
from IPython import embed
embed()
There's lots more info in the docs.