I have a notebook script that I run on different datasets. I want to save the script, INCLUDING the output cells, in the data folder each time I run it.
I have the following command placed at the end of my script that I run in jupyter. I intend to save pretty much what I can see on the screen to a HTML file.
"here is my notebook script with inputs and ouput including graphs"
cmd='jupyter nbconvert --to html odnp_postprocessing.ipynb --output-dir '+dataFolder
os.system(cmd)
However, nbconvert does not export the actual cells. It will print out only the input cell without ouput the first time I run, but if I re-run, it will finally export both input and output. However then if I change something in the script, it will always export the first version. Then the only way around I found is to restart the kernel and re-run (twice) the new script with the modification.
Basically, it looks like nbconvert exports some kind of buffer that is not necesseraly the actual input and ouput cells that the user sees.
What I want to do, which is programmatically saved my notebook (inputs and ouputs) into HTML, each time I run it.
Is there a command to save the current version of the notebook? I tried to add %notebook before nbconvert command but a whole bunch of old inputs are saved as well.
I know I can run the notebook within nbconvert, but I'd like to avoid it as I already run it manually in jupyter.
Any idea?
I'm using jupyter through enthought canopy in Chrome browser.
Thanks
You're not telling us exactly how you are running nbconvert; from the current notebook ? Thus it is hard to figured things out.
Nbconvert converts the current file as it is on disk; as a wild guess: you haven't save your file. If you do not save your file then nbconvert will likely not have access to the outputs of cells; and wild guess again when you run it a second time autosave have kicked in.
Remember:
- Nbconvert does not execute the file
- Nbconvert is a separate process it can't magically access what is in your browser, which is potentially a different machine.
Usually think of it this way:
Run the notebook;
Save the notebook
Close the notebook
Run nbconvert.
reopen the notebook.
If you are using this command at the end of a notebook to save it in another format, then what you are looking for are save hooks that will trigger some code – server-side – every time you save a notebook.
Side note, learn about how to run shell command in IPython; ! can be used to execute shell command in CWD and does variable interpolation.
Related
I've got a notebook that has got a bit unwieldy and I'm doing some refactoring which isn't fun.
I was wondering if it would be possible to execute code in this notebook from the command line for debugging.
Ideally, I would run something like:
run-in-jupyter $notebook file.py
and see the output from the command line. There is an interpreter in jupyterlab that can do this, so this make me think that it is possible.
I have a brief search but couldn't find much
How to run an .ipynb Jupyter Notebook from terminal? I explicitly don't want to do this (I want to run commands in an existing instace)
There is this library but this seems quite involved and some of the results I found on the internet where people not being able to use the library
jupyter console (pip install jupyter-console) connects to a running jupyter kernel from the kernel. Details on running kernels can be found amongst jupyter's run time files, on my box these live in ~/.local/share/jupyter/runtime. You can find the path to the kernel data file corresponding to an open workbook with %config IPKernelApp.connection_file which will look something like ~/.local/share/jupyter/runtime/kernel-55da8a07-b67d-4584-9ec6-f24e4a26cbbd.json.
You can then connect from the command line with
jupyter console --existing ~/.local/share/jupyter/runtime/kernel-55da8a07-b67d-4584-9ec6-f24e4a26cbbd.json
You can pipe commands into it as shown
echo h=87 | jupyter console --existing 55da8a07-b67d-4584-9ec6-f24e4a26cbbd 'h=57' --simple-prompt -y
I'm using sublime text 3 and I want to run ipynb files. I have installed helium. In the document, it says to execute a block I should execute Helium: Execute Block in the command palette. But can't seem to find that command.
Step 1.) run %connect_info in a localhosted jupyter notebook and copy the json output
Step 2.) In sublime text editor, press "Ctrl+shift+P" and type hermes: start kernel and copy paste this json you copied from step 1.
You should see a blank page popout.
Step 3.) Open your py file you want to execute, and select the lines you want to execute using mouse and again press "Ctrl+shift+P" and type hermes: execute cell (you will see an option now), click on that and your output should run on the black page which popped out in step 2.
Note : If your localhost jupyter notebook is running in a specific folder, make sure that the py file you want to run using helium/hermes is under that jupyter notebook access, basically the py file has to be under the same folder you ran jupyter from.
Using jupyter-lab
%run otherNotebook.ipynb
gives the following error message
Error: file not found otherNotebook.ipynb.py
How can I use the magic method and prevent it from adding .py to the file
As described here %run is for running a named file inside IPython as a program. Jupyter notebooks are not Python programs.
Notebooks can be converted to Python programs/scripts using Jupytext. Following that conversion you could then use %run.
Alternatively, you can use nbconvert to execute a notebook or use Papermill to execute a notebook. Papermill allows you to easily pass in parameters at the time of run. I have an example of both commented out in code under 'Step #5' here and 'Step#2' here.
If you are actually trying to bring the code into your present notebook, then you may want to explore importing Jupyter notebooks as modules. importnb is recommended here for making importing notebooks more convenient. Or, I just came across the subnotebook project that let's you run a notebook as you would call a Python function, pass parameters and get results back, including output contents.
I was using iPython command line interface and after some operations I want to save my operation history to a notebook file. But I was not using iPython notebook from the beginning. Can I still make it?
From #Thomas K (I don't know why he didn't post an answer):
%notebook -e myhistory.ipynb
The short answer is in a couple of ways, the slightly longer answer is Yes - but you might not get what you expect!
Really long answer: The explanation is that when you are working in a notebook, now called a jupyter notebook of course, your work is stored in a series of cells each of which has one or more lines of code or markdown while when you are working in a console all of your work is a series of lines of python code.
From within a console session you can save, using %save some or all of your work to one or more python files that you can then paste, import, etc, into notebook cells. You can also save using %save -r to .ipy files your work including the magics as magics rather than the results of magics that again you can use from within your notebook later.
You can also use the %notebook magic to save all of your current history in one of an ipynb json file or a python .py text file with the -e export flag. However, it is not clear from the documentation if the history will end up in a single cell, one cell per command or some other division. A little testing suggests one cell per numbered line of your console, so a single command or definition, per cell.
Personally I will stick with outputting anything useful into python files using the %save command - or better yet start a notebook when I think I might be doing something that I would need later.
If you are using jupyter console (ie, the command line version of jupyter notebook) you can use the following command within a cell to save the notebook as a python file (.py)
save yourfilename
this will save the contents of the notebook as "yourfilename.py" in your current working directory (from which you started the jupyter console)
Alternatively, you can save the whole program by this command :
%save -r program-name 1-999999
The following commands were written to file program-name.ipy
I am new to ipython notebook, and I would like to convert my ipynb to pdf. But I get the following error when I try to Download as PDF via LaTex.
nbconvert failed: pdflatex not found on PATH
There is no documentation anywhere how to add pdflatex to my PATH. I use windows. Thank you!
A simple and surprisingly good solution is to print the notebook to pdf through the browser with ctrl+p. Just make sure your plots and figures are not on interactive mode otherwise they will not be displayed (set them to %matplotlib inline).
Exporting jupyter notebooks through latex is quite troublesome and takes a lot of tinkering to get something remotely close to publish ready. When I absolutely need publication quality I do it on a latex editor, but this tutorial goes in great length about doing it on jupyter.
A few useful tips to get better results:
Higher resolution plots
Hide your code-cells from the pdf
Take a look at these extensions to improve your jupyter documents
For Mac OS X, the solution for me was to install MacTex first and then export the path to find it:
### TeX
export PATH="/Library/TeX/Distributions/.DefaultTeX/Contents/Programs/texbin:$PATH"
You can add this to your .bash_profile or similar config file to load it every time.
See more here https://github.com/jupyter/nbconvert/issues/406
As said by Thomas K in the comments, you need to have Latex installed, and after add the path to the directory containing pdflatex.exe file to the PATH variable of your system.
I have looked for a lightweight distribution and tried installing TeXworks, but I didn't find any pdflatex.exe file.
So I have tried TeX Live, which worked fine creating the pdflatex.exe file under the target installation directory. This path should be like C:\...\texlive\2016\bin\win32.
Finally, you should just add this path to the PATH environment variable of your system (you can use the link shared by Thomas K).
As said here, you need to quit jupyter notebook and open a new command prompt after making any path changes, in order for jupyter to find the newly added item to the PATH.
Then, in Jupyter, you can check your environment variables by running the following (refer to this link for details):
import os
os.environ['PATH'].split(';')
and check if it contains the path to pdflatex.exe file.
If you get some trouble when exporting your notebook to pdf due to missing files/packages (this happened to me), refer to this link to search and install them under TeX Live.
For Linux, the reported error is due to the lack of XeLatex, part of the texlive-xetex package.
Installation in ubuntu will be:
sudo apt install texlive-xetex
Instead of using nbconvert what you can do is :
Download your ipynb file as HTML from File option.
Right-click and select print or use Ctrl+P.
Save as PDF
Easy.
Here is the full solution that worked for me (for Mac).
brew cask install mactex
$ cd ~/
$ touch .bash_profile
This will open the bash profile on TextEditor
$ open -e .bash_profile
Paste the following to the top and save
export PATH="/Library/TeX/Distributions/.DefaultTeX/Contents/Programs/texbin:$PATH"
Close any notebook you have and reopen it
Here is the notebook explaining it step by step:
https://github.com/ybaktir/notes/blob/master/Convert%20Jupyter%20Notebook%20to%20Pdf.ipynb
I agree that latex installation (at least on windows) is painful and the result in my case was not a great looking document. The ctrl-p method alone doesn't work great if you're running in JupyterLab, but if you export the notebook to HTML, then print from the browser, choosing PDF, the result is quite good.
I know my solution is not at a level. But it works !!
in your browser of notebook tab, simply do "ctrl + p" to get download in pdf
First export the notebook file to HTML (available through File> Download as..).
If you are using JupyterLab, then this is available under File > Export Notebook As....
Use (any) free online converters to convert html file to a pdf file. (One such free online converter is sejda (https://www.sejda.com/html-to-pdf)
Note, there are many such converters are available online.