I have an GUI-based (TraitsUI/PyQt/Envisage) application written in Python. I would like to spawn an IPython Notebook in which I expose a small API and a number of objects. Those objects include a SQLAlchemy session and a bunch of SQLAlchemy models.
I've looked a lot, but I can't find any examples of this. I can start a notebook:
from IPython.frontend.html.notebook import notebookapp
app = notebookapp.NotebookApp.instance()
app.initialize()
app.start()
and that works well enough (although I'd prefer if 'start' was nonblocking... I assume I can do it in another thread if needed), but I can't alter the namespace.
I've also found examples like this:
from IPython.zmq.ipkernel import IPKernelApp
namespace = dict(z=1010)
kapp = IPKernelApp.instance()
kapp.initialize()
# Update the ns we want with special variables auto-created by the kernel
namespace.update(kapp.shell.user_ns)
# Now set the kernel's ns to be ours
kapp.shell.user_ns = namespace
kapp.start()
But I'm not sure how to actually open the Notebook from here.
Does anybody have any suggestions?
Thanks!
>>> import IPython
>>> z=1010
>>> IPython.embed()
Python 3.5.2 (default, Oct 8 2019, 13:06:37)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.9.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: z
Out[1]: 1010
Related
I have a Julia module in which I would like to import the Python function sympy.physics.wigner.wigner_9j. My minimal example module is as follows:
module my_module
using PyCall
using SymPy
export test
test()=sympy.physics.wigner.wigner_9j(1,1,1,1,1,1,1,1,1)
end
Then in my Julia notebook running:
using my_module
test()
gives
KeyError: key :physics not found
However adding to the notebook
#pyimport sympy.physics.wigner as sympy_wigner
sympy_wigner.wigner_9j(1,1,1,1,1,1,1,1,1)
gives the correct output. For some reason using #pyimport inside modules gives errors, which I typically avoid by using an __init__ inside my module, e.g adding to my_module.jl
const camb=PyNULL()
function __init__() # this should probably go in SFBBispectrum.jl
copy!(camb, pyimport_conda("camb", "camb", "conda-forge"))
pars=camb.CAMBparams()
end
which allows me to access camb.CAMBparams. Unfortunately I am failing to do something similar for sympy.physics.wigner.wigner_9j.
It has been awhile, but does this help https://github.com/JuliaPy/SymPy.jl/blob/master/src/physics.jl
I am trying to import some of my personal modules into my IPython Clusters. I am using Anacondas on Windows Vista 64 bit
from IPython.parallel import Client
rc = Client()
dview = rc[:]
with dview.sync_imports():
import lib.rf
It is giving me this error:
No module named 'lib.rf'
I can import the module in the rest of my IPython notebook, as I have this .bat file to start ipython notebook:
cd C:\Users\Jon\workspace\bf
set PYTHONPATH=%PYTHONPATH%;C:\Users\Jon\workspace\bf
C:\Anaconda\envs\p33\scripts\ipython notebook
I am using this similar code to start my ip clusters:
cd C:\Users\Jon\workspace\bf
set PYTHONPATH=%PYTHONPATH%;C:\Users\Jon\workspace\bf
C:\Anaconda\envs\p33\Scripts\ipcluster start --n=7
Why is this not working?
More info:
If I print out sys.path, I get a list that contains C:\Users\Jon\workspace\bf
If I print out the paths of my clusters, I get the same list:
%px sys.path
['',
'',
'',
'C:\\Anaconda\\envs\\p33\\lib\\site-packages\\distribute-0.6.28-py3.3.egg',
'C:\\Anaconda\\envs\\p33\\lib\\site-packages\\pykalman-0.9.5-py3.3.egg',
'C:\\Anaconda\\envs\\p33\\lib\\site-packages\\patsy-0.2.1-py3.3.egg',
'C:\\Anaconda\\envs\\p33\\lib\\site-packages\\joblib-0.8.3_r1-py3.3.egg',
'C:\\Users\\Jon\\workspace\\bf',
'C:\\Users\\Jon\\workspace\\bf\\my_numba',
'C:\\Anaconda\\envs\\p33\\python33.zip',
'C:\\Anaconda\\envs\\p33\\DLLs',
'C:\\Anaconda\\envs\\p33\\lib',
'C:\\Anaconda\\envs\\p33',
'C:\\Anaconda\\envs\\p33\\lib\\site-packages',
'C:\\Anaconda\\envs\\p33\\lib\\site-packages\\Sphinx-1.2.3-py3.3.egg',
'C:\\Anaconda\\envs\\p33\\lib\\site-packages\\win32',
'C:\\Anaconda\\envs\\p33\\lib\\site-packages\\win32\\lib',
'C:\\Anaconda\\envs\\p33\\lib\\site-packages\\Pythonwin',
'C:\\Anaconda\\envs\\p33\\lib\\site-packages\\runipy-0.1.1-py3.3.egg',
'C:\\Anaconda\\envs\\p33\\lib\\site-packages\\setuptools-7.0-py3.3.egg',
'C:\\Anaconda\\envs\\p33\\lib\\site-packages\\IPython\\extensions']
In [45]:
Further analysis:
%px lib.__path__
Out[0:11]: _NamespacePath(['C:\\Anaconda\\envs\\p33\\lib\\site-packages\\win32\\lib'])
lib.__path__
Out[57]: ['.\\lib']
Looks like the ipcluster and notebook are looking at lib in different places. I have tried renaming lib to mylib. It has not helped.
It seems that with dview.sync_imports() is being run someplace other than your IPython Notebook environment and is therefore relying a different PYTHONPATH. It is definitely not being run on one of the cluster engines and so wouldn't expect it to leverage your cluster settings of PYTHONPATH.
I'm thinking you'll need to have that directory in your PYTHONPATH (not your PATH) for the calling python environment because that is the location from which you are importing the modules.
The impact of the bit you have about setting the PYTHONPATH in the DOS shell from which you invoke ipclusters isn't clear to me. I can see that one might expect this to let the engines know about your directory, but I'm wondering if that PYTHONPATH gets initilized to the environment from which you call IPython.parallel.Client.
The following is my current IPython command prompt (regular terminal IPython, not the notebook, or Qt):
In [45]:
I'm done with my workflow, and am moving on to a new task. I'd like a way to reset the IPython input prompt number as follows:
In [1]:
I understand this may be possible from non-duplicate question to which I cannot reply nor answer for my case: How do I reset the IPython input prompt numbering?
For further clarity, an example of the command I'm after:
In [99]: %reset_command_number
In [1]: 2+2
Out[1]: 4
In [2]: %reset_command_number
In [1]:
The 3 Magic Questions:
Does such magic (%reset_command_number) exist?
If the answer to 1 is yes, what dark magic is it?
If the answer to 1 is no, is this achievable from a lower level IPython api? (yes i've searched the api docs and no i did not find a way to do it, yet...)
One possibility would be to modify the execution_count of the InteractiveShell:
import IPython; IPython.get_ipython().execution_count = 0
Using a one line command with ; makes it easier to call this repeatedly (if you use up-arrow to scroll the command history)
This has the side effect that after first command you will also see this error. I do not know if it has any side effects, though. (it already says that the history logging was moved to another session)
ERROR! Session/line number was not unique in database. History logging moved to new session 20741
Demo
In [1]: a = 1
In [2]: b = 88
In [3]: c = 99
In [4]: import IPython; IPython.get_ipython().execution_count = 0
In [1]: a
Out[1]: 1
ERROR! Session/line number was not unique in database. History logging moved to new session 20767
In [2]: b
Out[2]: 88
In [3]: c
Out[3]: 99
In [4]: import IPython; IPython.get_ipython().execution_count = 0
In [1]: a = 15
ERROR! Session/line number was not unique in database. History logging moved to new session 20768
In [2]: a
Out[2]: 15
In [3]: b
Out[3]: 88
Using an alias?
It is also possible to use an alias for the command. For example, create file i_reset.py and put it into your site-packages, ~/.ipython/extensions/, your PYTHONPATH or any other folder you can see in your sys.path
# i_reset.py
def i_reset(line):
"""
Reset IPython shell numbering to [1]
"""
import IPython; IPython.get_ipython().execution_count = 0
def load_ipython_extension(ipython):
"""This function is called when the extension is
loaded. It accepts an IPython InteractiveShell
instance. We can register the magic with the
`register_magic_function` method of the shell
instance."""
ipython.register_magic_function(i_reset, 'line')
Then, you may use the extension by first loading it and then calling it like so:
In [4]: %load_ext i_reset
In [5]: i_reset
In [1]: a
Out[1]: 1
I'm not sure you can, and what is done with the notebook is equivalent (from what I understood) to just open a new python session as if you open a new tab and start ipython again, and re-execute everything.
So, if I were you and had to start a new task and want to start at In [1] I would just start a new ipython environment.
The equivalent with ipython notebook to start a new task would be to start a new notebook.
The only reason I can imagine why you would do this is that you want to access some variable from the previous task.
What I would is to save the object you want to re-use using pickle or by saving a variable into a file an reload it in a new ipython session.
Did you looked at the Doc on Input caching system ? You may have answer to other question relative to yours.
Note that I'm aware that's not really an answer, but it's a bit more than just a comment, anyway, I hope this helps !
If you have more specific needs than just reseting the prompt number, just detail them.
Assuming you don't need the session data, the simplest is to exit and restart IPython.
$ ipython
In [1]: 1+1
Out[1]: 2
In [2]: 2+2
Out[2]: 4
In [3]: 3+3
Out[3]: 6
In [4]: exit
$ ipython
In [1]: print("Prompt numbering reset")
Prompt numbering reset
In [2]:
I would like to work with several ipython notebooks at once sharing the same namespace. Is there currently (ipython-1.1.0) a way to do this?
I tried creating different notebooks on the same ipython kernel, but the notebooks don't share a namespace. Also, I've been able to use a terminal console alongside a notebook on the same namespace using the answers in Using IPython console along side IPython notebook, but I couldn't find the notebook equivalent of the --existing argument.
Thanks a lot
Unfortunately this no longer works, you get error message ipython.kernel replaced by ipython.parallel.
A less elegant way than above to alter this is to change IPython/frontend/html/notebook/kernelmanager.py around line 273 from
kernel_id = self.kernel_for_notebook(notebook_id)
to
kernel_id = None
for notebook_id in self._notebook_mapping:
kernel_id = self._notebook_mapping[notebook_id]
break
For Anaconda python, replace start_kernel in kernelmanager.py with
def start_kernel(self, kernel_id=None, path=None, **kwargs):
global saved_kernel_id
if saved_kernel_id:
return saved_kernel_id
if kernel_id is None:
kwargs['extra_arguments'] = self.kernel_argv
if path is not None:
kwargs['cwd'] = self.cwd_for_path(path)
kernel_id = super(MappingKernelManager, self).start_kernel(**kwargs)
self.log.info("Kernel started: %s" % kernel_id)
self.log.debug("Kernel args: %r" % kwargs)
self.add_restart_callback(kernel_id,
lambda : self._handle_kernel_died(kernel_id),
'dead',
)
else:
self._check_kernel_id(kernel_id)
self.log.info("Using existing kernel: %s" % kernel_id)
saved_kernel_id = kernel_id
return kernel_id
and add
saved_kernel_id = None
above
class MappingKernelManager(MultiKernelManager):
True IPython gurus, please supply the correct fix. A lot of people using notebooks want the ability to share the kernel, it's natural, because one notebook quickly grows too big to work with a single complex application, so it is easier to be able to break down the application into multiple notebooks.
Also, gurus, while you're listening, it would be nice to have a collapse-expand feature as in Mathematica so you can only view the part of the notebook you care about and you can zoom out the rest.
The IPython Notebook does not have the equivalent of --existing. Notebooks do not share kernels. It is not a limitation of the notebook itself, it is just a design decision made in the notebook server code. The server code can be modified, for instance, to have all notebooks share the same kernel. You can do this with a little monkeypatching in your IPython configuration. Start by creating a profile:
$ ipython profile create singlekernel
[ProfileCreate] Generating default config file: u'~/.ipython/profile_singlekernel/ipython_config.py'
[ProfileCreate] Generating default config file: u'~/.ipython/profile_singlekernel/ipython_qtconsole_config.py'
[ProfileCreate] Generating default config file: u'~/.ipython/profile_singlekernel/ipython_notebook_config.py'
[ProfileCreate] Generating default config file: u'~/.ipython/profile_singlekernel/ipython_nbconvert_config.py'
and edit $(ipython locate profile singlekernel)/ipython_notebook_config.py to contain:
# Configuration file for ipython-notebook.
c = get_config()
import os
import uuid
from IPython.kernel.multikernelmanager import MultiKernelManager
def start_kernel(self, **kwargs):
"""Minimal override of MKM.start_kernel that always returns the same kernel"""
kernel_id = kwargs.pop('kernel_id', str(uuid.uuid4()))
if self.km is None:
self.km = self.kernel_manager_factory(connection_file=os.path.join(
self.connection_dir, "kernel-%s.json" % kernel_id),
parent=self, autorestart=True, log=self.log
)
if not self.km.is_alive():
self.log.info("starting single kernel")
self.km.start_kernel(**kwargs)
else:
self.log.info("reusing existing kernel")
self._kernels[kernel_id] = self.km
return kernel_id
MultiKernelManager.km = None
MultiKernelManager.start_kernel = start_kernel
This just overrides the kernel starting mechanism to start only one kernel and return it at every subsequent request,
rather than starting a new one for each kernel ID.
Now whenever you start the notebook server with
ipython notebook --profile singlekernel
all of the notebooks in that session will share the same kernel.
This is very simple to recreate.
If my script foo.py is:
import scipy
Then run:
python pyinstaller.py --onefile foo.py
When I launch foo.exe I get:
WARNING: file already exists but should not: C:\Users\username\AppData\Local\Temp\_MEI86402\Include\pyconfig.h
I've tested a few versions but the latest I've confirmed is 2.1dev-e958e02 running on Win7, Python 2.7.5 (32 bit), Scipy version 0.12.0
I've submitted a ticket with the Pyinstaller folks but haven't heard anything yet. Any clues how to debug this further?
You can hack the spec file to remove the second instance by adding these lines after a=Analysis:
for d in a.datas:
if 'pyconfig' in d[0]:
a.datas.remove(d)
break
The answer by wtobia# worked for me. See https://github.com/pyinstaller/pyinstaller/issues/783
Go to C:\Python27\Lib\site-packages\PyInstaller\build.py
Find the def append(self, tpl): function.
Change if tpl[2] == "BINARY": to if tpl[2] in ["BINARY", "DATA"]:
Expanding upon Ilya's solution, I think this is a little bit more robust solution to modifying the spec file (again place after the a=Analysis... statement).
a.datas = list({tuple(map(str.upper, t)) for t in a.datas})
I only tested this on a small test program (one with a single import and print statement), but it seems to work. a.datas is a list of tuples of strings which contain the pyconfig.h paths. I convert them all to lowercase and then dedup. I actually found that converting all of them all to lowercase was sufficient to get it to work, which suggests to me that pyinstaller does case-sensitive deduping when it should be case-insensitive on Windows. However, I did the deduping myself for good measure.
I realized that the problem is that Windows is case-insensitive and these 2 statements are source directories are "duplicates:
include\pyconfig.h
Include\pyconfig.h
My solution is to manually tweak the .spec file with after the a=Analysis() call:
import platform
if platform.system().find("Windows")>= 0:
a.datas = [i for i in a.datas if i[0].find('Include') < 0]
This worked in my 2 tests.
A more flexible solution would be to check ALL items for case-insensitive collisions.
I ran the archive_viewer.py utility (from PyInstaller) on one of my own --onefile executables that has the same error and found that pyconfig.h is included twice:
(31374007, 6521, 21529, 1, 'x', 'include\\pyconfig.h'),
(31380528, 6521, 21529, 1, 'x', 'Include\\pyconfig.h'),
(31387049, 984, 2102, 1, 'x', 'pytz\\zoneinfo\\CET'),
Sadly though, I don't know how to fix it.
PyInstaller Manual link:
http://www.pyinstaller.org/export/d3398dd79b68901ae1edd761f3fe0f4ff19cfb1a/project/doc/Manual.html#archiveviewer