Fail to import IPython parallel in Jupyter - ipython

I recently made an update of IPython to 4.0.0 and installed Jupyter 4.0.6.
I wanted to use Ipython parallel, and after starting the engines in the notebook, I imported:
from IPython import parallel
And it fails:
~/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/IPython/utils/traitlets.py:5: UserWarning: IPython.utils.traitlets has moved to a top-level traitlets package.
warn("IPython.utils.traitlets has moved to a top-level traitlets package.")
~/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/IPython/utils/pickleutil.py:3: UserWarning: IPython.utils.pickleutil has moved to ipykernel.pickleutil
warn("IPython.utils.pickleutil has moved to ipykernel.pickleutil")
~/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/IPython/utils/jsonutil.py:3: UserWarning: IPython.utils.jsonutil has moved to jupyter_client.jsonutil
warn("IPython.utils.jsonutil has moved to jupyter_client.jsonutil")
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
<ipython-input-1-5652e9e33a4d> in <module>()
----> 1 from IPython import parallel
~/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/IPython/parallel/__init__.py in <module>()
31
32 from .client.asyncresult import *
---> 33 from .client.client import Client
34 from .client.remotefunction import *
35 from .client.view import *
~/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/IPython/parallel/client/client.py in <module>()
38 from IPython.utils.capture import RichOutput
39 from IPython.utils.coloransi import TermColors
---> 40 from IPython.utils.jsonutil import rekey, extract_dates, parse_date
41 from IPython.utils.localinterfaces import localhost, is_local_ip
42 from IPython.utils.path import get_ipython_dir
ImportError: cannot import name rekey
So I tried:
pip install rekey
But no distribution were found.
Note that it fails the same way in the notebook, be it open with ipython notebook or jupyter notebook, and in the console.
Also note that there is a warning:
UserWarning: IPython.utils.jsonutil has moved to jupyter_client.jsonutil
But rekey does not exist in the module jupyter_client.jsonutil
Question: How can I have IPython parallel to work within Jupyter ?
What am I missing ?

I found the problem I think (at least it works):
First, I had to import ipyparallel instead of IPython.parallel
See here: http://jupyter.readthedocs.org/en/latest/migrating.html#imports
EDIT: I get this OSError, but the fix was apparently useless, and it works without. I still don't get why I had this error, though.
Then, I had another error, when starting the client:
OSError: Connection file '~/.ipython/profile_default/security/ipcontroller-client.json' not found.
You have attempted to connect to an IPython Cluster but no Controller could be found.
Please double-check your configuration and ensure that a cluster is running.
So I just copy the directory ~/.ipython/profile_default to ~/.jupyter/profile_default
And it works!

Related

Import from the root of the repository when running a jupyter notebook

I have a repository with the following setup:
│
└───foo_lib
│ │ bar.py
│
└───notebooks
│ my_notebook.ipynb
So basically I have some common python code in foo_lib and some notebook in notebooks
In my_notebook I want to use the code from foo_lib. So I do:
from foo_lib import bar
But that doesn't work because the root of the repo isn't in my python path when the notebook is executed.
---------------------------------------------------------------------------
ModuleNotFoundError Traceback (most recent call last)
<ipython-input-1-e2c421feccf4> in <module>
----> 1 from foo_lib import bar
ModuleNotFoundError: No module named 'foo_lib'
The hack I've been using is to put %cd .. in the first cell. Then the working directory is the root of the repo and I can import fine. But it's not idempotent, so if I run the cell more than once, imports break again.
I found an idempotent solution. I can use globals()["_dh"][0] which points to the directory containing the notebook, when running in jupyter:
import os
os.chdir(os.path.join(globals()["_dh"][0], ".."))
Unfortuantely, this doesn't work when I run my notebook programatically using nbconvert:
import json
import nbconvert
import nbformat
def run_notebook():
ep = nbconvert.preprocessors.ExecutePreprocessor()
with open("notebooks/my_notebook.ipynb") as fp:
nb = nbformat.read(fp, as_version=4)
nb, resources = ep.preprocess(nb)
print(json.dumps(nb, indent=2))
if __name__ == "__main__":
run_notebook()
When I run this script from the root of the repository, globals()["_dh"][0] points to the root of the repository
So I'm looking for a solution to this import problem that:
is idempotent
works when executing from the browser/jupyter
works when executing using nbconvert
is short: I would have to copy paste the code in every notebook (since before that code runs, I can't do imports).
Is there a better way to do this?
I've figured out that the local repository code and be added to the site-package by calling:
pip install -e .

I have pytorch installed in a environment but import torch produces an error

I have anaconda python3 kernel with pytorch and numpy installed in the environment. In jupyter notebook first line 'import torch' produces error.
I am using anaconda navigator to launch jupyter notebook and enter my environment and see pytorch is installed but not being imported. Tried various dir extensions 'from torch... import * but more error
import torch
ImportError Traceback (most recent call last)
<ipython-input-1-20507c95d9af> in <module>
1
----> 2 import torch
3
4
~/anaconda3/envs/udacity1/lib/python3.6/site-packages/torch/__init__.py in <module>
100 pass
101
--> 102 from torch._C import *
103
104 __all__ += [name for name in dir(_C)
ImportError: /home/frida/anaconda3/envs/udacity1/lib/python3.6/site-packages/torch/lib/libtorch.so.1: undefined symbol: nvrtcGetProgramLogSize
I was able to add this to the end of my path to establish a 'backend ' for my notebook kernel. Thanks to Kris Stern!
-m ipykernel install --user
first get your kernel path with
(which python3)
then connect it using
sudo (your path)/anaconda3/bin/python3 -m ipykernel install --user

AttributeError: module 'pytest' has no attribute 'mark' for "from sklearn.model_selection import cross_val_score"

When i try to import "sklearn" and use it, i am getting "Attribute Error",
## Modeling
# Multiple Linear REgression (Least Square fitting)
from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LinearRegression
The error from Jupyter notebook is as below,
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-33-9fc361c79a41> in <module>()
2 # Multiple Linear REgression (Least Square fitting)
3
----> 4 from sklearn.model_selection import cross_val_score
5 from sklearn.linear_model import LinearRegression
6 #from sklearn.model_selection.train_test_split import train_test_split
D:\Anaconda_details\lib\site-packages\sklearn\model_selection\__init__.py in <module>()
17 from ._split import check_cv
18
---> 19 from ._validation import cross_val_score
20 from ._validation import cross_val_predict
21 from ._validation import cross_validate
D:\Anaconda_details\lib\site-packages\sklearn\model_selection\_validation.py in <module>()
29 from ..utils._joblib import logger
30 from ..externals.six.moves import zip
---> 31 from ..metrics.scorer import check_scoring, _check_multimetric_scoring
32 from ..exceptions import FitFailedWarning
33 from ._split import check_cv
D:\Anaconda_details\lib\site-packages\sklearn\metrics\__init__.py in <module>()
5
6
----> 7 from .ranking import auc
8 from .ranking import average_precision_score
9 from .ranking import coverage_error
D:\Anaconda_details\lib\site-packages\sklearn\metrics\ranking.py in <module>()
34 from ..utils.sparsefuncs import count_nonzero
35 from ..exceptions import UndefinedMetricWarning
---> 36 from ..preprocessing import label_binarize
37
38 from .base import _average_binary_score
D:\Anaconda_details\lib\site-packages\sklearn\preprocessing\__init__.py in <module>()
4 """
5
----> 6 from ._function_transformer import FunctionTransformer
7
8 from .data import Binarizer
D:\Anaconda_details\lib\site-packages\sklearn\preprocessing\_function_transformer.py in <module>()
3 from ..base import BaseEstimator, TransformerMixin
4 from ..utils import check_array
----> 5 from ..utils.testing import assert_allclose_dense_sparse
6 from ..externals.six import string_types
7
D:\Anaconda_details\lib\site-packages\sklearn\utils\testing.py in <module>()
758 import pytest
759
--> 760 skip_if_32bit = pytest.mark.skipif(_IS_32BIT,
761 reason='skipped on 32bit platforms')
762 skip_travis = pytest.mark.skipif(os.environ.get('TRAVIS') == 'true',
AttributeError: module 'pytest' has no attribute 'mark'
I have tried install/uninstall using pip
pip uninstall scikit-learn
pip install scikit-learn
(base) C:\Users\shashi>pip install scikit-learn
I am not able to uninstall sckit-learn package, it seems to be some path issue,
(base) C:\Users\shashi>pip uninstall scikit-learn
Uninstalling scikit-learn-0.20.0:
Would remove:
d:\anaconda_details\lib\site-packages\scikit_learn-0.20.0-py3.5.egg-info
d:\anaconda_details\lib\site-packages\sklearn
Proceed (y/n)? y
Exception:
Traceback (most recent call last):
File "D:\Anaconda_details\lib\shutil.py", line 544, in move
os.rename(src, real_dst)
OSError: [WinError 17] The system cannot move the file to a different disk drive: 'd:\\anaconda_details\\lib\\site-packages\\sklearn' -> 'C:\\Users\\sh289528\\AppData\\Local\\Temp\\pip-uninstall-w3d2aqtz\\anaconda_details\\lib\\site-packages\\sklearn'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "D:\Anaconda_details\lib\site-packages\pip\_internal\basecommand.py", line 228, in main
status = self.run(options, args)
File "D:\Anaconda_details\lib\site-packages\pip\_internal\commands\uninstall.py", line 68, in run
auto_confirm=options.yes, verbose=self.verbosity > 0,
File "D:\Anaconda_details\lib\site-packages\pip\_internal\req\req_install.py", line 661, in uninstall
uninstalled_pathset.remove(auto_confirm, verbose)
File "D:\Anaconda_details\lib\site-packages\pip\_internal\req\req_uninstall.py", line 219, in remove
renames(path, new_path)
File "D:\Anaconda_details\lib\site-packages\pip\_internal\utils\misc.py", line 273, in renames
shutil.move(old, new)
File "D:\Anaconda_details\lib\shutil.py", line 556, in move
rmtree(src)
File "D:\Anaconda_details\lib\shutil.py", line 494, in rmtree
return _rmtree_unsafe(path, onerror)
File "D:\Anaconda_details\lib\shutil.py", line 384, in _rmtree_unsafe
_rmtree_unsafe(fullname, onerror)
File "D:\Anaconda_details\lib\shutil.py", line 389, in _rmtree_unsafe
onerror(os.unlink, fullname, sys.exc_info())
File "D:\Anaconda_details\lib\shutil.py", line 387, in _rmtree_unsafe
os.unlink(fullname)
PermissionError: [WinError 5] Access is denied: 'd:\\anaconda_details\\lib\\site-packages\\sklearn\\feature_extraction\\_hashing.cp35-win_amd64.pyd'
You are using pip version 18.0, however version 19.0.3 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.
I am getting error again when i try to pip install.
(base) C:\Users\shashi>pip install scikit-learn
Collecting scikit-learn
Downloading https://files.pythonhosted.org/packages/d3/fa/b50821115c16e9b8ca307d3788e3dd1ec71cade3e564953ed7330a1fa3e0/scikit_learn-0.20.3-cp35-cp35m-win_amd64.whl (4.8MB)
100% |################################| 4.8MB 546kB/s
Requirement already satisfied: scipy>=0.13.3 in d:\anaconda_details\lib\site-packages (from scikit-learn) (1.1.0)
Requirement already satisfied: numpy>=1.8.2 in d:\anaconda_details\lib\site-packages (from scikit-learn) (1.15.2)
twisted 18.7.0 requires PyHamcrest>=1.9.0, which is not installed.
Installing collected packages: scikit-learn
Could not install packages due to an EnvironmentError: [WinError 5] Access is denied: 'D:\\Anaconda_details\\Lib\\site-packages\\sklearn\\feature_extraction\\_hashing.cp35-win_amd64.pyd'
Consider using the `--user` option or check the permissions.
You are using pip version 18.0, however version 19.0.3 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.
I have tried with conda uninstall and install. but that didn't help me either.
(base) C:\Users\shashi>conda update scikit-learn
Solving environment: done
==> WARNING: A newer version of conda exists. <==
current version: 4.5.5
latest version: 4.6.9
Please update conda by running
$ conda update -n base -c defaults conda

pyql is not accessable after its installation from native for Ubuntu14.04 python2.7

After installing pyql from the package source(according to its own wizard described in this file) on Ubuntu 14.04 into /usr/local/lib/python2.7/dist-packages/ folder, all tests finished with success. This folder includes all additional installed packages, which are accessible, from the python using native import command.
But, this specific installation python don’t see, and I cannot import anything from it.
Have you any idea, what I need to define in addition?
Thanks,
Yigal B.
There are results of my sys.path:
import sys
print(sys.path)
['/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu',
'/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages',
'/usr/lib/python2.7/dist-packages',
'/usr/lib/python2.7/dist-packages/PILcompat',
'/usr/lib/python2.7/dist-packages/gst-0.10',
'/usr/lib/python2.7/dist-packages/gtk-2.0',
'/usr/lib/pymodules/python2.7',
'/usr/lib/python2.7/dist-packages/wx-2.8-gtk2-unicode']
python -c 'from quantlib.settings import __quantlib_version__; print __quantlib_version__'
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: No module named quantlib.settings
Can you post the content of you sys.path? And the output of
python -c 'from quantlib.settings import __quantlib_version__; print __quantlib_version__'

Using Gurobi solver in the Enthought Canopy Editor on Linux

I am currently working with Gurobi solver using python. Previously, I have just been using a mix of a text editor and the terminal to write and run my models but I am currently experimenting with using Enthought Canopy Editor. I have a model that is works when I run in in the terminal using python model.py but when I try doing%run model.py in Canopy's iPython shell I get:
%run /home/cdhagmann/Copy/Code_Env/Farmer/model.py
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
/home/cdhagmann/Canopy/appdata/canopy-1.0.1.1189.rh5-x86_64/lib/python2.7/site-packages/IPython/utils/py3compat.pyc in execfile(fname, *where)
181 else:
182 filename = fname
--> 183 __builtin__.execfile(filename, *where)
/home/cdhagmann/Copy/Code_Env/Farmer/model.py in <module>()
----> 1 from gurobipy import *
2 from data_IO import read_data,currency
3 import time
4
5 # Import model data
ImportError: No module named gurobipy
My guess is that the terminal and Canopy look for modules in different places but I don't know how to rectify it. I found this reference on the Gurobi group site about Enthought Canopy and Gurobi on OS X 10.8 but didn't know how much those instruction would differ from doing it on Linux and even what he meant about installing Gurobi in Canopy's site-packages.
UPDATE: Based on information found the first link I clicked on, I did in fact leave the boxed clicked making Canopy my default directory. You have to manually prepend the PATH variable, which I missed them saying that I had to do. So I did that. I now have Canopy's version of iPython running in my terminal (check by using sys.path); however, I get a huge list of errors when I try run python.
cdhagmann#Crispin ~ $ python
Traceback (most recent call last):
File "/usr/lib/python2.7/site.py", line 563, in <module>
main()
File "/usr/lib/python2.7/site.py", line 545, in main
known_paths = addusersitepackages(known_paths)
File "/usr/lib/python2.7/site.py", line 272, in addusersitepackages
user_site = getusersitepackages()
File "/usr/lib/python2.7/site.py", line 247, in getusersitepackages
user_base = getuserbase() # this will also set USER_BASE
File "/usr/lib/python2.7/site.py", line 237, in getuserbase
USER_BASE = get_config_var('userbase')
File "/usr/lib/python2.7/sysconfig.py", line 578, in get_config_var
return get_config_vars().get(name)
File "/usr/lib/python2.7/sysconfig.py", line 505, in get_config_vars
import re
File "/usr/lib/python2.7/re.py", line 105, in <module>
import sre_compile
File "/usr/lib/python2.7/sre_compile.py", line 14, in <module>
import sre_parse
File "/usr/lib/python2.7/sre_parse.py", line 17, in <module>
from sre_constants import *
File "/usr/lib/python2.7/sre_constants.py", line 18, in <module>
from _sre import MAXREPEAT
ImportError: cannot import name MAXREPEAT
Suggestions?
And to answer #Jonathan questions:
sys.prefix = '/usr'
sys.path:
/usr/lib/python2.7
/usr/lib/python2.7/plat-x86_64-linux-gnu
/usr/lib/python2.7/lib-tk
/usr/lib/python2.7/lib-old
/usr/lib/python2.7/lib-dynload
/usr/local/lib/python2.7/dist-packages
/usr/lib/python2.7/dist-packages
/usr/lib/python2.7/dist-packages/PILcompat
/usr/lib/python2.7/dist-packages/gtk-2.0
/usr/lib/pymodules/python2.7
/usr/lib/python2.7/dist-packages/ubuntu-sso-client
gurobipy.__file__ = '/usr/local/lib/python2.7/dist-packages/gurobipy/gurobipy.so'
If the terminal is correctly configured for Canopy Python, then python in terminal and Canopy Python should be looking for modules in the same places. If during installation, you disabled "make Canopy be default Python", or if you have not restarted terminal since installation, then it won't automatically be configured correctly.
These articles should help you understand and configure:
http://docs.enthought.com/canopy/configure/faq.html#where-are-all-of-the-python-packages-in-my-user-python-environment
https://support.enthought.com/entries/23646538-Make-Canopy-s-Python-be-your-default-Python-i-e-on-the-PATH-
https://support.enthought.com/entries/23389761-Installing-packages-into-Canopy-Python-from-the-command-line
If these articles do not suffice for you to solve this, then please report -- when you run python from terminal, what are these?
sys.prefix
sys.path
gurobipy.__file__