If I 'source activate' a non-root environment (in my case "data"), then launch Jupyter notebook, the env seems to switch to root. I can tell because if I try to open a new python notebook, the dropdown under New says Python [Root]. I am also unable to import packages in my env, but not in root.
(data) Edwards-MacBook-Pro:~ mango$ conda list
# packages in environment at /Users/mango/anaconda/envs/data:
#
boto 2.42.0 py35_0
bz2file 0.98 py35_0
cycler 0.10.0 py35_0
freetype 2.5.5 1
gensim 0.12.4 np111py35_0
libpng 1.6.22 0
matplotlib 1.5.1 np111py35_0
mkl 11.3.3 0
numpy 1.11.1 py35_0
openssl 1.0.2i 0
pandas 0.18.1 np111py35_0
pip 8.1.2 py35_0
pyparsing 2.1.4 py35_0
pyqt 4.11.4 py35_4
python 3.5.2 0
python-dateutil 2.5.3 py35_0
pytz 2016.6.1 py35_0
qt 4.8.7 4
readline 6.2 2
requests 2.11.1 py35_0
scikit-learn 0.17.1 np111py35_2
scipy 0.18.1 np111py35_0
seaborn 0.7.1 py35_0
setuptools 27.2.0 py35_0
sip 4.18 py35_0
six 1.10.0 py35_0
smart_open 1.3.4 py35_0
sqlite 3.13.0 0
tk 8.5.18 0
wheel 0.29.0 py35_0
xz 5.2.2 0
zlib 1.2.8 3
(data) Edwards-MacBook-Pro:~ mango$ ipython
Python 3.5.2 |Anaconda custom (x86_64)| (default, Jul 2 2016, 17:52:12)
Type "copyright", "credits" or "license" for more information.
IPython 4.2.0 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
In [1]: import seaborn
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
<ipython-input-1-085c0287ecb5> in <module>()
----> 1 import seaborn
ImportError: No module named 'seaborn'
In [2]:
The same behavior occurs with gensim, so it is not just seaborn.
I managed to solve the issue. Conda installs the [root] environment with ipython and jupyter. If you create an env, those are not available by default. So when creating and env, either be sure to list those packages explicitly, or clone root. Cloning root may make a bulkier env and it may be less desirable for production, but better for a sandbox context.
I discovered this issue by trying the same test above with python and finding that my packages indicated that I was in the data env. I then decided to try the Anaconda Navigator program with conda install anaconda-navigator. While I like a cli, this GUI based program seems like a better way to manage packages.
Related
I am using Anaconda (latest version of september). I have executed 'conda update --all' to get the latest packages, in particular Scipy. But when I import scipy from a python shell, it's only importing the 1.7.3 version, not the latest one (1.9.1).
I am not fluent with the conda framework. I tried running conda install scipy which did not change anything, and conda install scipy=1.9.1 (which seems to hang).
I am using the base environment. This is a fresh isntall of the latest Anconda (with package update via conda, not any use of pip that could interfere).
Listing the packages via conda yields:
>>> conda list scipy
# packages in environment at /home/***/anaconda3:
#
# Name Version Build Channel
scipy 1.7.3 py39hc147768_0
However, when I look at the content of the anaconda3/pkgs folder:
>>> ls anaconda3/pkgs/ | grep scipy
anaconda3/pkgs/scipy-1.7.3-py39hc147768_0.conda
anaconda3/pkgs/scipy-1.9.1-py39h14f4228_0.conda
anaconda3/pkgs/scipy-1.7.3-py39hc147768_0 (contains Scipy 1.7.3)
anaconda3/pkgs/scipy-1.9.1-py39h14f4228_0 (contains Scipy 1.9.1)
anaconda3/pkgs/scipy-1.9.1-py39h14f4228_09jfxaf1g (empty folder)
anaconda3/pkgs/scipy-1.9.1-py39h14f4228_0dydy5wnw (empty folder)
So I am assuming that conda has both Scipy 1.7.3 and 1.9.1. But why can't I import the lastest one ?
How may I correct this situation ?
EDIT: creating a new environment and reinstalling the packages as needed solves my problem. However, how come the base environment is stuck with the earlier version ?
I am new to pytest and tried to write a first test case. Since I am new to programming overall, I developed without a virtual environment (naughty, naughty) using the globally installed python version 3.9.13.
The structure of my program is like this:
mypkg/
sub_mypkg/
file_a.py
pytest.ini
testing/
__init__.py
test_a.py
in which file_a.py imports pandas among other modules. The tests in test_a.py try among other things to run file_a.py.pytest.ini adds the root_dir to the PYTHONPATH.
In this setup pytest ran smoothly without any errors.
I then installed a virtualenv using python 3.11 in this project and installed all necessary modules (including pytest) in it and uninstalled pytest globally. After activating the virtualenv and running pytest from the terminal a ModuleNotFoundError was thrown for pandas.
Here is a list of the modules I use in venv:
Package Version
--------------- -------
attrs 22.1.0
colorama 0.4.6
contourpy 1.0.6
cycler 0.11.0
et-xmlfile 1.1.0
exceptiongroup 1.0.4
fonttools 4.38.0
iniconfig 1.1.1
kiwisolver 1.4.4
matplotlib 3.6.2
numpy 1.23.4
openpyxl 3.0.10
packaging 21.3
pandas 1.5.1
Pillow 9.3.0
pip 22.3.1
pluggy 1.0.0
pyparsing 3.0.9
pytest 7.2.0
python-dateutil 2.8.2
pytz 2022.6
PyYAML 6.0
setuptools 56.0.0
six 1.16.0
tomli 2.0.1
I checked that pandas was installed and I could import pandas in the REPL.
Also I could run the test line by line in the REPL.
Furthermore, I checked that the problem was not pandas itself. If I changed the placement of the import pandas with for example import numpy in file_a.py, it threw the ModuleNotFoundError for numpy.
I tried to use different versions of python in my venv (python 3.11.0, 3.10.7, 3.9.13, 3.9.6). Interestingly, pytest ran only inside the venv for python 3.9.13 (The one I developed it in).
I tried to include and delete __init__.py files in all directories and also tried different versions of pytest (6.2.5, 7.0.0).
I checked sys.path that the right root_dir was included.
Thanks in advance for your answers!
The Jupyter docs recommend using conda to install jupyter, if the runtime python is version 3.5 or lower. Installation is pretty straightforward with conda.
Follow Anaconda’s instructions for downloading and installing the Python 3.5 version.
$ conda install jupyter
$ pip list | grep ipython
ipython 7.29.0
ipython-genutils 0.2.0
$ ipython notebook --ip=0.0.0.0
This question is to discover if it's possible to install a 3.5 compatible jupyter using only pip, and without conda.
Given the following environment:
$ python --version
Python 3.5.2
$ sudo apt-get install python3-pip
$ pip3 --version
pip 8.1.1 from /usr/lib/python3/dist-packages (python 3.5)
Installing ipython is possible with the following restrictions:
$ python3 -m pip install ipython==7.9 traitlets==4.3.3 jedi==0.17.2
Jupyter, however, doesn't install gracefully.
The following allows for installation:
pip3 install jupyter ipython==7.9 traitlets==4.3.3 jedi==0.17.2 pyzmq==20 argon2-cffi==21.1.0
But at runtime, jupyter crashes:
File "/home/vagrant/.local/lib/python3.5/site-packages/ipython_genutils/importstring.py", line 31, in import_item
module = __import__(package, fromlist=[obj])
File "/home/vagrant/.local/lib/python3.5/site-packages/notebook/__init__.py", line 27, in <module>
from .nbextensions import install_nbextension
File "/home/vagrant/.local/lib/python3.5/site-packages/notebook/nbextensions.py", line 15, in <module>
from jupyter_core.paths import (
File "/home/vagrant/.local/lib/python3.5/site-packages/jupyter_core/__init__.py", line 1, in <module>
from .version import version_info, __version__
File "/home/vagrant/.local/lib/python3.5/site-packages/jupyter_core/version.py", line 21
_suffix_ = f'.dev{version_info.serial}'
Is it possible to install a 3.5 friendly jupyter using pip?
Resources
pyzmq https://github.com/jupyter/notebook/issues/5961
iPython Versions
https://github.com/ipython/ipython/blob/master/README.rst
IPython 7.10+ supports Python 3.6 and above.
When using Python 2.7, please install IPython 5.x LTS Long Term Support version.
Python 3.3 and 3.4 were supported up to IPython 6.x.
Python 3.5 was supported with IPython 7.0 to 7.9.
traitlets vs. python 3.5
Traitlets 5 uses f-strings in code, not compatible with Python 3.5
uninstall the traitlets 5.0.3 version, and install 4.3.3 version
https://githubmemory.com/repo/ipython/traitlets/issues/620
argon2-cffi vs. python 3.5
https://packagegalaxy.com/python/argon2-cffi
It targets Python 3.6 and newer, and PyPy3.
The last version that works with Python 2.7 is 20.1.0, and the last version that works with Python 3.5 is 21.1.0.
jedi vs. python 3.5
https://jedi.readthedocs.io/en/latest/docs/changelog.html
0.17.2 (2020-07-17)
This will be the last release that supports Python 2 and Python 3.5. 0.18.0 will be Python 3.6+.
I am using an EMR notebook attached to my cluster for some experimentation purposes. I needed to install some python modules for testing, specifically spacy and it's data module en_core_web_sm.
I ssh'ed into the master and core nodes and downloaded the modules individually. However I am not able to import from the my EMR notebook. I get the following error :
An error was encountered:
No module named 'spacy'
Traceback (most recent call last):
ModuleNotFoundError: No module named 'spacy'
I know there is a way to install them just for the scope of EMR notebook, but this wouldn't suffice in a production scenario, so please avoid answers which suggest notebook installing as mentioned in this guide : https://aws.amazon.com/blogs/big-data/install-python-libraries-on-a-running-cluster-with-emr-notebooks/
Please let me know if I am missing some setup steps. Appreciate your response.
You can use bootstraps to install additional modules while creating your EMR
https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-bootstrap.html
I was able to solve this by changing the bootstrap script to use sudo instead of --user. (You could also manually change run the scripts below)
Before I was running
pip3 install spacy --user
python3 -m spacy download en --user
I changed that script to
sudo pip3 install spacy
sudo python3 -m spacy download en
To verify this solution quickly issue the following commands from your EMR notebook (to compare before and after)
sc.list_packages()
You should see an output similar to this
SparkSession available as 'spark'.
Package Version
-------------------------- ----------
beautifulsoup4 4.9.0
blis 0.4.1
boto 2.49.0
catalogue 1.0.0
certifi 2020.4.5.2
chardet 3.0.4
cymem 2.0.3
en-core-web-sm 2.3.0
idna 2.9
importlib-metadata 1.6.1
jmespath 0.9.5
lxml 4.5.0
murmurhash 1.0.2
mysqlclient 1.4.2
nltk 3.4.5
nose 1.3.4
numpy 1.16.5
pip 9.0.1
plac 1.1.3
preshed 3.0.2
py-dateutil 2.2
python37-sagemaker-pyspark 1.3.0
pytz 2019.3
PyYAML 5.3.1
requests 2.24.0
setuptools 28.8.0
six 1.13.0
soupsieve 1.9.5
spacy 2.3.0
srsly 1.0.2
thinc 7.4.1
tqdm 4.46.1
urllib3 1.25.9
wasabi 0.6.0
wheel 0.29.0
windmill 1.6
zipp 3.1.0
This is not the best possible solution IMO, since the first warning that gets displayed after using sudo is
WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.
If anyone has a better solution please free to post.
Granted I've been off PYTHON for two + years. I'm trying to get back into the swing, and I remember having command and variable completion available on windows a few years back. We were stuck in 2.6 as we had dependencies.
Today, I'm trying to import readline. I get this message from pip.
C:\Users\Joe>pip3 install readline
Collecting readline
Using cached https://files.pythonhosted.org/packages/f4/01/2cf081af8d880b44939a5f1b446551a7f8d59eae414277fd0c303757ff1b/readline-6.2.4.1.tar.gz
Complete output from command python setup.py egg_info:error: this module is not meant to work on Windows
my version is:
Python 3.7.0 (v3.7.0:1bf9cc5093, Jun 27 2018, 04:59:51) [MSC v.1914 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
import readline
Traceback (most recent call last):
File "", line 1, in
ModuleNotFoundError: No module named 'readline'
I have recently faced the same issue on Windows 10. readline is not available for Windows, however there is pyreadline3 (a continuation of the abandoned pyreadline package) which is for windows and you may try it.
Try to install pyreadline3 and use it instead of readline:
pip: pip install pyreadline3 or python -m pip install pyreadline
mamba: mamba install -c conda-forge pyreadline3
conda: conda install -c conda-forge pyreadline3
Then, in your python file do this:
from pyreadline3 import Readline
readline = Readline()
That way, you can use readline in Windows like in linux systems.
Go to your CMD terminal and type
pip install pyreadline
with in few seconds it will be installed and you will be able to see message i.e. "Successfully installed pyreadline-"
then go back to your PYSPARK terminal and type again import readline
it will import now without any error
Thanks
I tried pyreadline after finding this question and had problems and eventually found that it is no longer supported.
What worked for me was installing pyreadline3 with:
pip install pyreadline3