Code written in zeppelin, its working and importing fine in shell when I start pyspark there but not in zeppelin with same code.
%pyspark
import pandas
Traceback (most recent call last):
File "/tmp/zeppelin_pyspark-4245945050627073162.py", line 239, in <module>
eval(compiledCode)
File "<string>", line 1, in <module>
ImportError: No module named pandas
This is because pandas is not installed on the machine.
If pip is not installed, first install pip.
sudo curl --silent --show-error https://bootstrap.pypa.io/get-pip.py | python
Then install pandas
sudo pip install pandas
Or use [docker] like this.
docker run -d -p 8080:8080 -t knockdata/zeppelin-highcharts
zeppelin-highcharts image include pandas and Highcharts functionality.
Related
installing the following using pip in collab notebook :
pip install torch==1.10.1+cu102 \
torchvision==0.11.2+cu102 \
torchaudio==0.10.1 \
-f https://download.pytorch.org/whl/torch_stable.html \
detectron2==0.6 \
-f https://dl.fbaipublicfiles.com/detectron2/wheels/cu102/torch1.10/index.html \
git+https://github.com/facebookresearch/detectron2#main#subdirectory=projects/DensePose
after, restarting runtime I run:
!python ./apply_net.py dump densepose_rcnn_R_101_FPN_DL_s1x.yaml
https://dl.fbaipublicfiles.com/densepose/densepose_rcnn_R_101_FPN_DL_s1x/165712116/model_final_844d15.pkl
./Tejrab/200404.png --output ./dump/200404.pkl -v
Expected behavior:
pkl file of the png image
Error
Traceback (most recent call last):
File "./apply_net.py", line 19, in
from densepose import add_densepose_config
ModuleNotFoundError: No module named 'densepose'
after installing densepose this error occured
2. ERROR: Command errored out with exit status 1: /usr/bin/python3 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-g43tc9ql/detectron2_0a570229969d47198f45a267ccc78493/setup.py'"'"'; file='"'"'/tmp/pip-install-g43tc9ql/detectron2_0a570229969d47198f45a267ccc78493/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-px39tp0q/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python3.7/detectron2 Check the logs for full command output.
PS
a new commit to apply_net.py has been added by authors c54429b could it be the reason ?
!pip install pyyaml==5.1
!pip install ninja
!pip install av
The three commands are :
Start the Virtual Environment by this command - source Detect_Outliers_In_Dataset/bin/activate
Change the directory by this command - cd Outlier_Detection
Start running Django Server by this command - python3 manage.py runserver
import subprocess
import pdb
p1 = subprocess.run("source Detect_Outliers_In_Dataset/bin/activate | cd Outlier_Detection | python3 manage.py runserver", capture_output=True, text=True, shell=True)
This is the error :
ompletedProcess(args='source Detect_Outliers_In_Dataset/bin/activate | cd Outlier_Detection | python3 manage.py runserver', returncode=2, stdout='', stderr="/bin/sh: 1: source: not found\n/bin/sh: 1: cd: can't cd to Outlier_Detection\npython3: can't open file 'manage.py': [Errno 2] No such file or directory\n")
Collecting psycopg2 Using cached psycopg2-2.8.5.tar.gz (380 kB)
ERROR: Command errored out with exit status 1: command: /home/ubuntu/egrdb/env/bin/python -c 'import sys, setuptools, tokenize; sys.argv0 = '"'"'/tmp/pip-install-gn70jweq/psycopg2/setup.py'"'"'; file='"'"'/tmp/pip-install-gn70jweq/psycopg2/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-4tr67ll8 cwd: /tmp/pip-install-gn70jweq/psycopg2/ Complete output (7 lines): running egg_info creating /tmp/pip-pip-egg-info-4tr67ll8/psycopg2.egg-info writing /tmp/pip-pip-egg-info-4tr67ll8/psycopg2.egg-info/PKG-INFO writing dependency_links to /tmp/pip-pip-egg-info-4tr67ll8/psycopg2.egg-info/dependency_links.txt writing top-level names to /tmp/pip-pip-egg-info-4tr67ll8/psycopg2.egg-info/top_level.txt writing manifest file '/tmp/pip-pip-egg-info-4tr67ll8/psycopg2.egg-info/SOURCES.txt' Error: b'You need to install postgresql-server-dev-X.Y for building a server-side extension or libpq-dev for building a client-side application.\n'
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
If you want to install the source version psycopg2-2.8.5.tar.gz then you will need to do as #Chris says. The simpler way though is to do:
pip install psycopg2-binary
Then you get a pre-compiled version and you don't need the -dev packages.
On a ckan instance running ok, I installed the harvester extension following this guide: https://github.com/ckan/ckanext-harvest
these are the steps I followed:
. /usr/lib/ckan/default/bin/activate
cd /usr/lib/ckan/default/src/ckan
sudo pip install -e git+https://github.com/okfn/ckanext-harvest.git#stable#egg=ckanext-harvest
cd /usr/lib/ckan/default/src/ckan/src/ckanext-harvest
sudo pip install -r pip-requirements.txt
This is the content of pip-requirements.txt:
pika==0.9.8
redis==2.10.1
I continue configuring the plugin, everything seems to work ok. I have it running at http://localhost/harvest. Then I create a new source, and when I want to start the gather command I get this error:
$ . /usr/lib/ckan/default/bin/activate
$ cd /usr/lib/ckan/default/src/ckan/src/ckanext-harvest
$ paster --plugin=ckanext-harvest harvester gather_consumer --config=/etc/ckan/default/production.ini
Traceback (most recent call last):
File "/usr/lib/ckan/default/bin/paster", line 9, in <module>
load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')()
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 104, in run
invoke(command, command_name, options, args[1:])
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 143, in invoke
exit_code = runner.run(args)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 238, in run
result = self.command()
File "/usr/lib/ckan/default/src/ckan/src/ckanext-harvest/ckanext/harvest/commands/harvester.py", line 125, in command
from ckanext.harvest.queue import get_gather_consumer, gather_callback
File "/usr/lib/ckan/default/src/ckan/src/ckanext-harvest/ckanext/harvest/queue.py", line 5, in <module>
import pika
ImportError: No module named pika
I'm pretty sure there must be something really silly with the virtualenv (python newbie here)
This is because you used sudo pip. Because of how python's virtualenv works, if you want to use sudo to install into the virtualenv, you need to give full path to pip. Something like this would work
sudo /usr/lib/ckan/default/bin/pip -e git+https://github.com/okfn/ckanext-harvest.git#stable#egg=ckanext-harvest
cd /usr/lib/ckan/default/src/ckanext-harvest
sudo /usr/lib/ckan/default/bin/pip install -r pip-requirements.txt
Something strange and unexpected is happening with the sys.path of any virtual environment I set. For example, a clean env:
$ virtualenv test
$ source test/bin/activate
(test) $
This is the expected PYTHONPATH:
(test) $ python
>>> import sys
>>> print '\n'.join(sys.path)
/home/user/test/local/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg
/home/user/test/local/lib/python2.7/site-packages/pip-1.1-py2.7.egg
/home/user/test/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg
/home/user/test/lib/python2.7/site-packages/pip-1.1-py2.7.egg
/home/user/test/lib/python2.7
/home/user/test/lib/python2.7/plat-linux2
/home/user/test/lib/python2.7/lib-tk
/home/user/test/lib/python2.7/lib-old
/home/user/test/lib/python2.7/lib-dynload
/usr/lib/python2.7
/usr/lib/python2.7/plat-linux2
/usr/lib/python2.7/lib-tk
/home/user/test/local/lib/python2.7/site-packages
/home/user/test/lib/python2.7/site-packages
But this is the one I really get:
(test) $ bpython
>>> import sys
>>> print '\n'.join(sys.path)
/usr/bin
/usr/lib/python2.7
/usr/lib/python2.7/plat-linux2
/usr/lib/python2.7/lib-tk
/usr/lib/python2.7/lib-old
/usr/lib/python2.7/lib-dynload
/usr/local/lib/python2.7/dist-packages
/usr/lib/python2.7/dist-packages
I can't figure out the reason of the two different sys.paths.
Because of that, no pip installation works!
I'm using Virtualenv 1.7.2, Ubuntu 12.04, Python 2.7.3.
Any help will be appreciated.
Rather than installing one copy of bpython per virtualenv, I've added this function to my shell profile (for example ~/.bashrc or ~/.zshrc). It wraps the bpython command with some logic to load the virtual environment's python path (if you have an active virtual environment).
bpython() {
if test -n "$VIRTUAL_ENV"
then
PYTHONPATH="$(python -c 'import sys; print ":".join(sys.path)')" \
command bpython "$#"
else
command bpython "$#"
fi
}
I found that I needed to deactivate and reactivate my virtualenv after installing bpython for it to work.
pip install bpython
deactivate
. bin/activate # or your equivalent activation command
My hypothesis is that you have not installed bpython after you have activated the new virtualenv.
I followed it up exactly like you mentioned:
mkvirtualenv bpython
(bpython)~ $ pip install bpython
(bpython)~ $bpython
and then ran the commands:
>>> import sys
>>> print '\n'.join(sys.path)
/Users/xxxx/.virtualenvs/bpython/bin
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/site-packages/pip-1.1-py2.7.egg
/Users/xxxx/.virtualenvs/bpython/lib/python27.zip
/Users/xxxx/.virtualenvs/bpython/lib/python2.7
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/plat-darwin
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/plat-mac
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/plat-mac/lib-scriptpackages
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/lib-tk
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/lib-old
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/lib-dynload
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/site-packages
and did the same thing again by running python under the activated virtualenv
(bpython)~ $ python
.....
>>> import sys
>>> print '\n'.join(sys.path)
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/site-packages/pip-1.1-py2.7.egg
/Users/xxxx/.virtualenvs/bpython/lib/python27.zip
/Users/xxxx/.virtualenvs/bpython/lib/python2.7
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/plat-darwin
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/plat-mac
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/plat-mac/lib-scriptpackages
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/lib-tk
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/lib-old
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/lib-dynload
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/site-packages
I saw no difference in the two results
I also discovered that if you have bpython installed locally, you need to create your virtualenv with --no-site-packages for it to work properly. If you created your virtualenv without that flag, you can create an empty file named no-global-site-packages.txt in ~/.virtualenvs/<env-name>/lib/python2.7/ as noted in this Stack Exchange answer.