ckan harvester: "No module named pika" error - plugins

On a ckan instance running ok, I installed the harvester extension following this guide: https://github.com/ckan/ckanext-harvest
these are the steps I followed:
. /usr/lib/ckan/default/bin/activate
cd /usr/lib/ckan/default/src/ckan
sudo pip install -e git+https://github.com/okfn/ckanext-harvest.git#stable#egg=ckanext-harvest
cd /usr/lib/ckan/default/src/ckan/src/ckanext-harvest
sudo pip install -r pip-requirements.txt
This is the content of pip-requirements.txt:
pika==0.9.8
redis==2.10.1
I continue configuring the plugin, everything seems to work ok. I have it running at http://localhost/harvest. Then I create a new source, and when I want to start the gather command I get this error:
$ . /usr/lib/ckan/default/bin/activate
$ cd /usr/lib/ckan/default/src/ckan/src/ckanext-harvest
$ paster --plugin=ckanext-harvest harvester gather_consumer --config=/etc/ckan/default/production.ini
Traceback (most recent call last):
File "/usr/lib/ckan/default/bin/paster", line 9, in <module>
load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')()
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 104, in run
invoke(command, command_name, options, args[1:])
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 143, in invoke
exit_code = runner.run(args)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 238, in run
result = self.command()
File "/usr/lib/ckan/default/src/ckan/src/ckanext-harvest/ckanext/harvest/commands/harvester.py", line 125, in command
from ckanext.harvest.queue import get_gather_consumer, gather_callback
File "/usr/lib/ckan/default/src/ckan/src/ckanext-harvest/ckanext/harvest/queue.py", line 5, in <module>
import pika
ImportError: No module named pika
I'm pretty sure there must be something really silly with the virtualenv (python newbie here)

This is because you used sudo pip. Because of how python's virtualenv works, if you want to use sudo to install into the virtualenv, you need to give full path to pip. Something like this would work
sudo /usr/lib/ckan/default/bin/pip -e git+https://github.com/okfn/ckanext-harvest.git#stable#egg=ckanext-harvest
cd /usr/lib/ckan/default/src/ckanext-harvest
sudo /usr/lib/ckan/default/bin/pip install -r pip-requirements.txt

Related

Trouble Installing Python3.6 Virtual Environment on Ubuntu 22.04

Virtual environment stopped being created after upgrading to Ubuntu 22.04. Running the command
virtualenv venv -p python3.6
produces an error that did not occur in the previous version - Ubuntu 20.04.:
fail
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/virtualenv/seed/embed/via_app_data/via_app_data.py", line 84, in _get
result = get_wheel(
File "/usr/lib/python3/dist-packages/virtualenv/seed/wheels/acquire.py", line 26, in get_wheel
wheel = from_bundle(distribution, version, for_py_version, search_dirs, app_data, do_periodic_update, env)
File "/usr/lib/python3/dist-packages/virtualenv/seed/wheels/bundle.py", line 13, in from_bundle
wheel = load_embed_wheel(app_data, distribution, for_py_version, of_version)
File "/usr/lib/python3/dist-packages/virtualenv/seed/wheels/bundle.py", line 33, in load_embed_wheel
wheel = get_embed_wheel(distribution, for_py_version)
File "/usr/lib/python3/dist-packages/virtualenv/seed/wheels/embed/__init__.py", line 77, in get_embed_wheel
raise Exception((
Exception: Wheel for pip for Python 3.6 is unavailable. apt install python3-pip-whl
pip-21.3.1-py3-none-any.whl already present in /home/su/.local/share/virtualenv/wheel/3.6/embed/3/pip.json
created virtual environment CPython3.6.15.final.0-64 in 3225ms
creator CPython3Posix(dest=/home/su/Downloads/courses-app/venv, clear=False, no_vcs_ignore=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/su/.local/share/virtualenv)
added seed packages: pip==21.3.1, setuptools==59.6.0, wheel==0.37.1
activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
Can anyone help solve this problem?
The issue was resolved after installing the following packages:
sudo apt install software-properties-common
sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt install python3.6
sudo apt install libpython3.6-dev
sudo apt install python3.6-distutils

zeppelin unable to import pandas, numpy, scipy

Code written in zeppelin, its working and importing fine in shell when I start pyspark there but not in zeppelin with same code.
%pyspark
import pandas
Traceback (most recent call last):
File "/tmp/zeppelin_pyspark-4245945050627073162.py", line 239, in <module>
eval(compiledCode)
File "<string>", line 1, in <module>
ImportError: No module named pandas
This is because pandas is not installed on the machine.
If pip is not installed, first install pip.
sudo curl --silent --show-error https://bootstrap.pypa.io/get-pip.py | python
Then install pandas
sudo pip install pandas
Or use [docker] like this.
docker run -d -p 8080:8080 -t knockdata/zeppelin-highcharts
zeppelin-highcharts image include pandas and Highcharts functionality.

Google Cloud SDK gcloud error

I use Google Cloud SDK under Window 7 64bit.
Google Cloud SDK and python install success. and run gcloud.
The error occurs as shown below.
C:\Program Files\Google\Cloud SDK>gcloud
Traceback (most recent call last):
File "C:\Program Files\Google\Cloud SDK\google-cloud-sdk\bin\..\./lib\googlecloudsdk\gcloud\gcloud.py", line 137, in <
module>
_cli = CreateCLI()
File "C:\Program Files\Google\Cloud SDK\google-cloud-sdk\bin\..\./lib\googlecloudsdk\gcloud\gcloud.py", line 98, in Cr
eateCLI
sdk_root = config.Paths().sdk_root
AttributeError: 'Paths' object has no attribute 'sdk_root'
Can ask for help? Thanks
I had the same problem. And this helped me solve this problem
Manually removed directory: C:\Program Files\Google\Cloud SDK
Then rerun: GoogleCloudSDKInstaller.exe
And make sure that you have connection to needed DL servers (I was first behind company firewall and installer didn't download all files - and no complains either by installer...)
Then I was OK again..
Source: https://code.google.com/p/google-cloud-sdk/issues/detail?id=62&thanks=62&ts=1407851956
echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key --keyring /usr/share/keyrings/cloud.google.gpg add -
sudo apt update
sudo apt-get install google-cloud-sdk
gcloud init

installing mongodb with pecl

this is what i get:
bitnami#ip-10-117-57-211:/bin$ sudo pecl install mongo
downloading mongo-1.4.0.tgz ...
Starting to download mongo-1.4.0.tgz (138,495 bytes)
..............................done: 138,495 bytes
84 source files, building
WARNING: php_bin /opt/bitnami/php/bin/php appears to have a suffix /bin/php, but config variable php_suffix does not match
running: phpize
Configuring for:
PHP Api Version: 20100412
Zend Module Api No: 20100525
Zend Extension Api No: 220100525
Can't load '/opt/bitnami/perl/lib/5.8.8/i686-linux/auto/Data/Dumper/Dumper.so' for module Data::Dumper: /opt/bitnami/perl/lib/5.8.8/i686-linux/auto/Data/Dumper/Dumper.so: undefined symbol: PL_sv_undef at /opt/bitnami/perl/lib/5.8.8/i686-linux/XSLoader.pm line 70.
at /opt/bitnami/perl/lib/5.8.8/i686-linux/Data/Dumper.pm line 36
Compilation failed in require at /usr/share/autoconf/Autom4te/C4che.pm line 33.
BEGIN failed--compilation aborted at /usr/share/autoconf/Autom4te/C4che.pm line 33.
Compilation failed in require at /usr/bin/autom4te line 38.
BEGIN failed--compilation aborted at /usr/bin/autom4te line 38.
ERROR: `phpize' failed
Any ideas how to fix?
Try configuring manually as it seems bitnami is a bit funky:
$ sudo apt-get update
$ sudo apt-get install build-essential libtool autoconf unzip wget
Download the latest source code from the web page, uncompress it and compile the module.
$ wget http://pecl.php.net/get/mongo-1.4.3.tgz
$ tar -zxf mongo*
$ export PHP_AUTOCONF=/usr/bin/autoconf
$ export PHP_PREFIX=/opt/bitnami/php
$ cd mongo-*
$ /opt/bitnami/php/bin/phpize
$ ./configure
$ make
$ sudo make install
Adapted from: http://wiki.bitnami.com/Components/PHP#How_to_install_OAuth_module.3f
well i finally got it working. I ran through the steps to install mongo extension on mac server and it worked for my bitnami lamp stack.
The instructions were on this blog:
http://andres.jaimes.net/857/setup-php-mongo-on-mac/

python and bpython using different PYTHONPATHs in Virtualenv

Something strange and unexpected is happening with the sys.path of any virtual environment I set. For example, a clean env:
$ virtualenv test
$ source test/bin/activate
(test) $
This is the expected PYTHONPATH:
(test) $ python
>>> import sys
>>> print '\n'.join(sys.path)
/home/user/test/local/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg
/home/user/test/local/lib/python2.7/site-packages/pip-1.1-py2.7.egg
/home/user/test/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg
/home/user/test/lib/python2.7/site-packages/pip-1.1-py2.7.egg
/home/user/test/lib/python2.7
/home/user/test/lib/python2.7/plat-linux2
/home/user/test/lib/python2.7/lib-tk
/home/user/test/lib/python2.7/lib-old
/home/user/test/lib/python2.7/lib-dynload
/usr/lib/python2.7
/usr/lib/python2.7/plat-linux2
/usr/lib/python2.7/lib-tk
/home/user/test/local/lib/python2.7/site-packages
/home/user/test/lib/python2.7/site-packages
But this is the one I really get:
(test) $ bpython
>>> import sys
>>> print '\n'.join(sys.path)
/usr/bin
/usr/lib/python2.7
/usr/lib/python2.7/plat-linux2
/usr/lib/python2.7/lib-tk
/usr/lib/python2.7/lib-old
/usr/lib/python2.7/lib-dynload
/usr/local/lib/python2.7/dist-packages
/usr/lib/python2.7/dist-packages
I can't figure out the reason of the two different sys.paths.
Because of that, no pip installation works!
I'm using Virtualenv 1.7.2, Ubuntu 12.04, Python 2.7.3.
Any help will be appreciated.
Rather than installing one copy of bpython per virtualenv, I've added this function to my shell profile (for example ~/.bashrc or ~/.zshrc). It wraps the bpython command with some logic to load the virtual environment's python path (if you have an active virtual environment).
bpython() {
if test -n "$VIRTUAL_ENV"
then
PYTHONPATH="$(python -c 'import sys; print ":".join(sys.path)')" \
command bpython "$#"
else
command bpython "$#"
fi
}
I found that I needed to deactivate and reactivate my virtualenv after installing bpython for it to work.
pip install bpython
deactivate
. bin/activate # or your equivalent activation command
My hypothesis is that you have not installed bpython after you have activated the new virtualenv.
I followed it up exactly like you mentioned:
mkvirtualenv bpython
(bpython)~ $ pip install bpython
(bpython)~ $bpython
and then ran the commands:
>>> import sys
>>> print '\n'.join(sys.path)
/Users/xxxx/.virtualenvs/bpython/bin
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/site-packages/pip-1.1-py2.7.egg
/Users/xxxx/.virtualenvs/bpython/lib/python27.zip
/Users/xxxx/.virtualenvs/bpython/lib/python2.7
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/plat-darwin
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/plat-mac
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/plat-mac/lib-scriptpackages
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/lib-tk
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/lib-old
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/lib-dynload
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/site-packages
and did the same thing again by running python under the activated virtualenv
(bpython)~ $ python
.....
>>> import sys
>>> print '\n'.join(sys.path)
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/site-packages/pip-1.1-py2.7.egg
/Users/xxxx/.virtualenvs/bpython/lib/python27.zip
/Users/xxxx/.virtualenvs/bpython/lib/python2.7
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/plat-darwin
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/plat-mac
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/plat-mac/lib-scriptpackages
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/lib-tk
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/lib-old
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/lib-dynload
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages
/Users/xxxx/.virtualenvs/bpython/lib/python2.7/site-packages
I saw no difference in the two results
I also discovered that if you have bpython installed locally, you need to create your virtualenv with --no-site-packages for it to work properly. If you created your virtualenv without that flag, you can create an empty file named no-global-site-packages.txt in ~/.virtualenvs/<env-name>/lib/python2.7/ as noted in this Stack Exchange answer.