I'm deploying Django on a production server together with virtualenv, and am having trouble activating virtualenv on the server with
source .../bin/activate
I did a little research, and found that the pythonpath is changed depending on if we are or aren't in a virtualenv.
sys.path (with virtualenv activated)
['',
'/.../virtualenv/test_path/bin',
'/.../virtualenv/test_path/local/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg',
'/.../virtualenv/test_path/local/lib/python2.7/site-packages/pip-1.2.1-py2.7.egg',
'/.../virtualenv/test_path/lib/python2.7',
'/.../virtualenv/test_path/lib/python2.7/plat-linux2',
'/.../virtualenv/test_path/lib/python2.7/lib-tk',
'/.../virtualenv/test_path/lib/python2.7/lib-old',
'/.../virtualenv/test_path/lib/python2.7/lib-dynload',
'/usr/lib/python2.7',
'/usr/lib/python2.7/plat-linux2',
'/usr/lib/python2.7/lib-tk',
'/.../virtualenv/test_path/local/lib/python2.7/site-packages',
'/.../virtualenv/test_path/local/lib/python2.7/site-packages/IPython/extensions']
sys.path (without activating virtualenv):
['',
'/usr/local/bin',
'/usr/lib/python2.7',
'/usr/lib/python2.7/plat-linux2',
'/usr/lib/python2.7/lib-tk',
'/usr/lib/python2.7/lib-old',
'/usr/lib/python2.7/lib-dynload',
'/usr/local/lib/python2.7/dist-packages',
'/usr/lib/python2.7/dist-packages',
'/usr/lib/python2.7/dist-packages/PIL',
'/usr/lib/python2.7/dist-packages/gst-0.10',
'/usr/lib/python2.7/dist-packages/gtk-2.0',
'/usr/lib/pymodules/python2.7',
'/usr/local/lib/python2.7/dist-packages/IPython/extensions']
Is is sufficient to just change the pythonpath to point to the virtualenv
.../python2.7/site-packages
folder to get the same results as running
source .../bin/activate
?
No, it is not. virtualenv is not only about site-packages, it is about a whole isolated python environment.
Doing source /path/to/venv/bin/activate just changes your $PATH environment variable to include your virtualenv bin directory as first lookup.
If you call python directly, it is just a shortcut for:
$ /path/to/venv/bin/python myscript.py
And if you call pip in an activated virtualenv, it is the same as:
$ /path/to/venv/bin/pip install XYZ
Related
I am trying to deploy a Django REST API using Azure App Service on Linux. I am using a postgresql Database and deploy via pipeline. Azure has postgresql 9.6. After running my pipeline, the Website shows an Server Error (500).
The AppLogs show, that the Container couldn't be started due an failed import of psycopg2.
[ERROR] Exception in worker process
Traceback (most recent call last):
File "/home/site/wwwroot/antenv/lib/python3.7/site-packages/django/db/backends/postgresql/base.py", line 25, in
import psycopg2 as Database
File "/home/site/wwwroot/antenv/lib/python3.7/site-packages/psycopg2/__init__.py", line 50, in
from psycopg2._psycopg import ( # noqa
ImportError: /home/site/wwwroot/antenv/lib/python3.7/site-packages/psycopg2/_psycopg.cpython-37m-x86_64-linux-gnu.so: undefined symbol: PQencryptPasswordConn
In the Build-stage of my pipeline, I set up my environment (python3.7) like this:
- script: |
python -m venv antenv
source antenv/bin/activate
python -m pip install --upgrade pip
pip install setup
pip install -r requirements.txt
Where requirements.txt looks like this:
Django==3.0.2
djangorestframework==3.11.0
psycopg2-binary==2.8.4
pandas==0.25.3
pytest==5.3.5
pytest-django==3.8.0
pytest-mock==2.0.0
python-dateutil==2.8.1
sqlparse==0.3.0
whitenoise==5.0.1
BuildJob and DeploymentJob seem to run flawless. the Build-logs indicate that psycopg2_binary-2.8.4-cp37-cp37m-manylinux1_x86_64.whl was correctly downloaded and installed.
Also the App runs fine on my machine when using the database on azure by configuring in the settings.py:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': 'databasename',
'USER': 'user#postgresqlserver',
'PASSWORD': 'Password',
'HOST': 'postgresqlserver.postgres.database.azure.com',
'PORT': '',
'OPTIONS': {'sslmode': 'require'}
}
} # Of course the info is actually saved in environment variables
This gives me the feeling, that something with the psycopg2 installation is not working... For others the *psycopg2-binary seemed to do the trick but unfortunateley not for me.
Am I right to assume that on azure I'm nether able to install postgresql10 as suggested here https://github.com/psycopg/psycopg2/issues/983 nor can install from source like suggested here https://github.com/psycopg/psycopg2/issues/1018?
There must be something I am missing, I would be grateful for any advice!
EDIT:
Taking a look at the library (as suggested here https://stackoverflow.com/a/59652816/13183775) I found that I don't have a PQencryptPasswordConn function but only a PQencryptPassword function. I have the feeling that this is expected for Postgresql9.6 (https://github.com/psycopg/psycopg2/blob/cb3353be1f10590cdc2a894ada42c3b4c171feb7/psycopg/psycopgmodule.c#L466).
To check, whether there are multiple versions libpq:
/>find . -name "libpq*"
./var/lib/dpkg/info/libpq5:amd64.symbols
./var/lib/dpkg/info/libpq5:amd64.shlibs
./var/lib/dpkg/info/libpq5:amd64.list
./var/lib/dpkg/info/libpq5:amd64.triggers
./var/lib/dpkg/info/libpq-dev.list
./var/lib/dpkg/info/libpq5:amd64.md5sums
./var/lib/dpkg/info/libpq-dev.md5sums
./usr/share/doc/libpq5
./usr/share/doc/libpq-dev
./usr/share/locale/ko/LC_MESSAGES/libpq5-9.6.mo
./usr/share/locale/it/LC_MESSAGES/libpq5-9.6.mo
./usr/share/locale/pl/LC_MESSAGES/libpq5-9.6.mo
./usr/share/locale/zh_TW/LC_MESSAGES/libpq5-9.6.mo
./usr/share/locale/tr/LC_MESSAGES/libpq5-9.6.mo
./usr/share/locale/cs/LC_MESSAGES/libpq5-9.6.mo
./usr/share/locale/de/LC_MESSAGES/libpq5-9.6.mo
./usr/share/locale/ru/LC_MESSAGES/libpq5-9.6.mo
./usr/share/locale/sv/LC_MESSAGES/libpq5-9.6.mo
./usr/share/locale/pt_BR/LC_MESSAGES/libpq5-9.6.mo
./usr/share/locale/fr/LC_MESSAGES/libpq5-9.6.mo
./usr/share/locale/es/LC_MESSAGES/libpq5-9.6.mo
./usr/share/locale/zh_CN/LC_MESSAGES/libpq5-9.6.mo
./usr/share/locale/ja/LC_MESSAGES/libpq5-9.6.mo
./usr/lib/x86_64-linux-gnu/pkgconfig/libpq.pc
./usr/lib/x86_64-linux-gnu/libpq.so.5
./usr/lib/x86_64-linux-gnu/libpq.so
./usr/lib/x86_64-linux-gnu/libpq.so.5.9
./usr/lib/x86_64-linux-gnu/libpq.a
./usr/include/postgresql/libpq-events.h
./usr/include/postgresql/libpq-fe.h
./usr/include/postgresql/libpq
./usr/include/postgresql/libpq/libpq-fs.h
./usr/include/postgresql/internal/libpq
./usr/include/postgresql/internal/libpq-int.h>
Sadly I'm not able to see here wether there are multiple libpq versions...
I read some posts regarding to the error I am seeing now when import pyspark, some suggest to install py4j, and I already did, and yet I am still seeing the error.
I am using a conda environment, here is the steps:
1. create a yml file and include the needed packages (including the py4j)
2. create a env based on the yml
3. create a kernel pointing to the env
4. start the kernel in Jupyter
5. running `import pyspark` throws error: ImportError: No module named py4j.protocol
The issue is resolved with adding environment section in kernel.json and explicitely specify the variables of the following:
"env": {
"HADOOP_CONF_DIR": "/etc/spark2/conf/yarn-conf",
"PYSPARK_PYTHON":"/opt/cloudera/parcels/Anaconda/bin/python",
"SPARK_HOME": "/opt/cloudera/parcels/SPARK2",
"PYTHONPATH": "/opt/cloudera/parcels/SPARK2/lib/spark2/python/lib/py4j-0.10.7-src.zip:/opt/cloudera/parcels/SPARK2/lib/spark2/python/",
"PYTHONSTARTUP": "/opt/cloudera/parcels/SPARK2/lib/spark2/python/pyspark/shell.py",
"PYSPARK_SUBMIT_ARGS": " --master yarn --deploy-mode client pyspark-shell"
}
I am trying to use puppet modules in vagrant.
My box is running puppet 4.0
I am installing modules using:
if [ ! -d /etc/puppet/modules/ ]; then
puppet module install puppetlabs-java
fi
in site.pp
I have:
class { 'java':
distribution => 'jdk',
}
I keep getting an error about could not find declared class java
why can't puppet find my module?
/etc/puppet/modules/ is the default path isn't it?
vagrant file
Vagrant.configure(2) do |config|
config.vm.box = "bento/centos-7.2"
config.vm.provider "virtualbox" do |vb|
vb.gui = true
vb.memory = "8192"
end
config.vm.provision :shell, :path => "upgrade_puppet.sh"
config.vm.provision :shell, :path => "puppet_modules.sh"
config.vm.provision :puppet do |puppet|
puppet.options = '--verbose --debug'
puppet.environment_path = "puppet/environments"
puppet.environment = "production"
end
end
Updated answer now that Vagrantfile has been provided
Locations have changed in puppet 4 and directory environments are now in use by default.
So how you are using the puppet provisioner is correct. However, vagrant will upload all the directories it needs to the guest, based on your Vagrantfile to:
/tmp/vagrant-puppet/environments/production
When Vagrant calls the puppet apply it will be looking for the modules it requires in:
/tmp/vagrant-puppet/environments/production/modules
and that module directory does not exist on your host.
You can change your if block to be:
if [ ! -d /vagrant/puppet/environments/production/modules ]; then
puppet module install puppetlabs-java --modulepath /vagrant/puppet/environments/production/modules
fi
/vagrant is shared between host and guest. This would install the java module and its dependencies on your host machine under:
puppet
|
+--environments
+
-- production
|
+ -- manifests
| +
| -- site.pp
|
+ -- modules
+
-- java
+
-- stdlib
When you do your vagrant up, this content gets uploaded to the host under:
/tmp/vagrant-puppet
Tested and confirmed based on your Vagrantfile.
As Jaxim mentions, it's because the default directory locations have changed in the newer version of Puppet.
If you're interested in installing moduels automatically with Puppet, I'd recommend the R10K vagrant plugin, you can specify versions of modules and make updating them much easier, and allows you to download modules not on the forge, such as git repos.
https://github.com/jantman/vagrant-r10k
A little bit late, but I am switching from Chef over to Puppet (company policy, do not ask! :) ) and ran into the exact same situation and coming from Chef background I was refusing to "pollute" my project folder with so many Puppet specific stuff. In my opinion, I should only need Vagrantfile and nothing else.
I was also getting the "Could not find declared class java at /tmp/vagrant-puppet/environments/production" error message. So, after much messing around I've found that in puppet.options you can provide any arguments that you would normally provide if calling puppet apply at the command line.
So, if anything helps try modifying the puppet.options in your Vagrantfile as follows:
config.vm.provision :puppet do |puppet|
puppet.options = '--verbose --modulepath=/etc/puppetlabs/code/environments/production/modules'
puppet.environment_path = "puppet/environments"
puppet.environment = "production"
end
This will help Puppet find its own nose and not think that everything is available at the /tmp folder, but that the modules have already been installed at its own folder location.
So this seems to be a really common problem with this setup, but I can't find any solutions that work on SO. I've setup a very new Ubuntu 15.04 server, then installed nginx, virtualenv (and -wrapper), and uWSGI (via apt-get, so globally, not inside the virtualenv).
My virtualenv is located at /root/Env/example. Inside of the virtualenv, I installed Django, then at /srv/www/example/app ran Django's startproject command with the project name example, so I have vaguely this structure:
-root
-Env
-example
-bin
-lib
-srv
-www
-example
-app
-example
manage.py
-example
wsgi.py
...
My example.ini file for uWSGI looks like this:
[uwsgi]
project = example
plugin = python
chdir = /srv/www/example/app/example
home = /root/Env/example
module = example.wsgi:application
master = true
processes = 5
socket = /run/uwsgi/app/example/example.socket
chmod-socket = 664
uid = www-data
gid = www-data
vacuum = true
But no matter whether I run this via uwsgi --ini /etc/uwsgi/apps-enabled/example.ini or via daemon, I get the exact same error:
Python version: 2.7.9 (default, Apr 2 2015, 15:37:21) [GCC 4.9.2]
Set PythonHome to /root/Env/example
ImportError: No module named site
I should note that the Django project works via the built-in development server ./manage.py runserver, and that when I remove home = /root/Env/example the thing works (but is obviously using the global Python and Django rather than the virtualenv versions, which means it's useless for a proper virtualenv setup).
Can anyone see some obvious path error that I'm not seeing? As far as I can tell, home is entirely correct based on my directory structure, and everything else in the ini too, so why is it not working with this ImportError?
In my case, I was seeing this issue because the django app I was trying to run was written in python 3 whereas uwsgi was configured for python 2. I fixed the problem by:
recompiling uwsgi to support both python 2 and python 3 apps
(I followed this guide)
adding this to my mydjangoproject_uwsgi.ini:
plugins = python35 # or whatever you specified while compiling uwsgi
For other folks using Django, you should also make sure you are correctly specifying the following:
# Django dir that contains manage.py
chdir = /var/www/project/myprojectname
# Django wsgi (myprojectname is the name of your top-level project)
module = myprojectname.wsgi:application
# the virtualenv you are using (full path)
home = /home/ubuntu/Env/mydjangovenv
plugins = python35
As #Freek said, site refers to a python module.
The error claims that python cannot find that package, which is because you have specified python_home to the wrong location.
I've encountered with the same problem and my uwsgi.ini is like below:
[uwsgi]
# variable
base = /home/xx/
# project settings
chdir = %(base)/
module = botservice.uwsgi:application
home = %(base)/env/bin
For this configuration uwsgi can find python executable in /env/bin but no packages could be found under this folder. So I changed home to
home = %(base)/env/
and it worked for me.
In your case, I suggest digging into home directive and point it to a location which contains both python executable and packages.
The site module is in the root of django.
First check is to activate the virtualenv manually (source /root/Env/example/bin/activate, start python and import site). If that fails, pip install django.
Assuming that django is correctly installed in the virtualenv, make sure that uWSGI activates the virtualenv. Relevant uWSGI configuration directives:
plugins = python
virtualenv = /root/Env/example
and in case you have error importing example.wsgi:
pythonpath = /srv/www/example/app/example
I am trying to import some of my personal modules into my IPython Clusters. I am using Anacondas on Windows Vista 64 bit
from IPython.parallel import Client
rc = Client()
dview = rc[:]
with dview.sync_imports():
import lib.rf
It is giving me this error:
No module named 'lib.rf'
I can import the module in the rest of my IPython notebook, as I have this .bat file to start ipython notebook:
cd C:\Users\Jon\workspace\bf
set PYTHONPATH=%PYTHONPATH%;C:\Users\Jon\workspace\bf
C:\Anaconda\envs\p33\scripts\ipython notebook
I am using this similar code to start my ip clusters:
cd C:\Users\Jon\workspace\bf
set PYTHONPATH=%PYTHONPATH%;C:\Users\Jon\workspace\bf
C:\Anaconda\envs\p33\Scripts\ipcluster start --n=7
Why is this not working?
More info:
If I print out sys.path, I get a list that contains C:\Users\Jon\workspace\bf
If I print out the paths of my clusters, I get the same list:
%px sys.path
['',
'',
'',
'C:\\Anaconda\\envs\\p33\\lib\\site-packages\\distribute-0.6.28-py3.3.egg',
'C:\\Anaconda\\envs\\p33\\lib\\site-packages\\pykalman-0.9.5-py3.3.egg',
'C:\\Anaconda\\envs\\p33\\lib\\site-packages\\patsy-0.2.1-py3.3.egg',
'C:\\Anaconda\\envs\\p33\\lib\\site-packages\\joblib-0.8.3_r1-py3.3.egg',
'C:\\Users\\Jon\\workspace\\bf',
'C:\\Users\\Jon\\workspace\\bf\\my_numba',
'C:\\Anaconda\\envs\\p33\\python33.zip',
'C:\\Anaconda\\envs\\p33\\DLLs',
'C:\\Anaconda\\envs\\p33\\lib',
'C:\\Anaconda\\envs\\p33',
'C:\\Anaconda\\envs\\p33\\lib\\site-packages',
'C:\\Anaconda\\envs\\p33\\lib\\site-packages\\Sphinx-1.2.3-py3.3.egg',
'C:\\Anaconda\\envs\\p33\\lib\\site-packages\\win32',
'C:\\Anaconda\\envs\\p33\\lib\\site-packages\\win32\\lib',
'C:\\Anaconda\\envs\\p33\\lib\\site-packages\\Pythonwin',
'C:\\Anaconda\\envs\\p33\\lib\\site-packages\\runipy-0.1.1-py3.3.egg',
'C:\\Anaconda\\envs\\p33\\lib\\site-packages\\setuptools-7.0-py3.3.egg',
'C:\\Anaconda\\envs\\p33\\lib\\site-packages\\IPython\\extensions']
In [45]:
Further analysis:
%px lib.__path__
Out[0:11]: _NamespacePath(['C:\\Anaconda\\envs\\p33\\lib\\site-packages\\win32\\lib'])
lib.__path__
Out[57]: ['.\\lib']
Looks like the ipcluster and notebook are looking at lib in different places. I have tried renaming lib to mylib. It has not helped.
It seems that with dview.sync_imports() is being run someplace other than your IPython Notebook environment and is therefore relying a different PYTHONPATH. It is definitely not being run on one of the cluster engines and so wouldn't expect it to leverage your cluster settings of PYTHONPATH.
I'm thinking you'll need to have that directory in your PYTHONPATH (not your PATH) for the calling python environment because that is the location from which you are importing the modules.
The impact of the bit you have about setting the PYTHONPATH in the DOS shell from which you invoke ipclusters isn't clear to me. I can see that one might expect this to let the engines know about your directory, but I'm wondering if that PYTHONPATH gets initilized to the environment from which you call IPython.parallel.Client.