Error using SciPy: ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject - scipy

I'm trying to install SciPy using a bootstrap script to use it to calculate a new field in a dataframe, running Spark on AWS EMR.
Although the SciPy website states that only NumPy is needed, when I tried to install SciPy using pip, pip kept failing, complaining about missing software, until I ended up with this bootstrap script:
sudo yum install -y python3-devel
sudo pip3 install -U Cython
sudo pip3 install -U pybind11
sudo pip3 install -U pythran
sudo pip3 install -U numpy
sudo pip3 install -U scipy
At this point, the bootstrap seems to be successful, but then at this line:
from scipy.stats import norm
I get the following error:
ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject
I tried --no-binary :all:, but that made no difference.
Any advice? Thanks!
Best, Oliver

Related

pip3 install firebase-admin fails at Command "python setup.py egg_info" failed with error code 1

I'm trying to install firebase-admin on a Raspberry Pi. I have 3 cards with fresh Debian 10 (buster) (i.e. fully updated/upgraded), and tried 3 different ways to get firebase-admin to install (I know it can be done since I did it on another Raspberry Pi, but did so many things before I finally got it to install, that I can't reproduce what I did!). So in each attempt to install firebase-admin, I get the "python setup.py egg_info" failed with error code 1"
Here is what I did on 2 fresh cards in the RPi:
export PATH="/home/pi/.local/bin:$PATH"
python3 -m pip install google-cloud-storage (successful)
python3 -m pip install firebase (successful)
python3 -m pip install firebase_admin ---> fails with error shown above
I then tried sudo in front of the python3 commands above, with the same result. Finally, on one of the non-sudo cards, I did a "python3 -m pip install --upgrade pip setuptools wheel" (successful), and then repeated step 4 above with the same result (i.e. python setup.py egg_info" failed with error code 1).
I am desperate (and doing something really basic and stupid?) .... any help would be much appreciated! Thanks (I have tried all suggestions in questions similar to this one, without success).
Using a raspberry-pi zero W, almost identical problem. firebase works fine, firebase_admin fails with:
python setup.py egg_info did not run successfully. │ exit code: 1
It seems to work if you install the firebase admin package with an older version of grpcio. This is the command I used.
pip install firebase-admin "grpcio <= 1.40.0"
I don't know if it's necessary, but I had that specific version of grpcio already installed manually, from a previous attempt to fix the issue:
pip3 install --no-cache-dir --force-reinstall -Iv grpcio==1.40.0
Hope it helps!

How to install egg file with easy_install?

On a HPC node, I have a file named pysdf-0.1-py3.8-linux-x86_64.egg.
I found a Nvidia Manual to install this file using python -m easy_install pysdf-0.1-py3.8-linux-x86_64.egg.
However, when I run this command I get the following error.
(/scratch/s.1915438/modulus) [s.1915438#sl1 eggs]$ python -m easy_install pysdf-0.1-py3.8-linux-x86_64.egg
/scratch/s.1915438/modulus/bin/python: No module named easy_install
Similarly, if I use pip install as follows
(/scratch/s.1915438/modulus) [s.1915438#sl1 eggs]$ python -m pip install pysdf-0.1-py3.8-linux-x86_64.egg
ERROR: Could not find a version that satisfies the requirement pysdf-0.1-py3.8-linux-x86_64.egg (from versions: none)
ERROR: No matching distribution found for pysdf-0.1-py3.8-linux-x86_64.egg
I do not have admin access on HPC server to use sudo apt. My python is installed at /scratch/s.1915438/modulus/bin/python on the HPC machine.
Does anyone knows why it says Could not find a version that satisfies the requirement.
After 3 months, I figured out how to do this. So, egg files can be installed using something called easy_install which was depreciated back in 2019. The last version of setuptools that supported easy_install was setuptools 42.0.0..
So, you need to downgrade to that version as follows.
pip3 install setuptools==42.0.0
And then install the egg files using the following command.
python3 -m easy_install example.egg
If you wish to upgrade the setuptools to the latest version, then type
pip3 install setuptools --upgrade

Error during scipy installation using pip3

Trying to install scipy on my raspberrypi (openhabian):
$ pip3 install scipy
but I get this message.
What's wrong?
Looks like a lot of Scipy dependencies are missing eg LAPACK.
Have you tried installing using sudo apt-get install python3-scipy? This will deal with any extra libraries that are needed for Scipy to work on your computer.

scipy missing libifport.so.5

I am trying to install scipy 0.10.1 on Ubuntu 10.10 using pip.
I have successfully installed numpy:
$ sudo pip install --upgrade numpy
[snip]
Successfully installed numpy
Cleaning up...
but when I try scipy I get:
$ sudo pip install --upgrade scipy
[snip]
File "/usr/local/lib/python2.6/dist-packages/numpy/linalg/linalg.py", line 23, in <module>
from numpy.linalg import lapack_lite
ImportError: libifport.so.5: cannot open shared object file: No such file or directory
I do have libifport.so.5, but I don't know why the installer isn't finding it.
$ echo $LD_LIBRARY_PATH
:/opt/intel/composerxe-2011.2.137/compiler/lib/intel64
I have also tried to install several other packages in the hope that one of them might help (e.g. libatlas-base-dev liblapack-dev libatlas-base-dev liblapack-dev, etc)
Any thoughts?
Thank you.
Most likely you have not defined the LD_LIBRARY_PATH for root. To check this:
$ sudo su
$echo $LD_LIBRARY_PATH
You can specify environment variables as follows:
$ sudo env LD_LIBRARY_PATH=path/to/compiler' pip install scipy
Check out this blog post to create an alias to save you typing each time:
http://final-world-domination.blogspot.ch/2011/02/sudo-doesnt-export-ldlibrarypath.html
This fixed the problem for me on Debian 9:
conda install -c intel -c conda-forge -c pytorch --override-channels intel-fortran-rt
# find / -iname "libifport*"
./opt/conda/lib/libifport.so
./opt/conda/lib/libifport.so.5
./opt/conda/pkgs/intel-fortran-rt-2021.3.0-intel_3350/lib/libifport.so
./opt/conda/pkgs/intel-fortran-rt-2021.3.0-intel_3350/lib/libifport.so.5

Installing psycopg2 into virtualenv when PostgreSQL is not installed on development system

Is it possible to install psycopg2 into a virtualenv when PostgreSQL isn't installed on my development system—MacBook Pro with OS X 10.6?
When I run pip install psycopg2 from within my virtualenv, I received the error shown below.
I'm trying to connect to a legacy database on a server using Django, and I'd prefer not to install PostgreSQL on my development system if possible.
Why not install PostgreSQL?
I received an error when installing PostgreSQL using homebrew. I have Xcode4—and only Xcode4—installed on my MacBook Pro and am thinking it's related to missing gcc 4.0. However, this is a problem for another StackOverflow question.
Update 8:37 AM on April 12, 2011: I'd still like to know if this is possible without installing PostgreSQL on my MacBook Pro. However, I ran brew update and forced a reinstallation of ossp-uuid with brew install --force ossp-uuid and now brew install postgresql works. With PostgreSQL successfully installed, I was able to pip install psycopg2 from within my virtualenv.
Error from pip install psycopg2
$ pip install psycopg2
Downloading/unpacking psycopg2
Running setup.py egg_info for package psycopg2
Error: pg_config executable not found.
Please add the directory containing pg_config to the PATH
or specify the full executable path with the option:
python setup.py build_ext --pg-config /path/to/pg_config build ...
or with the pg_config option in 'setup.cfg'.
Complete output from command python setup.py egg_info:
running egg_info
writing pip-egg-info/psycopg2.egg-info/PKG-INFO
writing top-level names to pip-egg-info/psycopg2.egg-info/top_level.txt
writing dependency_links to pip-egg-info/psycopg2.egg-info/dependency_links.txt
warning: manifest_maker: standard file '-c' not found
Error: pg_config executable not found.
Please add the directory containing pg_config to the PATH
or specify the full executable path with the option:
python setup.py build_ext --pg-config /path/to/pg_config build ...
or with the pg_config option in 'setup.cfg'.
----------------------------------------
Command python setup.py egg_info failed with error code 1
Storing complete log in /Users/matthew/.pip/pip.log
Preliminary Research
Below are the articles I read as preliminary research:
Installing psycopg2 to use Django with PostgreSQL on OS X
Installing psycopg2 on OS X
Using psycopg2 with virtualenv on Ubuntu JauntyLucid
Postgres, psycopg2, virtualenv install hints
apt-get install libpq-dev
helped me on debian squeeze
From comments:
apt-get install python-dev-is-python3
psycopg depends on pg_config command, and if you don't have it, you can't install psycopg.
If system installation is a problem to you, why don't you try compiling PostgreSQL and including generated bin files in your $PATH? Like:
export PATH=/path/to/compiled/postgresql/bin:"$PATH"
pip install psycopg2
brew install postgresql
export LDFLAGS="-L/usr/local/opt/openssl/lib"
export CPPFLAGS="-I/usr/local/opt/openssl/include"
pip3 install psycopg2
apt-get install libpq-dev
helped me in debian squeeze too . After that do pip install psycopg2. I faced problem of pg_config not found problem when i was setting up my environment on heroku , now its working fine .
You need to install the python-dev package in order to make use of python extensions such as psycopg2. I don't know how to install packages in mac but I run the following commands to install a python package on my Ubuntu machine.
sudo apt-get install python-dev
Or
sudo apt-get install python3-dev
if you are using Python3.x.
Once the installation is finished run the following command within your virtual environment.
pip install psycopg2
You don't need the complete PostgreSQL installed: you only need the client side libraries.
I solved it in MAC OSX using :
$ wget https://ftp.postgresql.org/pub/source/v9.5.3/postgresql-9.5.3.tar.bz2
$ tar xfv postgresql-9.5.3.tar.bz2
$ cd postgresql-9.5.3
$ ./configure
$ make
$ cd src/bin/pg_config
$ export PATH=`pwd`:"$PATH"
$ pip install psycopg2
Use pip install psycopg2-binary, it worked for me when pip install psycopg2 wasn't working.
Using the method you described in your April 12th update, I was able to install PostgreSQL(+1). Note that I originally was running Python 2.7.1 (32bit) and homebrew threw several errors and warnings regarding using a 32bit version of Python. I have since switched the 64/32 bit version of 2.7.1 and it works like a champ.
Regarding the pyscopg2, I was able to install it into my virtual environment from source by editing setup.cfg. I set pg_config to correct path inside homebrew's Cellar (pg_config=/usr/local/Cellar/postgresql/9.0.4/bin/pg_config). After saving the changes, I ran python setup.py install with zero issues. It's worth noting that I did not set the Mac's default python setting to 32 bit. I used the new 64 bit from start to finish.
After looking over some of the documentation, I think if I added homebrew's postgresql path to the system path I could have used pip to install it.
Reference:
http://favosdream.blogspot.com/2009/09/make-psycopg2-and-readline-work-in-snow.html
Update 6-8-2011:
While porting a project written on OS X to Windows 7, I found out that I had to install PostgreSQL on Windows as well. This ended up creating another user on my start up screen and other things that I just didn't like. While doing some digging I found Windows drivers for PostgreSQL here. I have since uninstalled the full PostgreSQL and installed the ODBC drivers which, thus far, work great.
To address the original question, after doing a bit more digging I think I found the equivalent ODBC for OS X here. I have not had a chance to try them out, but the concept works very well on Windows 7. I will update this when I get a chance to try them out. Until then, I hope this helps.
All from the above doesn't work for me (OS Catalina 10.15.1)
There was a conflict with classical postgres and homebrew version.
Please delete homebrew version by command
$ brew uninstall postgresql
and then install it from the official website:
$ sudo mkdir -p /etc/paths.d && echo /Applications/Postgres.app/Contents/Versions/latest/bin | sudo tee /etc/paths.d/postgresapp
It is actual for a simple psycopg2 install or django-heroku package.
if you have recently updated python or changed default python (let's say from 3.6 to 3.8). The following code
sudo apt-get install python-dev OR sudo apt-get install python3-dev
will be installing/working for the previous python version.
so if you want this command to work for the recently updated/changed python version try mentioning that specific version like python3.8 in command like
sudo apt-get install python3.8-dev
try above with following
pip install wheel
export PATH=/path/to/compiled/postgresql/bin:"$PATH"
sudo apt-get install libpq-dev
sudo apt-get install python3.x-dev **Change x with your version, eg python3.8**
pip install psycopg2-binary
pip install psycopg2
As I never needed to install postgresql database on this server I installed the following libraries on Ubuntu 14_04 version before running pip install psycopg2 on the same server
apt-get install libpq-dev python-dev
and then executed pip install psycopg2 within virtual env.
Output
Collecting psycopg2
Using cached psycopg2-2.6.1.tar.gz
Building wheels for collected packages: psycopg2
Running setup.py bdist_wheel for psycopg2
Stored in directory: /root/.cache/pip/wheels/e2/9a/5e/7b620848bbc7cfb9084aafea077be11618c2b5067bd532f329
Successfully built psycopg2
Installing collected packages: psycopg2
Successfully installed psycopg2-2.6.1
I know you are asking for development environment but if you are deploying on server say, Heroku.
Just add below line in the requirements.txt of your project.
django-heroku==0.3.1
As this package itself will install the required packages like psycopg2 on server deployment.
Try to specify the version and it'll works, do it inside the venv
pip install psycopg2-binary==2.8.6
based on my experience, apt install libpq-dev then try to install psycopg2 inside venv will fix many issues, however for more information you can refer to this link:
https://www.psycopg.org/docs/install.html