Configuring Amazon Elastic Beanstalk with PostGIS - postgresql

Question

Does anyone have any experience setting up Amazon Elastic Beanstalk with PostGIS (so that I can take advantage of Geodjango)?
There are a number of features that the default setup (RDS, featuring MySQL) does not currently support out of box:
1. PostgreSQL + PostGIS
2. The ability to install C/C++ libraries such as GEOS and Proj.4
Thanks in advance

Answer 1

If you want to use geodjango with Amazon Elastic Beanstalk you need to create a custom AMI where you can install PostGIS and then point your Elastic Beanstalk Application to that AMI when spinning up.
Here is a good tutorial on how to customize an EBS AMI. There is also an AWS tutorial for that but I found the first one easier to understand. On my custom AMI I installed geos, gdal, proj4 and postgis from source, and postgres using yum install postgres. Below are the commands i used to install all libraries into the AMI.
For the django app to find the libraries, I also set an additional environmental variable in the AWS EBS Console. In the menubar of my environment, I went to configuration --> software configuration and edited the Environment Properties by adding the property LD_LIBRARY_PATH set as /usr/local/lib/:$LD_LIBRARY_PATH.
Since the beanstalk app instances are not forseen to run the database themselves, I also set up a Amazon RDS Postgres hosted database which is a relatively new service, it supports PostGIS.
If you put that all together, you should get a very scaleable GeoDjango app!
sudo yum install postgresql postgresql-devel postgresql-server postgresql9-contrib gcc gcc-c++ make libtool curl libxml2 libxml2-devel python-devel
wget http://download.osgeo.org/proj/proj-4.8.0.zip
unzip proj-4.8.0.zip
cd proj-4.8.0
./configure
make
sudo make install
cd ..
wget http://download.osgeo.org/geos/geos-3.4.2.tar.bz2
tar -xvf geos-3.4.2.tar.bz2
cd geos-3.4.2
./configure
make
sudo make install
cd ..
wget http://download.osgeo.org/gdal/1.10.1/gdal1101.zip
unzip gdal1101.zip
cd gdal-1.10.1
./configure --with-python=yes
make
sudo make install
cd ..
wget http://download.osgeo.org/postgis/source/postgis-2.1.1.tar.gz
tar -xvf postgis-2.1.1.tar.gz
cd postgis-2.1.1
./configure
make
sudo make install

Answer 2

You can also do it without a custom AMI, just use ebextensions. I tested this with Amazon Instance (2013.09) ami-35792c5c so use that one instead of the newer ones. If you have your Django in Elastic Beanstalk 101 completed, you know about ebextensions. The ebextensions below will quickly get going you can use the following ebextensions. Just place the following in your .ebextensions folder at the base of your repository. I also include postgres 9.3 and memcached in these config files:
00_repo_ostgis.config:
files:
"/etc/yum.repos.d/pgdg-93-redhat.repo":
mode: "000644"
owner: root
group: root
content: |
[pgdg93]
name=PostgreSQL 9.3 $releasever - $basearch
baseurl=http://yum.postgresql.org/9.3/redhat/rhel-6-$basearch
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG-93
[pgdg93-source]
name=PostgreSQL 9.3 $releasever - $basearch - Source
failovermethod=priority
baseurl=http://yum.postgresql.org/srpms/9.3/redhat/rhel-6-$basearch
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG-93
commands:
epel_repo:
command: yum-config-manager -y --enable epel
remi_repo:
command: yum-config-manager -y --enable remi
packages:
rpm:
pgdg-redhat93-9.3-1: 'http://yum.postgresql.org/9.3/redhat/rhel-6-x86_64/pgdg-redhat93-9.3-1.noarch.rpm'
remi: 'http://rpms.famillecollet.com/enterprise/remi-release-6.rpm'
qt4-devel: 'http://mirror.centos.org/centos/6/os/x86_64/Packages/qt-4.6.2-28.el6_5.x86_64.rpm'
01_app_postgis.config:
packages:
yum:
libtiff-devel: ''
libjpeg-devel: ''
libzip-devel: ''
freetype-devel: ''
postgresql-devel: ''
gdal: ''
gdal-python: ''
geos: ''
proj: ''
libmemcached: ''
libmemcached-devel: ''
cyrus-sasl-devel: ''
zlib-devel: ''
container_commands:
01_collectstatic:
command: 'PYTHONPATH=.:..:../lib cd site/<your_project> && ./manage.py collectstatic -c --noinput && cd ../..'
leader_only: true
02_migrate:
command: 'PYTHONPATH=.:..:../lib cd site/<your_project> && ./manage.py migrate --noinput && cd ../..'
leader_only: true
option_settings:
- namespace: aws:elasticbeanstalk:container:python
option_name: WSGIPath
value: site/<your_project>/wsgi.py
- namespace: aws:elasticbeanstalk:container:python:staticfiles
option_name: /static/
value: site/<your_project>/static/
- option_name: DJANGO_SETTINGS_MODULE
value: settings_prod
The structure of my project is a bit different. My settings file and urls.py I moved to the root of my project directory so i had to change the path to settings in wsgi.py. So adjust this accordingly. Just use the container_commands and option_settings you were using before.
Your requirements.txt file should contain at minimum:
Django==1.7.1
Pillow
psycopg2
I store most other python dependencies in ../lib which I include in my PYTHONPATH so my repo structure is like this:
<your_project>/
├── requirements.txt
├── .ebextensions/
│   ├── 00_repos_postgis.config
│   └── 01_app_postgis.config
└── site/
├── <your_project>
  │ ├── wsgi.py
  │ ├── settings_prod.py # used for EB, like settings_local.py but uses env vars
│   └── settings.py
   └── lib/
└── <all pip dependencies>
Checkout the deployment tool I built, it uses fabric. I took what I liked from EB CLI tools and adjusted till it was tailored for django: https://github.com/radlws/django-awseb-tasks
NOTE: It is extremely important that you use AMI ami-35792c5c when you launch your environment. Its the only one that worked for me for this setup. If other instances work please feel free to edit them into this answer. Also see my original question.

Answer 3

As i mentioned here my solution for a 2017.03 image was:
commands:
01_yum_update:
command: sudo yum -y update
02_epel_repo:
command: sudo yum-config-manager -y --enable epel
03_install_gdal_packages:
command: sudo yum -y install gdal gdal-devel
files:
"/etc/httpd/conf.d/wsgihacks.conf":
mode: "000644"
owner: root
group: root
content: |
WSGIPassAuthorization On
packages:
yum:
git: []
postgresql95-devel: []
gettext: []
libjpeg-turbo-devel: []
libffi-devel: []

Answer 4

If you want to use radtek's solution and want to use the latest Amazon AMI (2014.9), you might face dependency problems. This solved it for me.

Answer 5

Open ssh shell of your aws beanstalk. Execute below command step by step. You will get working postgis.
cd /home/ec2-user
sudo yum -y install gcc gcc-c++ make cmake libtool libcurl-devel libxml2-devel rubygems swig fcgi-devel libtiff-devel freetype-devel curl-devel libpng-devel giflib-devel libjpeg-devel cairo-devel freetype-devel readline-devel openssl-devel python27 python27-devel
# PROJ
wget http://download.osgeo.org/proj/proj-4.8.0.tar.gz
tar -zxvf proj-4.8.0.tar.gz
cd proj-4.8.0
./configure
make
sudo make install
cd ..
# GEOS
wget http://download.osgeo.org/geos/geos-3.4.2.tar.bz2
tar -xvf geos-3.4.2.tar.bz2
cd geos-3.4.2
./configure
make
sudo make install
cd ..
# GDAL
wget http://download.osgeo.org/gdal/1.10.1/gdal-1.10.1.tar.gz
tar -zxvf gdal-1.10.1.tar.gz
cd gdal-1.10.1
./configure
make
sudo make install
cd ..
# PostGIS
wget http://download.osgeo.org/postgis/source/postgis-2.1.0.tar.gz
tar -xvf postgis-2.1.0.tar.gz
cd postgis-2.1.0
./configure
make
sudo make install
cd ..
Then create a symbolic link:
ln -s /usr/local/lib/libgdal.so /usr/lib/libgdal.so.1
/sbin/ldconfig

Answer 6

Here in alternative answer with which the python3 bindings for gdal will work in the AWS EB instance. pygdal is a nice package for “virtualenv and setuptools friendly version of standard GDAL python bindings”, but does not support 1.7.3, which is the gdal version installed using sudo yum -y install gdal gdal-devel, e.g this answer.
Thus, I had to install an updated gdal and also setup linkages to go with it. This method draws on information in yellowcap’s, radek’s and various other Stackoverflow posts.
As yellowcap said, you must set ENVIRONMENT VARIABLES that make the library linkages you need. In the AWS EB console, Configuration --> Software configuration, edit the Environment Properties adding:
LD_LIBRARY_PATH  = “/usr/local/lib/:$LD_LIBRARY_PATH”
LD_PRELOAD = ”/usr/lib/libgdal.so.1
Put the following in your .ebextensions folder at the base of your repository:
django.config:
option_settings:
"aws:elasticbeanstalk:application:environment":
DJANGO_SETTINGS_MODULE: “<your project>.settings.server_settings"
PYTHONPATH: "$PYTHONPATH"
"aws:elasticbeanstalk:container:python":
WSGIPath: “<your project>/wsgi.py"
"aws:elasticbeanstalk:container:python:staticfiles":
"/static/": "www/static/"
commands:
01_yum_update:
command: sudo yum -y update
02_pip_upgrade:
command: /opt/python/run/venv/bin/pip install --upgrade pip
ignoreErrors: false
packages:
yum:
postgresql95-devel: []
git: []
libffi-devel: []
container_commands:
01_migrate:
command: "python manage.py migrate"
leader_only: true
02_collectstatic:
command: "source /opt/python/run/venv/bin/activate && python manage.py collectstatic --noinput”
gdal.config
files:
"/opt/elasticbeanstalk/hooks/appdeploy/pre/01_install_gdal_prerequisites.sh":
mode: "000755"
owner: root
group: root
content: |
#!/usr/bin/env bash
sudo yum install -y gcc-c++ gcc libpng libtiff postgresql95-devel git libffi-devel libxml2-devel libxslt-devel htop
wget http://download.osgeo.org/proj/proj-4.9.3.tar.gz
tar -zvxf proj-4.9.3.tar.gz
cd proj-4.9.3
./configure
make -j 2
sudo make install
cd ..
rm proj-4.9.3.tar.gz
wget http://download.osgeo.org/gdal/2.3.2/gdal-2.3.2.tar.gz
tar -zvxf gdal-2.3.2.tar.gz
cd gdal-2.3.2
./configure --with-static-proj4=/usr/local/lib --with-python --with-threads --with-pg=/usr/bin/pg_config
make -j 2
sudo make install
cd ..
rm gdal-2.3.2.tar.gz
ln -s /usr/local/lib/libgdal.so /usr/lib/libgdal.so.1
sudo ldconfig
in requirements.py include the pygdal package
pygdal==2.3.2.4
You must be sure that the gdal version number matches the pygdal version number, see pygdal documentation.
Result
The following works the EC2 instance virtual env, where your django project is located:
$ python manage.py shell
In [1]: from osgeo import gdal
In [2]: