Background
I have developed a python flask app that uses the ArcGIS API for JavaScript to display geospatial data and query feature services from an ArcGIS Online account. Results from these queries are saved to a MongoDB Atlas cluster.
The flask app is deployed to an Apache server running Centos8 using mod_wsgi. It is currently working on a Heroku/gunicorn deployment.
Below is the code for the app.py code for the Flask app:
from flask import Flask, render_template, request
from data.database import global_init
from data.database import Query
from data.update import update
from whitenoise import WhiteNoise
app = Flask(__name__)
app.wsgi_app = WhiteNoise(app.wsgi_app)
my_static_folders = (
'./static/css/',
'./static/images/',
'./static/js/',
'./static/layers/'
)
for static in my_static_folders:
app.wsgi_app.add_files(static)
#app.route("/query", methods=["GET", "POST"])
def query():
if request.method == 'POST':
feature = request.json
feature_name = feature['name']
feature_region = feature['region']
feature_query = Query.objects(name=feature_name, region=feature_region)
if feature_query:
response = feature_query[0].export()
else:
response = ''
return response
#app.route("/")
def home():
return render_template("index.html")
if __name__ == "__main__":
global_init()
files = ['./static/css/styles.css', './static/js/app.js']
app.run(debug=True,
extra_files=files)
Below is the wsgi.py file:
#!/usr/bin/env python
import sys
import site
site.addsitedir('/var/www/dm.nhmarchive.org/lau-map/venv/lib/python3.6/site-packages')
sys.path.insert(0, '/var/www/dm.nhmarchive.org/lau-map')
from app import app as application
Problem
The issue arise when the app tries connecting to the MongoDB Atlas cluster to query/retrieve data. When a polygon is clicked on the map, the connection to the cluster is refused and the following error is logged in the error.log:
[Thu Oct 21 20:19:46.800129 2021] [wsgi:error] [pid 1430689:tid 139930235827968] [client 99.8.162.56:55543] , referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471649 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] [2021-10-21 20:20:14,470] ERROR in app: Exception on /query [POST], referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471711 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] Traceback (most recent call last):, referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471715 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] File "/var/www/dm.nhmarchive.org/lau-map/venv/lib/python3.6/site-packages/flask/app.py", line 2447, in wsgi_app, referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471718 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] response = self.full_dispatch_request(), referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471721 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] File "/var/www/dm.nhmarchive.org/lau-map/venv/lib/python3.6/site-packages/flask/app.py", line 1952, in full_dispatch_request, referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471724 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] rv = self.handle_user_exception(e), referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471727 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] File "/var/www/dm.nhmarchive.org/lau-map/venv/lib/python3.6/site-packages/flask/app.py", line 1821, in handle_user_exception, referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471729 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] reraise(exc_type, exc_value, tb), referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471732 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] File "/var/www/dm.nhmarchive.org/lau-map/venv/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise, referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471751 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] raise value, referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471757 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] File "/var/www/dm.nhmarchive.org/lau-map/venv/lib/python3.6/site-packages/flask/app.py", line 1950, in full_dispatch_request, referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471760 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] rv = self.dispatch_request(), referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471762 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] File "/var/www/dm.nhmarchive.org/lau-map/venv/lib/python3.6/site-packages/flask/app.py", line 1936, in dispatch_request, referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471765 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] return self.view_functions[rule.endpoint](**req.view_args), referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471768 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] File "/var/www/dm.nhmarchive.org/lau-map/app.py", line 51, in query, referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471770 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] feature_query = Query.objects(name=feature_name, region=feature_region), referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471773 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] File "/var/www/dm.nhmarchive.org/lau-map/venv/lib/python3.6/site-packages/mongoengine/queryset/manager.py", line 37, in __get__, referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471776 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] queryset = queryset_class(owner, owner._get_collection()), referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471778 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] File "/var/www/dm.nhmarchive.org/lau-map/venv/lib/python3.6/site-packages/mongoengine/document.py", line 214, in _get_collection, referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471786 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] if cls._meta.get("auto_create_index", True) and db.client.is_primary:, referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471788 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] File "/var/www/dm.nhmarchive.org/lau-map/venv/lib/python3.6/site-packages/pymongo/mongo_client.py", line 1031, in is_primary, referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471791 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] return self._server_property('is_writable'), referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471793 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] File "/var/www/dm.nhmarchive.org/lau-map/venv/lib/python3.6/site-packages/pymongo/mongo_client.py", line 856, in _server_property, referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471796 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] writable_server_selector), referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471798 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] File "/var/www/dm.nhmarchive.org/lau-map/venv/lib/python3.6/site-packages/pymongo/topology.py", line 243, in select_server, referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471800 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] address)), referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471803 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] File "/var/www/dm.nhmarchive.org/lau-map/venv/lib/python3.6/site-packages/pymongo/topology.py", line 200, in select_servers, referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471805 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] selector, server_timeout, address), referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471808 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] File "/var/www/dm.nhmarchive.org/lau-map/venv/lib/python3.6/site-packages/pymongo/topology.py", line 217, in _select_servers_loop, referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471810 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] (self._error_message(selector), timeout, self.description)), referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471814 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] pymongo.errors.ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused, Timeout: 30s, Topology Description: <TopologyDescription id: 6171cb5aa3baaebd849f82b2, topology_type: Single, servers: [<ServerDescription ('localhost', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('localhost:27017: [Errno 111] Connection refused',)>]>, referer: http://dm.nhmarchive.org/
The connection to the MongoDB Atlas cluser seems to be refused when app.py tries to perform a query. Below is the code used to establish a conncetion with the database instance in database.py:
import mongoengine
...
from mongoengine import connect
from dotenv import load_dotenv
# Connects to remote Atlas database
def global_init():
load_dotenv()
DB_URI = os.getenv('DB_URI')
connect(alias='laumap', host=DB_URI)
Troubleshooting
Adding the server's IP address to Atlas's 'IP Access List'
While I had 0.0.0.0/0 listed as IP Address (which should theoretically allow all addresses to my understanding), I added the IP address of the server to the list. This still did not work.
Adding pymongo kwargs to MongoEngine connect()
A similar problem was posted on a flask app with Atlas depolyed on python anywhere. They suggested adding the following kwargs when connecting to a database:
connect(alias='laumap', host=DB_URI, connect=False, maxPoolSize=1)
This also did not seem to work.
Adding HTTP Method Overrides
This was done as per the official Flask documentation:
class HTTPMethodOverrideMiddleware(object):
allowed_methods = frozenset([
'GET',
'HEAD',
'POST',
'DELETE',
'PUT',
'PATCH',
'OPTIONS'
])
bodyless_methods = frozenset(['GET', 'HEAD', 'OPTIONS', 'DELETE'])
def __init__(self, app):
self.app = app
def __call__(self, environ, start_response):
method = environ.get('HTTP_X_HTTP_METHOD_OVERRIDE', '').upper()
if method in self.allowed_methods:
environ['REQUEST_METHOD'] = method
if method in self.bodyless_methods:
environ['CONTENT_LENGTH'] = '0'
return self.app(environ, start_response)
app = Flask(__name__)
app.wsgi_app = HTTPMethodOverrideMiddleware(app.wsgi_app)
I don't have any experience setting HTTP headers in a Flask, so I'm unsure if the above is configured correctly. But as it is currently, it did not change the logged errors.
Testing the .env DB_URI variable in a python interactive session
Just as a sanity check, I made sure that the DB_URI was correctly formed in my .env file. Below is the code used in the python interactive shell on the server:
>>> import mongoengine
>>> from dotenv import load_dotenv
>>> load_dotenv()
True
>>> import os
>>> DB_URI = os.getenv('DB_URI')
>>> mongoengine.connect(alias='laumap', host=DB_URI)
MongoClient(host=['lau-shard-00-00.sybdh.mongodb.net:27017', 'lau-shard-00-
01.sybdh.mongodb.net:27017', 'lau-shard-00-02.sybdh.mongodb.net:27017'],
document_class=dict, tz_aware=False, connect=True, retrywrites=True, w='majority',
authsource='admin', replicaset='atlas-xvjuv5-shard-0', ssl=True,
read_preference=Primary())
Checking SE Linux Boolean values
As per this stack overflow post, I tried setting the following values to true:
httpd_can_network_connect --> off
httpd_can_network_connect_db --> off
Question
Why is my flask app not able to properly connect to my MongoDB Atlas cluster when it is able to locally, on my Heroku deployment and when I run flask run on the Apache/CentOS 8 server?
Is there some security settings in CentOS 8 that need to properly configured or is there something in the Flask app or wsgi script that may need to be changed?
From the error message (last line) you can see it is not attempting to connect to atlas, it's attempting to connect to localhost (the default if you don't specify a host). So the error will be with the connection string.
Looking at the code, are you 100% sure the DB_URI environment variable is set correctly? Maybe add some logging to confirm.
I am having difficulty placing array columns in a format that is consistent. I have the following output:
Mon,Jun,25,14:39:29,2012,971,29,0,25,0,0,0,4,Mon,Jun,25,14:39:29,2012,25,mod_was_ap22_http.c
Mon,Jun,25,14:40:29,2012,972,28,0,25,0,0,0,3,Mon,Jun,25,14:40:29,2012,3,mod_sm22.cpp,22,mod_was_ap22_http.c
Mon,Jun,25,14:41:29,2012,973,27,0,24,0,0,0,3,Mon,Jun,25,14:41:29,2012,24,mod_was_ap22_http.c
Mon,Jun,25,14:42:29,2012,974,26,0,20,0,0,0,6,Mon,Jun,25,14:42:29,2012,1,mod_sm22.cpp,19,mod_was_ap22_http.c
Mon,Jun,25,14:43:29,2012,971,29,0,26,0,0,0,3,Mon,Jun,25,14:43:29,2012,2,mod_sm22.cpp,24,mod_was_ap22_http.c
Mon,Jun,25,14:44:30,2012,957,43,0,41,0,0,0,2,Mon,Jun,25,14:44:30,2012,1,mod_sm22.cpp,40,mod_was_ap22_http.c
Mon,Jun,25,14:45:30,2012,963,37,0,35,0,0,0,2,Mon,Jun,25,14:45:30,2012,2,mod_sm22.cpp,32,mod_was_ap22_http.c
Mon,Jun,25,14:46:30,2012,972,28,0,24,1,1,0,2,Mon,Jun,25,14:46:30,2012,24,mod_was_ap22_http.c,1,ApacheModule.cpp
Mon,Jun,25,14:47:30,2012,961,39,1,37,0,0,0,1,Mon,Jun,25,14:47:30,2012,37,mod_was_ap22_http.c,1,ApacheModule.cpp
Mon,Jun,25,14:48:30,2012,968,32,0,30,0,0,0,2,Mon,Jun,25,14:48:30,2012,30,mod_was_ap22_http.c
Mon,Jun,25,14:49:30,2012,972,28,0,25,0,0,0,3,Mon,Jun,25,14:49:30,2012,1,mod_sm22.cpp,24,mod_was_ap22_http.c
I would like the columns to display:
DayOfWeek,Month,Day,Time,Year,Rdy,Bsy,Rd,Wr,Ka,Log,Dns,Cls,AP22,SM22,ApacheModule
Currently the columns in bold are not in that order (the rest are correct). Each line isn't consistent with that format. The line sometimes has ap22 first, sometimes has sm22 first, and sometimes has none or all three modules. The number before the module relates to the module. How can I move the data into a consistent format?
Note that the 2nd date, mod_was_http.c, mod_sm22.cpp, and ApacheModule.cpp in each line will be removed in the final array.
Here is my code so far:
# This program parses a error log for necessary information and outputs in CSV format.
# chunks of your input to ignore, see below...
my %ignorables = map { $_ => 1 } qw([notice mpmstats: rdy bsy rd wr ka log dns cls bsy: in);
# 3-arg open is safer than 2, lexical my $fh better than a global FH glob
open my $error_fh, '<', 'iset_error_log';
sub findLines {
my($item,#result)=("");
# Iterates over the lines in the file, putting each into $_
while (<$error_fh>) {
# Select only those fields that have the word 'notice'
if (/\[notice/) {
# Place those lines with the word 'rdy' on the next line
if (/\brdy\b/){
push #result,"$item\n";
$item="";
}
else {
$item.=",";
}
# Split the line into fields, separated by spaces, skip the %ignorables
my #line = grep { not defined $ignorables{$_} } split /\s+/;
# More cleanup
s/|^\[|notice|[]]//g for #line; # remove unnecessary elements from the array
# Output the line.
#line = join(",", #line);
s/,,/,/g for #line;
map $item.=$_, #line;
}
}
#result
}
my #array = &findLines;
foreach $line (#array){
print $line; #This is where I would like to organize the lines if possible.
}
My input file looks like this:
[Mon Jun 25 07:51:17 2012] [notice] mpmstats: rdy 990 bsy 10 rd 0 wr 7 ka 0 log 0 dns 0 cls 3
[Mon Jun 25 07:51:17 2012] [notice] mpmstats: bsy: 2 in mod_sm22.cpp, 5 in mod_was_ap22_http.c
[Mon Jun 25 08:08:17 2012] [notice] mpmstats: rdy 974 bsy 26 rd 1 wr 24 ka 0 log 0 dns 0 cls 1
[Mon Jun 25 08:08:17 2012] [notice] mpmstats: bsy: 1 in mod_sm22.cpp, 23 in mod_was_ap22_http.c, 1 in ApacheModule.cpp Mon,Jun,25,14:38:29,2012,962,38,0,36,0,0,0,2,Mon,Jun,25,14:38:29,2012,3,mod_sm22.cpp,33,mod_was_ap22_http.c
[Mon Jun 25 21:54:41 2012] [notice] mpmstats: rdy 999 bsy 1 rd 0 wr 0 ka 0 log 0 dns 0 cls 1
[Mon Jun 25 21:55:41 2012] [notice] mpmstats: rdy 999 bsy 1 rd 0 wr 0 ka 0 log 0 dns 0 cls 1
[Mon Jun 25 21:59:41 2012] [notice] mpmstats: rdy 999 bsy 1 rd 0 wr 1 ka 0 log 0 dns 0 cls 0
[Mon Jun 25 21:59:41 2012] [notice] mpmstats: bsy: 1 in mod_was_ap22_http.c
[Mon Jun 25 22:00:41 2012] [notice] mpmstats: rdy 999 bsy 1 rd 0 wr 1 ka 0 log 0 dns 0 cls 0
[Mon Jun 25 22:00:41 2012] [notice] mpmstats: bsy: 1 in mod_was_ap22_http.c
[Mon Jun 25 22:03:41 2012] [notice] mpmstats: rdy 998 bsy 2 rd 0 wr 2 ka 0 log 0 dns 0 cls 0
[Mon Jun 25 22:03:41 2012] [notice] mpmstats: bsy: 2 in mod_was_ap22_http.c
[Mon Jun 25 22:08:42 2012] [notice] mpmstats: rdy 998 bsy 2 rd 0 wr 2 ka 0 log 0 dns 0 cls 0
[Mon Jun 25 22:08:42 2012] [notice] mpmstats: bsy: 2 in mod_was_ap22_http.c
[Mon Jun 25 22:21:42 2012] [notice] mpmstats: rdy 999 bsy 1 rd 0 wr 1 ka 0 log 0 dns 0 cls 0
[Mon Jun 25 22:21:42 2012] [notice] mpmstats: bsy: 1 in mod_was_ap22_http.c
[Mon Jun 25 22:22:42 2012] [notice] mpmstats: rdy 999 bsy 1 rd 0 wr 1 ka 0 log 0 dns 0 cls 0
[Mon Jun 25 22:22:42 2012] [notice] mpmstats: bsy: 1 in mod_was_ap22_http.c
[Mon Jun 25 22:31:42 2012] [notice] mpmstats: rdy 999 bsy 1 rd 0 wr 0 ka 0 log 0 dns 0 cls 1
[Mon Jun 25 22:32:42 2012] [notice] mpmstats: rdy 999 bsy 1 rd 0 wr 1 ka 0 log 0 dns 0 cls 0
[Mon Jun 25 22:32:42 2012] [notice] mpmstats: bsy: 1 in mod_was_ap22_http.c
[Mon Jun 25 23:06:43 2012] [notice] mpmstats: rdy 999 bsy 1 rd 0 wr 1 ka 0 log 0 dns 0 cls 0
[Mon Jun 25 23:06:43 2012] [notice] mpmstats: bsy: 1 in mod_was_ap22_http.c
You probably want to reorder the columns while you still have the split, before you use join to turn them back into a line of text.
You just need to do a swap.
# 0 ,1 ,2 ,3 ,4 ,5 ,6 ,7 ,8 ,9 ,10 ,11 ,12 ,13 ,14 ,15
# DayOfWeek,Month,Day,Time,Year,Rdy,Bsy,Rd,Wr,Ka,Log,Dns,Cls,AP22,SM22,ApacheModule
#
# Sometimes the last 2 fields are missing and 13 comes before 14 and 15 in the
# input, so fix that.
if (#line < 16) {
push #line, '', ''; # or whatever you want for blanks
}
#line = #line[0..12,14,15,13]; # rearrange the array
Also, your regular expression s/,,/,/g is going to break this if you use blank strings ('') as your empty fields. The short lines missing the last fields will go back to being short missing the correct 13 and 14 fields.
Based upon the kinds of questions being asked here and previously, I highly recommend you get a copy of Modern Perl (available for download or purchase) or Learning Perl to get a better grasp of the language as a whole. I have recently read much of the former and enjoyed it and gained most of my initial Perl knowledge from an earlier edition of the latter.