Background
I have developed a python flask app that uses the ArcGIS API for JavaScript to display geospatial data and query feature services from an ArcGIS Online account. Results from these queries are saved to a MongoDB Atlas cluster.
The flask app is deployed to an Apache server running Centos8 using mod_wsgi. It is currently working on a Heroku/gunicorn deployment.
Below is the code for the app.py code for the Flask app:
from flask import Flask, render_template, request
from data.database import global_init
from data.database import Query
from data.update import update
from whitenoise import WhiteNoise
app = Flask(__name__)
app.wsgi_app = WhiteNoise(app.wsgi_app)
my_static_folders = (
'./static/css/',
'./static/images/',
'./static/js/',
'./static/layers/'
)
for static in my_static_folders:
app.wsgi_app.add_files(static)
#app.route("/query", methods=["GET", "POST"])
def query():
if request.method == 'POST':
feature = request.json
feature_name = feature['name']
feature_region = feature['region']
feature_query = Query.objects(name=feature_name, region=feature_region)
if feature_query:
response = feature_query[0].export()
else:
response = ''
return response
#app.route("/")
def home():
return render_template("index.html")
if __name__ == "__main__":
global_init()
files = ['./static/css/styles.css', './static/js/app.js']
app.run(debug=True,
extra_files=files)
Below is the wsgi.py file:
#!/usr/bin/env python
import sys
import site
site.addsitedir('/var/www/dm.nhmarchive.org/lau-map/venv/lib/python3.6/site-packages')
sys.path.insert(0, '/var/www/dm.nhmarchive.org/lau-map')
from app import app as application
Problem
The issue arise when the app tries connecting to the MongoDB Atlas cluster to query/retrieve data. When a polygon is clicked on the map, the connection to the cluster is refused and the following error is logged in the error.log:
[Thu Oct 21 20:19:46.800129 2021] [wsgi:error] [pid 1430689:tid 139930235827968] [client 99.8.162.56:55543] , referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471649 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] [2021-10-21 20:20:14,470] ERROR in app: Exception on /query [POST], referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471711 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] Traceback (most recent call last):, referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471715 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] File "/var/www/dm.nhmarchive.org/lau-map/venv/lib/python3.6/site-packages/flask/app.py", line 2447, in wsgi_app, referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471718 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] response = self.full_dispatch_request(), referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471721 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] File "/var/www/dm.nhmarchive.org/lau-map/venv/lib/python3.6/site-packages/flask/app.py", line 1952, in full_dispatch_request, referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471724 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] rv = self.handle_user_exception(e), referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471727 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] File "/var/www/dm.nhmarchive.org/lau-map/venv/lib/python3.6/site-packages/flask/app.py", line 1821, in handle_user_exception, referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471729 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] reraise(exc_type, exc_value, tb), referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471732 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] File "/var/www/dm.nhmarchive.org/lau-map/venv/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise, referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471751 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] raise value, referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471757 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] File "/var/www/dm.nhmarchive.org/lau-map/venv/lib/python3.6/site-packages/flask/app.py", line 1950, in full_dispatch_request, referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471760 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] rv = self.dispatch_request(), referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471762 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] File "/var/www/dm.nhmarchive.org/lau-map/venv/lib/python3.6/site-packages/flask/app.py", line 1936, in dispatch_request, referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471765 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] return self.view_functions[rule.endpoint](**req.view_args), referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471768 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] File "/var/www/dm.nhmarchive.org/lau-map/app.py", line 51, in query, referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471770 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] feature_query = Query.objects(name=feature_name, region=feature_region), referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471773 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] File "/var/www/dm.nhmarchive.org/lau-map/venv/lib/python3.6/site-packages/mongoengine/queryset/manager.py", line 37, in __get__, referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471776 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] queryset = queryset_class(owner, owner._get_collection()), referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471778 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] File "/var/www/dm.nhmarchive.org/lau-map/venv/lib/python3.6/site-packages/mongoengine/document.py", line 214, in _get_collection, referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471786 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] if cls._meta.get("auto_create_index", True) and db.client.is_primary:, referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471788 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] File "/var/www/dm.nhmarchive.org/lau-map/venv/lib/python3.6/site-packages/pymongo/mongo_client.py", line 1031, in is_primary, referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471791 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] return self._server_property('is_writable'), referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471793 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] File "/var/www/dm.nhmarchive.org/lau-map/venv/lib/python3.6/site-packages/pymongo/mongo_client.py", line 856, in _server_property, referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471796 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] writable_server_selector), referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471798 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] File "/var/www/dm.nhmarchive.org/lau-map/venv/lib/python3.6/site-packages/pymongo/topology.py", line 243, in select_server, referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471800 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] address)), referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471803 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] File "/var/www/dm.nhmarchive.org/lau-map/venv/lib/python3.6/site-packages/pymongo/topology.py", line 200, in select_servers, referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471805 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] selector, server_timeout, address), referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471808 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] File "/var/www/dm.nhmarchive.org/lau-map/venv/lib/python3.6/site-packages/pymongo/topology.py", line 217, in _select_servers_loop, referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471810 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] (self._error_message(selector), timeout, self.description)), referer: http://dm.nhmarchive.org/
[Thu Oct 21 20:20:14.471814 2021] [wsgi:error] [pid 1430925:tid 139930051188480] [client 99.8.162.56:55665] pymongo.errors.ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused, Timeout: 30s, Topology Description: <TopologyDescription id: 6171cb5aa3baaebd849f82b2, topology_type: Single, servers: [<ServerDescription ('localhost', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('localhost:27017: [Errno 111] Connection refused',)>]>, referer: http://dm.nhmarchive.org/
The connection to the MongoDB Atlas cluser seems to be refused when app.py tries to perform a query. Below is the code used to establish a conncetion with the database instance in database.py:
import mongoengine
...
from mongoengine import connect
from dotenv import load_dotenv
# Connects to remote Atlas database
def global_init():
load_dotenv()
DB_URI = os.getenv('DB_URI')
connect(alias='laumap', host=DB_URI)
Troubleshooting
Adding the server's IP address to Atlas's 'IP Access List'
While I had 0.0.0.0/0 listed as IP Address (which should theoretically allow all addresses to my understanding), I added the IP address of the server to the list. This still did not work.
Adding pymongo kwargs to MongoEngine connect()
A similar problem was posted on a flask app with Atlas depolyed on python anywhere. They suggested adding the following kwargs when connecting to a database:
connect(alias='laumap', host=DB_URI, connect=False, maxPoolSize=1)
This also did not seem to work.
Adding HTTP Method Overrides
This was done as per the official Flask documentation:
class HTTPMethodOverrideMiddleware(object):
allowed_methods = frozenset([
'GET',
'HEAD',
'POST',
'DELETE',
'PUT',
'PATCH',
'OPTIONS'
])
bodyless_methods = frozenset(['GET', 'HEAD', 'OPTIONS', 'DELETE'])
def __init__(self, app):
self.app = app
def __call__(self, environ, start_response):
method = environ.get('HTTP_X_HTTP_METHOD_OVERRIDE', '').upper()
if method in self.allowed_methods:
environ['REQUEST_METHOD'] = method
if method in self.bodyless_methods:
environ['CONTENT_LENGTH'] = '0'
return self.app(environ, start_response)
app = Flask(__name__)
app.wsgi_app = HTTPMethodOverrideMiddleware(app.wsgi_app)
I don't have any experience setting HTTP headers in a Flask, so I'm unsure if the above is configured correctly. But as it is currently, it did not change the logged errors.
Testing the .env DB_URI variable in a python interactive session
Just as a sanity check, I made sure that the DB_URI was correctly formed in my .env file. Below is the code used in the python interactive shell on the server:
>>> import mongoengine
>>> from dotenv import load_dotenv
>>> load_dotenv()
True
>>> import os
>>> DB_URI = os.getenv('DB_URI')
>>> mongoengine.connect(alias='laumap', host=DB_URI)
MongoClient(host=['lau-shard-00-00.sybdh.mongodb.net:27017', 'lau-shard-00-
01.sybdh.mongodb.net:27017', 'lau-shard-00-02.sybdh.mongodb.net:27017'],
document_class=dict, tz_aware=False, connect=True, retrywrites=True, w='majority',
authsource='admin', replicaset='atlas-xvjuv5-shard-0', ssl=True,
read_preference=Primary())
Checking SE Linux Boolean values
As per this stack overflow post, I tried setting the following values to true:
httpd_can_network_connect --> off
httpd_can_network_connect_db --> off
Question
Why is my flask app not able to properly connect to my MongoDB Atlas cluster when it is able to locally, on my Heroku deployment and when I run flask run on the Apache/CentOS 8 server?
Is there some security settings in CentOS 8 that need to properly configured or is there something in the Flask app or wsgi script that may need to be changed?
From the error message (last line) you can see it is not attempting to connect to atlas, it's attempting to connect to localhost (the default if you don't specify a host). So the error will be with the connection string.
Looking at the code, are you 100% sure the DB_URI environment variable is set correctly? Maybe add some logging to confirm.
if I enabled email forwarding, I can't receive any email but when I disabled email forwarding. Then webmail working fine. Anyone, please look into email logs? I don't see any sending and receiving problems. when email forwarding disabled on cyberpanel.
This problem only appear after enabling email forwarding.
Mar 29 07:14:01 blastoff postfix/bounce[2809]: 3DC8B3410DD: sender non-delivery notification: C9CED3410DE
Mar 29 07:14:01 blastoff postfix/qmgr[1082]: 3DC8B3410DD: removed
Mar 29 07:14:01 blastoff postfix/smtp[2934]: connect to gmail-smtp-in.l.google.com[2607:f8b0:4023:c03::1b]:25: Network is unreachable
Mar 29 07:14:02 blastoff postfix/smtp[2934]: C9CED3410DE: to=<riadloud#gmail.com>, relay=gmail-smtp-in.l.google.com[74.125.137.26]:25, delay=0.86, delays=0.01/0.07/0.36/0.42, dsn=2.0.0, status=sent (250 2.0.0 OK 1617002042 i21si16857578otj.220 - gsmtp)
Mar 29 07:14:02 blastoff postfix/qmgr[1082]: C9CED3410DE: removed
Mar 29 07:14:32 blastoff dovecot: imap-login: Login: user=<riad#blastoff.us>, method=PLAIN, rip=::1, lip=::1, mpid=2967, TLS, session=<QfcVoqe+6pUAAAAAAAAAAAAAAAAAAAAB>
Mar 29 07:14:32 blastoff dovecot: imap(riad#blastoff.us)<2967><QfcVoqe+6pUAAAAAAAAAAAAAAAAAAAAB>: Logged out in=89 out=1045 deleted=0 expunged=0 trashed=0 hdr_count=0 hdr_bytes=0 body_count=0 body_bytes=0
Mar 29 07:16:32 blastoff dovecot: imap-login: Login: user=<riad#blastoff.us>, method=PLAIN, rip=::1, lip=::1, mpid=3043, TLS, session=<b41Aqae+7JUAAAAAAAAAAAAAAAAAAAAB>
Mar 29 07:16:32 blastoff dovecot: imap(riad#blastoff.us)<3043><b41Aqae+7JUAAAAAAAAAAAAAAAAAAAAB>: Logged out in=89 out=1045 deleted=0 expunged=0 trashed=0 hdr_count=0 hdr_bytes=0 body_count=0 body_bytes=0
Mar 29 07:17:18 blastoff postfix/anvil[2916]: statistics: max connection rate 1/60s for (smtp:209.85.217.53) at Mar 29 07:13:56
Mar 29 07:17:18 blastoff postfix/anvil[2916]: statistics: max connection count 1 for (smtp:209.85.217.53) at Mar 29 07:13:56
Mar 29 07:17:18 blastoff postfix/anvil[2916]: statistics: max cache size 1 at Mar 29 07:13:56
Mar 29 07:17:24 blastoff dovecot: imap-login: Login: user=<riad#blastoff.us>, method=PLAIN, rip=::1, lip=::1, mpid=3087, TLS, session=<3BlfrKe+7pUAAAAAAAAAAAAAAAAAAAAB>
Mar 29 07:17:24 blastoff dovecot: imap(riad#blastoff.us)<3087><3BlfrKe+7pUAAAAAAAAAAAAAAAAAAAAB>: Logged out in=240 out=1300 deleted=0 expunged=0 trashed=0 hdr_count=0 hdr_bytes=0 body_count=0 body_bytes=0
Mar 29 07:18:17 blastoff spamd[3146]: logger: removing stderr method
Mar 29 07:18:17 blastoff spamd[3148]: config: no rules were found! Do you need to run 'sa-update'?
Mar 29 07:18:18 blastoff spamd[3146]: child process [3148] exited or timed out without signaling production of a PID file: exit 255 at /usr/sbin/spamd line 3034.
Mar 29 07:18:19 blastoff spamd[3150]: logger: removing stderr method
Mar 29 07:18:19 blastoff spamd[3152]: config: no rules were found! Do you need to run 'sa-update'?
Mar 29 07:18:20 blastoff spamd[3150]: child process [3152] exited or timed out without signaling production of a PID file: exit 255 at /usr/sbin/spamd line 3034.
Mar 29 07:18:21 blastoff spamd[3156]: logger: removing stderr method
Mar 29 07:18:21 blastoff spamd[3158]: config: no rules were found! Do you need to run 'sa-update'?
Mar 29 07:18:22 blastoff spamd[3156]: child process [3158] exited or timed out without signaling production of a PID file: exit 255 at /usr/sbin/spamd line 3034.
Mar 29 07:18:23 blastoff spamd[3159]: logger: removing stderr method
Mar 29 07:18:23 blastoff spamd[3161]: config: no rules were found! Do you need to run 'sa-update'?
Mar 29 07:18:24 blastoff spamd[3159]: child process [3161] exited or timed out without signaling production of a PID file: exit 255 at /usr/sbin/spamd line 3034.
Mar 29 07:18:24 blastoff spamd[3162]: logger: removing stderr method
Mar 29 07:18:24 blastoff spamd[3164]: config: no rules were found! Do you need to run 'sa-update'?
Mar 29 07:18:25 blastoff spamd[3162]: child process [3164] exited or timed out without signaling production of a PID file: exit 255 at /usr/sbin/spamd line 3034.
Mar 29 07:18:32 blastoff dovecot: imap-login: Login: user=<riad#blastoff.us>, method=PLAIN, rip=::1, lip=::1, mpid=3176, TLS, session=<qhxjsKe+8JUAAAAAAAAAAAAAAAAAAAAB>
Mar 29 07:18:32 blastoff dovecot: imap(riad#blastoff.us)<3176><qhxjsKe+8JUAAAAAAAAAAAAAAAAAAAAB>: Logged out in=89 out=1053 deleted=0 expunged=0 trashed=0 hdr_count=0 hdr_bytes=0 body_count=0 body_bytes=0
Mar 29 07:19:15 blastoff postfix/smtpd[3197]: connect from mail-vs1-f45.google.com[209.85.217.45]
Mar 29 07:19:15 blastoff postfix/smtpd[3197]: 9043434088F: client=mail-vs1-f45.google.com[209.85.217.45]
Mar 29 07:19:15 blastoff postfix/smtpd[3197]: warning: connect to /var/log/policyServerSocket: No such file or directory
Mar 29 07:19:16 blastoff postfix/smtpd[3197]: warning: connect to /var/log/policyServerSocket: No such file or directory
Mar 29 07:19:16 blastoff postfix/smtpd[3197]: warning: problem talking to server /var/log/policyServerSocket: No such file or directory
Mar 29 07:19:16 blastoff postfix/cleanup[3201]: 9043434088F: hold: header Received: from mail-vs1-f45.google.com (mail-vs1-f45.google.com [209.85.217.45])??by mail.blastoff.us (Postfix) with ESMTPS id 9043434088F??for <riad#blastoff.us>; Mon, 29 Mar 2021 07:19:15 +0000 (UTC from mail-vs1-f45.google.com[209.85.217.45]; from=<riadloud#gmail.com> to=<riad#blastoff.us> proto=ESMTP helo=<mail-vs1-f45.google.com>
Mar 29 07:19:16 blastoff postfix/cleanup[3201]: 9043434088F: message-id=<CACGWsS=QumtoJMTYX49XNFv7Kbk_-+xhJ4TrZdFezAytvToTow#mail.gmail.com>
Mar 29 07:19:16 blastoff opendkim[920]: 9043434088F: s=20161025 d=gmail.com SSL
Mar 29 07:19:16 blastoff postfix/smtpd[3197]: disconnect from mail-vs1-f45.google.com[209.85.217.45] ehlo=2 starttls=1 mail=1 rcpt=1 data=1 quit=1 commands=7
Mar 29 07:19:17 blastoff postfix/qmgr[1082]: 07B043410DD: from=<riadloud#gmail.com>, size=2541, nrcpt=2 (queue active)
Mar 29 07:19:19 blastoff postfix/pipe[3212]: 07B043410DD: to=<riad#blastoff.us>, relay=spamassassin, delay=3.7, delays=1.6/0.01/0/2, dsn=5.3.0, status=bounced (command line usage error. Command output: lda: Fatal: Unknown argument: unix Usage: dovecot-lda [-c <config file>] [-d <username>] [-p <path>] [-m <mailbox>] [-e] [-k] [-f <envelope sender>] [-a <original envelope recipient>] [-r <final envelope recipient>] )
Mar 29 07:19:19 blastoff postfix/pipe[3213]: 07B043410DD: to=<riadshout#gmail.com>, orig_to=<riad#blastoff.us>, relay=spamassassin, delay=3.7, delays=1.6/0.02/0/2, dsn=5.3.0, status=bounced (command line usage error. Command output: lda: Fatal: Unknown argument: unix Usage: dovecot-lda [-c <config file>] [-d <username>] [-p <path>] [-m <mailbox>] [-e] [-k] [-f <envelope sender>] [-a <original envelope recipient>] [-r <final envelope recipient>] )
Mar 29 07:19:19 blastoff postfix/cleanup[3201]: 3AEDA3410DE: message-id=<20210329071919.3AEDA3410DE#mail.blastoff.us>
Mar 29 07:19:19 blastoff postfix/bounce[3217]: 07B043410DD: sender non-delivery notification: 3AEDA3410DE
Mar 29 07:19:19 blastoff postfix/qmgr[1082]: 3AEDA3410DE: from=<>, size=6095, nrcpt=1 (queue active)
Mar 29 07:19:19 blastoff postfix/qmgr[1082]: 07B043410DD: removed
Mar 29 07:19:19 blastoff postfix/smtp[3220]: connect to gmail-smtp-in.l.google.com[2607:f8b0:4023:c03::1a]:25: Network is unreachable
Mar 29 07:19:19 blastoff postfix/smtp[3220]: 3AEDA3410DE: to=<riadloud#gmail.com>, relay=gmail-smtp-in.l.google.com[74.125.137.26]:25, delay=0.67, delays=0/0.02/0.29/0.35, dsn=2.0.0, status=sent (250 2.0.0 OK 1617002359 g9si17981450plj.221 - gsmtp)
Mar 29 07:19:19 blastoff postfix/qmgr[1082]: 3AEDA3410DE: removed
There are many configuration issues here.
The one that is causing the reject is:
command line usage error. Command output: lda: Fatal: Unknown argument: unix Usage: dovecot-lda [-c <config file>] [-d <username>] [-p <path>]
But there are also spamassassin issues (no rules found) and other milter issues (/var/log/policyServerSocket missing).
It doesn’t look like a forwarding issue but a general misconfiguration issue.
It is quite difficult to provide a precise advice here, you should review the whole configuration with someone familiar with this stuff.
In the following code sample, I start a proxy server using HTTP::Proxy and attempt to use it to request an HTTPS URL, but the proxy server either doesn't actually make the request, or doesn't return the response. However, if I make the URL use HTTP (not secure), the request succeeds. I've installed both IO::Socket::SSL and LWP::UserAgent::https (yay secret deps!), but am still unable to get HTTPS requests to go through the proxy. How can I get HTTP::Proxy to work with HTTPS URLs?
Here's my code:
#!/usr/bin/env perl
use strict;
use warnings;
use Data::Printer;
use HTTP::Proxy ':log';
use Mojo::UserAgent ();
my $URL = 'https://www.yahoo.com';
my $PROXY_PORT = 8667;
my $pid = fork();
if ($pid) { # I am the parent
print "Press ^c to kill proxy server...\n";
my $proxy = HTTP::Proxy->new( port => $PROXY_PORT );
$proxy->logmask(ALL);
$proxy->via(q{});
$proxy->x_forwarded_for(0);
$proxy->start;
waitpid $pid, 0;
}
elsif ($pid == 0) { # I am the child
sleep 3; # Allow the proxy server to start
my $ua = Mojo::UserAgent->new;
$ua->proxy
->http("http://127.0.0.1:$PROXY_PORT")
->https("http://127.0.0.1:$PROXY_PORT");
my $tx = $ua->get($URL);
if ($tx->error) {
p $tx->error;
}
else {
print "Success!\n";
}
}
else {
die 'Unknown result after forking';
}
Saving the above script as testcase-so.pl and running it:
$ MOJO_CLIENT_DEBUG=1 ./testcase-so.pl
Press ^c to kill proxy server...
-- Blocking request (https://www.yahoo.com)
-- Connect c66a92739c09c76fa24029e8079808c7 (https://www.yahoo.com:443)
-- Client >>> Server (https://www.yahoo.com)
CONNECT www.yahoo.com:443 HTTP/1.1\x0d
User-Agent: Mojolicious (Perl)\x0d
Content-Length: 0\x0d
Host: www.yahoo.com\x0d
Accept-Encoding: gzip\x0d
\x0d
-- Client >>> Server (https://www.yahoo.com)
[Tue Oct 9 12:02:54 2018] (12348) PROCESS: Forked child process 12352
[Tue Oct 9 12:02:54 2018] (12352) SOCKET: New connection from 127.0.0.1:45312
[Tue Oct 9 12:02:54 2018] (12352) REQUEST: CONNECT www.yahoo.com:443
[Tue Oct 9 12:02:54 2018] (12352) REQUEST: Accept-Encoding: gzip
[Tue Oct 9 12:02:54 2018] (12352) REQUEST: Host: www.yahoo.com
[Tue Oct 9 12:02:54 2018] (12352) REQUEST: User-Agent: Mojolicious (Perl)
[Tue Oct 9 12:02:54 2018] (12352) REQUEST: Content-Length: 0
[Tue Oct 9 12:02:54 2018] (12352) RESPONSE: 200 OK
[Tue Oct 9 12:02:54 2018] (12352) RESPONSE: Date: Tue, 09 Oct 2018 12:02:54 GMT
[Tue Oct 9 12:02:54 2018] (12352) RESPONSE: Transfer-Encoding: chunked
[Tue Oct 9 12:02:54 2018] (12352) RESPONSE: Server: HTTP::Proxy/0.304
-- Client <<< Server (https://www.yahoo.com)
HTTP/1.1 200 OK\x0d
Date: Tue, 09 Oct 2018 12:02:54 GMT\x0d
Transfer-Encoding: chunked\x0d
Server: HTTP::Proxy/0.304\x0d
\x0d
[Tue Oct 9 12:03:14 2018] (12352) CONNECT: Connection closed by the client
[Tue Oct 9 12:03:14 2018] (12352) PROCESS: Served 1 requests
[Tue Oct 9 12:03:14 2018] (12352) CONNECT: End of CONNECT proxyfication
\ {
message "Proxy connection failed"
}
[Tue Oct 9 12:03:15 2018] (12348) PROCESS: Reaped child process 12349
[Tue Oct 9 12:03:15 2018] (12348) PROCESS: 1 remaining kids: 12352
[Tue Oct 9 12:03:15 2018] (12348) PROCESS: Reaped child process 12352
[Tue Oct 9 12:03:15 2018] (12348) PROCESS: 0 remaining kids:
^C[Tue Oct 9 12:04:04 2018] (12348) STATUS: Processed 2 connection(s)
$
And with the $URL switched to not use https:
$ MOJO_CLIENT_DEBUG=1 ./testcase-so.pl
Press ^c to kill proxy server...
-- Blocking request (http://www.yahoo.com)
-- Connect f792ee97a0362ab493575d8116e69e59 (http://127.0.0.1:8667)
-- Client >>> Server (http://www.yahoo.com)
GET http://www.yahoo.com HTTP/1.1\x0d
Accept-Encoding: gzip\x0d
Content-Length: 0\x0d
Host: www.yahoo.com\x0d
User-Agent: Mojolicious (Perl)\x0d
\x0d
[Tue Oct 9 12:09:38 2018] (12656) PROCESS: Forked child process 12659
-- Client >>> Server (http://www.yahoo.com)
[Tue Oct 9 12:09:38 2018] (12659) SOCKET: New connection from 127.0.0.1:58288
[Tue Oct 9 12:09:38 2018] (12659) REQUEST: GET http://www.yahoo.com
[Tue Oct 9 12:09:38 2018] (12659) REQUEST: Accept-Encoding: gzip
[Tue Oct 9 12:09:38 2018] (12659) REQUEST: Host: www.yahoo.com
[Tue Oct 9 12:09:38 2018] (12659) REQUEST: User-Agent: Mojolicious (Perl)
[Tue Oct 9 12:09:38 2018] (12659) REQUEST: Content-Length: 0
[Tue Oct 9 12:09:38 2018] (12659) RESPONSE: 301 Moved Permanently
[Tue Oct 9 12:09:38 2018] (12659) RESPONSE: Cache-Control: no-store, no-cache
[Tue Oct 9 12:09:38 2018] (12659) RESPONSE: Date: Tue, 09 Oct 2018 14:10:01 GMT
[Tue Oct 9 12:09:38 2018] (12659) RESPONSE: Transfer-Encoding: chunked
[Tue Oct 9 12:09:38 2018] (12659) RESPONSE: Via: http/1.1 media-router-fp1006.prod.media.bf1.yahoo.com (ApacheTrafficServer [c s f ])
[Tue Oct 9 12:09:38 2018] (12659) RESPONSE: Location: https://www.yahoo.com/
[Tue Oct 9 12:09:38 2018] (12659) RESPONSE: Server: ATS
[Tue Oct 9 12:09:38 2018] (12659) RESPONSE: Content-Language: en
[Tue Oct 9 12:09:38 2018] (12659) RESPONSE: Content-Length: 8
[Tue Oct 9 12:09:38 2018] (12659) RESPONSE: Content-Type: text/html
[Tue Oct 9 12:09:38 2018] (12659) RESPONSE: Content-Security-Policy: sandbox allow-forms allow-same-origin allow-scripts allow-popups allow-popups-to-escape-sandbox allow-presentation; report-uri https://csp.yahoo.com/beacon/csp?src=ats&site=frontpage®ion=US&lang=en-US&device=desktop&yrid=&partner=;
[Tue Oct 9 12:09:38 2018] (12659) RESPONSE: X-Frame-Options: SAMEORIGIN
[Tue Oct 9 12:09:38 2018] (12659) RESPONSE: X-XSS-Protection: 1; report="https://csp.yahoo.com/beacon/csp?src=fp-hpkp-www"
-- Client <<< Server (http://www.yahoo.com)
HTTP/1.1 301 Moved Permanently\x0d
Cache-Control: no-store, no-cache\x0d
Date: Tue, 09 Oct 2018 14:10:01 GMT\x0d
Transfer-Encoding: chunked\x0d
Via: http/1.1 media-router-fp1006.prod.media.bf1.yahoo.com (ApacheTrafficServer [c s f ])\x0d
Location: https://www.yahoo.com/\x0d
Server: ATS\x0d
Content-Language: en\x0d
Content-Length: 8\x0d
Content-Type: text/html\x0d
Content-Security-Policy: sandbox allow-forms allow-same-origin allow-scripts allow-popups allow-popups-to-escape-sandbox allow-presentation; report-uri https://csp.yahoo.com/beacon/csp?src=ats&site=frontpage®ion=US&lang=en-US&device=desktop&yrid=&partner=;\x0d
X-Frame-Options: SAMEORIGIN\x0d
X-XSS-Protection: 1; report="https://csp.yahoo.com/beacon/csp?src=fp-hpkp-www"\x0d
\x0d
-- Client <<< Server (http://www.yahoo.com)
8\x0d
redirect\x0d
0\x0d
\x0d
Success!
[Tue Oct 9 12:09:38 2018] (12659) SOCKET: Getting request failed: Client closed
[Tue Oct 9 12:09:39 2018] (12656) PROCESS: Reaped child process 12657
[Tue Oct 9 12:09:39 2018] (12656) PROCESS: 1 remaining kids: 12659
[Tue Oct 9 12:09:39 2018] (12656) PROCESS: Reaped child process 12659
[Tue Oct 9 12:09:39 2018] (12656) PROCESS: 0 remaining kids:
^C[Tue Oct 9 12:09:45 2018] (12656) STATUS: Processed 2 connection(s)
$
There is a bug in HTTP::Proxy in that it returns the wrong response to a CONNECT request:
-- Client <<< Server (https://www.yahoo.com)
HTTP/1.1 200 OK\x0d
Date: Tue, 09 Oct 2018 12:02:54 GMT\x0d
Transfer-Encoding: chunked\x0d
Server: HTTP::Proxy/0.304\x0d
\x0d
The response to a CONNECT request can have no body which means that it should not have a HTTP header announcing a body like Transfer-Encoding: chunked does. This bug happens with all clients which do a CONNECT request using HTTP/1.1. If the CONNECT is instead done with HTTP/1.0 the problem vanishes since Transfer-Encoding: chunked is not defined with HTTP/1.0 yet and thus HTTP::Proxy does not send it.
The same problem happens when trying to use curl with HTTP::Proxy, thus this is not a problem solely of Mojo::UserAgent. I`ve made a patch to HTTP::Proxy to respond properly. See this pull request for the details and for the (small) diff you need to apply.
Wildfly not working with mod cluster below is the error_log
[Wed Jan 25 16:53:55.860419 2017] [core:error] [pid 4980:tid 1088] [client 192.168.1.10:64522] AH00135: Invalid method in request bela\x19\x84\x04
[Wed Jan 25 16:54:11.387227 2017] [core:error] [pid 4980:tid 1088] [client 192.168.1.10:64523] AH00135: Invalid method in request bela\x19\x84\x04
[Wed Jan 25 16:54:24.664313 2017] [core:error] [pid 4980:tid 1088] [client 192.168.1.10:64524] AH00135: Invalid method in request bela\x19\x84\x04
[Wed Jan 25 16:54:43.907621 2017] [core:error] [pid 4980:tid 1088] [client 192.168.1.10:64525] AH00135: Invalid method in request bela\x19\x84\x04
[Wed Jan 25 16:54:59.417380 2017] [core:error] [pid 4980:tid 1088] [client 192.168.1.10:64526] AH00135: Invalid method in request bela\x19\x84\x04
[Wed Jan 25 16:55:16.458689 2017] [core:error] [pid 4980:tid 1088] [client 192.168.1.10:64530] AH00135: Invalid method in request bela\x19\x84\x04
I am using TCP instaed of upd and server advertise is off on mod_cluster side
It's my first experiment with CGI::Sessions and I'm getting an error message in my logs that I don't know how to fix. It looks like the problem is on line 28 of my code, the $cookie variable in not imported:
[Mon Aug 18 21:42:32 2014] [error] [client 127.0.0.1] Variable "$cookie" is not imported at /usr/lib/cgi-bin/login.pl line 28.
5028 [Mon Aug 18 21:42:32 2014] [error] [client 127.0.0.1] \t(Did you mean &cookie instead?)
5029 [Mon Aug 18 21:42:32 2014] [error] [client 127.0.0.1] Variable "$cookie" is not imported at /usr/lib/cgi-bin/login.pl line 29.
5030 [Mon Aug 18 21:42:32 2014] [error] [client 127.0.0.1] \t(Did you mean &cookie instead?)
5031 [Mon Aug 18 21:42:32 2014] [error] [client 127.0.0.1] Global symbol "$cookie" requires explicit package name at /usr/lib/cgi-bin/login.pl line 28.
5032 [Mon Aug 18 21:42:32 2014] [error] [client 127.0.0.1] Global symbol "$cookie" requires explicit package name at /usr/lib/cgi-bin/login.pl line 29.
5033 [Mon Aug 18 21:42:32 2014] [error] [client 127.0.0.1] Bareword "username" not allowed while "strict subs" in use at /usr/lib/cgi-bin/login.pl line 34.
5034 [Mon Aug 18 21:42:32 2014] [error] [client 127.0.0.1] Bareword "password" not allowed while "strict subs" in use at /usr/lib/cgi-bin/login.pl line 35.
5035 [Mon Aug 18 21:42:32 2014] [error] [client 127.0.0.1] Execution of /usr/lib/cgi-bin/login.pl aborted due to compilation errors.
5036 [Mon Aug 18 21:42:32 2014] [error] [client 127.0.0.1] Premature end of script headers: login.pl
Here's my code:
1 #!/usr/bin/perl -wT
2 use strict;
3
4 use CGI::Session;
5 use CGI qw(:standard);
6
7 my %names = map { $_ => 1 } param;
8 my $open = "1234";
9
10 sub windowLayout() {
11 header,
12 start_html("Input Form"),
13 start_form,
14 "Please enter your username:",
15 textfield(-name=>'username',
16 -maxlength=>20),p,
17 "Please enter your password:",
18 password_field(-name=>'password',
19 -maxlength=>20),p,
20 submit,hr
21 }
22
23 if ($names{username} and $names{password}) {
24
25 my $cgi = new CGI;
26 my $session = new CGI::Session(undef, $cgi, {Directory=>'/tmp'});
27
28 $cookie = $cgi->cookie(CGISESSID => $session->id);
29 print $cgi->header(-cookie=>$cookie);
30
31 my $username = param("username");
32 my $password = param("password");
33
34 $session->param(username, $username);
35 $session->param(password, $password);
36
37 if ($password ne $open) {
38 print
39 windowLayout(),
40 end_form,
41 p({-style=>'Color: red;'},
42 "Sorry wrong password!");
43 print end_html;
44 } else {
45 print
46 redirect("hello.pl");
47 }
48 }else {
49 print
50 windowLayout(),
51 end_html;
52 }
As you're using strict in your code (which is, of course, a great habit to get in to), you'll need to declare all of your variables with my (or something equivalent). You don't do that for $cookie.
So change:
$cookie = $cgi->cookie(CGISESSID => $session->id);
To:
my $cookie = $cgi->cookie(CGISESSID => $session->id);
For future reference, if you get a Perl error message that you don't understand, you can get more detail about the problem by adding use diagnostics to you code (but remember to remove it again once the problem is fixed).
You also have a couple of other errors. The strings 'username' and 'password' need to be quoted string on lines 34 and 35.