Why does MongoDB port 27017 need to be opened in AWS EC2 Security Group? - mongodb

(I've searched SO, AWS support and more widely without success.)
I've just successfully deployed a MEANjs application to a Bitnami MEAN instance on EC2, following Ahmed Haque's excellent tutorial on scotch.io. As part of the tutorial/deployment I altered the AWS Security Group to include port 27017 for MongoDB traffic. The CIDR notation for the port 27017 was 0.0.0.0/0 - which AFAIK means 'allow access from any IP address'.
Question: Why does MongoDB port 27017 need to be opened in AWS EC2
Security Group for a 'production' type environment? Surely this is directly exposing the DB to the
Internet. The only thing that should be talking to Mongo is the
"/server/api" code, which is running on the same instance -
and so shouldn't need the port opening.
If I change the Security Group rule for port 27017 by closing off 27017, changing the source to: localhost, the internal IP address, the public IP address, or hack a CIDR to be equivalent to any of those - then the web app hangs (static content returns but no responses to db backed api calls). Changing the SG rule back to 0.0.0.0/0 almost immediately 'fixes' the hang.
All is otherwise sweet with my install. I've closed port 3000 (the node app) in the Security Group and am using Apache to proxy port 80 traffic to port 3000. Set up like this, port 3000 does not need to be open in the Security Group; to me this implies that on-instance traffic doesn't need ports to be externally exposed - so how come that's not true of the Mongo port?
I cant see anything in the '/client' code which is talking direct to Mongo.
What am I missing?
Thanks in advance - John

OK, after further investigation and overnight/red wine reflection I think I have an answer for those learners like me following the above tutorial (or similar). Following the Agile principle that 'done' means 'working code in a production environment' I was trying to understand the last 5 meters as a developer trying to get code working in a representative production environment (which wouldn't have unnecessary ports open) - this answer is written from that perspective. (Builds welcome from wiser readers.)
What's Happening
The step in the tutorial which (a) changed the Mongo bind IP address from 127.0.0.1 to 0.0.0.0, and (b) specifies a connection URL which uses the external IP address of the same instance, appears to have two effects:
It makes the MongoDB on the instance you're configuring potentially available to other instances (0.0.0.0 tells Mongo to "listen on all available network interfaces".)
It means that the IP traffic from your MEAN app /server component on the same instance will talk to Mongo as though it was coming from off-instance (even though it's on the same instance). Hence the Security Group needs to make port 27017 open to allow this traffic to flow. (This is the nub of the issue in terms of MEANjs stack component interaction.)
Fix
On a single instance MEANjs server, if you change the Mongo bind IP address back to 127.0.0.1 and the Mongo connection url to be 127.0.0.1:27017 then you can close off port 27017 in the EC2 Security Group and the app still works.
To share one MongoDB across more than one MEANjs app server (without wanting to stray into serverfault territory):
Change the Mongo bind IP address to 0.0.0.0,
Use the private IP address of the Mongo server in other app/instance connection strings
Add a EC2 Security Group CIDR rule of private IP address/24, or private IP address/16 to allow access across instances in the specified internal IP address range.
The above is developer 'hack', not a recommendation for good practice.

Related

Why can't App Engine connect to Compute Engine VM instance?

I have a VM instance (e2-micro) on GCP running with postgres. I added my own external ip address to pg_hba.conf so I can connect to the database on my local machine. Next to that I have a nodeJS application which I want to connect to that database. Locally that works, the application can connect to the database on the VM instance. But when I deploy the app to GCP I get a 500 Server Error when I try to visit the page in the browser.
These are the things I already did/tried:
Created a Firewall rule to allow connections on my own external ip address
Created a VPC connector and added that connector to my app.yaml
Made sure everything is in the same project and region (europe-west1)
If I allow all ip addresses on my VM instance with 0.0.0.0/0 then App Engine can connect, so my guess is that I'm doing something wrong the connector? I use 10.8.0.0/28 as ip range while the internal ip address of the VM instance is 10.132.0.2, is that an issue? I tried an ip range with 10.0.0.0 but that also didn't work.
First check if your app uses a /28 IP address range (see the documentation):
When you create a connector, you also assign it an IP range. Traffic
sent through the connector into your VPC network will originate from
an address in this range. The IP range must be a CIDR /28 range that
is not already reserved in your VPC network.
When you create a VPC connector a proper firewall rulle is also created to allow traffic:
An implicit firewall rule with priority 1000 is created on your VPC
network to allow ingress from the connector's IP range to all
destinations in the network.
As you wrote yourself when you create a rule that allows traffic from any IP it works (your app can connect). So - look for the rule that allows traffic from the IP range that your app is in - if it's not there create it.
Or - you can connect your app to your DB over public IP's - in such case you also have to create a proper rule that will allow the traffic from the app to DB.
Second - check the IP of the DB that app uses.
My guess is that you didn't change the IP of the DB (that app uses) and it tries to connect not via VPC connector but via external IP and that's why it cannot (and works only when you create a firewall rule).
This answer pointed me in the right direction: https://stackoverflow.com/a/64161504/3323605.
I needed to deploy my app with
gcloud beta app deploy
since the VPC connector method was on beta. Also, I tried to connect to the external IP in my app.yaml but that needed to be the internal IP ofcourse.

Connecting Google Cloud Platform's compute engine and app engine via VPC connector

I'd like to know in detail how to connect google compute engine virtual machine instance and app engine.
I've set up a virtual machine instance on Google compute engine, and my Postgres server is running there, following this tutorial: https://cloud.google.com/community/tutorials/setting-up-postgres
I've deployed my flask app under the same project on Google Cloud Platform, creating an app engine instance.
I searched on how to connect compute engine and app engine together, and it seems it should be possible through a VPC connector: connect Google App Engine and Google Compute Engine
This is what my VPC connector looks like:
Serverless VPC access
Name Network Region IP address range Min. throughput Max. throughput
connector-name default europe-west2 10.8.0.0/28 200 300
On my compute engine, I have my VM instance like so:
Name Zone Internal IP External IP
some-name europe-west2-c 10.154.0.2 (nic0) 34.89.113.193
On my flask app, I'm trying to connect to my remote DB like so:
db = PostgresqlExtDatabase(
"some-name", # databse name
user="postgres",
password="some-password",
host="10.154.0.2", # remote host internal ip
port=5432,
)
db.connect()
This is my app.yaml for the vpc access part, I've followed this reference: https://cloud.google.com/appengine/docs/standard/python/connecting-vpc#configuring
vpc_access_connector:
name: projects/some-name/locations/europe-west2/connectors/connector-name
If I understood correctly, if the VPC connector is present, I should just be able to connect using the internal IP address of my VM instance(this case, 10.154.0.2)?
The problem is, when the app is deployed for production, It is still complaining that it cannot connect:
2020-09-26 12:54:51 default[20200926t134815] Is the server running on host "10.154.0.2" and accepting
2020-09-26 12:54:51 default[20200926t134815] TCP/IP connections on port 5432?
If it's connected internally I assume I don't have to add that internal IP to firewall rules, although I did try that as well. As for firewall rules, I have allowed my local machine's IP address so I can connect to the remote Postgres server via PgAdmin.
I've actually tried External IP(34.89.113.193) as well although that doesn't make sense to me.
I'm a bit of a noob on networks and backend stuff in general, any help would be much appreciated.
UPDATED 1
This is my firewall rules:
Direction
Ingress, Egress
Action on match
Allow
Source filters
IP ranges
92.40.176.9/32
78.146.103.141/32
10.154.0.2
Protocols and ports
tcp:5432
Image for reference: Screenshot for the list of firewall rules
It turns out the firewall / postgres configurations were all ok, but because this VPC connector method was on beta, I needed to run:
gcloud beta app deploy
instead of the usual
gcloud app deploy.
This command then updated gcloud Beta Commands and prompted me to enable API:
API [appengine.googleapis.com] not enabled on project [742932836941]. Would you like to enable and retry (this will take a few minutes)? (y/N)?
After enabling this everything worked fine.
Per the information provided seems like both VPC firewall rules and the connector are well configured.
However, based on the messages
2020-09-26 12:54:51 default[20200926t134815] Is the server running on host "10.154.0.2" and accepting
2020-09-26 12:54:51 default[20200926t134815] TCP/IP connections on port 5432?
Seems like the VM or server using 10.154.0.2 is not accepting requests on port 5432 or the port has not been opened, you can use this site to do a port scan.
Based on the guide you followed to create PostgreSQL you are using Ubuntu as OS, therefore I suggest you open the port in ubuntu and see if the issue persists.

What does the --bindip configuration option in mongodb do?

Here is the mongodb documentation https://docs.mongodb.com/manual/reference/configuration-options/ it specifies that
The IP address that mongos or mongod binds to in order to listen for
connections from applications. You may attach mongos or mongod to any
interface. When attaching mongos or mongod to a publicly accessible
interface, ensure that you have implemented proper authentication and
firewall restrictions to protect the integrity of your database.
To bind to multiple IP addresses, enter a list of comma separated
values.
and that
127.0.0.1. is the default configuration
I'm clueless when it comes to networking so I wanted to know if someone could explain to me what this means in a more comprehensive way. Also, what would it mean if I were to change this? Why would I want to bind multiple IPs? and finally if anyone has a clue, why is 127.0.0.1 the default option?
Edit:
(You can skip this edit part)
Some of the motivation behind this question lies in getting these warnings while trying to run mongodb on docker:
2016-05-22T05:36:12.478+0000 I CONTROL [initandlisten] ** WARNING: Insecure configuration, access control is not enabled and no --bind_ip has been specified.
2016-05-22T05:36:12.478+0000 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted,
2016-05-22T05:36:12.478+0000 I CONTROL [initandlisten] ** and the server listens on all available network interfaces.
and also some issues where I'd get this error
2016-05-20T01:04:18.012+0000 I NETWORK [thread1] trying reconnect to 127.0.0.1:27017 (127.0.0.1) failed
2016-05-20T01:04:18.018+0000 I NETWORK [thread1] reconnect 127.0.0.1:27017 (127.0.0.1) ok
(more on that here if you are curious https://dba.stackexchange.com/questions/139075/replica-set-in-mongodb-using-docker-primary-has-error-and-stops-being-primary-w/139145#139145 but this issue is not the topic of this post!)
On bindIp
127.0.0.1 by convention is the IP address of localhost and is bound to the loopback interface, which is only accessible from the same machine.
Using this address as default is best practice, since doing so makes it impossible to accidentally expose a service to the public. You have to make the conscious choice to change the bind IP to make your service publicly available. Which you should only do after you made sure that you took proper security measures.
Note This is very simplified, skipping advanced topics
Typically, a machine has the loopback interface and one or more "real" network interfaces.
Say you have one network interface which is "internal" (only accessible by your application servers, since you put them into the same network) and you have one network interface which is "external" (reachable via the public internet for maintenance purposes). Now, if you would bind your MongoDB instance to all interfaces (you would use the IP address 0.0.0.0 to do that), your MongoDB instance would be accessible from the public internet – hardly a desired situation. Attackers could try to brute force your passwords and may eventually get access to your MongoDB instance. Better to prevent any access from the public internet at all.
What you would rather want to have that your MongoDB instance is accessible for your application servers and from the machine it runs on. So you would bind MongoDB to both the loopback interface's IP (127.0.0.1) and the IP of the private network, which in general would be one of
the range from 10.0.0.0 to 10.255.255.255
the range from 172.16.0.0 to 172.31.255.255
the range from 192.168.0.0 to 192.168.255.255
Let us take our example and say both the application servers and the MongoDB instance are in a private network in the range 192.168.X.X and you have given the MongoDB instance the IP address 192.168.0.1. So you would want to have your MongoDB instance be accessible via 192.168.0.1 so that the application servers can talk to it and via 127.0.0.1 to use the administration tools from the machine MongoDB runs on effortlessly.
So with the YAML configuration syntax, you would pass multiple IPs
NOTE do not add space between commas on multiple IPs
# WARNING!!! WARNING!!! WARNING!!!
# DO NOT DO THIS UNLESS YOU HAVE CLIENT AUTHENTICATION ENABLED
# (or you really, really, really know what you are doing)
net:
bindIp: 127.0.0.1,192.168.0.1
On the warnings
In short, this is MongoDBs way of saying:
Mate, you have two problems: you have not configured security yet and your MongoDB instance is only accessible from the local machine. The former is not as severe because of the latter. But you really should configure security before you bind the MongoDB instance to other IPs than "localhost"!
There is sort of an implied "Unless you really know what you are doing!", because iirc, the warning vanishes if you either activate client authentication or change the bindIp.
In my case i change bindIp to 0.0.0.0 in /etc/mongod.conf
sudo nano /etc/mongod.conf
# network interfaces
net:
port: 27017
bindIp: 0.0.0.0

configure mongodb only accept remote connection comes from LAN

mongodb has bind ip but it is not so practical due to when new server add, it need shutdown db and add the new server ip into bind ip list and restart db. This is unacceptable because all other servers need to relaunch either.
In almost all deployment, servers machine and db machine are in same LAN. So can mongodb be configured as only accept ranges of ip of [172.16.0.0 - 172.31.255.255], [192.168.0.0 - 192.168.255.255], [10.0.0.0 - 10.255.255.255]?
These 3 ranges ip is LAN ip
The bind_ip configuration value only determines which IP address(es) your MongoDB server is listening to. It does not control access from remote IPs -- that is the job of a firewall.
The address ranges you have listed as requiring remote access are all private IP address space which means these networks are not directly reachable/routable outside your LAN. Assuming you can route traffic between your private networks you should not need to bind to multiple IP addresses.
Given you are allowing access from a broad range of IP addresses, you should also read the Security section of the MongoDB manual (in particular, the Security Checklist and tutorial on enabling Access Control).
bindIp can accept multiple comma separated values. See the "Security considerations" section Here
Other than that you might want to consider configuring your firewall, maybe iptables if it runs on Linux machine.
Hope this helps

Connect to MongoDB sub-path

I'm trying to set up mongodb on webfaction. It works fine, currently running on port 36505 on the server. I can connect to it on the server just fine. But per the firewall, I can't connect to 36505 on the server from the 'net.
I can expose the port by mapping the port to port 80 under a sub-path. So I can make 1.1.1.1:36505 to 1.1.1.1:80/mongodb
My question is how do I actually connect to that? I'm not sure if mongo supports subpaths on port 80 or not...
The problem with what you are doing here is that the WebFaction firewall will prevent direct connections to your MongoDB instance because it isn't speaking straight HTTP and you don't have access to make it use port 80. The simple fix for this is to get a dedicated IP address and then have the support team open a hole in the firewall in order for you to connect directly to the database that way.
Here is an example on the WebFaction community page:
http://community.webfaction.com/questions/3448/using-websockets-with-a-nodejs-custom-app