I'm using Bitnami mongodb instance deployed on Google Compute Engine.
I can connect to that mongodb instance through MongoDB clients from my personal computer using the External IP of MongoDB instance.
My app is an Angular2 app with Node.js backend. Following is the connection string format used in server.js
mongoose.connect('mongodb://<username>:<password>#<External IP>:27017/<dbname>');
When I host my app on localhost it can connect to mongodb instance on the cloud and access data without issues.
But, when the app is deployed on google app engine the app no longer connects to the mongodb.
Following is the error.
MongoError: failed to connect to server [<External IP>:27017] on first connect
Can somebody suggest what is the root cause.
App Engine instances are part of the same private internal network as Compute Engine instances. Your App Engine application can communicate with services hosted on Compute Engine within the same project without going through the public internet. In many cases, your application connects to the service using the instance's internal IP address without needing to assign the database an external, public IP address. If desired, you can assign a public IP address to allow the database to be accessed from outside of Google Cloud Platform.
Also, App Engine applications can connect to databases via public Ip if the database server and firewall are configured properly to accept connections. Your App Engine application connects to the database using the service's public IP address.
By default, mongodb daemon is listening on TCP port 27017. Therefore, you will need to add a firewall rule on Compute Engine firewall for this port and protocol. This can be done using Google Cloud console or using gcloud command tool:
gcloud compute firewall-rules create allow-mongodb --allow tcp:27017
Related
I have created my organisation infrastructure in GCP following the Cloud Foundation Toolkit using the Terraform modules provided by Google.
The following table list the IP ranges for all environments:
Now I am in the process of deploying my application that consists of basically Cloud Run services and a Cloud SQL (Postgres) instance.
The Cloud SQL instance was created with a private IP from the "unallocated" IP range that is reserved for peered services (such as Cloud SQL).
In order to establish connectivity between Cloud Run and Cloud SQL, I have also created the Serverless VPC Connector (ip range 10.1.0.16/28) and configured the Cloud SQL proxy.
When I try to connect to the database from the Cloud Run service I get this error after ~10s:
CloudSQL connection failed. Please see https://cloud.google.com/sql/docs/mysql/connect-run for additional details: Post "https://www.googleapis.com/sql/v1beta4/projects/[my-project]/instances/platform-db/createEphemeral?alt=json&prettyPrint=false": context deadline exceeded
I have granted roles/vpcaccess.user for both the default Cloud Run SA and the one used by the application in the host project.
I have granted roles/compute.networkUser for both SAs in the service project. I also granted roles/cloudsql.client for both SAs.
I have enabled servicenetworking.googleapis.com and vpcaccess.googleapis.com in the service project.
I have run out of ideas and I can't figure out what the issue is.
It seems like a timeout error when Cloud Run tries to create a POST request to the Cloud SQL API. So it seems like the VPC connector (10.1.0.16/28) cannot connect to the Cloud SQL instance (10.0.80.0/20).
Has anyone experienced this issue before?
When you use the Cloud SQL built-in connexion in Cloud Run (but also App Engine and Cloud Function) a connexion similar to Cloud SQL proxy is created. This connexion can be achieved only on a Cloud SQL public IP, even if you have a serverless VPC connector and your database reachable through the VPC.
If you have only a private IP on Cloud SQL, you need to use the private IP to reach the database, not the built-in Cloud SQL connector. More detail in the documentation
I also wrote an article on this
If you are using a private IP, you need to check the docker bridge network's IP range. Here is what the documentation says:
If a client cannot connect to the Cloud SQL instance using private IP, check to see if the client is using any IP in the range 172.17.0.0/16. Connections fail from any IP within the 172.17.0.0/16 range to Cloud SQL instances using private IP. Similarly, Cloud SQL instances created with an IP in that range are unreachable. This range is reserved for the docker bridge network.
To resolve some of the issues, you are experiencing, follow the documentation here and post any error messages you receive, for example, you could try:
Try the gcloud sql connect command to connect to your instance. This command authorizes your IP address for a short time. You can run this command in an environment with Cloud SDK and mysql client installed. You can also run this command in Cloud Shell, which is available in the Google Cloud Console and has Cloud SDK and the mysql client pre-installed.
Temporarily allow all IP addresses to connect to an instance. For IPv4 authorize 0.0.0.0/0 (for IPv6, authorize ::/0. After you have tested this, please make sure you remove it again as it opens up to the world!
Are you using connection pools?
If not, I would create a cache of connections so that when your application needs to link to the database, it can get a temporary connection from the pool. Once the application has finished its operation, the connection returns to the pool again for later use. For this to work correctly, the connection needs to be open and closed efficiently and not waste any resources.
I'd like to know in detail how to connect google compute engine virtual machine instance and app engine.
I've set up a virtual machine instance on Google compute engine, and my Postgres server is running there, following this tutorial: https://cloud.google.com/community/tutorials/setting-up-postgres
I've deployed my flask app under the same project on Google Cloud Platform, creating an app engine instance.
I searched on how to connect compute engine and app engine together, and it seems it should be possible through a VPC connector: connect Google App Engine and Google Compute Engine
This is what my VPC connector looks like:
Serverless VPC access
Name Network Region IP address range Min. throughput Max. throughput
connector-name default europe-west2 10.8.0.0/28 200 300
On my compute engine, I have my VM instance like so:
Name Zone Internal IP External IP
some-name europe-west2-c 10.154.0.2 (nic0) 34.89.113.193
On my flask app, I'm trying to connect to my remote DB like so:
db = PostgresqlExtDatabase(
"some-name", # databse name
user="postgres",
password="some-password",
host="10.154.0.2", # remote host internal ip
port=5432,
)
db.connect()
This is my app.yaml for the vpc access part, I've followed this reference: https://cloud.google.com/appengine/docs/standard/python/connecting-vpc#configuring
vpc_access_connector:
name: projects/some-name/locations/europe-west2/connectors/connector-name
If I understood correctly, if the VPC connector is present, I should just be able to connect using the internal IP address of my VM instance(this case, 10.154.0.2)?
The problem is, when the app is deployed for production, It is still complaining that it cannot connect:
2020-09-26 12:54:51 default[20200926t134815] Is the server running on host "10.154.0.2" and accepting
2020-09-26 12:54:51 default[20200926t134815] TCP/IP connections on port 5432?
If it's connected internally I assume I don't have to add that internal IP to firewall rules, although I did try that as well. As for firewall rules, I have allowed my local machine's IP address so I can connect to the remote Postgres server via PgAdmin.
I've actually tried External IP(34.89.113.193) as well although that doesn't make sense to me.
I'm a bit of a noob on networks and backend stuff in general, any help would be much appreciated.
UPDATED 1
This is my firewall rules:
Direction
Ingress, Egress
Action on match
Allow
Source filters
IP ranges
92.40.176.9/32
78.146.103.141/32
10.154.0.2
Protocols and ports
tcp:5432
Image for reference: Screenshot for the list of firewall rules
It turns out the firewall / postgres configurations were all ok, but because this VPC connector method was on beta, I needed to run:
gcloud beta app deploy
instead of the usual
gcloud app deploy.
This command then updated gcloud Beta Commands and prompted me to enable API:
API [appengine.googleapis.com] not enabled on project [742932836941]. Would you like to enable and retry (this will take a few minutes)? (y/N)?
After enabling this everything worked fine.
Per the information provided seems like both VPC firewall rules and the connector are well configured.
However, based on the messages
2020-09-26 12:54:51 default[20200926t134815] Is the server running on host "10.154.0.2" and accepting
2020-09-26 12:54:51 default[20200926t134815] TCP/IP connections on port 5432?
Seems like the VM or server using 10.154.0.2 is not accepting requests on port 5432 or the port has not been opened, you can use this site to do a port scan.
Based on the guide you followed to create PostgreSQL you are using Ubuntu as OS, therefore I suggest you open the port in ubuntu and see if the issue persists.
Also, i need to connect with the same project hosted app with the cluster.
and access the database.
i don't know which IP i give in the mysql.createconnection in my app.
in my node.js app
var connection = mysql.createConnection({host:"IP",user:"username",password: "password",database:"databasename"});
"How to connect to Cloud SQL" is kind of a broad question, and you haven't given very many details as to your environment, but I'll try to point you in the right direction.
First there are generally 3 ways to connect to Cloud SQL - via Environment connectors, Private IP, Public IP.
Environment Connectors (App Engine & Cloud Functions)
If you are using Google App Engine or Google Functions, you should use the /cloudsql socket provided by the environment. See this page here for examples.
Private IP (Compute or Kubernetes Engine)
To connect via Private IP, your app needs to have access to a VPC. This can be either a Compute VM or a GKE cluster. Then you app can access the "Private IP" for the instance just like it would a local database.
Public IP (Anything with access to the internet)
Finally, you can connect via Public IP. This can be done as long as you have access to the internet, but by default public connections need to be authenticated. This can be done 3 different ways:
Using the Cloud SQL proxy
Using an SSL cert
Whitelisting an IP address
Hope this helps.
I'm trying new GUI of MongoDB called Compass.
However, my MongoDB server is on the Azure virtual machine which has private IP only.
All Azure servers in our subnet are accessible via one server called monitoring Server which has public IP and one haproxy is installed on it.
So how do I access MongoDB from my desktop(centos) using Compass?
For Azure VM, you just can connect it through the public IP or Azure Load Balancer with the NAT rules. Also for MongoDB.
If you can make your monitoring as a router and can redirect your connection to the VM of MongoDB so that you can connect the MongoDB. If not, you cannot connect your MongoDB.
You can take a look at the Azure Load Balancer with NAT rules if you don't want to connect directly to the VM which MongoDB on it.
I´ve deploy my demo app on GAE and works fine with mLab , but when I try to deploy mongodb on GCE (MongoDB (Google Click to Deploy) )the deploy is success but I don´t know how to get te URI to set on my app running on GAE.
I try with internal and external IP but it seems dont work !
Thanks
GAE Standard deployments are sand-boxed. Therefore you can not connect to GCE instances' internal IPs. You can imagine it as two different devices on two different private networks that are not capable to communicate with one another using their internal IPs. However, they can always communicate if one of the devices (GCE instance in this case) has a public IP, and it's private network (firewall) allowed traffic through the port required by the device.
On the other hand, if the GAE deployment is in flex environment, you should be able to connect to the db using the API through internal IPs.
I have tried and succeeded with this flex environment example for both internal and external IP addresses. Like you, I used Cloud Launcher to deploy Mongodb which created GCE instances with public IPs and network tags mongodb and mongodb-db. Then I created a db, username and a password by connecting to the primary db instance through SSH.
To use the internal IP, I just created/modified keys.json file per the example, as follows:
{
"mongoHost": "internal IP address",
"mongoPort": "27017",
"mongoDatabase": "db",
"mongoUser": "username",
"mongoPass": "password"
}
So I didn't have to worry about the URI as the code in server.js took care of it through passing this string:
mongodb://${user}:${pass}#${host}:${port}
But for your demo app, you may have to check the MongoDB official documentation for the standard connection string format URI.
As for using public IPs, I had to create a network firewall rule that allows tcp ingress on port 27017 with target tags identical to the network tags in order to limit access through the port to the MongoDB instances only. Next, I modified the keys.json file as above by replacing the internal IP with the public one.