We have an airflow task that adds data to the mongodb server.
We can connect to the mongodb server only behind IP Access or VPC Peering.
We are having issues with VPC Peering, so we thought we can just enable direct IP access between the airflow workers and the mongodb server
Has anyone done that?
If not, do you have another suggestion?
Related
How to configure IP access in mongodb atlas cluster though api or shell command ?
It's very time consuming to configure IP access to mongodb atlas cluster though mongodb GUI web platform each time.
It'd be very easy if we have some option to add IP to access mongodb atlas cluster though some api or some mongo shell command, so how to achieve that?
mongodb atlas network access section
The API documentation - for managing the access list: https://docs.atlas.mongodb.com/reference/api/access-lists/
I have a Mongodb Atlas database which is set up with VPC peering to a VPC in AWS. This works find and I'm able to access it from inside the VPC. I was, however, hoping to provide a jumpbox so that developers could use an SSH tunnel to connect to the Atlas database from their workstations outside of the VPC.
Developer workstation --> SSH Tunnel to box in VPC --> Atlas
I'm having trouble with that, however because I'm not sure what tunnel I need to set up. It looks to me like Mongo connects by looking up replica information in a DNS seed list (mongodb+srv://). So it isn't as simple as doing
ssh user#jumpbox -L 27017:env.somehost.mongodb.net:27017
Is there a way to enable direct connections on Atlas so that I can enable developers to access this database through an SSH tunnel?
For a replica set connection this isn't going to work with just MongoDB and a driver, but you can try running a proxy like https://github.com/coinbase/mongobetween on the jumpbox.
For standalone deployments you can connect through tunnels since the driver uses the address you supply and that's the end of it. Use directConnection URI option to force a standalone connection to a node of any deployment. While this allows you to connect to any node, you have to connect to the right node for replica sets (you can't write to secondaries) so this approach has limited utility for replica set deployments.
For mongos deployments that are not on Atlas the standalone behavior applies. With Atlas there are SRV records published which the driver follows, therefore for the tunneling purposes an Atlas sharded cluster behaves like a replica set and you can't trivially proxy connections to it. mongobetween may also work in this case.
We have a dedicated M10 cluster in Mongodb Atlas, on which I have created a peering connection with AWS to incorporate security using VPC. I have followed this Mongodb document for configuring peering connection between AWS and cluster.
https://docs.atlas.mongodb.com/security-vpc-peering/
The peering connection is created successfully and is active now. But the thing is, I am unable to connect to cluster without whitelisting my IP. When I try to connect without whitelisting the IP, it gives below error:
Something went wrong MongooseServerSelectionError: Could not connect
to any servers in your MongoDB Atlas cluster. One common reason is
that you're trying to access the database from an IP that isn't
whitelisted. Make sure your current IP address is on your Atlas
cluster's IP whitelist:
https://docs.atlas.mongodb.com/security-whitelist/
While after whitelisting the IP, I am able to connect to cluster successfully from local environment.
What do I need to access a cluster within VPC using application? I can not use the option IP whitelisting as every user's IP can not be whitelisted.
I have already whitelisted CIDR block as mentioned by the above documentation.
IP whitelisting is separate from peering. Peering determines the network, whitelisting determines who on the network is allowed access.
If you want to allow access from anything that has physical connectivity to the database, whitelist the entire world (0.0.0.0/0).
I've set up the peering connection between MongoDB Atlas and Googles "default" VPC and the connection is labeled as "active" on both ends.
The ip range of the vpc is whitelisted in MongoDB Atlas.
But my node hosted in google-app-engine still gets timed out when accessing the MongoDB.
I use the connection url of mongodb atlas for peered connection in the form of (notice the "-pri"):
mongodb+srv://<username>:<password>#<my-cluster>-pri.rthhs.mongodb.net/<dbname>?retryWrites=true&w=majority
Which part am i missing to establich the connection? Do i need a google vpc connector?
Thanks for any help!
First of all, make sure you are running M10-Cluster or above!!! VPC-peering is not available for M0/M2/M5...
And YES you do need that connector! All "Serverless"-Services from Gcloud (like GAE in standard environment) need it.
create a connector in the same region as your GAE-App following these instructions. You can find the current region of your GAE-App with gcloud app describe
your app.yaml has to point to that connector like this
app.yaml
runtime: nodejs10
vpc_access_connector:
name: projects/GCLOUD_PROJECT_ID/locations/REGION_WHERE_GAE_RUNS/connectors/NAME_YOU_ENTERED_IN_STEP_1
Go to your Atlas project, navigate to Network Access and whitelist the IP-range you set for the connector in Step 1
You may also need to whitelist the IP-range from Step 1 for the VPC-Network. You can do that in GCP by navigating to VPC-Network -> Firewall
If you have questions about how to setup the VPC-Peering between Atlas and Gcloud try this tutorial. They do it for Kubernetes-Engine (no connector needed). But adding my steps from above will hopefully do the trick.
Try Cannot connect to Mongo Atlas using VPC peering from GCP cluster and MongoDB and Google Cloud Functions VPC Peering?.
First step I suggest identifying whether you have physical connectivity (and so need to fix ip whitelist) or don't have connectivity (and need to fix peering configuration).
I'm trying to setup an VPC Peering from my MongoDB Atlas Cluster to my Kubernetes EKS Cluster on AWS. The Peering is established successfully but i get no connection to the cluster on my pod's.
The peering is setup.
The default entry for the whitelist ist added as well. Once the connection works i will replace it with a security Group.
The peering on AWS is accepted and "DNS resolution from requester VPC to private IP" is enabled.
The route as been added to the Public Route Table of the K8S Cluster.
When i connect to a pod and try to establish a connection with the following command:
# mongo "mongodb://x.mongodb.net:27017,y.mongodb.net:27017,z.mongodb.net:27017/test?replicaSet=Cluster0-shard-0" --ssl --authenticationDatabase admin --username JackBauer
I get "CONNECT_ERROR" for every endpoint.
What am I missing?
NOTE:
I've just created a new paid cluster and the VPC is working perfectly. Might this feature be limited to paid clusters only?
Well... as the documentation states:
You cannot configure Set up a Network Peering Connection on M0 Free
Tier or M2/M5 shared clusters.
Peering is not working on shared Cluster. Which makes, after i think about it, totally sense.