I am trying to use MongoDB change stream using Kafka. I chose AWS MSK since my entire infrastructure is within AWS and it will be easy to integrate with other AWS services.
I created an AWS MSK cluster within the VPC and created an AWS MSK connect using mongodb Kafka connector. I am getting the following error (Exception opening socket, connect timed out)
To test whether it has to do with VPC, I gave the public access to MongoDB by whitelisting 0.0.0.0, and then the connector is working fine. So I believe it has something to do with the VPC.
I tried multiple times with AWS MSK with VPC with public subnets, NAT gateway, etc, and it is still not working. Am getting the same error. I believe the AWS MSK connecter uses the same VPC configuration as AWS MSK since there's no separate configuration for AWS MSK Connect.
The answer is to create a Private Endpoint in Atlas then a VPC endpoint in AWS.
There is a section in this guide that explains how to set up the private endpoint: https://www.mongodb.com/developer/languages/java/integrating-mongodb-amazon-apache-kafka/?bpass=true#create-mongodb-atlas-cluster
For me even though we had a peering connection set up between AWS and Atlas with all of the correct routes etc, we still had to go via this private endpoint option, it's a limitation of the MSK I believe
Related
When utilizing AWS' connection string to connect to my Amazon Document DB instance in Robo3T I'm getting a 'Connection Timed Out' error.
Error connecting to ....cluster-....us-east-1.docdb.amazonaws.com:27017 (123.45.67.89:27017) :: caused by :: Connection timed out :
AWS does not currently support connecting to Document DB from public endpoints when not using SSH.
Connecting from Public Endpoints
You are trying to connect to an Amazon DocumentDB cluster directly
from your laptop or local development machine.
Trying to connect to an Amazon DocumentDB cluster directly from a
public endpoint, such as your laptop or local development machine,
will fail. Amazon DocumentDB is virtual private cloud (VPC)-only and
does not currently support public endpoints. Thus, you can't connect
directly to your Amazon DocumentDB cluster from your laptop or local
development environment outside of your VPC.
To connect to an Amazon DocumentDB cluster from outside an Amazon VPC,
you can use an SSH tunnel. For more information, see Connecting to an
Amazon DocumentDB Cluster from Outside an Amazon VPC. Additionally, if
your development environment is in a different Amazon VPC, you can
also use VPC Peering and connect to your Amazon DocumentDB cluster
from another Amazon VPC in the same region or a different region.
More details here: https://docs.aws.amazon.com/documentdb/latest/developerguide/troubleshooting.connecting.html#troubleshooting.cannot-connect.public-endpoints
So, I'm trying to set up a MongoDB PrivateLink to my AWS EKS but it seems like my pod is not managing to connect.
I followed this guide https://aws.amazon.com/blogs/apn/connecting-applications-securely-to-a-mongodb-atlas-data-plane-with-aws-privatelink/
and set up the following materials:
Created a new VPC with 3 availability zones
Created EKS and attached it to the VPC (the EKS has Private + Public networking enabled)
I used the private subnets for composing the VPC under the MongoDB privateLink and used the VPC name of my newly created VPC
Afterwards i ran the command that mongodb shows and waited for it to create.
It shows "Available" under "Endpoint Status" and "Ready for connection requests" under endpoint service status
I use my correct username/password and the mongo URI is correct (I whitelisted my computer and tested and it worked)
I'm rather new to the AWS - PrivateLink setup and don't seem to get it working. Should i perhaps use the public subnet ids? What could be the issue?
I've set up the peering connection between MongoDB Atlas and Googles "default" VPC and the connection is labeled as "active" on both ends.
The ip range of the vpc is whitelisted in MongoDB Atlas.
But my node hosted in google-app-engine still gets timed out when accessing the MongoDB.
I use the connection url of mongodb atlas for peered connection in the form of (notice the "-pri"):
mongodb+srv://<username>:<password>#<my-cluster>-pri.rthhs.mongodb.net/<dbname>?retryWrites=true&w=majority
Which part am i missing to establich the connection? Do i need a google vpc connector?
Thanks for any help!
First of all, make sure you are running M10-Cluster or above!!! VPC-peering is not available for M0/M2/M5...
And YES you do need that connector! All "Serverless"-Services from Gcloud (like GAE in standard environment) need it.
create a connector in the same region as your GAE-App following these instructions. You can find the current region of your GAE-App with gcloud app describe
your app.yaml has to point to that connector like this
app.yaml
runtime: nodejs10
vpc_access_connector:
name: projects/GCLOUD_PROJECT_ID/locations/REGION_WHERE_GAE_RUNS/connectors/NAME_YOU_ENTERED_IN_STEP_1
Go to your Atlas project, navigate to Network Access and whitelist the IP-range you set for the connector in Step 1
You may also need to whitelist the IP-range from Step 1 for the VPC-Network. You can do that in GCP by navigating to VPC-Network -> Firewall
If you have questions about how to setup the VPC-Peering between Atlas and Gcloud try this tutorial. They do it for Kubernetes-Engine (no connector needed). But adding my steps from above will hopefully do the trick.
Try Cannot connect to Mongo Atlas using VPC peering from GCP cluster and MongoDB and Google Cloud Functions VPC Peering?.
First step I suggest identifying whether you have physical connectivity (and so need to fix ip whitelist) or don't have connectivity (and need to fix peering configuration).
I'm trying to setup an VPC Peering from my MongoDB Atlas Cluster to my Kubernetes EKS Cluster on AWS. The Peering is established successfully but i get no connection to the cluster on my pod's.
The peering is setup.
The default entry for the whitelist ist added as well. Once the connection works i will replace it with a security Group.
The peering on AWS is accepted and "DNS resolution from requester VPC to private IP" is enabled.
The route as been added to the Public Route Table of the K8S Cluster.
When i connect to a pod and try to establish a connection with the following command:
# mongo "mongodb://x.mongodb.net:27017,y.mongodb.net:27017,z.mongodb.net:27017/test?replicaSet=Cluster0-shard-0" --ssl --authenticationDatabase admin --username JackBauer
I get "CONNECT_ERROR" for every endpoint.
What am I missing?
NOTE:
I've just created a new paid cluster and the VPC is working perfectly. Might this feature be limited to paid clusters only?
Well... as the documentation states:
You cannot configure Set up a Network Peering Connection on M0 Free
Tier or M2/M5 shared clusters.
Peering is not working on shared Cluster. Which makes, after i think about it, totally sense.
I want to connect my Lambda function to Mongo Atlas. It was all working fine but I needed to move my function inside a VPC so I could use redis.
Now I cannot connect to my database.
I looked at the security group on the VPC and added the mongo db port but with no joy
Also - the mongo IP Whitelist is as follows for now
0.0.0.0/0 (includes your current IP address)
Is there anything else I should try?
Thank you
I needed to move my function inside a VPC so I could use redis.
If you are:
Using dedicated MongoDB Atlas instances (i.e. not shared instances M0, M2 and M5 clusters).
And, the MongoDB Atlas deployment is hosted on AWS
Then, you could follow this instruction to set up VPC peering connection.
Please note that MongoDB Atlas supports VPC peering with other AWS VPCs in the same region. AWS does not support cross-region VPC peering. For multi-region clusters, you must create VPC peering connections per-region.
See also the tutorial shown on: Introducing VPC peering to MongoDB Atlas
struggled for days
using this tutorial
https://www.mongodb.com/blog/post/introducing-vpc-peering-for-mongodb-atlas
but it work in the end, when i found a missing step in the tutorial; when i used default VPC.
When crating custom VPC, rout table, and subnet; the subnet was what got me... auto assign IP needs to be enabled on the subnet;
PS you need at least two subnets for Lambda, so create one more if you haven't