I am trying to create a linked service in ADF to connect to a MongoDB and I am getting 30 second server timeouts.
I have the connection string and I can connect using Compass - my computer IP address is whitelisted - but I cannot connect through Azure linked service using their MongoDB connector with this connection string.
The Azure IP address ranges for my region have been added to the whitelist as well using the latest set published by Microsoft. I am using an azurehostedingegrationruntime that is in the same Region the MongoDB is hosted in.
Problem is the MongoDB is hosted by a software house and I am not convinced they know what they are doing. SSL is NOT enabled on the MongoDB and they are using the community edition v1.34.1, database is small < 0.75Gb. The MongoDB instance is installed on a Linux box - I was looking at a selfhostedintegrationruntime but that requires a gateway installing on the server that in turn needs the use of a windows server.
If anybody has any experience of connecting to a MongoDB through Azure data factory your help would be appreciated. The only option from the Azure end is the connection string and I know that is correct as I can connect using Compass with it, but it times out when trying to connect using Azure linked service so looks like it cannot see the MongoDB.
Connects ok with the given connection using Compass, just not using Azure even though the Azure IP addresses have been whitelisted.
Solved by the software house, so they do actually know what they are doing.
Don't need to use SelfHostedIntegrateionRuntime, the AzureHostedIntegrationRuntime works just fine. Also no need to whitelist the Azure IPs - these are subject to revision anyway.
", but on the instance firewall, I have the option to allow the exact service and this should cover any future ip changes. For now, I have allowed access only for the "
Hope this makes sense.
Related
I created a Compute Engine Instance on which I am hosting my MongoDB server.
I also have a nodeJS server which currently hosted in APP Engine of the same project and in the same region.
Now I want to connect my MongoDB database with the AppEngine server.
How can I do this?
Please Guide me.
Thanks in advance.
So main question is how you're atttempting to connect from GAE to MongoDB, which is not included in your question...
This aside, you'll need the connection string, as per MongoDB documentation [1], and this doc shows how to get it [2].
Since you're running both GAE and the GCE instance running MongoDB in the same project you can use the internal IP address and you can remove the external IP address from the GCE instance to remove a potential security issue with people accessing MongoDB directly.
The connection string would be:
mongodb://[username:password#]GCE_INTERAL_IP[:port1][/[defaultauthdb][?options]]
Replace GCE_INTERAL_IP with the actual internal IP of the GCE instance running MongoDB. You can find this in the GCP console.
https://docs.mongodb.com/guides/server/drivers/#obtain-your-mongodb-connection-string
https://docs.mongodb.com/manual/reference/connection-string/#mongodb-uri
I'm trying to connect to Google Cloud SQL from my machine (Ubuntu) using this command:
mysql --host='Public IP' --user='' --password
However, I'm getting this error:
ERROR 2003 (HY000): Can't connect to MySQL server on 'Public IP' (110)
I need any help resolving my issue.
First you need to let the Cloud SQL instance which IP addresses it can accept. You can do that without SSL by following the instructions here. However, to be more secure, I would recommend you using SSL. More info on that here.
Probably the easiest way to securely connect from your local machine to a public ip of a cloud SQL instance is to download and use the proxy, following the instructions here:
https://cloud.google.com/sql/docs/mysql/connect-admin-proxy
What you have to do is add a network to the public ip section, under the connections tab after selecting your Cloud SQL instance.
See Cloud SQL Connections Tab here
So, for the name input you put firstname-lastname kind of thing to denote whose ip it is. Then input your IP address 1.2.3.4/32 into the network input.
After doing so and saving you will be able to connect.
Yes, you can add SSL and use certificates. That is all best practice and what should be done for a production stack. But if this is just getting off the ground and in rapid development, that's all you need to do in the beginning.
I am trying to test a connection using Cloud Data Fusion to connect to an on-prem SQL Server. Our GCP Project does not use the default network but rather a custom VPC.
It's important to note that security is very important as this database contains healthcare data.
We currently have App Engine Flex code that uses pymssql to query the database on this SQL Server through the VPC, and want to test using Data Fusion.
I have copied and added the generated data fusion service account to IAM with role Cloud Data Fusion API Service Agent.
I have configured in Data Fusion
system.profile.properties.network = <VPC name>
I have verified that the username and password for authentication to SQL Server are valid for the database.
THE VPC network allows port 22 and 1433.
At this point I am just trying to get a successful connection to query a table in the database.
Here is the error message I get:
Connect timed out. Verify the connection properties. Make sure that an instance of SQL Server is running on the host and accepting TCP/IP connections at the port. Make sure that TCP connections to the port are not blocked by a firewall.
Thoughts?
It looks like you can access SQL Server from GCP (via AppEngine Flex Py code) but having trouble connecting through Data Fusion. It also looks like authentication doesnt seem to be a problem (as the error message is around connection).
One area I can think of is to try Cloud Data Fusion private IP, that allows you to securely connect to your VPC from Cloud Data Fusion. This is a new feature and you will need Data Fusion product team support to leverage this feature.
I installed oracle-jdbc thin driver to connect with On prem oracle DB but when I test the connection I get network adapter error
I tried the changing the host but still same
When running the pipeline from GCS-BQ I getting network port error. Can we change the VPC the pipeline is running on ?
Regarding the oracle db connection error, is the db available on the public network for connection? Currently wrangler service in Cloud Data Fusion cannot talk to the on-prem db over a private connection and we are actively working towards it.
However if the db is available on the public network then it seems like the issue with the oracle db configurations. Can you please take a look at this answer and see if it helps - Oracle SQL Developer: Failure - Test failed: The Network Adapter could not establish the connection?
Also are you able to connect to the oracle db through some other query tool such as SqlWorkbench?
Breaking down your question:
1. Connecting to on-prem databases
It is possible nowadays to connect to on-premise databases. Make sure you created an interconnect between the on-prem network and the network used by Data Fusion instance and make sure you applied the right firewall rules (seems you are getting firewall issues by the logs). I suggest trying to connect directly in the database first to confirm that the network setup works.
2. Change network configurations on the Data Fusion job.
You can specify parameters for your job. There are options to change the network and subnetwork that the job will be executed under Configure > Compute config > Customize option. If you use shared VPC you can also specify the Host project.
Recently i decided to move my database from inside my server machine to the MongoDB Atlas service.
Atlas provides a IP Whitelist feature which i use to remotely connect to the database cluster.
Should i plug my server application to Atlas using this feature?
What happens if my server IP changes? Is it secure?
For a general information on how to connect to an Atlas deployment, please see Connect to a Cluster
For connecting using a driver, please see Connect via Driver. There is an extensive list of examples using all of the officially-supported drivers.
As mentioned in the Prerequisites section, you need to use SSL/TLS and IP whitelist to connect to your Atlas instance. This whitelist would need to be updated should your application server's IP changes.
The whitelist provides an additional security layer in addition to your username/password, since this list will essentially reject any connection not originating from a known IP address. It is strongly recommended to utilize this whitelist, and arguably the effort required to maintain the whitelist is comparably small to the security advantages it provides.