I'm currently using helm charts from bitnami (Here) and trying to use a Cloud SQL instance in GCP. No matter what connection string I build, I get various connection errors connecting to the database. This causes the initial pod startup to fail, and never fully initialize.
It does not appear that spring-cloud-dataflow comes bundled with spring-cloud-gcp so it cannot be configured via Instance Name, so I assume we need to use the Private IP address, which fails every time.
For the below tests, a Cloud SQL MySQL instance was created, and a Cloud SQL Postgresql instance was created.
Chart configuration 1 for MySQL instance: (username and password omitted for brevity)
mariadb.enabled=false
externalDatabase.dataflow.url=jdbc:mysql://<privateip>/dataflow?useSSL=false
externalDatabase.dataflow.username=root
externalDatabase.driver=org.mariadb.jdbc.Driver
Results in (using port 0 or port 3306):
java.sql.SQLNonTransientConnectionException: Could not connect to address=(host=<private IP>)(port=0)(type=master) : Could not connect to <private IP>:0 : unexpected end of stream, read 0 bytes from 4 (socket was closed by server)
Caused by: java.io.EOFException: unexpected end of stream, read 0 bytes from 4 (socket was closed by server)
Chart configuration 2 (postgresql instance):
mariadb.enabled=false
externalDatabase.dataflow.url=jdbc:postgresql://<Private IP>:5432/
externalDatabase.dataflow.username=postgres
externalDatabase.driver=org.postgresql.Driver
Results in:
org.postgresql.util.PSQLException: The connection attempt failed.
Caused by: java.net.SocketException: Connection reset
I've verified that I can reach the databases via jdbc through Squirrel SQL (for postgresql) and MySQL workbench.
Will this never work unless spring-cloud-dataflow ships with spring-cloud-gcp?
You'll need to use the Cloud SQL Proxy or the Cloud SQL Java Connector.
There are a couple of answers here that show how to do that:
Connecting to Cloud SQL from Dataflow Job
Is it safe to connect DataFlow job to Cloud MySQL with proxy (by os.system)
Also, if your Cloud SQL instance is private IP only, you'll need to make sure your Data Flow deployment is in the same VPC as your Cloud SQL instance. For Public IP instances, using the Java connector or the proxy alone should be fine.
Related
I am new to the next JS platform but have prior knowledge on react JS, I have a problem when trying to host the created full stack next JS application on AWS using AWS Amplify and for database created a RDS instance with PostgreSQL setup I am getting error status code of 500 (Internal Server Error) but running in the local environment seems fine and the code is functional in the local environment.
The link to code on Git-Hub click here
On the hosted application following error gets console
Client side exception has occurred see here for more info: https://nextjs.org/docs/messages/client-side-exception-occurred
when checked with the logs of the RDS instance, The following log gets issued in the Viewing-Log error PostgreSQL
`2023-01-29 17:00:44 UTC:49.37.216.33(65214):Master name #DB Name:[8768]:LOG: could not receive data from client: Connection reset by peer`
I'm trying to connect to Db2 but I get the following error message
[jcc][t4][2034][11148][4.25.23] Execution failed due to a distribution protocol error that caused deallocation of the conversation.
A DRDA Data Stream Syntax Error was detected. Reason: 0x3. ERRORCODE=-4499, SQLSTATE=58009
What does that mean, and how do I get connected?
Well the short answer is that you might just need to add the sslConnection=true JDBC parameter
I.e. your JDBC URL needs to look something like this
db2://db2whoc-abcdefg.services.eu-de.bluemix.net:50001/BLUDB:sslConnection=true;
Depending on what tool you are using, you might edit the URL directly, or add as an extra JDBC property
Now the above only works if you are using a reasonably recent JDBC driver.
The DigiCertGlobalRootCA.crt used by e.g. Db2 on Cloud, Db2 Warehouse and Db2 Warehouse on Cloud was not bundled in older drivers. You can cross-reference the driver version (e.g. 4.25.23 in the question) against the drivers listed here. DB2 JDBC Driver Versions and Downloads and upgrade if not on the latest release
If your Db2 server is using a different certificate (for example a self-signed certificate) then you would need to download the certificate, and set the JDBC property sslCertLocation to point to it's location, or you can import the certificate into a local store, and use sslTrustStoreLocation and sslTrustStorePassword and a import the cert into their local certificate store.
All these JDBC properties are described on the
Common IBM Data Server Driver for JDBC and SQLJ properties for all supported database products page
Here's my scenario. I have set up an instance of Postgres DB running in the Google SQL cloud. It's up and running and if I whitelist my local IP, I can connect directly with no issue.
I then have deployed a docker container (postGrest) which is a web server that connects to the postgres DB.
When I configured this on Google Cloud Run, it did have a drop-down option where I could specify DB connectivity and it says that, behind the scenes, it configures Cloud SQL Proxy for this connection.
The container allows environment variables to be passed in to specify which server, etc.
One required parameter is the DB_URI to the postgred instance. When running locally, it looks like this:
postgres://authenticator:mysecretpassword#localhost:5432/testdb
When I tried to configure this on the cloud version I tried using the IP 127.0.0.1 (The google cloud SQL proxy documentation says this is how you connect via the proxy). This didn't work.
I then tried using the public-ip assigned to the postgres DB....this didn't work either.
Does anyone know how to specify the correct connection string using this DB_URI format?
I am just going to add this as an answer rather than a comment since it's easier for readability and perhaps helping other users. Please don't feel encouraged to change the accepted answer.
By following the documentation provided by the OP, the final pattern for the URI became:
# Breaking lines for improved readability
POSTGRESS_URI=postgresql:///dbname
?host=/cloudsql/myprojectid:region:myinstanceid
&user=username
&password=password
&sslmode=disable
* dont forget to prefix the unix socket path with /cloudsql/
Any parameters can be used normally as in the example of sslmode.
Also, be aware that two important things are mentioned in the Cloud SQL documentation:
Note: The PostgreSQL standard requires a .s.PGSQL.5432 suffix in the socket path. Some libraries apply this suffix automatically, but others require you to specify the socket path as follows: /cloudsql/INSTANCE_CONNECTION_NAME/.s.PGSQL.5432.
In my case, the program I am using already adds as a suffix .s.PGSQL.5432, so I didn't need to add it to my URI.
Warning: Linux based operating systems have a maximum socket path length of 107 characters. If the total length of the path exceeds this length, you will not be able to connect with a socket from Cloud Run (fully managed).
Cloud Run does not support connecting to Cloud SQL using IP addresses. This means 127.0.0.1 will not work. Cloud Run uses Unix Sockets. You must use a connection string.
The Cloud SQL Proxy connection string looks like this:
myprojectid:region:myinstanceid
You can get the instanceid from the Cloud SQL Instance Details page in the console.
You will also need to add permissions to your Cloud Run service account to access Cloud SQL. You will need at least Cloud SQL Client.
It seems that some postgres client libraries don't support putting user and password parameters in the URL query params. I had problems with pq for Node.js and would get the error "no PostgreSQL user name specified in startup packet".
An alternative way of writing the connection string is as follows:
Breaking lines for readability:
postgres://db_user:db_password#
%2Fcloudsql%2Fproj_name%3Aus-central1%3Asql_instance_name
/db_name&sslmode=disable
It's like a normal TCP connection string, but you put the path to the Unix socket as the host, encoding the / and : characters. If the first character of the hostname is /, then the hostname will be treated as a filesystem path.
I am not able to use the federated query capability from Google BigQuery to Google Cloud SQL Postgres. Google announced this federated query capability for BigQuery recently in beta state.
I use EXTERNAL_QUERY statement like described in documentation but am not able to connect to my Cloud SQL instance. For example with query
SELECT * FROM EXTERNAL_QUERY('my-project.europe-north1.my-connection', 'SELECT * FROM mytable;');
or
SELECT * FROM EXTERNAL_QUERY("my-project.europe-north1.pg1", "SELECT * FROM INFORMATION_SCHEMA.TABLES;");
I receive this error :
Invalid table-valued function EXTERNAL_QUERY Connection to PostgreSQL server failed: server closed the connection unexpectedly This probably means the server terminated abnormally before or while processing the request.
Sometimes the error is this:
Error encountered during execution. Retrying may solve the problem.
I have followed the instructions on page https://cloud.google.com/bigquery/docs/cloud-sql-federated-queries and enabled BigQuery Connection API. Some documents use different quotations for EXTERNAL_QUERY (“ or ‘ or ‘’’) but all the variants end with same result.
I cannot see any errors in stackdriver postgres logs. How could I correct this connectivity error? Any suggestions how to debug it further?
Just adding another possibility for people using Private IP only Cloud SQL instances. I've just encountered that and was wondering why it was still not working after making sure everything else looked right. According to the docs (as of 2021-11-13): "BigQuery Cloud SQL federation only supports Cloud SQL instances with public IP connectivity. Please configure public IP connectivity for your Cloud SQL instance."
I just tried and it works, as far as the bigquery query runs in EU (as of today 6 October it works).
My example:
SELECT * FROM EXTERNAL_QUERY("projects/xxxxx-xxxxxx/locations/europe-west1/connections/xxxxxx", "SELECT * FROM data.datos_ingresos_netos")
Just substitute the first xxxxs with your projectid and the last ones with the name you gave to the connection in The bigquery interface (not cloudsql info, that goes into the query)
Unfortunately BigQuery federated queries to Cloud SQL work currently only in US regions (2019 September). The documents (https://cloud.google.com/bigquery/docs/cloud-sql-federated-queries) say it should work also in other regions but this is not the case.
I tested the setup from original question multiple times in EU and europe-north1 but was not able to get it working. When I changed the setup to US or us-central1 it works!
Federated queries to Cloud SQL are in preview so the feature is evolving. Let's hope Google gets this working in other regions soon.
The BigQuery dataset and the Cloud SQL instance must be in the same region, or same location if the dataset is in a multi-region location such as US and EU.
Double check this according to Known issues listed.
server closed the connection unexpectedly This probably means the server terminated abnormally before or while processing the request.
This message usually means that you will find more information in the logs of the remote database server. No useful information could be sent to the "client" server because the connection disappeared so there was no way to send it. So you have look in the remote server.
I am following the tutorial on how to set up and connect a PostgreSQL server on AWS found HERE
When I try to sign in on workbench, I get this message:
At first I thought it was because my DB instance was not available yet, so I waited until it finished backing up. This did not seem to work as I am still getting this message. I will appreciate assistance on this.
Have you created a secutiry group and allow databases connection port?
From the docs:
VPC Security Group(s): Select Create New Security Group. This will
create a security group that will allow connection from the IP address
of the device you are currently using, to the database created.