Has anyone managed to get Debezium to work over AWS RDS Postgres with rds.force_ssl turned on in the parameter group?
The connector seems to work for a bit, and then we begin to receive errors like Database connection failed when writing to copy and Exception thrown while calling task.commit().
I have scoured the web searching for this issue, and I see many people encountered it, and many Jira issues opened about it.
The response generally is "Check your network configuration" or "Disable your SSL".
I just can't get it to work for some reason, and obviously disabling encryption in transit is not possible in production use cases (at least in ours).
I would appreciate any kind of help or insight into how to solve this!
Related
We have been running a service using NestJS and TypeORM on fully managed CloudRun without issues for several months. Yesterday PM we started getting Improper path /cloudsql/{SQL_CONNECTION_NAME} to connect to Postgres Cloud SQL instance "{SQL_CONNECTION_NAME}" errors in our logs.
We didn't make any server/SQL changes around this timestamp. Currently there is no impact to the service so we are not sure if this is a serious issue.
This error is not from our code, and our third party modules shouldn't know if we use Cloud SQL, so I have no idea where this errors come from.
My assumption is Cloud SQL Proxy or any SQL client used in Cloud Run is making this error. We use --add-cloudsql-instances flag when deploying with "gcloud run deploy" CLI command.
Link to the issue here
This log was recently added in the Cloud Run data path to provide more context for debugging CloudSQL connectivity issues. However, the original logic was overly aggressive, emitting this message even for properly working CloudSQL connections. Your application is working correctly and should not receive this warning.
Thank you for reporting this issue. The fix is ready and should roll out soon. You should not see this message anymore after the fix is out.
I wrote a simple finance tracking application in 2017 that uses a Heroku backend with a Postgres db. The application suddenly stopped working, and I traced the problem to the database.
I was unable to connect to the database, seeing this error:
psql: FATAL: too many connections for role
I thought maybe the app had a connection leak so I shut the frontend down (Im the only one that uses it) and reset all the db connections. I was then able to login to the db, and noticed all these strange roles (hundreds?) that I dont recognize.
When I logged out of psql, I tried logging back in and again was denied with the "too many connections" error. The only way I can log back in is if I kill all connections again and immediately login. If I wait 2-3 minutes the error comes back. I don't think my heroku app is establishing all these connections with the db, because Im tailing the logs and it's just sitting there.
Does anyone have any theories about what might be going on here? Have I been hacked maybe? How would you debug this further, and how might I fix the problem?
Thanks!
The database has obviously been hacked.
Shut it down and delete it right away.
Restore to a different cluster from a known good backup.
From now on, choose good passwords and use a restrictive pg_hba.conf that for example doesn't allow remote access for superusers.
Never, ever, operate your application with a superuser.
Examine your application for SQL injection vulnerabilities.
this may be because of a bot(made by hackers) that is scanning the internet and trying CVE exploit (N-day exploit) to see if it is vulnerable, and then launching that type of attack or it may be because someone on the VNAT with you trying to something weird, but one thing for sure it is a bot because you can not launch that many connections by hand.
I've tried, failed, deleted the database and tried again 7 times now and I get this error each time. I'm on the lite plan and taking the IBM Data Science Certification course and I can't get past this part. Any assistance would be greatly appreciated.
Deleting database (can only have one in lite plan I believe) retried several times
I just verified that I am able to create a fully working Lite instance on my end. Is it possible that it's a networking issue on your end? Was that the full error message? It seems to be cut off. In what region and datacenter are you trying to create the service instance?
This exception is being thrown by the PostgreSQL 8.3.7 server to my application.
Does anyone know what this error means and what I can do about it?
ERROR: cached plan must not change result type
STATEMENT: select code,is_deprecated from country where code=$1
I figured out what was causing this error.
My application opened a database connection and prepared a SELECT statement for execution.
Meanwhile, another script was modifying the database table, changing the data type of one of the columns being returned in the above SELECT statement.
I resolved this by restarting the application after the database table was modified. This reset the database connection, allowing the prepared statement to execute without errors.
I'm adding this answer for anyone landing here by googling ERROR: cached plan must not change result type when trying to solve the problem in the context of a Java / JDBC application.
I was able to reliably reproduce the error by running schema upgrades (i.e. DDL statements) while my back-end app that used the DB was running. If the app was querying a table that had been changed by the schema upgrade (i.e. the app ran queries before and after the upgrade on a changed table) - the postgres driver would return this error because apparently it does caching of some schema details.
You can avoid the problem by configuring your pgjdbc driver with autosave=conservative. With this option, the driver will be able to flush whatever details it is caching and you shouldn't have to bounce your server or flush your connection pool or whatever workaround you may have come up with.
Reproduced on Postgres 9.6 (AWS RDS) and my initial testing seems to indicate the problem is completely resolved with this option.
Documentation: https://jdbc.postgresql.org/documentation/head/connect.html#connection-parameters
You can look at the pgjdbc Github issue 451 for more details and history of the issue.
JRuby ActiveRecords users see this: https://github.com/jruby/activerecord-jdbc-adapter/blob/master/lib/arjdbc/postgresql/connection_methods.rb#L60
Note on performance:
As per the reported performance issues in the above link - you should do some performance / load / soak testing of your application before switching this on blindly.
On doing performance testing on my own app running on an AWS RDS Postgres 10 instance, enabling the conservative setting does result in extra CPU usage on the database server. It wasn't much though, I could only even see the autosave functionality show up as using a measurable amount of CPU after I'd tuned every single query my load test was using and started pushing the load test hard.
For us, we were facing similar issue. Our application works on multiple schema. Whenever we were doing schema changes, this issue started occruding.
Setting up prepareThreshold=0 parameter inside JDBC parameter disables statement caching at database level. This solved it for us.
I got this error, I manually ran the failing select query and it fixed the error.
I’m working on a experiment regarding to a course I’m taking about tuning DB2. I’m using the EC2 from Amazon (aws) to conduct the experiment.
My problem is, however, that I have to test a non-compression against row-compression in DB2 and to do that I’ve created a bsh file that run those experiments. But when I reach to my compression part I get the error ”Transaction log is full”; and no matter how low I set the inserts for it is complaining about my transaction log.
I’ve scouted Google for a day now trying to find some way to flush / clear the log or just get rit of it, i don’t need it. I’ve tried to increase the size but nothing has helped.
Please, I hope someone has an answer to solve this frustrating problem
Thanks
- Mestika
There is no need to "clear the log" in DB2. When a transaction is rolled back, DB2 releases the log space used by the transaction.
If you've increased the log size and it has not helped, please post more information about what you're trying to do.
No need of restarting. Just try to force the applications using DB2 force applications all.
Increase the Actie Log File Size and try to force application connections and terminate the connections.
Try to run the job now.
db2 force applications all
db2 update db cfg for sample using logfilsiz 5125
db2 force applications all
db2 terminate
db2 connect to sample
Run your job and monitor.
Just restart the instance, it would release the pending logs and you should be fine