I wrote a simple finance tracking application in 2017 that uses a Heroku backend with a Postgres db. The application suddenly stopped working, and I traced the problem to the database.
I was unable to connect to the database, seeing this error:
psql: FATAL: too many connections for role
I thought maybe the app had a connection leak so I shut the frontend down (Im the only one that uses it) and reset all the db connections. I was then able to login to the db, and noticed all these strange roles (hundreds?) that I dont recognize.
When I logged out of psql, I tried logging back in and again was denied with the "too many connections" error. The only way I can log back in is if I kill all connections again and immediately login. If I wait 2-3 minutes the error comes back. I don't think my heroku app is establishing all these connections with the db, because Im tailing the logs and it's just sitting there.
Does anyone have any theories about what might be going on here? Have I been hacked maybe? How would you debug this further, and how might I fix the problem?
Thanks!
The database has obviously been hacked.
Shut it down and delete it right away.
Restore to a different cluster from a known good backup.
From now on, choose good passwords and use a restrictive pg_hba.conf that for example doesn't allow remote access for superusers.
Never, ever, operate your application with a superuser.
Examine your application for SQL injection vulnerabilities.
this may be because of a bot(made by hackers) that is scanning the internet and trying CVE exploit (N-day exploit) to see if it is vulnerable, and then launching that type of attack or it may be because someone on the VNAT with you trying to something weird, but one thing for sure it is a bot because you can not launch that many connections by hand.
Related
For this project, I am using a prisma / Postgres database. I have made no changes to my code, and I have pulled a coworkers working version of the code to no avail. I am unable to do anything with the database, I cannot migrate, I cannot run mutations, and I cannot even open the psql console, as every command is met with
P1001: Can't reach database server at localhost:5432
Please make sure your database server is running at localhost:5432
I am not sure what I could have possibly done, I don't know enough about ports or even the contents of app.json well enough to have messed anything up. Now, no mutations can go through.
Interestingly enough, this all happened after I ran npx primsa migrate deploy on the deployed database which is on a EC2 VM from AWS. Since then, the native app associated with the database refuses to work, though it is worth nothing that the webapp connects to the deployed database just fine. This being said, nothing works locally, as the database / Port / Server don't exist anymore according to my machine, which makes no sense. I have no idea how to try to re-spin it, or why every single query / mutation from my Native App now ONLy returns Response not successful: Received status code 400 despite it having the same exact syntax it did when it worked, as well as the WebApp having the same syntax and server (ExpressJS). Does anyone have any ideas what could be causing this?
The error code 400 refers to a bad request coming from the client: too large request, malformed syntax, invalid request message framing, etc.
First step: make sure that your database server is indeed running. Try connecting to it with other SQL Clients or Libraries. Sometimes Prisma is just being difficult.
Second thing: are you hosting the database on the local server? I can assume you are because of the localhost. Make sure no other programs are using this port or maybe waiting for it.
Sorry if this doesn't help. Good luck!
After a password rotation for an Azure Postgresql Single Database user this error showed up in my Java application:
org.postgresql.util.PSQLException: FATAL: authentication failed for user "<username>": too many failed login attempts
How to re-enable this Database User?
EDIT:
It seems time-sensitive, after waiting ~10 minutes the problem seems solved!
It seems time sensitive, after waiting ~10 minutes the problem is solved!
Two observations to help other people:
It's probably also IP related, as from other IP addresses the same user could connect perfectly fine (with the correct pwd :) ).
At the same time a different user (which was also rotated) was able to login from the same K8s cluster (same IP)
Has anyone managed to get Debezium to work over AWS RDS Postgres with rds.force_ssl turned on in the parameter group?
The connector seems to work for a bit, and then we begin to receive errors like Database connection failed when writing to copy and Exception thrown while calling task.commit().
I have scoured the web searching for this issue, and I see many people encountered it, and many Jira issues opened about it.
The response generally is "Check your network configuration" or "Disable your SSL".
I just can't get it to work for some reason, and obviously disabling encryption in transit is not possible in production use cases (at least in ours).
I would appreciate any kind of help or insight into how to solve this!
I know that using Firebird 2.5+ I can check if there are users accessing my database using SQL, but unfortunately, Firebird 2.0 doesn't have this feature. Yes, I know it's an old version, but it's a legacy software and I'm not allowed to upgrade this in a short time... :(
I need to know if someone is connected to my 2.0 Firebird database, due to a process I'll run:
Block connections to DB (but ONLY if no one is connected)
Run my process
Allow users to reconnect again
I can start my process only when there are no users connected.
My database is part of a client/server system (no Web).
Any hints?
-at[tach] : this parameter prevents any new connections to the database from being made with the exception of the SYSDBA and the database owner. The shutdown will fail if there are any sessions connected after the timeout period has expired. It makes no difference if those connected sessions belong to the SYSDBA, the database owner or any other user. Any connections remaining will terminate the shutdown with the following details:
https://firebirdsql.org/manual/gfix-dbstartstop.html
There is also Services API to do it so your database access library should expose the shutdown function. Specify a short shutdown, and if it failed - then there were some users. If it succeeded - now you can go on with maintenance, having a warranty client applications will not be able to connect.
Alternatively you can upgrade Firebird 2.0 -> 2.1 which is more close to 2.0 than 2.5 but already have Monitoring Tables implemented.
However this your approach has one weak point - race conditions. Using M.T. you envision your work as following:
Keep querying M.T. (which slows down server work significantly) until there are no other connections.
start maintenance work, that would fail if other connections are active
complete maintenance work
Problem is, that even after at step 1 you gained "no other connection" state, it does not mean that between steps 1 and 2, and especially between steps 2 and 3 now new connections would be made.
Even if you made your checks and ensure #1 condition, when you would go on with maintenance there would be some new user connected back and working now. Not every time of course, but as time goes by it will eventually happen one day.
But there is yet one more good thing in FB 2.1 - database-level triggers.
c:\Program Files\Firebird\Firebird_2_1\doc\sql.extensions\README.db_triggers.txt
You can create a regular "all_current_connections" table, using on connect and on disconnect triggers to keep it up to date.
You perhaps would also have to add some logic to your applications, so they would update that table with your internal application ID, to tell main workflow apps/connections from servicing utilities. However it is also possible that mere CURRENT_USER and CURRENT_CONNECTION pair, which the trigger knows and can store to the table, would be enough for that table, if you can infer kind of application from mere user name.
Then on disconnect trigger might be checking whether all "main workflow" apps disconnected and POST_EVENT to notify servicing utilities. However those utilities would still have to shutdown the database first, anyway.
You can shut down the database using gfix. The gfix tool will try to shutdown the database and if connections still exist after a timeout, the shutdown will fail.
For example, use:
gfix -shut -attach 5 <your-database>
This will:
prevent new connection being created,
wait 5 seconds for the existing connections to end,
if after 5 seconds there are still active connections the shutdown will abort,
otherwise, after 5 seconds the database will be shut down.
After shutdown, only SYSDBA or the database owner can create a connection to the database. This is only a viable option if your application it self doesn't use SYSDBA or the database owner account.
You bring the database back online using:
gfix -online <your-database>
For more information, see also Gfix - Database Housekeeping: Database Startup and Shutdown
Well, not an elegant way, but works...
I try to rename the database file.
If there is someone accessing the database, the rename operation will give me
an exception, saying that the file is in use by some process.
If rename succeeds, new users will not be able to access the database
anymore (the connection string used by my systems is not changed).
I run the exclusive process I have to.
Rename the database file to its original name, allowing new users to
connect again.
I post my solution in the hope that helps someone facing a similar problem.
Our new version of the product will probably a Web application and the database was not choosen yet, but certainly will no be Firebird.
Thanks to all that tried to give me an answer.
I am trying to get a mongoDB working with authentication, using both Java and PHP drivers. I've added user roles to the mongoDB but haven't yet turned on authentication (so clients can login with usernames and passwords, but they don't have to, and user roles are not yet enforced).
To check that everything is working, before actually turning authentication on, I've been looking at the mongod.log file. I see things like:
2015-11-17T08:47:19.052+0000 I NETWORK [initandlisten] connection accepted from ###:### #158126 (46 connections now open)
2015-11-17T08:47:19.960+0000 I ACCESS [conn158126] Successfully authenticated as ### on ###
But.... I also see quite a few connections without the "ACCESS" line. However, when cross referencing with logs of the clients, it seems they are trying to connect with authentication.
What can be going on?
Is it perhaps the case that the ACCESS log only occurs if some database action is taken? So, e.g. if a client connects but doesn't try to read or write, would I not see the 2nd line?
Is it perhaps the case that the ACCESS log only occurs if some database action is taken?
At least for the JAVA driver I'm using, "yes" is the answer. I ran a test, connecting, requesting a DB and collection, but doing nothing more, and no authentication check is triggered. It's only when you try to read/write that the authentication happens.