I have a MongoDB client in three EC2 instances and I have created a replica set. Last time I had a problem, of space constraint which stopped my mongod process, thereby halting the application and now in an instance couple of days back, some of my tables were gone from database, so I set logging and all to my database just to catch if anything like that happens again. In a fresh incident this morning I was unable to login to my system and that's when I found out that whole database was empty. I checked other SO question like this which suggest setting up a TTL.Which I haven't done at all.
Now how do I debug this situation and do a proper root cause analysis? I can't even find anything in my debug logs as well. The tables just vanished. How do I set up proper logging mechanism and how do I ensure that all my tables are never ever deleted again?
Today I got a mail from Amazon that I was probably running an unsecured version of MongoDB and that may have caused this issue. So who ever is facing this issue please go through the Security Checklist Provided by MongoDB. There are some points that are absolutely necessary in there.
1. Enable Access Control and Enforce Authentication
2. Encrypt Communication
3. Limit Network Exposure
These three are the core and depending upon how many people access your database you can Configure Role-Based Access Control.
These are all the things I have done. Before this incident I had not taken security that seriously but after I was hit by it. I made sure I have all the necessary precautions in place.
Hope this helps someone.
Related
We have a PostgreSQL database with PostGIS running and today we ran into the issue that too less connections were available. Mostly we are using QGIS to access the database. We realized that issue because multiple users got the following error:
FATAL: remaining connection slots are reserved for non-replication superuser connections
When checking the number of connections in pgAdmin I realized a thing I saw before, but as I never ran into problems didn't care too much about.
QGIS creates multiple connections to PostgreSQL for the same user to the same database.
Now I am wondering why this is the case and how I can maybe change that behaviour.
Could this happen for example if a person got access rights to a database through different user groups?
One approach might be the issue that some users run into that if you add layers to a QGIS project that was created before might ask you multiple times for your login credentials if those changed. This seem to me that probably different credentials are saved with the project and therefor multiple connections might be used. Can anyone confirm or dispruve this? - Suggestions for a test scenario are also welcome to check this.
Any ideas, hints or soutions are welcome.
By the way: Yes we increase the number of max_connections, but I want to understand why this happens and get closer to the core of the situation.
I've got a database on Google SQL that is used by our application running on kubernetes in GKE.
The mysql instance is running on 5.6, and I need to update it to 5.7, so I tried using the new migration jobs.
I've set up the connection profile and all the required permissions for the source DB, then followed the instructions to make a continuous migration.
The Job says it's running, migrating the ~450GB database. After about a day, it's still running, the storage used seems to have stopped growing, and the replication delay is at 0. The source database is not currently in use (That's why I'm unsing it to try this out before doing the same with a more important db).
According to this, if the dump phase is done, I should be able to promote the instance, but the promote button remains greyed out, and there's no way to check the running state (it only says "running", and I don't see any way to check if it's dumping, on CDC, or anything else).
The documentation seems a bit lacking, and I couldn't find anything by googling around. Has anyone been using this?
In short, my questions are:
Why can't I promote the instance?
and how can I check in what phase is the migration?
Here's a screencap of my job:
link because SO doesn't let me embed images yet
Thanks.
p.d.: the tag that the documentation says should be used in stackoverflow is: google-cloud-database-migration-service, which is too long and stackoverflow doesn't allow, so I used google-cloud-sql instead :/
I am seeing an issue like this, but possibly more frustrating. After a week for a 2TB database, storage resets to near-zero and the full dump restarts, without any errors or indication of what happened.
I am having a big problem, quite difficult to find/search.
I have a server in Ubuntu, where inside that server I have installed:
GITLAB (have all proyect)
POSTGRESSQL (Independent gitlab database is used for a personal project)
TOMCAT with APP WEB (Springboot, this use postgres)
This server is still for testing, it is used for specific specific things (I mean, its use and access is limited and controlled)
I am having various problems:
This server is still for testing, it is used for specific specific things (I mean, its use and access is limited and controlled)
Very frequently, almost every day, the user postgres from the postgresql server "erases" the password. Without anyone doing it manually, "it happens exponentially". I notice why the application stops responding, and then I access postgresql and note that the postgres user has no password.
I looked for many places, and I can't find anything. I really don't know where else to look. If someone passed it to you or has information about it, I would be grateful if you could provide it to me.
------More information added----------
I was looking at the postgres logs, before I have no authentication and I see this.
There are times when no one could have been using the springboot server,
--2020-01-17 00:30:21.286
And also the two log that show before that moment. Could it be something that is deleting my password?
Thank you.
PostgreSQL does not randomly delete its own passwords, and I really doubt Tomcat or Gitlab do either. Indeed they shouldn't even have access to the server as the 'postgres' user or any other superuser, and so shouldn't be able to even if they wanted.
It seems like that there is an intruder in your system. After gaining access they create their own user with their own password. Then disabling your normal superuser from logging on is a common way to try to prevent you from regaining control and kicking them out. Do any users exist that you do not recognize?
The bit of the log file you posted clearly shows someone trying to guess your password, starting at 2:58. You aren't logging IP addresses (%h) so it doesn't show where they are coming from. It doesn't show that they succeed, but unless you have log_connections = on, it wouldn't show successes.
I've just created a new MongoDB account and I'm now trying to connect the free cluster I created via MongoDB Compass Community application but I'm getting a 'Authentication Fail' error being displayed.
This is what I've checked so far:
From my MongoDB Clusters section when I clicked on the Connect (…) button which then gives you various options. From there, I selected 'Connect with MongoDB Compass' and copied the connection string.
This was detected as expected by the Compass and the information was filled automatically in all the relevant fields and I also filled the password by copy/pasting it into the relevant field. 100% sure it is correct.
I checked that the username used was indeed set up as an admin and it is.
I checked my Authentication database was correct and it is.
I've checked that my public IP was added to the whitelist and it is. The only thing I've noticed is that when I added my public IP address, it added a /32 at the end. Is that the port?
But I'm not quite sure what else to test for to resolve this problem.
Any suggestions?
Thanks.
I eventually found out what the problem was after speaking to someone from MongoDB support Team!
Everything was done correctly except for one thing. I was being impatient after changing my Cluster User's password. It can take up to 2 minutes for the system to be updated and therefore to allow Compass to access it.
Once I waited a couple of minutes, I was able to login as expected in Compass.
I still can't quite believe I wasted so much time on such a simple issue but the main thing is that it is resolved.
I did send them some feedback as a lot of things could have been done a lot better:
Highlight it better in their documentation i.e. red??
Make the "warning" message displayed on the webpage after updating the user details more obvious. It was right in my face and never spotted it appear or disappear as once I'd update the user detail on the website, I'd swap immediately to Compass to try to login. By the time, I'd be done, well over 2 minutes would elapsed and the message would be long gone, so not very useful the way it is currently done.
Instead of just saying: 'Authentication Fail', which is correct, the message could read differently when it knows the user is being updated i.e. 'Authentication Fail - Please try again in a few minutes as we're updating this user's details'... Something like this anyway.
So, remember to be patient when changing your user's details in MongoDB and if you are, then yes, you will have a database up and running in the cloud in 5 minutes or less! :)
The premises:
I'm using NodeJS, Mongodb Native 2.0+. I have created one connection (pool) which is keept open and reused in all modules. When production ready, the app and the db will be hosted on the same server, probably a VPS.
Questions about MongoDB in a production ready environment.
Should I anticipate that Mongod can crash? And if so, should I have some kind of autostart for it. Can "bugs" in my code be responsible for Mongod crashing?
If i should anticipate crasches, this probably also means that i should anticipate disruptions in my app when db cant be reached. What is the proper way to handle these? How long should i expect these disruptions to be?
If I start my app, manually close the Mongod and initiate a route call, the default response seems to be "waiting/loading" until Mongod is up again. I guess this default behaviour is ok if I dont expect disruption to be more than some seconds?
If Mongod is not up when starting the app, an exception will be thrown and the app will not start. This seems fine because without the db, the app cant do anything. Or should this be handled in another way?
I have made extensive search for this online, but have not found anything useful. Maybe I dont know what search terms to use...
There is alot of questions crammed up in one post here, but I hope someone could give me some answer or provide me with some links to good reading. The big question here is, how do i handle db failure at start and runtime