My scenario is that I use a local DB and I always keep it running. An hour ago, I upgraded iTerm2 and all opened windows/tabs closed. After I restart iTerm2, I found that mongod is still running, so I use kill $(pidof mongod) and restart with mongod --dbpath ./db. However, most data I inserted has disappeared. I have checked .zsh_history to confirm that I am using the same db folder. I have also checked all mongod.lock address on my computer to see other DB folders, and none of them stores my latest data. I have seen mongo.log and the commands I used to query new data was still there. Therefore, how can I find my lost data? Thanks.
P.S. This situation happened last month when my mac was forced to shut down because of low battery. It happens again so I ask and want to know why it happens and what I can do.
Related
I use MongoDB 4.2 on my local machine (windows 10). I have not changed any configurations, so the default behavior of only accepting local connections should be in place. (I only need to access it locally)
I was running a script that was reading data from my MongoDB, there are no writes to the db in this script. When all the numbers were crunched I noticed weird results, and saw that my database was suddenly gone. I checked my dbpath and the data was gone from there too! Could it be a hack, or was it MongoDB that dropped both the database and the raw data in the dbpath?
I've seen similar questions on this forum, mostly resolved by the author forgetting to reroute to the correct dbpath, which is not the case here. I've checked the log but the log seems to be very limited (I restarted mongod and could only see logging happening after the restart).
MongoDB does not delete all of the files in its data directory.
Most likely either you are checking in the wrong place or something external to MongoDB deleted its files.
MongoDB was working beautifully for me for several months until I had an unexpected shutdown a week or two ago. Since then, I've been getting the error in the title that snowballs into an invalid argument, then a library panic, then some fatal assertions which cause MongoDB to crash.
Now, I've done my research: the normal answers are to run the repair function and to make sure SELinux isn't screwing up the process. Neither of those have worked. The error gets thrown during WiredTiger's checkpoint process, so reads/writes to the database aren't the issue, and because it's during the checkpoint process, it guarantees that MongoDB won't stay up for more than a day.
To be clear: all the files in the database are owned by mongod:mongod, have permissions set to 600 (default, and I tried setting them to 755 to see if that fixed it, and it didn't). I'm running mongodb as a service on a CentOS 7 box, and the service file specifies that it should run as user mongod. The mongod.conf file specifies a mounted filesystem as the database, and it was happy with that until the unexpected shutdown. I'm running MongoDB version 4.0.1, so WiredTiger really doesn't like it if I disable Journaling either (disregarding the fact that I shouldn't disable it in the first place).
I feel like I've exhausted all my options, and that the only thing I can do is backup my data and reinstall MongoDB. Are there any that I've missed?
After creating a backup of my data via mongodump, shutting down mongo, removing the entire database with rm -rf 'path-to-database', rebooting mongo (without the replication config), and restoring the data with mongorestore, mongodb still crashes. This time, however, it's with an Invariant failure after the open: operation not permitted. The only conclusion I can think of is that the data itself has become corrupted in some way. Thankfully, this isn't "mission critical" data, so to speak, and I can easily obtain new data.
Unfortunately, this doesn't answer my original question of "what other options do I have?". However, I'm still posting this in case others run into this same kind of issue.
EDIT: invariant issue was caused by me forgetting to re-initialize my replication set. After fixing that, it's clean. Because of this, I no longer believe it was a data corruption issue, but a checkpoint corruption issue.
EDIT 2: So the issue arose again after about a week, and after another week of trying various debugging methods, I tried simply moving the mongo process to another server. So far, that's been working. The previous server was acting up (I couldn't even run top at one point - another process had a lock on a necessary library file to run it), so here's to hoping that the current server doesn't follow suite.
At my work I was running a complex query. I cancelled it and went home yesterday. This morning in the back the query was impossible to be terminated, also with the 'terminate backend' functionality. A colleague of mine restarted the host machine where postgres is installed. After the machine restart, the postgres database sever would not start up.
In my log files I see the error:
'pg_ctl: this data directory appears to be running a pre-existing postmaster'
I am not sure how to handle this problem. I could try to fix it or try to extrapolate the data from the save files. What is the most logical step to take and do you know how to fix this?
Earlier it gave this error message :
2016-01-28 15:52:33 GMT FATAL: lock file "postmaster.pid" already exists
2016-01-28 15:52:33 GMT HINT: Is another postmaster (PID 2100) running in data directory "C:/PostgreSQL/9.1/data"?
UPDATE... I located the file postmaster.pid and deleted it. Now I am restarting the computer and hoping it will start.
UPDATE... It works now. I rebooted the computer and postgres just instantly started. Happy as a child but at the same time not fully satisfied because of the following forum: https://superuser.com/questions/553045/fatal-lock-file-postmaster-pid-already-exists . Here it is stated to NEVER delete the postmaster.pid because of possible data corruption. So because of that I will backup all databases I have in postgres now.
So if anyone can share some more light on my ICT adventure of today I would be very satisfied. That is why I will not state that this question is answered, since I have no idea what went wrong and perhaps will run into it again someday.
The explanation is pretty straightforward. PostgreSQL writes the process ID to a file called postmaster.pid — the presence of the file is supposed to indicate that the server is running. When the PostgreSQL shuts down cleanly, it removes the postmaster.pid file.
However, when your colleague restarted the host machine, the PostgreSQL server got killed without having had a chance to remove the postmaster.pid file. Therefore, when you tried to start PostgreSQL, the presence of the file made it look complain that the server was already running.
This answer provides more complete advice. In general, you should never delete postmaster.pid for no good reason, because it's supposed to help prevent two servers from running at once on the same data files. However, if you are certain that the process indicated by the postmaster.pid file is already dead, then by all means just delete the stale PID file manually.
In windows Delete all running postgres processes and start the service
I have a single machine that has MongoDB and its data is at /data/db as usual.
When my machine crashed due to an electric power trip, my MongoDB refuse to start at launch (Mac OS X Server via LaunchAgent) and also /data/db mysteriously disappear!
Also all log file are wipe out. This happen on my development SSD MBA and I thought is just a weird SSD case. But my XServe server is getting it as well when the power trip.
Am I missing some data protection articles somewhere? For sure it can't be this unreliable by just deleting /data/db!!??
MongoDB will never ever remove your database files!
In case of a crash you have to start mongod using the --repair option.
In addition: using the new journaling option of MongoDB in V 1.8+ that should help a lot when you run MongoDB as standalone service.
No that is not normal.
If it won't start, it's likely mongodb is indicating that you need to run a repair because mongod.lock is present and has a certain state in /data/db. But that would mean /data/db exists.
If /data/db exists but were empty (which in this case would be bad obviously), it would start right up.
If you log(s) are missing, sounds like a more general disk issue.
So check the startup message if about mongod.lock there is data there. Also with v1.8+ use journaling. (albeit you wouldn't lose all datafiles even without journaling)
I installed MongoDB both on Win 7 and on Mac OS X, and both places, I got mongod (the server) and mongo (the client).
But at both places, running mongod will fail if I double click on the file, and the error message was gone too quickly before I can see anything. (was better on Mac because Terminal didn't exit automatically and showed the error message).
Turned out it was due to /data/db not exist and the QuickStart guide says: By default MongoDB will store data in /data/db, but it won't automatically create that directory
I just have a big question that MongoDB seems to want a lot of people using it (as do many other products), but why would it not automatically create the folder for you? If it didn't exist... creating it can do not much harm... especially you can state so in the user agreement. The question is why. I can think of one strange reason, but the reason may be too strange to list here...
One good reason would be that you do not want it in /data/db. In this case, you want it to fail with an error when you forgot to specify the correct directory on the command line. The same goes for mis-spelled directory names. If MongoDB just created a new directory and started to serve from there, that would not be very helpful. It would be quite confusing, because databases and collections are auto-created, so there would not even be errors when you try to access them.