mongodb dropped collection keeps coming back after mongodump and mongorestore - mongodb

I have just created a data set, but a collection, 'items', is still lurking around and I am getting rid of it with the command db.items.drop() (after exec'ing into my docker container), which seems to successfully remove it. But when I run the 2 following scripts (after removing the collection):
save script:
#!/usr/bin/env bash
MONGO_DB_NAME="test-1"
DATA_NAME="test-data"
docker exec ${MONGO_DB_NAME} mongodump --quiet --out /tmp/${DATA_NAME}
mkdir -p ./backup
docker cp ${MONGO_DB_NAME}:/tmp/${DATA_NAME} ./backup/${DATA_NAME}
docker exec ${MONGO_DB_NAME} rm -rf /tmp/${DATA_NAME}
restore script:
#!/usr/bin/env bash
MONGO_DB_NAME="test-1"
DB_NAME="docker-node-mongo"
DATA_NAME="test-data"
docker exec -it $(docker ps -aqf "name=${MONGO_DB_NAME}") mongo ${DB_NAME} --eval "db.dropDatabase();"
docker cp ./backup/$DATA_NAME $(docker ps -aqf "name=${MONGO_DB_NAME}"):/tmp
docker exec -it $(docker ps -aqf "name=${MONGO_DB_NAME}") mongorestore --drop /tmp/${DATA_NAME}
the items collection comes right back. Does anyone know why this is? I'd like it gone for good.
Edit: When I delete individual items from the collection, and run the above scripts, they do not come back. But when I drop the collection as a whole, the collection keeps coming back (with nothing in it).

The mongodump command does not delete files that already exist in the --out path. Files for collections that are being backed up will be overwritten, but files for collection that no longer exist will not be touched, leaving the 'items' file(s) there to be restore the next time you run mongorestore.
To make sure this doesn't happen, move aside or remove the existing backup directory and create a new, empty one before running mongodump.

I had exactly similar "haunting" collection, which kept coming back again and again. The haunting collection also didn't work ok (e.g. cloning within it caused always an error). I don't know what caused it, but from some console logs I finally noticed that somehow another collection was using it's Name. I was then able to fix it by renaming the collection via mongosh and deleting then:
db.rrecord.renameCollection("record")
"This operation will rename the rrecord collection to record. If the target name (i.e. record) is the name of an existing collection, then the operation will fail."
ref. https://docs.mongodb.com/manual/reference/method/db.collection.renameCollection/

Related

Dockerized PGAdmin Mapped volume + COPY not working

I have a scenario where a certain data set comes from a CSV and I need to allow a non-dev to hit PG Admin and update this data set. I want to be able to put this CSV in a mapped folder from the host system and then use the PG Admin GUI to run a COPY command. So far PG Admin is telling me:
ERROR: could not open file "/var/lib/pgadmin/data-files/some_data.csv" for reading: No such file or directory
Here are my steps so far, along with a sanity check inspect:
docker volume create --name=data-files
docker run -e PGADMIN_DEFAULT_EMAIL="pgadmin#example.com" -e PGADMIN_DEFAULT_PASSWORD=some_pass -v data-files:/var/lib/pgadmin/data-files -d -p 5050:80 --name pgadmin dpage/pgadmin4
docker volume inspect data-files --format '{{.Mountpoint}}'
/app/docker/volumes/data-files/_data
docker cp ./updated-data.csv pgadmin:/var/lib/pgadmin/data-files
And, now I think that PG Admin could see the updated-data.csv, so I try COPY, which I know works locally on my dev system where PG Admin is on baremetal:
COPY foo.bar(
...
)
FROM '/var/lib/pgadmin/data-files/updated-data.csv'
DELIMITER ','
CSV HEADER
ENCODING 'windows-1252';
Is there any glaring mistake here? When I do docker cp there's no feedback to stdout. No error, no mention of success or a hash or anything.
It looks like you thought the file should be inside the pgadmin container however the file you are going to copy must be inside the postgres container so the query you run will find the file. I suggest you copy the file to postgres container :
docker cp <path_from_your_local>/file.csv <postgres_container_name>:/file.csv
Then in the query tool from your pgadmin you can copy without problems !
I hope this help to others came here...

Cannot mongodump to a single archive

I'm trying to dump a MongoDB database to an archive. Using the following command as given in documentation.
sudo mongodump --uri=mongodb://username:password#host:27017/dbname?authMechanism=SCRAM-SHA-1&authSource=authdb --archive=file.archive
But it doesn't dump as expected rather it creates a dump folder with .json file for each collection, which should be a single archive file as given.
It also shows the following error -
--archive=file.archive: command not found
Mongo version -
MongoDB shell version v3.6.3
I had this problem & I figured out the reason it was happening.
The command I was running was
sudo /usr/bin/mongodump --uri=mongodb+srv://{username}:{password}#{atlasendpoint}.mongodb.net/?retryWrites=true&w=majority --archive={filename}.archive --gzip 2>&1
I was getting the same error as you. I change the shell command by wrapping quotation marks around the URL; this fixed it for me.
sudo /usr/bin/mongodump --uri="mongodb+srv://{username}:{password}#{atlasendpoint}.mongodb.net/?retryWrites=true&w=majority" --archive={filename}.archive --gzip 2>&1

psql, can't copy db content to another - cannot run inside a transaction block-

I'd like to copy the content of my local machine to my remote one (inside a docker).
For some reason, it is more complicated that I was expected:
When I try to copy the data to the remote one, I get this "ERROR: CREATE DATABASE cannot run inside a transaction block".
Ok... So I get into my docker container, added the rule \set AUTOCOMMIT inside. But I still get this error.
This is the command I did:
// backup
pg_dump -C -h localhost -U postgres woof | xz >backup.xz
and then in my remote computer:
xz -dc backup.xz | docker exec -i -u postgres waf-postgres psql --set ON_ERROR_STOP=on --single-transaction
But each time I get this "CREATE DATABASE cannot run inside a transaction block" no matter what I try. Even if I put the autocommit to "on".
Here my problem: I don't know what a transaction block is. And I don't understand why copying one db to another need to be so hard pain: My remote db is empty. So why there is so much fuss and why psql just can't force what I want?
My aim is just to copy my local db to the remote one.
what happens here is: you add CREATE DATABASE statement with -C key and then try to run psql with --single-transaction, so the content of script are wrapped to BEGIN;...END;, where you can't use CREATE DATABASE
So iether remove -C and run psql against existing database, or remove --single-transaction for psql. Make decision based on what you really need...
from man pg_dump:
-C
--create
Begin the output with a command to create the database itself and reconnect to the created database. (With a script of this
form, it doesn't matter which database in the destination installation
you connect to before
running the script.) If --clean is also specified, the script drops and recreates the target database before reconnecting to
it.
from man psql:
--single-transaction
This option can only be used in combination with one or more -c and/or -f options. It causes psql to issue a BEGIN command
before the first such option and a COMMIT command after the last one, thereby wrapping all the commands into a single
transaction. This ensures that either all the commands complete successfully, or no changes are applied.

Mongodump not stored in the specific location

I am doing a mongodump, and want to store it in a specific location. So, from reading online, I can do this using the following command:
sudo mongodump -d mydbs -u user -p password -o /myfolder/mongoBackups
I am using an EC2 instance, and when I run this command from my centos home folder, this is the output I get from mongodump:
2017-01-07T16:01:42.053+0000 writing mydbs.users to
2017-01-07T16:01:42.055+0000 done dumping mydbs.users (2 documents)
However, when I cannot find the folder myfolder/mongoBackups anywhere in the specific home/centos location where I run it. Any idea why?
It's not within home/centos
It's in /myfolder/mongoBackups
Specifying an output directory of /myfolder/mongoBackups is basically putting it in a folder called myfolder on the root of the hard drive.
If you truly want the folder to be within home/centos try
sudo mongodump -d mydbs -u user -p password -o /home/centos/myfolder/mongoBackups

Boot2Docker (on Windows) running Mongo with shared folder (This file system is not supported)

I am trying to start a Mongo container using shared folders on Windows using Boot2Docker. When starting using run -it -v /c/Users/310145787/Desktop/mongo:/data/db mongo i get a warning message inside the container saying:
WARNING: This file system is not supported.
After starting mongo shutsdown immediately.
Any hints or tips on how to solve this?
Apparently, according to this gist and Sev (sevastos), mongo doesn't support mounted volume through the VirtualBox shared folder:
See mongoDB Productions Notes:
MongoDB requires a filesystem that supports fsync() on directories.
For example, HGFS and Virtual Box’s shared folders do not support this operation.
the easiest solutions of all and a proper way for data persistance is Data Volumes:
Assuming you have a container that has VOLUME ["/data"]
# Create a data volume
docker create -v /data --name yourData busybox true
# and use
docker run --volumes-from yourData ...
This isn't always ideal (but the following is for Mac, by Edward Chu (chuyik)):
I don't think it's a good solution, because the data just moved to another container right?
But it still inside the container rather than local system(mac disk).
I found another solution, that is to use sshfs to map data between boot2docker vm and your mac, which may be better since data is not stored inside linux container.
Create a directory to store data inside boot2docker:
boot2docker ssh
mkdir -p /mnt/sda1/dev
Use sshfs to link boot2docker and mac:
echo tcuser | sshfs docker#localhost:/mnt/sda1/dev <your mac dir path> -p 2022 -o password_stdin
Run image with mongo installed:
docker run -v /mnt/sda1/dev:/data/db <mongodb-image> mongod
The corresponding boot2docker issue points out to docker issue 12590 ( Problem with -v shared folders in 1.6 #12590), which points to the work around of using double-slash.
using a double slash seems to work. I checked it locally and it works.
docker run -d -v //c/Users/marco/Desktop/data:/data <image name>
it also works with
docker run -v /$(pwd):/data
As an workaround I just copy from a folder before mongo deamon starts. Also, in my case I don't care of journal files, so i only copy database files.
I've used this command on my docker-compose.yml
command: bash -c "(rm /data/db/*.lock && cd /prev && cp *.* /data/db) && mongod"
And everytime before stoping the container I use:
docker exec <container_name> bash -c 'cd /data/db && cp $(ls *.* | grep -v *.lock) /prev'
Note: /prev is set as a volume. path/to/your/prev:/prev
Another workaround is to use mongodump and mongorestore.
in docker-compose.yml: command: bash -c "(sleep 30; mongorestore
--quiet) & mongod"
in terminal: docker exec <container_name> mongodump
Note: I use sleep because I want to make sure that mongo started, and it takes a while.
I know this involves manual work etc, but I am happy that at least I got mongo with existing data running on my Windows 10 machine, and still can work on my Macbook when I want.
It's seems like you don't need the data directory for MongoDb, removing those lines from your docker-composer.yml should run without problems.
The data directory is only used by mongo for cache.