Looking at the following example from the MongoDB documentation: https://docs.mongodb.com/manual/reference/program/mongoimport/#ex-mongoimport-merge, the --mode merge works for 1 level deep collections.
I'm trying to merge collections that are n-level deep and the levels after the 1 level get overwritten as in using the --mode upsert flag.
Is that a bug in the merge function or was it not intended to work recursively?
Thanks!
Related
I'm trying to write a mongodump / mongorestore script that would copy our data from the production environment to staging once a week.
Problem is, I need to filter out one of the collections.
I was sure I'd find a way to apply a query only on a specific collection during the mongodump, but it seems like the query statement affects all cloned collections.
So currently I'm running one dump-restore for all the other collections, and one for this specific collection with a query on it.
Am I missing something? Is there a better way to achieve this goal?
Thanks!
It is possible.
--excludeCollection=<string>
Excludes the specified collection from the mongodump output. To exclude multiple collections, specify the --excludeCollection multiple times.
Example
mongodump --db=test --excludeCollection=users --excludeCollection=salaries
See Details here.
Important mongodump writes to /dump folder. If it's already there, it will overwrite everything.
If you need that data rename the folder or give mongodump an --out directory. Otherwise you don't need to worry.
I am trying to load data from a file created using mongoexport in one server into another. New documents must be inserted, existing ones must be updated with data in the file.I see that documentation refers to upsert and merge options, but the difference is not obvious to me.
There are a lot of "why not JOIN?" questions here, to understand who to use mongodb... But even the guide at model-embedded-one-to-many-relationships-between-documents not show the basic clues...
There are to approaches:
generate an array that contains the "embedded data" of the join, to produce "on fly" data. (using find() and what more simple algoritm?)
use the db.collection.insert to produce "persistent data".
So, if I am at mongo terminal, what the simplest way to do that?
real-life dataset example
At https://github.com/datasets/country-codes we have
country-codes.csv: a table with ~250 rows.
datapackage.json: a complex structure with the country-codes table metadata, at resources.schema.fields.
So, before to do queries at mongo in the terminal, we can perform something like
wget -c https://raw.githubusercontent.com/datasets/country-codes/master/data/country-codes.csv
wget -c https://raw.githubusercontent.com/datasets/country-codes/master/datapackage.json
mongoimport -d ccodes_db -c ccodes --type csv --file country-codes.csv --headerline
mongoimport -d ccodes_db -c ccodes_meta datapackage.json --jsonArray
mongo
show dbs
use ccodes_db
So, lets "join" ccodes_meta with ccodes collections at mongo... The task is to embed fields name and description (of ccodes_meta) into the ccodes collection... With the simplest algorithms (not need best performance), see itens 1 and 2 of the question.
I am trying to use mongoimport to translate a one-to-many relational structure into mongoDB using csv files. My approach is to import the "one" file then use the upsert option to append the "many" records as a nested array but it looks like it only replaces the original document instead of appending.
Is this a limitation of mongoimport or could I be doing something wrong?
You can do upserts when using mongoimport, but you cannot use complex operators to perform modifications to the data as you would with a normal update operation. This is a limitation of mongoimport - essentially each piece of data you import must be insert ready even though you are using the upsert functionality, which is basically working as a de-duplication mechanism for your input data.
If you wish to merge in a more sophisticated manner then it would be best to use one of the drivers and merge the data using your language of choice. This also has the advantage of avoiding potential issues with type fidelity and allowing you to code around exceptions etc.
Is it possible to do bulk update/upsert (not insert) in MongoDB?
If yes, please point me to any docs related to this?
Thanks
You can use the command line program mongoimport it should be in your MongoDB bin dir ...
There are two options you'll want to look into to use upsert ...
--upsert insert or
update objects that already exist
--upsertFields arg comma-separated fields for the query
part of the
upsert. You should make sure this is indexed
More info here: http://www.mongodb.org/display/DOCS/Import+Export+Tools
Or just do ...
$ mongoimport --help
mongo can execute .js file.
you can push all you update commands in a js file.
t.js
db.record.update({md5:"a35f10a8339ab678612d1f86be08b81a"},{$set:{algres:[]}},false,true);
db.record.update({md5:"a35f10a8339ab678612d1f86be08b81b"},{$set:{algres:[]}},false,true);
then,
mongo 127.0.0.1/test t.js
Bulk updates can also be done in batches as found in the documentation:
MongoDB Bulk Methods
I use these to import CSV files that I need to massage a bit before importing the data. Its kinda slow when dealing with updates, but it did my 50K document updates in about 83 seconds, which is far slower than mongoimport command.