I am trying to update a single field in each document in my collection using a csv and Mongoimport with –upsert included.
However the process removes all other fields in the document.
I have a Books Collection with documents like:
{
"_id" : "knOIv8ZUUK",
"Price" : 2.2,
"Title" : "Rats Ahoy"
}
{
"_id" : "okYEGuWznv",
"Price" : 3.3,
"Title" : "Friendly Fish"
}
a csv file:
_id,Price
knOIv8ZUUK,2.2
okYEGuWznv,3.3
And import using:
mongoimport --db local --collection Books --upsert --type csv
--headerline --file c:\import\newPrice
With results deleting the Title field
{
"_id" : "knOIv8ZUUK",
"Price" : 2.2
}
{
"_id" : "okYEGuWznv",
"Price" : 3.3
}
I, incorrectly, thought Upsert would just update an imported field.
So is there another process I can use to update just 1 field in large number of documents?
thanks
From mongoimport --upsertFields doc
You can need to use mode merge:
Merge existing documents that match a document in the import file with the new document. mongoimport will insert all other documents. Merge Matching Documents during Import describes how to use --mode merge.
and specify the field name, default is '_id'
--mode merge --upsertFields <fieldname>
so for your case just
--mode merge --upsertFields
New feature has been added since version 3.4 Documentation here.
Check this option
--mode insert|upsert|merge
In your case you can use this may be:
--mode merge
Related
Not able to backup mongo documents with command below.
Error:
Failed: error parsing query as Extended JSON: invalid JSON input. Position: 15. Character: O
mongodump --db test-data --collection foo --out=dump --query '{"_id":{"$lt": ObjectId("5e25b7a5f4c9b92aaa8a4131")}}'
ObjectId method returns a new ObjectId value which is not the ID value you passed in. The new ObjectId is invalid since is not part of JSON object.
Based on this documentation, you can passed your ID value which is 5e25b7a5f4c9b92aaa8a4131 to the nested query $oid (object id).
Query
{
"_id": {
"$lt": {
"$oid": "5e25b7a5f4c9b92aaa8a4131"
}
}
}
Final Solution
mongodump --db test-data --collection foo --out=dump --query '{"_id":{"$lt":{"$oid": "5e25b7a5f4c9b92aaa8a4131"}}}'
You will have to use MongoDB Extended JSON for this parse as json.
Your query becomes {"_id" : {"$lt" : {"$oid" : "5e25b7a5f4c9b92aaa8a4131"}}}
I'm trying to export phone numbers from a collection. Below is the sample document
{ "_id" : ObjectId("5ad5cf864717256ff02b4923"),"userName":"9619324746", "firstName" : "D H", "contactPhone" : 9619324746}
The export command that I used is below
mongoexport --db dbname --collection accounts --type=json --out accounts.json --fields contactPhone,userName
And the contents of JSON looks like below
{"_id":{"$oid":"5ad5cf864717256ff02b4923"},"userName":"9619324746","contactPhone":9.619324746e+09}
Can somebody help me to get the contactPhone value not converted? Thank you.
-Srini
If mongoexport exported 123 as 123.0, then 123 was a Double type in the document. You should try inserting the value as a 32- or 64-bit integer
db.collection.insert({
"tweetId" : NumberLong(1234567)
})
mongoexport exports JSON, using strict mode JSON representation, which inserts some type information into the JSON so MongoDB JSON parsers (like mongoimport) can reproduce the correct BSON data types while the exported JSON still conforms to the JSON standard
{ "tweetId" : { "$numberLong" : "1234567" } }
To preserve all the type information, use mongodump/mongorestore instead. To export all field values as strings, you'll need to write your own script with a driver that fetches each doc and stringifies all the values.
I have dozens of JSON files given by my colleague, they are comma separated documents, look like as follows:
{
"_id" : ObjectId("566a8d08b9ac7b7dc2ddb90a"),
"login_ip" : "180.173.143.x",
"login_time" : NumberLong("1478757697373"),
"logout_time" : NumberLong("1478757878035"),
"role" : NumberInt("5"),
"server_ip" : "115.28.94.x",
"thirdPartyId" : NumberInt("-1"),
"ver" : NumberInt("1036")
},
{
"_id" : ObjectId("566a8d0db9ac7b7dc2ddb90b"),
"login_ip" : "116.226.162.x",
"login_time" : NumberLong("1456103011531"),
"logout_time" : NumberLong("1456111567354"),
"role" : NumberInt("10002"),
"server_ip" : "115.28.94.x",
"thirdPartyId" : NumberInt("6"),
"ver" : NumberInt("1056")
},
...
I've tried to import them to my local mongodb with mongoimport tool, but it has trouble to locate the starting position of the second document, complains about the syntax of these files, in spite of the fact that the first document is parsed into the db.
+ mongoimport.exe --db eques --collection users_2017_04_25 --type json --bypassDocumentValidation --file 'E:\sample\mongdo/users_2017_04_25.json'
2017-06-12T14:01:32.029+0800 connected to: localhost
2017-06-12T14:01:32.040+0800 Failed: error processing document #2: invalid character ',' looking for beginning of value
2017-06-12T14:01:32.040+0800 imported 0 documents
PS: there're many files to be imported, please don't suggest me turn them into JSON arrays.
Please help.
The JSON documents above is expressed in BSON much of it mongoimport tool doesn't understand.
You should export your data in standard JSON format first(in Mongo world, it's call Strict mode), then feed it to mongoimport tool.
I can take mongodb data backup but I am not sure about mongodb schama backup.
Is there any way to take dump of MONGODB schema only not the data ?
You need to use mongorestore... which is used for things like importing json, or csv, etc.
You can read more about mongorestore in the docs below; I'd take a look and read up on them as they are very helpful.
http://www.mongodb.org/display/DOCS/Import+Export+Tools#ImportExportTools-mongorestore
You can also check out http://learnmongo.com for tips and help!
or you can visit the links
How to use the dumped data by mongodump? hope this may be helpful for you.
MongoDB is an NoSQL Database.
There is no fixed schema for any collection, so there are no functions available in mongo shell to find the collection schema.
Fixed Schema is applicable for RDBMS databases. In NoSQL DB, such as mongodb it is not required, but you can enforce same schema using your implementation logic, if required.
A document in a same collection, can be of different schema's. Please see example below
db.mycollection.insert([
{ "_id":1, "name":"A"},
{ "_id":2, "name":"CD", "age":29},
{ "_id":3, "name":"AB", "age":28},
{ "_id":4, "name":"ABC", "age":27, "emailId":"abc#xyz.com"},
{ "_id":5, "name":"ABCD", "age":29, "emailId":"abcd#xyz.com"}]);
db.mycollection.find();
{ "_id" : 1, "name" : "A" }
{ "_id" : 2, "name" : "CD", "age" : 29 }
{ "_id" : 3, "name" : "AB", "age" : 28 }
{ "_id" : 4, "name" : "ABC", "age" : 27, "emailId" : "abc#xyz.com" }
{ "_id" : 5, "name" : "ABCD", "age" : 29, "emailId" : "abcd#xyz.com" }
An approach to find the schema
In Mongo Shell
var k = db.mycollection.findOne();
for ( i in k){print (i)};
_id
name
this approach will work for you if all the documents in your collection follows the same schema.
Here's how I did it:
mongodump --uri="mongodb://localhost/mydb" -o ./mydb-dump
find ./mydb-dump -name *.bson -exec truncate -s 0 {} \;
Explanation: I'm dumping the whole database, then truncating all the .bson files (which hold collection data) to zero bytes.
Limitation: Obviously, this is only practical if the source database is small, otherwise you're generating a huge data dump only to throw away most of it.
To restore this-
mongorestore --uri="mongodb://some-other-server/mydb" ./mydb-dump
If there's a better way to do this, I'd love to know what it is!
MongoDB Compass GUI has a way to export the schema to JSON.
At the time of this post, there doesn't seem to be a way to do this by bulk, so this will have to be done for each collection one by one.
From the docs:
You can export your schema after analyzing it. This is useful for
sharing your schema and comparing schemas across collections.
If you have not already done so, analyze your schema:
Select your desired collection and click the Schema tab. Click
Analyze Schema.
Once your schema has been analyzed, export your schema:
In the top menu bar, click Collection. From the dropdown, click Share
Schema as JSON.
Your schema is copied to your clipboard as a JSON object.
See full docs here ~ https://www.mongodb.com/docs/compass/master/schema/export/
I wanted to use MongoVue to create a quick export of all the ids for portion of a given collection.
I have a document that has an _id field which is a composite key.
For example.
{
"_id" : {
"GroupID" : 3,
"ThingyID" : 320486
},
"HowManyOwned" : 42,
"IsAwesome" : true
}
I want to create an export of all the ThingyIDs for Group 3.
Of course if I make my query something like this.
db.GroupThingy.find({ "_id.GroupID" : 3 }, { "_id.ThingyID" : 1 })
I'll get back all the composite keys. I wanted to use MongoVue to just quickly create this export. If I use that query I get back an export of.
Document[2 Keys]
Document[2 Keys]
Document[2 Keys]
What I was hoping to get was
either
3,12345
3,3838
3,3777
3,1111
Or even better would be just
12345
3838
3777
1111
I could write a program for this but there has to be a quick way to accomplish this that I just don't know about.
Aggregation Framework doesn't help me get a csv export... but something like this will only support 20k documents
db.GroupThingy.group(
{
key: {
"_id.ThingyID": 1,
"_id.GroupID":1
},
cond: { "_id.GroupID": 3 },
reduce: function(curr, result){
result.ThingyID2 = curr._id.ThingyID
},
initial: { "ThingyID2": 0 }
});
Well I finally found out how to get the export using mongoexport thanks to this question:
how to export collection to csv in mongodb
However I kept getting the error:
ERROR: too many positional options... and this post helped me find the answer:
What does "too many positional options" mean when doing a mongoexport?
So my final solution to run the mongoexport was:
mongoexport
--host myHostName
--db theDB
--collection GroupThingy
--fields "_id.ThingyID"
--csv
--query "{'_id.GroupID':3}"
(options on there own lines for readability)