I'm trying to export phone numbers from a collection. Below is the sample document
{ "_id" : ObjectId("5ad5cf864717256ff02b4923"),"userName":"9619324746", "firstName" : "D H", "contactPhone" : 9619324746}
The export command that I used is below
mongoexport --db dbname --collection accounts --type=json --out accounts.json --fields contactPhone,userName
And the contents of JSON looks like below
{"_id":{"$oid":"5ad5cf864717256ff02b4923"},"userName":"9619324746","contactPhone":9.619324746e+09}
Can somebody help me to get the contactPhone value not converted? Thank you.
-Srini
If mongoexport exported 123 as 123.0, then 123 was a Double type in the document. You should try inserting the value as a 32- or 64-bit integer
db.collection.insert({
"tweetId" : NumberLong(1234567)
})
mongoexport exports JSON, using strict mode JSON representation, which inserts some type information into the JSON so MongoDB JSON parsers (like mongoimport) can reproduce the correct BSON data types while the exported JSON still conforms to the JSON standard
{ "tweetId" : { "$numberLong" : "1234567" } }
To preserve all the type information, use mongodump/mongorestore instead. To export all field values as strings, you'll need to write your own script with a driver that fetches each doc and stringifies all the values.
Related
Not able to backup mongo documents with command below.
Error:
Failed: error parsing query as Extended JSON: invalid JSON input. Position: 15. Character: O
mongodump --db test-data --collection foo --out=dump --query '{"_id":{"$lt": ObjectId("5e25b7a5f4c9b92aaa8a4131")}}'
ObjectId method returns a new ObjectId value which is not the ID value you passed in. The new ObjectId is invalid since is not part of JSON object.
Based on this documentation, you can passed your ID value which is 5e25b7a5f4c9b92aaa8a4131 to the nested query $oid (object id).
Query
{
"_id": {
"$lt": {
"$oid": "5e25b7a5f4c9b92aaa8a4131"
}
}
}
Final Solution
mongodump --db test-data --collection foo --out=dump --query '{"_id":{"$lt":{"$oid": "5e25b7a5f4c9b92aaa8a4131"}}}'
You will have to use MongoDB Extended JSON for this parse as json.
Your query becomes {"_id" : {"$lt" : {"$oid" : "5e25b7a5f4c9b92aaa8a4131"}}}
I have dozens of JSON files given by my colleague, they are comma separated documents, look like as follows:
{
"_id" : ObjectId("566a8d08b9ac7b7dc2ddb90a"),
"login_ip" : "180.173.143.x",
"login_time" : NumberLong("1478757697373"),
"logout_time" : NumberLong("1478757878035"),
"role" : NumberInt("5"),
"server_ip" : "115.28.94.x",
"thirdPartyId" : NumberInt("-1"),
"ver" : NumberInt("1036")
},
{
"_id" : ObjectId("566a8d0db9ac7b7dc2ddb90b"),
"login_ip" : "116.226.162.x",
"login_time" : NumberLong("1456103011531"),
"logout_time" : NumberLong("1456111567354"),
"role" : NumberInt("10002"),
"server_ip" : "115.28.94.x",
"thirdPartyId" : NumberInt("6"),
"ver" : NumberInt("1056")
},
...
I've tried to import them to my local mongodb with mongoimport tool, but it has trouble to locate the starting position of the second document, complains about the syntax of these files, in spite of the fact that the first document is parsed into the db.
+ mongoimport.exe --db eques --collection users_2017_04_25 --type json --bypassDocumentValidation --file 'E:\sample\mongdo/users_2017_04_25.json'
2017-06-12T14:01:32.029+0800 connected to: localhost
2017-06-12T14:01:32.040+0800 Failed: error processing document #2: invalid character ',' looking for beginning of value
2017-06-12T14:01:32.040+0800 imported 0 documents
PS: there're many files to be imported, please don't suggest me turn them into JSON arrays.
Please help.
The JSON documents above is expressed in BSON much of it mongoimport tool doesn't understand.
You should export your data in standard JSON format first(in Mongo world, it's call Strict mode), then feed it to mongoimport tool.
I am trying to update a single field in each document in my collection using a csv and Mongoimport with –upsert included.
However the process removes all other fields in the document.
I have a Books Collection with documents like:
{
"_id" : "knOIv8ZUUK",
"Price" : 2.2,
"Title" : "Rats Ahoy"
}
{
"_id" : "okYEGuWznv",
"Price" : 3.3,
"Title" : "Friendly Fish"
}
a csv file:
_id,Price
knOIv8ZUUK,2.2
okYEGuWznv,3.3
And import using:
mongoimport --db local --collection Books --upsert --type csv
--headerline --file c:\import\newPrice
With results deleting the Title field
{
"_id" : "knOIv8ZUUK",
"Price" : 2.2
}
{
"_id" : "okYEGuWznv",
"Price" : 3.3
}
I, incorrectly, thought Upsert would just update an imported field.
So is there another process I can use to update just 1 field in large number of documents?
thanks
From mongoimport --upsertFields doc
You can need to use mode merge:
Merge existing documents that match a document in the import file with the new document. mongoimport will insert all other documents. Merge Matching Documents during Import describes how to use --mode merge.
and specify the field name, default is '_id'
--mode merge --upsertFields <fieldname>
so for your case just
--mode merge --upsertFields
New feature has been added since version 3.4 Documentation here.
Check this option
--mode insert|upsert|merge
In your case you can use this may be:
--mode merge
I wanted to use MongoVue to create a quick export of all the ids for portion of a given collection.
I have a document that has an _id field which is a composite key.
For example.
{
"_id" : {
"GroupID" : 3,
"ThingyID" : 320486
},
"HowManyOwned" : 42,
"IsAwesome" : true
}
I want to create an export of all the ThingyIDs for Group 3.
Of course if I make my query something like this.
db.GroupThingy.find({ "_id.GroupID" : 3 }, { "_id.ThingyID" : 1 })
I'll get back all the composite keys. I wanted to use MongoVue to just quickly create this export. If I use that query I get back an export of.
Document[2 Keys]
Document[2 Keys]
Document[2 Keys]
What I was hoping to get was
either
3,12345
3,3838
3,3777
3,1111
Or even better would be just
12345
3838
3777
1111
I could write a program for this but there has to be a quick way to accomplish this that I just don't know about.
Aggregation Framework doesn't help me get a csv export... but something like this will only support 20k documents
db.GroupThingy.group(
{
key: {
"_id.ThingyID": 1,
"_id.GroupID":1
},
cond: { "_id.GroupID": 3 },
reduce: function(curr, result){
result.ThingyID2 = curr._id.ThingyID
},
initial: { "ThingyID2": 0 }
});
Well I finally found out how to get the export using mongoexport thanks to this question:
how to export collection to csv in mongodb
However I kept getting the error:
ERROR: too many positional options... and this post helped me find the answer:
What does "too many positional options" mean when doing a mongoexport?
So my final solution to run the mongoexport was:
mongoexport
--host myHostName
--db theDB
--collection GroupThingy
--fields "_id.ThingyID"
--csv
--query "{'_id.GroupID':3}"
(options on there own lines for readability)
I'm trying to use mongoimport to upsert data with string values in _id.
Since the ids look like integers (even though they're in quotes), mongoimport treats them as integers and creates new records instead of upserting the existing records.
Command I'm running:
mongoimport --host localhost --db database --collection my_collection --type csv --file mydata.csv --headerline --upsert
Example data in mydata.csv:
{ "_id" : "0364", someField: "value" }
The result would be for mongo to insert a record like this: { "_id" : 364, someField: "value" } instead of updating the record with _id "0364".
Does anyone know how to make it treat the _id as strings?
Things that don't work:
Surrounding the data with double double quotes ""0364"", double and single quotes "'0364'" or '"0364"'
Appending empty string to value: { "_id" : "0364" + "", someField: "value" }
Unfortunately there is not now a way to force number-like strings to be interpreted as strings:
https://jira.mongodb.org/browse/SERVER-3731
You could write a script in Python or some other language with which you're comfortable, along the lines of:
import csv, pymongo
connection = pymongo.Connection()
collection = connection.mydatabase.mycollection
reader = csv.DictReader(open('myfile.csv'))
for line in reader:
print '_id', line['_id']
upsert_fields = {
'_id': line['_id'],
'my_other_upsert_field': line['my_other_upsert_field']}
collection.update(upsert_fields, line, upsert=True, safe=True)
Just encountered this same issue and discovered an alternative. You can force Mongo to use string types for non-string values by converting your CSV to JSON and quoting the field. For example, if your CSV looks like this:
key value
123 foo
abc bar
Then you'll get an integer field for key 123 and a string field for key abc. If you convert that to JSON, making sure that all the keys are quoted, and then use --type json when you import, you'll end up with the desired behavior:
{
"123":"foo",
"abc":"bar"
}
I was able to prefix the numeric string and that worked for me. Example:
00012345 was imported as 12345 (Type Int)
string00012345 was imported as string00012345 (Type String)
My source was a SQL database so I just did
select 'string'+column as name
Of course, you also need to do a bit of post-processing to parse the string, but far less effort than converting a rather large tsv file to json.
I also added +1 to the jira link above for the enhancement.
As an alternative to #Jesse, you can do something similar in the mongo console, e.g.
db.my_collection.find().forEach(function (obj) {
db.my_collection.remove({_id: obj._id); // remove the old one
obj._id = '' + obj._id; // change to string
db.my_collection.save(obj); // resave
});
For non _id fields you can simply do:
db.my_collection.find().forEach(function (obj) {
obj.someField = '' + obj.someField; // change to string
db.my_collection.save(obj); // resave
});
I encountered the same issue.
I feel the simplest way is to convert the CSV file to a JSON file using an online tool and then import.
This is the tool I used:
http://www.convertcsv.com/csv-to-json.htm
It lets you wrap the integer values of your CSV file in double quotes for your JSON file.
If you have trouble importing this JSON file and encountering an error, just add --jsonArray to your import command. It will work for sure.
mongoimport --host localhost --db mydb -c mycollection --type json --jsonArray --file <file_path>