I am trying to export the MongoDB output to CSV format. But have trouble.
See the following document in my collection:
db.save.find().pretty();
{
"_id" : ObjectId("58884b11e1370511b89d8267"),
"domain" : "google.com",
"emails" : [
{
"email" : "f#google.com",
"first" : "James",
"Last" : "fer"
},
{
"email" : "d#gmail.com",
"first" : "dear",
"last" : "near"
}
]
}
Exporting the document to csv
C:\MongoDB\Server\bin>mongoexport.exe -d Trial -c save -o file.csv --type csv --fields domain,emails
2017-01-25T12:50:54.927+0530 connected to: localhost
2017-01-25T12:50:54.929+0530 exported 1 record
The output file is:
domain,emails
google.com,"[{""email"":""f#google.com"",""first"":""James"",""Last"":""fer""},{""email"":""d#gmail.com"",""first"":""dear"",""last"":""near""}]"
But if I import the same file, the output is different then it was in the actual collection. See the example:
> db.sir.find().pretty()
{
"_id" : ObjectId("5888529fa26b65ae310d026f"),
"domain" : "google.com",
"emails" : "[{\"email\":\"f#google.com\",\"first\":\"James\",\"Last\":\"fer\"},{\"email\":\"d#gmail.com\",\"first\":\"dear\",\"last\":\"near\"}]"
}
I do not want that extra \ in my import document. That's it. Please tell me if it is avoidable and if yes, then what should be the format of CSV to be given for import.
This is not expected format. So let me know how I can make the proper format. Kindly help me with this query.
Related
I am having problems exporting subdocuments that are stored in MongoDB to a .CSV.
My data: a mongo collection that contains a unique user ID and scores from personality quizzes.
I would like a CSV that has three columns: user_id, name, raw_score. To add a further layer of complexity, within the 'scales' subdocument some users will have more than two entries (some quizzes produced more than 2 personality scores).
An example of my data minus documents that I am not interested in:
"assessment":{
"user_id" : "5839b1a654842f35617ad100",
"submissions" : {
"results" : {
"scales" : [
{
"scale" : {
"name" : "Security",
"code" : "SEC",
"multiplier" : 1
},
"raw_score" : 2
},
{
"scale" : {
"name" : "Power",
"code" : "POW",
"multiplier" : -1
},
"raw_score" : 3
}
],
}
}
}
}
I have tried using mongoexport but this produces a CSV that only has a user_id column.
rekuss$ mongoexport -d production_hoganx_app -c assessments --type=csv -o app_personality.csv -f user_id,results.scales.scale.name,results.scales.raw_score
Any ideas where I am going wrong?
Please let me know if you need anymore information.
Many thanks
You should try removing '=' sign from type. You could try --type csv
Scope:
I have a collection on MongoDB that I want to export to a .csv file. I have done this already, with a similar database, using the MongoExport.exe, executing it right on the server (windows machine, hosting the MongoDB database).
Problem:
Once I run the following script
mongoexport.exe --fieldFile fields.txt --db AppleStore --collection AppleStoreApps --out applestore.csv --csv --port 21766
I start getting the following error messages
Invalid BSON object type for CSV output:10
It works for some cases, but seems like the majority of records gets this error.
More Information:
This is an example of JSON object on mongoDB, that should be exported:
{
"_id" : ObjectId("545c05ea74671a1d1c572da9"),
"url" : "https://itunes.apple.com/us/app/dc-eventos/id782560424?mt=8",
"name" : "DC Eventos",
"developerName" : "FERNANDO COSTA",
"developerUrl" : "https://itunes.apple.com/us/artist/fernando-costa/id729986271",
"price" : 0,
"isFree" : true,
"thumbnailUrl" : "http://a4.mzstatic.com/us/r30/Purple6/v4/ee/a2/5e/eea25e3f-8f12-9dce-c86f-37e5e3d9a8dc/icon350x350.jpeg",
"compatibility" : "Requires iOS 5.0 or later. Compatible with iPhone, iPad, and iPod touch. This app is optimized for iPhone 5.",
"category" : "Business",
"updateDate" : ISODate("2014-03-22T03:00:00.000Z"),
"version" : "1.82.82.542",
"size" : "16.3 MB",
"languages" : [
"English"
],
"minimumAge" : 4,
"ageRatingReasons" : [],
"rating" : {
"starsRatingCurrentVersion" : 0,
"starsVersionAllVersions" : 0,
"ratingsCurrentVersion" : 0,
"ratingsAllVersions" : 0
},
"topInAppPurchases" : null
}
mongoexport is likely choking on empty array -- "ageRatingReasons" : [] -- and null objects. examine the records one by one and check for a pattern.
csv cannot 'do' arrays and objects hence the need for json and xml. try exporting json and then convert with a variety of json to csv converters that will handle complex or custom flattening of objects such as [] to 0 or skipped commas val,,val whatever is needed. the jsontocsv convertor must also permit turning off validating, simply because ObjectId("545c05ea74671a1d1c572da9") is invalid json.
I am importing data into mongodb by this command
mongoimport -d dataBase --collection ip2location --type csv --file "/home/oodles/git/csv/IP2LOCATION-LITE-DB11.CSV" --fields _id,ipFrom,ipTo,countryCode,countryName,regionName,cityName,latitude,longitude,zipCode,timeZone
import is successfull
but the problem is
db.ip2Location.find().pretty()
getting
"_id" : ObjectId("52be7f25c80e0735273985bf"), ///here is requirement need "_id" : NumberLong(1)
"ipFrom" : NumberLong(16777216),
"ipTo" : NumberLong(16777471),
"countryCode" : "AU",
"countryName" : "AUSTRALIA",
"regionName" : "QUEENSLAND",
"cityName" : "SOUTH BRISBANE",
"latitude" : -27.48333,
"longitude" : 153.01667,
"zipCode" : 4101,
"timeZone" : "+10:00"
first line is "_id" : ObjectId("52be7f25c80e0735273985bf"),
but I need like this: "id" : NumberLong(1)
cvs data sample
"16777216","16777471","AU","AUSTRALIA","QUEENSLAND","SOUTH BRISBANE","-27.483330","153.016670","4101","+10:00"
csv data sample
"16777216","16777471","AU","AUSTRALIA","QUEENSLAND","SOUTH BRISBANE","-27.483330","153.016670","4101","+10:00"
The sample line doesn't match the list of --fields provided; it has one field less. The first field should be the _id you want to use (or 1 per your example).
Corrected line:
1, "16777216","16777471","AU","AUSTRALIA","QUEENSLAND","SOUTH BRISBANE","-27.483330","153.016670","4101","+10:00"
I tested and this works as expected in MongoDB 2.0.4.
I can't reproduce your results of having the _id added automatically with the first field value missing; when I tried with MongoDB 2.0.4 it assigned values to fields in the order listed so that the _id became 16777216, ipFrom was 16777471, etc. I suspect you may have been viewing a document inserted in an earlier mongoimport run where your --fields list did not include _id.
You should also be aware that mongoimport only inserts data (it does not do updates or upserts). If there is already a document with the given _id than mongoimport will ignore that line of your CSV.
An easier way to keep the fields and CSV data in sync is to have the list of fields as the first line in your CSV and then use mongoimport --headerline ... instead of --fields.
I am having a collection named "Books" and my sub documents like this
{
Country : 'XYZ'
Books : [
{"name" : "book1", "url" : "book1url", "auth_email" : "emailid1"},
{"name" : "book2", "url" : "book2url", "auth_email" : "emailid2"},
{"name" : "book3", "url" : "book3url", "auth_email" : "emailid3"},
{"name" : "book4", "url" : "book4url", "auth_email" : "emailid4"}
..........................................
]
}
I want to export it to a csv file with the following format
Country | name | url | auth_email| name | url | auth_email | ................
XYZ book1 book1url emailid1 book2 book2url emailid2 ...................
I did something like this
mongoexport -host localhost -db test -collection Books -csv > TopBooksOnline.csv -f Country,Books.name,Books.url,Books.auth_email
But I got empty content in the fields "Books.name", "Books.url" and " Books.auth_email"
If I do this
mongoexport -host localhost -db test -collection Books -csv > TopBooksOnline.csv -f Country,Books
Then it will club all my sub-docs into one.
How should I proceed ?
I don't think that the CSV format allows you to split up a single document as multiple documents, as this is something that you would need if you have nested documents. It would most definitely not create multiple fields for each sub-document as you are expecting. If you want to do that, then you can quite easily write your own CSV exporter which allows you a lot more flexibility anyway.
I have a schema, with a document structure that looks like this:
{ "_id" : ObjectId( "4f8dcb06ee21783d7400003c" ),
"venue" : ObjectId( "4f8dcb06ee21783d7400003b" ),
"name" : "Some event",
"webite: "www.whatever.com",
"attendees" : [
{ "_id" : ObjectId( "4f8dfb06ee21783d7134503a" ), "firstName" : "Joe", "lastName" : "Blogs", "emailAddress" : "some#thing1.com" },
{ "_id" : ObjectId( "4f8dfb06ee21783d7134503b" ), "firstName" : "John", "lastName" : "West", "emailAddress" : "some#thing2.com" }
{ "_id" : ObjectId( "4f8dfb06ee21783d7134503c" ), "firstName" : "Simon", "lastName" : "Green", "emailAddress" : "some#thing3.com" }
{ "_id" : ObjectId( "4f8dfb06ee21783d7134503d" ), "firstName" : "Harry", "lastName" : "Smith", "emailAddress" : "some#thing4.com" }
],
"eventType" : "Party"
}
I have a CSV file, that I'd like to import into the attendees collection...
Is that possible using mongoimport?
I read this: Using mongoimport to read CSV into nested structure? and it seems as though the answer could be no...
Since this is a one time operation, I wouldn't mind if I had to import it into an "attendees" collection, then run another command to insert into the attendees collection within my document? (there's only one event document at the moment....)
You are correct, and unfortunately the answer is no.
Mongoimport is a very simple program. (You can see for yourself here: https://github.com/mongodb/mongo/blob/master/src/mongo/tools/import.cpp) For users who would like to import data that is more complicated than a 2-dimensional table of strings and numbers, the official advice is to write a custom script that will read in your input file and create documents in exactly the format that you would like them. Hopefully the import.cpp file will give you some ideas on how to get started.
This maybe very late, but will be useful for future users running mongoDB > 4.x
You can have sub-document imported using a csv file by using a dot operator on the name.
for eg. if you want a document with a sub document called extra containing taxes, state & created_at, then you can name your fields as extra.taxes.auto(), extra.state.auto(), and extra.created_at.date(2006-01-02).
I was stuck on this question and couldn't find help online so posting it here.
Hope this helps someone.