mongoexport fields from subdocuments to csv - mongodb

I am trying to export a field from a subdocument with no luck.
Here is my syntax;
mongoexport -d test -c accounts -f account_number,situses.coordinates -o coordinates.csv --type=csv
The output includes the account_number but not the coordinates field from the subdocument.
According to the docs, this is supposed to work.
The following will export the entire situses subdocument, but I only want the one field.
mongoexport -d test -c accounts -f account_number,situses -o coordinates.csv --type=csv
Am I just referencing the subdocument field wrong or something?
I'm running Mongodb 3.0.4
ADDITIONAL INFO
The following syntax worked on an earlier version of Mongodb (2.6.x ?). Notice the subdoc.0.fieldname syntax.
mongoexport -d test -c accounts -f account_number,situses.0.coordinates -o coordinates.csv --csv
It appears support for directly referencing a subdocument has been removed.

There is mistake in your syntax.
From mongo version 3.0.0, mongoexport removed the --type = csv option. Use the --type=csv option to specify CSV format for the output.
You should use :
mongoexport --db tests --collection accounts --type=csv --fields account_number,situses --out coordinates.csv
For nested fields you should use :
mongoexport --db tests --collection accounts --csv --fields 'account_number,situses.0.coordinates' --out /home/vishwas/c1.csv
EDIT for mongo 3.0 with sub documents:
You need to create separate collection with required fields from subdocuments like -
db.test.aggregate({"$unwind":"$situses"},{"$project":{"_id":0,"account_number":1,"siteUsesCo":"$situses.coordinates"}},{"$out" : "forcsv"})
If you want only one field from subdocument then use aggregation like -
db.test.aggregate({"$unwind":"$situses"},{"$limit":1},{"$project":{"_id":0,"account_number":1,"siteUsesCo":"$situses.coordinates"}},{"$out" : "forcsv"})
And then export from forcsv collection like-
mongoexport --db test --collection forcsv --csv --fields 'account_number,siteUsesCo' --out coordinates.csv
And after exporting delete collection forcsv.

And one more solution, where you can configure output in flexible way
mongo host:port/test --quiet query.js -u username -p passw0rd > accounts.csv
and query.js:
db = db.getSiblingDB('test');
db.getCollection('accounts').find({}, {account_number:1, situses:1, _id:0}).forEach(
function(item_data) { print(`${item_data.account_number},${item_data.situses[0].coordinates}`); });

It looks like this is a known bug to be fixed in 3.0.5.
See this; https://jira.mongodb.org/browse/TOOLS-657

Related

Syntax error:missing ; before statement #(shell):1:15 [duplicate]

I'm working with mongodb 2.4.3 and I cannot get mongoexport to work. The error message I get with each command is:
"JavaScript execution failed: SyntaxError: Unexpected identifier"
At first I thought the problem was with my query parameters, which were long. But find() worked fine with the query so I know that the syntax is ok. I then created a new collection using the query and tried exporting the collection with the following:
mongoexport --db Oct2012 --collection sept8subset --csv --fields "text","created_at","user.screen_name","user.followers_count" --out sept8.csv
mongoexport --db Oct2012 --collection sept8subset --csv --fields text,created_at,user.screen_name,user.followers_count --out sept8.csv
mongoexport -d Oct2012 -c sept8subset --csv --fields text,created_at,user.screen_name -o sept8.csv
mongoexport --db Oct2012 --collection sept8subset --dbpath ~/db (should need dbpath as mongod instance is running)
mongoexport --db OCt2012 -collection sept8subset -o sept8.txt
mongoexport --db Oct2012 --collection sept8subset
In each case, I get "JavaScript execution failed: SyntaxError: Unexpected identifier". Where could that SyntaxError be??
The collection I'm trying to copy has 50,339 objects. In case it is just too big for mongoexport to handle, I took 5 documents out of the collection to make a new collection. I then tried to export them using same command structure as above. I still get the same error message.
Now I'm wondering if the problem is that mongoexport can't work with data involving dates. MongoDB documentation states that I may want to write my own export script using a client driver.
Does anyone have an idea of what my problem is here? Many thanks if you can help out.
I found out that running mongoexport from the mongo client--as I was doing--is incorrect. Mongoexport is executed at the command prompt.
I ran into this too and I solved it executing the command from the path/bin where 'mongoexport' is located.In my case :
my_path: /usr/local/Cellar/mongodb/2.4.4-x86_64/bin
"mongoexport -d book -c shelf"
'mongod' is located one step upper from 'mongoimport' , 'mongoexport', etc...

how to run mongoexport csv with a query

I'm trying to export a mongodb query to csv file. here's what I have:
mongoexport --db db_name --collection agents --query ‘{ $and: [ {clients_count: {$gt:2}}, {vendors_count:{$gt:10}} ] }’ --csv --fieldFile userFields.txt --out outputFilePathAndName.csv
I got the following error:
Error parsing command line: too many positional options
What am I doing wrong?
got it. correct query:
mongoexport --db db_name --collection collectionName --query '{$and:[{clients_count:{$gt:2}},{vendors_count:{$gt:10}}]}' --csv --fieldFile userFields.txt --out filepat/fileName.csv
the key is to use single quotes ' and leave no spaces in the query.
EDIT
having issue with the fields, but seems to be grabbing the correct documents.

mongoexport not working if you add query of selection

If you want to export any collections of any of your mongodb database, you can use mongoexport command. For example:
mongoexport --db dbname --collection collectionName --query '{"fields":1}' --out output.json
However, if you add any of selection criteria to the query, mongoexport command doesn't work. For example, if you run the following command:
mongoexport --db dbname --collection collectionName --query '{},{"fields":0}' --out output.json
the resultant JSON file includes data of every field, despite me excluding a filed named fields.
So why does this such strange behavior occur? And how can I fix it?
For your information, db.colName.find({},{"fields":0}) in mongoDB shell works as usual.
I'm on MongoDB 2.4.3 and OS X 10.9.
Thanks.
That's because --query argument takes only query and there is no option to pass projection. See mongoexport --query in official documentation. If you want to use a projection you have to add --fields option

mongoexport syntax error message

I'm working with mongodb 2.4.3 and I cannot get mongoexport to work. The error message I get with each command is:
"JavaScript execution failed: SyntaxError: Unexpected identifier"
At first I thought the problem was with my query parameters, which were long. But find() worked fine with the query so I know that the syntax is ok. I then created a new collection using the query and tried exporting the collection with the following:
mongoexport --db Oct2012 --collection sept8subset --csv --fields "text","created_at","user.screen_name","user.followers_count" --out sept8.csv
mongoexport --db Oct2012 --collection sept8subset --csv --fields text,created_at,user.screen_name,user.followers_count --out sept8.csv
mongoexport -d Oct2012 -c sept8subset --csv --fields text,created_at,user.screen_name -o sept8.csv
mongoexport --db Oct2012 --collection sept8subset --dbpath ~/db (should need dbpath as mongod instance is running)
mongoexport --db OCt2012 -collection sept8subset -o sept8.txt
mongoexport --db Oct2012 --collection sept8subset
In each case, I get "JavaScript execution failed: SyntaxError: Unexpected identifier". Where could that SyntaxError be??
The collection I'm trying to copy has 50,339 objects. In case it is just too big for mongoexport to handle, I took 5 documents out of the collection to make a new collection. I then tried to export them using same command structure as above. I still get the same error message.
Now I'm wondering if the problem is that mongoexport can't work with data involving dates. MongoDB documentation states that I may want to write my own export script using a client driver.
Does anyone have an idea of what my problem is here? Many thanks if you can help out.
I found out that running mongoexport from the mongo client--as I was doing--is incorrect. Mongoexport is executed at the command prompt.
I ran into this too and I solved it executing the command from the path/bin where 'mongoexport' is located.In my case :
my_path: /usr/local/Cellar/mongodb/2.4.4-x86_64/bin
"mongoexport -d book -c shelf"
'mongod' is located one step upper from 'mongoimport' , 'mongoexport', etc...

How to export collection to CSV in MongoDB?

How do you export all the records in a MongoDB collection to a .csv file?
mongoexport --host localhost --db dbname --collection name --type=csv > test.csv
This asks me to specify name of the fields I need to export. Can I just export all the fields without specifying the names of fields?
#karoly-horvath has it right. Fields are required for csv.
According to this bug in the MongoDB issue tracker https://jira.mongodb.org/browse/SERVER-4224 you MUST provide the fields when exporting to a csv. The docs are not clear on it. That is the reason for the error.
Try this:
mongoexport --host localhost --db dbname --collection name --csv --out text.csv --fields firstName,middleName,lastName
UPDATE:
This commit: https://github.com/mongodb/mongo-tools/commit/586c00ef09c32c77907bd20d722049ed23065398 fixes the docs for 3.0.0-rc10 and later. It changes
Fields string `long:"fields" short:"f" description:"comma separated list of field names, e.g. -f name,age"`
to
Fields string `long:"fields" short:"f" description:"comma separated list of field names (required for exporting CSV) e.g. -f \"name,age\" "`
VERSION 3.0 AND ABOVE:
You should use --type=csv instead of --csv since it has been deprecated.
More details: https://docs.mongodb.com/manual/reference/program/mongoexport/#export-in-csv-format
Full command:
mongoexport --host localhost --db dbname --collection name --type=csv --out text.csv --fields firstName,middleName,lastName
Also, you are not allowed spaces between comma separated field names.
BAD:
-f firstname, lastname
GOOD:
-f firstname,lastname
mongoexport --help
....
-f [ --fields ] arg comma separated list of field names e.g. -f name,age
--fieldFile arg file with fields names - 1 per line
You have to manually specify it and if you think about it, it makes perfect sense. MongoDB is schemaless; CSV, on the other hand, has a fixed layout for columns. Without knowing what fields are used in different documents it's impossible to output the CSV dump.
If you have a fixed schema perhaps you could retrieve one document, harvest the field names from it with a script and pass it to mongoexport.
If you want, you can export all collections to csv without specifying --fields (will export all fields).
From http://drzon.net/export-mongodb-collections-to-csv-without-specifying-fields/ run this bash script
OIFS=$IFS;
IFS=",";
# fill in your details here
dbname=DBNAME
user=USERNAME
pass=PASSWORD
host=HOSTNAME:PORT
# first get all collections in the database
collections=`mongo "$host/$dbname" -u $user -p $pass --eval "rs.slaveOk();db.getCollectionNames();"`;
collections=`mongo $dbname --eval "rs.slaveOk();db.getCollectionNames();"`;
collectionArray=($collections);
# for each collection
for ((i=0; i<${#collectionArray[#]}; ++i));
do
echo 'exporting collection' ${collectionArray[$i]}
# get comma separated list of keys. do this by peeking into the first document in the collection and get his set of keys
keys=`mongo "$host/$dbname" -u $user -p $pass --eval "rs.slaveOk();var keys = []; for(var key in db.${collectionArray[$i]}.find().sort({_id: -1}).limit(1)[0]) { keys.push(key); }; keys;" --quiet`;
# now use mongoexport with the set of keys to export the collection to csv
mongoexport --host $host -u $user -p $pass -d $dbname -c ${collectionArray[$i]} --fields "$keys" --csv --out $dbname.${collectionArray[$i]}.csv;
done
IFS=$OIFS;
works for me remoting to a docker container with mongo:4.2.6
mongoexport -h mongodb:27017 --authenticationDatabase=admin -u username -p password -d database -c collection -q {"created_date": { "$gte": { "$date": "2020-08-03T00:00:00.000Z" }, "$lt": { "$date": "2020-08-09T23:59:59.999Z" } } } --fields=somefield1,somefield2 --type=csv --out=/archive.csv
Easy export csv or json file With Mongo Compass tool
Mongo Compass As the GUI for MongoDB, MongoDB Compass allows you to make smarter decisions about document structure, querying, indexing, document validation, and more. Commercial subscriptions include technical support for MongoDB Compass.
https://www.mongodb.com/try/download/compass
I could not get mongoexport to do this for me. I found that,to get an exhaustive list of all the fields, you need to loop through the entire collection once. Use this to generate the headers. Then loop through the collection again to populate these headers for each document.
I've written a script to do just this. Converting MongoDB docs to csv irrespective of schema differences between individual documents.
https://github.com/surya-shodan/mongoexportcsv
Also if you want to export inner json fields use dot (. operator).
JSON record:
{
"_id" : "00118685076F2C77",
"value" : {
"userIds" : [
"u1"
],
"deviceId" : "dev"
}
mongoexport command with dot operator (using mongo version 3.4.7):
./mongoexport --host localhost --db myDB --collection myColl
--type=csv --out out.csv --fields value.deviceId,value.userIds
Output csv:
value.deviceId,value.userIds
d1,"[""u1""]"
d2,"[""u2""]"
Note: Make sure you do not export an array. It would corrupt the CSV format like field userIds shown above
Solution for MongoDB Atlas users!
Add the --fields parameter as comma separated field names enclosed in double inverted quotes:
--fields "<FIELD 1>,<FIELD 2>..."
This is complete example:
mongoexport --host Cluster0-shard-0/shard1URL.mongodb.net:27017,shard2URL.mongodb.net:27017,shard3URL.mongodb.net:27017 --ssl --username <USERNAME> --password <PASSWORD> --authenticationDatabase admin --db <DB NAME> --collection <COLLECTION NAME> --type <OUTPUT FILE TYPE> --out <OUTPUT FILE NAME> --fields "<FIELD 1>,<FIELD 2>..."
This working for me Try it
mongoexport --host cluster0-shard-dummy-link.mongodb.net:27017 --db yourdbname --forceTableScan --collection users --type json --out /var/www/html/user.json --authenticationDatabase admin --ssl --username Yourusername --password Yourpassword
Above cmd return whole data of the users collection
if you want filter field then add --fields=email,name
For all those who are stuck with an error.
Let me give you guys a solution with a brief explanation of the same:-
command to connect:-
mongoexport --host your_host --port your_port -u your_username -p your_password --db your_db --collection your_collection --type=csv --out file_name.csv --fields all_the_fields --authenticationDatabase admin
--host --> host of Mongo server
--port --> port of Mongo server
-u --> username
-p --> password
--db --> db from which you want to export
--collection --> collection you want to export
--type --> type of export in my case CSV
--out --> file name where you want to export
--fields --> all the fields you want to export (don't give spaces in between two field name in between commas in case of CSV)
--authenticationDatabase --> database where all your user information is stored
Below command used to export collection to CSV format.
Note: naag is database, employee1_json is a collection.
mongoexport --db naag--collection employee1_json --type csv --out /home/orienit/work/mongodb/employee1_csv_op1