When importing data from file (csv in my case) mongoimport automatically choose data type for each field.
Is it possible to choose data type manually for specific field?
I encountered situation, when in my file there are phone numbers, which I want and which I should treat as strings, but mongoimport (quite properly) treat those phone numbers as a numbers (NumberLong).
When importing CSV/TSV to mongodb, the option --columnsHaveTypes can help to define the columnstypes. But the document seems very unclear. I tried several times until finally did succeed.
You should add option --columnsHaveTypes and change every column after --fields and remember using "\" before "(" and ")".
for example, change:
mongoimport -h foohost -d bardb -c fooc --type tsv --fields col1,col2,col3 --file path/to/file.txt
into
mongoimport -h foohost -d bardb -c fooc --type tsv --fields col1.int32\(\),col2.double\(\),col3.string\(\) --columnsHaveTypes --file path/to/file.txt
What you can do is import these data using CSV and then run the update statement on the existing data in mongo db to convert it into the format that you want.
Now version 3.4 onward mongoimport supports specifying the field types explicitly while importing the data. See below link:
https://docs.mongodb.com/manual/reference/program/mongoimport/#cmdoption--columnsHaveTypes
See the Type Fidelity section in the documentation:
mongoimport and mongoexport do not reliably preserve all rich BSON
data types because JSON can only represent a subset of the types
supported by BSON. As a result, data exported or imported with these
tools may lose some measure of fidelity. See MongoDB Extended JSON for
more information.
Use mongodump and mongorestore to preserve types.
When I tried to import CSV into Mongo Atlas, I ran into a similar issue. Here's how I deal with it.
To avoid shell error you can enclose fields in double-quotes.
In the below example, I used two-column "Name, Barcode".You Can use whatever column you need also don't forget to update <connecttionString>,<collectionName>, <CSVpath> with your own values.
for more mongo types refer to mongoimport documentation.
mongoimport --uri <connecttionString> --collection <collectionName> --type csv --file <CSVpath> -f "Name.string(),Barcode.string()" --columnsHaveTypes
You can also choose to put the column types in a field file to make it easier. Just make sure you have specified all columns in your field file.
In my case, I named it "field.txt".
In the field file, you write the columns with their types this way: <column>.<type>. To get the list of all types used in the mongoimport syntax, please visit https://www.mongodb.com/docs/database-tools/mongoimport/
field.txt
name.string()
usercode.int64()
city.string()
town.string()
address.string()
price.decimal()
date_created.date_go(2021-08-10 15:04:05)
You can choose to name it anything you want as long as you point the fieldFile to it. eg. fieldFile=myfieldname.txt
mongoimport --uri <connectionString> --collection <collectionName> --type csv --file <csv path> --columnsHaveTypes --fieldFile=field.txt --mode=insert
How do I export the Collection from Mongo DB to xsl file.
I am using MongoVue export, but it is just displaying the Document and Number of keys in it, but not all the Key and Value pairs in the document..
Please help how to export a mongodb collection to xsl.
I'm not sure of using MongoVue (I haven't used that before) but if you are ok with an extra manual step you could always use mongoexport to output to a CSV file. You can then open and save as XSL via excel.
Here's the syntax for using mongoexport:
mongoexport -d <database> -c <collection> --csv --fields <field1,field2,...> -o <filename>
This will use the localhost mongo instance by defaykt so if you need to connect to another server see the docs:
http://docs.mongodb.org/manual/reference/program/mongoexport/
I am getting an error after executing this command:
mongoexport --db records --collection source_list --csv --out C:\bcopy.csv
record is my DB n source_list is my collection
It displays this message:
assertion: 9998 you need to specify fields
I also tried to specify fields but it is giving me the same error.
What changes should i make in the command to get a backup of my collection or is there any other way to do so ?
Here's sample command that specifies fields to export:
mongoexport -h 127.0.0.1 --port 27018 --db mydb --collection system.profile --csv --out profile.csv --fields ns,millis,numYield,nscanned
In my case --headerline helped. I had around 60 columns, enumerating them with -f would be quite cumbersome.
--headerline
If using “--type csv” or “--type tsv,” use the first line as field names. Otherwise, > mongoimport will import the first line as a distinct document.
Seems like you should be using -f paramater to choose the fields that will be exported to csv file. There is a bug reported for this case to change the explanation as the error message is not informative enough.
https://jira.mongodb.org/browse/SERVER-4224
I have a simple script to get an export from MongoDB:
#!/bin/sh -x
QUERY="'{ \"type\":\"listing\" }'"
mongoexport --db event --collection listing --query $QUERY --fields type,name --csv
(I'm using the -x switch for debugging purposes)
Here's the output when I run the script:
$ ./simple_query.sh
+ QUERY='{ "type":"listing" }'
+ mongoexport --db event --collection listing --query '{ "type":"listing" }' --fields type,name --csv
ERROR: too many positional options
However, if I merely copy and paste the mongoexport line to the prompt, it works! What's going on?
(Using MongoDB 2.0.0)
Are you using older version of mongodb? If so maybe it's this bug. It is fixed on 2.0.0-* versions.
Edit: Actually this might be different problem. You can fix it by removing the spaces on the query. Spaces seem to make the shell think it is actually multiple arguments:
QUERY="'{\"type\":\"listing\"}'"
CSV file with contact information:
Name,Address,City,State,ZIP
Jane Doe,123 Main St,Whereverville,CA,90210
John Doe,555 Broadway Ave,New York,NY,10010
Running this doesn't add documents to the database:
$ mongoimport -d mydb -c things --type csv --file locations.csv --headerline
Trace says imported 1 objects, but in the MongoDB shell running db.things.find() doesn't show any new documents.
What am I missing?
Your example worked for me with MongoDB 1.6.3 and 1.7.3. Example below was for 1.7.3. Are you using an older version of MongoDB?
$ cat > locations.csv
Name,Address,City,State,ZIP
Jane Doe,123 Main St,Whereverville,CA,90210
John Doe,555 Broadway Ave,New York,NY,10010
ctrl-d
$ mongoimport -d mydb -c things --type csv --file locations.csv --headerline
connected to: 127.0.0.1
imported 3 objects
$ mongo
MongoDB shell version: 1.7.3
connecting to: test
> use mydb
switched to db mydb
> db.things.find()
{ "_id" : ObjectId("4d32a36ed63d057130c08fca"), "Name" : "Jane Doe", "Address" : "123 Main St", "City" : "Whereverville", "State" : "CA", "ZIP" : 90210 }
{ "_id" : ObjectId("4d32a36ed63d057130c08fcb"), "Name" : "John Doe", "Address" : "555 Broadway Ave", "City" : "New York", "State" : "NY", "ZIP" : 10010 }
I was perplexed with a similar problem where mongoimport did not give me an error but would report importing 0 records. I had saved my file that didn't work using the OSX Excel for Mac 2011 version using the default "Save as.." "xls as csv" without specifying "Windows Comma Separated(.csv)" format specifically. After researching this site and trying the "Save As again using "Windows Comma Separated (.csv)" format, mongoimport worked fine. I think mongoimport expects a newline character on each line and the default Mac Excel 2011 csv export didn't provide that character at the end of each line.
We need to execute the following command:
mongoimport --host=127.0.0.1 -d database_name -c collection_name --type csv --file csv_location --headerline
-d is database name
-c is collection name
--headerline If using --type csv or --type tsv, uses the first line as field names. Otherwise, mongoimport will import the first line as a distinct document.
For more information: mongoimport
you will most likely need to authenticate if you're working in production sort of environments. You can use something like this to authenticate against the correct database with appropriate credentials.
mongoimport -d db_name -c collection_name --type csv --file filename.csv --headerline --host hostname:portnumber --authenticationDatabase admin --username 'iamauser' --password 'pwd123'
I use this on mongoimport shell
mongoimport --db db_name --collection collection_name --type csv --file C:\\Your_file_path\target_file.csv --headerline
type can choose csv/tsv/json
But only csv/tsv can use --headerline
You can read more on the offical doc.
Check that you have a blank line at the end of the file, otherwise the last line will be ignored on some versions of mongoimport
When I was trying to import the CSV file, I was getting an error. What I have done.
First I changed the header line's column names in Capital letter and removed "-" and added "_" if needed. Then Typed below command for importing CSV into mongo
$ mongoimport --db=database_name --collection=collection_name --type=csv --file=file_name.csv --headerline
Robert Stewart have already answered for how to import with mongoimport.
I am suggesting easy way to import CSV elegantly with 3T MongoChef Tool (3.2+ version). Might help someone in future.
You just need to select collection
Select file to import
You can also unselect data which is going to import. Also many options are there.
Collection imported
See how to import video
First you should come out of the mongo shell and then execute the mongoimport command like this:
Manojs-MacBook-Air:bin Aditya$ mongoimport -d marketdata -c minibars
--type csv
--headerline
--file '/Users/Aditya/Downloads/mstf.csv'
2017-05-13T20:00:41.989+0800 connected to: localhost
2017-05-13T20:00:44.123+0800 imported 97609 documents
Manojs-MacBook-Air:bin Aditya$
Robert Stewart's answers is great.
I'd like to add that you also can type your fields with --columHaveTypes and --fields like this :
mongoimport -d myDb -c myCollection --type csv --file myCsv.csv
--columnsHaveTypes --fields "label.string(),code.string(),aBoolean.boolean()"
(Careful to not have any space after the comma between your fields)
For other types, see doc here : https://docs.mongodb.com/manual/reference/program/mongoimport/#cmdoption-mongoimport-columnshavetypes
For the 3.4 version, please use the following syntax:
mongoimport -u "username" -p "password" -d "test" -c "collections" --type csv --file myCsv.csv --headerline
After 3 days, I finally made it on my own. Thanks to all the users who supported me.
My requirement was to import the .csv (with no headline) to remote MongoDB instance. For mongoimport v3.0.7below command worked for me:
mongoimport -h <host>:<port> -u <db-user> -p <db-password> -d <database-name> -c <collection-name> --file <csv file location> --fields <name of the columns(comma seperated) in csv> --type csv
For example:
mongoimport -h 1234.mlab.com:61486 -u arpitaggarwal -p password -d my-database -c employees --file employees.csv --fields name,email --type csv
Below is the screenshot of how it looks like after import:
where name and email are the columns in the .csv file.
Given .csv file I have which has only one column with no Header, below command worked for me:
mongoimport -h <mongodb-host>:<mongodb-port> -u <username> -p <password> -d <mongodb-database-name> -c <collection-name> --file file.csv --fields <field-name> --type csv
where field-name refers to the Header name of the column in .csv file.
C:\wamp\mongodb\bin>mongoexport --db proj_mmm --collection offerings --csv --fieldFile offerings_fields.txt --out offerings.csv
Just use this after executing mongoimport
It will return number of objects imported
use db
db.collectionname.find().count()
will return the number of objects.
use :
mongoimport -d 'database_name' -c 'collection_name' --type csv --headerline --file filepath/file_name.csv
mongoimport -d test -c test --type csv --file SampleCSVFile_119kb.csv --headerline
check collection data:-
var collections = db.getCollectionNames();
for(var i = 0; i< collections.length; i++)
{
print('Collection: ' + collections[i]);
// print the name of each collection
db.getCollection(collections[i]).find().forEach(printjson);
//and then print the json of each of its elements
}
1]We can save xsl as .csv file
2] Got to MongoDB bin pathon cmd - > cd D:\Arkay\soft\MongoDB\bin
3] Run below command
> mongoimport.exe -d dbname -c collectionname --type csv --file "D:\Arkay\test.csv" --headerline
4] Verify on Mongo side using below coomand.
>db.collectioname.find().pretty().limit(1)
Strangely no one mentioned --uri flag:
mongoimport --uri connectionString -c questions --type csv --file questions.csv --headerline
Sharing for future readers:
In our case, we needed to add the host parameter to make it work
mongoimport -h mongodb://someMongoDBhostUrl:somePORTrunningMongoDB/someDB -d someDB -c someCollection -u someUserName -p somePassword --file someCSVFile.csv --type csv --headerline --host=127.0.0.1
Make sure to copy the .csv file to /usr/local/bin or whatever folder your mondodb is in
All these answers above are great. And the way to go on a full featured application.
But if you want to prototype fast, want flexibility as the collection still changes as well as to minimize your early code base, there is a much simpler way that is not much discussed.
You can basically forego mongoimport by now. I could have saved 3 hours if it was mentioned here on this question. So let me share for others:
Mongodb has a GUI called Mongo Compass has both csv and json import features out of the box in a matter of clicks. It is an official part of the Mongo ecosytem. At the time of writing it is free and it works very well for my use case.
https://www.mongodb.com/products/compass
You simply get MongoDB compass running on your machine by following the simple installation. A couple of fields for DB connection and authentication directly in the GUI.
Import the csv/json file. It took less than a second on a 30KB file to be parsed before user (me) validates.
Validate the "type" of each property. Great feature, I could directly mention the property types such as booleans, integers, etc. In my experience, they seem all default to string. You can update before importing. Dates were more finicky and needed special attention on the coding side.
One click further the csv is a collection in your mongo db local or on the cloud. Voila!
If you have multiple files and you want to import all of them using python, you can do the following.
import os
import subprocess
# directory of files
dir_files = 'C:\data'
# create list of all files
_, _, fns = next(os.walk(dir_files))
files = [os.path.join(dir_files, fn) for fn in fns]
# mongotool address
mongotool = r'C:\Program Files\MongoDB\Server\4.4\bin\mongoimport.exe'
# name of mongodb database
mydatabase = 'mydatabase'
# name of mongodb collection
mycollection = 'mycollection'
# import all files to mongodb
for fl in files:
commands =[mongotool, '--db', mydatabase,
'--collection', mycollection,
'--file', fl,
'--type', 'tsv',
'--headerline']
subprocess.Popen(commands, shell=True)