How to import all files from a folder to MongoDB? - mongodb

I have a folder containing CSVfiles. I want to upload the data of all the CSVs to MongoDB. I have tried the following command:
for i in *.csv; do mongoimport -d mydatabase -c ${i%.*} --type csv --file $i --headerline ; done
I have modified it to suit my scenario. The modified command is as follows:
for i in "C:\Users\lenovo-pc\Desktop\Testing sample csv\*.csv"; do mongoimport -d Better -c TESTCSV --type csv --file $i --headerline ; done
But It is giving error: i was unexpected at this time.
I would like to know how I can upload all the CSVs from a folder at once. I do not want to upload them one by one. Kindly help.

Try this:
for %i in (*.csv) do mongoimport -d Better -c TESTCSV --type csv --file "%i" --headerline
Make sure you run this from the directory in which the *.csv files are present.

I've made this script for Windows that imports all CSVs in the same folder and names each collection with the name of the respective CSV.
You need to copy these lines in a .bat file, then edit the variables MONGO_HOME and db as you need:
TITLE MongoDB CSV Importer
SET "MONGO_HOME=C:\Program Files\MongoDB\Server\3.6"
SET db=datasets
for %%v in (*.csv) do "%MONGO_HOME%\bin\mongoimport.exe" -d %db% -c %%~nv --type CSV --file %%v --headerline
TITLE "Import completed!"
PAUSE
and this works the same for Linux shell scripts (.sh):
db="datasets"
for entry in *".csv"
do
coll=$(echo "$entry" | cut -f 1 -d '.')
echo $name
mongoimport -d $db -c $coll --type CSV --file $coll".csv" --headerline
done
echo "end of import."
Hope this helps

Related

MongoDB: add a date using mongoimport with --columnsHaveTypes --headerline

I have a test.tsv TSV file that has a header looking like this
sample.string() organism.string() capture.string() sex.string()
and I am using the following command to import my file into mongodb
mongoimport --quiet -d somedb --collection=somecollection --file=test.tsv --type tsv --mode upsert --upsertFields sample --columnsHaveTypes --headerline
I wish to add the date following mongodb format at the moment of my file import.
I found a way to do it while in mongodb but I can't seem to find anything about mongoimport.
Is there a command (or a workaround) that looks like the following and allows us to add the date of the file content ?
mongoimport --quiet -d somedb --collection=somecollection --file=test.tsv --type tsv --mode upsert --upsertFields sample --columnsHaveTypes --headerline --addDate
Thanks in advance.
I found a workaround for this:
First, get the date from bash with some formatting:
date=$(date -u +"%d-%m-%Y %T")
Now, add this variable to all the lines in file.tsv like so:
awk -v date="$date" '{print $0 "\t" date}' file.tsv
Then, we can change the header by adding the date format type
$ sed -i '1s/.*/Sample.string()\tOrganism.string()\tCapture.string()\tSex.string()\tDate.date(02-01-2006 15:04:05)/' file.tsv
$ head file.tsv
Sample.string() Organism.string() Capture.string() Sex.string() Date.date(02-01-2006 15:04:05)
Then, we can import with mongoimport:
$ mongoimport -d somedb --collection=somecollection --file=file.tsv --type tsv --mode upsert --upsertFields Sample --columnsHaveTypes --headerline
2019-09-30T11:23:37.934+0200 connected to: xxxx
2019-09-30T11:23:37.940+0200 imported 1 document

How to import CSV file to MongoDB on Windows?

The mongoshell shows the result of the mongoimport command while trying to import a CSV file into MongoDB:
I wanted to import a csv file into MongoDB. So I have used the command:
mongoimport -d dbname -c collectionname --type csv --file filename.csv --headerline
after executing this command I constantly got an error of missing ; before statement. I stored the file in the path mongo\bin only. Is the command itself wrong or should I save the file any other location than I have saved it now? Can anyone give a way to fix this command?
mongoimport is a binary that runs from OS shell and not the mongo shell.
So
C:\User\Home>mongoimport -d dbname -c collectionname --type csv --file filename.csv --headerline
is correct while
mongo> mongoimport -d dbname -c collectionname --type csv --file filename.csv --headerline
is not.

mongoDB mongoimport error .. getting file doesn't exist error

I am a newbie in Mongodb. when i am trying to mongoimport i am getting the below error.
I have the file placed in c:\mongo\data\db\mongo.csv ... can anyone pls help me.
C:\mongodb\bin>mongoimport.exe -d test -c foo --file c:\mongo\data\db\mongo.csv --type csv
connected to: 127.0.0.1
file doesn't exist: c:\mongo\data\db\mongo.csv
either you should give path like as you are in C directive only
C:\mongodb\bin>mongoimport.exe -d test -c foo --file /mongo/data/db/mongo.csv --type csv
or you can give path in a qoute (" ") as mentioned by Gianpj
C:\mongodb\bin>mongoimport.exe -d test -c foo --file "c:\mongo\data\db\mongo.csv" --type csv
Are you sure the file does exist? There are no spaces in the file path?
Try with double quotes:
--file "c:\mongo\data\db\mongo.csv"
Lastly where did got that .csv file? from mongoexport --csv ?

MongoDB bulk csv imports (folder full of csv)

I have tried to import a single csv file in mongodb using mongoimport using mongoimport -d mydb -c things --type csv --file locations.csv --headerline. It works like a charm. My problem is that I want to import a folder full of csv(s) in mongodb. I searched, but could not find anything on that. I also tried to give folder path with wildcard (*), but it does not accept that. An example path is /home/user/event_files/* where event_files is the folder containing the csv files. How can I accomplish the import of a folder full of csv(s) in mongodb ?
Thankyou,
Mohsin
EDIT:
#!bin/bash
FILES="/root/event_files/*"
for f in $FILES
do
mongoimport -d mydb -c events --type csv --file "$f" --headerline
done
I made this script to do the job. Replace your own folder path in "FILES" variable.
In the mongoimport command (between do and done), "mydb" is your database name, "events" is your collection name.
You will need to replace things to suit your needs.
Thankyou,
Mohsin.
This worked for me. Loops through the files in event_files and imports each csv using the header line as the data field in mongo.
#!bin/bash
FILES="/root/event_files/*"
for f in $FILES
do
mongoimport -d mydb -c events --type csv --file "$f" --headerline
done

How to use mongoimport to import CSV files?

CSV file with contact information:
Name,Address,City,State,ZIP
Jane Doe,123 Main St,Whereverville,CA,90210
John Doe,555 Broadway Ave,New York,NY,10010
Running this doesn't add documents to the database:
$ mongoimport -d mydb -c things --type csv --file locations.csv --headerline
Trace says imported 1 objects, but in the MongoDB shell running db.things.find() doesn't show any new documents.
What am I missing?
Your example worked for me with MongoDB 1.6.3 and 1.7.3. Example below was for 1.7.3. Are you using an older version of MongoDB?
$ cat > locations.csv
Name,Address,City,State,ZIP
Jane Doe,123 Main St,Whereverville,CA,90210
John Doe,555 Broadway Ave,New York,NY,10010
ctrl-d
$ mongoimport -d mydb -c things --type csv --file locations.csv --headerline
connected to: 127.0.0.1
imported 3 objects
$ mongo
MongoDB shell version: 1.7.3
connecting to: test
> use mydb
switched to db mydb
> db.things.find()
{ "_id" : ObjectId("4d32a36ed63d057130c08fca"), "Name" : "Jane Doe", "Address" : "123 Main St", "City" : "Whereverville", "State" : "CA", "ZIP" : 90210 }
{ "_id" : ObjectId("4d32a36ed63d057130c08fcb"), "Name" : "John Doe", "Address" : "555 Broadway Ave", "City" : "New York", "State" : "NY", "ZIP" : 10010 }
I was perplexed with a similar problem where mongoimport did not give me an error but would report importing 0 records. I had saved my file that didn't work using the OSX Excel for Mac 2011 version using the default "Save as.." "xls as csv" without specifying "Windows Comma Separated(.csv)" format specifically. After researching this site and trying the "Save As again using "Windows Comma Separated (.csv)" format, mongoimport worked fine. I think mongoimport expects a newline character on each line and the default Mac Excel 2011 csv export didn't provide that character at the end of each line.
We need to execute the following command:
mongoimport --host=127.0.0.1 -d database_name -c collection_name --type csv --file csv_location --headerline
-d is database name
-c is collection name
--headerline If using --type csv or --type tsv, uses the first line as field names. Otherwise, mongoimport will import the first line as a distinct document.
For more information: mongoimport
you will most likely need to authenticate if you're working in production sort of environments. You can use something like this to authenticate against the correct database with appropriate credentials.
mongoimport -d db_name -c collection_name --type csv --file filename.csv --headerline --host hostname:portnumber --authenticationDatabase admin --username 'iamauser' --password 'pwd123'
I use this on mongoimport shell
mongoimport --db db_name --collection collection_name --type csv --file C:\\Your_file_path\target_file.csv --headerline
type can choose csv/tsv/json
But only csv/tsv can use --headerline
You can read more on the offical doc.
Check that you have a blank line at the end of the file, otherwise the last line will be ignored on some versions of mongoimport
When I was trying to import the CSV file, I was getting an error. What I have done.
First I changed the header line's column names in Capital letter and removed "-" and added "_" if needed. Then Typed below command for importing CSV into mongo
$ mongoimport --db=database_name --collection=collection_name --type=csv --file=file_name.csv --headerline
Robert Stewart have already answered for how to import with mongoimport.
I am suggesting easy way to import CSV elegantly with 3T MongoChef Tool (3.2+ version). Might help someone in future.
You just need to select collection
Select file to import
You can also unselect data which is going to import. Also many options are there.
Collection imported
See how to import video
First you should come out of the mongo shell and then execute the mongoimport command like this:
Manojs-MacBook-Air:bin Aditya$ mongoimport -d marketdata -c minibars
--type csv
--headerline
--file '/Users/Aditya/Downloads/mstf.csv'
2017-05-13T20:00:41.989+0800 connected to: localhost
2017-05-13T20:00:44.123+0800 imported 97609 documents
Manojs-MacBook-Air:bin Aditya$
Robert Stewart's answers is great.
I'd like to add that you also can type your fields with --columHaveTypes and --fields like this :
mongoimport -d myDb -c myCollection --type csv --file myCsv.csv
--columnsHaveTypes --fields "label.string(),code.string(),aBoolean.boolean()"
(Careful to not have any space after the comma between your fields)
For other types, see doc here : https://docs.mongodb.com/manual/reference/program/mongoimport/#cmdoption-mongoimport-columnshavetypes
For the 3.4 version, please use the following syntax:
mongoimport -u "username" -p "password" -d "test" -c "collections" --type csv --file myCsv.csv --headerline
After 3 days, I finally made it on my own. Thanks to all the users who supported me.
My requirement was to import the .csv (with no headline) to remote MongoDB instance. For mongoimport v3.0.7below command worked for me:
mongoimport -h <host>:<port> -u <db-user> -p <db-password> -d <database-name> -c <collection-name> --file <csv file location> --fields <name of the columns(comma seperated) in csv> --type csv
For example:
mongoimport -h 1234.mlab.com:61486 -u arpitaggarwal -p password -d my-database -c employees --file employees.csv --fields name,email --type csv
Below is the screenshot of how it looks like after import:
where name and email are the columns in the .csv file.
Given .csv file I have which has only one column with no Header, below command worked for me:
mongoimport -h <mongodb-host>:<mongodb-port> -u <username> -p <password> -d <mongodb-database-name> -c <collection-name> --file file.csv --fields <field-name> --type csv
where field-name refers to the Header name of the column in .csv file.
C:\wamp\mongodb\bin>mongoexport --db proj_mmm --collection offerings --csv --fieldFile offerings_fields.txt --out offerings.csv
Just use this after executing mongoimport
It will return number of objects imported
use db
db.collectionname.find().count()
will return the number of objects.
use :
mongoimport -d 'database_name' -c 'collection_name' --type csv --headerline --file filepath/file_name.csv
mongoimport -d test -c test --type csv --file SampleCSVFile_119kb.csv --headerline
check collection data:-
var collections = db.getCollectionNames();
for(var i = 0; i< collections.length; i++)
{
print('Collection: ' + collections[i]);
// print the name of each collection
db.getCollection(collections[i]).find().forEach(printjson);
//and then print the json of each of its elements
}
1]We can save xsl as .csv file
2] Got to MongoDB bin pathon cmd - > cd D:\Arkay\soft\MongoDB\bin
3] Run below command
> mongoimport.exe -d dbname -c collectionname --type csv --file "D:\Arkay\test.csv" --headerline
4] Verify on Mongo side using below coomand.
>db.collectioname.find().pretty().limit(1)
Strangely no one mentioned --uri flag:
mongoimport --uri connectionString -c questions --type csv --file questions.csv --headerline
Sharing for future readers:
In our case, we needed to add the host parameter to make it work
mongoimport -h mongodb://someMongoDBhostUrl:somePORTrunningMongoDB/someDB -d someDB -c someCollection -u someUserName -p somePassword --file someCSVFile.csv --type csv --headerline --host=127.0.0.1
Make sure to copy the .csv file to /usr/local/bin or whatever folder your mondodb is in
All these answers above are great. And the way to go on a full featured application.
But if you want to prototype fast, want flexibility as the collection still changes as well as to minimize your early code base, there is a much simpler way that is not much discussed.
You can basically forego mongoimport by now. I could have saved 3 hours if it was mentioned here on this question. So let me share for others:
Mongodb has a GUI called Mongo Compass has both csv and json import features out of the box in a matter of clicks. It is an official part of the Mongo ecosytem. At the time of writing it is free and it works very well for my use case.
https://www.mongodb.com/products/compass
You simply get MongoDB compass running on your machine by following the simple installation. A couple of fields for DB connection and authentication directly in the GUI.
Import the csv/json file. It took less than a second on a 30KB file to be parsed before user (me) validates.
Validate the "type" of each property. Great feature, I could directly mention the property types such as booleans, integers, etc. In my experience, they seem all default to string. You can update before importing. Dates were more finicky and needed special attention on the coding side.
One click further the csv is a collection in your mongo db local or on the cloud. Voila!
If you have multiple files and you want to import all of them using python, you can do the following.
import os
import subprocess
# directory of files
dir_files = 'C:\data'
# create list of all files
_, _, fns = next(os.walk(dir_files))
files = [os.path.join(dir_files, fn) for fn in fns]
# mongotool address
mongotool = r'C:\Program Files\MongoDB\Server\4.4\bin\mongoimport.exe'
# name of mongodb database
mydatabase = 'mydatabase'
# name of mongodb collection
mycollection = 'mycollection'
# import all files to mongodb
for fl in files:
commands =[mongotool, '--db', mydatabase,
'--collection', mycollection,
'--file', fl,
'--type', 'tsv',
'--headerline']
subprocess.Popen(commands, shell=True)