Is there a way from the command line that I can dump a MongoDb database to a javascript file that can be interpreted by the mongo shell? I am looking for a way to do exactly what the RockMongo Export function does, but I need to be able to call it from a command line script. I've looked everywhere for something that does this but all I can seem to find is mongoexport and mongodump which don't seem to do what I want, as these just create JSON files.
The reason I need to do this is because codeception's MongoDb module requires a file in this format to restore the database after each test. I want to write a script to automate this process so that I don't have to constantly go through RockMongo and generate the dump.
Thanks in advance!
In case anyone else happens to find this, I finally found a solution that works for my scenario. I had to take Markus' suggestion and kind of roll my own solution, but I discovered a mongodb command called bsondump that made things much easier.
So in my script I first use mongodump to create a BSON file of my collection
mongodump --db mydb --collection mycollection --out - > mycollection.bson
I then use bsondump to convert that into JSON that can be used in Shell Mode
bsondump mycollection.bson > mycollection.json
Finally, I'm using PHP so in my PHP script I loop through that json file and wrap each line in an insert statement.
$lines = file('mycollection.json');
$inserts = [];
foreach($lines as $line)
{
$inserts[] = 'db.getCollection("mycollection").insert(' . trim($line) . ');' . PHP_EOL;
}
file_put_contents('output.js', $inserts);
I'm guessing there is probably a better way to do this, but so far this seems to be working nicely for me. Thanks for steering me in the right direction Markus!
Related
I'm pretty new to PostgreSQL, but I have to query some results using psql in an interactive command line session. I am connecting through a cluster and I would like to extract the output of the query into a file for further analysis.
The command I use to connect is psql -h hostname.with.dbs -p 5432 -U my-username and inside I do the query. But it is not clear to me how to pipe that into a file in my user folder in the machine used to connect to Presto.
If I have to add more details, let me know, as I am not an expert and might forgot to add important information. Thank you all!
I found a solution to that. Basically appending \g file_name; at the end of the query. It saves the file in the directory where I launched the command to connect to the database. I didn't try to add full path to the file name, but I assume it would work as well.
I am using MongoDB 2.2.2 for 32-bit Windows7 machine. I have a complex aggregation query in a .js file. I need to execute this file on the shell and direct the output to a CSV file. I ensure that the query returns a "flat" json (no nested keys), so it is inherently convertible to a neat csv.
I know about load() and eval(). eval() requires me to paste the whole query into the shell and allows only printjson() inside the script, while I need csv. And, the second way: load()..It prints the output on the screen, and again in json format.
Is there a way Mongo can do this conversion from json to csv? (I need csv file to prepare charts on the data). I am thinking:
1. Either mongo has a built-in command for this that I can't find right now.
2. Mongo can't do it for me; I can at most send the json output to a file which I then need to convert to csv myself.
3. Mongo can send the json output to a temporary collection, the contents of which can be easily mongoexported to csv format. But I think only map-reduce queries support output collections. Is that right? I need it for an aggregation query.
Thanks for any help :)
I know this question is old but I spend an hour trying to export a complex query to csv and I wanted to share my thoughts. First I couldn't get any of the json to csv converters to work (although this one looked promising). What I ended up doing was manually writing the csv file in my mongo script.
This is a simple version but essentially what I did:
print("name,id,email");
db.User.find().forEach(function(user){
print(user.name+","+user._id.valueOf()+","+user.email);
});
This I just piped the query to stdout
mongo test export.js > out.csv
where test is the name of the database I use.
Mongo's in-built export is working fine, unless you want to any data manipulation like format date, covert data types etc.
Following command works as charm.
mongoexport -h localhost -d databse -c collection --type=csv
--fields erpNum,orderId,time,status
-q '{"time":{"$gt":1438275600000}, "status":{"$ne" :"Cancelled"}}'
--out report.csv
Extending other answers:
I found #GEverding's answer most flexible. It also works with aggregation:
test_db.js
print("name,email");
db.users.aggregate([
{ $match: {} }
]).forEach(function(user) {
print(user.name+","+user.email);
}
});
Execute the following command to export results:
mongo test_db < ./test_db.js >> ./test_db.csv
Unfortunately, it adds additional text to the CSV file which requires processing the file before we can use it:
MongoDB shell version: 3.2.10
connecting to: test_db
But we can make mongo shell stop spitting out those comments and only print what we have asked for by passing the --quiet flag
mongo --quiet test_db < ./test_db.js >> ./test_db.csv
Here is what you can try:
print("id,name,startDate")
cursor = db.<collection_name>.find();
while (cursor.hasNext()) {
jsonObject = cursor.next();
print(jsonObject._id.valueOf() + "," + jsonObject.name + ",\"" + jsonObject.stateDate.toUTCString() +"\"")
}
Save that in a file, say "export.js". Run the following command:
mongo <host>/<dbname> -u <username> -p <password> export.js > out.csv
Have a look at
this
for outputing from mongo shell to file.
There is no support for outputing csv from mongos shell. You would have to write the javascript yourself or use one of the many converters available. Google "convert json to csv" for example.
Just weighing in here with a nice solution I have been using. This is similar to Lucky Soni's solution above in that it supports aggregation, but doesn't require hard coding of the field names.
cursor = db.<collection_name>.<my_query_with_aggregation>;
headerPrinted = false;
while (cursor.hasNext()) {
item = cursor.next();
if (!headerPrinted) {
print(Object.keys(item).join(','));
headerPrinted = true;
}
line = Object
.keys(item)
.map(function(prop) {
return '"' + item[prop] + '"';
})
.join(',');
print(line);
}
Save this as a .js file, in this case we'll call it example.js and run it with the mongo command line like so:
mongo <database_name> example.js --quiet > example.csv
I use the following technique. It makes it easy to keep the column names in sync with the content:
var cursor = db.getCollection('Employees.Details').find({})
var header = []
var rows = []
var firstRow = true
cursor.forEach((doc) =>
{
var cells = []
if (firstRow) header.push("employee_number")
cells.push(doc.EmpNum.valueOf())
if (firstRow) header.push("name")
cells.push(doc.FullName.valueOf())
if (firstRow) header.push("dob")
cells.push(doc.DateOfBirth.valueOf())
row = cells.join(',')
rows.push(row)
firstRow = false
})
print(header.join(','))
print(rows.join('\n'))
When executing a script in a remote server. Mongo will add its own logging output, which we might want to omit from our file.
--quiet option will only disable connection related logs. Not all mongo logs. In such case we might need to filter out unneeded lines manually. A Windows based example:
mongo dbname --username userName --password password --host replicaset/ip:port --quiet printDataToCsv.js | findstr /v "NETWORK" > data.csv
This will pipe the script output and use findstr to filter out any lines, which have NETWORK string in them. More information on findstr: https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/findstr
A Linux version of this would use grep.
I tried to import a simple json file using mongoimport and i get the following error
PER-MacBook-Pro:/AJ$ mongoimport --db test --collection samplePM --file /users/AJ/Documents/Development/ETLwork/Dummydata/Penguin_Players.json
2015-06-16T09:53:57.291-0400 connected to: localhost
2015-06-16T09:53:57.293-0400 Failed: error processing document #1: invalid character '\\' looking for beginning of object key string
2015-06-16T09:53:57.293-0400 imported 0 documents
Sample json file is as follows:
{
"position":"Right Wing",
"id":8465166,
"weight":200,
"height":"6' 0\"",
"imageUrl":"http://1.cdn.nhle.com/photos/mugs/8465166.jpg",
"birthplace":"Seria, BRN",
"age":37,
"name":"Craig Adams",
"birthdate":"April 26, 1977",
"number":27
},
{
"position":"Right Wing",
"id":8475761,
"weight":195,
"height":"6' 2\"",
"imageUrl":"http://1.cdn.nhle.com/photos/mugs/8475761.jpg",
"birthplace":"Gardena, CA, USA",
"age":23,
"name":"Beau Bennett",
"birthdate":"November 27, 1991",
"number":19
}
Am I doing something wrong here?
I was able to get away with using --jsonArray tag, giving the file full path, and modified it adding bracket at the beginning and at the end,
[{"my":"json","file":"imported"},{"my":"cool","file":"succeeded"}]
mongoimport --db myCoolDb --collection myCoolColl --file /path/to/my/imported/file.json --jsonArray
The comment about non "UTF-8" characters was helpful.
It seems like there is a problem with creating json documents using textedit in Mac. I could not find these non UTF-8 characters but i created the same file using vi test.json in mac shell. I pasted the contents, saved the file and used mongoimport. It works now.
Thanks
I got the same error while importing json data. Instead use the .bson data using mongorestore command.
mongorestore -d <db> -c <collection> <.bson-file>
Use --drop if you want to drop the existing data in the collection.
I was getting Failed: error processing document #112783: invalid character ',' looking for beginning of value because one of my objects was formatted improperly. Notice how "psychosurgery" is missing curly braces:
{
"word": "psychosurgeons",
"firstLetter": "p"
}
" psychosurgery",
{
"word": "psychosurgical",
"firstLetter": "p"
}
Since there are over 600,000 lines in the file I'm trying to import, this would have been tough to find manually.
So I ran the same mongoimport command with full verbosity (-vvvvv) enabled, and the script stopped right on the problematic object. See mongoimport --help for more info.
Hope this helps someone.
I got the same problem because I used texteditor on the mac. The solution was to convert the file to plain text. Make sure the extension ends in .json because texteditor wants to put .txt at the end.
Just open a text file, copy all the data to the newly created text file. While saving the text file select the option 'UTF-8' in the Encoding drop down and later change the text file to JSON or CSV by renaming.
Then import the file as usual as it is.
if you used mongoexport to download use mongoimport to upload
or if you use mongodump to download use mongorestore to upload
because i used to download with mongodump and tried to upload with mongoimport i was getting error processing document #1: invalid character '\u008c'error after i tried with mongorestore it was fine
Is there any method to insert CSV file into Mongo DB other than using mongoimport tool ? . I need to perform bulk insertions in mongoDB . I referred some sites and i found that there are some issues in using mongo import tool for importing large set of data.Please enlighten me how to insert CSV into mongoDB from application directly ? I need to know if there are any methods or wrappers in c++ or java for inserting CSV into MongoDB. Thanks in advance
the MongoDB shell is able to evaluate JavaScript. So you could write your own parser in JavaScript load your program into the shell and start it. If your don't like JavaScript, there are a lot of drivers for other programming languages where you could load your file and use the driver to insert your data into the database.
you can use bulk option available with mongo2.6
read the file
iterate and save to some variable
bulk.excecute.
mongoimport -c 'check' --db 'test' --type csv --file test.csv --headerline
use mongoimport --help for more help
(edit : -h option is for host)
test.csv
name, age, sex
John, 23, M
I am trying to log a complete session in psql into a .txt file. The command given to me was initially this:
psql db_name| tee file_name.txt
However, my SSH client does nothing until I quit it. That means, it does not recognize any command. More like a document, no action happens no matter what I write. So far, only '\q' is recognised which lets me get out of it. Any ideas what is happening? How am I to write the query if shell will not read anything. Also, I tried the following (this is before connecting to database) :
script filename.txt
It does show the message : script started, file is filename.txt, but I dont know where this file is stored and how to retrieve it.
Any help with the above will be welcome and really appreciated! Thanks a lot :)
There is option to psql for log query and results:
-L filename
--log-file filename
Write all query output into file filename, in addition to the normal output destination.
Try this:
psql db_name -L file_name.txt