Printing Mongo query output to a file while in the mongo shell - mongodb

2 days old with Mongo and I have a SQL background so bear with me. As with mysql, it is very convenient to be in the MySQL command line and output the results of a query to a file on the machine. I am trying to understand how I can do the same with Mongo, while being in the shell
I can easily get the output of a query I want by being outside of the shell and executing the following command:
mongo localhost:27017/dbname --eval "printjson(db.collectionName.findOne())" > sample.json
The above way is fine, but it requires me to exit the mongo shell or open a new terminal tab to execute this command. It would be very convenient if I could simply do this while still being inside the shell.
P.S: the Question is an offshoot of a question I posted on SO

AFAIK, there is no a interactive option for output to file, there is a previous SO question related with this: Printing mongodb shell output to File
However, you can log all the shell session if you invoked the shell with tee command:
$ mongo | tee file.txt
MongoDB shell version: 2.4.2
connecting to: test
> printjson({this: 'is a test'})
{ "this" : "is a test" }
> printjson({this: 'is another test'})
{ "this" : "is another test" }
> exit
bye
Then you'll get a file with this content:
MongoDB shell version: 2.4.2
connecting to: test
> printjson({this: 'is a test'})
{ "this" : "is a test" }
> printjson({this: 'is another test'})
{ "this" : "is another test" }
> exit
bye
To remove all the commands and keep only the json output, you can use a command similar to:
tail -n +3 file.txt | egrep -v "^>|^bye" > output.json
Then you'll get:
{ "this" : "is a test" }
{ "this" : "is another test" }

We can do it this way -
mongo db_name --quiet --eval 'DBQuery.shellBatchSize = 2000; db.users.find({}).limit(2000).toArray()' > users.json
The shellBatchSize argument is used to determine how many rows is the mongo client allowed to print. Its default value is 20.

If you invoke the shell with script-file, db address, and --quiet arguments, you can redirect the output (made with print() for example) to a file:
mongo localhost/mydatabase --quiet myScriptFile.js > output

There are ways to do this without having to quit the CLI and pipe mongo output to a non-tty.
To save the output from a query with result x we can do the following to directly store the json output to /tmp/x.json:
> EDITOR="cat > /tmp/x.json"
> x = db.MyCollection.find(...).toArray()
> edit x
>
Note that the output isn't strictly Json but rather the dialect that Mongo uses.

In the new mongodb shell 5.0+ mongosh, it integrate the Node.js fs module, so you can simply do below in the new mongosh shell:
fs.writeFileSync('output.json', JSON.stringify(db.collectionName.findOne()))
This also avoid problems such as the ObjectId(...) being included in the tojson result, which is not valid JSON string.
The above code works according to the docs describes:
The MongoDB Shell, mongosh, is a fully functional JavaScript and Node.js 14.x REPL environment for interacting with MongoDB deployments. You can use the MongoDB Shell to test queries and operations directly with your database.
The old mongo shell already marked as Legacy, so use the mongosh if possible.

It may be useful to you to simply increase the number of results that get displayed
In the mongo shell > DBQuery.shellBatchSize = 3000
and then you can select all the results out of the terminal in one go and paste into a text file.
It is what I am going to do :)
(from : https://stackoverflow.com/a/3705615/1290746)

Combining several conditions:
write mongo query in JS file and send it from terminal
switch/define a database programmatically
output all found records
cut initial output lines
save the output into JSON file
myScriptFile.js
// Switch current database to "mydatabase"
db = db.getSiblingDB('mydatabase');
// The mark for cutting initial output off
print("CUT_TO_HERE");
// Main output
// "toArray()" method allows to get all records
printjson( db.getCollection('jobs').find().toArray() );
Sending the query from terminal
-z key of sed allows treat output as a single multi-line string
$> mongo localhost --quiet myScriptFile.js | sed -z 's/^.*CUT_TO_HERE\n//' > output.json

Related

How to load a file from the same directory as the script, in MongoDB shell

I'm running a bash script that executes a javascript file in the Mongo shell:
outerscript.sh
SCRIPTPATH="$( cd "$(dirname "$0")" ; pwd -P )"
LOGDIR=~/logs
LOGFILE=$LOGDIR/outerscript.log
mongo < $SCRIPTPATH/jsscript.js >> $LOGFILE 2>&1
jsscript.js
load(myVars.js);
MongoDB is v4.0.19
The script may be run locally or on a remote server, and it may be run from the command line or from a cron job, and I want to load in myVars.js which is always in the same folder as outerscript.sh.
I can't find a way to load myVars.js when running outerscript.sh from a cron job.
pwd() returns the user's home directory, not the location of the currently running script, so this doesn't work for the cron job:
load(pwd() + '/myVars.js');
This also doesn't work for the cron job, I'm guessing for the same reason:
load('./myVars.js');
Alternatively, is there another way to pass variables into my JavaScript file, that will still let me log output to the logfile? As far as I can tell, you can't combine --eval with logging output to a file, and I haven't found a way to read environment variables directly from inside the JS file.
Thanks!
You don't need a redirect, try mongo $SCRIPTPATH/jsscript.js >> $LOGFILE 2>&1
See
$ mongo --help
MongoDB shell version v4.4.1
usage: mongo [options] [db address] [file names (ending in .js)]
db address can be:
foo foo database on local machine
192.168.0.5/foo foo database on 192.168.0.5 machine
192.168.0.5:9999/foo foo database on 192.168.0.5 machine on port 9999
mongodb://192.168.0.5:9999/foo connection string URI can also be used
You can use cat() in your js script.
e.g.
echo '{"path": "some folder"}' > /tmp/foldername.js
Then in your js script:
var txt = cat("/tmp/foldername.js")
var folderObj = JSON.parse(txt)
Then you can use folderObj object in your script.
Why do you not simply use SCRIPTPATH="$(dirname "$0")"?
Wernfried Domscheit answered my question perfectly and I have accepted that answer.
I also worked out how to pass in environment variables as parameters and am posting it here just in case it helps anybody.
outerscript.sh
SCRIPTPATH="$( cd "$(dirname "$0")" ; pwd -P )"
ENVVARS='var mongoAddress='\'"${MONGO_ADDRESS}"\''; var databaseName='\'"${DATABASE_NAME}"\'
mongo --eval "${ENVVARS}" $SCRIPTPATH/jsscript.js >> $LOGFILE 2>&1
jsscript.js
print(mongoAddress);
print(databaseName);

Query result set export to local machine Mongo DB [duplicate]

I am using MongoDB 2.2.2 for 32-bit Windows7 machine. I have a complex aggregation query in a .js file. I need to execute this file on the shell and direct the output to a CSV file. I ensure that the query returns a "flat" json (no nested keys), so it is inherently convertible to a neat csv.
I know about load() and eval(). eval() requires me to paste the whole query into the shell and allows only printjson() inside the script, while I need csv. And, the second way: load()..It prints the output on the screen, and again in json format.
Is there a way Mongo can do this conversion from json to csv? (I need csv file to prepare charts on the data). I am thinking:
1. Either mongo has a built-in command for this that I can't find right now.
2. Mongo can't do it for me; I can at most send the json output to a file which I then need to convert to csv myself.
3. Mongo can send the json output to a temporary collection, the contents of which can be easily mongoexported to csv format. But I think only map-reduce queries support output collections. Is that right? I need it for an aggregation query.
Thanks for any help :)
I know this question is old but I spend an hour trying to export a complex query to csv and I wanted to share my thoughts. First I couldn't get any of the json to csv converters to work (although this one looked promising). What I ended up doing was manually writing the csv file in my mongo script.
This is a simple version but essentially what I did:
print("name,id,email");
db.User.find().forEach(function(user){
print(user.name+","+user._id.valueOf()+","+user.email);
});
This I just piped the query to stdout
mongo test export.js > out.csv
where test is the name of the database I use.
Mongo's in-built export is working fine, unless you want to any data manipulation like format date, covert data types etc.
Following command works as charm.
mongoexport -h localhost -d databse -c collection --type=csv
--fields erpNum,orderId,time,status
-q '{"time":{"$gt":1438275600000}, "status":{"$ne" :"Cancelled"}}'
--out report.csv
Extending other answers:
I found #GEverding's answer most flexible. It also works with aggregation:
test_db.js
print("name,email");
db.users.aggregate([
{ $match: {} }
]).forEach(function(user) {
print(user.name+","+user.email);
}
});
Execute the following command to export results:
mongo test_db < ./test_db.js >> ./test_db.csv
Unfortunately, it adds additional text to the CSV file which requires processing the file before we can use it:
MongoDB shell version: 3.2.10
connecting to: test_db
But we can make mongo shell stop spitting out those comments and only print what we have asked for by passing the --quiet flag
mongo --quiet test_db < ./test_db.js >> ./test_db.csv
Here is what you can try:
print("id,name,startDate")
cursor = db.<collection_name>.find();
while (cursor.hasNext()) {
jsonObject = cursor.next();
print(jsonObject._id.valueOf() + "," + jsonObject.name + ",\"" + jsonObject.stateDate.toUTCString() +"\"")
}
Save that in a file, say "export.js". Run the following command:
mongo <host>/<dbname> -u <username> -p <password> export.js > out.csv
Have a look at
this
for outputing from mongo shell to file.
There is no support for outputing csv from mongos shell. You would have to write the javascript yourself or use one of the many converters available. Google "convert json to csv" for example.
Just weighing in here with a nice solution I have been using. This is similar to Lucky Soni's solution above in that it supports aggregation, but doesn't require hard coding of the field names.
cursor = db.<collection_name>.<my_query_with_aggregation>;
headerPrinted = false;
while (cursor.hasNext()) {
item = cursor.next();
if (!headerPrinted) {
print(Object.keys(item).join(','));
headerPrinted = true;
}
line = Object
.keys(item)
.map(function(prop) {
return '"' + item[prop] + '"';
})
.join(',');
print(line);
}
Save this as a .js file, in this case we'll call it example.js and run it with the mongo command line like so:
mongo <database_name> example.js --quiet > example.csv
I use the following technique. It makes it easy to keep the column names in sync with the content:
var cursor = db.getCollection('Employees.Details').find({})
var header = []
var rows = []
var firstRow = true
cursor.forEach((doc) =>
{
var cells = []
if (firstRow) header.push("employee_number")
cells.push(doc.EmpNum.valueOf())
if (firstRow) header.push("name")
cells.push(doc.FullName.valueOf())
if (firstRow) header.push("dob")
cells.push(doc.DateOfBirth.valueOf())
row = cells.join(',')
rows.push(row)
firstRow = false
})
print(header.join(','))
print(rows.join('\n'))
When executing a script in a remote server. Mongo will add its own logging output, which we might want to omit from our file.
--quiet option will only disable connection related logs. Not all mongo logs. In such case we might need to filter out unneeded lines manually. A Windows based example:
mongo dbname --username userName --password password --host replicaset/ip:port --quiet printDataToCsv.js | findstr /v "NETWORK" > data.csv
This will pipe the script output and use findstr to filter out any lines, which have NETWORK string in them. More information on findstr: https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/findstr
A Linux version of this would use grep.

mongodb script file prints result along with log

bank_data collection has 6 records in it. so i am expecting the output should show only 6. but the 6 is printed along with timestamp and other text log.
script.js has following code
use test;
db.bank_data.find({first_name : "logan"}).count();
ran the above file in command line
mongo --quiet script.js > result.txt
Output result.txt file has following out put
2015-05-27T17:12:09.752+0530 I CONTROL Hotfix KB2731284 or later update is installed, no need to zero-out data files
switched to db test
6
I need to print the value alone 6
This should work , eval option is one way to get this done:
mongo test --quiet --eval "printjson(db.bank_data.find({first_name :
"logan"}).count())" > result.txt

Skip printing to stdout when using "mongo --eval"

I need the js variable to be passed through command line arguments. The option --eval did this. But it also printing the variable value to the shell by default. I want to skip the variable value printed to the shell.
Ex:-
I have used "user_name" inside the script.js
Passing the value using mongo --eval="user_name='john'" script.js
But it un necessarily printing the value
  john
Any ideas.
There is a --quiet flag - but evaluated items are still printed to stdout.
You could pipe to /dev/null or a file to capture the stdout eg:
mongo --eval="user_name='john'" script.js > /dev/null
For this use case you could use var in your declaration. This will not print the value:
mongo --eval="var user_name='john';" script.js

MongoDB : Show databases like MySQL

My MongoDB has more than 100 databases in it.
Whenever i use show dbs command my screen is filled with all databases names and it makes hard to find a particular database.
How to display only those databases which contain a particular substring as we can query in MySQL for displaying particular databases with ( show databases like '%SUBSTR%' ) query.
We do not have options like that. But your problem can be resolved by outputting the result into a txt file and later opening it
$ mongo | tee outnew.txt
In the mongo shell you can the give the show dbs command and exit.
mongo> show dbs;
mongo> exit
Then using gedit or excel access the outnew.txt file.
Hope it helped.
Another option is:
> db.getMongo().getDBNames().forEach(
function(databaseName) {
if (databaseName.match(/SUBSTR/i))
print(databaseName);
}
);
> var showdbs = function(pattern) {
db.getMongo().getDBNames().forEach(
function(databaseName) {
if (databaseName.match(new RegExp(pattern, 'i')))
print(databaseName);
});
};
> showdbs('SUBSTR'); // ALL: showdbs();
If you are on a *nix OS you could run the following command.
mongo --eval "db.adminCommand('listDatabases')['databases']" | grep "SUBSTR"
Note: You need to be admin to run this command.