I have a standard Python docker image that needs to start after postgers is properly started in its standard image.
I understand that I can add this Bash command in the docker-compose file:
command: bash -c 'while !</dev/tcp/db/5432; do sleep 1; done; npm start'
depends_on:
- mypostgres
But I don't have bash installed in the standard python docker image, and I'm trying to keep the installation minimal.
Is there a way to wait for postgres without having bash installed in my image?
I have a standard Python docker image that needs to start after postgres is properly started in its standard image.
You mentioned "Python docker image", but you appear to be calling npm start, which is a node.js application, not a Python application.
The standard Python images do have bash installed (as do the official Node images):
$ docker run -it --rm python:3.10 bash
root#c9bdac2e23f9:/#
However, just checking for the port to be available may be insufficient in any case, so really what you want is to execute a query against the database and only continue once the query is successful.
A common solution is to install the postgres cli and run psql in a loop, like this:
until psql -h $HOST -U $USER -d $DB_NAME -c 'select 1' >/dev/null 2>&1; do
echo 'Waiting for database...'
sleep 1
done
You can use environment variables or a .pgpass file to provide the appropriate password.
If you are building a custom image, it may be better to place this logic in your ENTRYPOINT script rather than embedding it in the command field of your docker-compose.yaml.
If you don't want to psql, you can write the same logic in Python or Node utilizing whatever Postgres bindings are available (e.g., something like psycopg2 for Python).
A better solution is to make your application robust in the face of database failures, because this allows your application to continue running if the database is briefly unavailable during a restart.
I am researching now the simple "use" command for the mongo command. Please help me.
I just want save the my query in a file, but before that i need to connect to a certain database. For that i tried to find a "use" command like in sql, but could not find anything.
I just want to execute something like
mongo ....--use [db] --eval 'db.find' > save.query
In your question, you didn't specify what platform you're using, are you using Linux? Windows? Anyway, if you want to use the command line for mongo db then I would recommend to use the mongodb shell. Download the mongodb shell https://www.mongodb.com/try/download/shell and select what platform you're using.
https://www.mongodb.com/docs/mongodb-shell/run-commands/#mongosh-usage
That sais you have use <database> command
I just got it. You can just add the database name:
mongo [dbname] --host etc.
and it worked.
This is the easiest way in linux:
Option 1 ( command line params )
echo "db.exampleollection.find({}).forEach(function(d){printjson(d)})" | mongo --quiet exampledatabase --host "examplehost" --port "examplePort" --authenticationDatabase=admin -u "exampleuser" -p "examplepassword" > output.json
Explained:
Send the command you need to execute via echo to the mongo shell and redirect the output to the result file.
Add the option --quiet to suppress the shell printed info
You can provide the database directly in as comman line argument.
Option 2 : same way but URI format:
echo "show collections" | mongo "mongodb://user:pass#host:port/database?authSource=admin" --quiet
On RHEL, the below command works:
psql -h hostname -U username -p port_no -d database -f /tmp/myfile.sql &> logfile01.txt
On FreeBSD, this throws error:
"Invalid null command"
Please suggest.
If you use this only on the command line then there is no need to change the shell.
To redirect stdout and stderr to a file in C-Shell synthax simply use ">& filename".
Different story is, if you want to write shell scripts. Bourne Shell and it's clones (like i.e. Bash) are better suited for writing script. See this Unix FAQ "Csh Programming Considered Harmful": http://www.faqs.org/faqs/unix-faq/shell/csh-whynot/
This redirection works in bash
&> logfile01.txt
, but it does not work in csh which is the default shell in FreeBSD.
# set | grep shell
shell /bin/csh
# ls -la &> logfile01.txt
Invalid null command.
Bash is not installed by default. You can install it
pkg install bash
and configure it as the default shell.
I'm trying to create a service / script to automatically start and controll my nodejs server, but it doesnt seem to work at all.
First of all, I used this source as main reference http://kvz.io/blog/2009/12/15/run-nodejs-as-a-service-on-ubuntu-karmic/
After testing around, I minimzed the content of the actual file to avoid any kind of error, resulting in this (the bare minimum, but it doesnt work)
description "server"
author "blah"
start on started mountall
stop on shutdown
respawn
respawn limit 99 5
script
export HOME="/var/www"
exec nodejs /var/www/server/server.js >> /var/log/node.log 2>&1
end script
The file is saved in /etc/init/server.conf
when trying to start the script (as root, or normal user), I get:
root#iof304:/etc/init# start server
start: Job failed to start
Then, I tried to check my syntax with init-checkconf, resulting in:
$ init-checkconf /etc/init/server.conf
File /etc/init/server.conf: syntax ok
I tried different other things, like initctl reload-configuration with no result.
What can I do? How can I get this to work? It can't be that hard, right?
This is what our typical startup script looks like. As you can see we're running our node processes as user nodejs. We're also using the pre-start script to make sure all of the log file directories and .tmp directories are created with the right permissions.
#!upstart
description "grabagadget node.js server"
author "Jeffrey Van Alstine"
start on started mysql
stop on shutdown
respawn
script
export HOME="/home/nodejs"
exec start-stop-daemon --start --chuid nodejs --make-pidfile --pidfile /var/run/nodejs/grabagadget.pid --startas /usr/bin/node -- /var/nodejs/grabagadget/app.js --environment production >> /var/log/nodejs/grabagadget.log 2>&1
end script
pre-start script
mkdir -p /var/log/nodejs
chown nodejs:root /var/log/nodejs
mkdir -p /var/run/nodejs
mkdir -p /var/nodejs/grabagadget/.tmp
# Git likes to reset permissions on this file, but it really needs to be writable on server start
chown nodejs:root /var/nodejs/grabagadget/views/layout.ejs
chown -R nodejs:root /var/nodejs/grabagadget/.tmp
# Date format same as (new Date()).toISOString() for consistency
sudo -u nodejs echo "[`date -u +%Y-%m-%dT%T.%3NZ`] (sys) Starting" >> /var/log/nodejs/grabagadget.log
end script
pre-stop script
rm /var/run/nodejs/grabagadget.pid
sudo -u nodejs echo "[`date -u +%Y-%m-%dT%T.%3NZ`] (sys) Stopping" >> /var/log/nodejs/grabgadget.log
end script
As of Ubuntu 15, upstart is no longer being used, see systemd.
I want to execute mongo commands in shell script, e.g. in a script test.sh:
#!/bin/sh
mongo myDbName
db.mycollection.findOne()
show collections
When I execute this script via ./test.sh, then the connection to MongoDB is established, but the following commands are not executed.
How to execute other commands through shell script test.sh?
You can also evaluate a command using the --eval flag, if it is just a single command.
mongo --eval "printjson(db.serverStatus())"
Please note: if you are using Mongo operators, starting with a $ sign, you'll want to surround the eval argument in single quotes to keep the shell from evaluating the operator as an environment variable:
mongo --eval 'db.mycollection.update({"name":"foo"},{$set:{"this":"that"}});' myDbName
Otherwise you may see something like this:
mongo --eval "db.test.update({\"name\":\"foo\"},{$set:{\"this\":\"that\"}});"
> E QUERY SyntaxError: Unexpected token :
Put your mongo script into a .js file.
Then execute mongo < yourFile.js
Ex:
demo.js //file has your script
use sample //db name
show collections
keep this file in "c:\db-scripts"
Then in cmd prompt go to "c:\db-scripts"
C:\db-scripts>mongo < demo.js
This will execute the code in mongo and shows the output
C:\db-scripts>mongo < demo.js
Mongo shell version: 3.0.4
Connecting to: test
switched to db sample
users //collection name
tasks //collection name
bye
C:\db-scripts>
This works for me under Linux:
mongo < script.js
For newer version of mongodb
mongosh < script.js
Put this in a file called test.js:
db.mycollection.findOne()
db.getCollectionNames().forEach(function(collection) {
print(collection);
});
then run it with mongo myDbName test.js.
There is an official documentation page about this as well.
Examples from that page include:
mongo server:27017/dbname --quiet my_commands.js
mongo test --eval "printjson(db.getCollectionNames())"
The shell script below also worked nicely for me... definite had to use the redirect that Antonin mentioned at first... that gave me the idea to test the here document.
function testMongoScript {
mongo <<EOF
use mydb
db.leads.findOne()
db.leads.find().count()
EOF
}
In case you have authentication enabled:
mongo -u username -p password --authenticationDatabase auth_db_name < your_script.js
For newer version
mongosh -u username -p password --authenticationDatabase auth_db_name < your_script.js
I use the "heredoc" syntax, which David Young mentions. But there is a catch:
#!/usr/bin/sh
mongo <db> <<EOF
db.<collection>.find({
fieldName: { $exists: true }
})
.forEach( printjson );
EOF
The above will NOT work, because the phrase "$exists" will be seen by the shell and substituted with the value of the environment variable named "exists." Which, likely, doesn't exist, so after shell expansion, it becomes:
#!/usr/bin/sh
mongo <db> <<EOF
db.<collection>.find({
fieldName: { : true }
})
.forEach( printjson );
EOF
In order to have it pass through you have two options. One is ugly, one is quite nice. First, the ugly one: escape the $ signs:
#!/usr/bin/sh
mongo <db> <<EOF
db.<collection>.find({
fieldName: { \$exists: true }
})
.forEach( printjson );
EOF
I do NOT recommend this, because it is easy to forget to escape.
The other option is to escape the EOF, like this:
#!/usr/bin/sh
mongo <db> <<\EOF
db.<collection>.find({
fieldName: { $exists: true }
})
.forEach( printjson );
EOF
Now, you can put all the dollar signs you want in your heredoc, and the dollar signs are ignored. The down side: That doesn't work if you need to put shell parameters/variables in your mongo script.
Another option you can play with is to mess with your shebang. For example,
#!/bin/env mongo
<some mongo stuff>
There are several problems with this solution:
It only works if you are trying to make a mongo shell script executable from the command line. You can't mix regular shell commands with mongo shell commands. And all you save by doing so is not having to type "mongo" on the command line... (reason enough, of course)
It functions exactly like "mongo <some-js-file>" which means it does not let you use the "use <db>" command.
I have tried adding the database name to the shebang, which you would think would work. Unfortunately, the way the system processes the shebang line, everything after the first space is passed as a single parameter (as if quoted) to the env command, and env fails to find and run it.
Instead, you have to embed the database change within the script itself, like so:
#!/bin/env mongo
db = db.getSiblingDB('<db>');
<your script>
As with anything in life, "there is more than one way to do it!"
In my setup I have to use:
mongo --host="the.server.ip:port" databaseName theScript.js
For newer version of mongodb
mongosh --host="the.server.ip:port" databaseName theScript.js
How about this:
echo "db.mycollection.findOne()" | mongo myDbName
echo "show collections" | mongo myDbName
Create a script file; write commands:
#!/bin/sh
mongo < file.js
For newer versions
mongosh < file.js
In file.js write your mongo query:
db.collection.find({"myValue":null}).count();
As suggested by theTuxRacer, you can use the eval command, for those who are missing it like I was, you can also add in your db name if you are not trying to preform operation on the default db.
mongo <dbname> --eval "printjson(db.something.find())"
Newer version of mongodb
mongosh <dbname> --eval "printjson(db.something.find())"
In my case, I can conveniently use \n as separator for the next mongo command I want to execute then pipe them to mongo
echo $'use your_db\ndb.yourCollection.find()' | mongo
Newer version of mongodb
echo $'use your_db\ndb.yourCollection.find()' | mongosh
Thank you printf! In a Linux environment, here's a better way to have only one file run the show. Say you have two files, mongoCmds.js with multiple commands:
use someDb
db.someColl.find()
and then the driver shell file, runMongoCmds.sh
mongo < mongoCmds.js
Newer version of mongodb
mongosh < mongoCmds.js
Instead, have just one file, runMongoCmds.sh containing
printf "use someDb\ndb.someColl.find()" | mongo
Bash's printf is much more robust than echo and allows for the \n between commands to force them on multiple lines.
mongo <<EOF
use <db_name>
db.getCollection("<collection_name>").find({})
EOF
--shell flag can also be used for javascript files
mongo --shell /path/to/jsfile/test.js
mongo db_name --eval "db.user_info.find().forEach(function(o) {print(o._id);})"
Recently migrated from mongodb to Postgres. This is how I used the scripts.
mongo < scripts.js > inserts.sql
Read the scripts.js and output redirect to inserts.sql.
scripts.js looks like this
use myDb;
var string = "INSERT INTO table(a, b) VALUES";
db.getCollection('collectionName').find({}).forEach(function (object) {
string += "('" + String(object.description) + "','" + object.name + "'),";
});
print(string.substring(0, string.length - 1), ";");
inserts.sql looks like this
INSERT INTO table(a, b) VALUES('abc', 'Alice'), ('def', 'Bob'), ('ghi', 'Claire');
If you want to handle it with one line it's an easy way.
file.sh --> db.EXPECTED_COLLECTION.remove("_id":1234)
cat file.sh | mongo <EXPECTED_COLLECTION>
Single shell script solution with ability to pass mongo arguments (--quiet, dbname, etc):
#!/usr/bin/env -S mongo --quiet localhost:27017/test
cur = db.myCollection.find({});
while(cur.hasNext()) {
printjson(cur.next());
}
The -S flag might not work on all platforms.
When using a replicaset, writes must be done on the PRIMARY, so I usually use syntax like this which avoids having to figure out which host is the master:
mongo -host myReplicaset/anyKnownReplica
The legacy mongo shell has been removed from MongoDB with the version 6 release in 2022
There is a new way to execute shell scripts using the new shell, mongosh
From the shell documentation on writing scripts:
You can use mongosh to execute a script from the command line without entering the mongosh console
To specify the filename, use the --file or -f parameter to specify the filename
You may also need to specify connection information