Bash Variable in Mongo eval command inserts empty data - mongodb

I have a script that inserts data into Mongo from a CSV but I have to encrypt one of the columns before insertion.
#!/bin/bash
while IFS=, read -r f1 f2 f3 f4 f5 f6 f7 f8 f9 f10 f11 f12
do
ENCRYPT_VAL="$f5" | openssl aes-256-cbc -a -pbkdf2 -salt -pass pass:{*****}
if [ "$f4" = "m" ]
then
mongo a11_USER_gender --eval "db.m.insert({username: '$f1', first: '$f2', last: '$f3', gender: '$f4', dob: '$ENCRYPT_VAL', state: '$f6', municipality: '$f7', season: '$f8', continent: '$f9', elective: '$f10', f1: '$f11', airline: '$f12'})"
echo "Male Original value: '$f5'"
echo "Male Encrypted value: $ENCRYPT_VAL"
else
mongo a11_USER_gender --eval "db.f.insert({username: '$f1', first: '$f2', last: '$f3', gender: '$f4', dob: '$ENCRYPT_VAL', state: '$f6', municipality: '$f7', season: '$f8', continent: '$f9', elective: '$f10', f1: '$f11', airline: '$f12'})"
echo "Female Original value: '$f5'"
echo "Female Encrypted value: $ENCRYPT_VAL"
fi
done < /root/FileName.csv
I'm able to see that the variable has been successfully assigned the value of the encryption and that the Mongo insertion was seemingly successful. However, when I access the data inside of Mongo itself it displays a blank for that column.
I'm absolutely lost as to what could be the cause. I've tried enclosing the variable inside of the eval command in various different ways. I know I could just encrypt and save the value to the CSV prior to insertion, but I still feel like my attempted solution should work?
Sample of Seemingly Correct Encryption:
MongoDB shell version v5.0.9
connecting to:mongodb://127.0.0.1:27017/a11_USER_gender?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("7d51958e-1550-4abc-ad2e-dbb8f0297646") }
MongoDB server version: 5.0.9
WriteResult({ "nInserted" : 1 })
Female Original value: 'february 20 1987'
Female Encrypted value:
U2FsdGVkX1/bKsrdCX7844Ozr6PkYRZnVZRcguYSBJE=
Sample of Erroneous Mongo Entry Below:
{
"_id" : ObjectId("62c21fa9463288ccfaddba16"),
"username" : "zc1615",
"first" : "zeena",
"last" : "crayton",
"gender" : "f",
"dob" : "",
"state" : "ne",
"municipality" : "coral_gables",
"season" : "summer",
"continent" : "asia",
"elective" : "mad3301",
"f1" : "williams",
"airline" : "gol"
}

This:
ENCRYPT_VAL="$f5" | openssl aes-256-cbc -a -pbkdf2 -salt -pass pass:{*****}
Does not set ENCRYPT_VAL to the output of the openssl command. It temporarily sets ENCRYPT_VAL to the value of $f5 (see below), and then runs openssl on null input (printing the result to the console). So:
You're not capturing the output of openssl, and
You're not actually setting ENCRYPT_VAL (this is why you end up with an empty string in Mongo)
If you want to capture the output of a command in a variable, use $(...), like this:
ENCRYPT_VAL="$( echo "$f5" | openssl aes-256-cbc -a -pbkdf2 -salt -pass pass:{*****} )"
The Bourne shell and derivatives allow you to provide temporary environment to a command by prefixing the command with variable expressions. For example, we can do this:
$ FOO=bar sh -c 'echo $FOO'
bar
That sets the variable FOO in the environment of the sh command. It doesn't set a shell variable in the current shell and it doesn't change the environment of the current shell.
When you write:
ENCRYPT_VAL="$f5" | openssl aes-256-cbc -a -pbkdf2 -salt -pass pass:{*****}
You're using exactly that syntax, except you're not even specifying a command, so it is entirely a no-op.

Related

Run a command in mongodb container with credentials from outside

I want to set the compatibility version in mongodb, which is running in a container, but from outside:
docker exec -it docker-compose_mongodb_1 bash -c 'mongo -uroot -p rootpassword --eval "db.adminCommand( { setFeatureCompatibilityVersion: "4.0" } )"'
MongoDB shell version v4.0.23
connecting to: mongodb://127.0.0.1:27017/?gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("7945d913-f77c-4242-be33-af8e20c07374") }
MongoDB server version: 4.0.23
{
"ok" : 0,
"errmsg" : "Command argument must be of type String, but was of type double in: { setFeatureCompatibilityVersion: 4.0, lsid: { id: UUID(\"7945d913-f77c-4242-be33-af8e20c07374\") }, $db: \"admin\" }. See http://dochub.mongodb.org/core/4.0-feature-compatibility.",
"code" : 14,
"codeName" : "TypeMismatch"
}
Any idea what I am doing wrong?
It looks like a quoting problem: you seem to have double quotes around "4.0", but you've already quoted the entire argument to the --eval argument, so the effect of these quotes isn't what you want (the value (4.0) ends up unquoted, hence the error ("argument must be of type String, but was of type double").
In theory you can just escape the inner quotes:
docker exec -it docker-compose_mongodb_1 bash -c 'mongo -uroot -p rootpassword --eval "db.adminCommand( {setFeatureCompatibilityVersion: \"4.0\" } )"'

How can I print an Ansible vaulted variable that includes a Kubernetes secret from the CLI?

I have a Ansible group_vars directory with the following file within it:
$ cat inventory/group_vars/env1
...
...
ldap_config: !vault |
$ANSIBLE_VAULT;1.1;AES256
31636161623166323039356163363432336566356165633232643932623133643764343134613064
6563346430393264643432636434356334313065653537300a353431376264333463333238383833
31633664303532356635303336383361386165613431346565373239643431303235323132633331
3561343765383538340a373436653232326632316133623935333739323165303532353830386532
39616232633436333238396139323631633966333635393431373565643339313031393031313836
61306163333539616264353163353535366537356662333833653634393963663838303230386362
31396431636630393439306663313762313531633130326633383164393938363165333866626438
...
...
This Ansible encrypted string has a Kubernetes secret encapsulated within it. A base64 blob that looks something like this:
IyMKIyBIb3N0IERhdGFiYXNlCiMKIyBsb2NhbGhvc3QgaXMgdXNlZCB0byBjb25maWd1cmUgdGhlIGxvb3BiYWNrIGludGVyZmFjZQojIHdoZW4gdGhlIHN5c3RlbSBpcyBib290aW5nLiAgRG8gbm90IGNoYW5nZSB0aGlzIGVudHJ5LgojIwoxMjcuMC4wLjEJbG9jYWxob3N0CjI1NS4yNTUuMjU1LjI1NQlicm9hZGNhc3Rob3N0Cjo6MSAgICAgICAgICAgICBsb2NhbGhvc3QKIyBBZGRlZCBieSBEb2NrZXIgRGVza3RvcAojIFRvIGFsbG93IHRoZSBzYW1lIGt1YmUgY29udGV4dCB0byB3b3JrIG9uIHRoZSBob3N0IGFuZCB0aGUgY29udGFpbmVyOgoxMjcuMC4wLjEga3ViZXJuZXRlcy5kb2NrZXIuaW50ZXJuYWwKIyBFbmQgb2Ygc2VjdGlvbgo=
How can I decrypt this in a single CLI?
We can use an Ansible adhoc command to retrieve the variable of interest, ldap_config. To start we're going to use this adhoc to retrieve the Ansible encrypted vault string:
$ ansible -i "localhost," all \
-m debug \
-a 'msg="{{ ldap_config }}"' \
--vault-password-file=~/.vault_pass.txt \
-e#inventory/group_vars/env1
localhost | SUCCESS => {
"msg": "ABCD......."
Make note that we're:
using the debug module and having it print the variable, msg={{ ldap_config }}
giving ansible the path to the secret to decrypt encrypted strings
using the notation -e#< ...path to file...> to pass the file with the encrypted vault variables
Now we can use Jinja2 filters to do the rest of the parsing:
$ ansible -i "localhost," all \
-m debug \
-a 'msg="{{ ldap_config | b64decode | from_yaml }}"' \
--vault-password-file=~/.vault_pass.txt \
-e#inventory/group_vars/env1
localhost | SUCCESS => {
"msg": {
"apiVersion": "v1",
"bindDN": "uid=readonly,cn=users,cn=accounts,dc=mydom,dc=com",
"bindPassword": "my secret password to ldap",
"ca": "",
"insecure": true,
"kind": "LDAPSyncConfig",
"rfc2307": {
"groupMembershipAttributes": [
"member"
],
"groupNameAttributes": [
"cn"
],
"groupUIDAttribute": "dn",
"groupsQuery": {
"baseDN": "cn=groups,cn=accounts,dc=mydom,dc=com",
"derefAliases": "never",
"filter": "(objectclass=groupOfNames)",
"scope": "sub"
},
"tolerateMemberNotFoundErrors": false,
"tolerateMemberOutOfScopeErrors": false,
"userNameAttributes": [
"uid"
],
"userUIDAttribute": "dn",
"usersQuery": {
"baseDN": "cn=users,cn=accounts,dc=mydom,dc=com",
"derefAliases": "never",
"scope": "sub"
}
},
"url": "ldap://192.168.1.10:389"
}
}
NOTE: The above section -a 'msg="{{ ldap_config | b64decode | from_yaml }}" is what's doing the heavy lifting in terms of converting from Base64 to YAML.
References
How to run Ansible without hosts file
https://docs.ansible.com/ansible/latest/user_guide/playbooks_filters.html#filters-for-formatting-data
Base64 Decode String in jinja
How to decrypt string with ansible-vault 2.3.0
If you need a one liner that works with any yaml file (not only in inventory) containing inlined vault vars, and if you are ready to install a pip package for that, there is a solution using yq, a yaml processor built on top of jq
prerequesite: Install yq
pip install yq
Usage
You can get your result with the following command:
yq -r .ldapconfig inventory/group_vars/env1 | ansible_vault decrypt
If you need to type your vault pass interactively, don't forget to add the relevant option
yq -r .ldapconfig inventory/group_vars/env1 | ansible_vault --ask-vault-pass decrypt
Note: the -r option to yq is mandatory to get a raw result without the quotation marks around the value.

Just fetch a single attrbute from a single document

I need to pull the latest date from a collection on mongo db and set it in a shell script.
LASTDOCDATE=mongo mongo.com:27017/tracking -u user -p pw --authenticationDatabase authdb --eval 'db.TRACKING_DATA.find({},{datecreated :1}).limit(1).sort({datecreated:-1}).map(function (doc){ return doc.datecreated; })'
echo $LASTDOCDATE
This to be set but when run through the terminal produces:
connecting to: mongo.com:27017/tracking
Mon Jul 27 2015 16:28:08 GMT-0700 (PDT)
have can I pull just the date attribute to be set in a shell script as a variable
Wrap your call with the printjson() method of the shell in order to get an output string:
LASTDOCDATE=mongo mongo.com:27017/tracking -u user -p pw \\
--authenticationDatabase authdb \\
--eval 'printjson(db.TRACKING_DATA.find({},{datecreated :1}).limit(1).sort({datecreated:-1}).map(function (doc){ return doc.datecreated; }))'
Or just print, while referencing the single element:
LASTDOCDATE=mongo mongo.com:27017/tracking -u user -p pw \\
--authenticationDatabase authdb \\
--eval 'print(db.TRACKING_DATA.find({},{datecreated :1}).limit(1).sort({datecreated:-1}).toArray()[0].datecreated'
Notating the single array element, and then the property:
.find({},{datecreated :1}).limit(1).sort({datecreated:-1}).toArray()[0].datecreated'
Or findOne() like this with $orderby:
.findOne(
{ "query": {}, "$orderby": { "datecreated": 1 }},
{ "_id": 0, "datecreated": 1 }
).datecreated
So .print() or .printjson() depending on the output format you want. Or even .valueOf() on the "datecreated" to just get the timestamp value rather than the string.

Mongo query works in mongo shell but not in a bash mongo --eval?

Here is an example query:
db.readings.find( {"_id.s" : ISODate("2012-11-01T00:05:00Z") } ).count()
Query works in the mongo shell. However, in a bash script or directly in the Ubuntu shell
mongo fivemin --eval "printjson(db.readings.find( {"_id.s" : ISODate("2012-11-01T00:05:00Z") } ).count())"
returns a SyntaxError: missing : after property id (shell eval):1
I can't seem to find a problem with the query. I reverted to { "_id" : {"s" : ...} }, and it still gives the same problem. find().count() works however.
Had the same issue in bash shell.
This will fail because of double quotes:
mongo fivemin --eval "printjson(db.readings.find( {"_id.s" : ISODate("2012-11-01T00:05:00Z") } ).count())"
But using the eval string in single quote it works:
mongo fivemin --eval 'printjson(db.readings.find( {"_id.s" : ISODate("2012-11-01T00:05:00Z") } ).count())'
And if you use a $ sign like $regex you need to escape it:
mongo fivemin --eval 'printjson(db.readings.find( {"_id.s" : {"\$regex":"2012-11-01T*"} } ).count())'
use single quotes into double,
e.g.:
mongo fivemin --eval "printjson(db.readings.find( {'_id.s' : ISODate('2012-11-01T00:05:00Z') } ).count())"
There is also another possible issue using --eval with bash double quotes, if the query contains mongo operators i.e. $ne, $gt etc, since operators start with $.
double quotes:
$ echo " '$ne': null "
$ => '': null
single quotes:
$ echo ' "$ne": null '
$ => "$ne": null
Bash tries to replace these operators with variables.
$ ne = 'test'
$ echo " '$ne': null "
$ => 'test': null
So, I always recommend to use --eval with single quotes.
Just sat and thought about it. It seems to be a problem with bash exiting out on the " (should have noticed immediately!). Instead I used ' (or I guess you can use /" for JSON) so the query looks like:
printjson(db.readings.find({'_id.s' : ISODate('2013-01-01T00:05:00Z') }).count())"
The best way to handle this is to build the mongo command in a var. Then use eval command to execute the mongo command:
mongo_update_query="db.collectionName.update({ name:\""${some_name}"\", \
{ \$addToSet: { nick_names : { \$each : [ ${name_array} ] }}});"
mongo_cmd_str=$(echo "mongo --host ${mongo_host} --port ${mongo_port} ${mongo_database} --eval '${mongo_update_query}'")
# the actual call to mongo query
eval ${mongo_cmd_str}

mongo dbname --eval 'db.collection.find()' does not work

Why does this work:
# mongo dbname
MongoDB shell version: 1.8.3
connecting to: nextmuni_staging
> db.collection.find()
{ "foo" : "bar" }
> bye
While this does not work:
# mongo localhost/dbname --eval 'db.collection.find()'
MongoDB shell version: 1.8.3
connecting to: localhost/dbname
DBQuery: dbname.collection -> undefined
It should be exactly the same, no?
Thanks!
The return val of db.collection.find() is a cursor type. Executing this command from within the shell will create a cursor and show you the first page of data. You can start going through the rest by repeating the 'it' command.
I think the scope of variables used during the execution of an eval'd script is only for the lifetime of the script (data can be persisted into collections of course) so once the script terminates those cursor variables no longer exist and so you would be able to send another eval script to page the data. So the behaviour you get during a shell session wouldn't really work from an eval script.
To get close to the behaviour you could run something like this:
mongo dbname --eval "db.collection.find().forEach(printjson)"
That shows you that the command does execute and produce a cursor which you can then iterate over sending the output to stdout.
Edit: I think the point I was trying to make was that the command you are issuing is working its just the output is not what you expect.
The printjson functions covers a lot of ground when scripting with mongo --eval '...'. Rather than chaining .forEach you can simply wrap your call.
$ mongo --eval 'db.stats_data.stats()' db_name
MongoDB shell version: 2.4.14
connecting to: db_name
[object Object]
$ mongo --eval 'db.stats_data.stats().forEach(printjson)' db_name
MongoDB shell version: 2.4.14
connecting to: db_name
Tue Jan 10 15:32:11.961 TypeError: Object [object Object] has no method 'forEach'
$ mongo --eval 'printjson(db.stats_data.stats())' db_name
MongoDB shell version: 2.4.14
connecting to: db_name
{
"ns" : "db_name.stats_data",
"count" : 5516290,
"size" : 789938800,
"avgObjSize" : 143.20110073980882,
"storageSize" : 1164914688,
"numExtents" : 18,
"nindexes" : 3,
"lastExtentSize" : 307515392,
"paddingFactor" : 1.0000000000000457,
"systemFlags" : 1,
"userFlags" : 0,
"totalIndexSize" : 1441559616,
"indexSizes" : {
"_id_" : 185292688,
"owner_id_key_idx" : 427678384,
"onwer_metric_key_idx" : 828588544
},
"ok" : 1
}