I'm quite new to MongoDB and a bit confused. I'm trying to create on AWS an automated backup routine to be used in production, and I want to make sure I'm doing it correctly.
By now I have set a Replica Set with 1 Arbiter and 3 Members (1 primary, 1 secondary, 1 hidden delayed by 4 hours). Each member has 3 separate EBS volumes (data 100GB, journal 20GB, log 10GB).
I created a Lambda Function with NodeJS that run every hour (with CloudWatch Event) to take a snapshot, that performs the following operations:
MongoClient connects to the hidden delayed member mongodb://admin:password#ec2.private.ip:27017/admin
Flush all pending write operations db.command({ fsync: 1, lock: true })
Create EC2 Connection to retrieve all EBS volumes tagged as LambdaSnapshot:
const ec2Conn = new AWS.EC2({ region: 'us-west-1' })
const params = {
Filters: [
{
Name: "tag-key",
Values: ["LambdaSnapshot"],
},
],
};
const volumes = (await ec2Conn.describeVolumes(params).promise()).Volumes;
const volumeIds = volumes.map((volume) => volume.VolumeId);
Create a snapshot on each volume:
return Promise.all(
volumeIds.map(async (volumeId) => {
const formattedDate = moment().format("DD/MM/YYYY HH:mm:ss");
const snapshot = await ec2Conn
.createSnapshot({
Description: "Snapshot " + volumeId + " taken on " + formattedDate,
VolumeId: volumeId,
TagSpecifications: [
{
ResourceType: "snapshot",
Tags: [
{
Key: "Name",
Value: volume + " " + formattedDate,
},
{
Key: snapshotTag,
Value: volume + " " + formattedDate,
},
],
},
],
})
.promise();
return snapshot.SnapshotId;
});
);
Unlock the instance for writes with db.command({ fsyncUnlock: 1 })
I have 2 main doubts.
I have tagged as LambdaSnapshot only the EBS Volume containing DATA (the one of 100GB) of the hidden delayed member. I'm not sure if I have to take snapshot of the journal and the log volumes as well.
I notice that even though I'm using await to run the command to create the snapshot, the function continues anyway and unlock the instance while the snapshot is in pending state. I'm not sure, but I think that the command createSnapshot() give only the input to AWS to start the snapshot and resolve the promise without waiting for complete. So I'm in doubts if I have to unlock the db outside the lambda function once the snapshot complete; in that case I don't know how to listen for the complete event to run a second lambda function that unlock the db.
Thanks in advance
As stated in docs EBS snapshot creation is asynchronous:
Snapshots occur asynchronously; the point-in-time snapshot is created immediately, but the status of the snapshot is pending until the snapshot is complete (when all of the modified blocks have been transferred to Amazon S3), which can take several hours for large initial snapshots or subsequent snapshots where many blocks have changed. While it is completing, an in-progress snapshot is not affected by ongoing reads and writes to the volume.
Backup documentation says:
To get a correct snapshot of a running mongod process, you must have journaling enabled and the journal must reside on the same logical volume as the other MongoDB data files. Without journaling enabled, there is no guarantee that the snapshot will be consistent or valid.
Unless you have another documentation reference saying you do NOT need to back up log or journal data with all the other data, I suggest backing up everything together.
In Bluemix Node-RED application I use Cloudant and dashDB services. I replicated Cloudant database into dashDB which contains multiple values stored in a table like DALERT,DEVICE,ID etc.
I am trying to search records from the CLOUDANT table in my dashDB using DALERT column which value is equal to critical.
I am trying these way in Node-RED editor but unable to retrieve data from dashDB:
[
{"id":"9941f62b.66be08","type":"http in","name":"","url":"/get/specificcritical","method":"get","swaggerDoc":"","x":103.5,"y":409,"z":"c96fb1cb.36905","wires":[["b52196bf.4ade68","292d3e2e.d6d2c2"]]},
{"id":"b52196bf.4ade68","type":"function","name":"","func":"msg.Dalert=msg.payload.Dalert;\n\nreturn msg;","outputs":1,"noerr":0,"x":326,"y":353,"z":"c96fb1cb.36905","wires":[["457cf34.fba830c","d937915d.26c87"]]},
{"id":"457cf34.fba830c","type":"dashDB in","service":"dashDB-9a","query":"select * from XXXXX.CLOUDANT WHERE DALERT=?;","params":"msg.Dalert","name":"","x":510,"y":404,"z":"c96fb1cb.36905","wires":[["60d36407.9f2c9c","886c48df.7793b8"]]},
{"id":"d937915d.26c87","type":"debug","name":"","active":false,"console":"true","complete":"payload","x":599,"y":327,"z":"c96fb1cb.36905","wires":[]},
{"id":"60d36407.9f2c9c","type":"debug","name":"","active":true,"console":"false","complete":"false","x":758,"y":397,"z":"c96fb1cb.36905","wires":[]},
{"id":"886c48df.7793b8","type":"http response","name":"","x":771,"y":477,"z":"c96fb1cb.36905","wires":[]},
{"id":"292d3e2e.d6d2c2","type":"debug","name":"","active":true,"console":"true","complete":"payload","x":321,"y":483,"z":"c96fb1cb.36905","wires":[]}
]
Please let me know if there is any solution.
If I understood correct your question, you just need to modify your function node with something similar to this:
msg.dalert="critical";
return msg;
Assuming your DALERT column in the CLOUDANT table is of type VARCHAR. You may need to change it if is a different type on your database.
Running the application like:
http://yourappname.mybluemix.net/get/specificcritical
will result in output similar to this for my table:
[
{
"DALERT": "critical",
"DEVICE": "device1",
"ID": 1
},
{
"DALERT": "critical",
"DEVICE": "device3",
"ID": 3
},
{
"DALERT": "critical",
"DEVICE": "device5",
"ID": 5
}
]
Here is the new node flow I created with the changes (I added an input node with blank message just to test the flow in the editor):
[{"id":"c7468303.38b98","type":"http in","name":"","url":"/get/specificcritical","method":"get","swaggerDoc":"","x":125,"y":245,"z":"8e2ae4a.f71d518","wires":[["c685ce8c.397a3","20dfeeba.df2012"]]},{"id":"c685ce8c.397a3","type":"function","name":"","func":"msg.dalert=\"critical\";\nreturn msg;","outputs":1,"noerr":0,"x":347.5,"y":189,"z":"8e2ae4a.f71d518","wires":[["e1f8c153.1e074","1f2d6f8e.e0d29"]]},{"id":"e1f8c153.1e074","type":"dashDB in","service":"dashDB-0a","query":"select * from CLOUDANT WHERE DALERT=?;","params":"msg.dalert","name":"","x":531.5,"y":240,"z":"8e2ae4a.f71d518","wires":[["f1810e4c.0e7ef","1401dc1a.ebfe24"]]},{"id":"1f2d6f8e.e0d29","type":"debug","name":"","active":false,"console":"true","complete":"payload","x":620.5,"y":163,"z":"8e2ae4a.f71d518","wires":[]},{"id":"f1810e4c.0e7ef","type":"debug","name":"dashDB Output","active":true,"console":"false","complete":"payload","x":779.5,"y":233,"z":"8e2ae4a.f71d518","wires":[]},{"id":"1401dc1a.ebfe24","type":"http response","name":"","x":792.5,"y":313,"z":"8e2ae4a.f71d518","wires":[]},{"id":"20dfeeba.df2012","type":"debug","name":"","active":true,"console":"true","complete":"payload","x":342.5,"y":319,"z":"8e2ae4a.f71d518","wires":[]},{"id":"1f530672.e0acfa","type":"inject","name":"","topic":"","payload":"","payloadType":"none","repeat":"","crontab":"","once":false,"x":122,"y":112,"z":"8e2ae4a.f71d518","wires":[["c685ce8c.397a3"]]}]
I got ans for question.
if we want to pass value as parameter just write msg.variable=msg.payload.variable; and return msg;
inside function node and msg.variable also declare in query parm inside dashDB IN node. eg:msg.Dalert=msg.payload.Dalert; and
critical value pass with url as http://yourappname.mybluemix.net/get/specificcritical?Dalert=critical
Its simple working node red flow
[
{"id":"9941f62b.66be08","type":"http in","name":"","url":"/get/specificcritical","method":"get","swaggerDoc":"","x":103.5,"y":409,"z":"c96fb1cb.36905","wires":[["b52196bf.4ade68","292d3e2e.d6d2c2"]]},
{"id":"b52196bf.4ade68","type":"function","name":"","func":"msg.Device=msg.payload.Device;\n\nreturn msg;","outputs":1,"noerr":0,"x":326,"y":353,"z":"c96fb1cb.36905","wires":[["457cf34.fba830c","d937915d.26c87"]]},
{"id":"457cf34.fba830c","type":"dashDB in","service":"dashDB-XX","query":"select * from XXXXX.CLOUDANT WHERE DEVICE=?","params":"msg.Device","name":"","x":510,"y":404,"z":"c96fb1cb.36905","wires":[["60d36407.9f2c9c","886c48df.7793b8"]]},
{"id":"d937915d.26c87","type":"debug","name":"","active":false,"console":"true","complete":"payload","x":599,"y":327,"z":"c96fb1cb.36905","wires":[]},
{"id":"60d36407.9f2c9c","type":"debug","name":"","active":true,"console":"false","complete":"false","x":758,"y":397,"z":"c96fb1cb.36905","wires":[]},
{"id":"886c48df.7793b8","type":"http response","name":"","x":771,"y":477,"z":"c96fb1cb.36905","wires":[]},
{"id":"292d3e2e.d6d2c2","type":"debug","name":"","active":true,"console":"true","complete":"payload","x":321,"y":483,"z":"c96fb1cb.36905","wires":[]}
]
I am using chef version 10.16.2
I have a role (in ruby format). I need to access an attrubute set in one of the cookbooks
eg.
name "basebox"
description "A basic box with some packages, ruby and rbenv installed"
deployers = node['users']['names'].find {|k,v| v['role'] == "deploy" }
override_attributes {
{"rbenv" => {
"group_users" => deployers
}
}
}
run_list [
"recipe[users]",
"recipe[packages]",
"recipe[nginx]",
"recipe[ruby]"
]
I am using chef-solo so i cannot use search as given on http://wiki.opscode.com/display/chef/Search#Search-FindNodeswithaRoleintheExpandedRunList
How do i access node attributes in a role definition ?
Roles are JSON data.
That is, when you upload the role Ruby file to the server with knife, they are converted to JSON. Consider this role:
name "gaming-system"
description "Systems used for gaming"
run_list(
"recipe[steam::installer]",
"recipe[teamspeak3::client]"
)
When I upload it with knife role from file gaming-system.rb, I have this on the server:
{
"name": "gaming-system",
"description": "Systems used for gaming",
"json_class": "Chef::Role",
"default_attributes": {
},
"override_attributes": {
},
"chef_type": "role",
"run_list": [
"recipe[steam::installer]",
"recipe[teamspeak3::client]"
],
"env_run_lists": {
}
}
The reason for the Ruby DSL is that it is "nicer" or "easier" to write than the JSON. Compare the lines and syntax, and it's easy to see which is preferable to new users (who may not be familiar with JSON).
That data is consumed through the API. If you need to do any logic with attributes on your node, do it in a recipe.
Not sure if I 100% follow, but if you want to access an attribute which is set by a role from a recipe, then you just call it like any other node attribute. For example, in the case you presented, assuming the node has the basebox role in its run_list, you would just call:
node['rbenv']['group_users']
The role attributes are merged into the node.
HTH