I'm running a k8s cluster in GKE and used their walkthrough of putting together a McRouter setup with memcached. Initially we were using consul keystores but our cache is too large and causes consul to use too much memory, so we decided to test out memcache in it's place. I spin up the mcrouter daemonset and have a single pod of memcache and everything works just fine. This is when I added some test keys. They get and delete ok. The issue comes when I leave keys in place and scale.
I scale up the memcache statefulset and add the second server name to the configmap for mcrouter. Once I see the new server using stats servers I then run a get and one of the keys is no longer there. I've telneted to 11211 on the original memcache pod and run a get and can retrieve the same key just fine. The config provided in the configmap is below:
{
"pools": {
"A": {
"servers": [
"memcached-0.memcached.default.svc.cluster.local:11211",
"memcached-1.memcached.default.svc.cluster.local:11211"
]
}
},
"route": "PoolRoute|A"
}
I've also moved to using a statefulset for mcrouter to limit to only one pod, and also switched to using the official docker image rather than the one in the k8s example Helm Chart, and had no luck. No matter what I do, I keep getting a "not found" via the mcrouter get on at least one key after scaling while other keys are still found fine. Help?
With PoolRoute, routes to the destination are based on key hash.
So, when you scale up the memcache stateful set with a new memcache, the key hash of your key change. That's why Mcrouter is not able to get your key even if it's already present in your cache.
Let's have a look at the wiki: https://github.com/facebook/mcrouter/wiki/List-of-Route-Handles
And maybe this kind of configuration may help you:
{
"pools": {
"A": {
"servers": [
"memcached-0.memcached.default.svc.cluster.local:11211",
"memcached-1.memcached.default.svc.cluster.local:11211"
]
},
"fallback": {
"servers": [
"memcached-fallback.memcached.default.svc.cluster.local:11211"
]
}
},
"route": {
"type": "FailoverRoute",
"normal": {
"type": "PoolRoute",
"pool": "A"
},
"failover": "PoolRoute|fallback"
}
}
Related
I have a spinnaker pipeline that deploys a db-job to k8s.
I would like to be able to delete the job before deploying another one, i.e. to add a spinnaker stage or to somehow configure job so it deletes itself.
I know that cronjob would be great for it, but it is in beta and not stable enough to be used for db operations.
I have tried to add a stage to spinnaker like this:
{
"account": "k8s-devops-v2",
"cloudProvider": "kubernetes",
"location": "loc",
"manifestArtifactAccount": "loc-open-source",
"manifestName": "job my-job-name",
"mode": "static",
"name": "Delete Db job",
"options": {
"cascading": true,
"gracePeriodSeconds": null
},
"type": "deleteManifest"
}
but it won't work.
I also don't want to use ttl because I wan't the latest job to be present until the new one is created.
Are there any other options? What is the best practice for this?
Depending on what version of kubernetes you're running you can do this in the k8s job itself: https://kubernetes.io/docs/tasks/job/automated-tasks-with-cron-jobs/#jobs-history-limits. The caveat is that once deleted unless you collect the logs via some kind of scraper, you won't be able to see what the job did unless you get to the cluster in time.
Is there an ARM template which will allow us to setup a MongoDB replica set instance using Azure Managed Disk instead of regular data disks?
Note: The following reference provides a way to setup MongoDB replica set using regular data disks
https://github.com/Azure/azure-quickstart-templates/blob/master/mongodb-replica-set-centos/nested/primary-resources.json#L190-L230
You can edit the ARM template to use Managed Disks as shown below. Also, please use the apiVersion as 2017-03-30
"apiVersion": "2017-03-30",
"type": "Microsoft.Compute/virtualMachines",
{
"name": "datadisk1",
"diskSizeGB": "[parameters('sizeOfDataDiskInGB')]",
"lun": 0,
"createOption": "Empty",
"managedDisk": {
"storageAccountType": "Standard_LRS"
}
"caching": "ReadWrite"
}
My actual task is:
Create AMI outside CloudFormation
Use CloudFormation to launch instance from AMI
Detach all existing volumes (including root and data volumes)
Attach the new volumes (root and data) which are created from latest snapshot.
I am stuck at building CloudFormation script to detach volumes from the instance.
Any suggestions or any one written CF script to detach the volumes (root & data)?
Here's one way you can approach this:
Create an AMI outside of AWS CloudFormation
Use AWS CloudFormation to launch an instance from that AMI
Create a Lambda-backed custom resource that takes a reference of the instance that was just launched as input, uses the AWS SDK to find out the volumes that are currently attached to the EC2 instance, and detach them one by one.
Attach whatever volumes you want to attach using resource of type AWS::EC2::VolumeAttachment
Key takeaway: If you want to perform a custom operation that CloudFormation doesn't natively support, try Lambda-backed custom resources.
Using the BlockDeviceMappings of EC2Instance you can specify all the EBS volumes for the instance. One of the EBS properties is SnapshotId. If that changes the volumes will be replaced on update. Just pass in the snapshotId as a parameter.
Example:
{
"Type": "AWS::EC2::Instance",
"Properties": {
"BlockDeviceMappings": [
{
"DeviceName": "/dev/sda1",
"Ebs" : { "SnapshotId" : {"Ref": "Snapshot1Id"} }
},
{
"DeviceName": "/dev/sda2",
"Ebs" : { "SnapshotId" : {"Ref": "Snapshot2Id"} }
},
{
"DeviceName": "/dev/sda3",
"Ebs" : { "SnapshotId" : {"Ref": "Snapshot3Id"} }
}
],
"ImageId": { "Ref": "InstanceAmi" }
}
We have a scenario with 2 web servers and 2 memcache servers. The memcache server nodes are configured within the configuration files on each web server.
In order for us to automatically horizontally scale the web servers we need to be able to automatically horizotally scale the memcache servers. Currently we are not able to do that due to the memcache nodes being defined in the web configuration files.
What is the best way to automatically horizontally scale the memcache servers with the least impact on the system.
You can try mcrouter.
What is Mcrouter?
It is a memcached protocol router that is used at Facebook to handle all traffic of cache servers across dozens of clusters distributed in Facebook data centres around the world.
How the scaling happens?
To adding a new server in to the cluster, just adding an IP entry in the route config file. 'mcrouter' automatically detects it and add to the cluster with consistent hashing.
The config file looks like below, with 2 servers entries.
{
"pools": {
"A": {
"servers": [
"SERVER1_IP:11211",
"SERVER2_IP:11211"
]
}
},
"route": {
"type": "OperationSelectorRoute",
"operation_policies": {
"add": "AllSyncRoute|Pool|A",
"delete": "AllSyncRoute|Pool|A",
"get": "RandomRoute|Pool|A",
"set": "AllSyncRoute|Pool|A"
}
}
}
Hope it helps!
I don't know if I understand the options of consul exec...
I have a consul server and several consul clients: https://play.golang.org/p/s3N3r3lK9e (example of config files)
I would like to create a service to run a program in each client:
"service": {
"name": "runner", "port": 7700,
"check": {
"script": "/usr/local/bin/myApp --run"
}
}
When a new KV is written in Consul, I want to execute an app in server side to run the service called "runner" in a specific node, in other words I want to execute in my application consul exec -service=runner to run another app (myApp --run) in the node client side. This is possible? This is the meaning of consul exec?
If you don't understand the question, I can rewrite it.
Usually it's used for common jobs on all nodes. For example, something like this: sudo apt-get update.
But, remember, it will be running on ALL nodes in cluster. So, if this command produce huge output, it will be the mess.
Secondly, there is no guarantee of execution.
For things like this I recommend to use system like Ansible, Chef, etc.