CloudFormation to detach root device from stopped instance - aws-cloudformation

My actual task is:
Create AMI outside CloudFormation
Use CloudFormation to launch instance from AMI
Detach all existing volumes (including root and data volumes)
Attach the new volumes (root and data) which are created from latest snapshot.
I am stuck at building CloudFormation script to detach volumes from the instance.
Any suggestions or any one written CF script to detach the volumes (root & data)?

Here's one way you can approach this:
Create an AMI outside of AWS CloudFormation
Use AWS CloudFormation to launch an instance from that AMI
Create a Lambda-backed custom resource that takes a reference of the instance that was just launched as input, uses the AWS SDK to find out the volumes that are currently attached to the EC2 instance, and detach them one by one.
Attach whatever volumes you want to attach using resource of type AWS::EC2::VolumeAttachment
Key takeaway: If you want to perform a custom operation that CloudFormation doesn't natively support, try Lambda-backed custom resources.

Using the BlockDeviceMappings of EC2Instance you can specify all the EBS volumes for the instance. One of the EBS properties is SnapshotId. If that changes the volumes will be replaced on update. Just pass in the snapshotId as a parameter.
Example:
{
"Type": "AWS::EC2::Instance",
"Properties": {
"BlockDeviceMappings": [
{
"DeviceName": "/dev/sda1",
"Ebs" : { "SnapshotId" : {"Ref": "Snapshot1Id"} }
},
{
"DeviceName": "/dev/sda2",
"Ebs" : { "SnapshotId" : {"Ref": "Snapshot2Id"} }
},
{
"DeviceName": "/dev/sda3",
"Ebs" : { "SnapshotId" : {"Ref": "Snapshot3Id"} }
}
],
"ImageId": { "Ref": "InstanceAmi" }
}

Related

Cleaning kubernetes jobs

I have a spinnaker pipeline that deploys a db-job to k8s.
I would like to be able to delete the job before deploying another one, i.e. to add a spinnaker stage or to somehow configure job so it deletes itself.
I know that cronjob would be great for it, but it is in beta and not stable enough to be used for db operations.
I have tried to add a stage to spinnaker like this:
{
"account": "k8s-devops-v2",
"cloudProvider": "kubernetes",
"location": "loc",
"manifestArtifactAccount": "loc-open-source",
"manifestName": "job my-job-name",
"mode": "static",
"name": "Delete Db job",
"options": {
"cascading": true,
"gracePeriodSeconds": null
},
"type": "deleteManifest"
}
but it won't work.
I also don't want to use ttl because I wan't the latest job to be present until the new one is created.
Are there any other options? What is the best practice for this?
Depending on what version of kubernetes you're running you can do this in the k8s job itself: https://kubernetes.io/docs/tasks/job/automated-tasks-with-cron-jobs/#jobs-history-limits. The caveat is that once deleted unless you collect the logs via some kind of scraper, you won't be able to see what the job did unless you get to the cluster in time.

Running two containers on Fargate using CDK

I'd like to use Fargate to run two containers - one for the main project's backend, and another for the database (MongoDB). The basic example included in the GitHub repo shows how to run a single container on Fargate using CDK, however I still have 2 issues:
The example doesn't show how to run two containers.
I'd like to scale the database containers but have them share the data storage (so that the data gets stored in a central place and stays synchronized between the different containers).
I've figured out how to (sort of) fixed the first issue, similarly to how ecs.LoadBalancedFargateService is implemented, however the second issue still remains.
For reference, this is what I have so far in stack.ts (the rest is the basic boilerplate cdk init app --language typescript generates for you):
import cdk = require("#aws-cdk/cdk");
import ec2 = require("#aws-cdk/aws-ec2");
import ecs = require("#aws-cdk/aws-ecs");
import elbv2 = require("#aws-cdk/aws-elasticloadbalancingv2");
const {ApplicationProtocol} = elbv2;
export class AppStack extends cdk.Stack {
constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
super(scope, id, props);
// Create VPC and Fargate Cluster
const vpc = new ec2.VpcNetwork(this, "FargateVPC", {
maxAZs: 2
});
const cluster = new ecs.Cluster(this, "Cluster", {vpc});
// Create task definition
const fargateTaskDefinition = new ecs.FargateTaskDefinition(this, "FargateTaskDef", {
memoryMiB: "512",
cpu: "256"
});
// Create container from local `Dockerfile`
const appContainer = fargateTaskDefinition.addContainer("Container", {
image: ecs.ContainerImage.fromAsset(this, "Image", {
directory: ".."
})
});
// Set port mapping
appContainer.addPortMappings({
containerPort: 5000
});
// Create container from DockerHub image
const mongoContainer = fargateTaskDefinition.addContainer("MongoContainer", {
image: ecs.ContainerImage.fromDockerHub("mongo")
});
// Set port mapping
mongoContainer.addPortMappings({
containerPort: 27017
});
// Create service
const service = new ecs.FargateService(this, "Service", {
cluster,
taskDefinition: fargateTaskDefinition,
desiredCount: 2
});
// Configure task auto-scaling
const scaling = service.autoScaleTaskCount({
maxCapacity: 5
});
scaling.scaleOnCpuUtilization("CpuScaling", {
targetUtilizationPercent: 70
});
// Create service with built-in load balancer
const loadBalancer = new elbv2.ApplicationLoadBalancer(this, "AppLB", {
vpc,
internetFacing: true
});
// Allow incoming connections
loadBalancer.connections.allowFromAnyIPv4(new ec2.TcpPort(5000), "Allow inbound HTTP");
// Create a listener and listen to incoming requests
const listener = loadBalancer.addListener("Listener", {
port: 5000,
protocol: ApplicationProtocol.Http
});
listener.addTargets("ServiceTarget", {
port: 5000,
protocol: ApplicationProtocol.Http,
targets: [service]
});
// Output the DNS where you can access your service
new cdk.Output(this, "LoadBalancerDNS", {
value: loadBalancer.dnsName
});
}
}
Thanks in advance.
Generally, running a database in a Fargate container is not recommended since there is not currently a good solution for persisting data. You could integrate a hook that copies data into something like S3 prior to a task stopping, but generally those kinds of solutions are very fragile and not recommended.
You may want to check out DocumentDB as an alternative to running your own MongoDB instances, though support for DocumentDB constructs in the CDK are not yet fully fleshed out.
Another alternative is to run regular ECS tasks and attach an EBS volume on your EC2 Instance. Then you can use docker volumes to mount the EBS volume to your container. With this approach, you'll need to tag the instance metadata and use an ECS placement constraint to ensure that your task gets placed on the instance that has the EBS volume attached.
If either of these approaches works for you, feel free to open a feature request on the CDK repository. Hope this helps!
Is AWS Fargate a hard requirement?
If not, you could opt for simple ECS + Ec2, it supports the use of persistent data volumes:
Fargate tasks only support nonpersistent storage volumes.
For EC2 tasks, use data volumes in the following common examples:
To provide persistent data volumes for use with a container
To define an empty, nonpersistent data volume and mount it on multiple containers
To share defined data volumes at different locations on different containers on the same container instance
To provide a data volume to your task that is managed by a third-party volume driver
I haven't tried it myself but it seems that CDK has stable support for ECS + Ec2.
PS the link to the basic example is broken, I tried to find the new location but in the new example repository without success.

McRouter loses key on scale up

I'm running a k8s cluster in GKE and used their walkthrough of putting together a McRouter setup with memcached. Initially we were using consul keystores but our cache is too large and causes consul to use too much memory, so we decided to test out memcache in it's place. I spin up the mcrouter daemonset and have a single pod of memcache and everything works just fine. This is when I added some test keys. They get and delete ok. The issue comes when I leave keys in place and scale.
I scale up the memcache statefulset and add the second server name to the configmap for mcrouter. Once I see the new server using stats servers I then run a get and one of the keys is no longer there. I've telneted to 11211 on the original memcache pod and run a get and can retrieve the same key just fine. The config provided in the configmap is below:
{
"pools": {
"A": {
"servers": [
"memcached-0.memcached.default.svc.cluster.local:11211",
"memcached-1.memcached.default.svc.cluster.local:11211"
]
}
},
"route": "PoolRoute|A"
}
I've also moved to using a statefulset for mcrouter to limit to only one pod, and also switched to using the official docker image rather than the one in the k8s example Helm Chart, and had no luck. No matter what I do, I keep getting a "not found" via the mcrouter get on at least one key after scaling while other keys are still found fine. Help?
With PoolRoute, routes to the destination are based on key hash.
So, when you scale up the memcache stateful set with a new memcache, the key hash of your key change. That's why Mcrouter is not able to get your key even if it's already present in your cache.
Let's have a look at the wiki: https://github.com/facebook/mcrouter/wiki/List-of-Route-Handles
And maybe this kind of configuration may help you:
{
"pools": {
"A": {
"servers": [
"memcached-0.memcached.default.svc.cluster.local:11211",
"memcached-1.memcached.default.svc.cluster.local:11211"
]
},
"fallback": {
"servers": [
"memcached-fallback.memcached.default.svc.cluster.local:11211"
]
}
},
"route": {
"type": "FailoverRoute",
"normal": {
"type": "PoolRoute",
"pool": "A"
},
"failover": "PoolRoute|fallback"
}
}

Is there an ARM template which will allow us to setup a MongoDB replica set instance using Azure Managed Disk instead of regular data disks

Is there an ARM template which will allow us to setup a MongoDB replica set instance using Azure Managed Disk instead of regular data disks?
Note: The following reference provides a way to setup MongoDB replica set using regular data disks
https://github.com/Azure/azure-quickstart-templates/blob/master/mongodb-replica-set-centos/nested/primary-resources.json#L190-L230
You can edit the ARM template to use Managed Disks as shown below. Also, please use the apiVersion as 2017-03-30
"apiVersion": "2017-03-30",
"type": "Microsoft.Compute/virtualMachines",
{
"name": "datadisk1",
"diskSizeGB": "[parameters('sizeOfDataDiskInGB')]",
"lun": 0,
"createOption": "Empty",
"managedDisk": {
"storageAccountType": "Standard_LRS"
}
"caching": "ReadWrite"
}

Temporary storage size when installing service fabric

I'd like to create a basic (cheap) environment for my service fabric application. However, Service Fabric seems to use the "Temporary Storage" drive of the VMs which is limited in size. The only way I can see to increase the temporary storage drive is to pay for a more performant VM, which I don't really want to do yet.
Is there a way to either, increase the Temporary Storage size, or to tell Service Fabric not to use the Temporary Storage drive, but a different drive?
I am running out of storage in the temp drive, so need to look at other options.
The best option for you is creating a custom cluster from an ARM template, setup all requirements like extra disks, disk types, and then deploy a new cluster based on this template.
You can get some templates in this repository and use this guide as reference
If you don't configure MaxDiskQuotaInMB then Service Fabric diagnostic logs can use up 64gigs of temporary storage (d: for Windows or /mnt on Linux).
You can try using VMs with larger temporary storage (64gigs or more) or limit how much storage service fabric allocates for logs by:
running this type of Powershell command described here
Set-AzureRmServiceFabricSetting -ResourceGroupName clusterResourceGroup -Name clusterName -Section "Diagnostics" -Parameter "MaxDiskQuotaInMB" -Value "25600"
defining it as part of the arm template like in this example
"parameters": [
{
"name": "MaxDiskQuotaInMB",
"value": 5120
}
]
changing it via resources.azure.com by following these instructions from Microsoft Docs
{
"name": "Diagnostics",
"parameters": [
{
"name": "MaxDiskQuotaInMB",
"value": "65536"
}
]
}