I have a spinnaker pipeline that deploys a db-job to k8s.
I would like to be able to delete the job before deploying another one, i.e. to add a spinnaker stage or to somehow configure job so it deletes itself.
I know that cronjob would be great for it, but it is in beta and not stable enough to be used for db operations.
I have tried to add a stage to spinnaker like this:
{
"account": "k8s-devops-v2",
"cloudProvider": "kubernetes",
"location": "loc",
"manifestArtifactAccount": "loc-open-source",
"manifestName": "job my-job-name",
"mode": "static",
"name": "Delete Db job",
"options": {
"cascading": true,
"gracePeriodSeconds": null
},
"type": "deleteManifest"
}
but it won't work.
I also don't want to use ttl because I wan't the latest job to be present until the new one is created.
Are there any other options? What is the best practice for this?
Depending on what version of kubernetes you're running you can do this in the k8s job itself: https://kubernetes.io/docs/tasks/job/automated-tasks-with-cron-jobs/#jobs-history-limits. The caveat is that once deleted unless you collect the logs via some kind of scraper, you won't be able to see what the job did unless you get to the cluster in time.
Related
When creating an EMR cluster via the UI, I can click 'enable debugging'.
Via the cli, I can add the parameter --enable-debugging.
How can I do it via cloudformation? I did give a LogUri, where I do see the logs, but the EMR web UI carries on telling me 'Debugging not configured' when running Spark jobs.
It just is added as the first step in the steps...so the equivalent of the below in CloudFormation config.
"Steps": [{
"Name": "Setup Hadoop Debugging",
"ActionOnFailure": "TERMINATE_CLUSTER",
"HadoopJarStep": {
"Jar": "command-runner.jar",
"Args": ["state-pusher-script"]
}
}]
The accepted answer is almost correct.
I received an error while deploying: CloudFormation currently only supports long-running clusters, set ActionOnFailure to CANCEL_AND_WAIT or CONTINUE.
So, the following code worked for me (also, YAML version for those, who would like to copypaste that format):
Steps:
- Name: SetupHadoopDebugging
ActionOnFailure: CONTINUE
HadoopJarStep:
Jar: command-runner.jar
Args:
- 'state-pusher-script'
Is there a way in Amazon Fargate to determine the startup order of containers?
Let's say I have two containers and I want container B to start only when A is already up.
In other Amazon ECS I could use links to achieve this, but links are not supported in Fargate's awsvpc network mode.
As of March 2019, this feature is now available! https://aws.amazon.com/about-aws/whats-new/2019/03/amazon-ecs-introduces-enhanced-container-dependency-management/
The way you do this is with dependsOn in your task definition. e.g. if you want a container to start only after a container foo has started, you do
"dependsOn": [
{
"containerName": "foo",
"condition": "START"
}
]
If you are using the AWS Task Definition wizard, you can configure this under the section "Container Definitions". Edit (or add) your container, then scroll down to the section "STARTUP DEPENDENCY ORDERING". You can choose from four options: start, complete, success, healthy.
There is no explicit way to control ordering in Fargate today, but it's on the roadmap.
I'm running a k8s cluster in GKE and used their walkthrough of putting together a McRouter setup with memcached. Initially we were using consul keystores but our cache is too large and causes consul to use too much memory, so we decided to test out memcache in it's place. I spin up the mcrouter daemonset and have a single pod of memcache and everything works just fine. This is when I added some test keys. They get and delete ok. The issue comes when I leave keys in place and scale.
I scale up the memcache statefulset and add the second server name to the configmap for mcrouter. Once I see the new server using stats servers I then run a get and one of the keys is no longer there. I've telneted to 11211 on the original memcache pod and run a get and can retrieve the same key just fine. The config provided in the configmap is below:
{
"pools": {
"A": {
"servers": [
"memcached-0.memcached.default.svc.cluster.local:11211",
"memcached-1.memcached.default.svc.cluster.local:11211"
]
}
},
"route": "PoolRoute|A"
}
I've also moved to using a statefulset for mcrouter to limit to only one pod, and also switched to using the official docker image rather than the one in the k8s example Helm Chart, and had no luck. No matter what I do, I keep getting a "not found" via the mcrouter get on at least one key after scaling while other keys are still found fine. Help?
With PoolRoute, routes to the destination are based on key hash.
So, when you scale up the memcache stateful set with a new memcache, the key hash of your key change. That's why Mcrouter is not able to get your key even if it's already present in your cache.
Let's have a look at the wiki: https://github.com/facebook/mcrouter/wiki/List-of-Route-Handles
And maybe this kind of configuration may help you:
{
"pools": {
"A": {
"servers": [
"memcached-0.memcached.default.svc.cluster.local:11211",
"memcached-1.memcached.default.svc.cluster.local:11211"
]
},
"fallback": {
"servers": [
"memcached-fallback.memcached.default.svc.cluster.local:11211"
]
}
},
"route": {
"type": "FailoverRoute",
"normal": {
"type": "PoolRoute",
"pool": "A"
},
"failover": "PoolRoute|fallback"
}
}
Is there an ARM template which will allow us to setup a MongoDB replica set instance using Azure Managed Disk instead of regular data disks?
Note: The following reference provides a way to setup MongoDB replica set using regular data disks
https://github.com/Azure/azure-quickstart-templates/blob/master/mongodb-replica-set-centos/nested/primary-resources.json#L190-L230
You can edit the ARM template to use Managed Disks as shown below. Also, please use the apiVersion as 2017-03-30
"apiVersion": "2017-03-30",
"type": "Microsoft.Compute/virtualMachines",
{
"name": "datadisk1",
"diskSizeGB": "[parameters('sizeOfDataDiskInGB')]",
"lun": 0,
"createOption": "Empty",
"managedDisk": {
"storageAccountType": "Standard_LRS"
}
"caching": "ReadWrite"
}
I have created a Task Definition on Elastic Container Service and have successfully run it in a Fargate cluster. However when I create a Scheduled Task in said cluster the option for "Launch Type" is hardcoded to EC2. Is there a way, perhaps through the command line to schedule the task to run on Fargate?
Heads up ! This is now supported in AWS:
https://aws.amazon.com/about-aws/whats-new/2018/08/aws-fargate-now-supports-time-and-event-based-task-scheduling/
Although not in some regions - As at Apr-19 it still wasn't supported in EU-west-2 (London). Check the table at the top of this page to see if it's supported in the region you want: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/scheduled_tasks.html
There seem to be no way of scheduling a task on FARGATE.
Only way it can be done right now seems to be by having your 'scheduler' external to ECS. I did it with a lambda. You can also use something like a jenkins or a simple cron task that fires the aws-cli command to ECS, in both these cases though you will need an instance always running.
I wrote a lambda that accepts the params (overrides) to be sent to the ECS task and has the schedule the task was supposed to have.
Update:
It seems there is a schedule tab in FARGATE Cluster details now that will allow you set cron schedules on ECS tasks.
While the AWS Documentation gives you ways to do this through CloudFormation, it seems like they've not yet released this feature anyway. I have been trying to do something similar and ran into the same issue.
Once it does become available, this link from the aws docs should be useful. Here's how they suggest doing it, but I keep running into errors saying NetworkConfiguration is not recognized and LaunchType is not recognized.
"EcsParameters": {
"Group": "string",
"LaunchType": "string",
"NetworkConfiguration": {
"awsvpcConfiguration": {
"AssignPublicIp": "string",
"SecurityGroups": [ "string" ],
"Subnets": [ "string" ]
}
},
Update: Here is an alternative that did end up working for me through the aws events put-targets command on the aws cli!
Make sure your aws cli is up to date. This method fails for older
versions of the cli. run this to update: pip install awscli --upgrade --user
After that, you should be good to go. Use the aws events put-targets --rule <value> --targets <value> command. Make sure that before you run this command you have a rule already defined on your cluster. If not, you can do that with the aws events put-rule cmd too. Refer to the AWS docs for put-rule, and for put-targets.
An example of a rule from the documentation is given below:
aws events put-rule --name "DailyLambdaFunction" --schedule-expression "cron(0 9 * * ? *)"
The put-targets command that worked for me is this:
aws events put-targets --rule cli-RS-rule --targets '{"Arn": "arn:aws:ecs:1234/cluster/clustername","EcsParameters": {"LaunchType": "FARGATE","NetworkConfiguration": {"awsvpcConfiguration": {"AssignPublicIp": "ENABLED", "SecurityGroups": [ "sg-id1233" ], "Subnets": [ "subnet-1234" ] }},"TaskCount": 1,"TaskDefinitionArn": "arn:aws:ecs:1234:task-definition/taskdef"},"Id": "sampleID111","RoleArn": "arn:aws:iam:1234:role/eventrole"}'
You can create a CloudWatch rule that uses a schedule as the event source and an ESC task as the target.
No this is not supported yet unfortunately. There is an open issue here. Hopefully it gets done soon as I would like to use it as well!
Disclosure: I work for SenseDeep that provides Powerdown # https://www.powerdown.io
Other services provide this functionality. PowerDown gives the ability to schedule Fargate services. This is at the service level, not the task level, but it is easy to create services for tasks. For example: you could schedule a CICD pipeline container to run 9-5 M-F.
It's not possible to have EC2 instances and Fargate instances at the same cluster.
It's possible to schedule a Fargate instance. Create a specific service and update that from aws tools. Ex:
aws ecs update-service --service my-http-service --task-definition
https://docs.aws.amazon.com/cli/latest/reference/ecs/update-service.html
Useful resources:
You could use the ECS aws tools and execute on lambda or travis.
Check out this medium post:
https://medium.com/#joseignaciocastelli92/how-to-create-a-continuous-deployment-process-using-ecs-fargate-docker-travis-410d84b4d99e
At the button has this repository that has the aws commands:
https://github.com/JicLotus/ecs-farate-scripts-to-deploy-and-build
Bests