I'm kicking off a single ECS Fargate task with a command such as:
aws ecs run-task --cluster Fargate \
--task-definition $ECR_REPO-run-setup
--overrides file:///tmp/ecs-overrides-db-migrate.txt \
--count 1 --launch-type FARGATE \
--network-configuration "awsvpcConfiguration={subnets=[$PUBLIC_SUBNET_1, $PUBLIC_SUBNET_2],securityGroups=[$FARGATE_SG],assignPubli cIp=ENABLED}"
There are no ECS services, tasks or instances at all running in my account at the moment. This is the response I get back:
{
"failures": [
{
"reason": "Capacity is unavailable at this time. Please try again later or in a different availability zone"
}
],
"tasks": []
}
I don't even see a way to specify a different availability zone for a Fargate Task?
If I should just retry, how long should I wait before retries?
Withing a VPC you can create one or more subnets that correspond to an availability zone.
When launching your Fargate task you will notice the network-configuration parameter and associated awsvpcConfiguration. To specify multiple zones you can pass in multiple subnets. For example:
aws ecs run-task --cluster Fargate \
--task-definition $ECR_REPO-run-setup
--overrides file:///tmp/ecs-overrides-db-migrate.txt \
--count 1 --launch-type FARGATE \
--network-configuration "awsvpcConfiguration={subnets=[$MY_SUBNET_IN_AZ1,
$MY_SUBNET_IN_AZ2]
The VPC documentation in aws contains more helpful information:
https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Subnets.html#vpc-subnet-basics
Related
I am running a regional GKE kubernetes cluster in is-central1-b us-central-1-c and us-central1-f. I am running 1.21.14-gke.700. I am adding a confidential node pool to the cluster with this command.
gcloud container node-pools create card-decrpyt-confidential-pool-1 \
--cluster=svcs-dev-1 \
--disk-size=100GB \
--disk-type=pd-standard \
--enable-autorepair \
--enable-autoupgrade \
--enable-gvnic \
--image-type=COS_CONTAINERD \
--machine-type="n2d-standard-2" \
--max-pods-per-node=8 \
--max-surge-upgrade=1 \
--max-unavailable-upgrade=1 \
--min-nodes=4 \
--node-locations=us-central1-b,us-central1-c,us-central1-f \
--node-taints=dedicatednode=card-decrypt:NoSchedule \
--node-version=1.21.14-gke.700 \
--num-nodes=4 \
--region=us-central1 \
--sandbox="type=gvisor" \
--scopes=https://www.googleapis.com/auth/cloud-platform \
--service-account="card-decrpyt-confidential#corp-dev-project.iam.gserviceaccount.com" \
--shielded-integrity-monitoring \
--shielded-secure-boot \
--tags=testingdonotuse \
--workload-metadata=GKE_METADATA \
--enable-confidential-nodes
This creates a node pool but there is one problem... I can still SSH to the instances that the node pool creates. This is unacceptable for my use case as these node pools need to be as secure as possible. I went into my node pool and created a new machine template with ssh turned off using an instance template based off the one created for my node pool.
gcloud compute instance-templates create card-decrypt-instance-template \
--project=corp-dev-project
--machine-type=n2d-standard-2
--network-interface=aliases=gke-svcs-dev-1-pods-10a0a3cd:/28,nic-type=GVNIC,subnet=corp-dev-project-private-subnet,no-address
--metadata=block-project-ssh-keys=true,enable-oslogin=true
--maintenance-policy=TERMINATE --provisioning-model=STANDARD
--service-account=card-decrpyt-confidential#corp-dev-project.iam.gserviceaccount.com
--scopes=https://www.googleapis.com/auth/cloud-platform
--region=us-central1 --min-cpu-platform=AMD\ Milan
--tags=testingdonotuse,gke-svcs-dev-1-10a0a3cd-node
--create-disk=auto-delete=yes,boot=yes,device-name=card-decrpy-instance-template,image=projects/confidential-vm-images/global/images/cos-89-16108-766-5,mode=rw,size=100,type=pd-standard
--shielded-secure-boot
--shielded-vtpm -
-shielded-integrity-monitoring
--labels=component=gke,goog-gke-node=,team=platform --reservation-affinity=any
When I change the instance templates of the nodes in the node pool the new instances come online but they do not attach to the node pool. The cluster is always trying to repair itself and I can't change any settings until I delete all the nodes in the pool. I don't receive any errors.
What do I need to do to disable ssh into the node pool nodes with the original node pool I created or with the new instance template I created. I have tried a bunch of different configurations with a new node pool and the cluster and have not had any luck. I've tried different tags network configs and images. None of these have worked.
Other info:
The cluster was not originally a confidential cluster. The confidential nodes are the first of its kind added to the cluster.
One option you have here is to enable private IP addresses for the nodes in your cluster. The --enable-private-nodes flag will make it so the nodes in your cluster get private IP addresses (rather than the default public, internet-facing IP addresses).
Note that in this case, you would still be able to SSH into these nodes, but only from within your VPC network.
Also note that this means you would not be able to access NodePort type services from outside of your VPC network. Instead, you would need to use a LoadBalancer type service (or provide some other way to route traffic to your service from outside of the cluster, if required).
If you'd like to prevent SSH access even from within your VPC network, your easiest option would likely be to configure a firewall rule to deny SSH traffic to your nodes (TCP/UDP/SCTP port 22). Use network tags (the --tags flag) to target your GKE nodes.
Something along the lines of:
gcloud compute firewall-rules create fw-d-i-ssh-to-gke-nodes \
--network NETWORK_NAME \
--action deny \
--direction ingress \
--rules tcp:22,udp:22,sctp:22 \
--source-ranges 0.0.0.0/0 \
--priority 65534 \
--target-tags my-gke-node-network-tag
Finally, one last option I'll mention for creating a hardened GKE cluster is to use Google's safer-cluster Terraform module. This is an opinionated setup of a GKE cluster that follows many of the principles laid out in Google's cluster hardening guide and the Terraform module takes care of a lot of the nitty-gritty boilerplate here.
I needed the metadata flag when creating the node pool
--metadata=block-project-ssh-keys=TRUE \
This blocked ssh.
However, enable-os-login=false won't work because it is reserved for use by the Kubernetes Engine
I started an AWS EKS cluster using AWS CloudFormation. I then waited for the stack creation to complete through the following command:
$ aws cloudformation wait stack-create-complete --stack-name "myclusterstack"
I also waited for the EKS cluster to be active through the following command:
aws eks wait cluster-active --name "mycluster"
After these commands complete, I started up an AWS EKS Nodegroup using AWS CloudFormation. However, I get the following error in the CloudFormation stack events:
Resource handler returned message: "Cluster 'mycluster' is not in ACTIVE status (Service: Eks, Status Code: 400, Request ID: a9aa2e4e-1b17-4fd4-bd89-851037867b25)" (RequestToken: 410bd19e-7710-b637-a30c-889f5b5a8893, HandlerErrorCode: InvalidRequest)
I noticed, however, that if I wait for around 5 minutes after creation of the cluster, I don't get this error when I create the nodegroup.
It seems like a few more minutes is needed after aws eks wait cluster-active to be able to successfully launch a nodegroup.
I also tried checking if the cluster endpoint is available as a means to check if the cluster is already active:
aws eks describe-cluster --name "mycluster" --query 'cluster.endpoint'
However, creation of the nodegroup still fails even when the cluster endpoint is already available.
Is sleeping for a few minutes the only way to be able to spin up a nodegroup after cluster creation? What should be the proper way to wait for a cluster to be active to be able to launch a nodegroup?
To prevent Cluster Auto Scaler from terminating some of nodes, I would need to annotate them with:
cluster-autoscaler.kubernetes.io/scale-down-disabled=true;
Is there a way to do so in USERDATA script?
For labeling the nodes, there is no issue, and it is possible to do so via:
--kubelet-extra-args \
"--node-labels=
Thanks
No, it not possible.
The list of supported parameters for the bootstrap script:
--use-max-pods Sets --max-pods for the kubelet when true. (default: true)
--b64-cluster-ca The base64 encoded cluster CA content. Only valid when used with --apiserver-endpoint. Bypasses calling \"aws eks describe-cluster\"
--apiserver-endpoint The EKS cluster API Server endpoint. Only valid when used with --b64-cluster-ca. Bypasses calling \"aws eks describe-cluster\"
--kubelet-extra-args Extra arguments to add to the kubelet. Useful for adding labels or taints.
--enable-docker-bridge Restores the docker default bridge network. (default: false)
--aws-api-retry-attempts Number of retry attempts for AWS API call (DescribeCluster) (default: 3)
--docker-config-json The contents of the /etc/docker/daemon.json file. Useful if you want a custom config differing from the default one in the AMI
You can add node labels, taints, etc by using the --kubelet-extra-args option on the bootstrap.sh invokation as you guessed. For an example, see the AWS Blog post: Improvements for Amazon EKS Worker Node Provisioning
Use a USERDATA script similar to the following:
UserData: !Base64
"Fn::Sub": |
#!/bin/bash
set -o xtrace
/etc/eks/bootstrap.sh ${ClusterName} ${BootstrapArguments}
/opt/aws/bin/cfn-signal --exit-code $? \
--stack ${AWS::StackName} \
--resource NodeGroup \
--region ${AWS::Region}
The above is a fragment from the CloudFormation template. Of course you can make your script more complex, with security hardening, etc. if you so desire.
For a complete CloudFormation template, download the sample from AWS:
curl -O https://amazon-eks.s3-us-west-2.amazonaws.com/cloudformation/2019-11-15/amazon-eks-nodegroup.yaml
It is absolutely possible. Here is part of my example userdata, specifically useful if you want to run both OnDemand and Spot instance. In my example I am adding lifecycle node label which changes based on the type. See below:
--use-max-pods 'true' \
--kubelet-extra-args ' --node-labels=lifecycle=OnDemand \
--system-reserved cpu=250m,memory=0.2Gi,ephemeral-storage=1Gi \
--kube-reserved cpu=250m,memory=1Gi,ephemeral-storage=1Gi \
--eviction-hard memory.available<0.2Gi,nodefs.available<10% \
--event-qps 0'
I hope that gives you a nice example.
I am trying to deploy my ecs but I faced a timeout issue
and my deployment get stuck here. How does this happen?
this is my step to get the taskArn using aws cli
cluster="clustername"
service_arn=$( aws ecs list-services --cluster $cluster --query 'serviceArns[0]' --output text )
task_arn=$( aws ecs list-tasks --cluster $cluster --service-name $service_arn --query 'taskArns[0]' --output text )
I realised that I should not set the minimum healthy percentage to 50%. I should set it to 0% so as to allow the task to change
I am trying to create scheduled task with cloudwatch.
I am using this page
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-events-rule-target.html
The problem i see is when i run task normally then aws asks
vpc
subnets
Launchtype
BUT when i use cloudwatch target then it dont ask for vpc, subnets etc. why is that ?
CloudFormation has not been updated to accommodate some Fargate functionality yet. If you get an error while trying to deploy an ECS task from CloudFormation,
try using the command line interface (aws events put-target) instead, which allows you to add a target that contains the required ECS parameters for launch type and network config.
Here is an example of how I configured my ECS tasks to be deployed from the CLI instead of CloudFormation:
1. Add vpc/subnet config to a variable, NETWORK_CONFIGURATION:
NETWORK_CONFIGURATION='{"awsvpcConfiguration":{"AssignPublicIp":"ENABLED","SecurityGroups": \["'${AWS_NETWORKCONFIG_SECURITY_GROUP}'"],"Subnets":["'${AWS_NETWORKCONFIG_SUBNET}'"]}}'
Run the following command to deploy your task, which will take the vpc config from the variable declared above
aws events put-targets \
--rule events-rule--${TASK_NAME} \
--targets '{"Arn":"arn:aws:ecs:'${AWS_REGION}':'${AWS_ACCOUNT_ID}':cluster/ecs-cluster-1","EcsParameters":{"LaunchType":"FARGATE","NetworkConfiguration":'${NETWORK_CONFIGURATION}',"TaskCount": 1,"TaskDefinitionArn": "arn:aws:ecs:'${AWS_REGION}':'${AWS_ACCOUNT_ID}':task-definition/ecs-task-'${TASK_NAME}'"},"Id": "ecs-targets-'${TASK_NAME}'","RoleArn": "arn:aws:iam::'${AWS_ACCOUNT_ID}':role/ecsEventsRole"}' \
;