I am trying to build an AWS EKS Cluster with AWS cdk in Java.
We have an existing VPC and subnets which need to get some Kubernetes tags like kubernetes.io/role/internal-elb=1 etc.
I can get the ISubnets by getting the vpc with:
IVpc vpc = Vpc.fromVpcAttributes(this, "my-vpc", vpcAttributes);
List<ISubnet> subnets = vpc.getPrivateSubnets();
subnets.forEach(iSubnet -> Tag.add(iSubnet, "kubernetes.io/role/internal-elb", "1"));
but awscdk.core.Tag.add() is expecting a Construct, which I am not creating because the subnet already exists.
Also tried the example here: https://docs.aws.amazon.com/de_de/cdk/latest/guide/tagging.html
private void addTagToAllVPCSubnets(Tag tag) {
TagProps includeOnlySubnets = TagProps.builder()
.includeResourceTypes(singletonList("AWS::EC2::Subnet"))
.build();
Tag.add(this, tag.getKey(), tag.getValue(), includeOnlySubnets);
}
... but still i can not see any of the new tags in the CF yaml of the cdk synth.
Any help will be appreciated!
You can do it automatically using lambda-supported custom resources.
It seems like this is a limitation in CDK at the moment. This is something that the EKS construct in CDK should deal with, but which is currently not possible as indicated by a warning during a CDK deployment:
[Warning at /stack/some-project-EKS-cluster] Could not auto-tag private subnets with "kubernetes.io/role/internal-elb=1", please remember to do this manually
For the same reason that this can't be done automatically, you can't do it by using Tag.add().
Since the EKS module in CDK is still experimental/development preview, you have three options right now:
Wait for a full release, which perhaps includes automatic subnet tagging.
Create your own VPC through CDK, which allows you to tag your own subnets.
Manually edit existing subnets through the VPC service interface in the AWS console
A good idea would probably be to create an issue on the AWS CDK Github and request tagging existing subnets (and other existing constructs in general) as a feature. I could not find other issues regarding this on their Github.
Related
I'm working with ECS with self managed EC2 based clusters. We have 1 cluster for each env, dev/stage/prod
I'm struggling to have my containers in ECS be aware of what cluster / environment they start in so that on task start up time they can properly configure themselves without having to bake the env specific config into the images.
It would be really easy if there was some command to run inside the container that could return the cluster name. It seems like that should be easy. I can think of a few sub optimal ways to do this. get the container/host IP and look up the instance. Try to grab /etc/ecs/ecs.config from the host instance etc...
It seems like there should be a better way. my google skills are failing... thx!
The ECS Task Metadata endpoint, available at ${ECS_CONTAINER_METADATA_URI_V4}/task within any ECS task, will return the cluster name, among other things.
Alternatively, if you were using an IaC tool such as Terraform or CloudFormation to build your ECS tasks, it would be trivial to inject the cluster name as an environment variable in the tasks.
Mark B's answer is better but before I got that I found this solution:
Add ECS_ENABLE_CONTAINER_METADATA=true to the /etc/ecs/ecs.config file on the ec2 host and you will have access to the ecs.config file as well as having the file available as and env var. See:
[Ecs Container Metadata File][1]
I think Mark's answer is better b/c this solution involves editing the userdata script for the host instances
[1]: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/container-metadata.html
A cloud run project that worked two months ago suddenly started complaining about the default log bucket being outside the VPC-SC perimeter. However, this project is not in an organisation, so I don't understand how I can remove the perimeter.
gcloud builds submit --tag [tag]
Errors with:
ERROR: (gcloud.builds.submit)
The build is running, and logs are being written to the default logs bucket.
Unfortunately, the default logs bucket is always outside any VPC-SC security
perimeter, so this tool cannot stream the logs for you.
While changing the controls is not possible:
If you still have the issue, I reviewed documentation according to your main question of how to remove gcloud VPC-SC security perimeter, and, if you activate VPC accessible services and then decide that the VPC networks in your perimeter no longer need access to the Cloud Storage service, you can remove services from your service perimeter's VPC accessible services using the following command:
gcloud access-context-manager perimeters update example_perimeter \
--remove-vpc-allowed-services=example.storage.googleapis.com \
--policy=example.11271009391
If the issue persists, you can leave a comment so that we can continue helping you, or here is a link that helps to troubleshoot any issue related to VPC Service Control.
Update your gcloud tools......
I am studying about CI/CD on AWS (CodePipeline/CodeBuild/CodeDeploy) and found it to be a very good tool for managing a pipeline on the cloud with everything managed (don't even need to install Jenkins on EC2).
I am now reading about container building and deployment. For the build phase, CodeBuild supports building container images. For the deploy phase, while I could find a CodeDeploy solution to ECS cluster, it seems there is no direct CodeDeploy solution for EKS (kindly correct if I am wrong).
May I know if there is a solution to integrate EKS cluster (i.e. the deploy phase can fetch the docker image from ECR or dockerhub and deploy to EKS)? I have come across some ideas using lamda functions to trigger the cluster to perform rolling update of the container image, but I could not find a step-by-step guide on this.
=========================
(Update 17 Sep 2020)
Somehow managed to create a lambda function to trigger an update to EKS to perform rolling update of the k8s deployment. Thanks Prashanna for the source base.
Just want to share the key setups in the process.
(1) Update the lambda execution role to include permission to describe EKS clusters
Create a policy with describe EKS cluster access, and attach to the role:
Policy snippet:
...
......
"Action": "eks:Describe*"
...
......
Or you can create a "EKSFullAccess" policy, and attach to the lambda execution role
(2) Update the k8s ConfigMap, and supplement the lambda execution role ARN to the mapRole section. The corresponding k8s role should be a role that has permission to update container images (say system:masters) used for the k8s deployment
You can edit the map with command like below:
kubectl edit -n kube-system configmap/aws-auth
You don't have to add/update another ConfigMap even if your deployment is in another namespace. It will take effect as well.
Sample lambda function call request and response:
Gitab provides the inbuilt integration of EKS and deployment with the help of Helm charts. If you plan to use other tools Using AWS lambda to update the image is the best bet!
I've added my github project.
Setup a lambda with below code and give RBAC access to this lambda in your EKS. Try invoking the lambda by passing the required information like namespace, deployment, image etc
Lambda for Kubernetes image update
The lambda must require EKS:describecluster policy.
The Lambda role must be provided atleast update image RBAC role in EKS cluster RBAC role setup
Since there's no built-in CI/CD for EKS at the moment, this is going to be a showcase of success/failure stories of a 3rd-party CI/CDs in EKS :) My take: https://github.com/fluxcd/flux
Pros:
Quick to set up initially (until you get into multiple teams/environments)
Tracks and deploys image releases out of box
Possibility to split what to auto-deploy in dev/prod using regex. E.g. all versions to dev, only minor to prod. Or separate tag prefixes for dev/prod.
All state is in git - a good practice to start with
Cons:
Getting complex for further pipeline expansion, e.g. blue-green, canary, auto-rollbacks, etc.
The dashboard is proprietary (weave works product)
Not for on-demand parametrized job runs like traditional CIs.
Setup:
Setup an automated image build (looks like you've already figured out)
Setup flux and helm-operator into the cluster, point them to your "gitops repo"
For each app, create a HelmRelease object that describes a regex of image tag to track
Done. A newly published image tag that falls into regex will be auto-deployed to the cluster and the new version is committed to a gitops repo.
In my terraform infrastructure, I spin up several Kubernetes clusters based on parameters, then install some standard contents to those Kubernetes clusters using the kubernetes provider.
When I change the parameters and one of the clusters is no longer needed, terraform is unable to tear it down because the provider and resources are both in the module. I don't see an alternative, however, because I create the kubernetes cluster in that same module, and the kubernetes object are all per kubernetes cluster.
All solutions I can think of involve adding a bunch of boilerplate to my terraform config. Should I consider generating my terraform config from a script?
I made a git repo that shows exactly the problems I'm having:
https://github.com/bukzor/terraform-gke-k8s-demo
TL;DR
Two solutions:
Create two separate modules with Terraform
Use interpolations and depends_on between the code that creates your Kubernetes cluster and the kubernetes resources:
resource "kubernetes_service" "example" {
metadata {
name = "my-service"
}
depends_on = ["aws_vpc.kubernetes"]
}
resource "aws_vpc" "kubernetes" {
...
}
When destroying resources
You are encountering a dependency lifecycle issue
PS: I don't know the code you've used to create / provision your Kubernetes cluster but I guess it looks like this
Write code for the Kubernetes cluster (creates a VPC)
Apply it
Write code for provisionning Kubernetes (create an Service that creates an ELB)
Apply it
Try to destroy everything => Error
What is happenning is that by creating a LoadBalancer Service, Kubernetes will provision an ELB on AWS. But Terraform doesn't know that and there is no link between the ELB created and any other resources managed by Terraform.
So when terraform tries to destroy the resources in the code, it will try to destroy the VPC. But it can't because there is an ELB inside that VPC that terraform doesn't know about.
The first thing would be to make sure that Terraform "deprovision" the Kubernetes cluster and then destroy the cluster itself.
Two solutions here:
Use different modules so there is no dependency lifecycle. For example the first module could be k8s-infra and the other could be k8s-resources. The first one manages all the squeleton of Kubernetes and is apply first / destroy last. The second one manages what is inside the cluster and is apply last / destroy first.
Use the depends_on parameter to write the dependency lifecycle explicitly
When creating resources
You might also ran into a dependency issue when terraform apply cannot create resources even if nothing is applied yet. I'll give an other example with a postgres
Write code to create an RDS PostgreSQL server
Apply it with Terraform
Write code, in the same module, to provision that RDS instance with the postgres terraform provider
Apply it with Terraform
Destroy everything
Try to apply everything => ERROR
By debugging Terraform a bit I've learned that all the providers are initialized at the beggining of the plan / apply so if one has an invalid config (wrong API keys / unreachable endpoint) then Terraform will fail.
The solution here is to use the target parameter of a plan / apply command.
Terraform will only initialize providers that are related to the resources that are applied.
Apply the RDS code with the AWS provider: terraform apply -target=aws_db_instance
Apply everything terraform apply. Because the RDS instance is already reachable, the PostgreSQL provider can also initiate itself
I am using the existing single master Mesosphere DCOS cloud formation template:
https://s3.amazonaws.com/downloads.mesosphere.io/dcos/stable/cloudformation/single-master.cloudformation.json
I am trying to figure out how to indicate that I want to spin this up in an existing VPC that is already configured with a NAT/Internet gateway.
New to cloud formation and can't find any docs on the Mesosphere site around what the template actually creates and why. In addition there doesn't appear to be an all up manual setup tutorial. Just this template.
Thanks!
You can easily change the CF template, just delete the VPC resource, include the vpc-id as a parameter and change all the references to the VPC resource to point the new parameter.
In the same way you could replace the subnets in the template and remove the NAT instance.
I have made the changes in cloudformation script to install mesosphere in existing VPC/NAT.
https://github.com/navidurrahman/dcos-cloudformation
Let me know, if you face any problems
This cloud formation template installs dcos version 1.3. I have written a terraform module for latest mesosphere installation.
https://github.com/navidurrahman/terraform_mesosphere