I'm using vault approle auth method to fetch secrets from vault. Below is my vault agent configmap.
---
apiVersion: v1
kind: ConfigMap
metadata:
name: my-configmap
data:
config-init.hcl: |
"auto_auth" = {
"method" = {
"config" = {
"role" = "test"
"role_id_file_path" = "roleid"
"secret_id_file_path" = "secretid"
"remove_secret_id_file_after_reading" = false
}
"type" = "approle"
}
"sink" = {
"config" = {
"path" = "/home/vault/.token"
}
"type" = "file"
"wrap_ttl" = "30m"
}
}
"vault" = {
"address" = "https://myvault.com"
}
"exit_after_auth" = true
"pid_file" = "/home/vault/.pid"
Then I'm referencing the above configmap in the deployment file.
annotations:
vault.hashicorp.com/agent-inject: 'true'
vault.hashicorp.com/agent-configmap: 'my-configmap'
But I get below error
vault-agent-init 2022-07-20T10:43:13.306Z [ERROR] auth.handler: error getting path or data from method: error="no known role ID" backoff=4m51.03s
Create a secret file in Kubernetes by using the RoleID and SecretID and pass the below annotation
vault.hashicorp.com/extra-secret: "secret-file"
Related
I read through the karpenter document at https://karpenter.sh/v0.16.1/getting-started/getting-started-with-terraform/#install-karpenter-helm-chart. I followed instructions step by step. I got errors at the end.
kubectl logs -f -n karpenter -l app.kubernetes.io/name=karpenter -c controller
DEBUG controller.provisioning Relaxing soft constraints for pod since it previously failed to schedule, removing: spec.topologySpreadConstraints = {"maxSkew":1,"topologyKey":"topology.kubernetes.io/zone","whenUnsatisfiable":"ScheduleAnyway","labelSelector":{"matchLabels":{"app.kubernetes.io/instance":"karpenter","app.kubernetes.io/name":"karpenter"}}} {"commit": "b157d45", "pod": "karpenter/karpenter-5755bb5b54-rh65t"}
2022-09-10T00:13:13.122Z
ERROR controller.provisioning Could not schedule pod, incompatible with provisioner "default", incompatible requirements, key karpenter.sh/provisioner-name, karpenter.sh/provisioner-name DoesNotExist not in karpenter.sh/provisioner-name In [default] {"commit": "b157d45", "pod": "karpenter/karpenter-5755bb5b54-rh65t"}
Below is the source code:
cat main.tf
terraform {
required_version = "~> 1.0"
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 4.0"
}
helm = {
source = "hashicorp/helm"
version = "~> 2.5"
}
kubectl = {
source = "gavinbunney/kubectl"
version = "~> 1.14"
}
}
}
provider "aws" {
region = "us-east-1"
}
locals {
cluster_name = "karpenter-demo"
# Used to determine correct partition (i.e. - `aws`, `aws-gov`, `aws-cn`, etc.)
partition = data.aws_partition.current.partition
}
data "aws_partition" "current" {}
module "vpc" {
# https://registry.terraform.io/modules/terraform-aws-modules/vpc/aws/latest
source = "terraform-aws-modules/vpc/aws"
version = "3.14.4"
name = local.cluster_name
cidr = "10.0.0.0/16"
azs = ["us-east-1a", "us-east-1b", "us-east-1c"]
private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
public_subnets = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]
enable_nat_gateway = true
single_nat_gateway = true
one_nat_gateway_per_az = false
public_subnet_tags = {
"kubernetes.io/cluster/${local.cluster_name}" = "shared"
"kubernetes.io/role/elb" = 1
}
private_subnet_tags = {
"kubernetes.io/cluster/${local.cluster_name}" = "shared"
"kubernetes.io/role/internal-elb" = 1
}
}
module "eks" {
# https://registry.terraform.io/modules/terraform-aws-modules/eks/aws/latest
source = "terraform-aws-modules/eks/aws"
version = "18.29.0"
cluster_name = local.cluster_name
cluster_version = "1.22"
vpc_id = module.vpc.vpc_id
subnet_ids = module.vpc.private_subnets
# Required for Karpenter role below
enable_irsa = true
node_security_group_additional_rules = {
ingress_nodes_karpenter_port = {
description = "Cluster API to Node group for Karpenter webhook"
protocol = "tcp"
from_port = 8443
to_port = 8443
type = "ingress"
source_cluster_security_group = true
}
}
node_security_group_tags = {
# NOTE - if creating multiple security groups with this module, only tag the
# security group that Karpenter should utilize with the following tag
# (i.e. - at most, only one security group should have this tag in your account)
"karpenter.sh/discovery/${local.cluster_name}" = local.cluster_name
}
# Only need one node to get Karpenter up and running.
# This ensures core services such as VPC CNI, CoreDNS, etc. are up and running
# so that Karpenter can be deployed and start managing compute capacity as required
eks_managed_node_groups = {
initial = {
instance_types = ["m5.large"]
# Not required nor used - avoid tagging two security groups with same tag as well
create_security_group = false
min_size = 1
max_size = 1
desired_size = 1
iam_role_additional_policies = [
"arn:${local.partition}:iam::aws:policy/AmazonSSMManagedInstanceCore", # Required by Karpenter
"arn:${local.partition}:iam::aws:policy/AmazonEKSWorkerNodePolicy",
"arn:${local.partition}:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly", #for access to ECR images
"arn:${local.partition}:iam::aws:policy/CloudWatchAgentServerPolicy"
]
tags = {
# This will tag the launch template created for use by Karpenter
"karpenter.sh/discovery/${local.cluster_name}" = local.cluster_name
}
}
}
}
#The EKS module creates an IAM role for the EKS managed node group nodes. We’ll use that for Karpenter.
#We need to create an instance profile we can reference.
#Karpenter can use this instance profile to launch new EC2 instances and those instances will be able to connect to your cluster.
resource "aws_iam_instance_profile" "karpenter" {
name = "KarpenterNodeInstanceProfile-${local.cluster_name}"
role = module.eks.eks_managed_node_groups["initial"].iam_role_name
}
#Create the KarpenterController IAM Role
#Karpenter requires permissions like launching instances, which means it needs an IAM role that grants it access. The config
#below will create an AWS IAM Role, attach a policy, and authorize the Service Account to assume the role using IRSA. We will
#create the ServiceAccount and connect it to this role during the Helm chart install.
module "karpenter_irsa" {
source = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
version = "5.3.3"
role_name = "karpenter-controller-${local.cluster_name}"
attach_karpenter_controller_policy = true
karpenter_tag_key = "karpenter.sh/discovery/${local.cluster_name}"
karpenter_controller_cluster_id = module.eks.cluster_id
karpenter_controller_node_iam_role_arns = [
module.eks.eks_managed_node_groups["initial"].iam_role_arn
]
oidc_providers = {
ex = {
provider_arn = module.eks.oidc_provider_arn
namespace_service_accounts = ["karpenter:karpenter"]
}
}
}
#Install Karpenter Helm Chart
#Use helm to deploy Karpenter to the cluster. We are going to use the helm_release Terraform resource to do the deploy and pass in the
#cluster details and IAM role Karpenter needs to assume.
provider "helm" {
kubernetes {
host = module.eks.cluster_endpoint
cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
exec {
api_version = "client.authentication.k8s.io/v1beta1"
command = "aws"
args = ["eks", "get-token", "--cluster-name", local.cluster_name]
}
}
}
resource "helm_release" "karpenter" {
namespace = "karpenter"
create_namespace = true
name = "karpenter"
repository = "https://charts.karpenter.sh"
chart = "karpenter"
version = "v0.16.1"
set {
name = "serviceAccount.annotations.eks\\.amazonaws\\.com/role-arn"
value = module.karpenter_irsa.iam_role_arn
}
set {
name = "clusterName"
value = module.eks.cluster_id
}
set {
name = "clusterEndpoint"
value = module.eks.cluster_endpoint
}
set {
name = "aws.defaultInstanceProfile"
value = aws_iam_instance_profile.karpenter.name
}
}
#Provisioner
#Create a default provisioner using the command below. This provisioner configures instances to connect to your cluster’s endpoint and
#discovers resources like subnets and security groups using the cluster’s name.
#This provisioner will create capacity as long as the sum of all created capacity is less than the specified limit.
provider "kubectl" {
apply_retry_count = 5
host = module.eks.cluster_endpoint
cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
load_config_file = false
exec {
api_version = "client.authentication.k8s.io/v1beta1"
command = "aws"
args = ["eks", "get-token", "--cluster-name", module.eks.cluster_id]
}
}
resource "kubectl_manifest" "karpenter_provisioner" {
yaml_body = <<-YAML
apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
name: default
spec:
requirements:
- key: karpenter.sh/capacity-type
operator: In
values: ["spot"]
limits:
resources:
cpu: 1000
provider:
subnetSelector:
Name: "*private*"
securityGroupSelector:
karpenter.sh/discovery/${module.eks.cluster_id}: ${module.eks.cluster_id}
tags:
karpenter.sh/discovery/${module.eks.cluster_id}: ${module.eks.cluster_id}
ttlSecondsAfterEmpty: 30
YAML
depends_on = [
helm_release.karpenter
]
}
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: inflate
spec:
replicas: 0
selector:
matchLabels:
app: inflate
template:
metadata:
labels:
app: inflate
spec:
terminationGracePeriodSeconds: 0
containers:
- name: inflate
image: public.ecr.aws/eks-distro/kubernetes/pause:3.2
resources:
requests:
cpu: 1
EOF
kubectl scale deployment inflate --replicas 5
kubectl logs -f -n karpenter -l app.kubernetes.io/name=karpenter -c controller
DEBUG controller.provisioning Relaxing soft constraints for pod since it previously failed to schedule, removing: spec.topologySpreadConstraints = {"maxSkew":1,"topologyKey":"topology.kubernetes.io/zone","whenUnsatisfiable":"ScheduleAnyway","labelSelector":{"matchLabels":{"app.kubernetes.io/instance":"karpenter","app.kubernetes.io/name":"karpenter"}}} {"commit": "b157d45", "pod": "karpenter/karpenter-5755bb5b54-rh65t"}
2022-09-10T00:13:13.122Z
ERROR controller.provisioning Could not schedule pod, incompatible with provisioner "default", incompatible requirements, key karpenter.sh/provisioner-name, karpenter.sh/provisioner-name DoesNotExist not in karpenter.sh/provisioner-name In [default] {"commit": "b157d45", "pod": "karpenter/karpenter-5755bb5b54-rh65t"}
I belive this is due to the pod topology defined in the Karpenter deployment here:
https://github.com/aws/karpenter/blob/main/charts/karpenter/values.yaml#L73-L77
, you can read further on what pod topologySpreadConstraints does here:
https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/
If you increase the desired_size to 2 which matches the default deployment replicas above, that should resove the error.
I'm trying to install Vault on a Kubernetes Cluster by running the Vault Helm chart out of Terraform. For some reason the ingress doesn't get created.
When I forward the pods port the ui comes up fine, so I assume everything is working, but the ingress not being available is tripping me up.
Edit: There are no errors while running terraform apply.
If there is another point where I should look, please tell me.
This is my helm_release resource:
name = "vault"
repository = "https://helm.releases.hashicorp.com"
chart = "vault"
namespace = "vault"
create_namespace = true
set {
name = "ui.enabled"
value = "true"
}
#Set ingress up to use cert-manager provided secret
set {
name = "ingress.enabled"
value = "true"
}
set {
name = "ingress.annotations.cert-manager\\.io/cluster-issuer"
value = "letsencrypt-cluster-prod"
}
set {
name = "ingress.annotations.kubernetes\\.io/ingress\\.class"
value = "nginx"
}
set {
name = "ingress.tls[0].hosts[0]"
value = var.vault_hostname
}
set {
name = "ingress.hosts[0].host"
value = var.vault_hostname
}
set {
name = "ingress.hosts[0].paths[0]"
value = "/"
}
}
I'm relatively new to all of these techs, having worked with puppet before, so if someone could point me in the right direction, I'd be much obliged.
I achieved enabling ingress with a local variable, here is the working example
locals {
values = {
server= {
ingress = {
enabled = var.server_enabled
labels = {
traffic = "external"
}
ingressClassName = "nginx"
annotations = {
"kubernetes.io/tls-acme" = "true"
"nginx.ingress.kubernetes.io/ssl-redirect" = "true"
}
hosts = [{
host = vault.example.com
paths = ["/"]
}]
tls = [
{
secretName = vault-tls-secret
hosts = ["vault.example.com"]
}
]
}
}
}
}
resource "helm_release" "vault" {
name = "vault"
namespace = "vault"
repository = "https://helm.releases.hashicorp.com"
chart = "vault"
version = "0.19.0"
create_namespace = true
# other value to set
#set {
# name = "server.ha.enabled"
#value = "true"
#}
values = [
yamlencode(local.values)
]
}
I want to automate inserting the ARNs of specific roles into an EKS aws-auth ConfigMap, right after deploying the cluster. However, it seems that Terraform is recommending using kubectl instead.
I have tried the following method but I'm getting an error that the data block is not expecting here.
data "aws_eks_cluster_auth" "cluster_auth" {
name = "my_cluster"
}
provider "kubernetes" {
host = aws_eks_cluster.my_cluster.endpoint
cluster_ca_certificate = base64decode(aws_eks_cluster.my_cluster.certificate_authority.0.data)
token = data.aws_eks_cluster_auth.cluster_auth.token
}
resource "kubernetes_config_map" "aws_auth_configmap" {
metadata {
name = "aws-auth"
namespace = "kube-system"
}
data {
mapRoles = <<YAML
- rolearn: arn:aws:iam::111111111111:role/MyRole
username: system:node:{{EC2PrivateDNSName}}
groups:
- system:bootstrappers
- system:nodes
- rolearn: arn:aws:iam::111111111111:role/MyRole
username: kubectl
groups:
- system:masters
YAML
}
}
you can give the aidanmelen/eks-auth/aws a try ;)
module "eks" {
source = "terraform-aws-modules/eks/aws"
# insert the 15 required variables here
}
module "eks_auth" {
source = "aidanmelen/eks-auth/aws"
eks = module.eks
map_roles = [
{
rolearn = "arn:aws:iam::66666666666:role/role1"
username = "role1"
groups = ["system:masters"]
},
]
map_users = [
{
userarn = "arn:aws:iam::66666666666:user/user1"
username = "user1"
groups = ["system:masters"]
},
{
userarn = "arn:aws:iam::66666666666:user/user2"
username = "user2"
groups = ["system:masters"]
},
]
map_accounts = [
"777777777777",
"888888888888",
]
}
I am having issues installing the cert-manager Helm chart and setting up a LetsEncrypt cluster issuer using Pulumi in our Azure Kubernetes cluster. We are using Kubernetes version 1.21.2 and cert-manager. 1.5.3.
When running pulumi up before any of the resources exist I get the following error:
kubernetes:cert-manager.io/v1:ClusterIssuer (cert-manager-letsencrypt):
error: creation of resource cert-manager/letsencrypt failed because the Kubernetes API server reported that the apiVersion for this resource does not exist. Verify that any required CRDs have been created: no matches for kind "ClusterIssuer" in version "cert-manager.io/v1"
error: update failedaToolsCertManager cert-manager
I can confirm that the cluster issuer hasn't been created by running kubectl get clusterissuer.
When running pulumi up again it succeeds and the letsencrypt ClusterIssuer is correctly created.
I don't want to have to run pulumi up consecutive times to reach a successful deployment. Can anyone see what the issue is here?
C# stack definition:
// Create new namespace
var certManagerNamespace = new Namespace("cert-manager",
new NamespaceArgs()
{
Metadata = new ObjectMetaArgs
{
Name = "cert-manager"
}
},
options);
// Install cert-manager using Helm
var certManagerChart = new Chart("cert-manager",
new ChartArgs
{
Chart = "cert-manager",
Version = "1.5.3",
Namespace = certManagerNamespace.Metadata.Apply(m => m.Name),
Values =
{
["installCRDs"] = "true"
},
FetchOptions = new ChartFetchArgs
{
Repo = "https://charts.jetstack.io"
}
},
options);
// Create ClusterIssuer using LetsEncrypt
var clusterIssuer = new ClusterIssuer($"{name}-letsencrypt",
new ClusterIssuerArgs()
{
ApiVersion = "cert-manager.io/v1",
Kind = "ClusterIssuer",
Metadata = new ObjectMetaArgs()
{
Name = "letsencrypt",
Namespace = "cert-manager",
},
Spec = new ClusterIssuerSpecArgs()
{
Acme = new ClusterIssuerSpecAcmeArgs()
{
Email = "administrator#blahblah.com",
Server = "https://acme-v02.api.letsencrypt.org/directory",
PrivateKeySecretRef = new ClusterIssuerSpecAcmePrivateKeySecretRefArgs()
{
Name = "letsencrypt"
},
Solvers =
{
new ClusterIssuerSpecAcmeSolversArgs()
{
Http01 = new ClusterIssuerSpecAcmeSolversHttp01Args()
{
Ingress = new ClusterIssuerSpecAcmeSolversHttp01IngressArgs()
{
Class = "nginx"
}
}
}
}
}
}
},
new CustomResourceOptions()
{
DependsOn = certManagerChart,
Provider = options.Provider
});
The cluster issuer definition from Pulumi:
+ kubernetes:cert-manager.io/v1:ClusterIssuer: (create)
[urn=urn:pulumi:preprod::MyAks::kubernetes:cert-manager.io/v1:ClusterIssuer::cert-manager-letsencrypt]
[provider=urn:pulumi:preprod::MyAks::k8sx:service:MyAks$pulumi:providers:kubernetes::k8s-provider::5191350f-c03b-4796-bc48-053584e2c996]
apiVersion: "cert-manager.io/v1"
kind : "ClusterIssuer"
metadata : {
labels : {
app.kubernetes.io/managed-by: "pulumi"
}
name : "letsencrypt"
namespace: "cert-manager"
}
spec : {
acme: {
email : "administrator#blahblah.com"
privateKeySecretRef: {
name: "letsencrypt"
}
server : "https://acme-v02.api.letsencrypt.org/directory"
solvers : [
[0]: {
http01: {
ingress: {
class: "nginx"
}
}
}
]
}
}
Using kubectl we can create docker registry authentication secret as follows
kubectl create secret docker-registry regsecret \
--docker-server=docker.example.com \
--docker-username=kube \
--docker-password=PW_STRING \
--docker-email=my#email.com \
How do i create this secret using terraform, i saw this link, it has data, in the flow of terraform the kubernetes instance is being created in azure and i get the data required from there and i created something like below
resource "kubernetes_secret" "docker-registry" {
metadata {
name = "registry-credentials"
}
data = {
docker-server = data.azurerm_container_registry.docker_registry_data.login_server
docker-username = data.azurerm_container_registry.docker_registry_data.admin_username
docker-password = data.azurerm_container_registry.docker_registry_data.admin_password
}
}
It seems that it is wrong as the images are not being pulled. What am i missing here.
If you run following command
kubectl create secret docker-registry regsecret \
--docker-server=docker.example.com \
--docker-username=kube \
--docker-password=PW_STRING \
--docker-email=my#email.com
It will create a secret like following
$ kubectl get secrets regsecret -o yaml
apiVersion: v1
data:
.dockerconfigjson: eyJhdXRocyI6eyJkb2NrZXIuZXhhbXBsZS5jb20iOnsidXNlcm5hbWUiOiJrdWJlIiwicGFzc3dvcmQiOiJQV19TVFJJTkciLCJlbWFpbCI6Im15QGVtYWlsLmNvbSIsImF1dGgiOiJhM1ZpWlRwUVYxOVRWRkpKVGtjPSJ9fX0=
kind: Secret
metadata:
creationTimestamp: "2020-06-01T18:31:07Z"
name: regsecret
namespace: default
resourceVersion: "42304"
selfLink: /api/v1/namespaces/default/secrets/regsecret
uid: 59054483-2789-4dd2-9321-74d911eef610
type: kubernetes.io/dockerconfigjson
If we decode .dockerconfigjson we will get
{"auths":{"docker.example.com":{"username":"kube","password":"PW_STRING","email":"my#email.com","auth":"a3ViZTpQV19TVFJJTkc="}}}
So, how can we do that using terraform?
I created a file config.json with following data
{"auths":{"${docker-server}":{"username":"${docker-username}","password":"${docker-password}","email":"${docker-email}","auth":"${auth}"}}}
Then in main.tf file
resource "kubernetes_secret" "docker-registry" {
metadata {
name = "regsecret"
}
data = {
".dockerconfigjson" = "${data.template_file.docker_config_script.rendered}"
}
type = "kubernetes.io/dockerconfigjson"
}
data "template_file" "docker_config_script" {
template = "${file("${path.module}/config.json")}"
vars = {
docker-username = "${var.docker-username}"
docker-password = "${var.docker-password}"
docker-server = "${var.docker-server}"
docker-email = "${var.docker-email}"
auth = base64encode("${var.docker-username}:${var.docker-password}")
}
}
then run
$ terraform apply
This will generate same secrets. Hope it will helps
I would suggest creating a azurerm_role_assignement to give aks access to the acr:
resource "azurerm_role_assignment" "aks_sp_acr" {
scope = azurerm_container_registry.acr.id
role_definition_name = "AcrPull"
principal_id = var.service_principal_obj_id
depends_on = [
azurerm_kubernetes_cluster.aks,
azurerm_container_registry.acr
]
}
Update
You can create the service principal in the azure portal or with az cli and use client_id, client_secret and object-id in terraform.
Get Client_id and Object_id by running az ad sp list --filter "displayName eq '<name>'". The secret has to be created in the Certificates & secrets tab of the service principal. See this guide: https://pixelrobots.co.uk/2018/11/first-look-at-terraform-and-the-azure-cloud-shell/
Just set all three as variable, eg for obj_id:
variable "service_principal_obj_id" {
default = "<object-id>"
}
Now use the credentials with aks:
resource "azurerm_kubernetes_cluster" "aks" {
...
service_principal {
client_id = var.service_principal_app_id
client_secret = var.service_principal_password
}
...
}
And set the object id in the acr as described above.
Alternative
You can create the service principal with terraform (only works if you have the necessary permissions). https://www.terraform.io/docs/providers/azuread/r/service_principal.html combined with a random_password resource:
resource "azuread_application" "aks_sp" {
name = "somename"
available_to_other_tenants = false
oauth2_allow_implicit_flow = false
}
resource "azuread_service_principal" "aks_sp" {
application_id = azuread_application.aks_sp.application_id
depends_on = [
azuread_application.aks_sp
]
}
resource "azuread_service_principal_password" "aks_sp_pwd" {
service_principal_id = azuread_service_principal.aks_sp.id
value = random_password.aks_sp_pwd.result
end_date = "2099-01-01T01:02:03Z"
depends_on = [
azuread_service_principal.aks_sp
]
}
You need to assign the role "Conributer" to the sp and can use it directly in aks / acr.
resource "azurerm_role_assignment" "aks_sp_role_assignment" {
scope = var.subscription_id
role_definition_name = "Contributor"
principal_id = azuread_service_principal.aks_sp.id
depends_on = [
azuread_service_principal_password.aks_sp_pwd
]
}
Use them with aks:
resource "azurerm_kubernetes_cluster" "aks" {
...
service_principal {
client_id = azuread_service_principal.aks_sp.app_id
client_secret = azuread_service_principal_password.aks_sp_pwd.value
}
...
}
and the role assignment:
resource "azurerm_role_assignment" "aks_sp_acr" {
scope = azurerm_container_registry.acr.id
role_definition_name = "AcrPull"
principal_id = azuread_service_principal.aks_sp.object_id
depends_on = [
azurerm_kubernetes_cluster.aks,
azurerm_container_registry.acr
]
}
Update secret example
resource "random_password" "aks_sp_pwd" {
length = 32
special = true
}