I'm setting up an Airflow environment on Google Cloud Composer for testing. I've added some secrets to my namespace, and they show up fine:
$ kubectl describe secrets/eric-env-vars
Name: eric-env-vars
Namespace: eric-dev
Labels: <none>
Annotations: <none>
Type: Opaque
Data
====
VERSION_NUMBER: 6 bytes
I've referenced this secret in my DAG definition file (leaving out some code for brevity):
env_var_secret = Secret(
deploy_type='env',
deploy_target='VERSION_NUMBER',
secret='eric-env-vars',
key='VERSION_NUMBER',
)
dag = DAG('env_test', schedule_interval=None, start_date=start_date)
operator = KubernetesPodOperator(
name='k8s-env-var-test',
task_id='k8s-env-var-test',
dag=dag,
image='ubuntu:16.04',
cmds=['bash', '-cx'],
arguments=['env'],
config_file=os.environ['KUBECONFIG'],
namespace='eric-dev',
secrets=[env_var_secret],
)
But when I run this DAG, the VERSION_NUMBER env var isn't printed out. It doesn't look like it's being properly linked to the pod either (apologies for imprecise language, I am new to both Kubernetes and Airflow). This is from the Airflow task log of the pod creation response (also formatted for brevity/readability):
'env': [
{
'name': 'VERSION_NUMBER',
'value': None,
'value_from': {
'config_map_key_ref': None,
'field_ref': None,
'resource_field_ref': None,
'secret_key_ref': {
'key': 'VERSION_NUMBER',
'name': 'eric-env-vars',
'optional': None}
}
}
]
I'm assuming that we're somehow calling the constructor for the Secret wrong, but I am not entirely sure. Guidance appreciated!
Turns out this was a misunderstanding of the logs!
When providing an environment variable to a Kubernetes pod via a Secret, that value key in the API response is None because the value comes from the secret_key_ref.
Related
I got a list in yml - credentials. And supposedly each bank has to have a different password that needs to be encrypted. What would be the right way to specify that? As of now I got it configured like this, but that doesn't work.
This is the config.yml
infopoint:
endpoint: https://test.test.com/ws/SSS/Somthing.pl
system: TEST
mock: false
credentials:
- bank: 1111
user: LSSER
existingSecret:
name: infopoint-creds-s1-hb
- bank: 2222
user: TESSER
existingSecret:
name: infopoint-creds-s1
envFrom:
- secretRef:
name: infopoint-creds-s1-hb
- secretRef:
name: infopoint-creds-s1
This is how I created both secret keys on the server.
C:\Users\mks\IdeaProjects>kubectl.exe create secret generic infopoint-creds-s1-hb --from-literal=INFOPOINT_CREDENTIALS_PASSWORD=SOMEPASS -o yaml -n test-env --dry-run=client | kubeseal -o yaml --scope namespace-wide > infopoint-creds-s1-hb.yaml
C:\Users\mks\IdeaProjects>kubectl.exe create secret generic infopoint-creds-s1 --from-literal=INFOPOINT_CREDENTIALS_PASSWORD=SOMEPASS -o yaml -n test-env --dry-run=client | kubeseal -o yaml --scope namespace-wide > infopoint-creds-s1.yaml
This is my Spring configuration.
#Configuration
#ConfigurationProperties(prefix = "infopoint")
class InfopointAPIConfiguration {
lateinit var endpoint: String
var proxyServerName: String? = null
var proxyPortNumber: String? = null
lateinit var system: String
lateinit var mock: String
lateinit var credentials: List<Credentials>
data class Credentials(
var bank: String? = null,
var user: String? = null,
var password: String? = null
)
fun credentialsByBank(bank: Int): Credentials {
return credentials.firstOrNull { it.bank == bank.toString() }
?: error("Could not load credential for bank $bank")
}
}
Kubernetes secrets can be used or configured in applications in multiple ways for example configmaps, sealed secrets and environment variables. Since you got struck with the sealed secrets part I am providing an answer related to the same.
First we need to create a sealed secret in the same namespace with the same name for preventing other users on the same cluster from using your sealed secret. For more information related to sealed secrets please go through this document.
Now we have our secret created, all we need to do is to use it in our application. The secret which we created needs to be referenced in the yaml file. There is a detailed description on how to configure secrets in a spring boot application along with a sample project available here.
I'm installing the External Secrets Operator alongside Apache Pinot into an EKS cluster using CDK. I'm running into an issue that I think is being caused by CDK attempting to create a resource defined by the ESO before the ESO has actually gotten up and running. Here's the relevant code:
// install Pinot
const pinot = cluster.addHelmChart('Pinot', {
chartAsset: new Asset(this, 'PinotChartAsset', { path: path.join(__dirname, '../pinot') }),
release: 'pinot',
namespace: 'pinot',
createNamespace: true
});
// install the External Secrets Operator
const externalSecretsOperator = cluster.addHelmChart('ExternalSecretsOperator', {
chart: 'external-secrets',
release: 'external-secrets',
repository: 'https://charts.external-secrets.io',
namespace: 'external-secrets',
createNamespace: true,
values: {
installCRDs: true,
webhook: {
port: 9443
}
}
});
// create a Fargate Profile
const fargateProfile = cluster.addFargateProfile('FargateProfile', {
fargateProfileName: 'externalsecrets',
selectors: [{ 'namespace': 'external-secrets' }]
});
// create the Service Account used by the Secret Store
const serviceAccount = cluster.addServiceAccount('ServiceAccount', {
name: 'eso-service-account',
namespace: 'external-secrets'
});
serviceAccount.addToPrincipalPolicy(new iam.PolicyStatement({
effect: iam.Effect.ALLOW,
actions: [
'secretsmanager:GetSecretValue',
'secretsmanager:DescribeSecret'
],
resources: [
'arn:aws:secretsmanager:us-east-1:<MY-ACCOUNT-ID>:secret:*'
]
}))
serviceAccount.node.addDependency(externalSecretsOperator);
// create the Secret Store, an ESO Resource
const secretStoreManifest = getSecretStoreManifest(serviceAccount);
const secretStore = cluster.addManifest('SecretStore', secretStoreManifest);
secretStore.node.addDependency(serviceAccount);
secretStore.node.addDependency(fargateProfile);
// create the External Secret, another ESO resource
const externalSecretManifest = getExternalSecretManifest(secretStoreManifest)
const externalSecret = cluster.addManifest('ExternalSecret', externalSecretManifest)
externalSecret.node.addDependency(secretStore);
externalSecret.node.addDependency(pinot);
Even though I've set the ESO as a dependency to the Secret Store, when I try to deploy this I get the following error:
Received response status [FAILED] from custom resource. Message returned: Error: b'Error from server (InternalError): error when creating "/tmp/manifest.yaml": Internal error occurred:
failed calling webhook "validate.clustersecretstore.external-secrets.io": Post "https://external-secrets-webhook.external-secrets.svc:443/validate-external-secrets-io-v1beta1-clusterse
cretstore?timeout=5s": no endpoints available for service "external-secrets-webhook"\n'
If I understand correctly, this is the error you'd get if you try to add a Secret Store before the ESO is fully installed. I'm guessing that CDK does not wait until the ESO's pods are running before attempting to apply the manifest. Furthermore, if I comment out the lines the create the Secret Store and External Secret, do a cdk deploy, uncomment those lines and then deploy again, everything works fine.
Is there any way around this? Some way I can retry applying the manifest, or to wait a period of time before attempting the apply?
The addHelmChart method has a property wait that is set to false by default - setting it to true lets CDK know to not mark the installation as complete until of its its K8s resources are in a ready state.
I am currently provision my EKS cluster/s using EKSCTL and I want to use Terraform to provision the cluster/s. I am using Terraform EKS module to create cluster. I have use EKSCTL to create identity mapping with following command
eksctl create iamidentitymapping -- region us-east-1 --cluster stage-cluster --arn arn:aws:iam::111222333444:role/developer --username dev-service
I want to convert this command to Terraform with following, but it is not the best way
resource "null_resource" "eks-identity-mapping" {
depends_on = [
module.eks,
aws_iam_policy_attachment.eks-policy-attachment
]
provisioner "local-exec" {
command = <<EOF
eksctl create iamidentitymapping \
--cluster ${var.eks_cluster_name} \
--arn ${data.aws_iam_role.mwaa_role.arn} \
--username ${var.mwaa_username} \
--profile ${var.aws_profile} \
--region ${var.mwaa_aws_region}
EOF
}
}
How can I use Kubernetes provider to achieve this
I haven't found a clear matching for this particular command, but you can achieve something similar by setting the aws-auth config map in kubernetes, adding all of the users/roles and their access rights in one go.
For example we use something like the following below to supply the list of admins to our cluster:
resource "kubernetes_config_map" "aws_auth" {
metadata {
name = "aws-auth"
namespace = "kube-system"
}
data = {
mapRoles = <<CONFIGMAPAWSAUTH
- rolearn: ${var.k8s-node-iam-arn}
username: system:node:{{EC2PrivateDNSName}}
groups:
- system:bootstrappers
- system:nodes
- rolearn: arn:aws:iam::111222333444:role/developer
username: dev-service
groups:
- system:masters
CONFIGMAPAWSAUTH
}
}
Note that this file contains all of the role mappings, so you should make sure var.k8s-node-iam-arn is set to the superuser of the cluster otherwise you can get locked out. Also you have to set what access these roles will get.
You can also add specific IAM users instead of roles as well:
- userarn: arn:aws:iam::1234:user/user.first
username: user.first
groups:
- system:masters
You can do this directly in the eks module.
You create the list of roles you want to add, e.g.:
locals {
aws_auth_roles = [
{
rolearn = ${data.aws_iam_role.mwaa_role.arn}
username = ${var.mwaa_username}
groups = [
"system:masters"
]
},
]
}
and then in the module, you add:
module "eks" {
source = "terraform-aws-modules/eks/aws"
version = "19.5.1"
[...]
# aws-auth configmap
manage_aws_auth_configmap = true
aws_auth_roles = local.aws_auth_roles
[...]
}
Note: In older versions of "terraform-aws-modules/eks/aws" (14, 17) it was map_users and map_roles, in 19 it is manage_aws_auth_configmap and aws_auth_users, aws_auth_roles.
See the documentation here: https://registry.terraform.io/modules/terraform-aws-modules/eks/aws/19.7.0#input_manage_aws_auth_configmap
UPDATE: for this to work and not give an error like Error: The configmap "aws-auth" does not exist, you need to also add this authentication part:
data "aws_eks_cluster_auth" "default" {
name = local.cluster_name
}
provider "kubernetes" {
host = module.eks.cluster_endpoint
cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
token = data.aws_eks_cluster_auth.default.token
}
I've declared a kubernetes secret in pulumi like:
const tlsSecret = new k8s.core.v1.Secret('tlsSecret', {
metadata: {
name: 'star.builds.qwil.co'
},
data: {
'tls.crt': tlsCert,
'tls.key': tlsKey
}
});
However, I'm finding that when the secret is created only tls.key is present in the secret. When I look at the Diff View from the pulumi deployment on app.pulumi.com I see the following:
tlsSecret (kubernetes:core:Secret)
+ kubernetes:core/v1:Secret (create)
[urn=urn:pulumi:local::ledger::kubernetes:core/v1:Secret::tlsSecret]
apiVersion: "v1"
data : {
tls.key: "[secret]"
}
kind : "Secret"
metadata : {
labels: {
app.kubernetes.io/managed-by: "pulumi"
}
name : "star.builds.qwil.co"
}
Why is only tls.key being picked up even though I've also specified a tls.crt?
Turns out the variable tlsCert was false-y (I was loading it from config with the wrong key for Config.get()). Pulumi was smart and didn't create a secret with an empty string.
We are running a self-managed Airflow 1.10.2 with KubernetesExecutor on a GKE cluster in GCP. All internal operators are working fine so far, except the KubernetesPodOperator, which we would like to use for running our custom docker images. It seems that the Airflow worker images don't have privileges to start other pods inside the Kubernetes cluster. DAG just does not seem to be doing anything after starting it. This is what we found in the logs initially:
FileNotFoundError: [Errno 2] No such file or directory: '/root/.kube/config'
Next try - in_cluster=True parameter in the KubernetesPodOperator section does not seem to help. After that, we tried to use this parameter in airflow.cfg, section [kubernetes]:
gcp_service_account_keys = kubernetes-executor-private-key:/var/tmp/private/kubernetes_executor_private_key.json
and the error message was now TypeError: a bytes-like object is required, not 'str'
This is the parameter definition from github:
# GCP Service Account Keys to be provided to tasks run on Kubernetes Executors
# Should be supplied in the format: key-name-1:key-path-1,key-name-2:key-path-2
gcp_service_account_keys =
Already tried using various kinds of parentheses and quotes here, no success.
DAG code:
from datetime import datetime, timedelta
from airflow.contrib.operators.kubernetes_pod_operator import KubernetesPodOperator
from airflow.operators.dummy_operator import DummyOperator
default_args = {
'owner': 'xxx',
'depends_on_past': False,
'start_date': datetime.utcnow(),
'email': ['airflow#example.com'],
'email_on_failure': False,
'email_on_retry': False,
'retries': 1,
'retry_delay': timedelta(minutes=5)
}
dag = DAG(
'kubernetes_sample', default_args=default_args, schedule_interval=timedelta(minutes=10))
start = DummyOperator(task_id='run_this_first', dag=dag)
passing = KubernetesPodOperator(namespace='default',
image="Python:3.6",
cmds=["Python","-c"],
arguments=["print('hello world')"],
labels={"foo": "bar"},
name="passing-test",
task_id="passing-task",
in_cluster=True,
get_logs=True,
dag=dag
)
failing = KubernetesPodOperator(namespace='default',
image="ubuntu:1604",
cmds=["Python","-c"],
arguments=["print('hello world')"],
labels={"foo": "bar"},
in_cluster=True,
name="fail",
task_id="failing-task",
get_logs=True,
dag=dag
)
passing.set_upstream(start)
failing.set_upstream(start)
Anyone facing the same problem? Am i missing something here?