I am currently implementing CI/CD pipeline using docker , Kubernetes and Jenkins for my micro services deployment. And I am testing the pipeline using the public repository that I created in Dockerhub.com. When I tried the deployment using Kubernetes Helm chart , I were able to add my all credentials in Value.yaml file -the default file getting for adding the all configuration when we creating a helm chart.
Confusion
Now I removed my helm chart , and I am only using deployment and service n plane YAML files. SO How I can add my Dockerhub credentials here ?
Do I need to use environment variable ? Or Do I need to create any separate YAML file for credentials and need to give reference in Deployment.yaml file ?
If I am using imagePullSecrets way How I can create separate YAML file for credentials ?
From Kubernetes point of view: Pull an Image from a Private Registry you can create secrets and add necessary information into your yaml (Pod/Deployment)
Steps:
1. Create a Secret by providing credentials on the command line:
kubectl create secret docker-registry regcred --docker-server=<your-registry-server> --docker-username=<your-name> --docker-password=<your-pword> --docker-email=<your-email>
2. Create a Pod that uses your Secret (example pod):
apiVersion: v1
kind: Pod
metadata:
name: private-reg
spec:
containers:
- name: private-reg-container
image: <your-private-image>
imagePullSecrets:
- name: regcred
You can pass the dockerhub creds as environment variables at jenkins only and Imagepullsecrets are to be made as per kubernetes doc, as they are one time things, you can directly add them to the required clusters
Related
I'm trying to deploy a Flink stream processor to a Kubernetes cluster with the help of the official Flink kubernetes operator.
The Flink app also uses Minio as its state backend. Everything worked fine until I tried to provide the credentials from Hashicorp Vault in the following way:
apiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment
metadata:
name: flink-app
namespace: default
spec:
serviceAccount: sa-example
podTemplate:
apiVersion: v1
kind: Pod
metadata:
name: pod-template
spec:
serviceAccountName: default:sa-example
containers:
- name: flink-main-container
# ....
flinkVersion: v1_14
flinkConfiguration:
presto.s3.endpoint: https://s3-example-api.dev.net
high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
high-availability.storageDir: s3p://example-flink/example-1/high-availability/
high-availability.cluster-id: example-1
high-availability.namespace: example
high-availability.service-account: default:sa-example
# presto.s3.access-key: *
# presto.s3.secret-key: *
presto.s3.path-style-access: "true"
web.upload.dir: /opt/flink
jobManager:
podTemplate:
apiVersion: v1
kind: Pod
metadata:
name: job-manager-pod-template
annotations:
vault.hashicorp.com/namespace: "/example/dev"
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/agent-init-first: "true"
vault.hashicorp.com/agent-inject-secret-appsecrets.yaml: "example/Minio"
vault.hashicorp.com/role: "example-serviceaccount"
vault.hashicorp.com/auth-path: auth/example
vault.hashicorp.com/agent-inject-template-appsecrets.yaml: |
{{- with secret "example/Minio" -}}
presto.s3.access-key: {{.Data.data.accessKey}}
presto.s3.secret-key: {{.Data.data.secretKey}}
{{- end }}
When I comment the presto.s3.access-key and presto.s3.secret-key config values in the flinkConfiguration, replace them with the above listed Hashicorp Vault annotations and try to provide them programmatically during runtime:
val configuration: Configuration = getSecretsFromFile("/vault/secrets/appsecrets.yaml")
val env = org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.getExecutionEnvironment(configuration)
I receive the following error message:
java.io.IOException: com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), WebIdentityTokenCredentialsProvider: You must specify a value for roleArn and roleSessionName, com.amazonaws.auth.profile.ProfileCredentialsProvider#5331f738: profile file cannot be null, com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper#bc0353f: Failed to connect to service endpoint: ]
at com.facebook.presto.hive.s3.PrestoS3FileSystem$PrestoS3OutputStream.uploadObject(PrestoS3FileSystem.java:1278) ~[flink-s3-fs-presto-1.14.2.jar:1.14.2]
at com.facebook.presto.hive.s3.PrestoS3FileSystem$PrestoS3OutputStream.close(PrestoS3FileSystem.java:1226) ~[flink-s3-fs-presto-1.14.2.jar:1.14.2]
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72) ~[flink-s3-fs-presto-1.14.2.jar:1.14.2]
at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101) ~[flink-s3-fs-presto-1.14.2.jar:1.14.2]
at org.apache.flink.fs.s3presto.common.HadoopDataOutputStream.close(HadoopDataOutputStream.java:52) ~[flink-s3-fs-presto-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.blob.FileSystemBlobStore.put(FileSystemBlobStore.java:80) ~[flink-dist_2.12-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.blob.FileSystemBlobStore.put(FileSystemBlobStore.java:72) ~[flink-dist_2.12-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.blob.BlobUtils.moveTempFileToStore(BlobUtils.java:385) ~[flink-dist_2.12-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.blob.BlobServer.moveTempFileToStore(BlobServer.java:680) ~[flink-dist_2.12-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.blob.BlobServerConnection.put(BlobServerConnection.java:350) [flink-dist_2.12-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:110) [flink-dist_2.12-1.14.2.jar:1.14.2]
I initially also tried to append the secrets to flink-config.yaml in the docker-entrypoint.sh based on this documentation - Configure Access Credentials:
if [ -f '/vault/secrets/appsecrets.yaml' ]; then
(echo && cat '/vault/secrets/appsecrets.yaml') >> $FLINK_HOME/conf/flink-conf.yaml
fi
The question is how to provide the S3 credentials during the runtime since the Flink operator mounts the flink-config.yaml from a config map and it is a flink-conf.yaml: Read-only file system.
Thank you
There is no support for this from the Kubernetes operator. In fact, this is not a limitation of the Flink Kubernetes operator, it is due to the fact of lack in support in Kubernetes native integration. There is a separate story for this in the Kubernetes operator side - FLINK-27491.
As a workaround, what you can do is, set up an init container and update the config map from the init container using kubernetes API after reading it from the vault. So the updated config map should have the secrets replaced by the init container and those will be visible to the job manager and all of its task managers. The whole Flink cluster journey starts only after updating the config map from the init container so it should be visible to the Flink cluster.
A simple example to update the config map from the init container can be found here. In this example, the config map is updated with a simple CURL command. In theory, you can use any lightweight client to update the config map like this.
A side note: If possible I would suggest to use AWS IAM role rather than IAM plain secrets as IAM role is more secure compared to IAM static credentials.
I would like to avoid keeping secret in the Git as a best practise, and store it in AWS SSM.
Is there any way to get the value from AWS System Manager and use to create Kubernetes Secret?
I manage to create secret by fetching value from AWS Parameter store using the following script.
cat <<EOF | ./kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
name: kiali
namespace: istio-system
type: Opaque
data:
passphrase: $(echo -n "`aws ssm get-parameter --name /dev/${env_name}/kubernetes/kiali_password --with-decrypt --region=eu-west-2 --output text --query Parameter.Value`" | base64 -w0)
username: $(echo -n "admin" | base64 -w0)
EOF
For sure, 12factors requires to externalize configuration outside Codebase.
For your question, there is an attempt to integrate AWS SSM (AWS Secret Manager) to be used as the single source of truth for Secrets.
You just need to deploy the controller :
helm repo add secret-inject https://aws-samples.github.io/aws-secret-sidecar-injector/
helm repo update
helm install secret-inject secret-inject/secret-inject
Then annotate your deployment template with 2 annotations:
template:
metadata:
annotations:
secrets.k8s.aws/sidecarInjectorWebhook: enabled
secrets.k8s.aws/secret-arn: arn:aws:secretsmanager:us-east-1:123456789012:secret:database-password-hlRvvF
Other steps are explained here.
But I think that I highlighted the most important steps which clarifies the approach.
You can use GoDaddy external secrets. Installing it, creates a controller, and the controller will sync the AWS secrets within specific intervals. After creating the secrets in AWS SSM and installing GoDaddy external secrets, you have to create an ExternalSecret type as follows:
apiVersion: 'kubernetes-client.io/v1'
kind: ExtrenalSecret
metadata:
name: cats-and-dogs
secretDescriptor:
backendType: secretsManager
data:
- key: cats-and-dogs/mysql-password
name: password`
This will create a Kubernetes secrets for you. That secret can be exposed to your service as an environment variable or through volume mount.
Use Kubernetes External Secret. This below solution uses Secret Manager (not SSM) but servers the purpose.
Deploy using Helm
$ `helm repo add external-secrets https://external-secrets.github.io/kubernetes-external-secrets/`
$ `helm install kubernetes-external-secrets external-secrets/kubernetes-external-secrets`
Create new secret with required parameter in AWS Secret Manager:
For example - create a secret with secret name as "dev/db-cred" with below values.
{"username":"user01","password":"pwd#123"}
Secret.YAML:
apiVersion: kubernetes-client.io/v1
kind: ExternalSecret
metadata:
name: my-kube-secret
namespace: my-namespace
spec:
backendType: secretsManager
region: us-east-1
dataFrom:
- dev/db-cred
Refer it in helm values file as below
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: my-kube-secret
key: password
I'm new to openshift and Kubernetes too (coming from Docker swarm world). and I'm looking to create secrets in openshift using definition file. these secrets are generated from a file. to give an example of what I'm trying to do let's say I have a file "apache.conf" and I want to add that file to containers as a secret mounted as a volume. In swarm I can just write the following in the stack file:
my-service:
secrets:
- source: my-secret
target: /home/myuser/
mode: 0700
secrets:
my-secret:
file: /from/host/apache.conf
In openshift I'm looking to have something similar like:
apiVersion: v1
kind: Secret
metadata:
- name: my-secret
files:
- "/from/host/apache.conf"
type: Opaque
The only way I've found that I can do something similar is by using kustomize and according to this post using Kustomize with openshift is cumbersome. is there a better way for creating secrets from a file?
No you can't
The reason is that the object Secret is stored in the etcd database and is not bound to any host. Therefore, the object doesn't understand the path.
You can create the secret from a file using the cli, and then the content will be saved in the Secret object.
oc create secret generic my-secret --from-file=fil1=pullsecret_private.json
I have Spinnaker 1.10.5 deployed to Azure Kubernetes Service using Halyard.
I am trying to get Azure Container Registry webhooks to trigger a pipeline. I found that you can set up echo to allow artifact webhooks using an echo-local.yml like this:
webhooks:
artifacts:
enabled: true
sources:
- source: azurecr
templatePath: /path/to/azurecr.jinja
However, I'm stuck on the templatePath value. Since I'm deploying with Halyard into Kubernetes, all the configuration files get mounted as volumes from Kubernetes secrets.
How do I get my Jinja template into my Halyard-deployed echo so it can be used in a custom webhook?
As of Halyard 1.13 there will be the ability to custom mount secrets in Kubernetes
Create a Kubernetes secret with your Jinja template.
apiVersion: v1
kind: Secret
metadata:
name: echo-webhook-templates
namespace: spinnaker
type: Opaque
data:
mytemplate: [base64-encoded-contents-of-template]
Set the templatePath in the ~/.hal/default/profiles/echo-local.yml to the place you're mounting the secret.
webhooks:
artifacts:
enabled: true
sources:
- source: mysource
templatePath: /mnt/webhook-templates/mytemplate
Add the mount to ~/.hal/default/service-settings/echo.yml
kubernetes:
volumes:
- id: echo-webhook-templates
type: secret
mountPath: /mnt/webhook-templates
Since Halyard 1.13 hasn't actually been released yet, I obviously haven't tried this, but it's how it should work. Also... I guess I may be stuck until then.
I'd like to deploy pods into my GKE Kubernetes cluster that use images from a private, third-party Docker registry (not GCP's private Docker registry).
How do I provide my GKE Kubernetes cluster with credentials to that private repository so that the images can be pulled when required?
You need to create a secret that holds the credentials needed to download images from the private registry. This process is explained on Kubernetes documentation, but it looks like
kubectl create secret docker-registry regsecret --docker-server=<your-registry-server> --docker-username=<your-name> --docker-password=<your-pword> --docker-email=<your-email>
Then, once your secret has been created, you need to specify that you want to use this secret to pull images from the registry when creating the pod's containers with the imagePullSecrets key containing the name of the secret created above, like
apiVersion: v1
kind: Pod
metadata:
name: private-reg
spec:
containers:
- name: private-reg-container
image: <your-private-image>
imagePullSecrets:
- name: regsecret