I have a file for a Job resource, which looks something like below.I need to run multiple instances with this definition with separate arguments for each.
apiVersion: batch/v1
kind: Job
metadata:
generateName: abc-
spec:
template:
spec:
containers:
- name: abc
image: index.docker.io/some/image:latest
imagePullPolicy: Always
imagePullSecrets:
- name: some_secret
restartPolicy: Never
backoffLimit: 4
I can successfully run this job resource with
kubectl create -f my-job.yml
But, I'm not sure how I pass my arguments corresponding to
command:['arg1','arg2']
I think updating the file with my dynamic args for each request is just messy.
I tried kubectl patch -f my-job.yml --type='json' -p='[{"op": "add", "path": "/spec/template/spec/containers/0/", "value": {"command": ["arg1","arg2"] } }]', which works well for a Deployment kind but for Job it doesn't work
I tried
sudo kubectl run explicitly-provide-name-which-i-dont-want-to --image=index.docker.io/some/image:latest --restart=Never -- arg1 arg2, but for this I won't be able to pass the imagePullSecrets.
kind of a generic answer here, just trying to guide you. In general what you express is the need to 'parameterize' your kubernetes deployment descriptors. There are different ways some are simple, some others are a bit hacky and finally there is github.com/kubernetes/helm.
Personally I would strongly suggest you go through installing Helm on your cluster and then 'migrate' your job or any vanilla kubernetes deployment descriptor into a helm Chart. This will eventually give you the 'parameterization' power that you need to spin jobs in different ways and with different configs.
But, if this sounds like too much for you, I can recommend something that I was doing before I discover Helm. Using things like 'bash' / 'envsubst' I was eventually - templating manually the parts of the yaml file, with place holders (e.g env variables) and then I was feedind the yaml to tools like 'envsubst' where they were replacing the placeholders with the values from the environment. Ugly? Yes. Maintenable? maybe for a couple of simple examples. Example of envsubst here.
apiVersion: batch/v1
kind: Job
metadata:
spec:
template:
spec:
containers:
- name: abc
image: index.docker.io/some/image:latest
imagePullPolicy: Always
imagePullSecrets:
- name: $SOME_ENV_VALUE
restartPolicy: Never
backoffLimit: 4
Hope that helps..but seriously if you have time, consider checking 'Helm'.
I would also consider sourcing the command arguments from environment variables. These variables are then provided by helm as javapapo has mentioned.
I found this guide via Google:
Start Kubernetes job from command line with parameters
But the helm chart solution suggested by javapopo is the best way I guess.
Related
Is there a way to set the values of some keys in a Knative service.yaml file using environment variables?
More detail
I am trying to deploy a Knative service to a Kubernetes cluster using GitLab CI. Some of the variables in my service.yaml file depend on the project and environment of the GitLab CI pipeline. Is there a way I can seamlessly plug those values into my service.yaml file without resorting to hacks like sed -i ...?
For example, given the following script, I want the $(KUBE_NAMESPACE), $(CI_ENVIRONMENT_SLUG), and $(CI_PROJECT_PATH_SLUG) values to be replaced by accordingly-named environment variables.
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
name: design
namespace: "$(KUBE_NAMESPACE)"
spec:
template:
metadata:
name: design-v1
annotations:
app.gitlab.com/env: "$(CI_ENVIRONMENT_SLUG)"
app.gitlab.com/app: "$(CI_PROJECT_PATH_SLUG)"
spec:
containers:
- name: user-container
image: ...
timeoutSeconds: 600
containerConcurrency: 8
I don't think there is a great way to expand environment variables inside of an existing yaml, but if you don't want to use sed, you might be able to use envsubst:
envsubst < original.yaml > modified.yaml
You would just run this command before you use the yaml to expand the environment variables contained within it.
Also I think you'll need your variables to use curly braces, instead of parentheses, like this: ${KUBE_NAMESPACE}.
EDIT: You might also be able to use this inline like this: kubectl apply -f <(envsubst < service.yaml)
More than a Knative issue this is more of Kubernetes limitation. Kubernetes allows some expansion but not in annotations or namespace definitions. For example, you can do it in container env definitions:
containers:
- env:
- name: PODID
valueFrom: ...
- name: LOG_PATH
value: /var/log/$(PODID)
If this is a CI/CD system like Gitlab the environment variables should be in a shell environment, so a simple shell expansion will do. For example.
#!/bin/bash
echo -e "
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
name: design
namespace: "${KUBE_NAMESPACE}"
spec:
template:
metadata:
name: design-v1
annotations:
app.gitlab.com/env: "${CI_ENVIRONMENT_SLUG}"
app.gitlab.com/app: "${CI_PROJECT_PATH_SLUG}"
spec:
containers:
- name: user-container
image: ...
timeoutSeconds: 600
containerConcurrency: 8
" | kubectl apply -f -
You can also use envsubst as a helper like mentioned in the other answer.
i was looking for a strightforward way to create kubernetes resources templates (such as pod, deployment, service, etc.), though i couldn't find any good tool that does it. the tools i came across, are no longer maintained and some have stiff learning curve (such as kustomize).
some time ago, kubectl have introduced the generator and it can be used as follow to produce the resource configuration. for instance:
$ kubectl run helloworld --image=helloworld \
--dry-run --output=yaml --generator=run-pod/v1
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
run: helloworld
name: helloworld
spec:
containers:
- image: helloworld
name: helloworld
resources: {}
dnsPolicy: ClusterFirst
restartPolicy: Always
status: {}
but kubectl generator should not be used, since most if it was deprecated.
one might issue kubectl get on all the resources, to commit them in a source control and use it to restore the kubernetes cluster, though that implies that the kubernetes resources were created already, where my interest is generating these resource configuration in the first place.
please recommend about your favorite tool for generating\creating kubernetes resources or explain what is the best practice to handle my use case.
Create yaml files and use git to version them is the new best practice. However this approach can be automatized with GitOps approach and flagger. Another way is to create helm chart and customize with template feature.
You should definitely run away from generating from command line either way. You should have source of truth and cli generation will not give you that.
I have a file for a Job resource, which looks something like below.I need to run multiple instances with this definition with separate arguments for each.
apiVersion: batch/v1
kind: Job
metadata:
generateName: abc-
spec:
template:
spec:
containers:
- name: abc
image: index.docker.io/some/image:latest
imagePullPolicy: Always
imagePullSecrets:
- name: some_secret
restartPolicy: Never
backoffLimit: 4
I can successfully run this job resource with
kubectl create -f my-job.yml
But, I'm not sure how I pass my arguments corresponding to
command:['arg1','arg2']
I think updating the file with my dynamic args for each request is just messy.
I tried kubectl patch -f my-job.yml --type='json' -p='[{"op": "add", "path": "/spec/template/spec/containers/0/", "value": {"command": ["arg1","arg2"] } }]', which works well for a Deployment kind but for Job it doesn't work
I tried
sudo kubectl run explicitly-provide-name-which-i-dont-want-to --image=index.docker.io/some/image:latest --restart=Never -- arg1 arg2, but for this I won't be able to pass the imagePullSecrets.
kind of a generic answer here, just trying to guide you. In general what you express is the need to 'parameterize' your kubernetes deployment descriptors. There are different ways some are simple, some others are a bit hacky and finally there is github.com/kubernetes/helm.
Personally I would strongly suggest you go through installing Helm on your cluster and then 'migrate' your job or any vanilla kubernetes deployment descriptor into a helm Chart. This will eventually give you the 'parameterization' power that you need to spin jobs in different ways and with different configs.
But, if this sounds like too much for you, I can recommend something that I was doing before I discover Helm. Using things like 'bash' / 'envsubst' I was eventually - templating manually the parts of the yaml file, with place holders (e.g env variables) and then I was feedind the yaml to tools like 'envsubst' where they were replacing the placeholders with the values from the environment. Ugly? Yes. Maintenable? maybe for a couple of simple examples. Example of envsubst here.
apiVersion: batch/v1
kind: Job
metadata:
spec:
template:
spec:
containers:
- name: abc
image: index.docker.io/some/image:latest
imagePullPolicy: Always
imagePullSecrets:
- name: $SOME_ENV_VALUE
restartPolicy: Never
backoffLimit: 4
Hope that helps..but seriously if you have time, consider checking 'Helm'.
I would also consider sourcing the command arguments from environment variables. These variables are then provided by helm as javapapo has mentioned.
I found this guide via Google:
Start Kubernetes job from command line with parameters
But the helm chart solution suggested by javapopo is the best way I guess.
I'm using a CI to update my kubernetes cluster whenever there's an update to an image. Whenever the image is pushed and has the latest tag it kubectl apply's the existing deployment but nothing gets updated.
this is what runs
$ kubectl apply --record --filename /tmp/deployment.yaml
My goal is when the apply is ran that a rolling deployment gets executed.
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: api
spec:
replicas: 1
template:
metadata:
labels:
app: api
spec:
containers:
- name: api
image: us.gcr.io/joule-eed41/api:latest
imagePullPolicy: Always
ports:
- containerPort: 1337
args:
- /bin/sh
- -c
- echo running api;npm start
env:
- name: NAMESPACE
valueFrom:
configMapKeyRef:
name: config
key: NAMESPACE
As others suggested, have a specific tag.
Set new image using following command
kubectl set image deployment/deployment_name deployment_name=image_name:image_tag
In your case it would be
kubectl set image deployment/api api=us.gcr.io/joule-eed41/api:0.1
As #ksholla20 mentionedm using kubectl set image is a good option for many (most?) cases.
But if you can't change the image tag consider using:
1 ) kubectl rollout restart deployment/<name>
(reference).
2 ) kubectl patch deployment <name> -p "{\"spec\":{\"template\":{\"metadata\":{\"labels\":{\"version\":\"$CURRENT_BUILD_HASH_OR_DATE\"}}}}}}" (reference)
(*) Notice that the patch command allow you to change specific properties in the deployment (or any other object chosen) like the label selector and the pod label or other properties like the value of the NAMESPACE environment variable in your example.
I've run into the same problem and none of the solutions posted so far will help. The solution is easy, but not easy to see or predict. The applied yaml will generate both a deployment and a replicaset the first time it's run. Unfortunately, applying changes to the manifest likely only replaces the replicaset, while the deployment will remain unchanged. This is a problem because some changes need to happen at the deployment level, but the old deployment hangs around. To have best results, delete the deployment and ensure all previous deployments and replicasets are deleted. Then apply the updated manifest.
Is there any way to pass image version from a varibale/config when passing a manifest .yaml to kubectl command
Example :
apiVersion: v1
kind: ReplicationController
metadata:
name: nginx
spec:
replicas: 1
selector:
app: nginx
template:
metadata:
name: nginx
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:${IMAGE_VERSION}
imagePullPolicy: Always
resources:
limits:
cpu: "1.2"
memory: 100Mi
ports:
- containerPort: 80
Use case is to launch specific image version which is set at kubernetes level, and that the variable is resolved by kubernetes itself at server side.
Thanks and Regards,
Ravi
k8s manifest files are static yaml/json.
If you would like to template the manifests (and manage multiple resources in a bundle-like fashion), I strongly recommend you to have a look at Helm
I've recently created a Workshop which focuses precisely on the "Templating" features of Helm.
Helm does a lot more than just templating, it is built as a full fledged package manager for Kubernetes applications (think Apt/Yum/Homebrew).
If you want to handle everything client side, have a look at https://github.com/errordeveloper/kubegen
Although, at some point, you will need the other features of Helm and a migration will be needed when that time comes - I recommend biting the bullet and going for Helm straight up.
After looking into this recently we decided to just go with sed. Wrap kubectl apply into a small bash script and replace the placeholders before running apply.
We did look into more sophisticated tooling but we only found Helm. However Helm is a complex piece of technology that does way more than just templating. It changes your workflow a lot as you no longer deploy using kubectl and have to have a Helm package repo around to push your packages to. Our assessment was that Helm is not useful for deploying our application and using it for just the templating is overkill.
Here is an example how to do this with sed (it is a excerpt from my typical circleci config):
replaces="s/{.Namespace}/$CIRCLE_BRANCH/;";
replaces="$replaces s/{.CiBuild}/$CIRCLE_BUILD_NUM/; ";
replaces="$replaces s/{.CiCommit}/$CIRCLE_SHA1/; ";
replaces="$replaces s/{.CiUser}/$CIRCLE_USERNAME/; ";
cat ./k8s/app.yaml | sed -e "$replaces" | ./kubectl --kubeconfig=`pwd`/.kube/config apply --namespace=$NAMESPACE -f -