Is there some tool available that could tell me whether a K8s YAML configuration (to-be-supplied to kubectl apply) is valid for the target Kubernetes version without requiring a connection to a Kubernetes cluster?
One concrete use-case here would be to detect incompatibilities before actual deployment to a cluster, just because some already-deprecated label has been finally dropped in a newer Kubernetes version, e.g. as has happened for Helm and the switch to Kubernetes 1.16 (see Helm init fails on Kubernetes 1.16.0):
Dropped:
apiVersion: extensions/v1beta1
New:
apiVersion: apps/v1
I want to check these kind of incompatibilities within a CI system, so that I can reject it before even attempting to deploy it.
just run below command to validate the syntax
kubectl create -f <yaml-file> --dry-run
In fact the dry-run option is to validate the YAML syntax and the object schema. You can grab the output into a variable and if there is no error then rerun the command without dry-run
You could use kubeval
https://kubeval.instrumenta.dev/
I don't think kubectl support client-side only validation yet (02/2022)
Related
I use a kubernetes manifest file to deploy my code. My manifest typically has a number of things like Deployment, Service, Ingress, etc.. How can I perform a type of "rollout" or "restart" of everything that was applied with my manifest?
I know I can update my deployment say by running
kubectl rollout restart deployment <deployment name>
but what if I need to update all resources like ingress/service? Can it all be done together?
I would recommend you to store your manifests, e.g. Deployment, Service and Ingress in a directory, e.g. <your-directory>
Then use kubectl apply to "apply" those files to Kubernetes, e.g.:
kubectl apply -f <directory>/
See more on Declarative Management of Kubernetes Objects Using Configuration Files.
When your Deployment is updated this way, your pods will be replaced with the new version during a rolling deployment (you can configure to use another deployment strategy).
This is a Community Wiki answer so feel free to edit it and add any additional details you consider important.
As Burak Serdar has already suggested in his comment, you can simply use:
kubectl apply -f your-manifest.yaml
and it will apply all the changes you made in your manifest file to the resources, which are already deployed.
However note that running:
kubectl rollout restart -f your-manifest.yaml
makes not much sense as this file contains definitions of resources such as Services to which kubectl rollout restart cannot be applied. In consequence you'll see the following error:
$ kubectl rollout restart -f deployment-and-service.yaml
deployment.apps/my-nginx restarted
error: services "my-nginx" restarting is not supported
So as you can see it is perfectly possible to run kubectl rollout restart against a file that contains definitions of both resources that support this operation and those which do not support it.
Running kubectl apply instead will result in update of all the resources which definition has changed in your manifest:
$ kubectl apply -f deployment-and-service.yaml
deployment.apps/my-nginx configured
service/my-nginx configured
I'm trying to edit services created via helm chart and when changing from NodePort to ClusterIP I get this error
The Service "<name>" is invalid: spec.ports[0].nodePort: Fordbidden: may not be used when 'type' is 'ClusterIP'
I've seen solutions from other people where they just run kubectl apply -f service.yaml --force - but I'm not using kubectl but helm to do it - any thoughts ? If it was just one service I would just update/re-deploy manually but there are xx of them.
Found the answer to my exact question in here https://www.ibm.com/support/knowledgecenter/SSSHTQ/omnibus/helms/all_helms/wip/reference/hlm_upgrading_service_type_change.html
In short they suggest either to:
There are three methods you can use to avoid the service conversion issue above. You will only need to perform one of these methods:
Method 1: Installing the new version of the helm chart with a different release name and update all clients to point to the new probe service endpoint if required. Then delete the old release. This is the recommended method but requires a re-configuration on the client side.
Method 2: Manually changing the service type using kubectl edit svc. This method requires more manual steps but preserves the current service name and previous revisions of the helm chart. After performing this workaround, users should be able to perform a helm upgrade.
Method 3: Deleting and purging the existing helm release, and then install the new version of helm chart with the same release name.
I have a namespace namespace - which has ~10-15 deployments.
Creating a big yaml file, and apply it on a "deploy".
How do i validate, wait, watch, block, until all deployments have been rolledout ?
currently i am thinking of:
get list of deployments
foreach deployment - make api call to get status
once all deployments are "green" - end process, signaling deployment/ship is done.
what are the status'es of deployments, is there already a similar tool that can do it? https://github.com/Shopify/kubernetes-deploy is kind of what i am searching for, but it forces a yml structure and so on.
what would be the best approach?
Set a readiness probe and use kubectl rollout status deployment <deployment_name> to see the deployment rollout status
You'd better use Helm for managing deployments. Helm allows you to create reusable templates that can be applied to more than one environment. Read more here: https://helm.sh/docs/chart_template_guide/#getting-started-with-a-chart-template
You can create one big chart for all your services or you can create separate Helm charts for each your service.
Helm also allows you to run tests after deployment is done. Read more here: https://helm.sh/docs/developing_charts/#a-breakdown-of-the-helm-test-hooks
You probably want to use kubectl wait https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#wait
It lets you wait for a specific condition of a specific object
In your case:
kubectl -n namespace \
wait --for=condition=Available --timeout=32s \
deployment/name
use --dry-run option in the apply/create command to check the syntax.
I have a container cluster in Google Container Engine with Stackdriver logging agent enabled. It is correctly pulling stdout logs from my containers. Now I would like to change the fluentd config to specify a log parser so that the logs shown in the GCP Logging view will have the correct severity and component.
Following this Stackdriver logging guide from kubernetes.io, I have attempted to:
Get the fluentd ConfigMap as a yml file
Added a new <filter> according to my log4js log format
Created a new ConfigMap named fluentd-cm-2 in kube-system namespace
Edited the DaemonSet for fluentd and set its ConfigMap to fluentd-cm-2. I did this using kubectl edit ds instead of kubectl replace -f because the latter failed with an error message: "the object has been modified", even after getting a fresh copy of the DaemonSet yaml.
Unexpected result: The DaemonSet is restarted, but its configuration is reverted back to the original ConfigMap, so my changes did not take effect.
I have also tried editing the ConfigMap directly (kubectl edit cm fluentd-gcp-config-v1.1 --namespace kube-system) and saved it, but it was also reverted.
I noticed that the DaemonSet and ConfigMap for fluentd are tagged with addonmanager.kubernetes.io/mode: Reconcile. I would conclude that GKE has overwritten my settings because of this "reconcile" mode.
So, my question is: how can I change the fluentd configuration in a Google Container Engine cluster, when the logging agent was installed by GKE on cluster provisioning?
Please take a look at the Prerequisites section on the documentation page you mentioned. It's mentioned there, that on GKE you cannot change the default Stackdriver Logging integration. The reason is that GKE maintains this configuration: updates the agent, watches its health and so on. It's not possible to provide the same level of support for all possible configurations.
However, you can always disable the default integration and deploy your own, patched version of DaemonSet. You can find out how to disable the default integration in the GKE documentation:
gcloud beta container clusters update [CLUSTER-NAME] \
--logging-service=none
Note, that after you disabled the default integration, you have to maintain the new deployment yourself: update the agent, set the resources, watch its health.
Here is a solution for using your own fluentd daemonset that is very much like the one included with GKE.
https://cloud.google.com/solutions/customizing-stackdriver-logs-fluentd
I have a running pod and I want to change one of it's container's environment variable and made it work immediately. Can I achieve that? If I can, how to do that?
Simply put and in kube terms, you can not.
Environment for linux process is established on process startup, and there are certainly no kube tools that can achieve such goal.
For example, if you make a change to your Deployment (I assume you use it to create pods) it will roll the underlying pods.
Now, that said, there is a really hacky solution reported under Is there a way to change the environment variables of another process in Unix? that involves using GDB
Also, remember that even if you could do that, there is still application logic that would need to watch for such changes instead of, as it usually is now, just evaluate configuration from envs during startup.
This worked with me
kubectl set env RESOURCE/NAME KEY_1=VAL_1 ... KEY_N=VAL_N
check the official documentation here
Another approach for runtime pods you can get into the Pod command line and change the variables in the runtime
RUN kubectl exec -it <pod_name> -- /bin/bash
Then
Run export VAR1=VAL1 && export VAR2=VAL2 && your_cmd
I'm not aware of any way to do it and I can't think of real world scenario where this makes too much sense.
Usually you have to restart a process for it to notice the changed environment variables and the easiest way to do that is restart the pod.
The solution closest to what seem to want is to create a deployment and then use kubectl edit (kubectl edit deploy/name) to modify it's environment variables. A new pod is started and the old one is terminated after you save.
Kubernetes is designed in such a way that any changes to the pod should be redeployed through the config. If you go messing with pods that have already been deployed you can end up with weird clusters that are hard to debug.
If you really want to you can run additional commands in your running pod using kubectl exec, but this is only recommended for debug purposes.
kubectl exec -it <pod_name> export VARIABLENAME=<thing>
If you are using Helm 3> according to the documentation:
Automatically Roll Deployments
Often times ConfigMaps or Secrets are
injected as configuration files in containers or there are other
external dependencies changes that require rolling pods. Depending on
the application a restart may be required should those be updated with
a subsequent helm upgrade, but if the deployment spec itself didn't
change the application keeps running with the old configuration
resulting in an inconsistent deployment.
The sha256sum function can be used to ensure a deployment's annotation
section is updated if another file changes:
kind: Deployment
spec:
template:
metadata:
annotations:
checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
[...]
In the event you always want
to roll your deployment, you can use a similar annotation step as
above, instead replacing with a random string so it always changes and
causes the deployment to roll:
kind: Deployment
spec:
template:
metadata:
annotations:
rollme: {{ randAlphaNum 5 | quote }}
[...]
Both of these methods allow your Deployment to leverage the built in update strategy
logic to avoid taking downtime.
NOTE: In the past we recommended using the --recreate-pods flag as
another option. This flag has been marked as deprecated in Helm 3 in
favor of the more declarative method above.
It is hard to change from outside. But it is easy to change from inside. Your App running in the pod can change it. Just oppose an Api to change environment variable.
You can use configmap with volumes to update environment variables on the go..
Refer: https://itnext.io/how-to-automatically-update-your-kubernetes-app-configuration-d750e0ca79ab