I am trying to set up a job scheduler (airflow) on an EKS cluster to replace a scheduler (Jenkins) we're running directly on an ec2. This job scheduler should be able to deploy pods to the EKS cluster it's running on.
However, whenever I try to deploy the pod (with a pod manifest), I get the following error message:
Error from server (Forbidden): error when creating "deployment.yaml": pods "simple-pod" is forbidden: pod does not have "kubernetes.io/config.mirror" annotation, node "ip-xx.ec2.internal" can only create mirror pods
I believe the restriction has to do with the NodeRestriction plugin on the kube-apiserver running on the EKS Control Plane.
I have looked through documentation to see if I can turn this plugin off, however it does not appear to be possible through kubectl, and only possible by modifying the kube-apiserver configuration on control plane itself.
Is it possible to turn off this plugin? Or, is it possible to label a node or pod to mark that it is not subject to this plugin? More broadly, is running a job scheduler on EKS that assigns job on the same cluster a bad design choice?
If we wanted to containerize and deploy our job scheduler, do we need to instantiate a separate EKS cluster/other service to run it on?
Related
I would like to run an OpenDJ cluster in my kubernetes cluster. In order to do so I used this procedure
I've created a StetefulSet, service and storageclass.
In order to initialize the replication, I need to run several commands dsreplication enable and dsreplication initialize-all inside one of the StatefulSet replicas because when I try running those commands through a kubernetes job as a separated pod it sends the following error - The local instance is not configured or you do not have permissions to access it as the server has not initialized as one of the opendj cluster's replicas (Which has the initial command that launch the opendj server).
How would you run those commands from the statefulsets pods? (When I keep thinking about scaling - how will I join new pods to the cluster when they'll launch with the hpa?)
Or maybe the better question is how can I run commands on my cluster from a remote pod?
Thanks.
I have an issue that, at face value, appears to indicate that I have two deployments running in parallel within my kube cluster, but 'kubectl get pods' only shows one deployment.
My deployment is composed of a pod with two containers. One of the containers runs a golang application that creates an http API endpoint, and the other runs Telegraf to read metrics from the API endpoint and push them to InfluxDB. When writing the data to Influx I tag the data with the source host as the name of the pod. I use Grafana to plot the metrics and I can clearly see incoming streaming data coming from two hosts (e.g. I can set a "WHERE host=" query clause as either "application-pod-name-231620957-7n32f" and "application-pod-name-1931165991-x154c").
Based on the above, I'm fairly certain that two deployments of the pod are running, each with the two containers (one providing application metrics and the other with telegraf sending metrics to InfluxDB).
However, kube seems to think that one of the deployments doesn't exist. As mentioned, "kubectl get pods" doesn't display the 2nd pod name in any way shape or form. Only one of them.
Has anyone seen this? Any ideas on further troubleshooting? I've attempted to use the pod name (that I have within telegraf) to query more information using kubectl but always get the response that the pod doesn't exist... but it must exist! It's sending live data!
We had been experiencing issues with a node within the cluster. Specifically, the node was experiencing GC failures and communications into the cluster from that node was broken. Due to these failures, someone on our team performed a 'kubectl delete' on the node from within the cluster. By doing so the node continued running, but also the kubelet running on the node remained in a broken state, and so the node couldn't re-auto-register itself into the cluster. This node happened to be running the 2nd pod, and the pods running on the node continued running without issue. In our case, the node was running on AWS, in which case the way to avoid this situation is to reboot the node either from the AWS console or AWS API.
I have installed Kubernetes with minikube, which is a single node cluster.
There is a yaml file to deploy controller master but it showing
Back-off restarting failed container Error syncing pod
Can someone solve this issue?
link for the yaml file is here https://github.com/kubernetes/kubernetes.github.io/blob/master/docs/admin/high-availability/kube-controller-manager.yaml
The Kubernetes controller manager is a core component of Kubernetes and already running in every Kubernetes cluster, usually in form of a standalone pod managed by the Kubernetes addon manager. Minikube uses localkube which integrates the controller manager together with other Kubernetes core components in a single binary to simplify setup of single-node clusters for testing purposes. If you want to change options of the integrated controller manager or other components, use the --extra-config option of minikube start.
The example you linked is a custom deployment of the controller manager used for highly available multi-master clusters. If you want to test this you need to set up your cluster manually, minikube is not the right tool for this.
With an existing Kubernetes Cluster (e.g. v 1.2.2 on GCE) that has set ENABLE_NODE_LOGGING=true and LOGGING_DESTINATION=gcp, what is the recommended way to stop those pods from running on each node and deploy a replacement DaemonSet that uses a custom fluentd configuration and docker image?
This should take into consider future Kubernetes upgrades as well.
If you set those configuration parameters when starting your cluster, it will create a manifest file on each node that configures fluentd to send container logs to google cloud logging. You can remove those manifest files and the kubelet will stop the fluentd containers (and you should also modify your instance template to change the parameters; otherwise any new nodes created, replacing broken nodes or scaling up your number of nodes, will continue to create fluentd containers).
Alternatively, if you modify the configuration parameter and run upgrade.sh to upgrade your nodes to a newer version of Kubernetes then your nodes will not have the manifest file and you won't be running the fluentd container any longer.
I have a Google Container Engine cluster with 21 nodes, there is one pod in particular that I need to always be running on a node with a static IP address (for outbound purposes).
Kubernetes supports DaemonSets
This is a way to have a pod be deployed to a specific node (or in a set of nodes) by giving the node a label that matches the nodeSelector in the DaemonSet. You can then assign a static IP to the VM instance that the labeled node is on. However, GKE doesn't appear to support the DaemonSet kind.
$ kubectl create -f go-daemonset.json
error validating "go-daemonset.json": error validating data: the server could not find the requested resource; if you choose to ignore these errors, turn validation off with --validate=false
$ kubectl create -f go-daemonset.json --validate=false
unable to recognize "go-daemonset.json": no kind named "DaemonSet" is registered in versions ["" "v1"]
When will this functionality be supported and what are the workarounds?
If you only want to run the pod on a single node, you actually don't want to use a DaemonSet. DaemonSets are designed for running a pod on every node, not a single specific node.
To run a pod on a specific node, you can use a nodeSelector in the pod specification, as documented in the Node Selection example in the docs.
edit: But for anyone reading this that does want to run something on every node in GKE, there are two things I can say:
First, DaemonSet will be enabled in GKE in version 1.2, which is planned for March. It isn't enabled in GKE in version 1.1 because it wasn't considered stable enough at the time 1.1 was cut.
Second, if you want to run something on every node before 1.2 is out, we recommend creating a replication controller with a number of replicas greater than your number of nodes and asking for a hostPort in the container spec. The hostPort will ensure that no more than one pod from the RC will be run per node.
DaemonSets is still alpha feature and Google Container Engine supports only production Kubernetes features. Workaround: build your own Kubernetes cluster (GCE, AWS, bare metal, ...) and enable alpha/beta features.