Error: stat /mnt/jenkins-store: no such file or directory - kubernetes

Very confused, I used an ec2 instance to bootstrap an eks cluster and everything worked completely fine yesterday. I deleted that cluster last night, just spun a new one up and now I'm getting this error when trying to build my jenkins pod
Error: stat /mnt/jenkins-store: no such file or directory
I find it strange how this error didn't show up yesterday and I set everything up the exact same way today. That error is what I got when I described my jenkins pod.
Here's my jenkins.yaml file for reference
apiVersion: v1
kind: ServiceAccount
metadata:
name: jenkins
namespace: default
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: jenkins
namespace: default
rules:
- apiGroups: [""]
resources: ["pods","services"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: ["apps"]
resources: ["deployments"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get","list","watch"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["create","delete","get","list","patch","update","watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: jenkins
namespace: default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: jenkins
subjects:
- kind: ServiceAccount
name: jenkins
---
# Allows jenkins to create persistent volumes
# This cluster role binding allows anyone in the "manager" group to read secrets in any namespace.
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: jenkins-crb
subjects:
- kind: ServiceAccount
namespace: default
name: jenkins
roleRef:
kind: ClusterRole
name: jenkinsclusterrole
apiGroup: rbac.authorization.k8s.io
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
# "namespace" omitted since ClusterRoles are not namespaced
name: jenkinsclusterrole
rules:
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["create","delete","get","list","patch","update","watch"]
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: jenkins
namespace: default
spec:
selector:
matchLabels:
app: jenkins
replicas: 1
template:
metadata:
labels:
app: jenkins
spec:
containers:
- name: jenkins
image: jenkins/jenkins:lts
env:
- name: JAVA_OPTS
value: -Djenkins.install.runSetupWizard=false
ports:
- name: http-port
containerPort: 8080
- name: jnlp-port
containerPort: 50000
volumeMounts:
- name: jenkins-home
mountPath: /var
subPath: jenkins_home
- name: docker-sock-volume
mountPath: "/var/run/docker.sock"
imagePullPolicy: Always
volumes:
# This allows jenkins to use the docker daemon on the host, for running builds
# see https://stackoverflow.com/questions/27879713/is-it-ok-to-run-docker-from-inside-docker
- name: docker-sock-volume
hostPath:
path: /var/run/docker.sock
- name: jenkins-home
hostPath:
path: /mnt/jenkins-store
serviceAccountName: jenkins
---
apiVersion: v1
kind: Service
metadata:
name: jenkins
namespace: default
spec:
type: NodePort
ports:
- name: ui
port: 8080
targetPort: 8080
nodePort: 31000
- name: jnlp
port: 50000
targetPort: 50000
selector:
app: jenkins
---

Similarly to this answer, check if your ec2-run-instances launched as your new ec2 spawned instance should not include some volumes, which would be then used by the jenkins.yaml.
The lack of such a volume would explain the Error: stat /mnt/jenkins-store message.

Related

Pod assigned node role instead of service account role on AWS EKS

Before I get started I have seen questions this and this, and they did not help.
I have a k8s cluster on AWS EKS on which I am deploying a custom k8s controller for my application. Using instructions from eksworkshop.com, I created my service account with the appropriate IAM role using eksctl. I assign the role in my deployment.yaml as seen below. I also set the securityContext as that seemed to solve problem in another case as described here.
apiVersion: apps/v1
kind: Deployment
metadata:
name: tel-controller
namespace: tel
spec:
replicas: 2
selector:
matchLabels:
app: tel-controller
strategy:
rollingUpdate:
maxSurge: 50%
maxUnavailable: 50%
type: RollingUpdate
template:
metadata:
labels:
app: tel-controller
spec:
serviceAccountName: tel-controller-serviceaccount
securityContext:
fsGroup: 65534
containers:
- image: <image name>
imagePullPolicy: Always
name: tel-controller
args:
- --metrics-bind-address=:8080
- --health-probe-bind-address=:8081
- --leader-elect=true
ports:
- name: webhook-server
containerPort: 9443
protocol: TCP
- name: metrics-port
containerPort: 8080
protocol: TCP
- name: health-port
containerPort: 8081
protocol: TCP
securityContext:
readOnlyRootFilesystem: true
runAsNonRoot: true
allowPrivilegeEscalation: false
But this does not seem to be working. If I describe the pod, I see the correct role.
AWS_DEFAULT_REGION: us-east-1
AWS_REGION: us-east-1
AWS_ROLE_ARN: arn:aws:iam::xxxxxxxxx:role/eksctl-eks-tel-addon-iamserviceaccount-tel-t-Role1-3APV5KCV33U8
AWS_WEB_IDENTITY_TOKEN_FILE: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
Mounts:
/var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6ngsr (ro)
But if I do a sts.GetCallerIdentityInput() from inside the controller application, I see the node role. And obviously i get an access denied error.
caller identity: (go string) {
Account: "xxxxxxxxxxxx",
Arn: "arn:aws:sts::xxxxxxxxxxx:assumed-role/eksctl-eks-tel-nodegroup-voice-NodeInstanceRole-BJNYF5YC2CE3/i-0694a2766c5d70901",
UserId: "AROAZUYK7F2GRLKRGGNXZ:i-0694a2766c5d70901"
}
This is how I created by service account
eksctl create iamserviceaccount --cluster ${EKS_CLUSTER_NAME} \
--namespace tel \
--name tel-controller-serviceaccount \
--attach-policy-arn arn:aws:iam::xxxxxxxxxx:policy/telcontrollerRoute53Policy \
--override-existing-serviceaccounts --approve
I have done this successfully in the past. The difference this time is that I also have role & role bindings attached to this service account. My rbac.yaml for this SA.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: tel-controller-role
labels:
app: tel-controller
rules:
- apiGroups: [""]
resources: [events]
verbs: [create, delete, get, list, update, watch]
- apiGroups: ["networking.k8s.io"]
resources: [ingressclasses]
verbs: [get, list]
- apiGroups: ["", "networking.k8s.io"]
resources: [services, ingresses]
verbs: [create, get, list, patch, update, delete, watch]
- apiGroups: [""]
resources: [configmaps]
verbs: [create, delete, get, update]
- apiGroups: ["coordination.k8s.io"]
resources: ["leases"]
verbs: [get, create, update]
- apiGroups: [""]
resources: [pods]
verbs: [get, list, watch, update]
- apiGroups: ["", "networking.k8s.io"]
resources: [services/status, ingresses/status]
verbs: [update, patch]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: tel-controller-rolebinding
labels:
app: tel-controller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: tel-controller-role
subjects:
- kind: ServiceAccount
name: tel-controller-serviceaccount
namespace: tel
What am I doing wrong here? Thanks.
PS: I am deploying using kubectl
PPS: from go.mod I am using github.com/aws/aws-sdk-go v1.44.28
So the Go SDK has 2 methods to create a session. Apparently I was using the deprecated method to create my session (session.New). Using the recommended method to create the new session, session.NewSession solved my problem.

Is there a kubernetes role definition to allow the command `kubectl rollout restart deploy <deployment>`?

I want a deployment in kubernetes to have the permission to restart itself, from within the cluster.
I know I can create a serviceaccount and bind it to the pod, but I'm missing the name of the most specific permission (i.e. not just allowing '*') to allow for the command
kubectl rollout restart deploy <deployment>
here's what I have, and ??? is what I'm missing
apiVersion: v1
kind: ServiceAccount
metadata:
name: restart-sa
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: default
name: restarter
rules:
- apiGroups: ["apps"]
resources: ["deployments"]
verbs: ["list", "???"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: testrolebinding
namespace: default
subjects:
- kind: ServiceAccount
name: restart-sa
namespace: default
roleRef:
kind: Role
name: restarter
apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: Pod
metadata:
name: example
spec:
containers:
- image: nginx
name: nginx
serviceAccountName: restart-sa
I believe the following is the minimum permissions required to restart a deployment:
rules:
- apiGroups: ["apps", "extensions"]
resources: ["deployments"]
resourceNames: [$DEPLOYMENT]
verbs: ["get", "patch"]
If you want permission to restart kubernetes deployment itself from within the cluster you need to set permission on rbac authorisation.
In the yaml file you have missed some specific permissions under Role:rules you need to add in the below format
verbs: ["get", "watch", "list"]
Instead of “Pod” you need to add “deployment” in the yaml file.
Make sure that you add “serviceAccountName: restart-sa” in the deployment yaml file under “spec:containers.” As mentioned below:
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
serviceAccountName: restart-sa
Then you can restart the deployment using the below command:
$ kubectl rollout restart deployment [deployment_name]

Unable to find Prometheus custom app exporter as a target in Prometheus server deployed in Kubernetes

I created a custom exporter in Python using prometheus-client package. Then created necessary artifacts to find the metric as a target in Prometheus deployed on Kubernetes.
But I am unable to see the metric as a target despite following all available instructions.
Help in finding the problem is appreciated.
Here is the summary of what I did.
Installed Prometheus using Helm on the K8s cluster in a namespace prometheus
Created a python program with prometheus-client package to create a metric
Created and deployed an image of the exporter in dockerhub
Created a deployment against the metrics image, in a namespace prom-test
Created a Service, ServiceMonitor, and a ServiceMonitorSelector
Created a service account, role and binding to enable access to the end point
Following is the code.
Service & Deployment
apiVersion: v1
kind: Service
metadata:
name: test-app-exporter
namespace: prom-test
labels:
app: test-app-exporter
spec:
type: ClusterIP
selector:
app: test-app-exporter
ports:
- name: http
protocol: TCP
port: 6000
targetPort: 5000
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: test-app-exporter
namespace: prom-test
spec:
selector:
matchLabels:
app: test-app-exporter
template:
metadata:
labels:
app: test-app-exporter
spec:
#serviceAccount: test-app-exporter-sa
containers:
- name: test-app-exporter
image: index.docker.io/cbotlagu/test-app-exporter:2
imagePullPolicy: Always
resources:
limits:
memory: "128Mi"
cpu: "500m"
ports:
- name: http
containerPort: 5000
imagePullSecrets:
- name: myregistrykey
Service account and role binding
apiVersion: v1
kind: ServiceAccount
metadata:
name: test-app-exporter-sa
namespace: prom-test
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: test-app-exporter-binding
subjects:
- kind: ServiceAccount
name: test-app-exporter-sa
namespace: prom-test
roleRef:
kind: ClusterRole
name: test-app-exporter-role
apiGroup: rbac.authorization.k8s.io
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: test-app-exporter-role
rules:
- apiGroups: [""]
resources:
- nodes
- nodes/metrics
- services
- endpoints
- pods
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources:
- configmaps
verbs: ["get"]
- apiGroups:
- networking.k8s.io
resources:
- ingresses
verbs: ["get", "list", "watch"]
Service Monitor & Selector
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: test-app-exporter-sm
namespace: prometheus
labels:
app: test-app-exporter
release: prometheus
spec:
selector:
matchLabels:
# Target app service
app: test-app-exporter
endpoints:
- port: http
interval: 15s
path: /metrics
namespaceSelector:
matchNames:
- default
- prom-test
#any: true
---
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
name: service-monitor-selector
namespace: prometheus
spec:
serviceAccountName: test-app-exporter-sa
serviceMonitorSelector:
matchLabels:
app: test-app-exporter-sm
release: prometheus
resources:
requests:
memory: 400Mi
I am able to get the target identified by Prometheus.
But though the end point can be reached within the cluster as well as from the node IP. Prometheus says the target is down.
In addition to that I am unable to see any other target.
Prom-UI
Any help is greatly appreciated
Following is my changed code
Deployment & Service
apiVersion: v1
kind: Namespace
metadata:
name: prom-test
---
apiVersion: v1
kind: Service
metadata:
name: test-app-exporter
namespace: prom-test
labels:
app: test-app-exporter
spec:
type: NodePort
selector:
app: test-app-exporter
ports:
- name: http
protocol: TCP
port: 5000
targetPort: 5000
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: test-app-exporter
namespace: prom-test
spec:
replicas: 1
selector:
matchLabels:
app: test-app-exporter
template:
metadata:
labels:
app: test-app-exporter
spec:
serviceAccountName: rel-1-kube-prometheus-stac-operator
containers:
- name: test-app-exporter
image: index.docker.io/cbotlagu/test-app-exporter:2
imagePullPolicy: Always
resources:
limits:
memory: "128Mi"
cpu: "500m"
ports:
- name: http
containerPort: 5000
imagePullSecrets:
- name: myregistrykey
Cluster Roles
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: pod-reader
namespace: prom-test
rules:
- apiGroups:
- ""
resources:
- nodes/metrics
- endpoints
- pods
- services
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: pod-reader-from-prom-test
namespace: prom-test
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: pod-reader
subjects:
- kind: ServiceAccount
name: rel-1-kube-prometheus-stac-operator
namespace: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: monitoring-role
namespace: monitoring
rules:
- apiGroups:
- ""
resources:
- nodes/metrics
- endpoints
- pods
- services
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: pod-reader-from-prom-test
namespace: monitoring
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: monitoring-role
subjects:
- kind: ServiceAccount
name: rel-1-kube-prometheus-stac-operator
namespace: monitoring
Service Monitor
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: test-app-exporter-sm
namespace: monitoring
labels:
app: test-app-exporter
release: prometheus
spec:
selector:
matchLabels:
# Target app service
app: test-app-exporter
endpoints:
- port: http
interval: 15s
path: /metrics
namespaceSelector:
matchNames:
- prom-test
- monitoring
---
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
name: service-monitor-selector
namespace: monitoring
spec:
serviceAccountName: rel-1-kube-prometheus-stac-operator
serviceMonitorSelector:
matchLabels:
app: test-app-exporter
release: prometheus
resources:
requests:
memory: 400Mi

Kubernetes Dashboard not connecting in the network

I've tried many options to access kubernetes dashboard from a different machine, which is within the network of my master server.
From https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/
I installed k8s dashboard using:
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta8/aio/deploy/recommended.yaml
And opened dashboard by running kube proxy then changed the service yaml file from ClusterIP to NodePort and added a port: 32414. But, it is not working both inside and outside master.
So, deleted
kubectl delete -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta8/aio/deploy/recommended.yaml
Then, created dashboard using below yaml file.
# Copyright 2017 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ------------------- Dashboard Secret ------------------- #
apiVersion: v1
kind: Secret
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard-certs
namespace: kube-system
type: Opaque
---
# ------------------- Dashboard Service Account ------------------- #
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kube-system
---
# ------------------- Dashboard Role & Role Binding ------------------- #
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: kubernetes-dashboard-minimal
namespace: kube-system
rules:
# Allow Dashboard to create 'kubernetes-dashboard-key-holder' secret.
- apiGroups: [""]
resources: ["secrets"]
verbs: ["create"]
# Allow Dashboard to create 'kubernetes-dashboard-settings' config map.
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["create"]
# Allow Dashboard to get, update and delete Dashboard exclusive secrets.
- apiGroups: [""]
resources: ["secrets"]
resourceNames: ["kubernetes-dashboard-key-holder", "kubernetes-dashboard-certs"]
verbs: ["get", "update", "delete"]
# Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map.
- apiGroups: [""]
resources: ["configmaps"]
resourceNames: ["kubernetes-dashboard-settings"]
verbs: ["get", "update"]
# Allow Dashboard to get metrics from heapster.
- apiGroups: [""]
resources: ["services"]
resourceNames: ["heapster"]
verbs: ["proxy"]
- apiGroups: [""]
resources: ["services/proxy"]
resourceNames: ["heapster", "http:heapster:", "https:heapster:"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: kubernetes-dashboard-minimal
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: kubernetes-dashboard-minimal
subjects:
- kind: ServiceAccount
name: kubernetes-dashboard
namespace: kube-system
---
# ------------------- Dashboard Deployment ------------------- #
kind: Deployment
apiVersion: apps/v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kube-system
spec:
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
k8s-app: kubernetes-dashboard
template:
metadata:
labels:
k8s-app: kubernetes-dashboard
spec:
containers:
- name: kubernetes-dashboard
image: k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.0
ports:
- containerPort: 8443
protocol: TCP
args:
- --auto-generate-certificates
# Uncomment the following line to manually specify Kubernetes API server Host
# If not specified, Dashboard will attempt to auto discover the API server and connect
# to it. Uncomment only if the default does not work.
# - --apiserver-host=http://my-address:port
volumeMounts:
- name: kubernetes-dashboard-certs
mountPath: /certs
# Create on-disk volume to store exec logs
- mountPath: /tmp
name: tmp-volume
livenessProbe:
httpGet:
scheme: HTTPS
path: /
port: 8443
initialDelaySeconds: 30
timeoutSeconds: 30
volumes:
- name: kubernetes-dashboard-certs
secret:
secretName: kubernetes-dashboard-certs
- name: tmp-volume
emptyDir: {}
serviceAccountName: kubernetes-dashboard
# Comment the following tolerations if Dashboard must not be deployed on master
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
---
# ------------------- Dashboard Service ------------------- #
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kube-system
spec:
type: NodePort
ports:
- port: 443
targetPort: 8443
nodePort: 32323
selector:
k8s-app: kubernetes-dashboard
Then , added sa_cluster_admin.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: dashboard-admin
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: cluster-admin-rolebinding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: dashboard-admin
namespace: kube-system
Applied both using kubectl apply -f <yaml_files>
Then, I can access the dashboard with the IP address and without running kubectl proxy.
But it is working only on the master.
When I tried to connect from another machine, it is not connecting.
https://masternode_ip:NodePort_added

argo workflow-controller can't connect to Kubernetes APIServer

I have installed argo in my own namespace in a central kubernetes cluster in my organization.
After installation when argo "workflow-controller" tries to fetch the configmaps using the API server, I get timeout error.
time="2018-08-15T01:24:40Z" level=fatal msg="Get https://192.168.0.1:443/api/v1/namespaces/2304613691/configmaps/workflow-controller-configmap: dial tcp 192.168.0.1:443: i/o timeout\ngithub.com/argoproj/argo/errors.Wrap\n\t/root/go/src/github.com/argoproj/argo/errors/errors.go:87\ngithub.com/argoproj/argo/errors.InternalWrapError\n\t/root/go/src/github.com/argoproj/argo/errors/errors.go:70\ngithub.com/argoproj/argo/workflow/controller.(*WorkflowController).ResyncConfig\n\t/root/go/src/github.com/argoproj/argo/workflow/controller/controller.go:295\nmain.Run\n\t/root/go/src/github.com/argoproj/argo/cmd/workflow-controller/main.go:96\ngithub.com/argoproj/argo/vendor/github.com/spf13/cobra.(*Command).execute\n\t/root/go/src/github.com/argoproj/argo/vendor/github.com/spf13/cobra/command.go:750\ngithub.com/argoproj/argo/vendor/github.com/spf13/cobra.(*Command).ExecuteC\n\t/root/go/src/github.com/argoproj/argo/vendor/github.com/spf13/cobra/command.go:831\ngithub.com/argoproj/argo/vendor/github.com/spf13/cobra.(*Command).Execute\n\t/root/go/src/github.com/argoproj/argo/vendor/github.com/spf13/cobra/command.go:784\nmain.main\n\t/root/go/src/github.com/argoproj/argo/cmd/workflow-controller/main.go:68\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:195\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:2337"
It is trying to access the follwing url: https://192.168.0.1:443/api/v1/namespaces/2304613691/configmaps/workflow-controller-configmap from within the pod container.
I have also modified the kubernetes host config to reflect kubernetes.default and added an open all ingress and egress network policy.
But still the exception is there.
time="2018-08-16T18:23:55Z" level=fatal msg="Get https://kubernetes.default:443/api/v1/namespaces/2304613691/configmaps/workflow-controller-configmap: dial tcp: i/o timeout\ngithub.com/argoproj/argo/errors.Wrap\n\t/root/go/src/github.com/argoproj/argo/errors/errors.go:87\ngithub.com/argoproj/argo/errors.InternalWrapError\n\t/root/go/src/github.com/argoproj/argo/errors/errors.go:70\ngithub.com/argoproj/argo/workflow/controller.(*WorkflowController).ResyncConfig\n\t/root/go/src/github.com/argoproj/argo/workflow/controller/controller.go:295\nmain.Run\n\t/root/go/src/github.com/argoproj/argo/cmd/workflow-controller/main.go:96\ngithub.com/argoproj/argo/vendor/github.com/spf13/cobra.(*Command).execute\n\t/root/go/src/github.com/argoproj/argo/vendor/github.com/spf13/cobra/command.go:750\ngithub.com/argoproj/argo/vendor/github.com/spf13/cobra.(*Command).ExecuteC\n\t/root/go/src/github.com/argoproj/argo/vendor/github.com/spf13/cobra/command.go:831\ngithub.com/argoproj/argo/vendor/github.com/spf13/cobra.(*Command).Execute\n\t/root/go/src/github.com/argoproj/argo/vendor/github.com/spf13/cobra/command.go:784\nmain.main\n\t/root/go/src/github.com/argoproj/argo/cmd/workflow-controller/main.go:68\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:195\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:2337"
apiVersion: v1
items:
- apiVersion: v1
kind: ServiceAccount
metadata:
name: argo
namespace: 2304613691
- apiVersion: v1
kind: ServiceAccount
metadata:
name: argo-ui
namespace: 2304613691
kind: List
---
apiVersion: v1
items:
- apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: argo-role
namespace: 2304613691
rules:
- apiGroups:
- ""
resources:
- pods
- pods/exec
verbs:
- create
- get
- list
- watch
- update
- patch
- apiGroups:
- ""
resources:
- configmaps
verbs:
- get
- watch
- list
- apiGroups:
- ""
resources:
- persistentvolumeclaims
verbs:
- create
- delete
- apiGroups:
- argoproj.io
resources:
- workflows
verbs:
- get
- list
- watch
- update
- patch
- apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: argo-ui-role
namespace: 2304613691
rules:
- apiGroups:
- ""
resources:
- pods
- pods/exec
- pods/log
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- secrets
verbs:
- get
- apiGroups:
- argoproj.io
resources:
- workflows
verbs:
- get
- list
- watch
kind: List
---
apiVersion: v1
items:
- apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: argo-binding
namespace: "2304613691"
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: argo-role
subjects:
- kind: ServiceAccount
name: argo
namespace: "2304613691"
- apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: argo-ui-binding
namespace: "2304613691"
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: argo-ui-role
subjects:
- kind: ServiceAccount
name: argo-ui
namespace: "2304613691"
kind: List
---
apiVersion: v1
items:
- apiVersion: extensions/v1beta1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "1"
generation: 1
name: workflow-controller
namespace: 2304613691
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: workflow-controller
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
labels:
app: workflow-controller
spec:
containers:
- args:
- --configmap
- workflow-controller-configmap
command:
- workflow-controller
env:
- name: ARGO_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
image: <our repo>/sample-agupta34/workflow-controller:v2.1.1
imagePullPolicy: IfNotPresent
name: workflow-controller
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: argo
serviceAccountName: argo
terminationGracePeriodSeconds: 30
- apiVersion: extensions/v1beta1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "1"
generation: 1
name: argo-ui
namespace: 2304613691
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: argo-ui
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
labels:
app: argo-ui
spec:
containers:
- env:
- name: ARGO_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: IN_CLUSTER
value: "true"
- name: ENABLE_WEB_CONSOLE
value: "false"
- name: BASE_HREF
value: /
image: <our repo>/sample-agupta34/argoui:v2.1.1
imagePullPolicy: IfNotPresent
name: argo-ui
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: argo-ui
serviceAccountName: argo-ui
terminationGracePeriodSeconds: 30
kind: List
---
apiVersion: v1
data:
config: |
artifactRepository: {}
executorImage: <our repo>/sample-agupta34/argoexec:v2.1.1
kind: ConfigMap
metadata:
name: workflow-controller-configmap
namespace: 2304613691
---
apiVersion: v1
kind: Service
metadata:
name: argo-ui
namespace: 2304613691
labels:
app: argo-ui
spec:
ports:
- port: 80
protocol: TCP
targetPort: 8001
selector:
app: argo-ui
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: argo-ui
namespace: 2304613691
annotations:
kubernetes.io/ingress.class: "netscaler.v2"
netscaler.applecloud.io/insecure-backend: "true"
spec:
backend:
serviceName: argo-ui
servicePort: 80
---
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: argo-and-argo-ui-netpol
spec:
podSelector:
matchLabels:
app: workflow-controller
app: argo-ui
ingress:
- {}
egress:
- {}