I am trying to delete (and recreate) the Argo namespace, but it won't fully delete because I tried launching an eventsource and eventbus there. Now these will not delete.
I have tried to delete them via yaml and individually - no success yet.
The frustrating result is that I cannot re-launch argo
customresourcedefinition.apiextensions.k8s.io/clusterworkflowtemplates.argoproj.io unchanged
customresourcedefinition.apiextensions.k8s.io/cronworkflows.argoproj.io unchanged
customresourcedefinition.apiextensions.k8s.io/workfloweventbindings.argoproj.io unchanged
customresourcedefinition.apiextensions.k8s.io/workflows.argoproj.io unchanged
customresourcedefinition.apiextensions.k8s.io/workflowtasksets.argoproj.io unchanged
customresourcedefinition.apiextensions.k8s.io/workflowtemplates.argoproj.io unchanged
clusterrole.rbac.authorization.k8s.io/argo-aggregate-to-admin unchanged
clusterrole.rbac.authorization.k8s.io/argo-aggregate-to-edit unchanged
clusterrole.rbac.authorization.k8s.io/argo-aggregate-to-view unchanged
clusterrole.rbac.authorization.k8s.io/argo-cluster-role unchanged
clusterrole.rbac.authorization.k8s.io/argo-server-cluster-role unchanged
clusterrolebinding.rbac.authorization.k8s.io/argo-binding unchanged
clusterrolebinding.rbac.authorization.k8s.io/argo-server-binding unchanged
Error from server (Forbidden): error when creating "https://github.com/argoproj/argo-workflows/releases/download/v3.2.0/install.yaml": serviceaccounts "argo" is forbidden: unable to create new content in namespace argo because it is being terminated
Error from server (Forbidden): error when creating "https://github.com/argoproj/argo-workflows/releases/download/v3.2.0/install.yaml": serviceaccounts "argo-server" is forbidden: unable to create new content in namespace argo because it is being terminated
Error from server (Forbidden): error when creating "https://github.com/argoproj/argo-workflows/releases/download/v3.2.0/install.yaml": roles.rbac.authorization.k8s.io "argo-role" is forbidden: unable to create new content in namespace argo because it is being terminated
Error from server (Forbidden): error when creating "https://github.com/argoproj/argo-workflows/releases/download/v3.2.0/install.yaml": rolebindings.rbac.authorization.k8s.io "argo-binding" is forbidden: unable to create new content in namespace argo because it is being terminated
Error from server (Forbidden): error when creating "https://github.com/argoproj/argo-workflows/releases/download/v3.2.0/install.yaml": configmaps "workflow-controller-configmap" is forbidden: unable to create new content in namespace argo because it is being terminated
Error from server (Forbidden): error when creating "https://github.com/argoproj/argo-workflows/releases/download/v3.2.0/install.yaml": services "argo-server" is forbidden: unable to create new content in namespace argo because it is being terminated
Error from server (Forbidden): error when creating "https://github.com/argoproj/argo-workflows/releases/download/v3.2.0/install.yaml": services "workflow-controller-metrics" is forbidden: unable to create new content in namespace argo because it is being terminated
Error from server (Forbidden): error when creating "https://github.com/argoproj/argo-workflows/releases/download/v3.2.0/install.yaml": deployments.apps "argo-server" is forbidden: unable to create new content in namespace argo because it is being terminated
Error from server (Forbidden): error when creating "https://github.com/argoproj/argo-workflows/releases/download/v3.2.0/install.yaml": deployments.apps "workflow-controller" is forbidden: unable to create new content in namespace argo because it is being terminated
Here is what is going on in the argo namespace itself
{
"apiVersion": "v1",
"kind": "Namespace",
"metadata": {
"creationTimestamp": "2021-10-21T14:52:51Z",
"deletionTimestamp": "2021-12-10T18:48:17Z",
"labels": {
"kubernetes.io/metadata.name": "argo"
},
"name": "argo",
"resourceVersion": "9222845",
"uid": "2ce61352-74a5-40d1-b35e-fe1efa39c3af"
},
"spec": {
"finalizers": [
"kubernetes"
]
},
"status": {
"conditions": [
{
"lastTransitionTime": "2021-12-10T18:48:23Z",
"message": "All resources successfully discovered",
"reason": "ResourcesDiscovered",
"status": "False",
"type": "NamespaceDeletionDiscoveryFailure"
},
{
"lastTransitionTime": "2021-12-10T18:48:23Z",
"message": "All legacy kube types successfully parsed",
"reason": "ParsedGroupVersions",
"status": "False",
"type": "NamespaceDeletionGroupVersionParsingFailure"
},
{
"lastTransitionTime": "2021-12-10T18:48:23Z",
"message": "All content successfully deleted, may be waiting on finalization",
"reason": "ContentDeleted",
"status": "False",
"type": "NamespaceDeletionContentFailure"
},
{
"lastTransitionTime": "2021-12-10T18:48:23Z",
"message": "Some resources are remaining: eventbus.argoproj.io has 1 resource instances, eventsources.argoproj.io has 1 resource instances",
"reason": "SomeResourcesRemain",
"status": "True",
"type": "NamespaceContentRemaining"
},
{
"lastTransitionTime": "2021-12-10T18:48:23Z",
"message": "Some content in the namespace has finalizers remaining: eventbus-controller in 1 resource instances, eventsource-controller in 1 resource instances",
"reason": "SomeFinalizersRemain",
"status": "True",
"type": "NamespaceFinalizersRemaining"
}
],
"phase": "Terminating"
}
}
UPDATE:
Even though one test of Kubernetes shows that the eventsource persists, another test shows that it does not
This is the response to
kubectl get eventsources argo
Error from server (NotFound): eventsources.argoproj.io "argo" not found
For anyone who stumbles onto this question, it is a permissions issue. Make certain your service account has permissions to work in both namespaces (argo and argo-events).
Related
I have deploy a k8s cluster with kubeadm, I want to get controller manager's metrics with following command:
curl -k https://localhost:10257/metrics
but got the following error:
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
"status": "Failure",
"message": "forbidden: User \"system:anonymous\" cannot get path \"/metrics\"",
"reason": "Forbidden",
"details": {
},
"code": 403
}
So my question is, how to get k8s controller manager's metrics?
This is a forbidden error due to permission issues which need to be authenticated with a valid user. For this,You need to create a service account, then give that service account access permissions to the metrics Path through RBAC, then this will make that service account to get the metrics.
As per this Role and Cluster Binding doc, you need to allow metrics path(replace with /healthz) as below and give a try.
Allow GET and POST requests to the non-resource endpoint /healthz and all subpaths (must be in a ClusterRole bound with a ClusterRoleBinding to be effective):
rules:
- nonResourceURLs: ["/healthz", "/healthz/*"] # '*' in a nonResourceURL is a suffix glob match
verbs: ["get", "post"]
i folowed guide on official Kubernetes Dashboard github (https://github.com/kubernetes/dashboard) and now I'm facing problem with accessing it. I used kubectl proxy to redirect internal port outside, but when I try to open address:
http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/
it just ends up with this error:
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
"status": "Failure",
"message": "error trying to reach service: dial tcp 192.168.23.7:8443: i/o timeout",
"reason": "ServiceUnavailable",
"code": 503
}
What am I supposed to do?
You get a timeout. Check if dashboard pods are working (kubectl get pods -n kubernetes-dashboard)
Check u have enough access control
Check here https://github.com/kubernetes/dashboard/blob/master/docs/user/access-control/README.md
I trying to create a service account with permissions to get information about endpoints pod IPs and getting back a permissions problem.
Creating the service account and give it the right permissions:
$ kubectl create role endpoints-reader --verb=get --verb=list --resource=endpoints
$ kubectl create serviceaccount endpoints-reader-sa
$ kubectl create rolebinding default-endpoints-reader --role=endpoints-reader --serviceaccount=endpoints-reader-sa:endpoints-reader-sa
Adding this sa to the deployment YAML file:
...
spec:
serviceAccountName: endpoints-reader-sa
containers:
- name: ...
I stated the pod and logged in into it (ssh). Now I want to run a REST call to pull the information:
$ TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token);
$ curl https://kubernetes.default.svc/api/v1/namespaces/XXX/endpoints --silent --header "Authorization: Bearer $TOKEN" --insecure
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
"status": "Failure",
"message": "endpoints is forbidden: User \"system:serviceaccount:XXX:endpoints-reader-sa\" cannot list resource \"endpoints\" in API group \"\" in the namespace \"XXX\"",
"reason": "Forbidden",
"details": {
"kind": "endpoints"
},
"code": 403
}
What I'm doing wrong?
OK... Found the issue.
So, this line:
kubectl create rolebinding default-endpoints-reader --role=endpoints-reader --serviceaccount=endpoints-reader-sa:endpoints-reader-sa
should be changed to this:
kubectl create rolebinding default-endpoints-reader --role=endpoints-reader --serviceaccount=XXX:endpoints-reader-sa
Where XXX is the namespace name.
Similar problem you can find here.
I'm trying to scale up/down some deployments over HTTP and also list the deployments on my cluster. I'm able to list pods, but can't figure out the deployments piece.
http://localhost:8080/api/v1/namespaces/default/deployments
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "the server could not find the requested resource",
"reason": "NotFound",
"details": {},
"code": 404
}
Deployments are in the apps/v1 namespace, and you need to include apps in the URL. The API documentation for the "list deployments" endpoint gives the URL as
GET /apis/apps/v1/namespaces/{namespace}/deployments
You can use the normal read-modify-write sequence to change the replicas: field in a deployment spec to scale it.
There is also a dedicated endpoint to scale deployments, though it's slightly underdocumented. Manage replicas count for deployment using Kubernetes API
suggests reading and patching the scale resource, or there is an example with a minimal JSON payload.
I created a cluster role "try-usr"
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: try-usr
rules:
- apiGroups:
- '*'
resources:
- '*'
verbs:
- get
- list
- watch
While accessing the Web UI(Dashboard), it's throwing an error as follows:
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
"status": "Failure",
"message": "services \"https:kubernetes-dashboard:\" is forbidden: User \"xyz\" cannot get services/proxy in the namespace \"kube-system\"",
"reason": "Forbidden",
"details": {
"name": "https:kubernetes-dashboard:",
"kind": "services"
},
"code": 403
}
Depending on the kubernetes version, the dashboard will require different permissions according to the docs
v1.7
create and watch permissions for secrets in kube-system namespace required to - create and watch for changes of kubernetes-dashboard-key-holder secret.
get, update and delete permissions for secrets named kubernetes-dashboard-key-holder and kubernetes-dashboard-certs in kube-system namespace.
proxy permission to heapster service in kube-system namespace required to allow getting metrics from heapster.
v1.8
create permission for secrets in kube-system namespace required to create kubernetes-dashboard-key-holder secret.
get, update and delete permissions for secrets named kubernetes-dashboard-key-holder and kubernetes-dashboard-certs in kube-system namespace.
get and update permissions for config map named kubernetes-dashboard-settings in kube-system namespace.
proxy permission to heapster service in kube-system namespace required to allow getting metrics from heapster.