I have an application that I can deploy to kubernetes (Google Kubernetes Engine) to which I'm trying to add Google's CDN. For this I'm adding a BackendConfig. But when my gitlab pipeline tries to apply it it returns the following error.
$ kubectl apply -f backend-config.yaml
Error from server (Forbidden): error when retrieving current configuration of:
Resource: "cloud.google.com/v1beta1, Resource=backendconfigs", GroupVersionKind: "cloud.google.com/v1beta1, Kind=BackendConfig"
I have a strongly suspect that the account the pipeline is running under does not have enough privileges to access backend configs. Being new to k8s and gke I'm not sure how to fix this. Especially as I cannot find what permission is needed for this.
Edit
I added a kubectl get backendconfigs to my pipeline and that fails with the same error. Running it from my gcloud sdk environment the same command works.
Note the cluster is managed by Gitlab and using RBAC. My understanding is that gitlab creates service accounts per namespace in k8s with the edit role.
Edit 2
Added ClusterRole and ClusterRoleBinding based on Arghya's answer.
Output of $ kubectl get crd
NAME CREATED AT
backendconfigs.cloud.google.com 2020-01-09T15:37:27Z
capacityrequests.internal.autoscaling.k8s.io 2020-04-28T11:15:26Z
certificaterequests.cert-manager.io 2020-01-15T06:53:47Z
certificates.cert-manager.io 2020-01-15T06:53:48Z
challenges.acme.cert-manager.io 2020-01-15T06:53:48Z
challenges.certmanager.k8s.io 2020-01-09T15:47:01Z
clusterissuers.cert-manager.io 2020-01-15T06:53:48Z
clusterissuers.certmanager.k8s.io 2020-01-09T15:47:01Z
issuers.cert-manager.io 2020-01-15T06:53:48Z
issuers.certmanager.k8s.io 2020-01-09T15:47:01Z
managedcertificates.networking.gke.io 2020-01-09T15:37:53Z
orders.acme.cert-manager.io 2020-01-15T06:53:48Z
orders.certmanager.k8s.io 2020-01-09T15:47:01Z
scalingpolicies.scalingpolicy.kope.io 2020-01-09T15:37:53Z
updateinfos.nodemanagement.gke.io 2020-01-09T15:37:53Z
Output of kubectl describe crd backendconfigs.cloud.google.com
Name: backendconfigs.cloud.google.com
Namespace:
Labels: <none>
Annotations: <none>
API Version: apiextensions.k8s.io/v1beta1
Kind: CustomResourceDefinition
Metadata:
Creation Timestamp: 2020-01-09T15:37:27Z
Generation: 1
Resource Version: 198
Self Link: /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/backendconfigs.cloud.google.com
UID: f0bc780a-32f5-11ea-b7bd-42010aa40111
Spec:
Conversion:
Strategy: None
Group: cloud.google.com
Names:
Kind: BackendConfig
List Kind: BackendConfigList
Plural: backendconfigs
Singular: backendconfig
Scope: Namespaced
Validation:
Open APIV 3 Schema:
Properties:
API Version:
Description: APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources
Type: string
Kind:
Description: Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds
Type: string
Metadata:
Type: object
Spec:
Description: BackendConfigSpec is the spec for a BackendConfig resource
Properties:
Cdn:
Description: CDNConfig contains configuration for CDN-enabled backends.
Properties:
Cache Policy:
Description: CacheKeyPolicy contains configuration for how requests to a CDN-enabled backend are cached.
Properties:
Include Host:
Description: If true, requests to different hosts will be cached separately.
Type: boolean
Include Protocol:
Description: If true, http and https requests will be cached separately.
Type: boolean
Include Query String:
Description: If true, query string parameters are included in the cache key according to QueryStringBlacklist and QueryStringWhitelist. If neither is set, the entire query string is included and if false the entire query string is excluded.
Type: boolean
Query String Blacklist:
Description: Names of query strint parameters to exclude from cache keys. All other parameters are included. Either specify QueryStringBlacklist or QueryStringWhitelist, but not both.
Items:
Type: string
Type: array
Query String Whitelist:
Description: Names of query string parameters to include in cache keys. All other parameters are excluded. Either specify QueryStringBlacklist or QueryStringWhitelist, but not both.
Items:
Type: string
Type: array
Type: object
Enabled:
Type: boolean
Required:
enabled
Type: object
Connection Draining:
Description: ConnectionDrainingConfig contains configuration for connection draining. For now the draining timeout. May manage more settings in the future.
Properties:
Draining Timeout Sec:
Description: Draining timeout in seconds.
Format: int64
Type: integer
Type: object
Iap:
Description: IAPConfig contains configuration for IAP-enabled backends.
Properties:
Enabled:
Type: boolean
Oauthclient Credentials:
Description: OAuthClientCredentials contains credentials for a single IAP-enabled backend.
Properties:
Client ID:
Description: Direct reference to OAuth client id.
Type: string
Client Secret:
Description: Direct reference to OAuth client secret.
Type: string
Secret Name:
Description: The name of a k8s secret which stores the OAuth client id & secret.
Type: string
Required:
secretName
Type: object
Required:
enabled
oauthclientCredentials
Type: object
Security Policy:
Type: object
Session Affinity:
Description: SessionAffinityConfig contains configuration for stickyness parameters.
Properties:
Affinity Cookie Ttl Sec:
Format: int64
Type: integer
Affinity Type:
Type: string
Type: object
Timeout Sec:
Format: int64
Type: integer
Type: object
Status:
Type: object
Version: v1beta1
Versions:
Name: v1beta1
Served: true
Storage: true
Status:
Accepted Names:
Kind: BackendConfig
List Kind: BackendConfigList
Plural: backendconfigs
Singular: backendconfig
Conditions:
Last Transition Time: 2020-01-09T15:37:27Z
Message: no conflicts found
Reason: NoConflicts
Status: True
Type: NamesAccepted
Last Transition Time: <nil>
Message: the initial names have been accepted
Reason: InitialNamesAccepted
Status: True
Type: Established
Stored Versions:
v1beta1
Events: <none>
Create a ClusterRole and ClusterRoleBinding for service account example-sa in namespace example-namespace
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: backendconfig-role
rules:
- apiGroups: ["cloud.google.com"]
resources: ["backendconfigs"]
verbs: ["get", "watch", "list", "create", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: backendconfig-rolebinding
subjects:
- kind: ServiceAccount
name: example-sa
namespace: example-namespace
roleRef:
kind: ClusterRole
name: backendconfig-role
apiGroup: rbac.authorization.k8s.io
To check the permission is applied
kubectl auth can-i get backendconfigs --as=system:serviceaccount:example-namespace:example-sa -n example-namespace
Related
I used the following guide to set up my chaostoolkit cluster: https://chaostoolkit.org/deployment/k8s/operator/
I am attempting to kill a pod using kubernetes, however the following error:
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:serviceaccount:chaostoolkit-run:chaostoolkit-b3af262edb\" cannot list resource \"pods\" in API group \"\" in the namespace \"task-dispatcher\"","reason":"Forbidden","details":{"kind":"pods"},"code":403}
I set my serviceAccountName to an RBAC that I created but for some reason my kubernetes defaults to "system:serviceaccount:chaostoolkit-run:chaostoolkit-b3af262edb".
apiVersion: v1
kind: ConfigMap
metadata:
name: my-chaos-exp
namespace: chaostoolkit-run
data:
experiment.yaml: |
---
version: 1.0.0
title: Terminate Pod Experiment
description: If a pod gets terminated, a new one should be created in its place in a reasonable amount of time.
tags: ["kubernetes"]
secrets:
k8s:
KUBERNETES_CONTEXT: "docker-desktop"
method:
- type: action
name: terminate-k8s-pod
provider:
type: python
module: chaosk8s.pod.actions
func: terminate_pods
arguments:
label_selector: ''
name_pattern: my-release-rabbitmq-[0-9]$
rand: true
ns: default
---
apiVersion: chaostoolkit.org/v1
kind: ChaosToolkitExperiment
metadata:
name: my-chaos-exp
namespace: chaostoolkit-crd
spec:
serviceAccountName: test-user
automountServiceAccountToken: false
pod:
image: chaostoolkit/chaostoolkit:full
imagePullPolicy: IfNotPresent
experiment:
configMapName: my-chaos-exp
configMapExperimentFileName: experiment.yaml
restartPolicy: Never
Error which is shared is using default service account "choastoolkit". Look like the role associated might not proper permissions.
The service account "test-user" which is been used in ChaosToolkitExperiment defintion should have proper role access to delete pod.
Please specify proper service account having proper role access.
I have configured Crossplane on my machine, created a cluster with a bunch of other resources, and now I am trying to cleanup everything.
For instance, I am trying to delete a Cluster by using kubectl delete clusters <cluster-name>, but it simply does not get removed. In minikube dashboard I see the following condition on that cluster:
Type: Terminating
Reason: InstanceDeletionCheck
Message: could not confirm zero CustomResources remaining: timed out waiting for the condition
My goal is to cleanup the managed resources created by playing with this github repo: https://github.com/upbound/platform-ref-multi-k8s, and I would really appreciate any help
This is the output of the kubectl describe cluster multik8s-cluster-aws-wd95c-qlbrt command:
Name: multik8s-cluster-aws-wd95c-qlbrt
Namespace:
Labels: crossplane.io/claim-name=multik8s-cluster-aws
crossplane.io/claim-namespace=default
crossplane.io/composite=multik8s-cluster-aws-wd95c
Annotations: crossplane.io/external-create-pending: 2022-05-03T08:36:12Z
crossplane.io/external-create-succeeded: 2022-05-03T08:36:14Z
crossplane.io/external-name: multik8s-cluster-aws
API Version: eks.aws.crossplane.io/v1beta1
Kind: Cluster
Metadata:
Creation Timestamp: 2022-05-03T08:24:13Z
Deletion Grace Period Seconds: 0
Deletion Timestamp: 2022-05-03T10:15:06Z
Finalizers:
finalizer.managedresource.crossplane.io
Generate Name: multik8s-cluster-aws-wd95c-
Generation: 6
Managed Fields:
...
Owner References:
API Version: multik8s.platformref.crossplane.io/v1alpha1
Controller: true
Kind: EKS
Name: multik8s-cluster-aws-wd95c-h2nbj
UID: 76852ac3-58a9-42ec-8307-c02e490e8f32
Resource Version: 507248
UID: f02fa30d-9878-4be9-bebc-838d7e58d565
Spec:
Deletion Policy: Delete
For Provider:
...
Status:
At Provider:
Arn: arn:aws:eks:us-west-2:305615705119:cluster/multik8s-cluster-aws
Created At: 2022-05-03T08:36:14Z
Endpoint: https://519EADEC62BE27B27903C30E01A8E22D.gr7.us-west-2.eks.amazonaws.com
Identity:
Oidc:
Issuer: https://oidc.eks.us-west-2.amazonaws.com/id/519EADEC62BE27B27903C30E01A8E22D
Platform Version: eks.6
Resources Vpc Config:
Cluster Security Group Id: sg-0b9baf2fff4385125
Vpc Id: vpc-0fca5959a43bbdf71
Status: ACTIVE
Conditions:
Last Transition Time: 2022-05-03T08:48:26Z
Message: update failed: cannot update EKS cluster version: InvalidParameterException: Unsupported Kubernetes minor version update from 1.21 to 1.16
Reason: ReconcileError
Status: False
Type: Synced
Last Transition Time: 2022-05-03T08:48:25Z
Reason: Available
Status: True
Type: Ready
Events: <none>
How to remove version from 'status.storedVersions' while Manually upgrading the existing objects to a new stored version in kubernetes custom resources
apiVersion for CRD: apiextensions.k8s.io/v1beta1
Using aws EKS
CRD YAML
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
# name must match the spec fields below, and be in the form: <plural>.<group>
name: crontabs.stable.example.com
spec:
# group name to use for REST API: /apis/<group>/<version>
group: stable.example.com
# list of versions supported by this CustomResourceDefinition
versions:
- name: v1
# Each version can be enabled/disabled by Served flag.
served: true
# One and only one version must be marked as the storage version.
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
image:
type: string
replicas:
type: integer
# either Namespaced or Cluster
scope: Namespaced
names:
# plural name to be used in the URL: /apis/<group>/<version>/<plural>
plural: crontabs
# singular name to be used as an alias on the CLI and for display
singular: crontab
# kind is normally the CamelCased singular type. Your resource manifests use this.
kind: CronTab
# shortNames allow shorter string to match your resource on the CLI
shortNames:
- ct
And upgrading the CRD to v2
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
# name must match the spec fields below, and be in the form: <plural>.<group>
name: crontabs.stable.example.com
spec:
# group name to use for REST API: /apis/<group>/<version>
group: stable.example.com
# list of versions supported by this CustomResourceDefinition
versions:
- name: v2
# Each version can be enabled/disabled by Served flag.
served: true
# One and only one version must be marked as the storage version.
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
image:
type: string
replicas:
type: integer
- name: v1
# Each version can be enabled/disabled by Served flag.
served: false
# One and only one version must be marked as the storage version.
storage: false
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
image:
type: string
replicas:
type: integer
# either Namespaced or Cluster
scope: Namespaced
names:
# plural name to be used in the URL: /apis/<group>/<version>/<plural>
plural: crontabs
# singular name to be used as an alias on the CLI and for display
singular: crontab
# kind is normally the CamelCased singular type. Your resource manifests use this.
kind: CronTab
# shortNames allow shorter string to match your resource on the CLI
shortNames:
- ct
After disabling version v1 I want remove it from the versions list to do that I need to remove it from status.storedVersions how can i do that the kubernetes document does not give details about that
There is also another another way of removing status.specVersion using cluster API.
Fields under status are modified via the status sub resource:
curl -d '[{ "op": "replace", "path":"/status/storedVersions", "value": ["v2"] }]' \
-H "Content-Type: application/json-patch+json" \
-X PATCH \
http://localhost:8080/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/crontabs.stable.example.com/status
To access API you need to first connected to cluster doc
Ive tried many various options on Minikube, Kubeadm, GKE but everywhere output was the same.
Unfortunately it is impossible using kubectl and docs Remove the old version from the CustomResourceDefinition spec.versions list is bit missleading.
Once a definition falls into specVersion it's impossible to remove it. storage: true puts it int the storedVersions list.
It was well described in this Github thread, especially comment from #sebgl.
IUC correctly the way CRDs are validated, it seems impossible to
easily update a CRD in order to remove an old version.
When a CRD is updated:
the list of expected storedVersions is built from
status.StoredVersions in the previous CRD + newer storedVersions in
the new CRD
this list of storedVersions is passed to a
validation function that checks all expected storedVersions are still
present in the new CRD
All served versions in the history of
the CRD end up in the Status. Any update to the CRD Status is
impossible. Any update to the CRD versions that would consist in
removing a version that exists in the status is impossible.
However, he also mentioned it can be done by using operator-lifecycle-manager.
Programmatically updating the CRD can be found here. This would bypass validation of CRD storedVersion.
Assuming I have a custom resource on my k8s cluster exposed on a proprietary api endpoint, e.g. somecompany/v1
Is there a way to validate a .yaml manifest describing this resource?
It his a functionality the custom resource provider should expose or it is natively supported by k8s for CRDs?
Let's take a look on a simple example:
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: myresources.stable.example.com
spec:
group: stable.example.com
versions:
- name: v1
served: true
storage: true
scope: Namespaced
names:
plural: myresources
singular: myresource
kind: MyResource
shortNames:
- mr
validation:
openAPIV3Schema:
required: ["spec"]
properties:
spec:
required: ["cert","key","domain"]
properties:
cert:
type: "string"
minimum: 1
key:
type: "string"
minimum: 1
domain:
type: "string"
minimum: 1
spec.validation field describes custom validation methods for your custom resource. You can block the creation of resources using validation if certain fields are left empty. In this example, OpenAPIV3Schema validation conventions is used to check the type of some fields in our custom resource. We ensure that spec , spec.cert , spec.key , and spec.domain fields of the custom resource do exist and that they are of a String type. Users can also use validatingadmissionwebhook as a validation schema. You can find more about restrictions for using this field in the official documentation.
I've registered custom resource definition in K8S:
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: resources.example.com
labels:
service: "my-resource"
spec:
group: example.com
version: v1alpha1
scope: Namespaced
names:
plural: resources
singular: resource
kind: MYRESOURCE
shortNames:
- res
Now on attempt to get an 'explain' for my custom resource with:
kubectl explain resource
I get the following error:
group example.com has not been registered
How can I add an explain information to my custom resource definition, or is this not supported for CRDs?
explain works using openapi schema information published by the server. Prior to v1.15, CRDs did not have the ability to publish that info.
In 1.15+, CRDs that specify structural schemas and enable pruning publish OpenAPI and work with explain.