Assuming I have a custom resource on my k8s cluster exposed on a proprietary api endpoint, e.g. somecompany/v1
Is there a way to validate a .yaml manifest describing this resource?
It his a functionality the custom resource provider should expose or it is natively supported by k8s for CRDs?
Let's take a look on a simple example:
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: myresources.stable.example.com
spec:
group: stable.example.com
versions:
- name: v1
served: true
storage: true
scope: Namespaced
names:
plural: myresources
singular: myresource
kind: MyResource
shortNames:
- mr
validation:
openAPIV3Schema:
required: ["spec"]
properties:
spec:
required: ["cert","key","domain"]
properties:
cert:
type: "string"
minimum: 1
key:
type: "string"
minimum: 1
domain:
type: "string"
minimum: 1
spec.validation field describes custom validation methods for your custom resource. You can block the creation of resources using validation if certain fields are left empty. In this example, OpenAPIV3Schema validation conventions is used to check the type of some fields in our custom resource. We ensure that spec , spec.cert , spec.key , and spec.domain fields of the custom resource do exist and that they are of a String type. Users can also use validatingadmissionwebhook as a validation schema. You can find more about restrictions for using this field in the official documentation.
Related
I have constraint template yaml code policy to prevent all namespaces deletion.
Now I want to create a policy to prevent only non-empty namespace deletion, i.e, which containing a resource like pod, ingress, pv, pvc, secret, service, etc.
Therefore empty namespace should be able to be deleted but namespace which contains content should not be able to be deleted.
Any suggestions on this?
Template:
apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
name: k8snamespacecannotbedeleted
spec:
crd:
spec:
names:
kind: K8sNamespaceCannotBeDeleted
targets:
- target: admission.k8s.gatekeeper.sh
rego: |
package kubernetes.admission
violation[{"msg": msg}] {
input.review.kind.kind == "Namespace"
input.review.operation == "DELETE"
msg := "[OPA] Namespace deletions are not permitted"
}
Constraint:
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sNamespaceCannotBeDeleted
metadata:
name: namespace-cannot-be-deleted
spec:
match:
kinds:
- apiGroups: [""]
kinds: ["Namespace"]
its probably a bit too late, but this is how i would approach this problem:
first step:
create a cronjob in your fav language that call API server, run that every x min to check for non empty namespaces, add an annotation to those namespaces, like "can_delete" : "no"
second step:
Edit your rego code and check if there is any annotations called "can_delete", if yes, and its set to "no", prevent deletion
I have an application that I can deploy to kubernetes (Google Kubernetes Engine) to which I'm trying to add Google's CDN. For this I'm adding a BackendConfig. But when my gitlab pipeline tries to apply it it returns the following error.
$ kubectl apply -f backend-config.yaml
Error from server (Forbidden): error when retrieving current configuration of:
Resource: "cloud.google.com/v1beta1, Resource=backendconfigs", GroupVersionKind: "cloud.google.com/v1beta1, Kind=BackendConfig"
I have a strongly suspect that the account the pipeline is running under does not have enough privileges to access backend configs. Being new to k8s and gke I'm not sure how to fix this. Especially as I cannot find what permission is needed for this.
Edit
I added a kubectl get backendconfigs to my pipeline and that fails with the same error. Running it from my gcloud sdk environment the same command works.
Note the cluster is managed by Gitlab and using RBAC. My understanding is that gitlab creates service accounts per namespace in k8s with the edit role.
Edit 2
Added ClusterRole and ClusterRoleBinding based on Arghya's answer.
Output of $ kubectl get crd
NAME CREATED AT
backendconfigs.cloud.google.com 2020-01-09T15:37:27Z
capacityrequests.internal.autoscaling.k8s.io 2020-04-28T11:15:26Z
certificaterequests.cert-manager.io 2020-01-15T06:53:47Z
certificates.cert-manager.io 2020-01-15T06:53:48Z
challenges.acme.cert-manager.io 2020-01-15T06:53:48Z
challenges.certmanager.k8s.io 2020-01-09T15:47:01Z
clusterissuers.cert-manager.io 2020-01-15T06:53:48Z
clusterissuers.certmanager.k8s.io 2020-01-09T15:47:01Z
issuers.cert-manager.io 2020-01-15T06:53:48Z
issuers.certmanager.k8s.io 2020-01-09T15:47:01Z
managedcertificates.networking.gke.io 2020-01-09T15:37:53Z
orders.acme.cert-manager.io 2020-01-15T06:53:48Z
orders.certmanager.k8s.io 2020-01-09T15:47:01Z
scalingpolicies.scalingpolicy.kope.io 2020-01-09T15:37:53Z
updateinfos.nodemanagement.gke.io 2020-01-09T15:37:53Z
Output of kubectl describe crd backendconfigs.cloud.google.com
Name: backendconfigs.cloud.google.com
Namespace:
Labels: <none>
Annotations: <none>
API Version: apiextensions.k8s.io/v1beta1
Kind: CustomResourceDefinition
Metadata:
Creation Timestamp: 2020-01-09T15:37:27Z
Generation: 1
Resource Version: 198
Self Link: /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/backendconfigs.cloud.google.com
UID: f0bc780a-32f5-11ea-b7bd-42010aa40111
Spec:
Conversion:
Strategy: None
Group: cloud.google.com
Names:
Kind: BackendConfig
List Kind: BackendConfigList
Plural: backendconfigs
Singular: backendconfig
Scope: Namespaced
Validation:
Open APIV 3 Schema:
Properties:
API Version:
Description: APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources
Type: string
Kind:
Description: Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds
Type: string
Metadata:
Type: object
Spec:
Description: BackendConfigSpec is the spec for a BackendConfig resource
Properties:
Cdn:
Description: CDNConfig contains configuration for CDN-enabled backends.
Properties:
Cache Policy:
Description: CacheKeyPolicy contains configuration for how requests to a CDN-enabled backend are cached.
Properties:
Include Host:
Description: If true, requests to different hosts will be cached separately.
Type: boolean
Include Protocol:
Description: If true, http and https requests will be cached separately.
Type: boolean
Include Query String:
Description: If true, query string parameters are included in the cache key according to QueryStringBlacklist and QueryStringWhitelist. If neither is set, the entire query string is included and if false the entire query string is excluded.
Type: boolean
Query String Blacklist:
Description: Names of query strint parameters to exclude from cache keys. All other parameters are included. Either specify QueryStringBlacklist or QueryStringWhitelist, but not both.
Items:
Type: string
Type: array
Query String Whitelist:
Description: Names of query string parameters to include in cache keys. All other parameters are excluded. Either specify QueryStringBlacklist or QueryStringWhitelist, but not both.
Items:
Type: string
Type: array
Type: object
Enabled:
Type: boolean
Required:
enabled
Type: object
Connection Draining:
Description: ConnectionDrainingConfig contains configuration for connection draining. For now the draining timeout. May manage more settings in the future.
Properties:
Draining Timeout Sec:
Description: Draining timeout in seconds.
Format: int64
Type: integer
Type: object
Iap:
Description: IAPConfig contains configuration for IAP-enabled backends.
Properties:
Enabled:
Type: boolean
Oauthclient Credentials:
Description: OAuthClientCredentials contains credentials for a single IAP-enabled backend.
Properties:
Client ID:
Description: Direct reference to OAuth client id.
Type: string
Client Secret:
Description: Direct reference to OAuth client secret.
Type: string
Secret Name:
Description: The name of a k8s secret which stores the OAuth client id & secret.
Type: string
Required:
secretName
Type: object
Required:
enabled
oauthclientCredentials
Type: object
Security Policy:
Type: object
Session Affinity:
Description: SessionAffinityConfig contains configuration for stickyness parameters.
Properties:
Affinity Cookie Ttl Sec:
Format: int64
Type: integer
Affinity Type:
Type: string
Type: object
Timeout Sec:
Format: int64
Type: integer
Type: object
Status:
Type: object
Version: v1beta1
Versions:
Name: v1beta1
Served: true
Storage: true
Status:
Accepted Names:
Kind: BackendConfig
List Kind: BackendConfigList
Plural: backendconfigs
Singular: backendconfig
Conditions:
Last Transition Time: 2020-01-09T15:37:27Z
Message: no conflicts found
Reason: NoConflicts
Status: True
Type: NamesAccepted
Last Transition Time: <nil>
Message: the initial names have been accepted
Reason: InitialNamesAccepted
Status: True
Type: Established
Stored Versions:
v1beta1
Events: <none>
Create a ClusterRole and ClusterRoleBinding for service account example-sa in namespace example-namespace
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: backendconfig-role
rules:
- apiGroups: ["cloud.google.com"]
resources: ["backendconfigs"]
verbs: ["get", "watch", "list", "create", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: backendconfig-rolebinding
subjects:
- kind: ServiceAccount
name: example-sa
namespace: example-namespace
roleRef:
kind: ClusterRole
name: backendconfig-role
apiGroup: rbac.authorization.k8s.io
To check the permission is applied
kubectl auth can-i get backendconfigs --as=system:serviceaccount:example-namespace:example-sa -n example-namespace
How to remove version from 'status.storedVersions' while Manually upgrading the existing objects to a new stored version in kubernetes custom resources
apiVersion for CRD: apiextensions.k8s.io/v1beta1
Using aws EKS
CRD YAML
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
# name must match the spec fields below, and be in the form: <plural>.<group>
name: crontabs.stable.example.com
spec:
# group name to use for REST API: /apis/<group>/<version>
group: stable.example.com
# list of versions supported by this CustomResourceDefinition
versions:
- name: v1
# Each version can be enabled/disabled by Served flag.
served: true
# One and only one version must be marked as the storage version.
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
image:
type: string
replicas:
type: integer
# either Namespaced or Cluster
scope: Namespaced
names:
# plural name to be used in the URL: /apis/<group>/<version>/<plural>
plural: crontabs
# singular name to be used as an alias on the CLI and for display
singular: crontab
# kind is normally the CamelCased singular type. Your resource manifests use this.
kind: CronTab
# shortNames allow shorter string to match your resource on the CLI
shortNames:
- ct
And upgrading the CRD to v2
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
# name must match the spec fields below, and be in the form: <plural>.<group>
name: crontabs.stable.example.com
spec:
# group name to use for REST API: /apis/<group>/<version>
group: stable.example.com
# list of versions supported by this CustomResourceDefinition
versions:
- name: v2
# Each version can be enabled/disabled by Served flag.
served: true
# One and only one version must be marked as the storage version.
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
image:
type: string
replicas:
type: integer
- name: v1
# Each version can be enabled/disabled by Served flag.
served: false
# One and only one version must be marked as the storage version.
storage: false
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
image:
type: string
replicas:
type: integer
# either Namespaced or Cluster
scope: Namespaced
names:
# plural name to be used in the URL: /apis/<group>/<version>/<plural>
plural: crontabs
# singular name to be used as an alias on the CLI and for display
singular: crontab
# kind is normally the CamelCased singular type. Your resource manifests use this.
kind: CronTab
# shortNames allow shorter string to match your resource on the CLI
shortNames:
- ct
After disabling version v1 I want remove it from the versions list to do that I need to remove it from status.storedVersions how can i do that the kubernetes document does not give details about that
There is also another another way of removing status.specVersion using cluster API.
Fields under status are modified via the status sub resource:
curl -d '[{ "op": "replace", "path":"/status/storedVersions", "value": ["v2"] }]' \
-H "Content-Type: application/json-patch+json" \
-X PATCH \
http://localhost:8080/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/crontabs.stable.example.com/status
To access API you need to first connected to cluster doc
Ive tried many various options on Minikube, Kubeadm, GKE but everywhere output was the same.
Unfortunately it is impossible using kubectl and docs Remove the old version from the CustomResourceDefinition spec.versions list is bit missleading.
Once a definition falls into specVersion it's impossible to remove it. storage: true puts it int the storedVersions list.
It was well described in this Github thread, especially comment from #sebgl.
IUC correctly the way CRDs are validated, it seems impossible to
easily update a CRD in order to remove an old version.
When a CRD is updated:
the list of expected storedVersions is built from
status.StoredVersions in the previous CRD + newer storedVersions in
the new CRD
this list of storedVersions is passed to a
validation function that checks all expected storedVersions are still
present in the new CRD
All served versions in the history of
the CRD end up in the Status. Any update to the CRD Status is
impossible. Any update to the CRD versions that would consist in
removing a version that exists in the status is impossible.
However, he also mentioned it can be done by using operator-lifecycle-manager.
Programmatically updating the CRD can be found here. This would bypass validation of CRD storedVersion.
If I create project 'A' and want to deploy a template from inside it that creates objects in project 'A' and also in another project 'B' how would I achieve this. I tried simply specifying the namespace of project 'B' in the template but got the error
the namespace of the provided object does not match the namespace sent on the request
According to documentation:
If an object definition’s metadata includes a fixed namespace field value, the field will be stripped out of the definition during template instantiation. If the namespace field contains a parameter reference, normal parameter substitution will be performed and the object will be created in whatever namespace the parameter substitution resolved the value to, assuming the user has permission to create objects in that namespace.
So just specify required namespaces as template's parameters. Usable example:
apiVersion: template.openshift.io/v1
kind: Template
metadata:
name: xxx
parameters:
- name: ns1
value: test1
- name: ns2
value: test2
objects:
- apiVersion: v1
data:
key1: value1
kind: ConfigMap
metadata:
namespace: ${ns1}
name: cm1
- apiVersion: v1
data:
key2: value2
kind: ConfigMap
metadata:
namespace: ${ns2}
name: cm2
I've registered custom resource definition in K8S:
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: resources.example.com
labels:
service: "my-resource"
spec:
group: example.com
version: v1alpha1
scope: Namespaced
names:
plural: resources
singular: resource
kind: MYRESOURCE
shortNames:
- res
Now on attempt to get an 'explain' for my custom resource with:
kubectl explain resource
I get the following error:
group example.com has not been registered
How can I add an explain information to my custom resource definition, or is this not supported for CRDs?
explain works using openapi schema information published by the server. Prior to v1.15, CRDs did not have the ability to publish that info.
In 1.15+, CRDs that specify structural schemas and enable pruning publish OpenAPI and work with explain.