Extract status of Kubernetes CR created via ansible-operator - kubernetes

I am new to json query. Facing trouble extracting the status.conditions[ansibleResult].type
I have a CRD defined and created CR against the same, which is picked up by operator-sdk running ansible in the background. I am updating the CRD to provide relevant status once CR is accepted and processed by operator-sdk.
CR output in json appears like below.
{
"apiVersion": "vault.cpe.oraclecloud.com/v1alpha1",
"kind": "OciVaultKeys",
"metadata": {
"annotations": {
"kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"vault.cpe.oraclecloud.com/v1alpha1\",\"kind\":\"OciVaultKeys\",\"metadata\":{\"annotations\":{},\"name\":\"operator-key-broken\",\"namespace\":\"tms\"},\"spec\":{\"freeformTags\":[{\"key\":\"Type\",\"value\":\"Optional-Values-Added\"}],\"ociVaultKeyName\":\"operator-key-broken\",\"ociVaultKeyShapeAlgorithm\":\"RSA\",\"ociVaultKeyShapeLength\":32,\"ociVaultName\":\"ocivault-sample-12\"}}\n"
},
"creationTimestamp": "2022-03-18T07:43:03Z",
"finalizers": [
"vault.cpe.oraclecloud.com/finalizer"
],
"generation": 1,
"name": "operator-key-broken",
"namespace": "tms",
"resourceVersion": "717880023",
"selfLink": "/apis/vault.cpe.oraclecloud.com/v1alpha1/namespaces/tms/ocivaultkeys/operator-key-broken",
"uid": "0d634e72-f592-48e0-be9b-ebfa017b2dfe"
},
"spec": {
"freeformTags": [
{
"key": "Type",
"value": "Optional-Values-Added"
}
],
"ociVaultKeyName": "operator-key-broken",
"ociVaultKeyShapeAlgorithm": "RSA",
"ociVaultKeyShapeLength": 32,
"ociVaultName": "ocivault-sample-12"
},
"status": {
"conditions": [
{
"lastTransitionTime": "2022-03-18T07:43:27Z",
"message": "",
"reason": "",
"status": "False",
"type": "Successful"
},
{
"lastTransitionTime": "2022-03-18T08:26:08Z",
"message": "Running reconciliation",
"reason": "Running",
"status": "False",
"type": "Running"
},
{
"ansibleResult": {
"changed": 0,
"completion": "2022-03-18T08:26:24.217728",
"failures": 1,
"ok": 14,
"skipped": 1
},
"lastTransitionTime": "2022-03-18T08:26:25Z",
"message": "The task includes an option with an undefined variable. The error was: No first item, sequence was empty.\n\nThe error appears to be in '/home/opc/cpe-workstation/mr_folder/workspace-2/osvc-kubernetes-operators/oci-services/roles/ocivaultkeys/tasks/fetch_vault_details_oci.yml': line 12, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: DEBUG | Fetch Vault Details | Extract Vault OCID n service_endpoint in source region\n ^ here\n",
"reason": "Failed",
"status": "True",
"type": "Failure"
}
]
}
}
I wish to reliably extract the status.conditions[].type (for the element ansibleResult) in CRD.
CRD definition extract is as below
- name: v1alpha1
served: true
storage: true
additionalPrinterColumns:
- description: 'Status of the OCI Vault Key'
jsonPath: .status.conditions[-1].type
name: STATUS
type: string
priority: 0
CRD is looking for a jsonPath expression to extract.
Thanks

Please try following :
kubectl get ocivaultkeys operator-key-broken -o jsonpath='{.status.conditions[?(#.ansibleResult)].type}'
Expected output : Failure
jsonpath help

Related

How to create idempotent, re-deployable ARM templates that utilize Key Vault? Circular dependencies present issues

I'm trying to incorporate some additional security features on my resources such as encryption using customer managed keys. For the Service Bus, this requires the Service Bus to have a managed identity created and granted access to the Key Vault. However, the managed identity is not known until after the Service Bus exists.
In my ARM template, I need to initialize the Service Bus without encryption in order to get a managed identity, grant that identity access to key vault, then update the Service Bus with encryption. However, this deployment process is not repeatable. On subsequent re-deployments, it will fail because encryption cannot be removed from a Service Bus once it is granted. Even if it did work, I would be performing unnecessary steps to remove encryption and add it back on every deployment. It seems that an IAC template that describes the expected final state of the resource cannot be used to both maintain and bootstrap a new environment.
The same problem exists for API Management, I want to add custom domains but they require Key Vault access. It means I can't re-deploy my ARM template without removing the custom domains when the init step gets repeated (or keep a separate set of templates for 'initialization' vs the 'real deployment'.
Is there a better solution for this? I looked into user assigned identities which seems like it could solve the problem, but they aren't supported in ARM templates. I checked if there is a way to make the 'init' step conditional by checking if the resource already exists, and this is also not supported via ARM template.
If you want to maintain a single ARM template, you can use nested deployments to define a resource then reference it again to update it.
In the following example, I create the Service Bus with a system-assigned managed identity, a Key Vault, and an RSA key. In a nested deployment within the same ARM template that depends on that key being generated, I then update the Service Bus to enable encryption. An advantage of being all in the same template is that resourceId and reference can be used with abbreviated syntax (i.e. just the resource name).
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"name": {
"type": "string",
"defaultValue": "[resourceGroup().name]"
},
"location": {
"type": "string",
"defaultValue": "[resourceGroup().location]"
},
"tenantId": {
"type": "string",
"defaultValue": "[subscription().tenantId]"
}
},
"variables": {
"kv_name": "[concat(parameters('name'), 'kv')]",
"kv_version": "2019-09-01",
"sb_name": "[concat(parameters('name'), 'sb')]",
"sb_version": "2018-01-01-preview",
"sb_keyname": "sbkey"
},
"resources": [
{
"type": "Microsoft.ServiceBus/namespaces",
"apiVersion": "[variables('sb_version')]",
"name": "[variables('sb_name')]",
"location": "[parameters('location')]",
"sku": {
"name": "Premium",
"tier": "Premium",
"capacity": 1
},
"identity": {
"type": "SystemAssigned"
},
"properties": {
"zoneRedundant": false
}
},
{
"type": "Microsoft.KeyVault/vaults",
"apiVersion": "[variables('kv_version')]",
"name": "[variables('kv_name')]",
"location": "[parameters('location')]",
"dependsOn": [
"[variables('sb_name')]"
],
"properties": {
"sku": {
"family": "A",
"name": "Standard"
},
"tenantId": "[parameters('tenantId')]",
"accessPolicies": [
{
"tenantId": "[reference(variables('sb_name'), variables('sb_version'), 'Full').identity.tenantId]",
"objectId": "[reference(variables('sb_name'), variables('sb_version'), 'Full').identity.principalId]",
"permissions": {
"keys": [
"get",
"wrapKey",
"unwrapKey"
]
}
}
],
// Both must be enabled to encrypt Service Bus at rest.
"enableSoftDelete": true,
"enablePurgeProtection": true
}
},
{
"type": "Microsoft.KeyVault/vaults/keys",
"apiVersion": "[variables('kv_version')]",
"name": "[concat(variables('kv_name'), '/', variables('sb_keyname'))]",
"location": "[parameters('location')]",
"dependsOn": [
"[variables('kv_name')]"
],
"properties": {
"kty": "RSA",
"keySize": 2048,
"keyOps": [
"wrapKey",
"unwrapKey"
],
"attributes": {
"enabled": true
}
}
},
{
"type": "Microsoft.Resources/deployments",
"apiVersion": "2020-10-01",
"name": "sb_deployment",
"dependsOn": [
"[resourceId('Microsoft.KeyVault/vaults/keys', variables('kv_name'), variables('sb_keyname'))]"
],
"properties": {
"mode": "Incremental",
"template": {
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"resources": [
{
"type": "Microsoft.ServiceBus/namespaces",
"apiVersion": "[variables('sb_version')]",
"name": "[variables('sb_name')]",
"location": "[parameters('location')]",
"sku": {
"name": "Premium",
"tier": "Premium",
"capacity": 1
},
"identity": {
"type": "SystemAssigned"
},
"properties": {
"zoneRedundant": false,
"encryption": {
"keySource": "Microsoft.KeyVault",
"keyVaultProperties": [
{
// Ideally should specify a specific version, but no ARM template function to get this currently.
"keyVaultUri": "[reference(variables('kv_name')).vaultUri]",
"keyName": "[variables('sb_keyname')]"
}
]
}
}
}
]
}
}
}
]
}
To deploy this using the Azure CLI:
az group create -g rg-example -l westus2
az deployment group create -g rg-example -f template.json --parameters name=example

How to programmatically modify a running k8s pod status conditions?

I'm trying to modify the running state of my pod, managed by a deployment controller both from command line via kubectl patch and from the k8s python client API. Neither of them seem to work
From the command line, I tried both strategic merge match and JSON merge patch, but neither of them works. For e.g. I'm trying to patch the pod conditions to make the status field to False
kubectl -n foo-ns patch pod foo-pod-18112 -p '{
"status": {
"conditions": [
{
"type": "PodScheduled",
"status": "False"
},
{
"type": "Ready",
"status": "False"
},
{
"type": "ContainersReady",
"status": "False"
},
{
"type": "Initialized",
"status": "False"
}
],
"phase": "Running"
}
}' --type merge
From the python API
# definition of various pod states
ready_true = { "type": "Ready", "status": "True" }
ready_false = { "type": "Ready", "status": "False" }
scheduled_true = { "type": "PodScheduled", "status": "True" }
cont_ready_true = { "type": "ContainersReady", "status": "True" }
cont_ready_false = { "type": "ContainersReady", "status": "False" }
initialized_true = { "type": "Initialized", "status": "True" }
initialized_false = { "type": "Initialized", "status": "False" }
patch = {"status": { "conditions": [ready_false, initialized_false, cont_ready_false, scheduled_true ], "phase" : "Running" }}
p_status = v1.patch_namespaced_pod_status(podname, "default", body=patch)
While running the above snippet, I don't see any errors and the response p_status has all the pod conditions modified as applied in the patch, but I don't see any events from API server related to this pod status change.
May be the deployment controller is rolling back the changes to a working config? I'm looking for ways to patch the pod conditions and test if my custom controller (not related to the question) is able to see those new pod conditions.
You should not.
Clients write the desired state in the spec: and controllers write the status:-part.

How can I filter events for the cluster autoscaler in kubernetes?

I see the following event from kubectl get events:
{
"apiVersion": "v1",
"count": 1,
"eventTime": null,
"firstTimestamp": "2019-12-04T19:52:51Z",
"involvedObject": {
"apiVersion": "v1",
"kind": "Pod",
"name": "example-deployment-55f789d54c-tlwnz",
"namespace": "default",
"resourceVersion": "82663",
"uid": "2fdbd034-16cf-11ea-bc4a-42010a800186"
},
"kind": "Event",
"lastTimestamp": "2019-12-04T19:52:51Z",
"message": "Unable to mount volumes for pod \"example-deployment-55f789d54c-tlwnz_default(2fdbd034-16cf-11ea-bc4a-42010a800186)\": timeout expired waiting for volumes to attach or mount for pod \"default\"/\"example-deployment-55f789d54c-tlwnz\". list of unmounted volumes=[nfs-volume]. list of unattached volumes=[nfs-volume default-token-kc7ks]",
"metadata": {
"creationTimestamp": "2019-12-04T19:52:51Z",
"name": "example-deployment-55f789d54c-tlwnz.15dd430deb31e8fd",
"namespace": "default",
"resourceVersion": "1529",
"selfLink": "/api/v1/namespaces/default/events/example-deployment-55f789d54c-tlwnz.15dd430deb31e8fd",
"uid": "a7c80266-16cf-11ea-bc4a-42010a800186"
},
"reason": "FailedMount",
"reportingComponent": "",
"reportingInstance": "",
"source": {
"component": "kubelet",
"host": "gke-test-a2e50ea5b9f1dd9-my-node-pool-5a20b1ac-vk9q"
},
"type": "Warning"
}
....
I've tried filtering by: kubectl get events --all-namespaces -o json --field-selector source.component=cluster-autoscaler but that errors with:
{
"apiVersion": "v1",
"items": [],
"kind": "List",
"metadata": {
"resourceVersion": "",
"selfLink": ""
}
}
Error from server (BadRequest): Unable to find "/v1, Resource=events" that match label selector "", field selector "source.component=cluster-autoscaler": field label not supported: source.component
How can I filter this?
Can be done using jq (though it does not return a JSON array - but individual JSON objects seperated by newlines):
kubectl get events --all-namespaces -o json | jq '.items[]|select(.source.component=="cluster-autoscaler")'

What is my Custom Resource Definition URL in Kubernetes

I am trying to hit my custom resource definition endpoint in Kubernetes but cannot find an exact example for how Kubernetes exposes my custom resource definition in the Kubernetes API. If I hit the custom services API with this:
https://localhost:6443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions
I get back this response
"items": [
{
"metadata": {
"name": "accounts.stable.ibm.com",
"selfLink": "/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/accounts.stable.ibm.com",
"uid": "eda9d695-d3d4-11e9-900f-025000000001",
"resourceVersion": "167252",
"generation": 1,
"creationTimestamp": "2019-09-10T14:11:48Z",
"deletionTimestamp": "2019-09-12T22:26:20Z",
"finalizers": [
"customresourcecleanup.apiextensions.k8s.io"
]
},
"spec": {
"group": "stable.ibm.com",
"version": "v1",
"names": {
"plural": "accounts",
"singular": "account",
"shortNames": [
"acc"
],
"kind": "Account",
"listKind": "AccountList"
},
"scope": "Namespaced",
"versions": [
{
"name": "v1",
"served": true,
"storage": true
}
],
"conversion": {
"strategy": "None"
}
},
"status": {
"conditions": [
{
"type": "NamesAccepted",
"status": "True",
"lastTransitionTime": "2019-09-10T14:11:48Z",
"reason": "NoConflicts",
"message": "no conflicts found"
},
{
"type": "Established",
"status": "True",
"lastTransitionTime": null,
"reason": "InitialNamesAccepted",
"message": "the initial names have been accepted"
},
{
"type": "Terminating",
"status": "True",
"lastTransitionTime": "2019-09-12T22:26:20Z",
"reason": "InstanceDeletionCheck",
"message": "could not confirm zero CustomResources remaining: timed out waiting for the condition"
}
],
"acceptedNames": {
"plural": "accounts",
"singular": "account",
"shortNames": [
"acc"
],
"kind": "Account",
"listKind": "AccountList"
},
"storedVersions": [
"v1"
]
}
}
]
}
This leads me to believe I have correctly created the custom resource accounts. There are a number of examples that don't seem to be quite right and I cannot find my resource in the Kubernetes REST api. I can use with my custom resource from kubectl but I need to expose it with RESTful APIs.
https://localhost:6443/apis/stable.example.com/v1/namespaces/default/accounts
returns
404 page not found
Where as:
https://localhost:6443/apis/apiextensions.k8s.io/v1beta1/apis/stable.ibm.com/namespaces/default/accounts
returns
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "the server could not find the requested resource",
"reason": "NotFound",
"details": {},
"code": 404
}
I have looked at https://docs.okd.io/latest/admin_guide/custom_resource_definitions.html and https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/
The exact URL would be appreciated.
This is a quite decent way retrieving K8s REST API resource executing kubectl get command on some top debugging levels, like #Suresh Vishnoi mentioned in the comment:
kubectl get <api-resource> -v=8
Apparently, eventually checked by #Amit Kumar Gupta, the correct URL accessing custom resource as per your CRD json output is the following:
https://<API_server>:port/apis/stable.ibm.com/v1/namespaces/default/accounts
Depending on the authentication method you may choose: X509 Client Certs, Static Token File, Bearer Token or HTTP API proxy in order to authenticate user requests against Kubernetes API.

Resource Not Found for Creating CronJob

I am running 1.6.2, and am hitting the /apis/batch/v2alpha1/namespaces/<namespace>/cronjobs endpoint, with a valid namespace and a request body of
{
"body": {
"apiVersion": "batch/v2alpha1",
"kind": "CronJob",
"metadata": {
"name": "hello"
},
"spec": {
"schedule": "*/1 * * * *",
"jobTemplate": {
"spec": {
"template": {
"spec": {
"containers": [
{
"name": "hello",
"image": "busybox",
"args": [
"/bin/sh",
"-c",
"date; echo Hello from the Kubernetes cluster"
]
}
],
"restartPolicy": "OnFailure"
}
}
}
}
}
}
}
I receive a response of
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "the server could not find the requested resource",
"reason": "NotFound",
"details": {},
"code": 404
}
According to the documentation, this endpoint should exist. I figure I probably have some setting set incorrectly, but I'm not sure which one and how to correct it. Any help is appreciated.
The v2alpha1 features are not enabled by default. Make sure you are starting your kube-apiserver with this switch to enable the CronJob resource: --runtime-config=batch/v2alpha1=true.