Not able to deploy functions on kubeless - kubernetes

Im a beginner in Kubernetes. I have been trying out kubeless on minikube. I have set up both in the latest version available. When i deploy the function, this is the output that i got:
INFO[0000] Deploying function...
INFO[0000] Function hello submitted for deployment
INFO[0000] Check the deployment status executing 'kubeless function ls hello'
When i run the kubeless function ls, i get this:
NAME NAMESPACE HANDLER RUNTIME DEPENDENCIES STATUS
hello default example.hello python3.6 MISSING: Check controller logs
MISSING: Check controller logs every time i create a function it is showing this status. I also checked by changing the RUNTIME to python2.7, but still it doesn't work. The deploy command is following
kubeless function deploy hello --runtime python3.6 --from-file python-example/example.py --handler example.hello
Please guide me on how to fix this issue.

As I can see from the kubeless.io:
To debug "MISSING: Check controller logs" kind of issues it is necessary to check what is the error in the controller logs. To retrieve these logs execute:
$ kubectl logs -n kubeless -l kubeless=controller
There are cases in which the validations done in the CLI won't be enough to spot a problem in the given parameters. If that is the case the function Deployment will never appear.
Hope that helps.

From kubeless code, this status happens if kubeless cannot get status of the k8s deployment for this function.
status, err := getDeploymentStatus(cli, f.ObjectMeta.Name, f.ObjectMeta.Namespace)
if err != nil && k8sErrors.IsNotFound(err) {
status = "MISSING: Check controller logs"
}
So there are some possible reasons as followed:
There is a runtime issue for this function, for example, syntax issue or dependency issue which cause the pod failed to run. Check the pod logs can help to figure out.(This happens for my case, not sure whether it is caused by the second reason which cause kubeless cannot get the failure message)
The kubeless version is not compatible with the k8s cluster version. From k8s 1.15, the extension/v1beta1 version for deployment is removed. However, early version kubeless still uses extension/v1beta1 to get the status of deployment. You can check the api-resources of your k8s cluster.
$kubectl api-resources | grep deployments
deployments deploy apps true Deployment
#kubectl api-versions | grep apps
apps/v1
Check the following change list of kubeless which uses new apps/v1 endpoint.
Use new apps/v1 endpoint
func getDeploymentStatus(cli kubernetes.Interface, funcName, ns string) (string, error) {
- dpm, err := cli.ExtensionsV1beta1().Deployments(ns).Get(funcName, metav1.GetOptions{})
+ dpm, err := cli.AppsV1().Deployments(ns).Get(funcName, metav1.GetOptions{})

First, get the name for the kubeless-controller pod:
kubectl -n kubeless get pods
You can get the logs from the Kubeles controller:
kubectl logs -n kubeless -c kubeless-function-controller kubeless-controller-manager-5dc8f64bb7-b9x4r

Related

Helm 3 Deployment Order of Kubernetes Service Catalog Resources

I am using Helm v3.3.0, with a Kubernetes 1.16.
The cluster has the Kubernetes Service Catalog installed, so external services implementing the Open Service Broker API spec can be instantiated as K8S resources - as ServiceInstances and ServiceBindings.
ServiceBindings reflect as K8S Secrets and contain the binding information of the created external service. These secrets are usually mapped into the Docker containers as environment variables or volumes in a K8S Deployment.
Now I am using Helm to deploy my Kubernetes resources, and I read here that...
The [Helm] install order of Kubernetes types is given by the enumeration InstallOrder in kind_sorter.go
In that file, the order does neither mention ServiceInstance nor ServiceBinding as resources, and that would mean that Helm installs these resource types after it has installed any of its InstallOrder list - in particular Deployments. That seems to match the output of helm install --dry-run --debug run on my chart, where the order indicates that the K8S Service Catalog resources are applied last.
Question: What I cannot understand is, why my Deployment does not fail to install with Helm.
After all my Deployment resource seems to be deployed before the ServiceBinding is. And it is the Secret generated out of the ServiceBinding that my Deployment references. I would expect it to fail, since the Secret is not there yet, when the Deployment is getting installed. But that is not the case.
Is that just a timing glitch / lucky coincidence, or is this something I can rely on, and why?
Thanks!
As said in the comment I posted:
In fact your Deployment is failing at the start with Status: CreateContainerConfigError. Your Deployment is created before Secret from the ServiceBinding. It's only working as it was recreated when the Secret from ServiceBinding was available.
I wanted to give more insight with example of why the Deployment didn't fail.
What is happening (simplified in order):
Deployment -> created and spawned a Pod
Pod -> failing pod with status: CreateContainerConfigError by lack of Secret
ServiceBinding -> created Secret in a background
Pod gets the required Secret and starts
Previously mentioned InstallOrder will leave ServiceInstace and ServiceBinding as last by comment on line 147.
Example
Assuming that:
There is a working Kubernetes cluster
Helm3 installed and ready to use
Following guides:
Kubernetes.io: Instal Service Catalog using Helm
Magalix.com: Blog: Kubernetes Service Catalog
There is a Helm chart with following files in templates/ directory:
ServiceInstance
ServiceBinding
Deployment
Files:
ServiceInstance.yaml:
apiVersion: servicecatalog.k8s.io/v1beta1
kind: ServiceInstance
metadata:
name: example-instance
spec:
clusterServiceClassExternalName: redis
clusterServicePlanExternalName: 5-0-4
ServiceBinding.yaml:
apiVersion: servicecatalog.k8s.io/v1beta1
kind: ServiceBinding
metadata:
name: example-binding
spec:
instanceRef:
name: example-instance
Deployment.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: ubuntu
spec:
selector:
matchLabels:
app: ubuntu
replicas: 1
template:
metadata:
labels:
app: ubuntu
spec:
containers:
- name: ubuntu
image: ubuntu
command:
- sleep
- "infinity"
# part below responsible for getting secret as env variable
env:
- name: DATA
valueFrom:
secretKeyRef:
name: example-binding
key: host
Applying above resources to check what is happening can be done in 2 ways:
First method is to use timestamp from $ kubectl get RESOURCE -o yaml
Second method is to use $ kubectl get RESOURCE --watch-only=true
First method
As said previously the Pod from the Deployment couldn't start as Secret was not available when the Pod tried to spawn. After the Secret was available to use, the Pod started.
The statuses this Pod had were the following:
Pending
ContainerCreating
CreateContainerConfigError
Running
This is a table with timestamps of Pod and Secret:
| Pod | Secret |
|-------------------------------------------|-------------------------------------------|
| creationTimestamp: "2020-08-23T19:54:47Z" | - |
| - | creationTimestamp: "2020-08-23T19:54:55Z" |
| startedAt: "2020-08-23T19:55:08Z" | - |
You can get this timestamp by invoking below commands:
$ kubectl get pod pod_name -n namespace -o yaml
$ kubectl get secret secret_name -n namespace -o yaml
You can also get get additional information with:
$ kubectl get event -n namespace
$ kubectl describe pod pod_name -n namespace
Second method
This method requires preparation before running Helm chart. Open another terminal window (for this particular case 2) and run:
$ kubectl get pod -n namespace --watch-only | while read line ; do echo -e "$(gdate +"%H:%M:%S:%N")\t $line" ; done
$ kubectl get secret -n namespace --watch-only | while read line ; do echo -e "$(gdate +"%H:%M:%S:%N")\t $line" ; done
After that apply your Helm chart.
Disclaimer!
Above commands will watch for changes in resources and display them with a timestamp from the OS. Please remember that this command is only for example purposes.
The output for Pod:
21:54:47:534823000 NAME READY STATUS RESTARTS AGE
21:54:47:542107000 ubuntu-65976bb789-l48wz 0/1 Pending 0 0s
21:54:47:553799000 ubuntu-65976bb789-l48wz 0/1 Pending 0 0s
21:54:47:655593000 ubuntu-65976bb789-l48wz 0/1 ContainerCreating 0 0s
-> 21:54:52:001347000 ubuntu-65976bb789-l48wz 0/1 CreateContainerConfigError 0 4s
21:55:09:205265000 ubuntu-65976bb789-l48wz 1/1 Running 0 22s
The output for Secret:
21:54:47:385714000 NAME TYPE DATA AGE
21:54:47:393145000 sh.helm.release.v1.example.v1 helm.sh/release.v1 1 0s
21:54:47:719864000 sh.helm.release.v1.example.v1 helm.sh/release.v1 1 0s
21:54:51:182609000 understood-squid-redis Opaque 1 0s
21:54:52:001031000 understood-squid-redis Opaque 1 0s
-> 21:54:55:686461000 example-binding Opaque 6 0s
Additional resources:
Stackoverflow.com: Answer: Helm install in certain order
Alibabacloud.com: Helm charts and templates hooks and tests part 3
So to answer my own question (and thanks to #dawid-kruk and the folks on Service Catalog Sig on Slack):
In fact, the initial start of my Pods (the ones referencing the Secret created out of the ServiceBinding) fails! It fails because the Secret is actually not there the moment K8S tries to start the pods.
Kubernetes has a self-healing mechanism, in the sense that it tries (and retries) to reach the target state of the cluster as described by the various deployed resources.
By Kubernetes retrying to get the pods running, eventually (when the Secret is finally there) all conditions will be satisfied to make the pods start up nicely. Therefore, eventually, evth. is running as it should.
How could this be streamlined? One possibility would be for Helm to include the custom resources ServiceBinding and ServiceInstance into its ordered list of installable resources and install them early in the installation phase.
But even without that, Kubernetes actually deals with it just fine. The order of installation (in this case) really does not matter. And that is a good thing!

Kubernetes reports different apiVersion for the same Deployment

Due to Kubernetes removing deprecated API I am trying to figure out an apiVersion of resources in a cluster and came across the following behavior.
When requesting definition for specific deployment, it reports an old api version:
> kubectl get deployment/some-deployment -o yaml
apiVersion: extensions/v1beta1
kind: Deployment
...
Bud when requesting it for all resources api version reported as new for the same resource:
> kubectl get all -o yaml
...
- apiVersion: apps/v1
kind: Deployment
...
Same goes when checking selfLink in metadata:
kubectl get all -o=custom-columns=NAME:.metadata.name,API-version:.metadata.selfLink
and
kubectl get deployment/some-deployment -o=custom-columns=NAME:.metadata.name,API-version:.metadata.selfLink
How should this be interpreted?
Kubernetes version is 1.15.7.
Quoted from here
kubectl get deployment foo is ambiguous, since the server has deployments in multiple api groups. When a resource exists in multiple api groups, kubectl uses the first group listed in discovery docs published by the server which contains the resource. For backwards compatibility, that is the extensions api group.
If you want to ensure you get a deployment in the apps api group, fully qualify the resource you are requesting by running kubectl get deployments.apps test-nginx
If you want a specific version, like v1, in the apps api group, include that as well: kubectl get deployments.v1.apps test-nginx
Also from here
What stored version is used for?
It is the serialized version in etcd. Whenever you fetch an object, the api server reads it from etcd, converts it into an internal version, then converts it to the version you requested.
So, If you run
$ kubectl get deployments nginx -o name
deployment.extensions/nginx
As you can see requested version for above command is deployment.extensions
And If you run
$ kubectl get all -o name
...
deployment.apps/nginx
The requested version is deployment.apps. so, it returns the version with apps/v1
Ref: https://github.com/kubernetes/kubernetes/issues/58131

no matches for kind "Deployment" in version "extensions/v1beta1"

While deploying mojaloop, Kubernetes responds with the following errors:
Error: validation failed: [unable to recognize "": no matches for kind
"Deployment" in version "apps/v1beta2", unable to recognize "": no
matches for kind "Deployment" in version "extensions/v1beta1", unable
to recognize "": no matches for kind "StatefulSet" in version
"apps/v1beta2", unable to recognize "": no matches for kind
"StatefulSet" in version "apps/v1beta1"]
My Kubernetes version is 1.16.
How can I fix the problem with the API version?
From investigating, I have found that Kubernetes doesn't support apps/v1beta2, apps/v1beta1.
How can I make Kubernetes use a not deprecated version or some other supported version?
I am new to Kubernetes and anyone who can support me I am happy
In Kubernetes 1.16 some apis have been changed.
You can check which apis support current Kubernetes object using
$ kubectl api-resources | grep deployment
deployments deploy apps true Deployment
This means that only apiVersion with apps is correct for Deployments (extensions is not supporting Deployment). The same situation with StatefulSet.
You need to change Deployment and StatefulSet apiVersion to apiVersion: apps/v1.
If this does not help, please add your YAML to the question.
EDIT
As issue is caused by HELM templates included old apiVersions in Deployments which are not supported in version 1.16, there are 2 possible solutions:
1. git clone whole repo and replace apiVersion to apps/v1 in all templates/deployment.yaml using script
2. Use older version of Kubernetes (1.15) when validator accept extensions as apiVersion for Deployment and StatefulSet.
to convert an older Deployment to apps/v1, you can run:
kubectl convert -f ./my-deployment.yaml --output-version apps/v1
You can change manually as an alternative. Fetch the helm chart:
helm fetch --untar stable/metabase
Access the chart folder:
cd ./metabase
Change API version:
sed -i 's|extensions/v1beta1|apps/v1|g' ./templates/deployment.yaml
Add spec.selector.matchLabels:
spec:
[...]
selector:
matchLabels:
app: {{ template "metabase.name" . }}
[...]
Finally install your altered chart:
helm install ./ \
-n metabase \
--namespace metabase \
--set ingress.enabled=true \
--set ingress.hosts={metabase.$(minikube ip).nip.io}
Enjoy!
I prefer kubectl explain.
# kubectl explain deploy
KIND: Deployment
VERSION: apps/v1
DESCRIPTION:
Deployment enables declarative updates for Pods and ReplicaSets.
FIELDS:
apiVersion <string>
APIVersion defines the versioned schema of this representation of an
object. Servers should convert recognized schemas to the latest internal
value, and may reject unrecognized values. More info:
https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
kind <string>
Kind is a string value representing the REST resource this object
represents. Servers may infer this from the endpoint the client submits
requests to. Cannot be updated. In CamelCase. More info:
https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
metadata <Object>
Standard object metadata.
spec <Object>
Specification of the desired behavior of the Deployment.
status <Object>
Most recently observed status of the Deployment.
With kubectl explain you can also see specific parameters of an object:
# kubectl explain Service.spec.externalTrafficPolicy
KIND: Service
VERSION: v1
FIELD: externalTrafficPolicy <string>
DESCRIPTION:
externalTrafficPolicy denotes if this Service desires to route external
traffic to node-local or cluster-wide endpoints. "Local" preserves the
client source IP and avoids a second hop for LoadBalancer and Nodeport type
services, but risks potentially imbalanced traffic spreading. "Cluster"
obscures the client source IP and may cause a second hop to another node,
but should have good overall load-spreading.
To put it simply, you don't force the current installation to use an outdated version of the API; you fix the version in your config files.
If you want to check which version your current kube supports, run :
root#ubn64:~# kubectl api-versions | grep -i apps
apps/v1
I was getting below error -
error: unable to recognize "deployment.yaml": no matches for kind "Deployment" in version "extensions/v1beta1"
solution that worked for me -
modified the line from apiVersion: extensions/v1beta1 to apiVersion: apps/v1 in deployment.yaml
Reason -
we had upgraded the K8 cluster hence this error occured.
This was annoying me because I am testing lots of helm packages so I wrote a quick script - which could be modified to sort your workflow perhaps
see below
New workflow
First fetch the chart as a tgz to your working directory
helm fetch repo/chart
then in your working directly run bash script below - which I named helmk
helmk myreleasename mynamespace chart.tgz [any parameters for kubectl create]
Contents of helmk - need to edit your kubeconfig clustername to work
#!/bin/bash
echo usage $0 releasename namespace chart.tgz [createparameter1] [createparameter2] ... [createparameter n]
echo This will use your namespace then shift back to default so be careful!!
kubectl create namespace $2 #this will create harmless error if namespace exists have to ignore
kubectl config set-context MYCLUSTERNAME --namespace $2
helm template -n $1 --namespace $2 $3 | kubectl convert -f /dev/stdin | kubectl create --save-config=true ${#:4} -f /dev/stdin
#note the --namespace parameter in helm template above seems to be ignored so we have to manually switch context
kubectl config set-context MYCLUSTERNAME --namespace default
It's a slightly dangerous hack since I manually switch to your new desired namespace context then back again so only to be used for single user devs really or comment that out.
You will get a warning about using the kubectl convert facility like this
If you need to edit the YAML to customise - just replace one of the /dev/stdin to intermediate files but It's probably better to get it up using "create" with a save-config as I have and then simply "apply" your changes which means that they will be recorded in kubernetes too.
Good luck
I was facing the same issue on a cluster that was upgraded to a version that does not support certain api versions (v1.17 and apps/v1beta2).
$ helm get manifest some-deployment
...
# Source: some-deployment/templates/deployment.yaml
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: some-deployment
labels:
...
Looking at the helm docs, it seems that the manifest is stored in the cluster for helm to reference, and it may include invalid api versions, leading to errors.
The 2 proposed methods are to either manually edit the manifest (a rather tedious multi-stage process), or use a helm plugin called mapkubeapis that does it automatically.
$ helm plugin install https://github.com/helm/helm-mapkubeapis
It can be run with the --dry-run flag to simulate the effects:
$ helm mapkubeapis --dry-run some-deployment
2021/02/15 09:33:29 NOTE: This is in dry-run mode, the following actions will not be executed.
2021/02/15 09:33:29 Run without --dry-run to take the actions described below:
2021/02/15 09:33:29
2021/02/15 09:33:29 Release 'some-deployment' will be checked for deprecated or removed Kubernetes APIs and will be updated if necessary to supported API versions.
2021/02/15 09:33:29 Get release 'some-deployment' latest version.
2021/02/15 09:33:30 Check release 'some-deployment' for deprecated or removed APIs...
2021/02/15 09:33:30 Found deprecated or removed Kubernetes API:
"apiVersion: apps/v1beta2
kind: Deployment"
Supported API equivalent:
"apiVersion: apps/v1
kind: Deployment"
2021/02/15 09:33:30 Finished checking release 'some-deployment' for deprecated or removed APIs.
2021/02/15 09:33:30 Deprecated or removed APIs exist, updating release: some-deployment.
2021/02/15 09:33:30 Map of release 'some-deployment' deprecated or removed APIs to supported versions, completed successfully.
and then run without the flag to apply the changes.

Kubectl create "invalid object"

I am creating namespace using kubectl with yaml. The following is my yaml configuration
apiVersion: v1
kind: Namespace
metadata:
name: "slackishapp"
labels:
name: "slackishapp"
But, when I run kubectl create -f ./slackish-namespace-manifest.yaml, I got an error like the following
error: SchemaError(io.k8s.api.autoscaling.v2beta2.PodsMetricStatus): invalid object doesn't have additional properties.
What goes wrong on my yaml? I am reading about it on the documentation as well. I don't see any difference with my configuration.
There is nothing wrong with your yaml but I suspect you have the wrong version of kubectl.
kubectl needs to be within 1 minor from the cluster you are using as described here.
You can check your versions with
kubectl version
Download the kubectl.exe on https://kubernetes.io/docs/tasks/tools/install-kubectl/#before-you-begin then replace it on Program Files\Docker\Docker\resources\bin if you have docker desktop
Like Andreas Wederbrand said, it is a version problem.
As a workaround, try to create your namespace within imperative mode
kubectl create ns slackishapp && kubectl label ns slackishapp name=slackishapp
And then compare existing yaml with one you have written in order to check what is missing
kubectl get ns slackishapp -o yaml --export

How to get Deployments ($ kubectl get deployments) by specifying custom property value?

I have deployed a Kubernetes service and when I query to get the Deployment $ kubectl get deployments, I can see the Deployment.
the json of the Deployment looks like below --
apiVersion: v1
kind: Deployment
metadata:
name: test
spec:
replicas: 1
template:
metadata:
labels:
app: test
release: testRelease
customProp: xyz
My question is how can I frame a query by which I can get the Deployment by specifying the 'customProp' value. Does kubectl support to pass a jsonpath as part of the query? so that I can pass a json path like jsonpath='{$.spec.template.metadata.labels.customProp}' and value against this jsonPath as 'xyz'.
This is what I am thinking to execute:
$ kubectl get deployments -n <namespace> <json path query>
However not sure how to frame the json path query and pass along with $kubectl get deployments.
Kubectl does support query feature, you can use below query
kubectl get pods --selector=customProp=xyz
Kubectl also supports JSON path expressions too, to get more details, follow the link. You can write query following the syntax shown on the link.
Yes, one can query to the kube-apiserver for a resource using jsonpath. Run following command to get what you want:
$ kubectl get deploy test -o=jsonpath='{.spec.template.metadata.labels.customProp}'
For more usage, see https://kubernetes.io/docs/reference/kubectl/jsonpath.
Add a label to your deployment object. Then with below command to query specific deployment
kubectl get deploy - l labelname=labelvalue
To get deployed image name as a string, you can choose from any attribute in deployment yaml.
kubectl get deploy/${image.name} -o jsonpath="{..image}"