How to find resource issuing request in Kubernetes cluster by IP address? - kubernetes

Update:
What I am trying to achieve: I have the IP address of a cluster resource (10.1.239.128 in the examples below) and want to figure out to which POD/service/... it belongs.
I set up a Kuebernetes cluster using Microk8s running, among others, vault and cert-manager via Helm charts. In two of the PODs created for vault and cert-manager, I get multiple the following log entry:
> kubectl logs pod/cert-manager-webhook-6787f645b9-s5fxl
http: TLS handshake error from 10.1.239.128:37763: EOF
> kubectl logs pod/release-name-vault-0
http: TLS handshake error from 10.1.239.128:48609: remote error: tls: unknown certificate authority
I am struggling with finding out where these requests come from.
I tried:
kubectl get svc --all-namespaces
kubectl get endpoints --all-namespaces
kubectl get endpointslices --all-namespaces
ping -a 10.1.239.128 using from a POD created using netshoot
to figure out from where the requests are coming from, but to no luck. The ip address is not listed or resolved to a DNS name to further investigate.
Any further ideas of how to get the resource issuing the request or further debug this? Thanks in advance.

Here is a workaround, not the finest way of doing it, but it may help to get in the right direction:
kubectl api-resources -oname |while read r;
do
echo -n "$r ----> ";
kubectl get $r -A -o yaml |grep -oPz '\d+\.\d+\.\d+\.\d+';
echo "" ;
done
Assuming the IP, you are searching for is a cluster resource.

Related

Unable to deploy WSO2 APIM in Minikube Kubernetes cluster

I'm trying to deploy WSO2 APIM on Kubernetes using the pattern-1 described on the github page https://github.com/wso2/kubernetes-apim. I have added my minikube ip to my etc/hosts file as follows:
[minikube ip] am.wso2.com gateway.am.wso2.com
I'm unable to access the Publisher and Devportal using this url:https://am.wso2.com/publisher
Is there any other configuration that needs to be done? Any help would be great:). Thanks in advance..
First, make sure all your WSO2 pods are running and they're in the ready state.
kubectl get po -n <your_namespace>
This should output.
Then make sure you have enabled Ingress addon.
minikube addons list
Then make sure Ingress pods are running.
kubectl get po -n ingress-nginx
Next, get the Ingress external IP.
kubectl get ing -A
Get the external IP and the Host from the above and add a entry to the /etc/hosts as shown below.
If everything is in place you should be able to access the Publisher by going to https://am.wso2.com/
Try to run the below command in the command line.
minikube tunnel

Kong Gateway using Kubernetes

Trying to deploy kong gateway via Kubernetes:
Created a namespace: kong-helm
Applied yaml files (using kubectl on kong-helm namespace) which includes: configmap.yaml, service.yaml, secret.yaml, ingress.yaml.
Upon applying the dbless.yaml(https://raw.githubusercontent.com/Kong/kubernetes-ingress-controller/master/deploy/single/all-in-one-dbless.yaml)ingress dbless pod is running.
kubectl get svc --all-namespaces - able to see the service(kong-test-poc) is created.
But when port forward is given: kubectl port-forward service/kong-test-poc 80:8080
Getting the following error: Error from server (NotFound): services "kong-test-poc" not found
Can you please tell how to rectify this error?
I believe you are missing the specific namespace where the service is running to it's going to your default namespace.
kubectl -n kong-helm port-forward service/kong-test-poc 8080:8080
I also recommend using an different port than 80 locally as this a unix reserved port. Also make sure that the kong-test-poc is configured to listen on 8080 (you didn't post the definition)

kubernetes api-server understanding

Hi i'm a newbie for k8s and i was wondering where and how kubectl sends requests to the kube api-server.
So for example, if i'm sending a request such as "kubectl get pods --all-namespaces"(and my default kubernetes endpoints is set as "192.168.64.2:8443"), my understanding is that this would translate to a https request such as "https://192.168.64.2:8443/api/v1/pods......etc" and kubectl would use authentication stored in .kube/config file. Am i right?
And i also have a metrics-server up and running on endpoint "172.17.0.8:4443" but how does kubectl know to use this ip when i run "kubectl get --raw /apis/metrics.k8s.io/v1beta1/nodes/<NODE_NAME> | jq"? are all kubectl commands directed to one ip?
Thanks in advance.
Authentication is one of the steps that kubectl achieves. You could see what happens when a command run with verbose ability. for example,
kubectl get pods -v9 --all-namespaces
Kubernetes know resource definitions and their implementation, you could check resource types with,
kubectl api-resources
So Kubernetes api-server knows which resources are metric-server and how that could call.
All kubectl requests go to the api-server. The api-server can either answer by itself or it delegates to other components, for example an extension api server.

openshift add service account to deployed app

I'm trying to add a service account to a deployed application but so far I keep getting the "application not available message" I did the following
created service account
oc create sa name-sa
oc add policy add-scc-to-user anyuid -z name-sa -n book
add service account to deployed app
oc set serviceacccount deploymentconfig wordapp name-sa
I check the pods and the application is running but I still not able to see any output from the route and the oc desribe pod command doesn't give any errors
I'm not sure that ServiceAccount's permission causes on this matter.
I think first you should check out the relationship of the access flow through Route -> Service -> Pod, and verify your application work well using curl command.
I show you the troubleshooting steps as follows.
Check your Route what Service is bound with it.
In this case, docker-registry Service is associated with the Route.
$ oc describe route <your routename>
:
Service: docker-registry
Weight: 100 (100%)
Endpoints: 10.128.1.94:5000 <--- You can check if this IP is matched with your application pod IP.
Then check the Service whether it can detect Endpoint pods correctly.
$ oc describe svc docker-registry
:
Port: 5000-tcp 5000/TCP
TargetPort: 5000/TCP
Endpoints: 10.128.1.94:5000 <--- You can check if this IP is matched with your application pod IP.
Verify the accessibility for the application on the pod using curl
$ oc rsh <your pod name> curl -vs http://localhost:5000/
:
< HTTP/1.1 200 OK <--- You check if you can get expected response of your application on the pod.
Additionally, you can also check your pod are running with setting SCC permission and ServiceAccount.
4 oc get pod <your podname> -o yaml | grep -E 'scc|serviceAccountName'
openshift.io/scc: anyuid
serviceAccountName: name-sa

How to format the output of kubectl describe to JSON

kubectl get command has this flag -o to format the output.
Is there a similar way to format the output of the kubectl describe command?
For example:
kubectl describe -o="jsonpath={...}" pods my-rc
would print a JSON format for the list of pods in my-rc replication controller. But -o is not accepted for the describe command.
kubectl describe doesn't support -o or equivalent. It's meant to be human-readable rather than script-friendly. You can achieve what you described with kubectl get pods -l <selector_of_your_rc> -o <output_format>, for example:
$ kubectl get pods -l app=guestbook,tier=frontend -o name
pod/frontend-a4kjz
pod/frontend-am1ua
pod/frontend-yz2dq
In my case, I needed to get the load balancer address from the service. I did it using kubectl get service:
$ kubectl -n <namespace> -ojson get service <service>
{
"apiVersion": "v1",
"kind": "Service",
[...]
"status": {
"loadBalancer": {
"ingress": [
{
"hostname": "internal-xxxxxxxxxxxxxxxxxxxxxxxxxxx-yyyyyyyyyy.us-east-1.elb.amazonaws.com"
}
[...]
}
Based on the output of kubectl help describe, it looks like it does not support structured output:
$ kubectl help describe
Show details of a specific resource or group of resources.
This command joins many API calls together to form a detailed description of a
given resource or group of resources.
$ kubectl describe TYPE NAME_PREFIX
will first check for an exact match on TYPE and NAME_PREFIX. If no such resource
exists, it will output details for every resource that has a name prefixed with NAME_PREFIX
Possible resource types include (case insensitive): pods (po), services (svc), deployments,
replicasets (rs), replicationcontrollers (rc), nodes (no), events (ev), limitranges (limits),
persistentvolumes (pv), persistentvolumeclaims (pvc), resourcequotas (quota), namespaces (ns),
serviceaccounts, ingresses (ing), horizontalpodautoscalers (hpa), daemonsets (ds), configmaps,
componentstatuses (cs), endpoints (ep), and secrets.
Usage:
kubectl describe (-f FILENAME | TYPE [NAME_PREFIX | -l label] | TYPE/NAME) [flags]
Examples:
# Describe a node
kubectl describe nodes kubernetes-minion-emt8.c.myproject.internal
# Describe a pod
kubectl describe pods/nginx
# Describe a pod identified by type and name in "pod.json"
kubectl describe -f pod.json
# Describe all pods
kubectl describe pods
# Describe pods by label name=myLabel
kubectl describe po -l name=myLabel
# Describe all pods managed by the 'frontend' replication controller (rc-created pods
# get the name of the rc as a prefix in the pod the name).
kubectl describe pods frontend
Flags:
-f, --filename=[]: Filename, directory, or URL to a file containing the resource to describe
-l, --selector="": Selector (label query) to filter on
Global Flags:
--alsologtostderr[=false]: log to standard error as well as files
--certificate-authority="": Path to a cert. file for the certificate authority.
--client-certificate="": Path to a client certificate file for TLS.
--client-key="": Path to a client key file for TLS.
--cluster="": The name of the kubeconfig cluster to use
--context="": The name of the kubeconfig context to use
--insecure-skip-tls-verify[=false]: If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure.
--kubeconfig="": Path to the kubeconfig file to use for CLI requests.
--log-backtrace-at=:0: when logging hits line file:N, emit a stack trace
--log-dir="": If non-empty, write log files in this directory
--log-flush-frequency=5s: Maximum number of seconds between log flushes
--logtostderr[=true]: log to standard error instead of files
--match-server-version[=false]: Require server version to match client version
--namespace="": If present, the namespace scope for this CLI request.
--password="": Password for basic authentication to the API server.
-s, --server="": The address and port of the Kubernetes API server
--stderrthreshold=2: logs at or above this threshold go to stderr
--token="": Bearer token for authentication to the API server.
--user="": The name of the kubeconfig user to use
--username="": Username for basic authentication to the API server.
--v=0: log level for V logs
--vmodule=: comma-separated list of pattern=N settings for file-filtered logging
There is a straightforward way, which might help.
You can run below command to get the yaml file of the service. Then copy paste to a new file.
kubectl edit svc {xx-servcice} -n {namespace} -o yaml
kubectl doesn't not support -o yaml/json for describe, but you can still use some other commands to get the info in describe, such as :
kubectl get events
As #janekuto suggested
describe cannot be used to display data in json format
Please see my answer here. jq is really a powerful utility to play around with json display of kubectl. You can do so much with jq without putting too much efforts.
kubectl - format the resource quota values in json format