I'm trying to install Gateway into my Kind cluster using Gateway API CRD by following a tutorial from https://istio.io/latest/docs/examples/bookinfo/
I had no problem with this command curl https://raw.githubusercontent.com/istio/istio/release-1.17/samples/bookinfo/gateway-api/bookinfo-gateway.yaml | kubectl apply -f -
But, command kubectl wait --for=condition=ready gtw bookinfo-gateway gave me this:
error: timed out waiting for the condition on gateways/bookinfo-gateway
I tried this in Minikube as well without any success.
Are these CRDs supported in both k8s (Minikube & Kind) platforms?
Related
Google managed prometheus seems like a great service however at the moment it does not work even in the example... https://cloud.google.com/stackdriver/docs/managed-prometheus/setup-managed
Setup:
create a new autopilot cluster 1.21.12-gke.2200
enable manage prometheus via gcloud cli command
gcloud beta container clusters update <mycluster> --enable-managed-prometheus --region us-central1
add port 8443 firewall webhook command
install ingress-nginx
try and use the PodMonitoring manifest to get metrics from ingress-nginx
Error from server (InternalError): error when creating "ingress-nginx/metrics.yaml": Internal error occurred: failed calling webhook "default.podmonitorings.gmp-operator.gke-gmp-system.monitoring.googleapis.com": Post "https://gmp-operator.gke-gmp-system.svc:443/default/monitoring.googleapis.com/v1/podmonitorings?timeout=10s": x509: certificate is valid for gmp-operator, gmp-operator.gmp-system, gmp-operator.gmp-system.svc, not gmp-operator.gke-gmp-system.svc
There is a thread suggesting this will all be fixed this week (8/11/2022), https://github.com/GoogleCloudPlatform/prometheus-engine/issues/300, but it seems like this should work regardless.
if I try to port forward ...
kubectl -n gke-gmp-system port-forward svc/gmp-operator 8443
error: Pod 'gmp-operator-67d5fff8b9-p4n7t' does not have a named port 'webhook'
[ Note to reader: This is a long question, please bear with me ]
I've been messing with Cloud Run for Anthos as a side project, and have got a service up and running on http (not https) -- it performs as expected, scaling up and scaling down as traffic requires.
I now want to add SSL to it, so traffic comes on SSL only. And I've been banging my head against a wall on this trying to get it to work. Mainly centering on this link
I have setup the cluster default domain (it shows up on the Cloud Run for Anthos page) and my service is accessible via http://myservice.mynamespace.mydomain.com (not my real domain, obviously)
I've patched the knative configs, enabling autoTLS and setting mydomain as needed:
kubectl patch configmap config-domain --namespace knative-serving --patch '{"data": {"mydomain.com": ""}}'
kubectl patch configmap config-domain --namespace knative-serving --patch '{"data": {"nip.io": null}}'
kubectl patch configmap config-domainmapping --namespace knative-serving --patch '{"data": {"autoTLS": "Enabled"}}'
kubectl patch configmap config-network --namespace knative-serving --patch '{"data": {"autoTLS": "Enabled"}}'
kubectl patch configmap config-network --namespace knative-serving --patch '{"data": {"httpProtocol": "Enabled"}}'
The documentation talks about using gcloud domain mapping:
gcloud run domain-mappings describe --domain DOMAIN
Yet, this only is available on beta
ERROR: (gcloud.run.domain-mappings.describe) This command group is in beta for fully managed Cloud Run; use `gcloud beta run domain-mappings`.
Using beta, however, also fails
ERROR: (gcloud.beta.run.domain-mappings.describe) NOT_FOUND: Resource 'mydomain.com' of kind 'DOMAIN_MAPPING' in region 'europe-west2' in project 'my-project' does not exist
Making my cluster zonal or regional makes no difference.
The documentation also mentions using kcert
kubectl get kcert
Yet this does not show anything until after I have deployed my service.
$ kubectl get kcert --all-namespaces
NAMESPACE NAME READY REASON
default route-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
And this NEVER shows ready. Trying to hit http://myservice.mynamespace.mydomain.com remains fine, but hitting https://myservice.mynamespace.mydomain.com returns:
curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to myservice.mynamespace.mydomain.com:443
Can anyone guide me into why I can't get SSL working on this?
Related question -- is it possible to use the already-configured cert-manager cluster issuer instead? Such as described here?
When I do Kubernetes port-forward with IBM cluster I get connection refused. I have access to other clusters like Azure Kubernetes Service and kubectl port-forward is working fine there. Also when I get a pod log using kubectl logs {pod_name} I get TLS handshake error but the other kubernetes commands like get pod and describe pod is working fine.
We are using Airflow to schedule Spark job on Kubernetes. Recently, I have encountered a scenario where:
airflow received error 404 with message "pods pod-name not found"
I manually checked that POD was actually working fine at that time. In fact, I was able to collect logs using kubectl logs -f -n namespace podname
What happened due to this is that airflow created another POD for running the same job which resulted in race condition.
Airflow is using Kubernetes Python client's read_namespaced_pod API()
def read_pod(self, pod):
"""Read POD information"""
try:
return self._client.read_namespaced_pod(pod.metadata.name, pod.metadata.namespace)
except BaseHTTPError as e:
raise AirflowException(
'There was an error reading the kubernetes API: {}'.format(e)
)
I believe read_namespaced_pod() calls Kubernetes API. In order to investigate this further, I would like to like check logs of Kubernetes API server.
Can you please share steps to check what is happening on Kubernetes side ?
Note: Kubernetes version is 1.18 and Airflow version is 1.10.10.
Answering the question from the perspective of logs/troubleshooting:
I believe read_namespaced_pod() calls Kubernetes API. In order to investigate this further, I would like to like check logs of Kubernetes API server.
Yes, you are correct, this function calls the Kubernetes API. You can check the logs of Kubernetes API server by running:
$ kubectl logs -n kube-system KUBERNETES_API_SERVER_POD_NAME
I would also consider checking the kube-controller-manager:
$ kubectl logs -n kube-system KUBERNETES_CONTROLLER_MANAGER_POD_NAME
The example output of it:
I0413 12:33:12.840270 1 event.go:291] "Event occurred" object="default/nginx-6799fc88d8" kind="ReplicaSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulCreate" message="Created pod: nginx-6799fc88d8-kchp7"
A side note!
Above commands will work assuming that your kubernetes-apiserver and kubernetes-controller-manager Pod is visible to you
Can you please share steps to check what is happening on Kubernetes side ?
This question targets the basics of troubleshooting/logs checking.
For that you can use following commands (and the ones mentioned earlier):
$ kubectl get RESOURCE RESOURCE_NAME:
example: $ kubectl get pod airflow-pod-name
also you can add -o yaml for more information
$ kubectl describe RESOURCE RESOURCE_NAME:
example: $ kubectl describe pod airflow-pod-name
$ kubectl logs POD_NAME:
example: $ kubectl logs airflow-pod-name
Additional resources:
Kubernetes.io: Docs: Concepts: Cluster administration: Logging Architecture
Kubernetes.io: Docs: Tasks: Debug application cluster: Debug cluster
I'm using mongodb-exporter for store/query the metrics via prometheus. I have set up a custom metric server and storing values for that .
That is the evidence of prometheus-exporter and custom-metric-server works compatible .
Query:
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/monitoring/pods/*/mongodb_mongod_wiredtiger_cache_bytes"
Result:
{"kind":"MetricValueList","apiVersion":"custom.metrics.k8s.io/v1beta1","metadata":{"selfLink":"/apis/custom.metrics.k8s.io/v1beta1/namespaces/monitoring/pods/%2A/mongodb_mongod_wiredtiger_cache_bytes"},"items":[{"describedObject":{"kind":"Pod","namespace":"monitoring","name":"mongo-exporter-2-prometheus-mongodb-exporter-68f95fd65d-dvptr","apiVersion":"/v1"},"metricName":"mongodb_mongod_wiredtiger_cache_bytes","timestamp":"TTTTT","value":"0"}]}
In my case when I create a hpa for this custom metrics from mongo exporter, hpa return this error to me :
failed to get mongodb_mongod_wiredtiger_cache_bytes utilization: unable to get metrics for resource mongodb_mongod_wiredtiger_cache_bytes: no metrics returned from resource metrics API
What is the main issue on my case ? I have checked all configs and flow is looking fine, but where is the my mistake .
Help
Thanks :)
In comments you wrote that you have enabled external.metrics, however in original question you had issues with custom.metrics
In short:
metrics supports only basic metric like CPU or Memory.
custom.metrics allows you to extend basic metrics to all Kubernetes objects (http_requests, number of pods, etc.).
external.metrics allows to gather metrics which are not Kubernetes objects:
External metrics allow you to autoscale your cluster based on any
metric available in your monitoring system. Just provide a metric
block with a name and selector, as above, and use the External metric
type instead of Object
For more detailed description, please check this doc.
Minikube
To verify if custom.metrics are enabled you need to execute command below and check if you can see any metrics-server... pod.
$ kubectl get pods -n kube-system
...
metrics-server-587f876775-9qrtc 1/1 Running 4 5d1h
Second way is to check if minikube have enabled metrics-server by
$ minikube addons list
...
- metrics-server: enabled
If it is disabled just execute
$ sudo minikube addons enable metrics-server
✅ metrics-server was successfully enabled
GKE
Currently at GKE heapster and metrics-server are turn on as default but custom.metrics are not supported by default.
You have to install prometheus adapter or stackdriver.
Kubeadm
Kubeadm do not include heapster or metrics server at the beginning. For easy installation, you can use this YAML.
Later you have to install prometheus adapter.
Apply custom.metrics
It's the same for Minikube, Kubeadm, GKE.
Easiest way to apply custom.metrics is to install prometheus adapter via Helm.
After helm installation you will be able to see note:
NOTES:
my-release-prometheus-adapter has been deployed.
In a few minutes you should be able to list metrics using the following command(s):
kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1
As additional information, you can use jq to get more user friendly output.
kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 | jq .