[ Note to reader: This is a long question, please bear with me ]
I've been messing with Cloud Run for Anthos as a side project, and have got a service up and running on http (not https) -- it performs as expected, scaling up and scaling down as traffic requires.
I now want to add SSL to it, so traffic comes on SSL only. And I've been banging my head against a wall on this trying to get it to work. Mainly centering on this link
I have setup the cluster default domain (it shows up on the Cloud Run for Anthos page) and my service is accessible via http://myservice.mynamespace.mydomain.com (not my real domain, obviously)
I've patched the knative configs, enabling autoTLS and setting mydomain as needed:
kubectl patch configmap config-domain --namespace knative-serving --patch '{"data": {"mydomain.com": ""}}'
kubectl patch configmap config-domain --namespace knative-serving --patch '{"data": {"nip.io": null}}'
kubectl patch configmap config-domainmapping --namespace knative-serving --patch '{"data": {"autoTLS": "Enabled"}}'
kubectl patch configmap config-network --namespace knative-serving --patch '{"data": {"autoTLS": "Enabled"}}'
kubectl patch configmap config-network --namespace knative-serving --patch '{"data": {"httpProtocol": "Enabled"}}'
The documentation talks about using gcloud domain mapping:
gcloud run domain-mappings describe --domain DOMAIN
Yet, this only is available on beta
ERROR: (gcloud.run.domain-mappings.describe) This command group is in beta for fully managed Cloud Run; use `gcloud beta run domain-mappings`.
Using beta, however, also fails
ERROR: (gcloud.beta.run.domain-mappings.describe) NOT_FOUND: Resource 'mydomain.com' of kind 'DOMAIN_MAPPING' in region 'europe-west2' in project 'my-project' does not exist
Making my cluster zonal or regional makes no difference.
The documentation also mentions using kcert
kubectl get kcert
Yet this does not show anything until after I have deployed my service.
$ kubectl get kcert --all-namespaces
NAMESPACE NAME READY REASON
default route-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
And this NEVER shows ready. Trying to hit http://myservice.mynamespace.mydomain.com remains fine, but hitting https://myservice.mynamespace.mydomain.com returns:
curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to myservice.mynamespace.mydomain.com:443
Can anyone guide me into why I can't get SSL working on this?
Related question -- is it possible to use the already-configured cert-manager cluster issuer instead? Such as described here?
Related
We are using Airflow to schedule Spark job on Kubernetes. Recently, I have encountered a scenario where:
airflow received error 404 with message "pods pod-name not found"
I manually checked that POD was actually working fine at that time. In fact, I was able to collect logs using kubectl logs -f -n namespace podname
What happened due to this is that airflow created another POD for running the same job which resulted in race condition.
Airflow is using Kubernetes Python client's read_namespaced_pod API()
def read_pod(self, pod):
"""Read POD information"""
try:
return self._client.read_namespaced_pod(pod.metadata.name, pod.metadata.namespace)
except BaseHTTPError as e:
raise AirflowException(
'There was an error reading the kubernetes API: {}'.format(e)
)
I believe read_namespaced_pod() calls Kubernetes API. In order to investigate this further, I would like to like check logs of Kubernetes API server.
Can you please share steps to check what is happening on Kubernetes side ?
Note: Kubernetes version is 1.18 and Airflow version is 1.10.10.
Answering the question from the perspective of logs/troubleshooting:
I believe read_namespaced_pod() calls Kubernetes API. In order to investigate this further, I would like to like check logs of Kubernetes API server.
Yes, you are correct, this function calls the Kubernetes API. You can check the logs of Kubernetes API server by running:
$ kubectl logs -n kube-system KUBERNETES_API_SERVER_POD_NAME
I would also consider checking the kube-controller-manager:
$ kubectl logs -n kube-system KUBERNETES_CONTROLLER_MANAGER_POD_NAME
The example output of it:
I0413 12:33:12.840270 1 event.go:291] "Event occurred" object="default/nginx-6799fc88d8" kind="ReplicaSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulCreate" message="Created pod: nginx-6799fc88d8-kchp7"
A side note!
Above commands will work assuming that your kubernetes-apiserver and kubernetes-controller-manager Pod is visible to you
Can you please share steps to check what is happening on Kubernetes side ?
This question targets the basics of troubleshooting/logs checking.
For that you can use following commands (and the ones mentioned earlier):
$ kubectl get RESOURCE RESOURCE_NAME:
example: $ kubectl get pod airflow-pod-name
also you can add -o yaml for more information
$ kubectl describe RESOURCE RESOURCE_NAME:
example: $ kubectl describe pod airflow-pod-name
$ kubectl logs POD_NAME:
example: $ kubectl logs airflow-pod-name
Additional resources:
Kubernetes.io: Docs: Concepts: Cluster administration: Logging Architecture
Kubernetes.io: Docs: Tasks: Debug application cluster: Debug cluster
I am following along this article and try this on GKE. After adding the argument - --log_output_level=default:debug the change seems accepted as I get deployment.extensions/istio-sidecar-injector edited
, but I how do I know for sure?
The output of
pod=$(kubectl -n istio-system get pods -l istio=sidecar-injector -o jsonpath='{.items[0].metadata.name}')
and then
kubectl -n istio-system logs -f $pod
is the same as before and when I do (again)kubectl -n istio-system edit deployment istio-sidecar-injector the added argument is not there...
Depends on how installed Istio on GKE. There are multiple ways to install Istio from GKE.
If you're installing from http://cloud.google.com/istio which installs a Google-managed version of istio to your cluster, editing like kubectl -n istio-system edit deployment istio-sidecar-injector is a really bad idea, because Google will either revert it or the next version will wipe your modifications (so don't do it).
If you're installing yourself from Istio open source release, Istio is distributed as a Helm chart, and has bunch of kubernetes .yaml manifests. You can go edit those YAML manifests –or update Helm values.yaml files to add that argument. Then you can perform the Istio installation with the updated values.
If you're interested in getting help debugging istio, please get to a contributor community forum like Istio on Rocket Chat: https://istio.rocket.chat/ .
I have deployed kubernetes cluster. The issue i have is that the dashboard is not accessible from external desktop system.
Following is my setup.
Two vm's with cluster deployed, one master one node.
dashboard running without any issue the kube-dns is also working as expected.
kubernetes version is 1.7.
Issue: When trying to access dashboard externally through kubectl proxy. i get unauthorized response.
This is with rbac role and rolebindings enabled.
How to i configure the cluster for http browser access to dashboard from external system.
Any hint/suggestions are most welcome.
kubectl proxy not working > 1.7
try this:
copy ~/.kube/config file to your desktop
then run the kubect like this
export POD_NAME=$(kubectl --kubeconfig=config get pods -n kube-system -l "app=kubernetes-dashboard,release=kubernetes-dashboard" -o jsonpath="{.items[0].metadata.name}")
echo http://127.0.0.1:9090/
kubectl --kubeconfig=config -n kube-system port-forward $POD_NAME 9090:9090
Then access the ui like this: http://127.0.0.1:9090
see this helps
If kubectl proxy gives the Unauthorized error, there can be 2 reasons:
Your user cert doesn't have the appropriate permissions. This is unlikely since you successfully deployed kube-dns and the dashboard.
kubelet authn/authz is enabled and it's not setup correctly. See the answer to my question.
I'm trying to deploy an application in a GKE 1.6.2 cluster running ContainerOS but the instructions on the website / k8s are not accurate anymore.
The error that I'm getting is:
Error from server (Forbidden): User "circleci#gophers-slack-bot.iam.gserviceaccount.com"
cannot get deployments.extensions in the namespace "gopher-slack-bot".:
"No policy matched.\nRequired \"container.deployments.get\" permission."
(get deployments.extensions gopher-slack-bot)
The repository for the application is available here available here.
Thank you.
I had a few breaking changes in the past with using the gcloud tool to authenticate kubectl to a cluster, so I ended up figuring out how to auth kubectl to a specific namespace independent of GKE. Here's what works for me:
On CircleCI:
setup_kubectl() {
echo "$KUBE_CA_PEM" | base64 --decode > kube_ca.pem
kubectl config set-cluster default-cluster --server=$KUBE_URL --certificate-authority="$(pwd)/kube_ca.pem"
kubectl config set-credentials default-admin --token=$KUBE_TOKEN
kubectl config set-context default-system --cluster=default-cluster --user=default-admin --namespace default
kubectl config use-context default-system
}
And here's how I get each of those env vars from kubectl.
kubectl get serviceaccounts $namespace -o json
The service account will contain the name of it's secret. In my case, with the default namespace, it's
"secrets": [
{
"name": "default-token-655ls"
}
]
Using the name, I get the contents of the secret
kubectl get secrets $secret_name -o json
The secret will contain ca.crt and token fields, which match the $KUBE_CA_PEM and $KUBE_TOKEN in the shell script above.
Finally, use kubectl cluster-info to get the $KUBE_URL value.
Once you run setup_kubectl on CI, your kubectl utility will be authenticated to the namespace you're deploying to.
In Kubernetes 1.6 and GKE, we introduce role based cess control. The authors of your took need to give the service account the ability to get deployments (along with probably quite a few others) to its account creation.
https://kubernetes.io/docs/admin/authorization/rbac/
I've used the following command to update the image run in a deployment:
kubectl --cluster websites --namespace production set image
deployment/mobile-web mobile-web=eu.gcr.io/websites/mobile-web:0.23
This worked well until I created a staging namespace mirroring the production environment. In other words the deployment mobile-web exists both in the production and staging namespace. Now I get the error:
Error from server: the server could not find the requested resource
(get deployments.extensions mobile-web)
What am I missing here? Or is the only way to update using a yaml- or JSON-file, which means a bit more work on the CI/CD pipeline? I've tried setting the namespace with:
kubectl config set-context production --namespace=production --cluster=websites
but to no avail.
The solution for my concern was to kill the current proxy and get new credentials and start the proxy again:
gcloud container clusters get-credentials websites
kubectl proxy --port=8080
Now either commands work as expected:
kubectl get deployment mobile-web --namespace=production
kubectl get deployment mobile-web --namespace=staging
However it doesn't explain why it stopped working in the first place.