Cloud Build Bazel Error: "kubectl toolchain was not properly configured so apply cannot be executed" - kubernetes

Question

I am trying to use rules_k8s for Bazel to deploy to my Kubernetes cluster.
Thus I have this cloudbuild.yaml file, which is executed by Google Cloud Build:
steps:
- name: gcr.io/cloud-builders/bazel
args: ['run', '//:kubernetes.apply']
(//:kubernetes is just a k8s_objects)
On my local machine running bazel run //:kubernetes.apply works fine, but
although the Google Cloud Build succeeds, it logs those errors. So the configuration is not applied to my Kubernetes cluster:
Target //:kubernetes.apply up-to-date:
bazel-bin/kubernetes.apply
INFO: Elapsed time: 29.863s, Critical Path: 0.14s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/kubernetes.apply
INFO: Build Event Protocol files produced successfully.
INFO: Build completed successfully, 1 total action
kubectl toolchain was not properly configured so k8s_deployment.apply cannot be executed.
kubectl toolchain was not properly configured so k8s_service.apply cannot be executed.
kubectl toolchain was not properly configured so projection_database_k8s_deployment.apply cannot be executed.
kubectl toolchain was not properly configured so projection_database_k8s_service.apply cannot be executed.
kubectl toolchain was not properly configured so k8s_deployment.apply cannot be executed.
kubectl toolchain was not properly configured so k8s_service.apply cannot be executed.
kubectl toolchain was not properly configured so k8s_deployment.apply cannot be executed.
kubectl toolchain was not properly configured so k8s_service.apply cannot be executed.
kubectl toolchain was not properly configured so event_store_k8s_deployment.apply cannot be executed.
kubectl toolchain was not properly configured so event_store_k8s_service.apply cannot be executed.
kubectl toolchain was not properly configured so k8s_deployment.apply cannot be executed.
kubectl toolchain was not properly configured so k8s_service.apply cannot be executed.
kubectl toolchain was not properly configured so event_store_k8s_deployment.apply cannot be executed.
kubectl toolchain was not properly configured so event_store_k8s_service.apply cannot be executed.
kubectl toolchain was not properly configured so k8s_deployment.apply cannot be executed.
kubectl toolchain was not properly configured so k8s_service.apply cannot be executed.
kubectl toolchain was not properly configured so event_store_k8s_deployment.apply cannot be executed.
kubectl toolchain was not properly configured so event_store_k8s_service.apply cannot be executed.
I also get a warning from the bazel cache:
DEBUG: /builder/home/.cache/bazel/_bazel_root/eab0d61a99b6696edb3d2aff87b585e8/external/io_bazel_rules_k8s/toolchains/kubectl/kubectl_toolchain.bzl:28:9: No kubectl tool was found or built, executing run for rules_k8s targets might not work.
P.S.: I get the same errors when using //:kubernetes.create
My setup
Deployments
load("#io_bazel_rules_k8s//k8s:object.bzl", "k8s_object")
k8s_object(
name = "k8s_deployment",
kind = "deployment",
cluster = "gke_cents-ideas_europe-west3-a_cents-ideas",
template = ":ideas.deployment.yaml",
images = {
"gcr.io/cents-ideas/ideas:latest": ":image"
},
)
Services
k8s_object(
name = "k8s_service",
kind = "service",
cluster = "gke_cents-ideas_europe-west3-a_cents-ideas",
template = ":ideas.service.yaml",
)
Aggregations
load("#io_bazel_rules_k8s//k8s:objects.bzl", "k8s_objects")
k8s_objects(
name = "k8s",
objects = [
":k8s_deployment",
":k8s_service",
]
)
Final composition
k8s_objects(
name = "kubernetes",
objects = [
"//services/ideas:k8s",
# ...
]
)
Update
I've now tried to make my own docker image with Bazel and kubectl:
FROM gcr.io/cloud-builders/bazel
RUN curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl
RUN chmod +x ./kubectl
RUN mv ./kubectl /usr/local/bin/kubectl
I pushed it to GCR and changed my cloudbuild.yaml to:
steps:
- name: eu.gcr.io/cents-ideas/bazel-kubectl
args: ["run", "//:kubernetes.apply"]
I firstly noticed, that the step took way longer than before. However at the end it throws an error:
$ /usr/local/bin/kubectl --kubeconfig= --cluster=gke_cents-ideas_europe-west3-a_cents-ideas --context= --user= apply -f -
error: cluster "gke_cents-ideas_europe-west3-a_cents-ideas" does not exist
Here is the full log.

Answer 1

As for the updated question, now you need to authenticate somehow to GKE inside the container.
First thing, I recommend installing gcloud tool to your container.
Btw, as for the huge container size 1.2 GB, that's because cloud-builders/bazel is huge :)
Have a look at our example on slim bazel container version:
https://github.com/aspect-development/bazel-k8s-example/blob/master/tools/Dockerfile.dazel
And here is Dockerfile for installing gcloud and kubectl, so you can grab needed parts from both files:
https://github.com/GoogleCloudPlatform/cloud-builders/blob/master/gcloud/Dockerfile
The second thing is authenticating, after gcloud is installed it should be easy.
Overall cloudbuild step should look similar to this:
- name: <link to your container>
entrypoint: /bin/sh
args:
- -c
- |
gcloud container clusters get-credentials cents-ideas --zone europe-west3-a --project cents-ideas
bazel run //:kubernetes.apply

Answer 2

It complains that it can't find kubectl on your machine. If GKE is used, also the gcloud sdk needs to be installed.
And also check that authentication is configured for those tools: kubectl auth, GKE auth.

Answer 3

The main issue is that kubectl doesn't come with the gcr.io/cloud-builders/bazel docker image. That is why it can't be found.
There are however, efforts to implement the kubectl toolchain into Bazel manually: https://github.com/bazelbuild/rules_k8s/tree/master/toolchains/kubectl#kubectl-toolchain. But as those features are currently in experimental status, they often don't work.
More info can be found in this issue: https://github.com/bazelbuild/rules_k8s/issues/512
For now, your best bet will be to install kubectl in the container.