I am using gocd for ci/cd. Result is tar archive. I need to copy resulting tar to GCE bucket.
I have gocd-agent docker image with included google sdk.
I know how to use gcloud with service account from local machine, but not from inside pod.
How to use service account assigned to pod with gcloud auth on pod?
Final goal is to use gsutil to copy above mentioned archive to bucket in same project.
My first thought would be to create Secret based on the service account, reference it in a pod yaml definiton to mount to some file and then run gcloud auth from the pod using that file. There's more info in Google cloud docs.
Another option which is quite new is to use Workload Identitiy. Seems you'd need to configure GKE cluster to enable this option. And it's working for some versions of GKE.
Related
I am working on a project in which I have created a k8s cluster to run selenium grid locally. I want to schedule the tests to run and until now I have tried to create a Jenkins cron job to do so. For that I am using k8s plugin in Jenkins.
However I am not sure about the steps to follow. Where should I be uploading the kube config file? There are a few options here:
Build Environment in Jenkins
Any ideas or suggestions?
Thanks
Typically, you can choose any option, depending on how you want to manage the system, I believe:
secret text or file option will allow you to copy/paste a secret (with a token) in Jenkins which will be used to access the k8s cluster. Token based access works by adding an HTTP header to your requests to the k8s API server as follows: Authorization: Bearer $YOUR_TOKEN. This authenticates you to the server. This is the programmatic way to access the k8s API.
configure kubectl option will allow you to perhaps specify the config file within Jenkins UI where you can set the kubeconfig. This is the imperative/scriptive way of configuring access to the k8s API. The kubeconfig itself contains set of keypair based credentials that are issued to a username and signed by the API server's CA.
Any way would work fine! Hope this helps!
If Jenkins is running in Kubernetes as well, I'd create a service account, create the necessary Role and RoleBinding to only create CronJobs, and attach your service account to your Jenkins deployment or statefulset, then you can use the token of the service account (by default mounted under /var/run/secrets/kubernetes.io/serviceaccount/token) and query your API endpoint to create your CronJobs.
However, if Jenkins is running outside of your Kubernetes cluster, I'd authenticate against your cloud provider in Jenkins using one of the plugins available, using:
Service account (GCP)
Service principal (Azure)
AWS access and secret key or with an instance profile (AWS).
and then would run any of the CLI commands to generate a kubeconfig file:
gcloud container clusters get-credentials
az aks get-credentials
aws eks update-kubeconfig
I have used Kubernetes and I deployed for example WordPress or nginx or etc. We install from yaml file. Where is it installed how can i find directory of pages(for example WordPress pages etc.) at same point at Google Cloud too. When I use Kubernetes at Google Cloud where is the path of installed files(ex. index.php).
If you are running the docker image directly without attaching anything like NFS, S3 or Disk then you will be able to get those files by default in the container file system(index.php and all).
With any K8s cluster you check files inside container either Gcloud or any :
kubectl get pods
kubectl exec -it <Wordpress pod name> -- /bin/bash
If you are attaching the File system like NFS, or object storage S3 or EFS you will be able to watch those files there unless you mount and apply config using the YAML file.
Regarding setup file (YAML),
Kubernetes uses the ETCD database as a data store. The flow is like this. Kubectl command connect to API server and sends the YAML file to API server. API parses and store the information in ETCD database so you wont be getting those file as it is in YAML format.
I deploy apps to Kubernetes running on Google Cloud from CI. CI makes use of kubectl config which contains auth information (either in directly CVS or templated from the env vars during build)
CI has seperate Google Cloud service account and I generate kubectl config via
gcloud auth activate-service-account --key-file=key-file.json
and
gcloud container clusters get-credentials <cluster-name>
This sets the kubectl config but the token expires in few hours.
What are my options of having 'permanent' kubectl config other than providing CI with key file during the build and running gcloud container clusters get-credentials ?
You should look into RBAC (role based access control) which will authenticate the role avoiding expiration in contrast to certificates which currently expires as mentioned.
For those asking the same question and upvoting.
This is my current sollution:
For some time I treated key-file.json as an identity token, put it to the CI config and used it within container with gcloud CLI installed. I used the key file/token to log in to GCP and let gcloud generate kubectl config - the same approach used for GCP container registry login.
This works fine but using kubectl in CI is kind of antipattern. I switched to deploying based on container registry push events. This is relatively easy to do in k8s with keel flux, etc. So CI has only to push Docker image to the repo and its job ends there. The rest is taken care of within k8s itself so there is no need for kubectl and it's config in the CI jobs.
I am working on the evolution of a siem soc, and my actual issue is to recover my logs from my storage account on Azure to a Elasticsearch-data deployed on a pod on KUBERNETES. So I would like to know the the right approach for that.
With filebeat there is no input for azureblod, only output
Logstash seems don't work without agent.
Thanks !
There is an approach you could consider for accomplishing your task. Kubernetes in Azure allows you to deploy Azure File Shares with your containers. If you move your logs to a file share, you should be able to accomplish your project.
I'd recommend checking Diego's post here, it shows how to access logs from a storage account, specifically FileShare.
here's a blurb from the tutorial:
1- *Create an Azure Storage account with your own parameters (deployment model: resource manager; type: general purpose). You will need the Azure Storage account name in the next step.
2- Modify the storageAccount parameter in this .yaml file with your Azure Storage account name and deploy it to your Kubernetes cluster: kubectl apply -f sidecar-storageclass.yaml. It will create a Kubernetes volume using your Azure File Storage account.
3- Deploy this .yaml file to your Kubernetes cluster: kubectl apply -f sidecar-pvc.yaml. It will create a volume claim for your volume in order to use it in your pod.
4- Modify your application deployment .yaml file by adding (modify the logFileDirectory parameter) this content and deploy it to your Kubernetes cluster. It will add the volume to your pod and store on it the logFilesDirectory.
5- Modify the logReaderName (you will filter the logs using this parameter), logFileDirectory (x2) and the logFileName with your data in this .yaml file and deploy it to your Kubernetes cluster: kubectl apply -f sidecar-logreaderpod.yaml. It will create the Log Reader pod and write the logFile content to the STDOUT.
The Log Reader pod uses tail command to write in the STDOUT. You can modify the tail command, for example, to write different files (extension .log) in the same STDOUT: tail -n+1 -f //*.log
Once you deploy the Log Reader, you can start to check the logs filtered by the pod name (you selected it when you deployed the last .yaml file):
kubectl get pods
kubectl logs <podname>
I am not able to get write access to a GCS bucket from within a GKE pod.
I have a GKE pod running. I have not changed any k8s configuration regarding service accounts. I have docker exec'd into the pod and installed gcloud/gsutil. gcloud auth list shows a 1234-compute#developer.gserviceaccount.com entry. From within GCS I have added that same account as storage admin, storage legacy bucket owner, storage object creator (i.e., I just tried a bunch of stuff). I am able to run gsutil ls gs://bucket. However when running gsutil cp file gs://bucket, it prints:
AccessDeniedException: 403 Insufficient OAuth2 scope to perform this operation.
Acceptable scopes: https://www.googleapis.com/auth/cloud-platform
gsutil acl get gs://bucket prints:
AccessDeniedException: Access denied. Please ensure you have OWNER permission on gs://bucket
Other things I have tried are adding the allUsers and allAuthenticatedUsers as creators and owners of the bucket, with no change. I am able to write to the bucket from my dev machine just fine.
When I run gsutil acl get gs://bucket from another machine, it prints the same address as an OWNER as the output from gcloud auth list from within the pod.
What is the special sauce I need to allow the pod to write to the bucket?
You need to set permissions for cluster (or better for particular node in case of Terraform):
oauth_scopes = [
"https://www.googleapis.com/auth/devstorage.read_write", // 'ere we go!
"https://www.googleapis.com/auth/logging.write",
"https://www.googleapis.com/auth/monitoring",
"https://www.googleapis.com/auth/service.management.readonly",
"https://www.googleapis.com/auth/servicecontrol",
"https://www.googleapis.com/auth/trace.append",
"https://www.googleapis.com/auth/compute",
]
The GKE cluster was created with default permissions, which only has read scope to GCS. Solutions:
Apply advice from Changing Permissions of Google Container Engine Cluster
Set GOOGLE_APPLICATION_CREDENTIALS as described in https://developers.google.com/identity/protocols/application-default-credentials
Had the same issue, I had to recreated a node pool with custom security config in order to get that access.
Also, in my pod I mounted the SA provided in a secret (default-token-XXXXX)
Then, once gcloud is installed in the pod (via docker file) works like a charm.
The key is the node-pool config and mounting the SA.