I have used Kubernetes and I deployed for example WordPress or nginx or etc. We install from yaml file. Where is it installed how can i find directory of pages(for example WordPress pages etc.) at same point at Google Cloud too. When I use Kubernetes at Google Cloud where is the path of installed files(ex. index.php).
If you are running the docker image directly without attaching anything like NFS, S3 or Disk then you will be able to get those files by default in the container file system(index.php and all).
With any K8s cluster you check files inside container either Gcloud or any :
kubectl get pods
kubectl exec -it <Wordpress pod name> -- /bin/bash
If you are attaching the File system like NFS, or object storage S3 or EFS you will be able to watch those files there unless you mount and apply config using the YAML file.
Regarding setup file (YAML),
Kubernetes uses the ETCD database as a data store. The flow is like this. Kubectl command connect to API server and sends the YAML file to API server. API parses and store the information in ETCD database so you wont be getting those file as it is in YAML format.
Related
I was given a docker compose file for superset which included volumes mounted from the repo itself.
docker-compose-non-dev.yml
I have to deploy this as containers in a pod in an EKS cluster. I can't figure out how the volumes should be done because the files are mounted locally from the repo when we run:
docker-compose up
[ EDIT ]
I just built the container with the files I needed inside it.
Docker compose is a tool geared towards local deployments (as you may know) and so it optimizes its workflows with that assumption. One way to work this around is by wrapping the docker image(s) that compose up with the additional files you have on your local environment. For example a wrapper dockerfile would be something like
FROM <original image>
ADD <local files to new image>
The resulting image is what you would run in the cloud on EKS.
Of course there are many other ways to work it around such as using Kubernetes volumes and (pre-)populating them with the local files, or bake the local files in the original image from the get go, etc.
All in all the traditional compose model of thinking (with local file mappings) isn't very "cloud deployments friendly".
You can convert docker-compose.yaml files with a tool called kompose.
It's as easy as running
kompose convert
in a directory containing docker-ccompose.yaml file.
This will create a bunch of files which you can deploy with kubectl apply -f . (or kompose up). You can read more here.
However, even though kompose will generate PersistentVolueClaim manifests, no PersistentVolumes will be created. You have to make those yourself (cluster may try to create PVs by itself, but it's strongly based on PVCs generated by kompose, I would not rely on that).
Docker compose is mainly used for devlopment, testing and single host deployments [reference], which is not exactly what Kubernetes was created for (latter being cloud oriented).
I have a Google Cloud Composer 1 environment (Airflow 2.1.2) where I want to run an Airflow DAG that utilizes the KubernetesPodOperator.
Cloud Composer makes available to all DAGs a shared file directory for storing application data. The files in the directory reside in a Google Cloud Storage bucket managed by Composer. Composer uses FUSE to map the directory to the path /home/airflow/gcs/data on all of its Airflow worker pods.
In my DAG I run several Kubernetes pods like so:
from airflow.contrib.operators import kubernetes_pod_operator
# ...
splitter = kubernetes_pod_operator.KubernetesPodOperator(
task_id='splitter',
name='splitter',
namespace='default',
image='europe-west1-docker.pkg.dev/redacted/splitter:2.3',
cmds=["dotnet", "splitter.dll"],
)
The application code in all the pods that I run needs to read from and write to the /home/airflow/gcs/data directory. But when I run the DAG my application code is unable to access the directory. Likely this is because Composer has mapped the directory into the worker pods but does not extend this courtesy to my pods.
What do I need to do to give my pods r/w access to the /home/airflow/gcs/data directory?
Cloud Composer uses FUSE to mount certain directories from Cloud Storage into Airflow worker pods running in Kubernetes. It mounts these with default permissions that cannot be overwritten, because that metadata is not tracked by Google Cloud Storage. A possible solution is to use a bash operator that runs at the beginning of your DAG to copy files to a new directory. Another possible solution can be to use a non-Google Cloud Storage path like a /pod path.
This is my very first post here and looking for some advise please.
I am learning Kubernetes and trying to get cloud code extension to deploy Kubernetes manifests on non-GKE cluster. Guestbook app can be deployed using cloud code extension to local K8 cluster(such as MiniKube or Docker-for-Desktop).
I have two other K8 clusters as below and I cannot deploy manifests via cloud code. I am not entirely sure if this is supposed to work or not as I couldn't find any docs or posts on this. Once the GCP free trial is finished, I would want to deploy my test apps on our local onprem K8 clusters via cloud code.
3 node cluster running on CentOS VMs(built using kubeadm)
6 node cluster on GCP running on Ubuntu machines(free trial and built using Hightower way)
Skaffold is installed locally on MAC and my local $HOME/.kube/config has contexts and users set to access all 3 clusters.
➜
guestbook-1 kubectl config get-contexts
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
docker-desktop docker-desktop docker-desktop
* kubernetes-admin#kubernetes kubernetes kubernetes-admin
kubernetes-the-hard-way kubernetes-the-hard-way admin
Error:
Running: skaffold dev -v info --port-forward --rpc-http-port 57337 --filename /Users/testuser/Desktop/Cloud-Code-Builds/guestbook-1/skaffold.yaml -p cloudbuild --default-repo gcr.io/gcptrial-project
starting gRPC server on port 50051
starting gRPC HTTP server on port 57337
Skaffold &{Version:v1.19.0 ConfigVersion:skaffold/v2beta11 GitVersion: GitCommit:63949e28f40deed44c8f3c793b332191f2ef94e4 GitTreeState:dirty BuildDate:2021-01-28T17:29:26Z GoVersion:go1.14.2 Compiler:gc Platform:darwin/amd64}
applying profile: cloudbuild
no values found in profile for field TagPolicy, using original config values
Using kubectl context: kubernetes-admin#kubernetes
Loaded Skaffold defaults from \"/Users/testuser/.skaffold/config\"
Listing files to watch...
- python-guestbook-backend
watching files for artifact "python-guestbook-backend": listing files: unable to evaluate build args: reading dockerfile: open /Users/adminuser/Desktop/Cloud-Code-Builds/src/backend/Dockerfile: no such file or directory
Exited with code 1.
skaffold config file skaffold.yaml not found - check your current working directory, or try running `skaffold init`
I have the docker and skaffold file in the path as shown in the image and have authenticated the google SDK in vscode. Any help please ?!
I was able to get this working in the end. What helped in this particular case was removing skaffold.yaml, then skaffold init, generated new skaffold.yaml. And, Cloud Code was then able deploy pods on both remote clusters. Thanks for all your help.
I am using gocd for ci/cd. Result is tar archive. I need to copy resulting tar to GCE bucket.
I have gocd-agent docker image with included google sdk.
I know how to use gcloud with service account from local machine, but not from inside pod.
How to use service account assigned to pod with gcloud auth on pod?
Final goal is to use gsutil to copy above mentioned archive to bucket in same project.
My first thought would be to create Secret based on the service account, reference it in a pod yaml definiton to mount to some file and then run gcloud auth from the pod using that file. There's more info in Google cloud docs.
Another option which is quite new is to use Workload Identitiy. Seems you'd need to configure GKE cluster to enable this option. And it's working for some versions of GKE.
I am working on the evolution of a siem soc, and my actual issue is to recover my logs from my storage account on Azure to a Elasticsearch-data deployed on a pod on KUBERNETES. So I would like to know the the right approach for that.
With filebeat there is no input for azureblod, only output
Logstash seems don't work without agent.
Thanks !
There is an approach you could consider for accomplishing your task. Kubernetes in Azure allows you to deploy Azure File Shares with your containers. If you move your logs to a file share, you should be able to accomplish your project.
I'd recommend checking Diego's post here, it shows how to access logs from a storage account, specifically FileShare.
here's a blurb from the tutorial:
1- *Create an Azure Storage account with your own parameters (deployment model: resource manager; type: general purpose). You will need the Azure Storage account name in the next step.
2- Modify the storageAccount parameter in this .yaml file with your Azure Storage account name and deploy it to your Kubernetes cluster: kubectl apply -f sidecar-storageclass.yaml. It will create a Kubernetes volume using your Azure File Storage account.
3- Deploy this .yaml file to your Kubernetes cluster: kubectl apply -f sidecar-pvc.yaml. It will create a volume claim for your volume in order to use it in your pod.
4- Modify your application deployment .yaml file by adding (modify the logFileDirectory parameter) this content and deploy it to your Kubernetes cluster. It will add the volume to your pod and store on it the logFilesDirectory.
5- Modify the logReaderName (you will filter the logs using this parameter), logFileDirectory (x2) and the logFileName with your data in this .yaml file and deploy it to your Kubernetes cluster: kubectl apply -f sidecar-logreaderpod.yaml. It will create the Log Reader pod and write the logFile content to the STDOUT.
The Log Reader pod uses tail command to write in the STDOUT. You can modify the tail command, for example, to write different files (extension .log) in the same STDOUT: tail -n+1 -f //*.log
Once you deploy the Log Reader, you can start to check the logs filtered by the pod name (you selected it when you deployed the last .yaml file):
kubectl get pods
kubectl logs <podname>