I am working on the evolution of a siem soc, and my actual issue is to recover my logs from my storage account on Azure to a Elasticsearch-data deployed on a pod on KUBERNETES. So I would like to know the the right approach for that.
With filebeat there is no input for azureblod, only output
Logstash seems don't work without agent.
Thanks !
There is an approach you could consider for accomplishing your task. Kubernetes in Azure allows you to deploy Azure File Shares with your containers. If you move your logs to a file share, you should be able to accomplish your project.
I'd recommend checking Diego's post here, it shows how to access logs from a storage account, specifically FileShare.
here's a blurb from the tutorial:
1- *Create an Azure Storage account with your own parameters (deployment model: resource manager; type: general purpose). You will need the Azure Storage account name in the next step.
2- Modify the storageAccount parameter in this .yaml file with your Azure Storage account name and deploy it to your Kubernetes cluster: kubectl apply -f sidecar-storageclass.yaml. It will create a Kubernetes volume using your Azure File Storage account.
3- Deploy this .yaml file to your Kubernetes cluster: kubectl apply -f sidecar-pvc.yaml. It will create a volume claim for your volume in order to use it in your pod.
4- Modify your application deployment .yaml file by adding (modify the logFileDirectory parameter) this content and deploy it to your Kubernetes cluster. It will add the volume to your pod and store on it the logFilesDirectory.
5- Modify the logReaderName (you will filter the logs using this parameter), logFileDirectory (x2) and the logFileName with your data in this .yaml file and deploy it to your Kubernetes cluster: kubectl apply -f sidecar-logreaderpod.yaml. It will create the Log Reader pod and write the logFile content to the STDOUT.
The Log Reader pod uses tail command to write in the STDOUT. You can modify the tail command, for example, to write different files (extension .log) in the same STDOUT: tail -n+1 -f //*.log
Once you deploy the Log Reader, you can start to check the logs filtered by the pod name (you selected it when you deployed the last .yaml file):
kubectl get pods
kubectl logs <podname>
Related
I have used Kubernetes and I deployed for example WordPress or nginx or etc. We install from yaml file. Where is it installed how can i find directory of pages(for example WordPress pages etc.) at same point at Google Cloud too. When I use Kubernetes at Google Cloud where is the path of installed files(ex. index.php).
If you are running the docker image directly without attaching anything like NFS, S3 or Disk then you will be able to get those files by default in the container file system(index.php and all).
With any K8s cluster you check files inside container either Gcloud or any :
kubectl get pods
kubectl exec -it <Wordpress pod name> -- /bin/bash
If you are attaching the File system like NFS, or object storage S3 or EFS you will be able to watch those files there unless you mount and apply config using the YAML file.
Regarding setup file (YAML),
Kubernetes uses the ETCD database as a data store. The flow is like this. Kubectl command connect to API server and sends the YAML file to API server. API parses and store the information in ETCD database so you wont be getting those file as it is in YAML format.
I'm trying to connect to a Digital Ocean Kubernates cluster using doctl but when I run
doctl kubernetes cluster kubeconfig save <> I get an error saying .kube/config: not a directory. I've authenticated using doctl and when I run doctl account get I see my account info. I'm confused as to what the problem is. Is this some sort of permission issue or did I miss a config step somewhere?
kubectl (by default) stores a configuration in ${HOME}/.kube/config. It appears you don't have the file and the command doesn't create it if it doesn't exist; I recommend you try creating ${HOME}/.kube first as doctl really ought to create the config file if it doesn't exist.
kubectl facilitates interacting with multiple clusters as multiple users in multiple namespaces through the use a tuple called 'context' which combines a cluster with a user with a(n optional) namespace. The command lets you switch between these easily.
After you're done with a cluster, generally (!) you must tidy up its entires in ${HOME}/.kube/config too as these configs tend to grow over time.
You can change the location of the kubectl config file using an environment variable (KUBECONFIG).
See Organizing Cluster Access Using kubeconfig Files
Okay first let me say please don't judge. Believe me, I am kicking myself in the ass.
So I lost my hard disk on my laptop which held the Kubernetes yaml files that I ran against a Kubernetes Cloud cluster. I don't have the latest backup which is the problem.
does anyone know how to get just the yaml I ran against the K8s cloud server. I can get to the cluster and run kubectl get pod my-pod -o yaml but of course, it adds a lot of things. I am just looking for the yaml that I ran.
I am stressing here and have learned my lesson. Backup, Backup and verify Backup.
You can use this and extend it to your needs:
kubectl get [resource type] -n [namespace] [resource Name] -o yaml > [output.yaml]
The -o yaml will do the job
Note
You will get some extra information provided by your cloud providers like history, version, and more.
Lens
https://k8slens.dev/
You can use Lens which will allow you to view & edit your resources so you will be able to copy the YAML from it.
TLDR: My understanding from learning all about K8s is that you need lots and lots of yaml files, however, I just deployed an app to a K8s clusters with 0 yaml files and it succeeded. Why is that? Does google cloud or K8s have defaults it uses when the app does not have any yaml file settings?
Longer:
I have a dockerized spring app that I deployed to a google cloud cluster I created via the UI.
It had 0 yaml files in there, so my expectation that kubectl deploy would fail, however, it succeeded and my stateless app is up there chugging away.
How does that work?
Well the gcp created for you in the background. I assume you pushed your docker image or CI to cluster and from there you just did few clicks right? same stuff you can do it on openshift environment. but in the background yaml file get's generated. if you edit the pod on your UI you will see that yaml file.
as above #Volodymyr Bilyachat said you can create deployment via imparative way or using declarative way(yaml). I would suggest always use declarative way.
you can see your deployment yaml file which you created from UI by doing
kubectl get deployment <deployment_name> -o yaml
kubectl get deployment <deployment_name> -o yaml > name.yaml #This will output your yaml file into name.yaml file
You can run your containers/pods using plain commands.
kubectl run podname --image=name
As you said 0 yaml files. But main idea of those files that you push them to source control and run test them via different environments using CI/CD.
Other benefit of yaml files that you can share configuration and someone else will be able to create infrastructure without having to write anything. Here is example how you can run elasticsearch with one command
kubectl apply -f https://download.elastic.co/downloads/eck/1.2.0/all-in-one.yaml
I am using gocd for ci/cd. Result is tar archive. I need to copy resulting tar to GCE bucket.
I have gocd-agent docker image with included google sdk.
I know how to use gcloud with service account from local machine, but not from inside pod.
How to use service account assigned to pod with gcloud auth on pod?
Final goal is to use gsutil to copy above mentioned archive to bucket in same project.
My first thought would be to create Secret based on the service account, reference it in a pod yaml definiton to mount to some file and then run gcloud auth from the pod using that file. There's more info in Google cloud docs.
Another option which is quite new is to use Workload Identitiy. Seems you'd need to configure GKE cluster to enable this option. And it's working for some versions of GKE.