I have a specific namespace that I am trying to output logs into Amazon S3 using Banzai cloud Logging operator.I Am currently using ClusterFlow and ClusterOutput but it is not working for me and not able to see the logs in S3 bucket.
Here are the images of the code that I have:
https://imgur.com/a/s6MBpML
I have tried reaching out to the Banzai Cloud Slack to no avail. Can someone help me check the yaml files and validate them?
You should be using Flow object and not ClusterFlow.
The Flow is a namespaced resource, which means logs will only be collected from the namespace that the Flow is deployed in. Where as ClusterFlow is scoped at the cluster level and can configure log collection across all namespaces.
Related
What would be the Fluentd configuration to collect logs and create a separate log file / folder path of each namespace separately ?
I want to use a Fluentd instance and have a configuration that would help me segregate and group logs of each namespace separately, and then zip them separately to be sent over http.
One starting with (kubernetes.var.log...) should contain the namespace, and therefore you can filter based on specific namespaces and decide how to handle those specific logs.
If, for any reason, the log path in your cluster does not contain the namespace in its path, you can also use the kubernetes plugin.
It will enrich your logs with metadata relevant to the cluster, and allow you to extract the namespace logs originated from and deal with them accordingly.
Refer to this for collecting logs with Fluentd ,Blog
According to GKE, I can enable cluster audit logs from the k8s.io API, which will forward cluster events to Cloud Logging. However, I'm unable to find RBAC logs for read requests on custom resources.
Specifically, if I have a CR foo, I seem to only be able to view create and delete events on foo. get and list are separate permissions as well (in both IAM and cluster RBAC), but those calls don't seem to be audited.
Is there a way to see those requests, and their responses, or is that not possible?
It's weird because the cluster's own kube-apiserver.log seems to log those requests:
... httplog.go:109] "HTTP" verb="GET" URI="/apis/foo.io/v1/namespaces/foo-ns/custom-resource/foo" latency="26.286746ms" userAgent="kubectl/v1.xx.x (linux/amd64)" audit-ID="baz" srcIP="1.2.3.4:55555" resp=200
Is there an dynamic way to pull log data from inside my containers?
All of my searches are returning that Azure Logs/Azure Sentinel can read data about AKS relative to the containers as they exist in K8s (online, running, failed, etc.) but not the actual in-container logs. Examples of results asking for this:
https://learn.microsoft.com/en-us/azure/azure-monitor/containers/container-insights-log-query
https://learn.microsoft.com/en-us/azure/azure-monitor/containers/container-insights-livedata-overview
https://github.com/MicrosoftDocs/azure-docs/blob/main/articles/azure-monitor/containers/container-insights-enable-new-cluster.md#enable-monitoring-of-a-new-azure-kubernetes-service-aks-cluster
...all of these provide documentation on monitoring containers (as they live in K8s) but not the app-level logs in the containers...
Is anyone aware of a technology or capability for Azure Logs/Azure Sentinel to consume in-container, on-disk container logs (e.g. inside the container: /var/log, /var/application/logs, etc.)?
Thanks!
Assuming you're referring to linux containers. You only need to have have the OMS agent enabled and pointing to the right workspace and this gets the logs streamed over easily.
The ContainerLog table which would show you the same thing as kubectl logs <pod>. Anything that's sent to stdout and stderr from your container should be available in the Log Analytics Workspace. So if these are not being sent to either, you could just write a small script as part of your container, that would send those logs to stdout.
Here's how I'm able to get SMTP logs from my container:
I have an azuredevops build job to get the log of a deployment pod.
command: kubectl logs deployment/myapp
I am getting the output in the summary page of azure devops pipeline, but the same I want to send a team with a log as an attachment. I am not getting any option in azure devops for that
Basically, your k8s log (pods) will gone after the pods has been terminated (although you can somehow keep it for a little while). For debug purpose or any other purpose you want, you need to Centralized logging your k8s log (use some tools: filebeat, fluentd, fluent-bit to forward your k8s log to elasticsearch).
EX: Some software (tools) for Centralized logging Elasticsearch, Graylog, ...
https://www.elastic.co/fr/what-is/elk-stack
And then you can save, export, analyze your log ... You can do anythings you want with your stored k8s log.
Hope this may help you, guy!
Edit: I use GCP as cloud solution and in GCP, by default, they will use fluentd to forward your k8s log to store in Logging. And the Logging has feature Export, I think you can search somethings similar to Logging in your cloud solution: Azure
I am using the Kubernetes Provider to describe services/pods in Terraform.
It can get confusing using the Hashicorp Configuration Language to define kubernetes_pod or kubernetes_service resources because the Kubernetes documentation describes everything in YAML which it means you need to translate it into HCL.
Is it possible to define pods as YAML and use them with kubernetes_pod and kubernetes_service resources as templates?
While Terraform normally uses HCL, this is a superset of JSON (much like YAML itself) so can also read JSON.
One possible option would be to take the YAML examples you already have and convert them into JSON and then use Terraform on those.
Unfortunately, that's unlikely to work because keywords are likely to be different for how Terraform is expecting things so you'd need to write something to do some basic translation of the input YAML to a Terraform resource JSON. At this point, it'd probably be worth just adding HCL output to the conversion so your outputted Terraform config is more readable if you ever intend to keep the Terraform config around instead of just one shot converting and applying the config.
The benefit of doing things this way would be that you have a reusable Kubernetes config that could be ran using kubectl or other tools but gives you the power of Terraform's lifecycle management, being able to plan changes and integration with non Kubernetes parts of your infrastructure (such as setting up instances to run the Kubernetes cluster on).
I've not used it much but I believe Kops will allow you to keep pod/service config in typical Kubernetes YAML files but can then use Terraform to manage the configuration and even allows you to output the Terraform configuration so you can run it outside of Kops itself.
The hashicorp/kubernetes provider does not support raw YAML/JSON, and they have no intention of implementing it.
The possible solutions are:
K2tf, a tool for converting Kubernetes RAW YAML manifests into Terraform HCL for the Kubernetes provider.
Use an alternative community Kubernetes provider, such as gavinbunny/kubectl, which does support raw YAML and can track each resource and the attributes in Terraform state, unlike the kubernetes-alpha provider.
Another solution is to use the hashicorp/kubernetes-alpha provider, you can pass in either a Terraform object or convert raw YAML manifest into a TF object for using in the provider resource. The downside is that the attributes are not tracked as individual objects and thus a change will cause the entire resource to be tainted.
Using the kubectl provider.
This core of this provider is the kubectl_manifest resource, allowing free-form yaml to be processed and applied against Kubernetes. This yaml object is then tracked and handles creation, updates and deleted seamlessly - including drift detection. This provider is ideal if you want to track the manifest in Terraform:
resource "kubectl_manifest" "test" {
yaml_body = file("path/to/manifest.yaml")
}
Using the kubernetes-alpha provider
The kubernetes_manifest represents one Kubernetes resource as described in the manifest attribute. The manifest value is the HCL transcription of a regular Kubernetes YAML manifest. To transcribe an existing manifest from YAML to HCL, use the Terrafrom built-in function yamldecode(), or use the tfk8s tool to convert YAML into manifest attributes for the kubernetes-alpha provider manifest resource.
Example using yamldecode:
resource "kubernetes_manifest" "service" {
provider = kubernetes-alpha
manifest = yamldecode(file("path/to/manifest.yaml"))
}
Why doesn't the kubernetes provider support RAW YAML?
Supporting YAML/JSON in hashicorp/kubernetes was considered before (the very first proposal of K8S provider was exactly that) and during the initial implementation of this provider and we decided not to do it.
The reason is that you can't accurately track resources created from RAW YAML as Terraform objects.
From Terraform's developer perspective it is very tricky to get around
the way K8S API works where you send an array [a, b, c] to the Create
API and then you Get back [a, b, c, d]. This happens for example with
pods that get some secret volumes attached automatically, but happens
with most other resources I had the chance to play with. The
whitelisting/blacklisting is tricky part.
You may also be interested in the following project, which allows you to convert YAML files to Terraform's HCL.
https://github.com/sl1pm4t/k2tf
Description:
A tool for converting Kubernetes API Objects (in YAML format) into HashiCorp's Terraform configuration language.
The converted .tf files are suitable for use with the Terraform Kubernetes Provider