Is it possible to use a fixed host name for Dataproc component gateway? By default, Dataproc component gateway can be accessed in URLs like https://42qlunruwrdhhauh3sryzzsz5a-dot-us-central1.dataproc.googleusercontent.com/jupyter/. I noticed that it is defined as as custom metadata key dataproc-notebooks-url in compute but could not find any way to change it in Dataproc init script. Is it possible to use a fixed name e.g. https://some-fixed-name-dot-us-central1.dataproc.googleusercontent.com/jupyter/. The issue is that the name is auto-generated every time a cluster gets re-created but I would like to access notebooks in a fixed url.
The host name is encoded from the cluster URI projects/<project>/regions/<region>/clusters/<cluster>, so if you recreate the cluster in the same project, region and with the same name, you should be able to get the same component gateway URL. But clusters with different names will get different URLs, that's by design.
Related
I know that in Amazon ECS container agent by setting the variable ECS_ENABLE_CONTAINER_METADATA=true ecs metadata files are created for the containers.
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/container-metadata.html
Is there any similar feature for the EKS?. I would like to retrieve instance metadata info from a file inside the container instead of using the IMDSv2 api.
you simply can't, you still need to use IMDSv2 api in your service,if you want to have get instance metadata
if you're looking at the Pod Metadata, ref https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/
or you can use pod labels too...
Try adding this as part of the user data:
"echo 'ECS_ENABLE_CONTAINER_METADATA=1' > /etc/ecs/ecs.config"
Found here: https://github.com/aws/amazon-ecs-agent/issues/1514
I'm trying to connect to a Digital Ocean Kubernates cluster using doctl but when I run
doctl kubernetes cluster kubeconfig save <> I get an error saying .kube/config: not a directory. I've authenticated using doctl and when I run doctl account get I see my account info. I'm confused as to what the problem is. Is this some sort of permission issue or did I miss a config step somewhere?
kubectl (by default) stores a configuration in ${HOME}/.kube/config. It appears you don't have the file and the command doesn't create it if it doesn't exist; I recommend you try creating ${HOME}/.kube first as doctl really ought to create the config file if it doesn't exist.
kubectl facilitates interacting with multiple clusters as multiple users in multiple namespaces through the use a tuple called 'context' which combines a cluster with a user with a(n optional) namespace. The command lets you switch between these easily.
After you're done with a cluster, generally (!) you must tidy up its entires in ${HOME}/.kube/config too as these configs tend to grow over time.
You can change the location of the kubectl config file using an environment variable (KUBECONFIG).
See Organizing Cluster Access Using kubeconfig Files
I am trying to utilize Rancher Terraform provider to create a new RKE cluster and then use the Kubernetes and Helm Terraform providers to create/deploy resources to the created cluster. I'm using this https://registry.terraform.io/providers/rancher/rancher2/latest/docs/resources/cluster_v2#kube_config attribute to create a local file with the new cluster's kube config.
The Helm and Kubernetes providers need the kube config in the provider configuration: https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs. Is there any way I can get the provider configuration to wait for the local file being created?
Generally speaking, Terraform always needs to evaluate provider configurations during the planning step because providers are allowed to rely on those settings in order to create the plan, and so it typically isn't possible to have a provider configuration refer to something created only during the apply step.
As a way to support bootstrapping in a situation like this though, this is one situation where it can be reasonable to use the -target=... option to terraform apply, to plan and apply only sufficient actions to create the Rancher cluster first, and then follow up with a normal plan and apply to complete everything else:
terraform apply -target=rancher2_cluster_v2.example
terraform apply
This two-step process is needed only for situations where the kube_config attribute isn't known yet. As long as this resource type has convergent behavior, you should be able to use just terraform apply as normal unless you in future make a change that requires replacing the cluster.
(This is a general answer about provider configurations refering to resource attributes. I'm not familiar with Rancher in particular and so there might be some specifics about that particular resource type which I'm not mentioning here.)
I found a sort of workaround solution. I output the rancher2_cluster.cluster.kube_config object into a variable. Then referenced that variable in my Kubernetes module. Instead of using kube_config attribute in the provider configuration, I used the token and host attributes and used yamldecode to parse the creds directly from the kube_config variable.
provider "kubernetes" {
token = "${yamldecode(var.kube_config)["users"][0]["user"]["token"]}"
host = "${yamldecode(var.kube_config)["clusters"][0]["cluster"]["server"]}"
}
I will suggest to split your functionality in 2 layers
Run the fist layer to generate the kube_config file.
Run the second layer that will consume this file.
Hey i have a question.
Im using logback-more-appenders(fluency plugin) to send logs to EFK stack (fluent-bit) which is working in kubernetes cluster, but it lacks kubernetes metadata ( like node/pod names).
I know i can use <additionalField></additionalField> in logbck.xml to add Service name (because this is static), but i cannot do it to dynamic parts like node or pod name.
I tried to do it on fluent-bit side using kubernetes filter, but this works only with tail/systemd inputs not a forward one (it parses tag with filename which contains namespce and pod name). Im using forward plugin to send logs from java software to elasticsearch, and in logback.xml i cannot enter dynamic pod name (or i don't know if i can).
Any tips how i can do it? I prefer to send logs using fluency instead of sniffing host container logs.
In my case, the best i could think of was to change from forward to tail plugin with structured logging (in json).
Have you tried to Pass POD ID and NODE NAME as environment variables in logback.xml as additional fields, that you can attribute the metadata to the logevents?
I need to get a list of all namespaces in a specifc Kubernetes cluster, using the Kubernetes API. Because I need to loop through multiple clusters in my Python program, I need to specify the cluster every time I call the API.
One option is to use list_namespace(), as described in https://github.com/kubernetes-client/python/blob/master/kubernetes/docs/CoreV1Api.md
However, this API doesn't allow me to specify the cluster. It picks up the cluster from the current-context in my .kube config file. If I remove or rename the config file, the API call fails completely.
I also found an extensions API at https://github.com/kubernetes-client/python/blob/master/kubernetes/docs/ExtensionsV1beta1Api.md
Unfortunately, there is no API there to retrieve a list of namespaces. Is there some other API that I am unaware of?
If you see the source code of the kube_config module you can use different arguments with the method load_kube_config to select your cluster:
def load_kube_config(config_file=None, context=None,
client_configuration=None,
persist_config=True):
"""Loads authentication and cluster information from kube-config file
and stores them in kubernetes.client.configuration.
:param config_file: Name of the kube-config file.
:param context: set the active context. If is set to None, current_context
from config file will be used.
:param client_configuration: The kubernetes.client.Configuration to
set configs to.
:param persist_config: If True, config file will be updated when changed
(e.g GCP token refresh).
"""
If I understood the code correctly, you can do somewhat like the following:
from kubernetes import client, config
for file in files:
config.load_kube_config(config_file=file)
v1 = client.CoreV1Api()
response = v1.list_namespace()
print(response)
EDIT: This is an example that uses the context argument with a single kubeconfig file to iterate over multiple clusters. In the kubernetes docs there is an entry on Merging kubeconfig files. Basically after having a config file with multiple contexts you can load the file with config.load_kube_config(config_file=file) and load contexts with client.load_kube_config(context="context2')
P.S. You don't need to use config.load_kube_config() if you want to use a config file in the default path ('~/.kube/config') or if you set a path in the KUBECONFIG environment variable.
Would you check this example
There you can navigate between multiple contexts and list all pods within all namespaces
Apparently you just need to replace
list_pod_for_all_namespaces()
with
list_namespace()