I have two kubernetes clusters running on Azure AKS.
One cluster named APP-Cluster which is hosting application pods.
One cluster named Vault-Cluster which the Hashicorp Vault is installed on.
I have installed Hashicorp Vault with Consul in HA mode according to below official document. The installation is successful.
https://learn.hashicorp.com/tutorials/vault/kubernetes-minikube?in=vault/kubernetes
But I am quite lost on how to connect and retrieve the secrets in Vault cluster from another cluster. I would like to use the sidecar injection method of Vault for my app cluster to communicate with vault cluster. I tried the follow the steps in below official document but in the document minikube is used instead of public cloud Kubernetes Service. How do I define the "EXTERNAL_VAULT_ADDR" variable for AKS like described in the document for minikube? Is it the api server DNS address which I can get from Azure portal?
https://learn.hashicorp.com/tutorials/vault/kubernetes-external-vault?in=vault/kubernetes
The way you interact with Vault is via HTTP(s) API. That means you need to expose the vault service running in your Vault-Cluster cluster using one of the usual methods.
As an example you could:
use a service of type LoadBalancer (this works because you are running kubernetes in a cloud provider that supports this feature);
install an ingress controller, expose it (again with a load balancer) and define an Ingress resource for your vault service.
use a node port service
The EXTERNAL_VAULT_ADDR value depends on which strategy you want to use.
Related
I setup a EKS cluster and integrated AWS Secrets Manager in it following the steps mentioned in https://github.com/aws/secrets-store-csi-driver-provider-aws and it worked as expected.
Now we have a requirement to integrate the AWS Secrets Manager on an on-premises k8s cluster and I am unable to follow the same steps as they seem to be explicitly for AWS EKS based clusters.
I googled around a bit and found you can call the Secrets Manager programmatically using one of the ways in https://docs.aws.amazon.com/secretsmanager/latest/userguide/asm_access.html, but this approach wont work for us.
Is there a k8s way to directly connect to AWS secrets Manager without setting up AWS-CLI and the OIDC cluster ID on the on-premises cluster?
Any help would be highly appreciated.
You can setup external OIDC providers with AWS and also setup K8s to with OIDC, but that is a lot of work.
AWS recently announced IAM Roles Anywhere which will let you use host based certificates to authenticate, but you will still have to call the Secrets Manager APIs.
If you are willing to retrieve secrets through etcd (which may store the secrets base64 encoded on the cluster) you can look at using the opensource External Secrets solution.
I have two K8S clusters setup, one on AWS EKS, the other is on GCP. I setup a rancher server which is used to manage this two clusters. I have an application (appA) which is packaged in a helm chart. The application is just a rest api server created by nodejs + express.
It is deployed to both clusters via Rancher. After deploy, appA are running in the two clusters separately.
I have another application (appB) (running outside of K8S) which is listening on a database stream and it needs to call appA (in the K8S clusters) to process the change events.
My question is how I can deploy an entry point, like nginx, which will forward the appB's requests to appA, one of the pod from the clusters should serve this request.
You can expose the appA Kubernetes service type as Loadbalancer.
You can run nginx outside of the k8s, create a upstream and add both EKS
and GKE loadbalancers urls as backend servers.
Follow the below link
https://docs.nginx.com/nginx/admin-guide/load-balancer/http-load-balancer/
I have a question around K8S Service Type "LoadBalancer".
I am working on developing a new "Kubernetes As a Service" Platform (like GKE etc.) for multi cloud.
Question is: K8S Service Type "LoadBalancer" works with Cloud Load Balancers (which are external to Kubernetes). GKE & other cloud based solution provides direct integration with them, so If I create a GKE Cluster & implement a Service Type "LoadBalancer", it will transparently create a new GCP Load Balancer & show Load Balancer IP in Kubernetes (as External IP). Same applies to other Cloud Providers also.
I want to allow a similar feature on my new "Kubernetes As a Service" platform, where users can choose a cloud provider, create a Kubernetes Cluster & then apply a K8S Service Type "LoadBalancer" & this will result creating a Load Balancer on the (user selected) cloud platform.
I am able to automate the flow till Kubernetes Cluster Creation, but clueless when it comes to "K8S Service & External Load Balancer" Integration.
Can anyone please help me how can I approach integrating K8S Service Type "LoadBalancer" with Specific Cloud Load Balancers? Do I need to write a new CRD or is there any similar code available in Git (in case anyone know any link for reference) ?
You have to understand how kubernetes is interacting with cloud provider. Like for example previously I deployed the Kubernetes on AWS with kops. I see that kubernetes uses aws access key & access secret to interact with aws. If I remember correctly, I saw some CLI options in kube-proxy or kubelet to support AWS. (I have searched man pages for all kubernetes binaries for aws options, but I couldn't find any to provide to you).
For example look at the kubelet man page, they provided an option called --google-json-key to authenticate GCP. You will get some idea if you deploy kubernetes on AWS with kops or kube-aws and dig through the setup and its configuration/options etc.(Same applies to other cloud providers)
I am new to Kubernetes and trying to move from VM based services to Kubernetes.
Current approach,
Have multiple VM's and running services on each VM. Services are running on multiple VM's and have VIP in front of them. Clients will be accessing VIP and VIP will be doing round robin on available services.
I read ISTIO and ingress and hope, the same thing can be done using ISTIO. I have setup a local minikube cluster and exploring all the use cases. I was able to deploy my service with scaling factor 2. Now, I would like to access my service using VIP. I was not sure how to create VIP and expose to other service in the Kubernetes cluster and services running outside the Kubernetes cluster? Can i use the same existing VIP? Or, Do i need to do any extra setting create a VIP in Kubenetes with any service name?
Thanks
Please note that Istio is an additional layer on top of other frameworks, including Kubernetes. In your case you should port your application to Kubernetes first, and then add Istio if needed.
Porting to Kubernetes:
Instead of a VIP, you define a Kubernetes service. You change the code or configure your microservices to use the defined Kubernetes services instead of the VIPs.
To access your services from the outside, you define a Kubernetes Ingress.
This probably should be enough to make your application run on Kubernetes.
Once you ported your application to Kubernetes, you can add Istio, see Istio Quick Start Guide. Istio can provide you advanced routing, logging and monitoring, policy enforcement, traffic encryption between services, and also support for various microservices patterns. See more at istio.io.
We run our stack on the Google Cloud Platform (hosted Kubernetes, GKE) and have a Consul cluster running outside of K8s (regular GCE instances).
Several services running in K8s use Consul, mostly for it's CP K/V Store and advanced locking, not so much for service discovery so far.
We recently ran into some issues with using the Consul service discovery from within K8s. Right now our apps talk directly to the Consul Servers to register and unregister services they provide.
This is not recommended best-practice, usually Consul clients (i.e. apps using Consul) should talk to the local Consul agent. In our setup there are no local Consul agents.
My Question: Should we run local Consul agents as sidekick containers in each pod?
IMHO this would be a huge waste of ressources, but it would match the Consul best-practies better.
I tried searching on Google, but all posts about Consul and Kubernetes talk about running Consul in K8s, which is not what I want to do.
As the official Consul Helm chart and the documentation suggests the standard approach is to run a DaemonSet of Consul clients and then use a connect-side-car injector to inject sidecars into your node simply by providing an annotation of the pod spec. This should handle all of the boilerplate and will be inline with best practices.
Consul: Connect Sidecar; https://www.consul.io/docs/platform/k8s/connect.html