I am trying to do some experiments with Kubernetes in google cloud.
I have docker image in google cloud registry and need to deploy that image to a kubernetes cluster.
Here are the steps I need to perform.
Create a Kubernetes cluster.
Copy the image from GCR and deploy to Kubernetes cluster.
Expose the cluster to internet via load balancer.
I know, it is possible to do via google cloud sdk cli. Is there way to do these steps via Java/node js?
There is a RESTful kubernetes-engine API:
https://cloud.google.com/kubernetes-engine/docs/reference/api-organization
e.g. create a cluster:
https://cloud.google.com/kubernetes-engine/docs/reference/rest/v1beta1/projects.zones.clusters/create
The container registry should be standard docker APIs.
Both Java and Node have kubernetes clients:
https://github.com/kubernetes-client/java
https://github.com/godaddy/kubernetes-client
Related
I have two kubernetes clusters running on Azure AKS.
One cluster named APP-Cluster which is hosting application pods.
One cluster named Vault-Cluster which the Hashicorp Vault is installed on.
I have installed Hashicorp Vault with Consul in HA mode according to below official document. The installation is successful.
https://learn.hashicorp.com/tutorials/vault/kubernetes-minikube?in=vault/kubernetes
But I am quite lost on how to connect and retrieve the secrets in Vault cluster from another cluster. I would like to use the sidecar injection method of Vault for my app cluster to communicate with vault cluster. I tried the follow the steps in below official document but in the document minikube is used instead of public cloud Kubernetes Service. How do I define the "EXTERNAL_VAULT_ADDR" variable for AKS like described in the document for minikube? Is it the api server DNS address which I can get from Azure portal?
https://learn.hashicorp.com/tutorials/vault/kubernetes-external-vault?in=vault/kubernetes
The way you interact with Vault is via HTTP(s) API. That means you need to expose the vault service running in your Vault-Cluster cluster using one of the usual methods.
As an example you could:
use a service of type LoadBalancer (this works because you are running kubernetes in a cloud provider that supports this feature);
install an ingress controller, expose it (again with a load balancer) and define an Ingress resource for your vault service.
use a node port service
The EXTERNAL_VAULT_ADDR value depends on which strategy you want to use.
I have a question around K8S Service Type "LoadBalancer".
I am working on developing a new "Kubernetes As a Service" Platform (like GKE etc.) for multi cloud.
Question is: K8S Service Type "LoadBalancer" works with Cloud Load Balancers (which are external to Kubernetes). GKE & other cloud based solution provides direct integration with them, so If I create a GKE Cluster & implement a Service Type "LoadBalancer", it will transparently create a new GCP Load Balancer & show Load Balancer IP in Kubernetes (as External IP). Same applies to other Cloud Providers also.
I want to allow a similar feature on my new "Kubernetes As a Service" platform, where users can choose a cloud provider, create a Kubernetes Cluster & then apply a K8S Service Type "LoadBalancer" & this will result creating a Load Balancer on the (user selected) cloud platform.
I am able to automate the flow till Kubernetes Cluster Creation, but clueless when it comes to "K8S Service & External Load Balancer" Integration.
Can anyone please help me how can I approach integrating K8S Service Type "LoadBalancer" with Specific Cloud Load Balancers? Do I need to write a new CRD or is there any similar code available in Git (in case anyone know any link for reference) ?
You have to understand how kubernetes is interacting with cloud provider. Like for example previously I deployed the Kubernetes on AWS with kops. I see that kubernetes uses aws access key & access secret to interact with aws. If I remember correctly, I saw some CLI options in kube-proxy or kubelet to support AWS. (I have searched man pages for all kubernetes binaries for aws options, but I couldn't find any to provide to you).
For example look at the kubelet man page, they provided an option called --google-json-key to authenticate GCP. You will get some idea if you deploy kubernetes on AWS with kops or kube-aws and dig through the setup and its configuration/options etc.(Same applies to other cloud providers)
Kubernetes is unable to launch container using image from private gcr.io container registry.
The error says "ImagePullBackOff".
Both Kubernetes and Container registry are in the same Google Cloud project.
The issue was with permissions.
It turns out that a service account that is used to launch Kubernetes needs to have reading permissions for Google Cloud Storage (this is important as the registry itself is using buckets to store images)
Exact details here
I am a new Dataproc user and I am trying to run a PySpark job that is supposed to use the MongoDB connector to retrieve data from a MongoDB replicaset hosted within a Googke Kubernetes Engine cluster.
Is it there a way to achieve this as my replicaset is not supposed to be accessible from the outside without using a port-forward or something?
In this case I assume by saying "outside" you're pointing to the internet or other networks than your GKE cluster's. If you deploy your Dataproc cluster on the same network as your GKE cluster, and expose the MongoDB service to the internal network, you should be able to connect to the databases from your Dataproc job without needing to expose it to outside of the network.
You can find more information in this link to know how to create a Cloud Dataproc cluster with internal IP addresses.
Just expose your Mogodb service in GKE and your should be able to reach it from within the same VPC network.
Take a look at this post for reference.
You should also be able to automate the service exposure through an init script
I want to deploy my application in cloud using Kubernetes based deployment. It consits of 3 layers Kafka, Ignite(as DB and processing) and Python(ML engine).
From Kafka layer we get data stream input which is then passed to Ignite for processing(feature engg). After processing the data is passed to the python
server for further ML predictions. How can I break this monolith application to microservices in Kubernetes?
Also can using Istio provide some advantage?
You can use the bitnami/kafka on docker hub from bitnami if you want pre-build image.
Export the image to your container registry with the gcloud command.
gcloud docker -- push [your image container registry path]
Deploy the images using UI or gcloud command
Expose the port{2181 9092-9099} or which one is exposed in the pulled image after the deployment on kubernetes.
Here is the link of the Ignite image on Google Compute, you have just to deploy it on the kubernetes engine and expose the appropriate ports
For python you have just to Build your python app using dockerfile as ignacio suggested.
it is possible and in fact those tools are easy to deploy in Kubernetes. Firstly, you need to gain some expertise in Kubernetes basics, specially in statefulsets and persistent volumes, since Kafka and Ignite are stateful components.
To deploy a Kafka cluster in Kubernetes follow instructions form this repository: https://github.com/Yolean/kubernetes-kafka
There are other alternatives, but this is the only one I've tested in production environments.
I have not experience with Ignite, this docs provides a step-by-step guide. Maybe someone else could share other resources.
About Python, just dockerize your ML model as any other Python app. In the official docker image for Python you'll find a basic Dockerfile to do that. Once you have your docker image pushed to a registry, just create a YAML file describing the deployment and apply it to Kubernetes.
As an alternative for the last step, you can use Draft to dockerize and deploy Python code.
Good luck!