We have setup a GKE cluster using Terraform with private and shared networking:
Network configuration:
resource "google_compute_subnetwork" "int_kube02" {
name = "int-kube02"
region = var.region
project = "infrastructure"
network = "projects/infrastructure/global/networks/net-10-23-0-0-16"
ip_cidr_range = "10.23.5.0/24"
secondary_ip_range {
range_name = "pods"
ip_cidr_range = "10.60.0.0/14" # 10.60 - 10.63
}
secondary_ip_range {
range_name = "services"
ip_cidr_range = "10.56.0.0/16"
}
}
Cluster configuration:
resource "google_container_cluster" "gke_kube02" {
name = "kube02"
location = var.region
initial_node_count = var.gke_kube02_num_nodes
network = "projects/ninfrastructure/global/networks/net-10-23-0-0-16"
subnetwork = "projects/infrastructure/regions/europe-west3/subnetworks/int-kube02"
master_authorized_networks_config {
cidr_blocks {
display_name = "admin vpn"
cidr_block = "10.42.255.0/24"
}
cidr_blocks {
display_name = "monitoring server"
cidr_block = "10.42.4.33/32"
}
cidr_blocks {
display_name = "cluster nodes"
cidr_block = "10.23.5.0/24"
}
}
ip_allocation_policy {
cluster_secondary_range_name = "pods"
services_secondary_range_name = "services"
}
private_cluster_config {
enable_private_nodes = true
enable_private_endpoint = true
master_ipv4_cidr_block = "192.168.23.0/28"
}
node_config {
machine_type = "e2-highcpu-2"
tags = ["kube-no-external-ip"]
metadata = {
disable-legacy-endpoints = true
}
oauth_scopes = [
"https://www.googleapis.com/auth/logging.write",
"https://www.googleapis.com/auth/monitoring",
]
}
}
The cluster is online and running fine. If I connect to one of the worker nodes i can reach the api using curl:
curl -k https://192.168.23.2
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
"status": "Failure",
"message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
"reason": "Forbidden",
"details": {
},
"code": 403
}
I also see a healthy cluster when using a SSH port forward:
❯ k get pods --all-namespaces --insecure-skip-tls-verify=true
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system event-exporter-gke-5479fd58c8-mv24r 2/2 Running 0 4h44m
kube-system fluentbit-gke-ckkwh 2/2 Running 0 4h44m
kube-system fluentbit-gke-lblkz 2/2 Running 0 4h44m
kube-system fluentbit-gke-zglv2 2/2 Running 4 4h44m
kube-system gke-metrics-agent-j72d9 1/1 Running 0 4h44m
kube-system gke-metrics-agent-ttrzk 1/1 Running 0 4h44m
kube-system gke-metrics-agent-wbqgc 1/1 Running 0 4h44m
kube-system kube-dns-697dc8fc8b-rbf5b 4/4 Running 5 4h44m
kube-system kube-dns-697dc8fc8b-vnqb4 4/4 Running 1 4h44m
kube-system kube-dns-autoscaler-844c9d9448-f6sqw 1/1 Running 0 4h44m
kube-system kube-proxy-gke-kube02-default-pool-2bf58182-xgp7 1/1 Running 0 4h43m
kube-system kube-proxy-gke-kube02-default-pool-707f5d51-s4xw 1/1 Running 0 4h43m
kube-system kube-proxy-gke-kube02-default-pool-bd2c130d-c67h 1/1 Running 0 4h43m
kube-system l7-default-backend-6654b9bccb-mw6bp 1/1 Running 0 4h44m
kube-system metrics-server-v0.4.4-857776bc9c-sq9kd 2/2 Running 0 4h43m
kube-system pdcsi-node-5zlb7 2/2 Running 0 4h44m
kube-system pdcsi-node-kn2zb 2/2 Running 0 4h44m
kube-system pdcsi-node-swhp9 2/2 Running 0 4h44m
So far so good. Then I setup the Cloud Router to announce the 192.168.23.0/28 network. This was successful and replicated to our local site using BGP. Running show route 192.168.23.2 displays the correct route is advertised and installed.
When trying to reach the API from the monitoring server 10.42.4.33 I just run into timeouts. All three, the Cloud VPN, the Cloud Router and the Kubernetes Cluster run in europe-west3.
When i try to ping one of the workers its working completely fine, so networking in general works:
[me#monitoring ~]$ ping 10.23.5.216
PING 10.23.5.216 (10.23.5.216) 56(84) bytes of data.
64 bytes from 10.23.5.216: icmp_seq=1 ttl=63 time=8.21 ms
64 bytes from 10.23.5.216: icmp_seq=2 ttl=63 time=7.70 ms
64 bytes from 10.23.5.216: icmp_seq=3 ttl=63 time=5.41 ms
64 bytes from 10.23.5.216: icmp_seq=4 ttl=63 time=7.98 ms
Googles Documentation gives no hit what could be missing. From what I understand the Cluster API should be reachable by now.
What could be missing and why is the API not reachable via VPN?
I have been missing the peering configuration documented here:
https://cloud.google.com/kubernetes-engine/docs/how-to/private-clusters#cp-on-prem-routing
resource "google_compute_network_peering_routes_config" "peer_kube02" {
peering = google_container_cluster.gke_kube02.private_cluster_config[0].peering_name
project = "infrastructure"
network = "net-10-13-0-0-16"
export_custom_routes = true
import_custom_routes = false
}
Related
I've been trying to apply a terraform project but I have been receiving this weird error I cannot find a solution to.
the deployment section:
############# the deployment section #############
resource "kubernetes_deployment" "dep_apps" {
for_each = var.apps
metadata {
name = each.value.appName
namespace = each.value.appName
labels = {
name = each.value.labels.name
tier = each.value.labels.tier
}
}
spec {
replicas = 3
strategy {
type = "RollingUpdate"
rolling_update {
max_surge = "25%"
max_unavailable = "25%"
}
}
selector {
match_labels = {
name = each.value.labels.name
tier = each.value.labels.tier
}
}
template {
metadata {
name = each.value.appName
namespace = each.value.appName
labels = {
name = each.value.labels.name
tier = each.value.labels.tier
}
}
spec {
container {
name = each.value.appName
image = each.value.image
resources {
limits = {
cpu = "500m"
memory = "512Mi"
}
requests = {
cpu = "200m"
memory = "256Mi"
}
}
}
}
}
}
}
############# the replica section #############
resource "kubernetes_horizontal_pod_autoscaler_v1" "autoscaler" {
for_each = var.apps
metadata {
name = "${each.value.appName}-as"
namespace = each.value.appName
labels = {
name = each.value.labels.name
tier = each.value.labels.tier
}
}
spec {
scale_target_ref {
api_version = "apps/v1"
kind = "ReplicaSet"
name = each.value.appName
}
min_replicas = 1
max_replicas = 10
}
}
############# the loadbalancer section #############
resource "kubernetes_service" "load_balancer" {
for_each = toset([for app in var.apps : app if app.appName == ["app1", "app2"]])
metadata {
name = "${each.value.appName}-lb"
labels = {
name = each.value.labels.name
tier = each.value.labels.tier
}
}
spec {
selector = {
app = each.value.appName
}
port {
name = "http"
port = 80
target_port = 8080
}
type = "LoadBalancer"
}
}
the variables section:
variable "apps" {
type = map(object({
appName = string
team = string
labels = map(string)
annotations = map(string)
data = map(string)
image = string
}))
default = {
"app1" = {
appName = "app1"
team = "frontend"
image = "nxinx"
labels = {
"name" = "stream-frontend"
"tier" = "web"
"owner" = "product"
}
annotations = {
"serviceClass" = "web-frontend"
"loadBalancer_and_class" = "external"
}
data = {
"aclName" = "acl_frontend"
"ingress" = "stream-frontend"
"egress" = "0.0.0.0/0"
"port" = "8080"
"protocol" = "TCP"
}
}
"app2" = {
appName = "app2"
team = "backend"
image = "nginx:dev"
labels = {
"name" = "stream-frontend"
"tier" = "web"
"owner" = "product"
}
annotations = {
"serviceClass" = "web-frontend"
"loadBalancer_and_class" = "external"
}
data = {
"aclName" = "acl_backend"
"ingress" = "stream-backend"
"egress" = "0.0.0.0/0"
"port" = "8080"
"protocol" = "TCP"
}
}
"app3" = {
appName = "app3"
team = "database"
image = "Mongo"
labels = {
"name" = "stream-database"
"tier" = "shared"
"owner" = "product"
}
annotations = {
"serviceClass" = "disabled"
"loadBalancer_and_class" = "disabled"
}
data = {
"aclName" = "acl_database"
"ingress" = "stream-database"
"egress" = "172.17.0.0/24"
"port" = "27017"
"protocol" = "TCP"
}
}
}
}
been trying to add mode maybe cpu utilization but didn't work
(before I added the error was on all 3 - app1,2,3)
type = "RollingUpdate"
rolling_update {
max_surge = "25%"
max_unavailable = "25%"
}
}
some errors I've received and kubectl data:
kubectl get deployments ->
NAME READY UP-TO-DATE AVAILABLE AGE
app1 0/3 3 0 34m
kubectl rollout status deployment app1 ->
error: deployment "app1" exceeded its progress deadline
kubectl get namespace ->
NAME STATUS AGE
app1 Active 6h16m
app2 Active 6h16m
app3 Active 6h16m
default Active 4d15h
kube-node-lease Active 4d15h
kube-public Active 4d15h
kube-system Active 4d15h
kubectl get pods ->
NAME READY STATUS RESTARTS AGE
app1-7f8657489c-579lm 0/1 Pending 0 37m
app1-7f8657489c-jjppq 0/1 ImagePullBackOff 0 37m
app1-7f8657489c-lv49l 0/1 ImagePullBackOff 0 37m
kubectl get pods --namespace=app2
NAME READY STATUS RESTARTS AGE
app2-68b6b59584-86dt8 0/1 ImagePullBackOff 0 38m
app2-68b6b59584-8kr2p 0/1 Pending 0 38m
app2-68b6b59584-jzzxt 0/1 Pending 0 38m
kubectl get pods --namespace=app3
NAME READY STATUS RESTARTS AGE
app3-5f589dc88d-gwn2n 0/1 InvalidImageName 0 39m
app3-5f589dc88d-pzhzw 0/1 InvalidImageName 0 39m
app3-5f589dc88d-vx452 0/1 InvalidImageName 0 39m
The errors are obvious: ImagePullBackOff and InvalidImageName. Unless you have copy+pasted and edited the terraform variable values, then you have a typo in the image name for all three apps:
app1: image = "nxinx"
app2: image = "nginx:dev"
app3: image = "Mongo"
The first one should sort itself out if you fix it from nxinx to nginx. The second one will not work, as there are no image tags dev for the nginx image. Last, but not the least, the Mongo image should be mongo (lowercase). It is important to understand that you cannot use arbitrary image names. If you do not specify a container registry, it will default to using Docker Hub in most cases. Since that is the case, you have to make sure you are using the expected naming convention for the images. Additionally, it is also important to understand how to debug things in Kubernetes. If you had just tried reading the logs of any of the Pods (but you should do it anyway for all of them) you would have probably seen the error yourself. For example:
kubectl get pod app1-7f8657489c-579lm -n app1
I want to create a controller and listen to the pod events when new pod is created (by a deployment) then add all labels belong to deployment to the created pod, is this possible at scale with client-go?
In order to observe pod events, you need to use informers. Informers have built-in optimizations to avoid overloading API servers.
There is a patch method available in the PodInterface that allows you to add a label to a pod.
Here is a sample code for your reference. In the main function, the informer code is added, and the LabelPod function implements the label logic.
package main
import (
"context"
"encoding/json"
"fmt"
"time"
v1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/labels"
"k8s.io/apimachinery/pkg/types"
"k8s.io/apimachinery/pkg/util/wait"
"k8s.io/client-go/informers"
"k8s.io/client-go/kubernetes"
"k8s.io/client-go/rest"
"k8s.io/client-go/tools/cache"
)
type patchStringValue struct {
Op string `json:"op"`
Path string `json:"path"`
Value string `json:"value"`
}
func main() {
clientSet := GetK8sClient()
labelOptions := informers.WithTweakListOptions(func(opts *metav1.ListOptions) {
opts.LabelSelector = GetLabelSelectorForDeployment("deployment-name", "namespace-name")
})
informers := informers.NewSharedInformerFactoryWithOptions(clientSet, 10*time.Second, informers.WithNamespace("namespace-name"), labelOptions)
podInformer := informers.Core().V1().Pods()
podInformer.Informer().AddEventHandler(
cache.ResourceEventHandlerFuncs{
AddFunc: handleAdd,
},
)
informers.Start(wait.NeverStop)
informers.WaitForCacheSync(wait.NeverStop)
}
func GetLabelSelectorForDeployment(Name string, Namespace string) string {
clientSet := GetK8sClient()
k8sClient := clientSet.AppsV1()
deployment, _ := k8sClient.Deployments(Namespace).Get(context.Background(), Name, metav1.GetOptions{})
labelSet := labels.Set(deployment.Spec.Selector.MatchLabels)
return string(labelSet.AsSelector().String())
}
func handleAdd(obj interface{}) {
k8sClient := GetK8sClient().CoreV1()
pod := obj.(*v1.Pod)
fmt.Println("Pod", pod.GetName(), pod.Spec.NodeName, pod.Spec.Containers)
payload := []patchStringValue{{
Op: "replace",
Path: "/metadata/labels/testLabel",
Value: "testValue",
}}
payloadBytes, _ := json.Marshal(payload)
_, updateErr := k8sClient.Pods(pod.GetNamespace()).Patch(context.Background(), pod.GetName(), types.JSONPatchType, payloadBytes, metav1.PatchOptions{})
if updateErr == nil {
fmt.Println(fmt.Sprintf("Pod %s labelled successfully.", pod.GetName()))
} else {
fmt.Println(updateErr)
}
}
func GetK8sClient() *kubernetes.Clientset {
config, err := rest.InClusterConfig()
if err != nil {
panic(err.Error())
}
// creates the clientset
clientset, err := kubernetes.NewForConfig(config)
if err != nil {
panic(err.Error())
}
return clientset
}
What about this??
Here are the pods before they are labeled manually:
❯ kubectl get po --show-labels
NAME READY STATUS RESTARTS AGE LABELS
nginx-deploy-6bdc4445fd-5qlhg 1/1 Running 0 13h app=nginx,pod-template-hash=6bdc4445fd
nginx-deploy-6bdc4445fd-pgkhb 1/1 Running 0 13h app=nginx,pod-template-hash=6bdc4445fd
nginx-deploy-6bdc4445fd-xdz59 1/1 Running 0 13h app=nginx,pod-template-hash=6bdc4445fd
I, here, label the pod "nginx-deploy-6bdc4445fd-5qlhg" with the label "test=2":
❯ kubectl label pod nginx-deploy-6bdc4445fd-5qlhg test=2
pod/nginx-deploy-6bdc4445fd-5qlhg labeled
Here are the pods after they are labeled manually:
❯ kubectl get po --show-labels
NAME READY STATUS RESTARTS AGE LABELS
nginx-deploy-6bdc4445fd-5qlhg 1/1 Running 0 13h app=nginx,pod-template-hash=6bdc4445fd,test=2
nginx-deploy-6bdc4445fd-pgkhb 1/1 Running 0 13h app=nginx,pod-template-hash=6bdc4445fd
nginx-deploy-6bdc4445fd-xdz59 1/1 Running 0 13h app=nginx,pod-template-hash=6bdc4445fd
I'm learning Istio so I followed the instruction here
As I'm using terraform so I converted the yaml file to terraform and install istio via Helm
locals {
istio_charts_url = "https://istio-release.storage.googleapis.com/charts"
}
resource "helm_release" "istio-base" {
name = "istio-base"
repository = local.istio_charts_url
chart = "base"
namespace = "istio-system"
create_namespace = true
}
resource "helm_release" "istiod" {
name = "istiod"
repository = local.istio_charts_url
chart = "istiod"
namespace = "istio-system"
depends_on = [helm_release.istio-base]
}
resource "kubernetes_namespace" "istio-ingress" {
metadata {
labels = {
istio-injection = "enabled"
}
name = "istio-ingress"
}
}
resource "helm_release" "istio-ingress" {
repository = local.istio_charts_url
chart = "gateway"
name = "istio-ingress"
namespace = kubernetes_namespace.istio-ingress.id
depends_on = [helm_release.istiod]
set {
name = "service.type"
value = "NodePort"
}
}
and application:
### blog page frontend
resource "kubernetes_service" "blog_page" {
metadata {
name = "blog-page"
namespace = kubernetes_namespace.istio-ingress.id
}
spec {
port {
port = 5000
name = "http"
}
selector = {
app = "blog_page"
}
}
}
resource "kubernetes_deployment" "blog_page_v1" {
metadata {
name = "blog-page-v1"
namespace = kubernetes_namespace.istio-ingress.id
}
spec {
replicas = 1
selector {
match_labels = {
app = "blog_page"
version = "v1"
}
}
template {
metadata {
labels = {
app = "blog_page"
version = "v1"
}
}
spec {
container {
image = "thiv17/blog-service:v1"
name = "blog-page"
image_pull_policy = "Always"
port {
container_port = 5000
}
}
}
}
}
}
resource "kubernetes_ingress" "istio-app" {
metadata {
name = "istio-app"
namespace = kubernetes_namespace.istio-ingress.id
annotations = {
"kubernetes.io/ingress.class" = "istio"
}
}
spec {
rule {
http {
path {
path = "/*"
backend {
service_name = kubernetes_service.blog_page.metadata[0].name
service_port = kubernetes_service.blog_page.spec[0].port[0].port
}
}
}
}
}
}
I expected that I can access via the node port with the Node IP is 10.0.83.140
kubectl describe svc istio-ingress --namespace=istio-ingress
-----
Port: http2 80/TCP
TargetPort: 80/TCP
NodePort: http2 30968/TCP
Endpoints: 10.0.91.237:80
Port: https 443/TCP
kubectl get pods --selector=“app=istio-ingress” --namespace=istio-ingress --output=wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
istio-ingress-5bd77ffbdf-h25vs 1/1 Running 0 24h 10.0.91.237 ip-10-0-83-140.us-west-2.compute.internal <none> <none>
However, when I ssh to this node, even though this node is listening to the port 30968
[ec2-user#ip-10-0-83-140 ~]$ netstat -plan | grep 30968
(No info could be read for "-p": geteuid()=1000 but you should be root.)
tcp 0 0 0.0.0.0:30968 0.0.0.0:* LISTEN -
But I can't access the address http://localhost:30968
* Trying ::1:30968...
* connect to ::1 port 30968 failed: Connection refused
* Failed to connect to localhost port 30968 after 0 ms: Connection refused
* Closing connection 0
curl: (7) Failed to connect to localhost port 30968 after 0 ms: Connection refused
[ec2-user#ip-10-0-83-140 ~]$
I tried to use the public IP also (Changed Security group to public Port 30968) and even changed to use LoadBlancer as well but still did not access it successfully.
Other debug info
kubectl get pods --namespace=istio-ingress
NAME READY STATUS RESTARTS AGE
blog-api-v1-86789596cf-8rh2j 2/2 Running 0 7h58m
blog-page-v1-54d45997f8-q6h6l 2/2 Running 0 7h58m
blog-page-v2-74b6d4b7c9-bgdrm 2/2 Running 0 7h58m
istio-ingress-5bd77ffbdf-h25vs 1/1 Running 0 24h
kubectl describe ingress istio-app --namespace=istio-ingress
Name: istio-app
Labels: <none>
Namespace: istio-ingress
Address:
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
Host Path Backends
---- ---- --------
*
/* blog-page:5000 (10.0.81.70:5000,10.0.95.8:5000)
Annotations: kubernetes.io/ingress.class: istio
Events: <none>
Full code:
https://gitlab.com/jimmy-pet-projects/terraform-eks-with-monitoring/-/blob/main/modules/kubernetes/istio.tf
https://gitlab.com/jimmy-pet-projects/terraform-eks-with-monitoring/-/blob/main/modules/kubernetes/istio_app.tf (edit
I found the issue: The name of helm should be istio-ingressgateway. I don't understand its document is using istio-ingress
$ helm install istio-ingress istio/gateway -n istio-ingress --wait
For all those who are facing issues with the Istio template. Here is the working template for the same. Since I faced a couple of issues with that template, I compiled it for my own use case. I hope it's helpful.
provider "helm" {
kubernetes {
config_path = "~/.kube/config"
}
}
provider "kubernetes" {
config_path = "~/.kube/config"
}
locals {
istio_charts_url = "https://istio-release.storage.googleapis.com/charts"
}
resource "kubernetes_namespace" "istio_system" {
metadata {
name = "istio-system"
labels = {
istio-injection = "enabled"
}
}
}
resource "helm_release" "istio-base" {
repository = local.istio_charts_url
chart = "base"
name = "istio-base"
namespace = kubernetes_namespace.istio_system.metadata.0.name
version = ">= 1.12.1"
timeout = 120
cleanup_on_fail = true
force_update = false
}
resource "helm_release" "istiod" {
repository = local.istio_charts_url
chart = "istiod"
name = "istiod"
namespace = kubernetes_namespace.istio_system.metadata.0.name
version = ">= 1.12.1"
timeout = 120
cleanup_on_fail = true
force_update = false
set {
name = "meshConfig.accessLogFile"
value = "/dev/stdout"
}
depends_on = [helm_release.istio-base]
}
resource "helm_release" "istio-ingress" {
repository = local.istio_charts_url
chart = "gateway"
name = "istio-ingress"
namespace = kubernetes_namespace.istio_system.metadata.0.name
version = ">= 1.12.1"
timeout = 500
cleanup_on_fail = true
force_update = false
depends_on = [helm_release.istiod]
}
I'm using Terraform helm_release resource to install bitnami/redis instance in my K8s cluster.
The code looks like this:
resource "helm_release" "redis-chart" {
name = "redis-${var.env}"
repository = "https://charts.bitnami.com/bitnami"
chart = "redis"
namespace = "redis"
create_namespace = true
set {
name = "auth.enabled"
value = "false"
}
set {
name = "master.containerPort"
value = "6379"
}
set {
name = "replica.replicaCount"
value = "2"
}
}
It completes successfully and in a separate directory I have my app terraform configuration.
In the app configuration, I want to get get the redis host from the above helm_release.
I'm doing it like this:
data "kubernetes_service" "redis-master" {
metadata {
name = "redis-${var.env}-master"
namespace = "redis"
}
}
And then in the kubernetes_secret resource I'm passing the data to my app deployment:
resource "kubernetes_secret" "questo-server-secrets" {
metadata {
name = "questo-server-secrets-${var.env}"
namespace = kubernetes_namespace.app-namespace.metadata.0.name
}
data = {
REDIS_HOST = data.kubernetes_service.redis-master.metadata.0.name
REDIS_PORT = data.kubernetes_service.redis-master.spec.0.port.0.port
}
}
But unfortunately, when I run the app deployment I'm getting the following logs:
[ioredis] Unhandled error event: Error: getaddrinfo ENOTFOUND
redis-dev-master
at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:66:26) [ioredis] Unhandled error event: Error: getaddrinfo ENOTFOUND
Which suggests that redis-dev-master is not a correct host for the redis instance.
How do I get the redis host from helm_release or any underlying services the release creates?
I've tried debugging and pinging a specific pod from my app deployment.
For reference, these are my redis resources:
NAME READY STATUS RESTARTS AGE
pod/redis-dev-master-0 1/1 Running 0 34m
pod/redis-dev-replicas-0 1/1 Running 1 34m
pod/redis-dev-replicas-1 1/1 Running 0 32m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/redis-dev-headless ClusterIP None <none> 6379/TCP 34m
service/redis-dev-master ClusterIP 172.20.215.60 <none> 6379/TCP 34m
service/redis-dev-replicas ClusterIP 172.20.117.134 <none> 6379/TCP 34m
NAME READY AGE
statefulset.apps/redis-dev-master 1/1 34m
statefulset.apps/redis-dev-replicas 2/2 34m
Maybe you were just using the wrong attribute to get that information. Checking the documentation at the Terraform Registry Website we can use the
cluster_ip attribute as described at the spec documentation description.
So you should end up with something like:
data.kubernetes_service.redis-master.spec.cluster_ip
And finally ending up with the following:
resource "kubernetes_secret" "questo-server-secrets" {
metadata {
name = "questo-server-secrets-${var.env}"
namespace = kubernetes_namespace.app-namespace.metadata.0.name
}
data = {
REDIS_HOST = data.kubernetes_service.redis-master.spec.cluster_ip
REDIS_PORT = data.kubernetes_service.redis-master.spec.port
}
}
I'm checking out with Seldon Core on Minikube and have successfully deployed a model on a cluster.
I tested with below code:
seldon-core-api-tester ../seldon-core/examples/models/keras_mnist/contract.json `minikube ip` `kubectl get svc -l app=seldon-apiserver-container-app -o jsonpath='{.items[0].spec.ports[0].nodePort}'` --oauth-key oauth-key --oauth-secret oauth-secret -p
and got the right prediction result looking like this.
RECEIVED RESPONSE:
meta {
puid: "gn83vb2ag419k547eqkhfduos2"
requestPath {
key: "mnist"
value: "mnist:0.1"
}
}
data {
names: "t:0"
names: "t:1"
names: "t:2"
names: "t:3"
names: "t:4"
names: "t:5"
names: "t:6"
names: "t:7"
names: "t:8"
names: "t:9"
ndarray {
values {
list_value {
values {
number_value: 0.00026227490161545575
}
values {
number_value: 0.0007252057548612356
}
values {
number_value: 0.028986405581235886
}
values {
number_value: 0.8030332922935486
}
values {
number_value: 7.914198795333505e-05
}
values {
number_value: 0.14541368186473846
}
values {
number_value: 0.002676495350897312
}
values {
number_value: 0.015001941472291946
}
values {
number_value: 0.0034872409887611866
}
values {
number_value: 0.00033424459979869425
}
}
}
}
}
However, when I was trying to use the python client,
from seldon_core.seldon_client import SeldonClient
sc = SeldonClient(deployment_name="mnist",namespace="seldon", seldon_rest_endpoint= '127.0.0.1:30790')
r = sc.predict(transport="rest")
I got this error.
HTTPConnection object at 0xb2bb5a780>: Failed to establish a new connection: [Errno 61] Connection refused'))
Could someone help me find out what's wrong?
$kubectl get svc
mnist-deployment-mnist ClusterIP 10.99.10.81 <none> 8000/TCP,5001/TCP 2d22h
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 4d22h
seldon-core-redis-master ClusterIP 10.107.217.176 <none> 6379/TCP 2d22h
seldon-core-seldon-apiserver NodePort 10.106.34.6 <none> 8080:30790/TCP,5000:31866/TCP 2d22h
seldon-mnist-0-1-4249605 ClusterIP 10.101.205.227 <none> 9000/TCP 2d22h
When you run the seldon-core-api-tester script, you provide minikube ip as an argument (along with the ambassador port). You'll need this address for the endpoint when you initialize the client instead of 127.0.0.1. So first run in your shell
minikube ip
and take a note of the ip, then find the ambassador port
kubectl get svc ambassador -o jsonpath='{.spec.ports[0].nodePort}'
then your client and call will look sth like this
from seldon_core.seldon_client import SeldonClient
import numpy as np
# this is the ip from `minikube ip` and port from `kubectl get svc ambassador -o jsonpath='{.spec.ports[0].nodePort}'`
minikube_ambassador_endpoint = "192.168.99.108:32667"
deployment_name = "mnist"
namespace = "default"
sc = SeldonClient(
gateway="ambassador",
gateway_endpoint=minikube_ambassador_endpoint,
transport="rest",
deployment_name=deployment_name,
namespace=namespace
)
response = sc.predict(
data=np.ones((5,)),
deployment_name=deployment_name,
payload_type="ndarray"
)
print(response)