We have an app running under a Google Cloud Kubernetes cluster, things are running fine in my testing scenario. We went to set up autoscaling for these pods - we'll probably never need to go to 0, but want it to scale up to (for now) 20 pods, and back down, obviously. We are deploying using faas-cli. First, tried:
faas-cli deploy --replace --update=false -f ./process-listing-image.yml \
--gateway=https://openfaas.ihouseprd.com \
--label "com.openfaas.scale.min=1" \
--label "com.openfaas.scale.max=20" \
--label "com.openfaas.scale.factor=5"
But that gave us 1 pod, and never moved. The it was suggested to use:
faas-cli deploy --replace --update=false -f ./process-listing-image.yml \
--gateway=https://openfaas.ihouseprd.com \
--label "com.openfaas.scale.min=0" \
--label "com.openfaas.scale.max=20" \
--label "com.openfaas.scale.factor=5"
But that still gave us but one pod. I most recently tried:
faas-cli deploy --replace --update=false -f ./process-listing-image.yml \
--gateway=https://openfaas.ihouseprd.com \
--label "com.openfaas.scale.min=5" \
--label "com.openfaas.scale.max=20" \
--label "com.openfaas.scale.factor=5"
Which produced 5 pods, but it hasn't scaled past that, despite there being thousands of requests waiting. Looking at the Cloud Console "Deployment Details" screen, I see the five pods, but can't tell if all 5 are working.
Any idea why these things aren't scaling?
Related
I want to setup several GKE clusters like here. So essentially, I would first create a VPC
gcloud compute networks create ${network_name} --subnet-mode=custom
and then the subnets
gcloud compute networks subnets create ${subnet_1} \
--region=${region_1} \
--network=${network_name} \
--range=10.0.0.0/16 \
--secondary-range pods=10.10.0.0/16,services=10.100.0.0/16
gcloud compute networks subnets create ${subnet_2} \
--region=${region_2} \
--network=${network_name} \
--range=10.1.0.0/16 \
--secondary-range pods=10.11.0.0/16,services=10.101.0.0/16
gcloud compute networks subnets create ${subnet_3} \
--region=${region_3} \
--network=${network_name} \
--range=10.2.0.0/16 \
--secondary-range pods=10.12.0.0/16,services=10.102.0.0/16
and then three GKE clusters:
gcloud beta container clusters create ${cluster_1} \
--region ${region_1} --num-nodes 1 \
--network ${network_name} --subnetwork ${subnet_1} \
--cluster-dns clouddns --cluster-dns-scope vpc \
--cluster-dns-domain ${cluster_domain_1}
--enable-ip-alias \
--cluster-secondary-range-name=pods --services-secondary-range-name=services
gcloud beta container clusters create ${cluster_2} \
--region ${region_2} --num-nodes 1 \
--network ${network_name} --subnetwork ${subnet_2} \
--cluster-dns clouddns --cluster-dns-scope vpc \
--cluster-dns-domain ${cluster_domain_2}
--enable-ip-alias \
--cluster-secondary-range-name=pods --services-secondary-range-name=services
gcloud beta container clusters create ${cluster_3} \
--region ${region_3} --num-nodes 1 \
--network ${network_name} --subnetwork ${subnet_3} \
--cluster-dns clouddns --cluster-dns-scope vpc \
--cluster-dns-domain ${cluster_domain_3}
--enable-ip-alias \
--cluster-secondary-range-name=pods --services-secondary-range-name=services
Furthermore, we need the node pools (here only done for cluster no. 1):
gcloud container node-pools create pd --cluster ${cluster_1} --machine-type n1-standard-4 --num-nodes=1 \
--node-labels=dedicated=pd --node-taints=dedicated=pd:NoSchedule
gcloud container node-pools create tikv --cluster ${cluster_1} --machine-type n1-highmem-8 --num-nodes=1 \
--node-labels=dedicated=tikv --node-taints=dedicated=tikv:NoSchedule
gcloud container node-pools create tidb --cluster ${cluster_1} --machine-type n1-standard-8 --num-nodes=1 \
--node-labels=dedicated=tidb --node-taints=dedicated=tidb:NoSchedule
Here begins the interesting part: We list the firewalls for cluster subnet no. 1:
gcloud compute firewall-rules list --filter='name~gke-${cluster_1}-.*-all'
and we allow incoming traffic from the other clusters
gcloud compute firewall-rules update ${firewall_rule_name} --source-ranges 10.10.0.0/16,10.11.0.0/16,10.12.0.0/16
If we repeat this for all clusters, then they are interconnected, i.e., we can access a service from cluster A in cluster B.
Now, I am facing the following situation. Say, we have project A and B and one cluster C.
I can use NetworkPolicies to ensure that the resources of the namespaces of project A (A1, A2, A3) can communicate with one another, as can the resources of the namespaces of project B (B1, B2), but there is no communication possible between, say, A1 and B2.
Now, my question is, how can we make that possible for multiple clusters that are connected as above? So assume, we have clusters C1, C2, C3 and for project A we have namespaces A1_C1, A2_C1, A3_C2, A4_C3, A5_C3 (in the respective cluster) and for project B we have namespaces B1_C1, B2_C2, B3_C2, B4_C3.
How can I make it possible, that all the resources of the namespaces associated to project A can communicate, say, A1_C1 to A3_C2, same for project B, but there is no communication possible between projects, say between resources of A1_C1 and B1_C1 or B2_C2?
Is such a thing possible? If so, how?
Your support is greatly appreciated.
Background:
We run Analytics pipelines on dedicated clusters once a day. All clusters are created at the same time, have once pod deployed, run their pipeline and are deleted once complete, use the default VPC network in the same region and are created with a command like this:
gcloud beta container clusters create <CLUSTER_NAME> \
--zone "europe-west1-b" \
--machine-type "n1-standard-2" \
--num-nodes=1 \
--scopes=https://www.googleapis.com/auth/cloud-platform \
--service-account=<SA_EMAIL> \
--disk-size 10GB \
--network default \
--subnetwork <SUB_NETWORK> \
--enable-master-global-access \
--enable-private-nodes \
--enable-private-endpoint \
--enable-ip-alias \
--enable-intra-node-visibility \
--enable-master-authorized-networks \
--master-ipv4-cidr=<MASTER_IP>/28 \
--cluster-ipv4-cidr <CLUSTER_IP>/14 \
--services-ipv4-cidr <SERVICES_IP/20 \
--enable-network-policy \
--enable-shielded-nodes
When we add a new cluster for a new pipeline we have encountered issues where the IP addresses collide, overlap and are unavailable. As we expect to continually add more pipelines and thus more clusters we want an automated way of avoiding this issue.
We have explored creating a dedicated network (and subnetwork) for each cluster so each cluster can have the same IP addresses (albeit in different networks) but are unsure if this is best practice.
Question:
Is it possible to create kubernetes clusters in Google Cloud so as the master, cluster and service IP addresses are auto-assigned?
Is it possible to expose Hue with Component Gateway for Dataproc? I went through the docs and didn't find any option to add service to it. I am creating Dataproc cluster with below command.
gcloud beta dataproc clusters create hive-cluster \
--scopes sql-admin,bigquery \
--image-version 1.5 \
--master-machine-type n1-standard-4 \
--num-masters 1 \
--worker-machine-type n1-standard-1 \
--num-workers 2 \
--region $REGION \
--zone $ZONE \
--optional-components=ANACONDA,JUPYTER \
--initialization-actions gs://bucket/init-scripts/cloud-sql-proxy.sh,gs://bucket/init-scripts/hue.sh \
--properties hive:hive.metastore.warehouse.dir=gs://$PROJECT-warehouse/datasets,dataproc:jupyter.notebook.gcs.dir=gs://bucket/notebooks/jupyter \
--metadata "hive-metastore-instance=$PROJECT:$REGION:hive-metastore" \
--enable-component-gateway
Hue is not an optional component of Dataproc, hence not accessible from component gateway. For now, you have to use Dataproc web interfaces:
Once the cluster has been created, Hue is configured to run on port 8888 on the master node in a Dataproc cluster. To connect to the Hue web interface, you will need to create an SSH tunnel and use a SOCKS 5 Proxy with your web browser as described in the dataproc web interfaces documentation. In the opened web browser go to 'localhost:8888' and you should see the Hue UI.
I have created minikube cluster. I have to run my automation script in the minikube for testcases using pytest. I have to pass service account. How to get the it? Anyone can please help?
While running minikube add extra flags:
minikube start \
--extra-config=apiserver.service-account-signing-key-file=/var/lib/minikube/certs/sa.key \
--extra-config=apiserver.service-account-key-file=/var/lib/minikube/certs/sa.pub \
--extra-config=apiserver.service-account-issuer=api \
--extra-config=apiserver.service-account-api-audiences=api,spire-server,nats \
--extra-config=apiserver.authorization-mode=Node,RBAC \
--extra-config=kubelet.authentication-token-webhook=true
Take a look: minikube-sa, kubernetes-psat.
I configured Gitlab runner (version 11.4.2) to use Kubernetes executor.
Here is my non-interactive registrer command:
gitlab-runner register
--non-interactive \
--registration-token **** \
--url https://mygitlab.net/ \
--tls-ca-file /etc/gitlab-runner/certs/ca.crt \
--executor "kubernetes" \
--kubernetes-image-pull-secrets pull-internal \
--kubernetes-image-pull-secrets pull-external \
--name "kube-docker-runner" \
--tag-list "docker" \
--config "/etc/gitlab-runner/config.toml" \
--kubernetes-image "docker:latest" \
--kubernetes-helper-image "gitlab/gitlab-runner-helper:x86_64-latest" \
--output-limit 32768
It works fine and I can see the execution log in the Gitlab UI
In kubernetes, I see the runner pod composed by 2 containers : helper and build. I expected to see execution job logs by watching the build container logs but it's not the case. I would like to centralize these job execution log with a tool like fluentdbit by reading the container stdout output.
If I start the docker:latest alone (without runner execution) in a pod deployed in the same kubernetes cluster, I can see the logs on stdout. Any idea for configuring the stdout of build container properly ?