In the docs for GKE it says all nodes (currently) have the same VM instance. Does this refer to the underlying machine type or the OS image (or both)?
I was assuming it was just the machine type (micro, small,.. etc) and Google layered their own image with infrastructure on top of that (e.g. kubernetes).
If this is the case what image does Google use on GKE? I was thinking it may be CoreOS, since that would seem to be a good match, but I am not sure.
I'd like to set up staging machines with the same image as production... but perhaps we don't need to know this or it doesn't matter what is used.
All nodes in the cluster currently have the same machine type and OS image. By default, the machine type is n1-standard-1 and the image is a recent container-vm image.
If you use gcloud to create your cluster, both settings can be overridden on the command line using the --machine-type and --source-image options respectively (documentation).
If you are using the cloud console to create your cluster, you can specify the machine type but not currently the source image.
Be aware that if you specify a different source image, you may not end up with a functional cluster because the kubernetes software that is installed on top of the source image requires specific underlying packages to be present in the system software. If you want consistency between staging/prod, you can use
gcloud container clusters describe <staging-cluster-name>
To see what image is being used in your staging cluster and ensure that you end up with the same image for your production cluster.
Related
Let's say I deployed 2 pods to Kubernetes and they both have the same underlying image which includes some read-only file system.
By default, do the pods share this file system? Or each pod copies the file system and hence has a separate copy of it?
I would appreciate any answer and especially would love to see some documentation or resources I can read in order to delve deeper into this issue.
Thanks in advance!
In short, it depends on where the pods are running. If they are running on the same node, then yes, they share the same read-only copy of the image, and if on separate nodes, then they have their own read-only copy of the image. Keep reading if you are interesting in knowing more technical details of this.
Inside Kubernetes Pods
A pod can be viewed as a set of containers bound together. It is a construct provided by Kubernetes to be able to have certain benefits out of the box. We can understand your question better if we zoom into a single node that is part of a Kubernetes cluster.
This node will have a kubelet binary running on it, which will receive certain "instructions" from the api-server on running pods. These "instructions" will be passed onto the cri (Container Runtime Interface) running on your node (let's assume it is the docker-engine). And this cri will be responsible for actually running the needed containers and report back to the kubelet which will report back to the api-server ultimately reporting to the pod-controller that the pod containers are Running.
Now, the question becomes, do multiple pods share the same image? I said the answer is yes for pods on the same node and this is how it works.
Say you run the first pod, the docker daemon running on your k8s node pulls this image from the configured registry and stores it in the local cache of the node. It then starts a container using this image. Note that a container that runs, utilizes the image as simply a read-only file-system, and depending on the storage driver configured in docker, you can have a "writeable layer" on top of this read-only filesystem that is used to allow you to read/write on the file-system of your container. This writeable layer is temporary and vanishes when you delete the container.
When you run the second pod, the daemon finds that the image is already available locally, and simply creates the small writeable layer for your container, on top of an existing image from the cache and provides this as a "writeable file system" to your container. This speeds things up.
Now, in case of docker, these read-only layers of the image (as one 'file-system') are shared across all containers running on the host. This makes sense since there is no need to copy a read-only file system and sharing it with multiple containers is safe. And each container can maintain its uniqueness by storing its data in the thin writeable layer that it has.
References
For further reading, you can use:
Read about storage drivers in docker. It explains how multiple containers share the r/o layer of the image.
Read details about different storage driver types to see how this "thin writeable layer" is implemented in practice by docker.
Read about container runtimes in Kubernetes to understand that docker isn't the only supported runtime. There are others but more or less, the same will hold true for them as well, as it makes sense to cache images locally and re-use the read-only image file system for multiple containers.
Read more about the kubelet component of Kubernetes to understand how it can support multiple run-times and how it helps the pod-controller setup and manage containers.
And of course, finally you can find more details about pods here. This will make a lot more sense after you've read the material above.
Hope this helps!
I have a simple 3-node cluster created using AKS. Everything has been going fine for 3 months. However, I'm starting to have some disk space usage issues that seem related to the Os disks attached to each nodes.
I have no error in kubectl describe node and all disk-related checks are fine. However, when I try to run kubectl logs on some pods, I sometimes get "no space left on device".
How can one manage storage used in those disks? I can't seem to find a way to SSH into those nodes as it seems to only be manageable via Azure CLI / web interface. Is there also a way to clean what takes up this space (I assume unused docker images would take place, but I was under the impression that those would get cleaned automatically...)
Generally, the AKS nodes just run the pods or other resources for you, the data is stored in other space just like remote storage server. In Azure, it means managed disks and Azure file Share. You can also store the growing data in the nodes, but you need to configure big storage for each node and I don't think it's a good way.
To SSH into the AKS nodes, there are ways. One is that set the NAT rule manually for the node which you want to SSH into in the load balancer. Another is that create a pod as the jump box and the steps here.
The last point is that the AKS will delete the unused images regularly and automatically. It's not recommended to delete the unused images manually.
Things you can do to fix this:
Create AKS with bigger OS disk (I usually use 128gb)
Upgrade AKS to a newer version (this would replace all the existing vms with new ones, so they won't have stale docker images on them)
Manually clean up space on nodes
Manually extend OS disk on nodes (will only work until you scale\upgrade the cluster)
I'd probably go with option 1, else this problem would haunt you forever :(
I'm looking for some clues on which solution can help me on my problem which is running a kubernetes cluster between 2 VMs.
I'm beginning with Kubernetes and all its possibilities but like everybody I started from a minikube single-node cluster to host my 4 containers respectively hosting mongoDB, redis, rabbitMQ and minio.
The idea is that I need something like minikube to create a cluster like this:
Moreover, these 2 VMs will run on RedHat EL 7 and won't be local and they may be hosted on different machines
Is it possible to build that architecture with kubeadm?
Response is yes, it can be achieved with kubeadm, it will allow you to aggregate your different VM/hosts into the cluster in the same way docker-swarm does by kind of subscribing them into the cluster.
See how (thanks #Jason Stanley)
Some Prerequisites
If like me you're on a client infrastructure, implementing your solution 'on-premise' on a Red Hat environment, you'll need two or three things.
The proper docker environment
An image registry
An easy way to configure your cluster
Docker environment
First one and because it's a restriction meant by docker and red hat is to use docker-ee the enterprise edition which allow support on prod by red hat, which is why most people pay a red hat subscription.
See typical architecture using docker-ee and how to install that on Red Hat
Local image registry
The other thing is to deploy a docker registry, this is an official image (documentation here) provided by Docker team, registry will allow your cluster to pull the image from it - images that you will push inside while setting everything up - note that this is relevant in a restricted environment where your VM/hosts can't have access to internet or use a proxy.
Cluster setting tool
One handy tool is using helm which allow you to set your cluster easily by referencing which images provide a given service, on which port with which policy.
Helm documentation
How can I specify a specific vm type for the cluster master (I don't want to use an high memory instance for relative an inactive node).
Also, is there any way to add nodes to a cluster and choosing the type of vm? (this can solve the first problem)
Update November 2015:
Now that Google Container Engine is no longer in alpha, you don't need to worry about the size of your cluster master, as it is part of the managed service.
You can now easily add/remove nodes from your cluster through the cloud console UI but they will all be the same machine type that you originally choose for your cluster.
If you are running OSS Kubernetes on GCE, then you can set the MASTER_SIZE environment variable in cluster/gce/config-default.sh before creating your cluster.
If you are running on GKE, we unfortunately don't yet offer the option to customize the size of your master differently than the size of your nodes. We hope to offer more flexibility in cluster provisioning soon.
There is currently not a way to resize your cluster after you create it. I'm actually working on this for OSS Kubernetes in Issue #3168.
I tried to create a new cluster in Container Engine in the Google Developers Console.
It finished pretty quickly with a yellow triangle with an exclamation point. I'm assuming that means it didn't work.
Any idea what I could be doing wrong?
There's a few things that could go wrong. The best option to figure out what's wrong in your situation is to try using the gcloud command line tool, which gives better error information. Information about how to install and use it is in Container Engine's documentation.
Other than the default network being removed (as mentioned by Robert Bailey), you may be trying to create more VM instances than you have quota for. You can check what your quota is on the developer console under Compute > Compute Engine > Quota. You're most likely to go over quota on either CPUs or in-use IP addresses, since each VM created is given an ephemeral IP address.
Have you deleted your default network?
The alpha version of Container Engine relies on the default network when creating VMs and routes between the nodes and you will see an error creating a cluster if you have deleted the default network.