I would like to deploy a bunch of yaml files https://github.com/quay/quay/tree/master/deploy/k8s on my kubernetes cluster and would like to know, what is the best approach to deploy these at once.
You can directly apply folder
kubectl create -f ./<foldername>
kubectl apply -f ./<foldername>
You can also add mutiliple files in one command
kubectl apply -f test.yaml,test-1.yaml
You can also merge all YAML files into a single file and manage it further.
Marge YAML file using ---
For example :
apiVersion: v1
kind: Service
metadata:
name: test-data
labels:
app: test-data
spec:
ports:
- name: http
port: 80
targetPort: 9595
- name: https
port: 9595
targetPort: 9595
selector:
app: test-data
tier: frontend
---
apiVersion: v1
kind: Service
metadata:
name: test-app
labels:
app: test-app
spec:
ports:
- name: http
port: 80
targetPort: 9595
- name: https
port: 9595
targetPort: 9595
selector:
app: test-app
tier: frontend
kubectl apply -f <folder-name>
A simple way to deploy all files in a given folder.
You may consider using Helm (The package manager for Kubernetes). Just like we use yum or apt-get for Linux, we use helm for k8s.
Using Helm, you can deploy multiple resources (bunch of YAMLs) in one go. Helm Charts help you define, install, and upgrade even the most complex Kubernetes application. Helm uses a packaging format called charts. A chart is a collection of files that describe a related set of Kubernetes resources. A single chart might be used to deploy something simple, like a memcached pod, or something complex, like a full web app stack with HTTP servers, databases, caches, and so on. Also, you don't need to combine all your YAMLs; they can remain separate as part of a given chart. Besides, if one chart depends on another, you can use the helm dependency feature.
The reason why i use Helm is because whenever i deploy a chart, helm tracks it as a release. Any change to a chart get a new release version. This way, upgrade (or rollback) becomes very easy and you can confidently say what went as part of a given release.
Also, if you have different microservices that have stuff in common, then helm provides a feature called Library Chart using which you can create definitions that can be re-used across charts, thus keeping your charts DRY.
Have a look at this introductory video: https://www.youtube.com/watch?v=Zzwq9FmZdsU&t=2s
I would advise linking the yaml's into one. The purpose of a deployment and service yaml is to deploy your application onto the cluster in one fell swoop. You can define many deployments and services within the one file. In your case, a tool such as Kustomize will help you combine them. Kustomize comes preinstalled with kubectl.
You can combine your yamls called a Multi-Resource yaml into one file using the --- operator. i.e.
apiVersion: v1
kind: Service
metadata:
name: foo
spec:
...
---
apiVersion: v1
kind: Service
metadata:
name: bar
spec:
...
Then make a kustomization.yaml which combines all your multi-resource yamls. There is a good guide on this here: https://levelup.gitconnected.com/kubernetes-merge-multiple-yaml-into-one-e8844479a73a
The documentation from k8 is here: https://kubernetes.io/docs/tasks/manage-kubernetes-objects/kustomization/
Related
I am using minikube (docker driver) with kubectl to test an agones fleet deployment. Upon running kubectl apply -f lobby-fleet.yml (and when I try to apply any other agones yaml file) I receive the following error:
error: resource mapping not found for name: "lobby" namespace: "" from "lobby-fleet.yml": no matches for kind "Fleet" in version "agones.dev/v1"
ensure CRDs are installed first
lobby-fleet.yml:
apiVersion: "agones.dev/v1"
kind: Fleet
metadata:
name: lobby
spec:
replicas: 2
scheduling: Packed
template:
metadata:
labels:
mode: lobby
spec:
ports:
- name: default
portPolicy: Dynamic
containerPort: 7600
container: lobby
template:
spec:
containers:
- name: lobby
image: gcr.io/agones-images/simple-game-server:0.12 # Modify to correct image
I am running this on WSL2, but receive the same error when using the windows installation of kubectl (through choco). I have minikube installed and running for ubuntu in WSL2 using docker.
I am still new to using k8s, so apologies if the answer to this question is clear, I just couldn't find it elsewhere.
Thanks in advance!
In order to create a resource of kind Fleet, you have to apply the Custom Resource Definition (CRD) that defines what is a Fleet first.
I've looked into the YAML installation instructions of agones, and the manifest contains the CRDs. you can find it by searching kind: CustomResourceDefinition.
I recommend you to first try to install according to the instructions in the docs.
I'm currently looking at GKE and some of the tutorials on google cloud. I was following this one here https://cloud.google.com/solutions/integrating-microservices-with-pubsub#building_images_for_the_app (source code https://github.com/GoogleCloudPlatform/gke-photoalbum-example)
This example has 3 deployments and one service. The example tutorial has you deploy everything via the command line which is fine and all works. I then started to look into how you could automate deployments via cloud build and discovered this:
https://cloud.google.com/build/docs/deploying-builds/deploy-gke#automating_deployments
These docs say you can create a build configuration for your a trigger (such as pushing to a particular repo) and it will trigger the build. The sample yaml they show for this is as follows:
# deploy container image to GKE
- name: "gcr.io/cloud-builders/gke-deploy"
args:
- run
- --filename=kubernetes-resource-file
- --image=gcr.io/project-id/image:tag
- --location=${_CLOUDSDK_COMPUTE_ZONE}
- --cluster=${_CLOUDSDK_CONTAINER_CLUSTER}
I understand how the location and cluster parameters can be passed in and these docs also say the following about the resource file (filename parameter) and image parameter:
kubernetes-resource-file is the file path of your Kubernetes configuration file or the directory path containing your Kubernetes resource files.
image is the desired name of the container image, usually the application name.
Relating this back to the demo application repo where all the services are in one repo, I believe I could supply a folder path to the filename parameter such as the config folder from the repo https://github.com/GoogleCloudPlatform/gke-photoalbum-example/tree/master/config
But the trouble here is that those resource files themselves have an image property in them so I don't know how this would relate to the image property of the cloud build trigger yaml. I also don't know how you could then have multiple "image" properties in the trigger yaml where each deployment would have it's own container image.
I'm new to GKE and Kubernetes in general, so I'm wondering if I'm misinterpreting what the kubernetes-resource-file should be in this instance.
But is it possible to automate deploying of multiple deployments/services in this fashion when they're all bundled into one repo? Or have Google just over simplified things for this tutorial - the reality being that most services would be in their own repo so as to be built/tested/deployed separately?
Either way, how would the image property relate to the fact that an image is already defined in the deployment yaml? e.g:
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
name: photoalbum-app
name: photoalbum-app
spec:
replicas: 3
selector:
matchLabels:
name: photoalbum-app
template:
metadata:
labels:
name: photoalbum-app
spec:
containers:
- name: photoalbum-app
image: gcr.io/[PROJECT_ID]/photoalbum-app#[DIGEST]
tty: true
ports:
- containerPort: 8080
env:
- name: PROJECT_ID
value: "[PROJECT_ID]"
The command that you use is perfect for testing the deployment of one image. But when you work with Kubernetes (K8S), and the managed version of GCP (GKE), you usually never do this.
You use YAML file to describe your deployments, services and all other K8S object that you want. When you deploy, you can perform something like this
kubectl apply -f <file.yaml>
If you have several file, you can use wildcard is you want
kubectl apply -f config/*.yaml
If you prefer to use only one file, you can separate the object with ---
apiVersion: v1
kind: Service
metadata:
name: my-nginx-svc
labels:
app: nginx
spec:
type: LoadBalancer
ports:
- port: 80
selector:
app: nginx
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-nginx
labels:
app: nginx
spec:...
...
Is there a way we can have a K8s pod per user/per firm? I realise, per user/per firm grouping is mixing up the business level semantics with infrastructure but say I had this need for regulatory reasons, etc to keep things separate. Then is there a way to create a pod on the fly when a user logs in for the first time and hold this pod reference and route any further requests to the relevant pod which will host a set of containers each running an instance of one of the modules.
Is this even possible?
If possible, what are those identifiers that
can be injected into the pod on the fly that I could use to identify that this is
USER-A-POD vs USER_B_POD or FIRM_A_POD vs FIRM_B_POD ?
Effectively, I need to have a pod template that helps me create identical pods of 1 replica but the only way they differ is they are serving traffic related to one user/one firm only.
Generally, if you want to send traffic to a specific pod say from a Kubernetes Service you would use Labels and Selectors. For example, using the selector app: usera-app in the Service:
apiVersion: v1
kind: Service
metadata:
name: usera-service
spec:
selector:
app: usera-app
ports:
- protocol: TCP
port: 80
targetPort: 80
Then say if the Deployment for your pods, using the label app: usera-app:
apiVersion: apps/v1
kind: Deployment
metadata:
name: usera-deployment
spec:
selector:
matchLabels:
app: usera-app
replicas: 2
template:
metadata:
labels:
app: usera-app
spec:
containers:
- name: myservice
image: nginx
ports:
- containerPort: 80
More info here
How you assign your pods and deployments is up to you and whatever configuration you may use. If you'd like to force create some of the labels in deployments/pods you can take a look at MutatingAdminssionWebhooks.
If you are looking at projects to facilitate all this you can take a look at:
Gatekeeper which is an implementation of the Open Policy Agent for Kubernetes admission. (Still in alpha as of this writing)
Other tools that can help you with attestation and admission mechanism (would have to be adapted for labels):
Kritis
Portieris
Yes, you can create multiple virtual clusters for each user with namespaces.
https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/
Namespaces are the way to divide cluster between users.
I would like to deploy an application cluster by managing my deployment via k8s Deployment object. The documentation has me extremely confused. My basic layout has the following components that scale independently:
API server
UI server
Redis cache
Timer/Scheduled task server
Technically, all 4 above belong in separate pods that are scaled independently.
My questions are:
Do I need to create pod.yml files and then somehow reference them in deployment.yml file or can a deployment file also embed pod definitions?
K8s documentation seems to imply that the spec portion of Deployment is equivalent to defining one pod. Is that correct? What if I want to declaratively describe multi-pod deployments? Do I do need multiple deployment.yml files?
Pagids answer has most of the basics. You should create 4 Deployments for your scenario. Each deployment will create a ReplicaSet that schedules and supervises the collection of PODs for the Deployment.
Each Deployment will most likely also require a Service in front of it for access. I usually create a single yaml file that has a Deployment and the corresponding Service in it. Here is an example for an nginx.yaml that I use:
apiVersion: v1
kind: Service
metadata:
annotations:
service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
name: nginx
labels:
app: nginx
spec:
type: NodePort
ports:
- port: 80
name: nginx
targetPort: 80
nodePort: 32756
selector:
app: nginx
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: nginxdeployment
spec:
replicas: 3
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginxcontainer
image: nginx:latest
imagePullPolicy: Always
ports:
- containerPort: 80
Here some additional information for clarification:
A POD is not a scalable unit. A Deployment that schedules PODs is.
A Deployment is meant to represent a single group of PODs fulfilling a single purpose together.
You can have many Deployments work together in the virtual network of the cluster.
For accessing a Deployment that may consist of many PODs running on different nodes you have to create a Service.
Deployments are meant to contain stateless services. If you need to store a state you need to create StatefulSet instead (e.g. for a database service).
You can use the Kubernetes API reference for the Deployment and you'll find that the spec->template field is of type PodTemplateSpec along with the related comment (Template describes the pods that will be created.) it answers you questions. A longer description can of course be found in the Deployment user guide.
To answer your questions...
1) The Pods are managed by the Deployment and defining them separately doesn't make sense as they are created on demand by the Deployment. Keep in mind that there might be more replicas of the same pod type.
2) For each of the applications in your list, you'd have to define one Deployment - which also makes sense when it comes to difference replica counts and application rollouts.
3) you haven't asked that but it's related - along with separate Deployments each of your applications will also need a dedicated Service so the others can access it.
additional information:
API server use deployment
UI server use deployment
Redis cache use statefulset
Timer/Scheduled task server maybe use a statefulset (If your service has some state in)
Is there any way to pass image version from a varibale/config when passing a manifest .yaml to kubectl command
Example :
apiVersion: v1
kind: ReplicationController
metadata:
name: nginx
spec:
replicas: 1
selector:
app: nginx
template:
metadata:
name: nginx
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:${IMAGE_VERSION}
imagePullPolicy: Always
resources:
limits:
cpu: "1.2"
memory: 100Mi
ports:
- containerPort: 80
Use case is to launch specific image version which is set at kubernetes level, and that the variable is resolved by kubernetes itself at server side.
Thanks and Regards,
Ravi
k8s manifest files are static yaml/json.
If you would like to template the manifests (and manage multiple resources in a bundle-like fashion), I strongly recommend you to have a look at Helm
I've recently created a Workshop which focuses precisely on the "Templating" features of Helm.
Helm does a lot more than just templating, it is built as a full fledged package manager for Kubernetes applications (think Apt/Yum/Homebrew).
If you want to handle everything client side, have a look at https://github.com/errordeveloper/kubegen
Although, at some point, you will need the other features of Helm and a migration will be needed when that time comes - I recommend biting the bullet and going for Helm straight up.
After looking into this recently we decided to just go with sed. Wrap kubectl apply into a small bash script and replace the placeholders before running apply.
We did look into more sophisticated tooling but we only found Helm. However Helm is a complex piece of technology that does way more than just templating. It changes your workflow a lot as you no longer deploy using kubectl and have to have a Helm package repo around to push your packages to. Our assessment was that Helm is not useful for deploying our application and using it for just the templating is overkill.
Here is an example how to do this with sed (it is a excerpt from my typical circleci config):
replaces="s/{.Namespace}/$CIRCLE_BRANCH/;";
replaces="$replaces s/{.CiBuild}/$CIRCLE_BUILD_NUM/; ";
replaces="$replaces s/{.CiCommit}/$CIRCLE_SHA1/; ";
replaces="$replaces s/{.CiUser}/$CIRCLE_USERNAME/; ";
cat ./k8s/app.yaml | sed -e "$replaces" | ./kubectl --kubeconfig=`pwd`/.kube/config apply --namespace=$NAMESPACE -f -