I'm reading the Mesos Architecture docs which, ironically, don't actually specify which components are supposed to run on which VMs/physicals.
It looks like, to run Mesos in HA, you need several categories of components:
Mesos Masters
ZooKeeper instances (quorum)
Hadoop clusters (job nodes? name nodes?)
But there's never any mention of how many you need of each type.
So I ask: How many VMs/physicals do you need to run Mesos with HA, and what components should be deployed to each?
Did you have a look at the HA docs? To run Mesos in HA, you'll need the Mesos Masters and ZooKeeper. Any Hadoop-related configurations are out of scope for Mesos HA itself.
To have a HA setup, you'll need a uneven number of nodes for the Masters and ZooKeeper (because of quorum mechanism). In our case, we're running 3 Master and 3 ZooKeeper nodes on 3 machines (one Master and one ZooKeeper instance per machine), and a number of Mesos Slaves/Agents on different machines.
Theoretically, the Slaves/Agents can run on the same machines as the Masters/ZooKeepers as well. I guess this is a matter of preferences and availability of machines, and your SLA needs.
If you want to run a large-scale production setup, it will probably make a lot of sense to even separate the Master and ZooKeeper instances.
Further references:
http://mesos.apache.org/documentation/latest/operational-guide/
http://mesos.apache.org/documentation/latest/configuration/ (see "Master Options")
Can Mesos 'master' and 'slave' nodes be deployed on the same machines?
Related
Deploy Mongo Database with a single master and two read replicas in the Kubernetes cluster of at least 3 worker nodes that are available in different availability zones.
Points to keep in mind while deploying the DB:
All replicas of DB should be deployed in a separate worker node of diff Availability Zone(For high availability).
Autoscale the read replicas if needed.
Data should be persistent.
Try to run the containers in nonprivileged mode if possible.
Use best practices as much as you can.
Push the task into a separate branch with a proper README file.
I'm new to kubernetes and trying to explore the new things in it. So, my question is
Suppose I have existing kubernetes cluster with 1 master node and 1 worker node. Consider this setup is on AWS, now I have 1 more VM instance available on Oracle Cloud Platform and I want to configure that VM as worker node and attach that worker node to existing cluster.
So, is it possible to do so? Can anybody have any suggestions regarding this.
I would instead divide your clusters up based on region (unless you have a good VPN between your oracle and AWS infrastructure)
You can then run applications across clusters. If you absolutely must have one cluster that is geographically separated, I would create a master (etcd host) in each region that you have a worker node in.
Worker Node and Master Nodes communication is very critical for Kubernetes cluster. Adding nodes from on-prem to a cloud provider or from different cloud provider will make lots of issues from network perspective.
As VPN connection between AWS and Oracle Cloud needed and every time worker node has to cross ocean (probably) to reach master node.
EDIT: From Kubernetes Doc, Clusters cannot span clouds or regions (this functionality will require full federation support).
https://kubernetes.io/docs/setup/best-practices/multiple-zones/
We are planning to build a small docker cluster for our application services. We considered to use 2 master vms for ha, 1 consul(if we choose Swarm) and 5-10 hosts for containers. We have not yet decided what to use - Docker Swarm or Kubernetes.
So the question is what "hardware" requirements (CPU cores, RAM) managers, both Swarm and Kubernetes, can meet to orchestrate this small cluster.
Just to clarify a bit on what Robert wrote about Kubernetes.
If you want to have up to 5 machines for running your applications even 1-core virtual machine (n1-standard-1 on GCE) should be enough.
You can handle 10-node cluster with 2-core virtual machine as Robert said. For official recommendations please take a look at:
https://kubernetes.io/docs/setup/best-practices/cluster-large/
However, note that resource usage of our master components is more related to number of pods (containers) you want to run on your cluster. If you want to have say single-digit-number of them, even n1-standard-1 GCE should be enough for 10-node cluster. But it's definitely safer to use n1-standard-2 in case of <=10 node clusters.
As for HA, I agree with Robert that having 3 master VMs is better than 2. Etcd (which is our backing storage) requires more than a half of all registered replicas to be up to work correctly, so in case of 2 instances, all of them needs to be up (which is generally not your goal). If you have 3 instances, one of them can be down.
Let me know if you have more questions about Kubernetes.
For Kubernetes, a single 2-core virtual machine (e.g. n1-standard-2 on GCE) can handle 5 nodes and probably 10. If you want to run an HA master configuration, you are likely to want 3 nodes to create a quorum for the etcd instances and you may want to provision slightly larger instances (like an n1-standard-4) to account for the overhead of clustering etcd.
Hi I am setting up Kubernetes on top of Mesos by following http://kubernetes.io/v1.1/docs/getting-started-guides/mesos.html and this is how my current test lab looks like
2 Numbers of mesos master with zookeeper
2 Numbers of mesos slaves with docker and flannel installed
Additional mesos slave running Kubernetes-mesos and kubernetes srvices
A server with ETCD service which supports both flannel and kubernetes
Can you please let me know if this is enough ?
Below are the two questions I have
Do we really need to have the kubernetes master server here to be configured as a mesos slave?
Do we need to install kubernetes package on mesos slaves as well ? The url talks about package installation and configuration only on the kubernetes master..With out kubernetes running on the slaves can the master create pods/services etc on the slaves through mesos scheduler?
Regarding Mesos Masters and Zookeeper instances, to have an even number of nodes is not really a good idea, because of the quorum mechanisms involved. My suggestion would be running three nodes of both services.
I assume you want to run this locally? If so, I guess it would make sense to use a preconfigured Vagrant project such as https://github.com/tobilg/coreos-mesos-cluster This launches a three node CoreOS cluster with all the Mesos/Zookeeper services already installed, and etcd and flanneld are also already installed on CoreOS itself.
This would mean that you only would have to do the following steps once the cluster is launched:
http://kubernetes.io/v1.1/docs/getting-started-guides/mesos.html#deploy-kubernetes-mesos respectively https://coreos.com/kubernetes/docs/latest/getting-started.html
http://kubernetes.io/v1.1/docs/getting-started-guides/mesos.html#start-kubernetes-mesos-services
1)Kubernetes master doesnt need to be a mesos slave .
2)You dont need kubernetes to be installed on the minions(mesos-slaves)
All you need is below
1)Mesos setup (Mesos masters and slaves along with zookeeper ,docker running on all mesos slaves)
2)etcd cluster ,will provide overlay network(flannel) and also will do the service discovery of kubernetes setup
3)kubernetes master ..
Below blogs helped a lot in setting it up
http://manfrix.blogspot.in/2015/11/mesoskubernetes-how-to-install-and-run.html
https://github.com/ruo91/docker-kubernetes-mesos
Can Apache Mesos 'master' nodes be co-located on the same machine as Mesos 'slave' nodes? Similarly (for high-availability (HA) deploys), can the Apache Zookeeper nodes used in Mesos 'master' election be deployed on the same machines as Mesos 'slave' nodes?
Mesos recommends 3 'masters' be used for HA deploys, and Zookeeper recommends 5 nodes be used for its quorum election system. It would be nice to have these services running along side Mesos 'slave' processes instead of committing 8 machines to effectively 'non-productive' tasks.
If such a setup is feasible, what are the pros/cons of such a setup?
Thanks!
You can definitely run a master, slave, and zk process all on the same node. You can even run multiple master and slave processes on the same node, provided you give them each unique ports, but that's only useful for a test cluster.
Typically we recommend running ZK on the same nodes as your masters, but if you have extra ZKs, you can certainly run them on slaves, or mix-and-match as you see fit, as long as all master/slave/framework nodes can reach the ZK nodes, and all slaves can reach the masters.
For a smaller cluster (<10 nodes) it could make sense to run a slave process on each master, especially since the standby masters won't be doing much. Even an active master for a small cluster uses only a small amount of cpu, memory, and network resources. Just make sure you adjust the --resources on that slave to account for the master's resource usage.
Once your cluster grows larger (especially >100 nodes) the network traffic to/from the master as well as its cpu/memory utilization becomes significant enough that you wouldn't want to run a mesos slave on the same node as the master. It should be fine to co-locate ZK with your master even at large scale.
You didn't specifically ask, but I'll also discuss where to run your framework schedulers (e.g. Spark, Marathon, or Chronos). These could be co-located with any of the other components, but they only really need to be able to reach the master and zk nodes, since all communication to slaves goes through the master. Some customers run the schedulers on master nodes, some run them on edge nodes (so users don't have access to the slaves), and others use meta-frameworks like Marathon to run other schedulers on slaves as Mesos tasks.