I just started working in a software company and they have 4 different deployment environments.
Reference / Integration / Acceptance / Production.
Could someon please explain to me what the typical usecases of these single environments are? Production is clear to me but the others not really. Especially the difference between Reference and Integration. I already googled it but couldn't find it out.
In software deployment, an environment or tier is a computer system in which a computer program or software component is deployed and executed. In simple cases, such as developing and immediately executing a program on the same machine, there may be a single environment, but in industrial use the development environment (where changes are originally made) and production environment (what end users use) are separated; often with several stages in between. This structured release management process allows phased deployment (rollout), testing/Acceptance , and rollback in case of problems.
Environments may vary significantly in size: the development environment is typically an individual developer's workstation, while the production environment may be a network of many geographically distributed machines in data centers, or virtual machines in cloud computing.
Related
I work with teams members to develop a microservices architecture but I have a problem with the way to work. Indeed, I have too many microservices and when I run them during my development, it consumes too memory even with a good workstation. So I use docker compose to build and execute my MSA but it takes a long time. One often hears about how technically build an MSA but never about the way to work efficiently to build it. How do you do in this case ? How do you work ? Do you use tools or any others to improve and facilitate your developments. I've heard about skaffold but I don't see what the difference is with docker compose or with a simple ci/cd in a cluster env for example. Feel free to give tips and your opinion. Thanks
I've had a fair amount of experience with microservices and local development and here's been some approaches I've seen:
Run all the things locally on docker or k8. If using k8, then a tool like skaffolding can make it easier to run and debug a service locally in the IDE but put it into your local k8 so that it can communicate with other k8 services. It works OK but running more than 4 or 5 full services locally in k8 or docker requires dedicating a substantial amount of CPU and memory.
Build mock versions of all your services. Use those locally and for integration tests. The mock services are intentionally much simpler and therefore easier to run lots of them locally. Obvious downside is that you have to build mock version of every service, and you can easily miss bugs that are caused by mock services not behaving like the real service. Record/replay tools like Hoveryfly can help in building mock services.
Give every developer their own Cloud environment. Run most services in the cloud but use a tool like Telepresence to swap locally running services in and out of the cloud cluster. This eliminates the problem of running too many services on a single machine but can be spendy to maintain separate cloud sandboxes for each developer. You also need a DevOps resource to help developers when their cloud sandbox gets out of whack.
Eliminate unnecessary microservice complexity and consolidate all your services into 1 or 2 monoliths. Enjoy being able to run everything locally as a single service. Accept the fact that a microservice architecture is overkill for most companies. Too many people choose a microservice architecture upfront before their needs demand it. Or they do it out of fear that they will need it in the future. Inevitably this leads to guessing how they should decompose the system into many microservices, and getting the boundaries and contracts wrong, which makes it just as hard or harder to fix in the future compared to a monolith. And they incur the costs of microservices years before they need to. Microservices make everything more costly and painful, from local development to deployment. For companies like Netflix and Amazon, it's necessary. For most of us, it's not.
I prefer option 4 if at all possible. Otherwise option 2 or 3 in that order. Option 1 should be avoided in my opinion but it is probably the option everyone tries first.
In GKE and assuming you have a private cluster. You can utilize port forwarding while hooked up to the GKE environment through the CLI. Create a script that forwards your local ports to the GKE environment. I believe on the services tab in your cluster is where you will find the "port-forwarding" button that will give you the CMD command. This way you can work on one microservice with all of its traffic being routed to the actual DEV cluster. This prevents you from having to run multiple projects at the same time.
I would say create a staging environment which will have all services running. This staging environment will specifically be curated for development. E.g. if it's deployed using k8s then you expose some ports using nodeport service if you need them for your specific microservice. And have a DevOps pipeline to always keep this environment up to date with the code.
This environment should always be built from master branch. If you have single repo for app or repo per service, it's fair assumption that the will always have most recent code when you create your dev/feature branch.
Then when you want to develop a feature or fix a bug you checkout your microservice. And if you are following the microservice pattern appropriately, that single microservice should be an executable and have it's own docker file and should be debuggable from your local IDE. Many enterprises follow this pattern, and enforce at the organization level that the master branch is always production ready and high quality.
Let's say, you discover a bug in some other microservice running in k8s cluster. You will very likely get tempted to find a way to debug that remote microservice. However, that should be written as a bug for the team that owns the microservice. If your team owns it then you fix it and then start working on your feature. If you really think you need to debug multiple microservices, then I think you have real tight coupling between the services or you don't really need the microservice architecture.
Looking at Microsoft documentation that says, under "Multiple deployment environments (production and staging)", that the "Service Fabric allows you to have multiple environments for your apps or to deploy different versions of your app side-by-side." How is this done and what does the topology look like? At any point in the cycle would different VMs in the cluster be at different versions or would different versions be installed on the same VMs? Particularly interested in Dev/Test/Prod environments, not simply upgrading Prod through a staging mechanism, though that would be good to better understand as well. Thanks.
What does Apache Mesos do that Kubernetes can't do or vice-versa?
Mesos is a Two level scheduler. Sure it grabs resource information from every machine and gives it to the top level scheduler such that frameworks like kubernetes can use to schedule containers across machines but Kubernetes can itself schedule containers across machines (No need for Mesos from this regard). so what are few things that Apache Mesos can do that Kubernetes cannot do or vice-versa?
Both Mesos and Kubernetes are n-th level containers orchestrators. This means you can achieve the same features but some kind of tasks could be done easier (read. better) on one of them. In fact, you can run Kubernetes on Mesos and vice verse.
Let's go through main differences that give some clue when you need to make a decision:
Architecture
As you pointed out Mesos is a Two-Level Scheduler and this is the main difference in architecture. This gives you the ability to create your custom scheduler (aka framework) to run your tasks. What's more, you can have more than one scheduler. All your schedulers compete for the resources that are fairly distributed using Dominant Resources Fairness algorithm (that could be replaced with custom allocator). You can also assign roles to the frameworks and tasks and assign weights to this roles to prioritize some schedulers. Roles are tightly connected with resources. Above features gives you the ability to create your own way of scheduling for different applications (e.g., Fenzo) with different heuristics based on a type of tasks you want to run. For example, when running batch tasks it's good to place them near data and time to start is not so important. On the other hand, running stateless services is independent of nodes and it's more critical to run them ASAP.
Kubernetes architecture is a single level scheduler. That means decisions where pod will be run are made in a single component. There is no such thing as resource offer. On the other hand, everything there is pluggable and built with a layered design.
Origin
Mesos was created at Twitter (formerly at Berkeley but the first production usage was at Twitter) to support their scale.
In March 2010, about a year into the Mesos project, Hindman and his Berkeley colleagues gave a talk at Twitter. At first, he was disappointed. Only about eight people showed up. But then Twitter's chief scientist told him that eight people was lot – about ten percent of the company's entire staff. And then, after the talk, three of those people approached him.
Soon, Hindman was consulting at Twitter, working hand-in-hand with those ex-Google engineers and others to expand the project. Then he joined the company as an intern. And, a year after that, he signed on as a full-time employee.
source
Kubernetes was created by Google to bring users to their cloud promising no lock-in experience. This is the same technique Amazon did with Kindle. You can read any book on it but using it with Amazon gives you the best experience. The same is true for Google. You can run Kubernetes on any cloud (public or private) but the best tooling, integration and support you'll get only on Google Cloud.
But Google and Microsoft are different. Microsoft wants to support everything on Azure, while Google wants Kubernetes everywhere. (In a sense, Microsoft is living up to the Borg name, assimilating all orchestrators, more than Google is.) And quite literally, Kubernetes is how Google is playing up to the on-premises cloud crowd giving it differentiation from AWS (which won’t sell its infrastructure as a stack with a license, although it says VMware is its private cloud partner) and Microsoft (which still doesn’t have its Azure Stack private cloud out the door). source
Community
Judging a project simply by its community size could be misleading. It's like you'd be saying that php is a great language because it has large community.
Mesos community is much smaller than Kubernetes. That's the fact. Kubernetes has financial support from many big companies including Google, Intel, Mirantis, RedHat and more while Mesos is developed mainly by Mesosphere with some support from Apple, Microsoft. Although Mesos is a mature project, its development is slow but stable. On the other hand, Kubernetes is much younger, but rapidly developed.
Meso contributors origin
The Kubernetes Community - Ian Lewis, Developer Advocate, Google
Scale
Mesos was targeted for big customers from the early beginning. It is used at Twitter, Apple, Verizon, Yelp, Netflix to run hundreds of thousands of containers on thousands of servers.
Kubernetes was started by Google to give developers Google Infrastructure experience (GIFFE). From the beginning, it was prepared for small scale up to hundreds of machines. This constraint is increased with every release but they started small to grow big. There are no public data about biggest Kubernetes installation.
Hype
Due to scale issues, Kuberntetes started to be popular among smaller companies (not cloud scale) while Mesos was targeted for enterprise users. Kubernetes is supported by Cloud Native Foundation while Mesos is Apache Foundation Project. These two foundations have different founding and sponsors. Generally, more money gives you better marketing and Kubernetes definitely did it right.
https://g.co/trends/RUuhA
Conclusion
It looks like Kubernetes already won the containers orchestrator war. But if you have some custom workloads and really big scale, Mesos could be a good choice.
The main difference is in the community size and the open source model : where DCOS is supported by Mesosphere and provide enterprise features in a commercial product only (because mesosphere isn't philanthropist), K8S has a larger community with strong contributions from different companies resulting in providing much more integrated enterprise features (multitenancy, RBAC, quota, preemption, gateways...) meaning they are easier to use, not necessarily they don't exist in DCOS.
I would globally say that :
DCOS is more battle tested for stateful and big data workloads but lacks of integration with other perimetric components including plug and play central monitoring and logging and enterprise features like security model, multi tenancy, auto updates... It was a very hard way to integrate everything for a production grade platform.
K8S is more battle tested for stateless apps and provides lots of plug and play tools like prometheus, EFK, helm... which makes the implementation of a production grade platform much easier. Next to that there is a big move on stateful workloads with statefulsets and the operator pattern which is comparable with mesos frameworks but again, K8S provides lots of tools to develop them with less costs because lots of functionalities are provided out of the box, it takes me 2 months to develop a MongoDB operator to provide MongoDB as a service in a multi tenant and secured way and I needed to learn Golang in the same time.
source
https://www.infoworld.com/article/3118345/cloud-computing/why-kubernetes-is-winning-the-container-war.html
https://www.theregister.co.uk/2017/10/17/docker_ee_kubernetes_support
https://www.techrepublic.com/article/these-two-vendors-are-most-likely-to-bring-kubernetes-containers-to-the-enterprise
https://www.cloudhealthtech.com/blog/container-wars-are-over-kubernetes-has-won
https://news.ycombinator.com/item?id=12462261
I am setting up an automated deployment environment for a number of decoupled services that are in active development. While I am comfortable with the automated deployment/configuration management aspect, I am looking for strategies on how best to structure the deployment environment to make things a bit easier for developers. Some things to take into consideration:
Developers are generally building web applications, web services, and daemons -- all of which talk to one another over HTTP, sockets, etc.
The developers may not have all running on their locally machine, but still need to be able to quickly do end to end testing by pointing their machine at the environment
My biggest concern with continuous deployment is that we have a large team and I do not want to constantly be restarting services while developers working locally against those remote servers. On the flip side, delaying deployments to this development environment makes integration testing much more difficult.
Can you recommend a strategy that you have used in this situation in the past that was worked well?
Continuous integration doesn't have to mean continuous deployment. You can compile/unit test/etc the code "continuously" thoughout the day without deploying it and only deploy at night. This is often a good idea anyway - to deploy at night or on demand - since people may be integration testing during the day and wouldn't want the codebase to change out from under them.
Consider, how much of the software can developers test locally? If a lot, they shouldn't need the environment constantly. If not a lot, it would be good to set up mocks/stubs so much more can be tested on a local server. Then the deployed environment is only needed for true integration testing and doesn't need to be update constantly throughout the day.
I'd suggest setting up a CI server (Hudson?) and use this to control all deployments to both your QA and production servers. This forces you to automate all aspects of deployment and ensures that the are no ad-hoc restarts of the system by developers.
I'd further suggest that you consider publishing your build output to a repository manager like Nexus , Artifactory or Archiva. In that way deployment scripts could retrieve any version of a previous build. The use of a repository manager would enable your QA team to certify a release prior to it's deployment onto production.
Finally, consider one of the emerging deployment automation tools. Tools like chef, puppet, ControlTier can be used to further version control the configuration of your infrastructure.
I agree with Mark's suggestion in using Hudson for build automation. We have seem successful continuous deployment projects that use Nolio ASAP (http://www.noliosoft.com) to automatically deploy the application once the build is ready. As stated, chef, puppet and the like are good for middle-ware installations and configurations, but when you need to continuously release the new application versions, a platform such as Nolio ASAP, that is application centric, is better suited.
You should have the best IT operation folks create and approve the application release processes, and then provide an interface for the developers to run these processes on approved environments.
I wonder what type of servers for internal usage you virtualize in the last -say- 6 months. Here's what we got virtual so far:
mediawiki
bugtracker (mantis)
subversion
We didn't virtualize spezialized desktop PCs which are running a certain software product, that is only used once in a while. Do you plan to get rid of those old machines any time soon?
And which server products do you use? Vmware ESX, Vmware Server, Xen installations...?
My standard answer to questions like this is, "virtualization is great; be aware of its limitations".
I would never rely on a purely-virtual implementation of anything that's an infrastructure-level service (eg the authoritative DNS server for your site; management and monitoring tools).
I work for a company that provides server and network management tools. We are constantly trying to overcome the marketing chutzpah of virtualization vendors in that infrastructure tools shouldn't live in infrastructure tools.
Virtualization wants to control all of your services. However, there are some things that should always exist on physical hardware.
When something goes wrong with your virtual setup, troubleshooting and recovery can take a long time. If you're still running some of those services you require for your company on physical hardware, you're not dead-in-the-water.
Virtualization also introduces clock lag, disk and network IO lag, and other issues you wouldn't see on physical hardware.
Lastly, the virtualization tool you pick then becomes in charge of all of the resources under its command for its hosted VMs. That translates to the hypervisor - not you - deciding what VM should have priority at any given moment. If you're concerned about any tool, service, or function being guaranteed to have certain resources, it will need to be on physical hardware.
For anything that "doesn't matter", like web, mail, dhcp, ldap, etc - virtualization is great.
Our build machine running FinalBuilder runs on a Windows XP Virtual Machine running in VMWare Server on Linux.
It is very practical to move it and also to backup, we just stop the Virtual Machine and copy the disk image.
Some days ago we needed to change the host pc, it took less than 2 hours to have our builder up and running on another pc.
We migrate to a new SBS 2005 Domain last month. We take the opotunity to create virtual machines for the following servers
Buid Machine
Svn Repository Machine
Bug Traking Machine (FogBugz)
Testing Databases
I recently had to build an internal network for our training division, enabling the classrooms to be networked and have access to various technologies. Because of the lack of hardware and equipment and running in an exclusive cash only environment I decided to go with a virtual solution on the server.
The server itself is running CentOS 5.1 with VMWare 1.0.6 loaded as the virtualisation provider. On top of this we have 4 Windows Server 2003 machines running, making up the Active Directory, Exchange, ISA, Database and Windows/AV updates component. File sharing and internet routing through the corporate network and ADSL is handled via the CentOS platform.
The setup allows us to expand to physical machines at a later stage quickly, and allows the main server to replaced with minimum downtime on the network, as it only requires the moving of the Virtual Machines and starting them up on the new box.
Project Management (dotProject)
Generic Testing Servers (IIS, PHP, etc)
Do you plan to get rid of those old machines any time soon? No
And which server products do you use? MS Virtual Server
We use ESX in our labs and lately we've virtualized our document sharing service (KnowledgeTree), the lab management tools and almost all of our department's internal web servers.
We also virtualized almost all of our QA department's test machines, with the exception of the performance and stability testing hardware.
We aren't going to get rid of the hardware any time soon, it will be used to decrease the budget needs and increase the number of projects that can be handled by one lab.
We use VMware ESX 3.5.x exclusively.
We virtualise a copy of a test client and server, so we can deploy to them before sending the files to the customer. They also gets used to test bug reports.
We find this is the biggest benefit to virtualisation as we can keep lots of per-customer versions around.
We also VM our web server, and corporate division has virtualised everything.