Typically manifest contains deployment concerns which are specific to environments. Those concerns can also be met using cf commands. Given a choice of using manifest v/s cf to define scale, bindings, number of instances ... etc, which one would you recommend. In the DevOps based world of cloud applications, do application developers also need to address deployment concerns?
manifest.yml is preferably used if you have a large number of deployments ie. to save you from repetitive task of configuring the app again and again after deployment.
In the cloud world the gap between administrators and developers is getting smaller. This means that developers should be more involved in the process of deployment and especially in process of monitoring application performance in order to improve it.
Related
As I dive into the world of Cloud Composer, Airflow, Google Kubernetes Engine, and Kubernetes I've not yet found a good answer to what exactly makes Cloud Composer better than Helm and GKE.
Here are some things I've found that could be unique to Composer but mostly seem like they could be handled by GKE.
On their homepage:
End-to-end integration with Google Cloud products including BigQuery, Dataflow, Dataproc, Datastore, Cloud Storage, Pub/Sub, and AI Platform gives users the freedom to fully orchestrate their pipeline.
On the features page:
Identity-Aware Proxy protects the interface
Cloud Composer associates a Cloud Storage bucket with the environment. The associated bucket stores the DAGs, logs, custom plugins, and data for the environment.
The downsides of Composer I've seen include:
It takes many hours to spin up a new instance
It doesn't support Kubernetes Executor
It is risky to change the underlying GKE config because it could be changed back by a composer update
There are often errors that happen when auto-scaling often happen but are documented as known
Upgrading environments is still beta
To be clear, I'm not saying Cloud Composer is bad. I'm just having trouble seeing why people like it. When I've asked folks why it is better than Helm + GKE they haven't had any compelling answers despite that they can tell many stories of Composer being unpredictable and having lots of issues.
Are you comparing the same things?
On one side, GKE, you have a container orchestrator. Declare that you want, it will deploy and maintain the stability of the cluster according with declared configuration. This configuration can be packaged with helm to write it in an easier mode. Because you deploy container, you can use the language that you want in your services.
On the other side, you have a workflow manager, with scheduler, retry policies, parallel task, context forwarding. you write DAG in python (only!) and you have operators to interact with external product/services. It's mainly designed for data processing and used a lot by data scientist and data engineering team.
Note: Cloud Composer is deployed on top of GKE (scheduler and worker), redis, app engine and Cloud SQL.
You compare 2 different worlds: Ops world (GKE/Helm) and the App/Data world (Composer/Airflow). Have a look to this new video
Update 1:
My bad, I didn't understand!!! Anyway, personally I don't want to manage things by myself: a cluster, the update of K8S, VM patching, replicas, snapshot, backup/restore,...
If someone can do this for me, I prefer, and managed services are perfect for me!!
Do you ask yourselves this question about Cloud SQL and a database managed by yourselves on a Compute Engine instance? If not (because Cloud SQL solve a lot of boring issues), my opinion is the same for Composer.
But it's an opinion, I didn't test both and compare the performance, cost and easiness.
I work with teams members to develop a microservices architecture but I have a problem with the way to work. Indeed, I have too many microservices and when I run them during my development, it consumes too memory even with a good workstation. So I use docker compose to build and execute my MSA but it takes a long time. One often hears about how technically build an MSA but never about the way to work efficiently to build it. How do you do in this case ? How do you work ? Do you use tools or any others to improve and facilitate your developments. I've heard about skaffold but I don't see what the difference is with docker compose or with a simple ci/cd in a cluster env for example. Feel free to give tips and your opinion. Thanks
I've had a fair amount of experience with microservices and local development and here's been some approaches I've seen:
Run all the things locally on docker or k8. If using k8, then a tool like skaffolding can make it easier to run and debug a service locally in the IDE but put it into your local k8 so that it can communicate with other k8 services. It works OK but running more than 4 or 5 full services locally in k8 or docker requires dedicating a substantial amount of CPU and memory.
Build mock versions of all your services. Use those locally and for integration tests. The mock services are intentionally much simpler and therefore easier to run lots of them locally. Obvious downside is that you have to build mock version of every service, and you can easily miss bugs that are caused by mock services not behaving like the real service. Record/replay tools like Hoveryfly can help in building mock services.
Give every developer their own Cloud environment. Run most services in the cloud but use a tool like Telepresence to swap locally running services in and out of the cloud cluster. This eliminates the problem of running too many services on a single machine but can be spendy to maintain separate cloud sandboxes for each developer. You also need a DevOps resource to help developers when their cloud sandbox gets out of whack.
Eliminate unnecessary microservice complexity and consolidate all your services into 1 or 2 monoliths. Enjoy being able to run everything locally as a single service. Accept the fact that a microservice architecture is overkill for most companies. Too many people choose a microservice architecture upfront before their needs demand it. Or they do it out of fear that they will need it in the future. Inevitably this leads to guessing how they should decompose the system into many microservices, and getting the boundaries and contracts wrong, which makes it just as hard or harder to fix in the future compared to a monolith. And they incur the costs of microservices years before they need to. Microservices make everything more costly and painful, from local development to deployment. For companies like Netflix and Amazon, it's necessary. For most of us, it's not.
I prefer option 4 if at all possible. Otherwise option 2 or 3 in that order. Option 1 should be avoided in my opinion but it is probably the option everyone tries first.
In GKE and assuming you have a private cluster. You can utilize port forwarding while hooked up to the GKE environment through the CLI. Create a script that forwards your local ports to the GKE environment. I believe on the services tab in your cluster is where you will find the "port-forwarding" button that will give you the CMD command. This way you can work on one microservice with all of its traffic being routed to the actual DEV cluster. This prevents you from having to run multiple projects at the same time.
I would say create a staging environment which will have all services running. This staging environment will specifically be curated for development. E.g. if it's deployed using k8s then you expose some ports using nodeport service if you need them for your specific microservice. And have a DevOps pipeline to always keep this environment up to date with the code.
This environment should always be built from master branch. If you have single repo for app or repo per service, it's fair assumption that the will always have most recent code when you create your dev/feature branch.
Then when you want to develop a feature or fix a bug you checkout your microservice. And if you are following the microservice pattern appropriately, that single microservice should be an executable and have it's own docker file and should be debuggable from your local IDE. Many enterprises follow this pattern, and enforce at the organization level that the master branch is always production ready and high quality.
Let's say, you discover a bug in some other microservice running in k8s cluster. You will very likely get tempted to find a way to debug that remote microservice. However, that should be written as a bug for the team that owns the microservice. If your team owns it then you fix it and then start working on your feature. If you really think you need to debug multiple microservices, then I think you have real tight coupling between the services or you don't really need the microservice architecture.
We are using mta to structure our application consisting of multiple micro-services.
The mtar file is deployed to SAP Cloud Foundry in blue-green mode using the SAP Cloud SDK pipeline (Piper Project -> cloudFoundryDeploy step).
We are running into the below issue while trying partial deployment with the blue-green strategy.
Issue: If MTA with same ID is redeployed, pipeline creates new MTA color (Blue to Green and vice-versa) and all applications with the old color are deleted. This approach creates issue during partial deployments.
For instance - using blue-green deployment approach, assume that we have deployed 10 micro-services to SAP CF space.
After bug fixes, we want to do partial redeployment for only a couple of micro-services.
In this case, if we use the same MTA ID and include only the two micro-services in mta.yaml, pipeline deletes the other 8 micro-services which are already deployed to SAP CF.
We tried updating the MTA version but it doesn't make any difference.
As a result of this issue, we unnecessarily need to redeploy all the 10 micro-services again although fix was done for only a couple of micro-services.
On the flip side, if we use a different MTA ID for redeploying the two micro-services, pipeline treats those as initial deployment and triggers an action which is same as the standard deployment type. This results in downtime and defies the purpose of blue-green deployment approach.
Appreciate if someone can help us here to resolve this issue of partial deployment using blue-green strategy and guide us on the best practices to follow.
I don't think this is currently possible, and the right place to build this would probably be multiapps-cli-plugin. I think it would be best if you opened an issue in that repo.
On another note: With that requirement of "partial deployments" I'm not sure if MTA is the best choice. Is there a reason why you need MTA? If you had one maven project per microservice, with one pipeline each this would be closer to the general concept of microservices where each unit can be deployed independently.
Hope this helps
Florian
I am developing a number of microservices which will run on Open Liberty. I have set up a test server in my eclipse environment which is configured to use all the features required by all the services which I am currently working on.
Whilst this works, it seems a heavy-handed approach and it would be good to test each service in an environment which closely resembles the target server. The services can differ in the set of features they require as well as the JVM settings necessary.
Each service will run in its own docker container and the docker configuration is defined in each project.
Is there a way to better test these services without explicitly setting up a new server for each individual service?
I am not aware of any way to segment the Liberty runtime (its features) nor the jvm (for different jvm settings) for different applications running in a single Liberty instance.
You can set app specific variables and retrieve them using MP Config, but that's not the same as jvm settings and certainly not the same as trying to segment specific features of the runtime to a specific application.
However, in general when testing, I would highly recommend trying to mimic your production environment as much as possible. Since you're planning on deploying into docker, I would do the same locally when testing, and given Liberty's lightweight, composable nature, it's unlikely that you'll hit resource issues locally when doing this (you should only enable the features on each Liberty instance that your app is using to minimize the size of that instance). This approach is one of the big benefits/value provided by containers and Liberty.
In other words, even if you could have one Liberty instance segmented per application, I would not recommend it for your testing because, as you said, "it would be good to test each service in an environment which closely resembles the target server"
We have a kind of evaluation job which consists of several thousand invocations of a legacy binary with various inputs, each of which running like a minute. The individual runs are perfectly parallelizable (one instance per core).
What is the state of the art to do this in a hybrid cloud scenario?
Kubernetes itself does not seem to provide an interface for prioritizing or managing waiting jobs. Jenkins would be good at these points, but feels like a hack. Of course, we could hack something ourselves, but the problem should be sufficiently generic to already have an out-of-the box solution.
There are a lot of frameworks that helps managing jobs in Kubernetes cluster. The most popular are:
Argo for orchestrating parallel jobs on Kubernetes. Workflows is implemented as a Kubernetes CRD (Custom Resource Definition).
Airflow - has a modular architecture and uses a message queue to orchestrate an arbitrary number of workers. Also take a look for kubernetes-executor.
I recommend you to look for this video which describe each of framework and help you decide which is better for you.
You may be interested in following aricles about using Mesos for Hybrid Cloud
Xue, Noha & Haugerud, Hårek & Yazidi, Anis. (2017). Towards a Hybrid Cloud Platform Using Apache Mesos. 143-148. 10.1007/978-3-319-60774-0_12.
Hybrid cloud technology is becoming increasingly popular as it merges private and public clouds to bring the best of two worlds together. However, due to the heterogeneous cloud installation, facilitating a hybrid cloud setup is not simple. Despite the availability of some commercial solutions to build a hybrid cloud, an open source implementation is still unavailable. In this paper, we try to bridge the gap by providing an open source implementation by leveraging the power of Apache Mesos. We build a hybrid cloud on the top of multiple cloud platforms, private and public.
Apache Mesos For All Your Hybrid Cloud Needs
Choosing the Best Approach to Hybrid Cloud