I was evaluating spring cloud config as a way to externalize my app configs.
Long term goal is to get my app on to kubernetes.
Is it advisable to go with spring cloud config or better adopt config map once I am in kubernetes.
It depends what you're trying to do. If you want your config centralised in a github repository or if kubernetes isn't your only deployment platform then you might prefer the spring cloud config server.
If you're just trying to externalise your configuration so that your spring boot apps play well in kubernetes then you could certainly use configmaps. Or if you don't have that many properties you might even just set environment variables. If some of the config is more sensitive then you'd want to look at secrets (a question that can get more complex, depending on how sensitive the config is).
You mention kubernetes as a 'long term goal' so perhaps you need a strategy for the short-term too. Perhaps you could use overriding with environment variables for now. It depends how much overriding you're likely to do and what your transitional platform is.
Related
There are web applications - WebAppA and WebAppB. Each web application depend on a Postgres database. We want to ship these applications to a customer who will deploy the applications on its own k8s cluster.
We want to create three packages - "WebappA", "WebAppB" and "Datastore". The webapp itself made of multiple services, not mentioning for the sake of simplicity.
We want to provide apt-get/brew/yum kind of experience, so that customer can deploy one or both the applications like al-carte. Most importantly while deploying, it should identify if the common package "DataStore" is running and not spin off yet another Postgres instance.
Is there any to package applications as packages for Kubernetes which can make installation easy with dependency handling?
Of course! One way to start would be using Helm charts. You can read more about them here.
Helm defines dependency relationships declaratively using charts, allows you to manipulate/maintain dependencies simply by managing some YAML manifests. It also allows you to have personalised repositories where you can put your images to be fetched from. It's really nice.
As I dive into the world of Cloud Composer, Airflow, Google Kubernetes Engine, and Kubernetes I've not yet found a good answer to what exactly makes Cloud Composer better than Helm and GKE.
Here are some things I've found that could be unique to Composer but mostly seem like they could be handled by GKE.
On their homepage:
End-to-end integration with Google Cloud products including BigQuery, Dataflow, Dataproc, Datastore, Cloud Storage, Pub/Sub, and AI Platform gives users the freedom to fully orchestrate their pipeline.
On the features page:
Identity-Aware Proxy protects the interface
Cloud Composer associates a Cloud Storage bucket with the environment. The associated bucket stores the DAGs, logs, custom plugins, and data for the environment.
The downsides of Composer I've seen include:
It takes many hours to spin up a new instance
It doesn't support Kubernetes Executor
It is risky to change the underlying GKE config because it could be changed back by a composer update
There are often errors that happen when auto-scaling often happen but are documented as known
Upgrading environments is still beta
To be clear, I'm not saying Cloud Composer is bad. I'm just having trouble seeing why people like it. When I've asked folks why it is better than Helm + GKE they haven't had any compelling answers despite that they can tell many stories of Composer being unpredictable and having lots of issues.
Are you comparing the same things?
On one side, GKE, you have a container orchestrator. Declare that you want, it will deploy and maintain the stability of the cluster according with declared configuration. This configuration can be packaged with helm to write it in an easier mode. Because you deploy container, you can use the language that you want in your services.
On the other side, you have a workflow manager, with scheduler, retry policies, parallel task, context forwarding. you write DAG in python (only!) and you have operators to interact with external product/services. It's mainly designed for data processing and used a lot by data scientist and data engineering team.
Note: Cloud Composer is deployed on top of GKE (scheduler and worker), redis, app engine and Cloud SQL.
You compare 2 different worlds: Ops world (GKE/Helm) and the App/Data world (Composer/Airflow). Have a look to this new video
Update 1:
My bad, I didn't understand!!! Anyway, personally I don't want to manage things by myself: a cluster, the update of K8S, VM patching, replicas, snapshot, backup/restore,...
If someone can do this for me, I prefer, and managed services are perfect for me!!
Do you ask yourselves this question about Cloud SQL and a database managed by yourselves on a Compute Engine instance? If not (because Cloud SQL solve a lot of boring issues), my opinion is the same for Composer.
But it's an opinion, I didn't test both and compare the performance, cost and easiness.
I’m currently using kubernetes and I came across of helm.
Let’s say I don’t like the idea of “infecting” my kubernetes cluster with a process that is not related to my applications but I would gladly accept it if it could be beneficial.
So I made some researches but I still can’t find anything I can’t easily do by using my yaml descriptor and kubectl so for now I can’t find an use except,maybe, for the environizing.
For example (taking it from guides I read:
you can easily install application, eg. helm install nginx —> I add an nginx image to my deployment descriptor, done
repositories -> I have docker ones (where I pull my images from)
you can easily helm rollback in case of release failure-> I just change the image version to the previous one in my kubernetes descriptor, easy
What bothers me is that, at level of commands, I do pretty much the same effort (helm update->kubectl apply).
In exchange for that I have a lot of boilerplate because of keeping the directory structure helm wants and I feel like missing the control I have with plain deployment descriptors ...what am I missing?
It is totally understandable your question. For small and simple deploys the benefits is not actually that great. But when the deploy of something is very complex Helm helps a lot.
Think that you have a couple squads that develop microservice for some company. If you can make a Chart that works for most of them, the deploy of each microservices would differ only by the image and the resources required. This way you get an standardized deployment and easier to all developers.
Another use case is deploying applications which requires a lot of moving parts. For example, if you want to deploy a Grafana server on Kubernetes you're probably going to need at least a Deployment and a Configmap, then you would need a service that matches this deployment. And if you want to expose it to the internet you need an ingress too.
One relatively simple application, would require 4 different YAMLs that you would to manually configure and make sure everything is correct instead you could do a simple helm install and reuse the configuration that someone has already made, sometimes even the company who created the Application.
There are a lot of other use cases, but these two are the ones that I would say are the most common.
Here's three suggestions of ways Helm can be useful:
Your continuous deployment system somewhat routinely produces new builds and wants to send them to the Kubernetes cluster. You can use templating to specify the image name and tag in a deployment, and so helm upgrade ... --set tag=201907211931 to request a specific tag.
You might have various service-specific controls like the log level or external database hostnames. The Helm values mechanism gives a uniform way to specify them, without having to know the details of the Kubernetes YAML files.
There is a repository of pre-packaged application charts, so if you want replicated PostgreSQL with in-cluster persistent storage, that's already built for you and you can just depend on it, rather than figuring out the right combination of StatefulSets and PersistentVolumeClaims yourself.
You can combine these in interesting (and potentially complex) ways: use an in-cluster database for developer testing but use a cloud-hosted and backed-up database for production, for example, and compute the database host name based on what combination of settings are provided.
There are, of course, alternative ways to do all of these things. Kustomize in particular can change the image value fairly straightforwardly, and is notable for having been included in the kubectl tool since Kubernetes 1.14 (see also Declarative Management of Kubernetes Objects Using Kustomize in the Kubernetes documentation). The "operator" pattern gives an alternate path to install software in your cluster, but even more so than Helm you're trusting an arbitrary program with API access.
Based on what parameters can we decide on a config server? aws cloud config vs spring cloud config server?
if the application is developed based on Spring framework, which one will be a better option? aws-cloud-config / spring-config?
does both support XML/properties configuration other than YAML?
Can both be configured for High Availability?
Is it possible to define a workflow (for approval of changes that need to be moved to PRODUCTION, using jenkins for example?)
If your application is based on Spring, it makes sense to use Spring Cloud Config as you would be using a specific framework instead of a Generic framework like AWS config
yes, Spring Cloud Config supports both YAML and property files
Ultimately the Spring Cloud Config is a Spring Boot App, hence you can make it high available using the same strategy that we use for any Spring Application
The simplest Approval work flow would be
Configure your Jenkins Job to send mail for approval
Set-up Promotion using https://wiki.jenkins.io/display/JENKINS/Promoted+Builds+Plugin
Deploy the Artificats to the specific environment
Typically manifest contains deployment concerns which are specific to environments. Those concerns can also be met using cf commands. Given a choice of using manifest v/s cf to define scale, bindings, number of instances ... etc, which one would you recommend. In the DevOps based world of cloud applications, do application developers also need to address deployment concerns?
manifest.yml is preferably used if you have a large number of deployments ie. to save you from repetitive task of configuring the app again and again after deployment.
In the cloud world the gap between administrators and developers is getting smaller. This means that developers should be more involved in the process of deployment and especially in process of monitoring application performance in order to improve it.