Similar to posts like this Separate dev and prod Firebase environment
I'm running into similar structuring issues. Unlike other posts like that that i've found, in my case, it's GCP as a whole rather than just Firebase. In addition, i'm looking at separation (or not) of blue and green deployments ALONG with the various environments.
The projects will be handling IoT data; mobile, field sensor/modules, web (in the future). Currently everything is unfortunately in one project.
So, i'm thinking of having 3 different projects for the staging, production, and test environments with each project having both blue and green deployments per, perhaps besides test but that's a different conversation.
Does GCP as a whole have documentation or recommendations about this? Or do you guys have any recommendations?
It's hard to answer because it depends a lot of your organization, your needs and your way of working.
Here you can find a google document about resources hierarchy.
However, I already see some GCP customers using only one project for dev/uat/prod because they share the same K8S cluster and separate the environment thanks to the namespace. By the way, the cluster maintenance cost is done only once for all the different steps of the project.
About the Blue/Green it depends on which component you want to apply this. If it's on the website, App Engine, or a Global Loadbalancer can do this. If it's about IoT Core or PubSub, I fear that you have to manage this by yourself or to create 2 different projects for this.
Related
First, I'm not sure this question is specific enough for Stack Overflow. Happy to remove or revise if someone has any suggestions.
We use Kubernetes to orchestrate our server side code, and have recently begun using Kustomize to modularize the code.
Most of our backend services fit nicely into that data model. For our main transactional system we have a base configuration that we overlay with tweaks for our development, staging, and different production flavors. This works really well and has helped us clean things up a ton.
We also use TensorFlow Serving to deploy machine learning models, each of which is trained and at this point deployed for each of our many clients. The only way that these configurations differ is in the name and metadata annotations (e.g., we might have one called classifier-acme and another one called classifier-bigcorp), and the bundle of weights that are pulled from our blob storage (e.g., one would pull from storage://models/acme/classifier and another would pull from storage://models/bigcorp/classifier). We also assign different namespaces to segregate between development, production, etc.
From what I understand of the Kustomize system, we would need to have a different base and set of overlays for every one of our customers if we wanted to encode the entire state of our current cluster in Kustomize files. This seems like a huge number of directories as we have many customers. If we have 100 customers and five different elopement environments, that's 500 directories with a kustomize.yml file.
Is there a tool or technique to encode this repeating with Kustomize? Or is there another tool that will work to help us generate Kubernetes configurations in a more systematic and compact way?
You can have more complex overlay structures than just a straight matrix approach. So like for one app have apps/foo-base and then apps/foo-dev and apps/foo-prod which both have ../foo-base in their bases and then those in turn are pulled in by the overlays/us-prod and overlays/eu-prod and whatnot.
But if every combo of customer and environment really does need its own setting then you might indeed end up with a lot of overlays.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 2 years ago.
Improve this question
I am totally new to this two technologies (I know docker and kubernetes btw).
Haven't find much an the web about this comparison topic.
I have read that Openshift is used by more companies,but a nightmare to install,pricier and on upgrade data loss can occur.
But nothing else.
What should be the deciding factor for which one to use for kubernete cluster orchestration?
I currently work for Rancher. I've also been building Internet infrastructure since 1996 and owned an MSP for 14 years that built and managed Internet datacenters for large US media companies. I've been working with containers since 2014, and since then I've tried pretty much everything that exists for managing containers and Kubernetes.
"The deciding factor" varies by individual and organization. Many companies use OpenShift. Many companies use Rancher. Many companies use something else, and everyone will defend their solution because it fits their needs, or because of the psychological principle of consistency, which states that because we chose to walk a certain path, that path must be correct. More specifically, the parameters around the solution we chose must be what we need because that was the choice we made.
Red Hat's approach to Kubernetes management comes from OpenShift being a PaaS before it was ever a Kubernetes solution. By virtue of being a PaaS, it is opinionated, which means it's going to be prescriptive about what you can do and how you can do it. For many people, this is a great solution -- they avoid the "analysis paralysis" that comes from having too many choices available to them.
Rancher's approach to Kubernetes management comes from a desire to integrate cloud native tooling into a modular platform that still lets you choose what to do. Much like Kubernetes itself, it doesn't tell you how to do it, but rather gives fast access to the tooling to do whatever you want to do.
Red Hat's approach is to create large K8s clusters and manage them independently.
Rancher's approach is to unify thousands of clusters into a single management control plane.
Because Rancher is designed for multi-cluster management, it applies global configuration where it benefits the operator (such as authentication and identity management) but keeps tight controls on individual clusters and namespaces within them.
Within the security boundaries Rancher gives developers access to clusters and namespaces, easy app deployment, monitoring and metrics, service mesh, and access to Kubernetes features without having to go and learn all about Kubernetes first.
But wait! Doesn't OpenShift give developers those things too?
Yes, but often with Red Hat-branded solutions that are modified versions of open source software. Rancher always deploys unadulterated versions of upstream software and adds management value to it from the outside.
The skills you learn using software with Rancher will transfer to using that same software anywhere else. That's not always the case with skills you learn while using OpenShift.
There are a lot of things in Kubernetes that are onerous to configure, independent of the value of using the thing itself. It's easy to spend more time fussing around with Kubernetes than you do using it, and Rancher wants to narrow that gap without compromising your freedom of choice.
What is it that you want to do, not only now, but in the future? You say that you already know Kubernetes, but something has you seeking a management solution for your K8s clusters. What are your criteria for success?
No one can tell you what you need to be successful. Not me, not Red Hat, not Rancher.
I chose to use Rancher and to work there because I believe that they are empowering developers and operators to hit the ground running with Kubernetes. Everything that Rancher produces is free and open source, and although they're a business, the vast majority of Rancher deployments make no money for Rancher.
This forces Rancher to create a product that has true value, not a product that they can convince other people to buy.
The proof is in the deployments - Red Hat has roughly 1,000 OpenShift customers, which means roughly 1,000 OpenShift deployments. Rancher has fewer paying customers than Red Hat, but Rancher has over 30,000 deployments that we know about.
You can be up and running with Rancher in under ten minutes, and you can import the clusters you already have and start working with them a few minutes later. Why not just take it for a spin and see if you like it?
I also invite you to join the Rancher Users slack. There you will not only find a community of Rancher users, but you will be able to find other people who compared Rancher and OpenShift and chose Rancher. They will be happy to help you with information that will lead you to feel confident about whatever choice you make.
I am in an enterprise that creates many large to small scale applications. Way back when I started using Visual Studio Online/VSTS/now Azure devops I broke all of these apps out into different projects. However, now it seems there is good support for managing multiple apps in one project, and this seems easier from a management standpoint.
From the research I did it seems i can do each service individually:
Repos: Select Import Repository from the consolidated project and enter in the URL of the app i want to import BEWARE
Also, the system automatically resets the State and Reason fields to the default initial values for the work item type that you move.
and I skimmed over the docs and missed that snippet. oops
Boards: Go to query (optionally select query across project) and move all items to the consolidated project (creating the appropriate teams, areas and past sprints to keep history)
Builds: Export json from source projects and import to consolidated project
Releases: Same as Builds
(we do not use test plans or artifacts at the moment)
Doing it this way I probably will loose build and release history, which is probably not a big deal once enough time passes, but I did not find any other good way.
My two questions are:
Are there better guidelines to move entire projects to one project?
Will I permanently or temporary loose anything else besides the build and release history that I am not realizing?
I went through the same thing and had to do what you described. At the time there was no support for moving services/functions between Projects or Orgs at all, let alone consolidating into a single Project. Unless something's changed, short of automating via your own API scripting it's all manual.
The other watch outs were mainly around access and security:
External API integrations such as Web Apps, Function Apps, JIRA, Service Now
External inbound app authorisations
External outbound app authorisations such as Azure Service Principals
Variable Group authorisations to YAML Build Pipelines
Library reference updates including KeyVault
etc
This refactoring ended up being much more work than the platform consolidation itself.
Imagine there is an application consisting from bunch of microservices. All of these microservices can be developed/deployed completely independently from each other. Each microservice can be "described" with several attributes - e.g. current API version, release version, commit hash etc. Along with that, there are several environments used in development process - e.g. Testing environment (often called Sandbox), Staging environment, Pre-Release environment and obviously Production environment.
Is there a convenient tool/way/approach to track, basically, what attribute is currently deployed to which environment? For instance, get a quick access to information like "what is the current version of Restful API at Pre-Release environment"? Or more complex one - "what was this version two month ago"? And of course see the "global picture" as well?
Theres no ready to use solution on the market yet according to my knowledge.
Some teams are using git ops https://www.twistlock.com/2018/08/06/gitops-101-gitops-use/ to get ahead of the chaos challenge a lot of different micro services usually ship with.
Another technology in a somewhat different, yet related direction are micro service meshes, istio https://istio.io/ being one of them.
There are also test approaches like contract testing or heavy integration tests, that are more expensive, but also provide more confidence.
Getting Started with Bluemix, what were your first applications using the platform? They could advise me? And forgive anything for the question.
Thank you for attention.
I think it depends on what is your programming experience and your personal preference. If you want to begin programming with the classic Object-Oriented paradigm you should try Liberty runtime. Instead if you like the scripting languages maybe you could give a try to Python, Ruby or Go runtimes. All of them provide a sample application that you can extend as you want, and have very detailed documentation. I suggest you to take a look at IBM Containers, they are very interesting and powerful, and they make you able to do potentially anything with the platform.
If you come from the on-premise world, please notice that Bluemix is built on Cloud Foundry, and there are two important considerations to think about:
Local file system storage is short-lived. When an application instance crashes or stops, the resources assigned to that instance are reclaimed by the platform including any local disk changes made since the app started. When the instance is restarted, the application will start with a new disk image. Although your application can write local files while it is running, the files will disappear after the application restarts.
Instances of the same application do not share a local file system. Each application instance runs in its own isolated container. Thus if your application needs the data in the files to persist across application restarts, or the data needs to be shared across all running instances of the application, the local file system should not be used.
For this reason local file system should not be used.
Personally since I had some experience in JEE + WAS my first application was a Web app developed on Liberty Runtime.
I suggest you to become familiar with IBM Bluemix DevOps Services, that allows you to develop, build and deploy working on a web IDE.
So the various runtimes and services within bluemix provide 2 types of samples to help you get started. Boilerplate, which are samples you can extend to develop new applications and there is a "deploy to bluemix" button on other samples which can be used to automatically get the sample installed and ready to try. What you start with is usually something like this to see it working and then go from there.
This question already has been answered in a way. But I think what you are looking for are the types of applications you can develop using Bluemix.
To directly answer your question - the first application I developed used the Concept Insights Watson service to extract insights from some news articles and create a concept based news search. I also experimented with Language Translation service where-in I converted the contents of a web page from English to Spanish.
If you look through the documentation pages for the various Watson services, you would come across various use cases where a particular service is applicable.
On a more general note I can see that Bluemix would help us to write some really smart applications in an easy way. The Watson Services provide a real simple interface to all application developers by taking away the highly complex tasks of machine learning and AI algorithms which would need a good level of expertise if done on our own.
Additionally, Bluemix is just like any other PaaS for e.g AWS or Google Compute Engine or Azure. Bluemix does provide relational databases, queues, time series database containers etc as a part of the platform. These would need to be used by the application that you are developing to cater to use cases of inter-process communication, data storage etc.
Hopefully this answer provides you some insight on what applications you can write with the Bluemix concept insight services
Our local applications run on WAS ND. When we moved to trying Liberty on BlueMix(as application, not service), typical problems were mostly around 12 Factors.
Config && Dev/Prod parity --> Earlier our configurations were in files inside application or configured in WAS. With Liberty, we were forced to externalize that and it was easy to setup environment variables in BlueMix.
Process --> As state-fullness was no longer an option, we had to change our application to store session outside, in relational as well as document DB.
Logs --> Logs are not available in local logs files as earlier.
Ephemeral instances --> As mentioned by Umberto