With Kubernetes you can use the Garbage Collector to automate the deletion of dependent resources when owning resources are removed. I'm wondering the easiest method to print out the dependency tree of an owning resource, potentially limiting to a tree depth if needs be.
I understand the potential for crashing the API service given the ability to fan out to all resources in a cluster and likely why this isn't an easy feat to achieve but I've been struggling to even find usable, community supported workarounds or even discussions/issues relating to this topic (likely my poor searching skills) so any help in achieving this would be great!
To make things more concrete a specific example of an abstract kubectl get query I'd like to achieve would be something like kubectl get scheduledworkflow <workflow name> --dependents:
This would find the Kubeflow Pipelines ScheduledWorkflow resource then recurse,
That would find all Argo Workflow resources,
Then for each Workflow resource many Pod and Volume resources (there are a few other types but wanted to paint the picture of these being disparate resource types).
We typically only keep a small number of Argo Workflow resources in the cluster at anyone one time as the majority of our Workflow's spawn 1k+ Pod so we have pretty aggressive GC policies in place. Even so listing these is just painful at the moment and need to use a custom script to do it but wondering if there was a higher level CLI, SDK or API available (or any group working on this issue in the community!).
There are no ready solutions for this.
I see two options how this can be proceeded:
1 - probably this is what you already mentioned: "need to use a custom script to do it".
Idea is to get jsons of required resource groups and then process it by any available/known language like bash/python/java/etc and/or using jq. All dependent objects have ownerReference field which allows to match resources.
More information about owners and dependents
jq tool and examples
2 - Write your own tool based on kubernetes garbage collector
Kubernetes garbage collector works based on graph built by GraphBuilder:
garbage collector source code
Graph is always up to date by using `reflectors:
GarbageCollector runs reflectors to watch for changes of managed API
objects, funnels the results to a single-threaded
dependencyGraphBuilder, which builds a graph caching the dependencies
among objects
graph_builder source code to get whole logic of it.
Built graph has node type:
graph data structure
Also it's worth to mention that working with api server is more convenient using kubernetes clients libraries which are available for different languages.
Related
my team needs to decide if we go for buy or create on a production data processing workflow framework. The service needs to be performant, receives documents (like images, pdfs) and processes the document by sending it to a predefined workflow definition (orchestration should not be expensive in terms of performance). Additionally we would like to deploy it on AKS and Models are running in docker containers. We checked and used Kubeflow - but it seems not ready for production processes and with airflow we have similar doubts. All we need is a framework that allows various processes by defining the order of the calls to our models in docker containers (sequentially and parallel) in a config file. Anyone who has experience and can guide us to the right (and hopefully simple) framework / tool? Anyone who has such a component in prod live?
I am looking for keeping some kind of baseline for everything applied to kubernetes(or just a namespace).
Like versioning microservices in a list and then check that in to github, in case of the need to roll something back.
Check out Velero, it is a backup tool for kubernetes. I don’t think it can use git as a backend, but you could add that (or use s3 or similar).
You can write and deploy an application that Watch the resources you are interested in, e.g. all Deployment, Service and Ingress... including all changes, and then store the changes as you want. I can recommend client-go for this kind of service.
However
Like versioning microservices in a list and then check that in to github, in case of the need to roll something back.
It is more common, and recommended to work the other way around, first save your intended desired state in Git, have an CICD service to apply your changes in the cluster or clusters. This way of working is called Infrastructure as Code. The new edition of the book Kubernetes Up&Running have a new chapter (18) that describes how to work in this way.
I am currently pending between using terraform and CloudFormation.
There is a question I haven't seen the answer yet (or maybe, I just haven't found it yet).
In terraform, you give a precise name to everything. This will delete the targets with those names.
But what about CF? If we already have an architecture in place and I want to add/delete an instance and use CF, how will this work? How will it know after which one to target?
I hope this question makes sense! I've already used terraform, but never before CloudFormation.
CloudFormation uses two mechanisms to identify its resources. The CFN template has a list of resources it created, it uses the actual ID, not a pretty name, and CFN also tags the resources (that support tags) with the stack ID.
CFN cannot be used to delete the resources in a different stack, only the stack that created them can manage them. Terraform allows you to import resources created by anything else into a new stack where they will be managed.
I used CFN for a year before converting to Terraform (also for a year now) and I'll never go back to CFN. Terraform offers many advantages over CFN that make CFN really hard to use now. Features such as plan before apply, re-usable modules, resource imports, granular output (CFN is mostly a black box), and generally faster AWS feature support (usually APIs are released at launch day and Terraform support follows soon after, /usually/ faster than CFN but not always).
We are currently serving around 140 webapps created by a bunch of different web agencies. The setup is the usual LEMP stack.
A 1.2 k8s cluster has been installed to migrate them as micro-services.
The problem we are facing is about serving static and dynamic content.
For this purpose we use, of course, two different containers (nginx and php-fpm) but we can't find an adequate solution to share data on both of them.
We hoped to be able to use versioned data containers but it is apparently not in the scope of k8s. Too bad.
gitRepo is not an option as we don't want to be dependent of a working git infra to instance pods. If it doesn't work we want to be autonomous and be able to serve traffic.
The other options (flocker, etc.) look heavy and complex in comparison to a simple data container. We also would like to be independent of data storage.
Is there an option I am not aware of? Does anyone have an advise on this?
Let me emphasise that we want to be able to version things in order to roll forward / backward easily.
Thank you for your time
In cq we can create live copies by either blueprints by opting "New Site..." or directly trough content nodes by opting for "New Live Copy...".
In both the cases inheritance is maintained and roll-out works too in same ways. So what is the advantage of using one over other.
Any views?
Live Copies
Live copies can be created for just a simple page or a tree of pages and might the page and it's subpages depending on the rollout configuration. A live copy can be linked to a rollout config or will use the system's default one.
There is no formal requirement on the source page's structure.
A live copy might reference a blueprint, while it can only reference to a single blueprint.
Blueprints
Blueprints target the rollout of complete multilingual website projects and are a tool to control multiple rollout configs and live copies.
A blueprint requires a certain structure for the source site:
- One root level page
- The root's immediate childs define the language branches of the site
- each language contains one or more child pages.
Blueprints allow you to control multiple live copies and centrally consistent rollout configs for the blueprint's live copies.
A blueprint rollout will push modifications to all it's live copies.
Usage scenarios of blueprints
Inheritance and rollout work the same way. Just because blueprint make use of live copies.
But blueprints help you to organize your rollout scenarios for large multilingual sites. Just imagine a corporate website that provides a two or even three digit number of locales which that need to be translated and kept in sync.
In such a scenario you will likely end up with a hardly understandable and maintainable number of live copy and rollout configurations.
Depending on a blueprint to e.g. standardize the rollout of a new language/market/locale provides you higher degree of governance over your process as the complete process centrally manageable through the blueprint template.
But as long as you do not have such a scenario you might be fine without having the complete blueprint overhead.
A Livecopy is defined in the target page node with a cq:LiveSyncConfig node. It basically defines: I am a livecopy of source (blueprint) page X, and the following rollout configs apply.
A Blueprint is defined in the source page node with a cq:BlueprintSyncConfig node, and this defines a target.
Essentially both achieve the same in the end, but I think there are a few differences: the first option can be used to create a 1:n relationship, whereas the second option is 1:1
Also, if page nodes are copy-pasted in AEM, then relationships are copied with them (not quite sure in which way exactly, you would have to try for both scenarios). Also when pages are deleted in a tree in the first scenario, AEM will add a cq:excludedPaths property to the config which causes the page to be skipped in future rollouts - not sure this is the same for cq:BlueprintSyncConfig as well.