I'm attempting to add some recommended labels to several k8s resources, and I can't see a good way to add labels for things that would change frequently, in this case "app.kubernetes.io/instance" and "app.kubernetes.io/version". Instance seems like a label that should change every time a resource is deployed, and version seems like it should change when a new version is released, by git release or similar. I know that I could write a script to generate these values and interpolate them, but that's a lot of overhead for what seems like a common task. I'm stuck using Kustomize, so I can't just use Helm and have whatever variables I want. Is there a more straightforward way to apply labels like these?
Kustomize's commonLabels transformer is a common way to handle this, sometimes via a component. It really depends on your overall layout.
Related
I have a Kubernetes controller written using client-go informers package. It maintains a watch on all Pods in the cluster, there are about 15k of these objects, and their YAML representation takes around 600 MB to print (I assume their in-memory representation is not that different.)
As a result, this (otherwise really small) controller watching Pods ends up with a huge memory footprint (north of 1 GiB). Even methods that you'd think offer a way of filtering, such as the one named like NewFilteredSharedInformerFactory doesn't really give you a way to specify a predicate function that chooses which objects are stored in the in-memory cache.
Instead, that method in client-go offers a TweakListOptionsFunc. It helps you control ListOptions but my predicate unfortunately cannot be satisfied with a labelSelector or fieldSelector. I need to drop the objects when they arrive to the controller through a predicate function.
Note: the predicate I have is something like "Pods that have an ownerReference by a DaemonSet" (which is not possible with fieldSelectors –also another question of mine) and there's no labelSelector that can work in my scenario.
How would I go about configuring an informer on Pods that only have DaemonSet owner references to reduce the memory footprint of my controller?
Here's an idea, you can get a list of all the DaemonSets in your cluster. Read the spec.selector.matchLabels field to retrieve the label that the DaemonSet pods are bound to have. Use those labels as part of your TweakListOptionsFunc with a logic like:
Pods with label1 OR label2 OR label3 ...
I know it's a bit of toil, but seems to be a working approach. I believe there isn't a way to specify fields in client-go.
It appears that today if you use SharedInformers, there's no way to filters which objects to keep in the shared cache and which ones to discard.
I have found an interesting code snippet in kube-state-metrics project that opts into the lower-layer of abstraction of initiating Watch calls directly (which would normally be considered as an anti-pattern) and using watch.Filter, it decides whether to return an object from a Watch() call (to a cache/reflector or not).
That said, many controller authors might choose to not go down this path as it requires you to specify your own cache/reflector/indexer around the Watch() call. Furthermore, projects like controller-runtime don't even let you get access to this low-level machinery, as far as I know.
Another aspect of reducing controllers' memory footprint can be done through field/data erasure on structs (instead of discarding objects altogether). This is possible in newer versions of client-go through cache.TransformFunc, which can let you delete fields of irrelevant objects (though, these objects would still consume some memory). This one is more of a band-aid that can make your situation better.
In my case, I mostly needed to watch for DaemonSet Pods in certain namespaces, so I refactored the code from using 1 informer (watching all namespaces) to N namespace-scoped informers running concurrently.
The usecase
We got multiple changelogs stored in the database, and want to create a histogram monitoring the duration between changes.
The problem
There doesn't seem to be a way to set the start time of a Historgram.Timer, e.g we want to set it to lastUpdated given the current changelog.
Avenues of approach
1 Subclassing Histogram
Should work. However the java-lib use protected/package-private extensively, thus making it hard without copying large portions of the library.
2 Using reflection
After a Histogram.Timer is created it should be possible to use reflection to set the start field. The field is marked as private final, and thus a SecurityManager could stop us in some environments.
Ideas?
Neither of the solutions seems like the correct way to go, and I suspect that I'm overlooking a simpler solution (but could find anything at SO or google). We're using grafana to visualize our metrics, if thats at all helpful in this scenario.
You don't need to subclass Histogram, as you don't need to use Histogram.Timer only because your histogram is measuring times.
Simply call myHistogram.observe(System.now() - lastUpdated) every time you record a new change in the database.
I have a public key that all my pods needs to have.
My initial thought was to create a ConfigMap or Secret to hold it but as far as I can tell neither of those can be used across namespaces. Apart from that, it's really boiler plate to paste the same volume into all my Deployments
So now I'm left with only, in my opinion, bad alternatives such as creating the same ConfigMap/Secret in all Namespaces and do the copy-paste thing in deployments.
Any other alternatives?
Extra information after questions.
The key doesn't need to be kept secret, it's a public key, but it needs to be distributed in a trusted way.
It won't rotate often but when it happens all images can't be re-built.
Almost all images/pods needs this key and there will be hundreds of images/pods.
You can use Kubernetes initializers to intercept object creation and mutate as you want. This can solve copy-paste in all your deployments and you can manage it from a central location.
https://medium.com/google-cloud/how-kubernetes-initializers-work-22f6586e1589
You will still need to create configmaps/secrets per namespace though.
While I don't really like the idea, one of the ways to solve it could be an init container that populates a volume with key(s) you need and then these volumes can be mounted in your containers as you see fit. That way it becomes independent of how you namespace stuff and relies only on how pods are defined/created.
That said, the Kubed mentioned by Ryan above sounds like more reasonable approach to a case like this one, and last but not least, something creates your namespaces after all, so having the creation of required elements of a namespace inside the same process sounds legit as well.
I'm developing a script that performs a certain core task, and using versions of that script in two different environments where some settings and steps along the way need to be different. What I am looking for is whether there exists an elegant way to handle the small differences between the two versions of the script. I'm sure developers face similar problems when developing software to be deployed on multiple platforms, but I don't have a specific name to pin on it.
What I do now is to open up the second script and manually replace the lines that need to be different. This is cumbersome, time-consuming, and a bit of a headache whenever I inevitably forget to comment out a line or change a string.
Example
[...]
path_to_something = "this/is/different"
use_something(path_to_something)
[...]
do_thing_A() # Only in environment A.
[...]
do_thing_B() # Only in environment B.
[...]
The omitted [...] parts are identical in both versions, and when I make a change to them, I have to either copy and paste each changed line, or if the changes are significant, copy the whole thing, and manually change the A and B parts.
Some ideas for possible solutions that I've come up with:
Write a script that automates the steps I manually take when moving the code back and forth. This exactly replicates the necessary steps, and it's quick and easy to add or remove steps as necessary.
Is this a use case for gitattributes?
Factor all the code that is identical between versions into separate files, so that the files containing the heterogenous code don't need to change at all, and thus don't need to be version-controlled, per se.
Some other tool or best practice that I don't know about to handle this type of workflow.
Looking around, I've found a question with a similar premise of maintaining different versions of code that does the same thing:
Proper way to maintain a project that meets two versions of a platform?
Solutions offered to that question:
Get rid of all the differences, then there is no problem to solve. This may or may not be possible in my specific case, and certainly won't be possible in every case for everyone in the future. So maybe there is a more general solution.
Maintain two different branches of the code, even though they are nearly identical. This is similar to what I do now, but I end up having to do a lot of copying and pasting back and forth between branches. Is that just inherent to software development?
Perform platform detection and wrap the differences in conditionals. This adds a lot of ugly stuff in the code, but if I could successfully detect the environment and implement all the necessary differences conditionally, I would not have to make any changes to the code before sending it to the different environments.
How do developers move code back and forth between similar, but different, parallel branches of a project?
Language- and SCM-agnostic
Use one (two, three) common class(es), but different interfaces (interface per environment)
Move all env-specific from hardcode into external configuration, stored separately
SCM-agnostic
Separate tasks, i.e.:
Get clean common core
Get changes on top core for every and each environment
Move it into $ENVS+1 branches in any SCM (Core+...+EnvN)
Changed workflow become
Commit common changes to Core
Merge Core to each Env-branch
Test env-branches, fix env-specific changes if needed
Private and personal, preferred for me and my habits
Variation of branch-tree solution
Pure Mercurial-way, because I'm too lazy to maintain Env-specific branches
Mercurial, one branch + MQ-queue with a set of patches (one mq-patch per Env, may be "set of queues /queue per Env/, one patch per queue")
Common code stored in immutable changesets, any changes, which convert plain core into environment-specific products, stored in patches and applied on demand (and edited when|if it needed).
Small advantages over branch-way - single branch, no merges, more
clean history
Disadvantage - pure Mercurial, now trendy Git (git-boys
are crying)
In the application I'm working on, a user can perform a "transition" which consists of "steps". A step can have an arbitrary number of dependencies on other steps. I'd like to be able to call a transition and have the steps execute in parallel as separate Celery tasks.
Ideally, I'd like something along the lines of celery-tasktree, except for directed acyclic graphs in general, rather than only trees, but it doesn't appear that such a library exists as yet.
The first solution that comes to mind is a parallel adaptation of a standard topological sort - rather than determining a linear ordering of steps which satisfy the dependency relation, we determine the entire set of steps that can be executed in parallel at the beginning, followed by the entire set of steps that can be executed in round 2, and so on.
However, this is not optimal when tasks take a variable amount of time and workers have to idle waiting for a longer running task while there are tasks that are now ready to run. (For my specific application, this solution is probably fine for now, but I'd still like to figure out how to optimise this.)
As noted in https://cs.stackexchange.com/questions/2524/getting-parallel-items-in-dependency-resolution, a better way is operate directly off the DAG - after each task finishes, check whether any of its dependent tasks are now able to run, and if so, run them.
What would be the best way to go about implementing something like this? It's not clear to me that there's an easy way to do this.
From what I can tell, Celery's group/chain/chord primitives aren't flexible enough to allow me to express a full DAG - though I might be wrong here?
I'm thinking I could create a wrapper for tasks which notifies dependent tasks once the current task finishes - I'm not sure what the best way to handle such a notification would be though. Accessing the application's Django database isn't particularly neat, and would make it hard to spin this out into a generic library, but Celery itself doesn't provide obvious mechanisms for this.
I also faced this problem but i couldn't really find a better solution or library except for one library, For anyone still interested, you can checkout
https://github.com/selinon/selinon. Although its only for python 3, It seems to be the only thing that does exactly what you want.
Airflow is another option but airflow is used in a more static environment just like other dag libraries.