I'm trying to figure out the precise processing order for templates within Helm (v3.8.1 and newer). Specifically, I'm looking for the order in which individual files are processed - if any.
I'm trying to figure out the best means to render some common-use values based on child charts' values. I have some subcharts that will need to interoperate, and I'm aware of the availability of the "global" map in order to publish values consumable by all charts. I've not tested modifying the contents of "global" programmatically, but I expect this should be possible.
Regardless, what I'm getting at is that using static variables defined in values.yaml (global, specifically, or even subcharts' value overrides) may not be enough in order to achieve the interoperation I need for all of these subcharts.
Specifically, I would like for each subchart, as it's processed, to compute specific values that could be consumed for interoperation and then publish them to the "global" map (i.e. global.subchart1.someValue, global.subchart2.someOtherValue, etc).
For instance: one of my subcharts is an LDAP provider, but an external LDAP could also be used, so the LDAP URL is something that needs to be "computed" to either be the external one (manually-specified), or the internal component's one.
This type of modification would allow me to consume those values (if available) wherever interoperation is required, via a single source of truth.
I'm looking for something like this:
Parse the top-level chart's Chart.yaml
Parse and process the top-level chart's non-template files (_* files)
Repeat the prior two steps for all subcharts, recursively
Parse the top-level chart's values.yaml
Parse and process/render the top-level chart's remaining template files
Repeat the prior two steps for all subcharts, recursively
Ideally, also including the order in which files are processed (I'm currently presuming they're processed alphabetically).
I realize this is probably not the ideal scenario, but I do want the ability for each subchart to be able to export information akin to "my service is exported on this port, with this user and password" (yes, I'm aware that secrets would likely be required here, etc...), or "this is the search DN for the LDAP users", etc.
Related
I am looking to not hardcode values in values.schema.json for validation and have it configurable from some charts or system environment variables.
Is this possible with helm3. Not able to see anything related to this in documentation.
Like for object type array I don't want to hardcode values of minItems and maxItems and use it from some configurable input.
I have several variable groups defined under Pipeline Library. I'm using a different group for each release environment (e.g., DEV, QA, PROD). The deployment for each environment needs all the same variables but with different values.
It's a long list of variables and I have no way to tell if my groups are getting out of sync with one another over time as we add and remove variables.
I really wish I had something like a C# interface to make all the groups match. I was hoping the new YAML templates could help somehow but I can't find anything that would solve this.
Anyone have a good solution to this? It seems like a very common use of variable groups.
Can I enforce the same variables across variable groups in the Azure
DevOps Pipeline Library?
No, we can't do that. It seems what you want is to sync the variable groups for DEV,QA,PROD environments, making these groups have same variable list but different values.
As I know this is not supported for now. Variable groups is originally designed to share variables across pipelines instead of other directions, so Azure Devops Service doesn't support such out-of-box feature to compare and sync variable groups.
Update1:
I submitted a feature request in User Voice forum that you can vote and track.
Update2:
1.Another direction if you want to check the variable list easily: If you want to check the variable list manually, you can consider using Rest API to get a Json response of your variable list for one specific variable group.
2.There're many related topics about how to convert Json to CSV/Excel file online, and after my test it's quite convenient. So maybe you can get three josn response and then convert then into text file/excel file to compare the difference between these three variable lists. It could be a temporary workaround ~
Hope it helps :)
I am experimenting with Azure Data Factory to replace some other data-load solutions we currently have, and I'm struggling with finding the best way to organize and parameterize the pipelines to provide the scalability we need.
Our typical pattern is that we build an integration for a particular Platform. This "integration" is essentially the mapping and transform of fields from their data files (CSVs) into our Stage1 SQL database, and by the time the data lands in there, the data types should be set properly and the indexes set.
Within each Platform, we have Customers. Each Customer has their own set of data files that get processed in that Customer context -- within the scope of a Platform, all Customer files follow the same schema (or close to it), but they all get sent to us separately. If you looked at our incoming file store, it might look like (simplified, there are 20-30 source datasets per customer depending on platform):
Platform
Customer A
Employees.csv
PayPeriods.csv
etc
Customer B
Employees.csv
PayPeriods.csv
etc
Each customer lands in their own SQL schema. So after processing the above, I should have CustomerA.Employees and CustomerB.Employees tables. (This allows a little bit of schema drift between customers, which does happen on some platforms. We handle it later in our stage 2 ETL process.)
What I'm trying to figure out is:
What is the best way to setup ADF so I can effectively manage one set of mappings per platform, and automatically accommodate any new customers we add to that platform without having to change the pipeline/flow?
My current thinking is to have one pipeline per platform, and one dataflow per file per platform. The pipeline has a variable, "schemaname", which is set using the path of the file that triggered it (e.g. "CustomerA"). Then, depending on file name, there is a branching conditional that will fire the right dataflow. E.g. if it's "employees.csv" it runs one dataflow, if it's "payperiods.csv" it loads a different dataflow. Also, they'd all be using the same generic target sink datasource, the table name being parameterized and those parameters being set in the pipeline using the schema variable and the filename from the conditional branch.
Are there any pitfalls to setting it up this way? Am I thinking about this correctly?
This sounds solid. Just be aware that you if you define column-specific mappings with expressions that expect those columns to be present, you may have data flow execution failures if those columns are not present in your customer source files.
The ways to protect against that in ADF Data Flow is to use column patterns. This will allow you to define mappings that are generic and more flexible.
I want to generate a unique identifier of a specific length and use this value across multiple pods internally. Since the length must be specific, and I'd prefer this to be handled internally rather than be adjustable by a user, I'd prefer to create the unique identifier on install/upgrade (only once if has not been set already) and not be changeable.
I want to use the identifiers internally as part of a naming schema for objects created within a specific deployment. I want to share these objects across other deployments, and need the identifier to determine if a given object belongs to a given deployment.
I was looking into setting a value in Secrets using randAlphaNum. Some problems I face with using Secrets are:
Related to this issue: https://github.com/helm/helm/issues/3053
It looks like the Secret value will be overwritten on upgrade. There is an open PR for a possible fix: https://github.com/helm/helm/pull/5290
But I don't have the ability to upgrade helm/kubernetes atm
Secret value is b64 encoded. I want to pass the value as an environment variable to various pods decoded. It doesn't really matter if the user knows the unique identifier. So, maybe I don't need a Secret? But, again, I don't want the user to be able to edit the value and the value should never change for a given deployment.
Any help or suggestions are appreciated! Thanks
You may then try to use ConfigMap instead. Seems it doesn't change on helm upgrade. Then you can use this guide to pass the value from ConfigMap to the pods.
I am using Rundeck for basic API calls periodically or manually. As of now, I have a lot of options the user has to provide, but most of them remains empty.
In fact, some of them depends on a previous one (e.g do you want X ? Then provide Y). Is there a way to display some options only if another has a given value ?
I would like not to create two different jobs but to keep only one.
One possible way is with Cascading Remote Options
This provides a mechanism for declaring hierarchical or dependent sets of option values.