I have a parent cloudformation script that launches two child cloudformation scripts, each in separate accounts. Is there a way I can get outputs from one of the child stacks and use them as inputs in the other child stack, all done from this parent template? The user should be able to input different account numbers as a parameter as this script will be run across several different accounts.
Parent template sample code:
Resources:
ChildAccountA:
Type: Custom::StackA
Properties:
ServiceToken: example
TemplateURL: https://s3.amazonaws.com/exampleA
ChildAccountB:
Type: Custom::StackB
Properties:
ServiceToken: example
TemplateURL: https://s3.amazonaws.com/exampleB
If I correctly understand, then yes. Your custom resource lambda should return the outputs to the parent stack.
The custom lambda obviously needs to have correct permissions to be able to deploy stacks in other accounts and get their outputs to be returned to the parent stack.
Since you haven't written if there are any issues with development of custom resources, I will just put the link to AWS crhelper. The helper simplify development of custom resources and returning results to it.
Just for completeness, for deployment of stacks in different regions and accounts, StackSets.
I am having problems using SSM valueForStringParameter method in CDK. It's working the first time I deploy the stack, but it is not picking up updates to the parameter value when I redeploy the stack because CloudFormation template hasn't changed and so CloudFormation thinks there were no updates, even if SSM parameter has changed.
For the context, I am deploying stack via CodePipeline, where I run cdk synth first, and then use CloudFormationCreateUpdateStackAction action to deploy template.
Anyone knows how to work around that? The only other option that I know will work is to switch to a custom resource lambda that calls SSM and returns value using aws-sdk, but that feels like a overly complicated option.
Update 1
I cannot use ValueFromLookupbecause value is only updated at runtime as part of cloudformation deployment by another stack (I deploy both stacks in CodePipeline, in 2 different regions), so synthesis time lookup would result in stale value.
All the valueOf* and from* methods work by adding a CloudFormation parameter. As you figured out already, changing the parameter value does not change the template and no change will be triggered.
What you probably want to use instead is the method valueFromLookup. Lookups are executed during synth and the result is put into the generated CFN template.
ssm.StringParameter.valueFromLookup(this, 'param-name');
But be aware, lookups are stored in the cdk.context.json. If you have commited that file to your repo, you need to erase that key via cdk context -e ... before synth/diff/deploy.
Since you cannot use lookup functions and the most common way to pass config to cdk is through context variables, I can only suggest dirty workarounds.
For example, you could create a dummy parameter in your stack to bump every time there's deployment.
var deploymentId = new CfnParameter(this, "deploymentId", new CfnParameterProps() { Type = "String", Description = "Deployment Id" });
SetParameterValue(deploymentId, this.Node.GetContext("deploymentId").ToString());
and when you synthesize the CF, you could generate an ID:
cdk synth -c deploymentId=$(uuidgen)
If you can avoid the "environment agnostic" syth and you really need an immutable artifact to deploy across multiple environments, you could use the built package from your cdk, for example, the npm package containing your cdk. Therefore, you could deploy it in each environment by overwriting the context parameters instead of using ssm parameters store.
See https://docs.aws.amazon.com/cdk/latest/guide/get_ssm_value.html, you can use method valueFromLookup which gets you parameter store value at synthesis time, when value is different from previous one, this shall trigger CF stack update.
However, I was under impression that valueForStringParameter should work on updated ssm parameter values as well, based on https://aws.amazon.com/blogs/mt/integrating-aws-cloudformation-with-aws-systems-manager-parameter-store/ Example 2:
In the kubectl Cheat Sheet (https://kubernetes.io/docs/reference/kubectl/cheatsheet/), there are 3 ways to modify resources. You can either update, patch or edit.
What are the actual differences between them and when should I use each of them?
I would like to add a few things to night-gold's answer. I would say that there are no better and worse ways of modifying your resources. Everything depends on particular situation and your needs.
It's worth to emphasize the main difference between editing and patching namely the first one is an interactive method and the second one we can call batch method which unlike the first one may be easily used in scripts. Just imagine that you need to make change in dozens or even a few hundreds of different kubernetes resources/objects and it is much easier to write a simple script in which you can patch all those resources in an automated way. Opening each of them for editing wouldn't be very convenient and effective. Just a short example:
kubectl patch resource-type resource-name --type json -p '[{"op": "remove", "path": "/spec/someSection/someKey"}]'
Although at first it may look unnecessary complicated and not very convenient to use in comparison with interactive editing and manually removing specific line from specific section, in fact it is a very quick and effective method which may be easily implemented in scripts and can save you a lot of work and time when you work with many objects.
As to apply command, you can read in the documentation:
apply manages applications through files defining Kubernetes
resources. It creates and updates resources in a cluster through
running kubectl apply. This is the recommended way of managing
Kubernetes applications on production.
It also gives you possibility of modifying your running configuration by re-applying it from updated yaml manifest e.g. pulled from git repository.
If by update you mean rollout ( formerly known as rolling-update ), as you can see in documentation it has quite different function. It is mostly used for updating deployments. You don't use it for making changes in arbitrary type of resource.
I don't think I have the answer to this but I hope this will help.
All three methods do the same thing, they modify some resources configuration but the command and way to it is not the same.
As describe in the documentation:
Editing is when you open the yaml configuration file that is in the kubernetes cluster and edit it (with vim or other) to get direct modification on your cluster. I would not recommand this outside of testing purpose, reapplying conf from orignal yaml file will delete modificaitons.
Patching seems the same to me, but without opening the file and targetting specific parts of the resources.
Updating in the documentation it seems that's it's all other method to update a resource without using patch or edit. Some of those can be used for debug/testing, for example forcing a resource replace, or update an image version. Others are used to update them with new configurations.
From experience, I only used editing and some command of update for testing, most of time I reapply the configurations.
In my Octopus project, the first step launches a bunch of nested stacks implemented with cloudformation.
I need to share the outputs of the master stack launched from Octopus, how can I do that?
Thanks.
The output variables from the CloudFormation template will be available to later steps the same as any other Octopus output variable, this is mentioned in the first paragraph of the documentation page.
Output variables can be accessed a number of different ways, depending on where you are accessing them, for example, in Powershell they can be accessed via the parameters dictionary $OctopusParameters["Octopus.Action[Step Name].Output.VariableName"].
You can also access them using the Variable Binding syntax, #{Octopus.Action[Step Name].Output.VariableName}
More information about output variables is available in the docs.
Problem
Your organization has many separate applications, some of which interact with each other (to form "systems"). You need to deploy these applications to separate environments to facilitate staged testing (for example, DEV, QA, UAT, PROD). A given application needs to be configured slightly differently in each environment (each environment has a separate database, for example). You want this re-configuration to be handled by some sort of automated mechanism so that your release managers don't have to manually configure each application every time it is deployed to a different environment.
Desired Features
I would like to design an organization-wide configuration solution with the following properties (ideally):
Supports "one click" deployments (only the environment needs to be specified, and no manual re-configuration during/after deployment should be necessary).
There should be a single "system of record" where a shared environment-dependent property is specified (such as a database connection string that is shared by many applications).
Supports re-configuration of deployed applications (in the event that an environment-specific property needs to change), ideally without requiring a re-deployment of the application.
Allows an application to be run on the same machine, but in different environments (run a PROD instance and a DEV instance simultaneously).
Possible Solutions
I see two basic directions in which a solution could go:
Make all applications "environment aware". You would pass the environment name (DEV, QA, etc) at the command line to the app, and then the app is "smart" enough to figure out the environment-specific configuration values at run-time. The app could fetch the values from flat files deployed along with the app, or from a central configuration service.
Applications are not "smart" as they are in #1, and simply fetch configuration by property name from config files deployed with the app. The values of these properties are injected into the config files at deploy-time by the install program/script. That install script takes the environment name and fetches all relevant configuration values from a central configuration service.
Question
How would/have you achieved a configuration solution that solves these problems and supports these desired features? Am I on target with the two possible solutions? Do you have a preference between those solutions? Also, please feel free to tell me that I'm thinking about the problem all wrong. Any feedback would be greatly appreciated.
We've all run into these kinds of things, particularly in large organizations. I think it's most important to manage your own expectations first, and also ask whether it's really necessary to tell every system and subsystem on a given box to "change to DEV mode" or "change to PROD mode". My personal recommendation is as follows:
Make individual boxes responsible for a different stage - i.e. "this is a DEV box", and "this is a PROD box".
Collect as much of the configuration that differs from box to box in one location, even if it requires soft links or scripts that collect the information to then print out.
A. This way, you can easily "dump this box's configuration" in two places and see what differs, for example after a new deployment.
B. You can also make configuration changes separate from software changes, at least to some degree, which is a good way to root out bugs that happen at release time.
Then have everything base its configuration on something/somewhere that is not baked-in or hard-coded - just make sure to collect and document it in that one location. It almost doesn't matter what the mechanism is, which is a good thing, because some systems just don't want to be forced to use some mechanisms or others.
Sorry if this is too general an answer - the question was very general. I've worked in several large software-based organizations before, and this seemed to be the best approach. Using a standalone server as "one unit of deployment" is the most realistic scenario (though sometimes its expensive), since applications affect each other, and no matter how careful you are, you destabilize a whole system when you move any given gear or cog.
The alternative gets very complex very quickly. You need to start rewriting the applications that you have control over in order to have them accept a "DEV" switch, and you end up adding layers of kludge to the ones you don't have control over. Usually, the ones you don't have control over at least base their properties on something defined on a system-wide level, unless they are "calling the mothership for instructions".
It's easier to redirect people to a remote location and have them "use DEV" vs "use PROD" than it is to "make this machine run like DEV" vs "make this machine run like PROD". And if you're mixing things up, like having a DEV task run together on the same box as a PROD task, then that's not a realistic scenario anyways: I guarantee that eventually you will be granting illegal DEV-only access to somebody on PROD, and you'll have a DEV task wipe out a PROD database.
Hope this helps. Let me know if you'd like to discuss more specifics involved.
I personally prefer solution 2 (the app should know itself, by its configuration, what environment it is running in). With solution 1 (pass the environment name as a startup parameter) the danger of using the wrong environment specifier is much too high. Accessing the TEST database from PROD code and vice versa may cause mayhem, if the two installed code bases are not of the same version, as is often the case.
My current project uses solution 1, but I don't like that. A previous project I worked on used a variation of solution 2: The build process generated one setup file for every environment, making sure that they contained the same code base but appropriate configuration paramters. That worked like a charm, but I know it contradicts the paradigm that the "exact same build files must be deployed everywhere".
I think I have asked a related, self-answered, question, before I read this one : How to organize code so that we can move and update it without having to edit the location of the configuration file? . So, on that basis, I provide an answer here. I don't like the idea of "smart" application (solution 1 here) for such a simple task as finding environment settings. It seems a complicated framework for something that should be simple. The idea of an install script (solution 2 here) is powerful, but it is useful to allow the user to change the content of the config file, but would it allow to change the location of this config file? What is this "central configuration service", where is it located? My answer is that I would go with option 2, if the goal is to set the content of the configuration file, but I feel that the issue of the location of this configuration file remains unanswered here.
If you're using JSON to store/transmit configuration (or can use JSON in your pre-deploy process to output to some other format) you can annotate key/property names for environment/context-specific values with arbitrary or environment-specific suffixes, and then dynamically prefer/discriminate them at build/deploy/run/render -time, while leaving un-annotated properties alone.
We have used this to avoid duplicating entire configuration files (with the associated problems well known) AND to reduce repetition. The technique is also perfect for internationalization (i18n) -- even within the same file, if desired.
Example, snippet of pre-processed JSON config:
var config = {
'ver': '1.0',
'help': {
'BLURB': 'This pre-production environment is not supported. Contact Development Team with questions.',
'PHONE': '808-867-5309',
'EMAIL': 'coder.jen#lostnumber.com'
},
'help#www.productionwebsite.com': {
'BLURB': 'Please contact Customer Service Center',
'BLURB#fr': 'S\'il vous plaît communiquer avec notre Centre de service à la clientèle',
'BLURB#de': 'Bitte kontaktieren Sie unseren Kundendienst!!1!',
'PHONE': '1-800-CUS-TOMR',
'EMAIL': 'customer.service#productionwebsite.com'
},
}
... and post-processed (in this case, at render time) given dynamic, browser-environment-known location.hostname='www.productionwebsite.com' and navigator.language of 'de'):
prefer(config,['www.productionwebsite.com','de']); // prefer(obj,string|Array<string>)
JSON.stringify(config); // {
'ver': '1.0',
'help': {
'BLURB': 'Bitte kontaktieren Sie unseren Kundendienst!!1!',
'PHONE': '1-800-CUS-TOMR',
'EMAIL': 'customer.service#productionwebsite.com'
}
}
If a non-annotated ('base') property has no competing annotated property, it is left alone (presumably global across environments) otherwise its value is replaced by an annotated value, if the suffix matches one of the inputs to the preference/discrimination function. Annotated properties that do not match are dropped entirely.
You can mix and match this behaviour to annotate configuration to achieve distinctions of global, default, specific that are (assuming you're sensible) readable with zero/minimal duplication.
The single, recursive prefer() function (as we're calling it, lacking the need or desire to make an entire project/framework out of it) we've developed so far (see jsFiddle, with inline docs) goes a bit further than this simple example, and (explained in greater detail here) handles deeply-nested configuration objects, as well as preferential ordering and (if you need to stay flat) combination of suffixes.
The function relies on JS ability to reference object properties as strings, dynamically, and tolerate # and & delimiters in property names which are not valid in dot-notation syntax but consequently (help) prevent developers from breaking this technique by accidentally referring to pre-processed/annotated attributes in code (unless they, non-conventionally don't prefer to use dot-notation.)
We have yet to have this break anything for us, nor have we been schooled on any fundamental flaws of this technique, beyond irresponsible/unintended usage or investment/fondness for existing frameworks/techniques that pre-exist. We have also not profiled it for performance (we only tend to run this once per build/session, etc.) so in your own usage, YMMV.
Most configurations transmitted client-side of course would not want to contain sensitive pre-production values, so one could (should!) use the same function to generate a production-only version (with no annotations) in pre-deploy, while still enjoying a SINGLE configuration file upstream in your process.
Further, if you're doing this for i18n, you may not want the entire wad going over the wire, so could process it server-side (cached or live, etc.) or pre-process it in build/deploy by splitting into separate files, but STILL enjoying a single source of truth as early in your workflow as possible.
We have not explored implementing the same function in Java (or C#, PERL, etc.) assuming it's even possible (with some exotic reflection maybe?) but a build environment that includes NodeJS could farm that step out easily.
Well if it suits your needs and you have no problem of storing the connection strings in the source control repository, you could create files like:
appsettings.dev.json
appsettings.qa.json
appsettings.staging.json
And choose the right one in the deployment script and rename it to the actual appsettings.json, which is then read by your app.