Using nested stacks is a best practice in AWS CloudFormation, and indeed they solve many problems (code reuse, stack limits, etc).
When updating a stack (either through UpdateStack or ChangeSets), one can provide UsePreviousTemplate=True in order to instruct CloudFormation to use whatever template is currently live for the stack. This is useful if all you want to is update some parameters, and especially if you have some automated script doing the update (we update a parameter with the SHA of the code on github that we want to put live, and CloudFormation does the rest for us -- build and deploy, so we update this parameter from a lambda as soon as we have the code ready for a deploy); in this case you want to make sure that all you do is update the stuff controlled by the parameter, and not accidentally provide a new (or old) template.
There seems to be no documentation on what happens if you give this parameter when there are nested stacks involved, nor does there seem to be any place where one could specify anything useful in this regard. There also seems to be nothing useful in the definition of the nested stack:
Type: AWS::CloudFormation::Stack
Properties:
NotificationARNs:
- String
Parameters:
Key : Value
Tags:
- Tag
TemplateURL: String
TimeoutInMinutes: Integer
Hence the following question:
What is the behaviour when doing an update with UsePreviousTemplate=True on a stack with nested stacks?
Is it possible to do a stack update, using something like UsePreviousTemplate=True not only for the parent stack but also for the nested stacks?
What is the behaviour when doing an update with UsePreviousTemplate=True on a stack with nested stacks?
The previous template for the root stack is being used. However this template points to the s3 location of the nested template, and the nested template is again retrieved from this location. This means that:
Any changes to that template on s3 result in those changes being deployed to your nested stack
If the template is not to be found on that spot anymore, the stack-update fails.
In general, you will always see the nested stacks in your ChangeSet, (supposedly) because CloudFormation doesn't know until the moment it's actually running if there are changes in the template. Also means that the stack-update of the parent will be slower.
You can avoid the first of these issues by using a versioned URL, if you have versioning switched on in your bucket by adding ?versionId=... to your url:
https://s3.#BUCKETREGION#.amazonaws.com/..../.../.../.../nested.template?versionId=XXXX
Note: the above is true regardless of whether there are Macros in the nested stacks (which legitimately would mean that at least they need re-evaluation upon update), or even CAPABILITY_AUTO_EXPAND is given or not (if not, this guarantees no macros in the nested stacks).
Is it possible to do a stack update, using something like UsePreviousTemplate=True not only for the parent stack but also for the nested stacks?
There is a possibility to set a stack policy that disallows updates on nested stacks ([here][3], under "Prevent Updates to Nested Stacks"), however in my experiments all that it did was disallow any update on the parent stack (since any update will automatically affect the client stacks).
So for now I'm inclined to say "no", but I'd be happy to be proven wrong here!
Related
Just in case I'm trying to solve the XY problem here, here's some context (domain is a role-playing game companion app). I have a document (campaign), which has a collection (characters), and I'm working with angular.io / angularfire.
The core problem here is that if I query the collection of characters on a campaign, I get back Observable<Character[]>. I can use that in an *ngFor let character of characters | async just fine, but this ends up being a little messy downstream - I really want to do something like have the attributes block as a standalone component (<character-attributes [character]="character">) and so on.
This ends up meaning down in the actual display components, I have a mixture of items that change via ngOnChanges (stuff that comes from the character) and items that are observable (things injected by global services like the User playing a particular Character).
I have a couple options for making this cleaner (the zeroth being: just ignore it).
One: I could flatten all the possible dependencies into scalars instead of observables (probably by treating things like the attributes as a real only-view component and injecting more data as a direct input - <character-attributes [character]="" [player]="" [gm]=""> etc. Displayable changes kind of take care of themselves.
Two: I could find some magical way to convert an Observable<Character[]> into an Observable<Observable<Character>[]> which is kind of what I want, and then pass the Character observable down into the various character display blocks (there's a few different display options, depending on whether you're a player (so you want much more details of your character, and small info on everything else) or a GM (so you want intermediate details on everything that can expand into details anywhere).
Three: Instead of passing a whole Character into my component, I could pass character.id and have child components construct an observable for it in ngOnInit. (or maybe switchMap in ngOnChanges, it's unclear if the angular runtime will reuse actual components for different items by changing out the arguments, but that's a different stack overflow question). In this case, I'd be doing multiple reads of the same document - once in a query to get all characters, and once in each view component that is given the characterId and needs to fetch an observable of the character in question.
So the question is: if I do firestore.collection('/foo/1/bars').valueChanges() and later do firestore.doc('/foo/1/bars/1').valueChanges() in three different locations in the code, does that call four firestore reads (for billing purposes), one read, or two (one for the query and one for the doc)?
I dug into the firebase javascript sdk, and it looks like it's possible that the eventmanager handles multiple queries for the same item by just maintaining an array of listeners, but I quite frankly am not confident in my code archaeology here yet.
There's probably an option four here somewhere too. I might be over-engineering this, but this particular toy project is primarily so I can wrestle with best-practices in firestore, so I want to figure out what the right approach is.
I looked at the code linked from the SDK and it might be the library is smart enough to optimize multiple observers of the same document to just read the document once. However this is an implementation detail that is dangerous to rely on, as it could change without notice because it's not part of the public API.
On one hand, if you have the danger above in mind and are still willing to investigate, then you may create some test program to discover how things work as of today, either by checking the reads usage from the Console UI or by temporarily modifying the SDK source adding some logging to help you understand what's happening under the hood.
On the other hand, I believe part of the question arises from a application state management perspective. In fact, both listening to the collection or listening to each individual document will notify the same changes to the app, IMO what differs here is how data will flow across the components and how these changes will be managed. In that aspect I would chose whatever approach feels better codewise.
Hope this helps somewhat.
I have created a stack for our Multi-AZ RDS setup within same region. The stack creates VPC, Subnets, IGW, Route Tables, SecGroups, EC2, RDS EC2 etc and currently about 200+ lines.
I was not aware of Export Parameter and Fn::ImportValue where we can do the cross-stack reference. I have also just learned that a stack can't be deleted if it has Exported some of its parameters and other stack are referencing it.
My question to CloudFormation experts is it more practical/professional to keep one Stack (so deletion of all the resources is easy) OR should we split our IAC into multiple inter-related stacks?
Appreciate your input as I have prepared this template for a Demo that I am giving to my team, either I should leave it as it is or split it up.
Created one Stack that setup the infrastructure and also create all the resources in one-go.
There are probably many correct answers, therefore I would try to help you narrow down the one that would be best for you.
You can delete such stack, but it will warn you because of having stacks that depend on resources inside the stack that is about to be deleted (but I don't see a reason doing that)
The max num of resources per stack is 200, the number may have changed in the meantime - we had this issue having to many resources, therefore we split in nested stacks architecture
Deleting resources manually will drift the stacks - that's not a good thing
Deleting resource should be done by removing it from the template(s) and updating the stacks correspondingly - this implies that having a one stack only is easier to maintain, but then again it depends on your use case.
I hope this helped you come up to a decision. If not, share more information.
Thanks, I have splitted into two stacks, first is the core-infrastructure that only creates network Skelton i.e VPC, IGW, Subnets, Routes. This stack emits output parameters with each of these resources's export names.
The second stack rds-resources just reference above and create EC2, RDS, SG.
Thanks again for the useful information.
I am using kubebuilder to create a Kubernetes operator. When an object of my kind is initiated I have to parse the spec and update the objects based on a few calculations.
From what I can tell I can either update the status of the object, the metadata, or a managed field (I may be wrong?). It appears that the sigs.k8s.io/controller-runtime/pkg/client library is responsible for how to update these fields (I'm not completely sure). I am having trouble understanding the docs.
I have the following questions:
Are there a guide to best practices about where to store configuration on the object between status, metadata (labels or annotations), and managed fields?
How do I update/patch the annotations of an object similar to how I would use r.Status().Update(ctx, &thing); to update the status?
The Kubebuilder docs are a bit raw but nonetheless are a handy guide when building CRDs and controllers with Kubebuilder. It walks you through a fairly detailed example which is great to study and refer back to, to see how to do certain things.
The answer to your question generally is, "it depends." What values are you calculating, and why? Why do you need to store them on the object? Is the lifecycle of this data coupled to the lifecycle of this object, or might this computed data need to live on and be used by other controllers even when the object is deleted? In general, is anything going to interact with those values? What is it going to do with them?
If nothing else aside from the reconciliation controller for the CRD is going to interact with the data you're putting, consider putting it within the object's Status.
Doing r.Status().Update(ctx, &thing) will avoid triggering any side-effects as it will only persist changes you've made to the object's Status subresource, rather than its spec or metadata.
A common thing to do with custom resources is to set and remove finalizers, which live in the object's metadata.
We are moving some of our API's to graphql and would like to know to handle the rollback of the deployed package (Schema)and the best practice to the same.
To be more specific let's say we have a Schema S with 3 fields and then we added 4th field "A" . Now for some reason we cannot go forward with this package and field "A". So we have to perform roll back of the package so that now the Schema doesn't have field "A".
Now some consumer might ask for this field "A" and he might get an error. We could of course ask our clients to update but there is a time gap during which we might have failed request.
How do we handle this scenario,specifically an urgent rollback with in few hours or a day?
In general, you should avoid removing fields without warning to avoid the exact scenario you describe.
As your schema evolves, it's not uncommon to have some fields that are no longer needed. For example, rather than introducing a drastic change to a particular field (moving from a nullable to a non-nullable return type, adding required arguments, etc.), we may opt to add another field and encourage clients to transition to using that one instead. In such scenarios, we want to eventually remove the original field. The safest way to do so is to deprecate the field first. Using SDL, we can do so using a directive:
fieldA: String #deprecated(reason: "Use fieldB instead!")
After a certain amount of time, you can then remove the field entirely. How long you wait to remove the field depends on your team and the expectations you've communicated around handling deprecated fields. For example, you may find it helpful to set a deadline, by which point all clients are expected to have stopped using any deprecated fields. This works well as long as your client teams have the bandwidth to handle such technical debt.
A deprecated field's resolver can be changed to return a null value (if the field itself is nullable) or some minimal mock data. This prevents making unnecessary API or database calls, while still ensuring client requests don't result in an error.
In the context of your question, this means you should probably avoid rolling back to a previous release and instead follow the process outlined above for the fields you want to remove.
Alternatively, you could consider versioning. GraphQL generally shies away from the concept of versioning. As the official site explains:
Why do most APIs version? When there's limited control over the data that's returned from an API endpoint, any change can be considered a breaking change, and breaking changes require a new version. If adding new features to an API requires a new version, then a tradeoff emerges between releasing often and having many incremental versions versus the understandability and maintainability of the API.
In contrast, GraphQL only returns the data that's explicitly requested, so new capabilities can be added via new types and new fields on those types without creating a breaking change. This has led to a common practice of always avoiding breaking changes and serving a versionless API.
With that in mind, it's also feasible to still implement versioning with GraphQL by serving different schemas from different endpoints. While it's costly and usually unnecessary to go that route, it may be the right solution for you and your team, particularly if you expect to have to do similar rollbacks in the future.
You cannot do anything w.r.t GraphQL. Since you need to have the field present in the GraphQL type system. There may be libraries available, which will allow you to specify whether the field should be present in the query or not. But, there's no way of allowing non-existent field in the Query.
But what you can do is opt for a Blue-Green deployment strategy. In this strategy, you have both the versions running at the same time.
Let's say: Green is the version with Field A and Blue is the version without Field A. So when your clients are updated they start requesting the Blue version. And once all your clients are updated, shut-down the Green (with Field A).
I'm using the Asana REST API to iterate over workspaces, projects, and tasks. After I achieved the initial crawl over the data, I was surprised to see that I only retrieved the top-level tasks. Since I am required to provide the workspace and project information, I was hoping not to have to recurse any deeper. It appears that I can recurse on a single task with the \subtasks endpoint and re-query... wash/rinse/repeat... but that amounts to a potentially massive number of REST calls (one for each subtask to see if they, in turn, have subtasks to query - and so on).
I can partially mitigate this by adding to the opt_fields query parameter something like:
&opt_fields=subtasks,subtasks.subtasks
However, this doesn't scale well. It means I have to elongate the query for each layer of depth. I suppose I could say "don't put tasks deeper than x layers deep" - but that seems to fly in the face of Asana's functionality and design. Also, since I need lots of other properties, it requires me to make a secondary query for each node in the hierarchy to gather those. Ugh.
I can use the path method to try to mitigate this a bit:
&opt_fields=(this|subtasks).(id|name|etc...)
but again, I have to do this for every layer of depth. That's impractical.
There's documentation about this great REPEATER + operator. Supposedly it would work like this:
&opt_fields=this.subtasks+.name
That is supposed to apply to ALL subtasks anywhere in the hierarchy. In practice, this is completely broken, and the REST API chokes and returns only the ids of the top-level tasks. :( Apparently their documentation is just wrong here.
The only method that seems remotely functional (if not practical) is to iterate first on the top-level tasks, being sure to include opt_fields=subtasks. Whenever this is a non-empty array, I would need to recurse on that task, query for its subtasks, and continue in that manner, until I reach a null subtasks array. This could be of arbitrary depth. In practice, the first REST call yields me (hopefully) the largest number of tasks, so the individual recursion may be mitigated by real data... but it's a heck of an assumption.
I also noticed that the limit parameter applied ONLY to the top-level tasks. If I choose to expand the subtasks, say. I could get a thousand tasks back instead of 100. The call could timeout if the data is too large. The safest thing to do would be to only request the ids of subtasks until recursion, and as always, ask for all the desired top-level properties at that time.
All of this seems incredibly wasteful - what I really want is a flat list of tasks which include the parent.id and possibly a list of subtasks.id - but I don't want to query for them hierarchically. I also want to page my queries with rational data sizes in mind. I'd like to get 100 tasks at a time until Asana runs out - but that doesn't seem possible, since the limit only applies to top-level items.
Unfortunately the repeater didn't solve my problem, since it just doesn't work. What are other people doing to solve this problem? And, secondarily, can anyone with intimate Asana insight provide any hope of getting a better way to query?
While I'm at it, a suggested way to design this: the task endpoint should not require workspace or project predicate. I should be able to filter by them, but not be required to. I am limited to 100 objects already, why force me to filter unnecessarily? In the same vein - navigating the hierarchy of Asana seems an unnecessary tax for clients who are not Asana (and possibly even the Asana UI itself).
Any ideas or insights out there?
Have you ensured that the + you send is URL-encoded? Whatever library you are using should usually handle this (which language are you using, btw? We have some first-party client libraries available)
Try &opt_fields=this.subtasks%2B.name if you're creating the URL manually, or (better yet) use a library that correctly encodes URL query parameters.