How to control resource creation order in Pulumi - pulumi

I'm trying to create some resources and need to enforce some sort of creation order. e.g. creating an aws.s3.Bucket for storing the logs before it can be used as an input to aws.cloudfront.Distribution.
How do I control resource creation order when using Pulumi?

Generally, Pulumi handles the ordering of resource creation automatically. In TypeScript this is even enforced by the language's type system via pulumi.Input<T> and pulumi.Output<T> types. But understanding the details of those types isn't actually necessary.
The Pulumi engine will resolve all "parameters" or "inputs" to a resource. So if you use one resource as a parameter in configuring another, the dependent resource will be created first. i.e. it works the way you would want it to.
However, there are situations where you need to explicitly mark one resource as being dependent upon another. This will happen when there is some sort of coupling that exists outside of the Pulumi program.
To specify an explicit dependency, you can provide an instance of pulumi.ResourceOptions to the resource, and set its dependsOn property. The Pulumi engine will resolve all of the resources in the dependsOn array before processing the resource.
Here's a simple example showing these two ways the Pulumi determines ordering. An AWS S3 bucket is a resource that contains files, called objects. The bucket must be created before any objects can be created inside of it.
// Create a bucket named "example-bucket", available at s3://example-bucket.
let bucket = new aws.s3.Bucket("bucket",
{
bucket: "example-bucket",
});
let file1 = new aws.s3.BucketObject("file1", {
// The bucket field of BucketObjectArgs is an instance of
// aws.s3.Bucket. Pulumi will know to create the "bucket"
// resource before this BucketObject resource.
bucket: bucket,
});
let file2 = new aws.s3.BucketObject("file2",
{
// The bucket field of BucketObjectArgs is a string. So
// Pulumi does not know to block creating the file2 resource
// until the S3 bucket exists.
bucket: "example-bucket",
} as aws.s3.BucketArgs,
{
// By putting "bucket" in the "dependsOn" array here,
// the Pulumi engine will create the bucket resource before
// this file2 resource.
dependsOn: [ bucket ],
} as pulumi.ResourceOptions);

Simple answer
The official docs are quite informative about this option:
The dependsOn option provides a list of explicit resource dependency
resources.
Pulumi automatically tracks dependencies between resources when you
supply an input argument that came from another resource’s output
properties. In some cases, however, you may need to explicitly
specify additional dependencies that Pulumi doesn’t know about, but
must respect. This might happen if a dependency is external to
the infrastructure itself — such as an application dependency — or is
implied due to an ordering or eventual consistency requirement.
These dependencies ensure that resource creation, update, and deletion
is done in the correct order.
The examples below demonstrates making res2 dependent on res1, even if
there is no property-level dependency:
#Python
res1 = MyResource("res1");
res2 = MyResource("res2", opts=ResourceOptions(depends_on=[res1]));
#Golang
res1, _ := NewMyResource(ctx, "res1", &MyResourceArgs{/*...*/})
res2, _ := NewMyResource(ctx, "res2", &MyResourceArgs{/*...*/}, pulumi.DependsOn([]Resource{res1}))
#JS
let res1 = new MyResource("res1", {/*...*/});
let res2 = new MyResource("res2", {/*...*/}, { dependsOn: [res1] });
If you want to understand what is happening under the hood
Read about creation and deletion order:
Pulumi executes resource operations in parallel whenever possible, but
understands that some resources may have dependencies on other
resources. If an output of one resource is provided as an input
to another, the engine records the dependency between these two
resources as part of the state and uses these when scheduling
operations. This list can also be augmented by using the
dependsOn resource option.
By default, if a resource must be replaced, Pulumi will attempt to
create a new copy the the resource before destroying the old one.
This is helpful because it allows updates to infrastructure to happen
without downtime. This behavior can be controlled by the
deleteBeforeReplace option. If you have disabled auto-naming by
providing a specific name for a resource, it will be treated as if it
was marked as deleteBeforeReplace automatically (otherwise the
create operation for the new version would fail since the name is in
use).

Related

cdk : how to stop generating random suffix in nested stack name

I am using CDK with Python where I have a nested stack with a fixed id/name calling a custom construct also with a fixed id/name. Why does it still get a random string generated at the end of the name still? Is there no way to stop it?
The custom construct is creating a DynamoDB table and due to the random suffix generated at the end of the stack name, the stack fails when it runs the second time saying table already exists. I need the table to be created with a retain policy so don't want it deleted every time the stack is executed. The table name also has to be consistent and can't change to include any random autogenerated string. The name for the table comes from a configuration fed into the stack which is referenced in an application elsewhere that I can't modify.
Here is the nested stack code
from aws_cdk import (
NestedStack,
)
from myconstructs import StepFunctionConstruct
class MyInfraStack(NestedStack):
def __init__(
self,
scope: Construct,
construct_id: str,
**kwargs,
) -> None:
super().__init__(scope, construct_id, **kwargs)
sf_const = StepFunctionConstruct(
self,
id="dev-StepFunctionConstruct",
state_machine_name="dev-sf"
)
This is custom construct code
class StepFunctionConstruct(Construct):
def __init__(
self,
scope: Construct,
id: str,
state_machine_name: str
):
super().__init__(scope, id)
# code here to create dynamo db table
How to get rid of the circled in red suffix from the image above when deploying this stack?
The Stack name doesn't change on any update. Moreover, that's not the issue here.
Also, the construct ID isn't the resource name, it being static doesn't mean the name will be. Read more in the docs.
Here's what's happening here:
You deploy the stack, it's created with an auto-generated name.
The stack deploys a DDB table with an explicitly specified name and a retention policy of Retain, meaning it is not deleted on stack destruction.
If you deploy the stack again with any changes at this point, the stack's name will not change.
Then you destroy the stack, but the table still exists due to the retention policy.
You deploy the stack again, the stack's auto-generated name changes. That's not the issue, though - the issue is that it's trying to deploy a DDB table with an explicitly specified name, but it cannot do that because a table with that name already exists.
Even without destroying the stack between deploys, you can run into this issue if you introduce a change to the DDB resource that requires replacement. The way CloudFormation handles replacements is by first creating the new version of the resource and then deleting the old one. But it cannot create the new version, because the name would be the same.
This is why it's recommended to let CloudFormation generate the names for you (like you do for the stack).
Reference: https://docs.aws.amazon.com/cdk/v2/guide/best-practices.html

In pulumi, how can I get the fully qualified resource type name?

I'd like to be able to generate an urn based on the current stack, project and resource type. This would be handy when renaming resources, where the aliases options requires a full urn to make the link between resources.
I've got the following:
`urn:pulumi:${pulumi.getStack()}::${pulumi.getProject()}::kubernetes:storage.k8s.io/v1:StorageClass::cluster-enable-ssd`
I'd like to generate the kubernetes:storage.k8s.io/v1:StorageClass part given an arbitrary pulumi Resource class, e.g. in this case k8s.storage.v1.StorageClass. The Resource class itself has name on it, but that only returns StorageClass.
Is there any way to get the global, unique identifier for a pulumi Resource type?
It's a bit hacky but you can get it like this (in Node.js):
(k8s.storage.v1.StorageClass as any).__pulumiType

What's the best way to consume Parameter Store value in AWS CDK

I am having problems using SSM valueForStringParameter method in CDK. It's working the first time I deploy the stack, but it is not picking up updates to the parameter value when I redeploy the stack because CloudFormation template hasn't changed and so CloudFormation thinks there were no updates, even if SSM parameter has changed.
For the context, I am deploying stack via CodePipeline, where I run cdk synth first, and then use CloudFormationCreateUpdateStackAction action to deploy template.
Anyone knows how to work around that? The only other option that I know will work is to switch to a custom resource lambda that calls SSM and returns value using aws-sdk, but that feels like a overly complicated option.
Update 1
I cannot use ValueFromLookupbecause value is only updated at runtime as part of cloudformation deployment by another stack (I deploy both stacks in CodePipeline, in 2 different regions), so synthesis time lookup would result in stale value.
All the valueOf* and from* methods work by adding a CloudFormation parameter. As you figured out already, changing the parameter value does not change the template and no change will be triggered.
What you probably want to use instead is the method valueFromLookup. Lookups are executed during synth and the result is put into the generated CFN template.
ssm.StringParameter.valueFromLookup(this, 'param-name');
But be aware, lookups are stored in the cdk.context.json. If you have commited that file to your repo, you need to erase that key via cdk context -e ... before synth/diff/deploy.
Since you cannot use lookup functions and the most common way to pass config to cdk is through context variables, I can only suggest dirty workarounds.
For example, you could create a dummy parameter in your stack to bump every time there's deployment.
var deploymentId = new CfnParameter(this, "deploymentId", new CfnParameterProps() { Type = "String", Description = "Deployment Id" });
SetParameterValue(deploymentId, this.Node.GetContext("deploymentId").ToString());
and when you synthesize the CF, you could generate an ID:
cdk synth -c deploymentId=$(uuidgen)
If you can avoid the "environment agnostic" syth and you really need an immutable artifact to deploy across multiple environments, you could use the built package from your cdk, for example, the npm package containing your cdk. Therefore, you could deploy it in each environment by overwriting the context parameters instead of using ssm parameters store.
See https://docs.aws.amazon.com/cdk/latest/guide/get_ssm_value.html, you can use method valueFromLookup which gets you parameter store value at synthesis time, when value is different from previous one, this shall trigger CF stack update.
However, I was under impression that valueForStringParameter should work on updated ssm parameter values as well, based on https://aws.amazon.com/blogs/mt/integrating-aws-cloudformation-with-aws-systems-manager-parameter-store/ Example 2:

What is the package `unstructured` used for in k8s.io /apimachinery?

I could not understand what the package can do, the offical doc show nothing about unstructured. What the package used for ? Is it used for converting map[string]interface{} to K8S Obj ?
https://godoc.org/k8s.io/apimachinery/pkg/apis/meta/v1/unstructured
It looks like unstructured provides an interface to kubernetes objects, when you don't know object type upfront, i.e. dynamic package in client-go uses it extensively
As #kseniia-churiumova suggested it is used when you don't know the object type. Here is the use case to understand it better. Let us say your organisation has a policy that all Kubernetes object must have annotation "owner" with value pointing to email ID of a person or group. You have tasked with finding all the resources that violates this policy.
You can have a configuration file that has list of GroupVersionKind and use unstructured to query them. If a new type needs a check you can add to configuration without changing the code.
Note: This is just an example. In production you will have to use something like Gatekeeper that implements OPA specification to enforce policies.

Read YAML config through Rest API

I have a really complicated system which use multiple languages and frameworks (Java Python Scala Bash). In each module I need to retrieve configuration values which are similar and change frequently. Currently I'm maintaining multiple conf files which holds lots of duplicates.
I wonder if there is out of the box RestAPI which can retrieve variables by demand from remote location.
All I manage to find by now are ways to load the entire file from remote source which is half a solution from me:
YAML.parse(open('https://link_to_file/file.yaml'))
My goal, which I fail to find a lead to it, is to make a direct call.
MyRemoteAPI.get("level1.level2.x")
P.S
YAML is not mandatory solution for me, I'm Open for suggestions.
I don't know about an out-of-the-box API, but it's fairly trivial to build. Make a service that will read the YAML file and traverse to the appropriate key. e.g. using a dynamic language like Ruby (+Rails), you could do something like
def value
config = YAML.load_file '/local/path/to/config.yaml'
render plain: config.dig(params[:key].split('.'))
end
dig essentially traverses a structure and safely returns nil if a key isn't found, so this returns the value at the "leaf" of the requested path.
You might also want to cache the structure in memory to prevent constantly reading from the file, e.g. could do something like ##config ||= YAML.parse(open('https://link_to_file/file.yaml')) or config = Rails.cache.fetch('config', expire_in: 1.hour) { ... }. And/or cache the API's HTTP response.