Best practice for using variables to configure and create new Github repository instance in Terraform instead of updating-in-place - github

I am trying to set up a standard Github repository template for my organization that uses Terraform to spin up new repos with the configured settings.
Every time I try to update the configuration file to create a new instance of the repository with a new name, instead it will try to update-in-place any repo that was already created using that file.
My question is what is the best practice for making my configuration file reusable with input variables like repo name? Should I make a module or is there some way of reusing that file otherwise?
Thanks for the help.

Terraform is a desired-state-configuration system, which means that your configuration should represent the full set of objects that should exist rather than an instruction to create a single object.
Therefore the typical way to add a new repository is to add a new resource block declaring that new repository, and leave the existing ones unchanged. Terraform will then see that there's a new resource not currently tracked in the state and will propose to create it.
If your repositories are configured in some systematic way that you can describe using a mechanical rule rather than manual configuration then you can potentially use the for_each meta-argument to declare multiple resource instances from the same resource block, using Terraform language expressions to describe the systematic rule.
For example, you could create a local value with a higher-level data structure that describes what should be different between your repositories and then use that data structure with for_each on a single resource block:
locals {
repositories = tomap({
example_1 = {
description = "First example repository"
}
example_2 = {
description = "Second example repository"
}
})
}
resource "github_repository" "all" {
for_each = local.repositories
name = each.key
description = each.value.description
private = true
}
For simplicity in this example I've only made the name and description variable between the instances, but you can add whatever extra attributes you need for each of the elements of local.repositories and then access them via each.value inside the resource block.
The private argument above illustrates how this approach can avoid the need to re-state argument values that will be the same for each declared repository, and have your local.repositories data structure focus only on the minimum attributes needed to describe the variations you need for your local policies around GitHub repositories.
A resource block with for_each set appears as a map of objects when used in expressions elsewhere, using the same keys as in the map given in for_each. Therefore if you need to access the repository ids, or any other attribute of the systematically-declared objects, you can write Terraform expressions that work with maps. For example, if you want to output all of the repository ids as a map of strings:
output "repository_ids" {
value = tomap({
for k, r in github_repository.all : k => r.repo_id
})
}

Related

Terraform Azure Devops Provider

We are trying to automate the Azure DevOps functions using Terraform. We are able to create Projects and Repos using Terraform. But we need to create multiple projects and repos specific to each project.
I have my terraform.tfvars file as given below
Proj1_Repos = ["Repo1","Repo2","Repo3"]
Proj2_Repos = ["Repo4","Repo5","Repo7"]
Project_Name = ["Proj1","Proj2"]
How i can write my terraform configuration file to create Proj1_Repos in Proj1 and Proj2_Repos in Proj2
I think you will have an easier time restructuring the variables to look something like:
"Projects" = {
"Proj1" = {
"repos" = ["Repo1","Repo2","Repo3"]
},
"Proj2" = {
"repos" = ["Repo4","Repo5","Repo6"]
}
}
This way you can more cleanly iterate over your declarations using the for_each operator for your devops repo resources.
Alternatively, if restructuring the input variables isn't an option, you can use the locals block to construct an association map for your variables. Something like this
If you are looking for a way to feed a variable value to reference another variable, you will not be able to do so without constructing a custom data object using the key and value of your variables. This route can get pretty wonky and not recommended.

Resolution error: Cannot use resource 'x' in a cross-environment fashion, the resource's physical name must be explicit set

I'm trying to pass an ecs cluster from one stack to another stack.
I get this error:
Error: Resolution error: Resolution error: Resolution error: Cannot use resource 'BackendAPIStack/BackendAPICluster' in a cross-environment fashion, the resource's physical name must be explicit set or use `PhysicalName.GENERATE_IF_NEEDED`.
The cluster is defined as below in BackendAPIStack:
this.cluster = new ecs.Cluster(this, 'BackendAPICluster', {
vpc: this.vpc
});
The stacks are defined as follows:
const backendAPIStack = new BackendAPIStack(app, `BackendAPIStack${settingsForThisEnv.stackVersion}`, {
env: {
account: process.env.CDK_DEFAULT_ACCOUNT,
region: process.env.CDK_DEFAULT_REGION
},
digicallPolicyQueue: digicallPolicyQueue,
environmentName,
...settingsForThisEnv
});
const metabaseStack = new MetabaseStack(app, 'MetabaseStack', backendAPIStack.vpc, backendAPIStack.cluster, {
vpc: backendAPIStack.vpc,
cluster: backendAPIStack.cluster
});
metabaseStack.addDependency(backendAPIStack);
Here's the constructor for metabaseStack:
constructor(scope: cdk.Construct, id: string, vpc: ec2.Vpc, cluster: ecs.Cluster, props: MetabaseStackProps) {
super(scope, id, props);
console.log('cluster', cluster)
this.vpc = vpc;
this.cluster = cluster;
this.setupMetabase()
}
and then I'm using the cluster here:
const metabaseService = new ecs_patterns.ApplicationLoadBalancedFargateService(this, 'Metabase', {
assignPublicIp: false,
cluster: this.cluster,
...
I can't find documentation on how to do what I'm trying to do.
You're creating a Region/Account-specific Stack with BackendAPIStack because you're binding the stack to a specific account and region via the env prop value.
Then you're creating a Region/Account-agnostic stack by creating the MetabaseStack without any env prop value.
In general, having two independent stacks like this is fine, but here you're linking them together by passing a reference from the BackendAPIStack to the MetabaseStack, which won't work.
This is a problem because CDK normally links Stacks together by performing Stack Exports and Imports of values, but CloudFormation does not support cross-region or cross-account Stack references
So, your possible solutions are:
(A) Set up your MetabaseStack to use the same account/region as your BackendAPIStack
Under the hood this will setup the Cluster's ARN to be a Stack export from BackendAPICluster and then MetabaseStack will be able to import it.
(B1) Create BackendAPICluster with a clusterName that you pick.
i.e. new Cluster(..., {vpc: this.vpc, clusterName: 'backendCluster' })
By not providing a name, you're using the default of "CloudFormation-generated name" which is the basis of the issue that CDK is reporting, albeit it in a confusing way.
When you do provide a name, then the ARN for the cluster is deterministic (not picked by CloudFormation at deployment time) so CDK then has enough information at build time to determine what the Cluster's ARN will be and can provide that to your MetabaseStack.
(B2) Create BackendAPICluster with a clusterName and let CDK pick
This is done by setting the clusterName to PhysicalName.GENERATE_IF_NEEDED
i.e. new Cluster(..., {clusterName: PhysicalName.GENERATE_IF_NEEDED })
PhysicalName.GENERATE_IF_NEEDED is a marker that indicates that a physical (name) will only be generated by the CDK if it is needed for cross-environment references. Otherwise, it will be allocated by CloudFormation.
This is what the error is trying to tell you, but I didn't understand it either...
If possible, I would go with (A). I suspect it was just an oversight anyway that you weren't passing the same env values to the MetabaseStack and you probably want both of these stacks in the same region to reduce latency and all that.
If not, then I would personally then go with (B2) next because I try to not give any of my resources explicit names unless they are part of some contract with another group. I.e. Assume the role named 'ServiceWorker' in Account XYZ or Download the data from Bucket 'ABC'.

Terraform append tags

I create VPC and subnets which I add tags.
Later, I create EKS cluster which appends its own tags and if I apply again, the tags are overwritten.
I need any method to read the current tags and then merge with my custom tags. The problem is if the VPC resources are being created for the first time I can't query if some tags exists.
Here is my subnets definition
resource "aws_subnet" "k8s" {
count = "${var.create_vpc && length(var.k8s_subnets) > 0 ? length(var.k8s_subnets) : 0}"
vpc_id = "${local.vpc_id}"
cidr_block = "${var.k8s_subnets[count.index]}"
availability_zone = "${element(var.azs, count.index)}"
tags = "${merge(map("Name", format("subnet-%s-${var.k8s_subnet_suffix}-%s", var.name, element(var.azs, count.index))), var.tags, var.k8s_subnet_tags)}"
}
This is the tag EKS adds:
kubernetes.io/cluster/eks-cluster : shared
I'm stuck with this kind of... which comes first, the chicken or the egg? Any idea or suggestion?
-- Edited
something like self.tags could be the solution but unfortunately is not possible:
self.ATTRIBUTE syntax is only allowed and valid within provisioners.
and it shows an error:
Error: resource 'aws_subnet.k8s' config: cannot contain self-reference self.tags
This is what I do:
I define the common tags in env.sh which wraps terraform
When creating another component, I do this: tags = "${merge(var.default_tags, map("Name", format("%s-Jenkins-ELB", var.env)))}"
You could write the VPC tags to SSM parameter store and retrieve them later to be use by the EKS cluster.

Custom CloudFormation Resources in Terraform

I'm trying out Terraform, and am in the process of translating one of my more interesting CloudFormation stacks to TF. Included as a key part of the stack is the following declaration that specifies a custom resource for the template - a Lambda that queries a list of AMIs and selects the latest one for the context, based on the description as a filter.
LatestAMI:
Type: Custom::LatestAMI
Properties:
ServiceToken: arn:aws:lambda:us-east-1:XXXXXXX:function:GetLatestAMI
Description: ubuntu-16.04
I've looked around the Terraform docs, but I can't seem to find out how I can specify this resource. Is there a Terraform analog for custom resources in CloudFormation?
The CF codes you posted calls a lambda function to get the latest ami id (filter with Description: ubuntu-16.04. There is simpler way to do in terraform.
You need data source aws_ami
https://www.terraform.io/docs/providers/aws/d/ami.html
Use this data source to get the ID of a registered AMI for use in other resources.
data "aws_ami" "latest_ami" {
most_recent = true
executable_users = ["all"]
filter {
name = "owner-alias"
values = ["amazon"]
}
filter {
name = "name"
values = ["*ubuntu-16.04*"]
}
}

New & renamed workflows with existing content

I have a site with a custom content type Content, which initially had a single workflow attached, content_workflow. There are several thousand existing instances of Content.
I now have a need to add a second workflow to this type, content_beta_workflow. How can I update all existing content to be part of the new workflow?
On a related note: if I want rename the initial workflow to content_alpha_workflow, how can I update all existing content to reflect this change?
If you are simply changing from one workflow to the other, follow these steps:
Go to Site Setup > Types
Select your custom content type from the drop down menu, the page will update to display the current workflow
Select your new workflow from the dropdown, a map will be generated showing each state in the current workflow
For each state, select the state in your new workflow that most closely matches (or is most appropriate)
When you save, all objects of your custom site will be updated to use the new workflow. For each state in the map from the original workflow, existing content in that state will be put into the state you chose in step 4 above. Security settings will be re-indexed and you are done.
As for renaming the old workflow, you can do so in the portal_workflow tool in the ZMI. But only change the human-facing Title of the workflow. Changing the ID may have side effects for the workflow history of your content.
edited
Okay, I see from your comment that you are looking to add a new workflow to a type in addition to the one it already has. Here's a bit of sample code to accomplish that:
my_type = 'Content' # This is your content portal_type name
my_wf = 'content_workflow_beta'
wf_chain = list(wf_tool.getChainForPortalType(my_type))
if my_wf not in wf_chain:
wf_chain.append(my_wf)
wf_tool.setChainForPortalTypes([my_type], wf_chain)
You can add this code in an upgrade step for the package that defines your content type and workflows. Add a call to updateRoleMappings on the workflow tool and you'll be set to use the new workflow through the standard Plone UI in addition to your original workflow.
As you've already found, you can also manually update the workflow history of all objects to rename workflow ID, but that's a pretty invasive step.
As workflow_history is a dict property on each content item, it was a case of adding or updating suitable items as required. First, I copied the GenericSetup for content_workflow to content_alpha_workflow. Next, I created content_beta_workflow and added it to the profile. Then I wrote the following upgrade step:
import logging
from DateTime import DateTime
def modify_content_workflow_history(context, logger=None):
if logger is None: logger = logging.getLogger('my.product')
# import the new workflows
context.portal_setup.runImportStepFromProfile('profile-my.product:default', 'workflow')
# set up some defaults for the new records
_history_defaults = dict(
action = None,
actor = 'admin',
comments = 'automatically created by update v2',
time = DateTime(),
)
_alpha_defaults = dict(review_state = 'alpha_state_1', **_history_defaults)
_beta_defaults = dict(review_state = 'beta_state_1', **_history_defaults)
for parent in context.parents.values():
for content in parent.content.values():
# don't acquire the parent's history
if 'parent_workflow' in content.workflow_history:
content.workflow_history = {}
# copy content_workflow to content_alpha_workflow
if 'content_workflow' in content.workflow_history:
alpha_defaults = context.workflow_history['content_workflow']
del content.workflow_history['content_workflow']
else:
alpha_defaults = (_alpha_defaults,) # must be a tuple
content.workflow_history['ctcc_content_alpha_workflow'] = alpha_defaults
# create the beta workflow with a modified actor
beta_defaults = dict(**_beta_defaults)
beta_defaults['actor'] = u'%suser' % parent.id
content.workflow_history['ctcc_content_beta_workflow'] = (beta_defaults,)
logger.info('Content workflow history updated')