I have inherited an AWS account with a lot of resources. Some of them were created manually, other by CloudFormation.
How can I check if a resource (in my case Security Group) was created by CloudFormation and belongs to a stack?
For some security groups aws ec2 describe-security-groups --group-ids real_id results in:
...
"Tags": [
{
"Value": "REAL_NAME",
"Key": "aws:cloudformation:logical-id"
},
{
"Value": "arn:aws:cloudformation:<REAL_ID>",
"Key": "aws:cloudformation:stack-id"
},
]
...
Other security groups don't have any tags.
Is it the only indicator? I mean, someone could easily remove tags form an SG created by CloudFormation.
As per the official documentation, in addition to any tags you define, AWS CloudFormation automatically creates the following stack-level tags with the prefix aws::
aws:cloudformation:logical-id
aws:cloudformation:stack-id
aws:cloudformation:stack-name
All stack-level tags, including automatically created tags, are propagated to resources that AWS CloudFormation supports. Currently, tags are not propagated to Amazon EBS volumes that are created from block device mappings.
--
This should be a good place to start with but since CF doesn't enforce the stack state so if someone deleted something manually then you would never know.
If I were you, I would export everything (supported) via Cloudformer and re-design the whole setup my way.
Another way:
You can pass PhysicalResourceId of a resource to describe_stack_resources and get the stack information if it belongs to a CF stack. This is an example:
cf = boto3.client('cloudformation')
cf.describe_stack_resources(PhysicalResourceId="i-0xxxxxxxxxxxxxxxx")
https://boto3.readthedocs.io/en/latest/reference/services/cloudformation.html#CloudFormation.Client.describe_stack_resources
I had the same issue. After no luck finding an answer I made a quick PowerShell script that will just look for a resource name in all of the stacks.
When CF was introduced the stacks didn't tag resources and even now I have issues with CloudFormation reliably tagging resources, there are still times it will tag one resource and not tag another even with the same resource type and in the same stack. In addition some resources like CloudWatch Alarms don't have tags.
$resourceName = "*MyResource*" #Part of the resource name, surrounded by asterisks (*)
$awsProfile = "Dev" #AWS Profile to use
$awsRegion = "us-east-1" #Region to query
Get-CFNStack -ProfileName $awsProfile -Region $awsRegion | Get-CFNStackResourceList -ProfileName $awsProfile -Region $awsRegion | Where-Object {$_.PhysicalResourceId -ilike $resourceName} | Select-Object StackName,PhysicalResourceId
Related
I am trying to build a proof-of-concept integration with Azure Cloud into another system. I am not an Azure subject matter expert, so I am struggling with the end-to-end integration.
I am having trouble associating a "Network Security Group" to the "Network Interface". I am able to create both, but they do not not associate to each other until I manually go into the Cloud Portal and associate.
I am using the following:
API Documentation:
https://learn.microsoft.com/en-us/rest/api/compute/virtualmachines
API Explorer:
https://resources.azure.com
I am calling the following end-points in order:
publicIPAddresses
https://management.azure.com/subscriptions/{subscriptionID}/resourceGroups/{resourceGroup}/providers/Microsoft.Network/publicIPAddresses/{resourceName}?api-version=2018-07-01
networkInterfaces
https://management.azure.com/subscriptions/{subscriptionID}/resourceGroups/{resourceGroup}/providers/Microsoft.Network/networkInterfaces/{resourceName}?api-version=2018-07-01
networkSecurityGroups
https://management.azure.com/subscriptions/{subscriptionID}/resourceGroups/{resourceGroup}/providers/Microsoft.Network/networkSecurityGroups/{resourceName}?api-version=2018-07-01
virtualMachines : https://management.azure.com/subscriptions/{subscriptionID}/resourceGroups/{resourceGroup}/providers/Microsoft.Compute/virtualMachines/{resourceName}?$expand=instanceView&api-version=2018-06-01
Everything else works except the NSG associating to the NIC.
Within the "networkSecurityGroups" message, I pass in the following parameter under the properties node.
"networkInterfaces": [{
"id": "/subscriptions/" + subscriptionID + "/resourceGroups/" + resourceGroup + "/providers/Microsoft.Network/networkInterfaces/" + networkInterfaces
}
]
I've tried reversing it by referencing the NSG in the Interface REST call, but still doesn't work. Oddly enough, I use the same syntax to associate the Interface to the VM itself, and that works as expected. Variations of the same syntax work with associating the PublicIP to the Interface, disks to VM, ect.
Any thoughts?
pretty sure you need to add this under NIC properties section:
"networkSecurityGroup": {
"id": "NSG_Resource_Id"
}
Starting on Nov 7, 2018 we started getting the following error when updating our CloudFormation stacks:
Updating user pool schema is not allowed from cloudformation. Use the
AddCustomAttributes API or the AWS Cognito Console to update user pool
schema.
Our CF stacks don't have any changes to the custom attributes of the Cognito pool. They only have changes to the PostConfirmation and CustomMessage triggers, as well the addition of API Gateway responses.
Does anybody know why we might be seeing this? How can we avoid this error message?
We had the same problem with deployment. For now we are deploying it without CustomMessage trigger and setting CustomMessage trigger manually after deployment.
we removed the CustomMessage changes from our template and that seemed to do the trick.
Mostly by luck, I've found an answer that allows me to get around this in an automated manner.
How our scripts used to work
First, let me explain how this used to work. I used to have the following set of cloudFormation scripts:
cognitoSetup.template --> <Serverless Framework> --> <cognitoSetup.template updated with triggers>
So we'd setup the Cognito pool, run the Serverless Framework to add the Cognito Lambda functions, and then update the cognitoSetup.template file with the ARNs for the lambdas exported when the Serverless Framework ran.
The Fix
Now, we include the ARNs for the Lambdas in the cognitoSetup.template. So now cognitoSetup.template looks like this:
"CognitoUserPool": {
"Type": "AWS::Cognito::UserPool"
...
"Properties": {
...
"LambdaConfig": {
"CustomMessage": "arn:aws:lambda:<our aws region>:<our account#>:function:main-<our stage>-onCognitoCustomMessage"
}
}
Note, we're setting this trigger before the lambda even exists. The trigger just needs an ARN, and it doesn't seem to care that it's not there yet. Then we run sls deploy which creates the actual Lambda function and everything works fine.
Now our scripts look like this:
cognitoSetup.template --> <Serverless Framework>
Why does this fix this error? I don't actually know. CloudFormation seems to be fine with this modification but not okay with modifying the same file later in our process. But it works.
I have an EC2 Instance running an old AMI from AWS, which in my current setup is from WINDOWS_2016_BASE. Now I know that, but the Instance doesn't know that.
Using PowerShell I want the EC2 Instance to create more like itself, but not from an AMI I have created (as that would of course always be at the old version) and not "directly" from the actual AMI ImageId it was created from. As AWS don't allow that if it is not the latest version of it (hence I need the indirect route to getting that AMI ImageId, even if it turns out to to be the same.. i.e if it the latest version already).
So I need to find, using PowerShell, the latest version of an AMI I already know (which I got from the metadata) and I do not want to specify the name of the AMI e.g WINDOWS_2016_BASE since I would like the machine to create more like itself without me having to hardcode what it is in the script.
I have no idea how to do this since using Get-EC2ImageByName -Names WINDOWS_2016_BASE is not what I want to do since I don't want to hardcode that names parameter or pass it in through userdata.
Get-EC2Image -ImageId <an old imageid> returns null since the AMI is no longer current.
This will be somewhat difficult without either stashing this name somewhere that your script can cue off of (which you've stated that you don't want to do), or relying on some mapping between names/amis (that Get-EC2ImageByName already does for you).
AMI metadata is designed such that it is easy to retrieve a 'family tree' of related AMIs, under the assumption that their names are similar excepting for maybe minor differences such as versions or dates. It is a common pattern that prefixes of related AMIs will be consistent, which makes them easier to search for and aggregate.
This point makes your approach difficult, because your scripts will be missing that bit of data -- the AMI name prefix.
You could dynamically retrieve your instance's AMI ID from the metadata service and pass that into Get-EC2Image to get AMI details, but you couldn't go further from there without some sort of name prefix to match against and use to search for related AMIs that are newer.
Maybe reconsider this approach, and keep the name prefix stashed in a tag or on the instance via user-data? For example, I just checked and the English Windows Server 2016 Base AMIs all share this prefix: Windows_Server-2016-English-Full-Base. If you stored that on your instance or as one of your tags, your script could retrieve it and run the following powershell to get the latest Windows Server 2016 AMI:
#(Get-EC2Image -Owner amazon -Filter #{ Name="name"; Values="Windows_Server-2016-English-Full-Base*" } | Sort CreationDate -Desc)[0].ImageId
Example from the system log (which i got from aws console view)
2016/12/26 14:36:12Z: AMI Origin Name: Windows_Server-2016-English-Full-Base
The equivalent powershell is:
Get-EC2ConsoleOutput
So here below in full (i'm new to powershell but i am pretty sure someone can wrap that into one character somehow)
# read the system console log
$consoleLog = Get-EC2ConsoleOutput $currentInstanceObj.InstanceId
$consoleLogOutput = [System.Text.Encoding]::UTF8.GetString([System.Convert]::FromBase64String($consoleLog.Output));
# extract the lines that contain AMI Origin in them (there should be 2) - sort them so name i first and version is second
$originLines = $consoleLogOutput -split '[\r\n]' | Where-Object {$_.Contains("AMI Origin")} | Sort-Object
# get the running name
$originLine = $originLines[0]
$originLineParts = $originLine.Split(':')
$originName = $originLineParts[$originLineParts.Length - 1].Trim()
"The origin name is $originName"
# get the running version (slighly pointless since the code below doesn't care as we want the latest - but it's for verbosity)
$originLine = $originLines[1]
$originLineParts = $originLine.Split(':')
$originVersion = $originLineParts[$originLineParts.Length - 1].Trim()
"The origin version is $originVersion"
# concatenate to get the original origin name (note: amazon have a naming pattern here - (name-version)
$amiName = $originName + "-" + $originVersion
"The original origin ami name is $amiName"
#find the latest of the same name and report the difference
$latestOriginObj = (Get-EC2Image -Filter #{ Name="name"; Values=($originName + "*")} | Sort-Object CreationDate -Descending | Select-Object -First 1)
if($latestOriginObj.ImageId -ne $currentInstanceObj.ImageId)
{
"The ami has been upgraded from " + ($currentInstanceObj.ImageId) + " to " + ($latestOriginObj.ImageId)
}
#....so go ahead and use the $latestOriginObj.ImageId when you create a new instance
And the knowledge source for that springs from these amazons docs
Excerpt below:
The AWS Management Console provides details about the AMI that you use to create an Amazon EC2 instance. The AMI ID field on the Description tab contains information including the Windows Server SKU, the architecture (32-bit or 64-bit), the date the AMI was created, and an AMI ID.
If an AMI has been made private or replaced by later versions and is no longer listed in the catalog, the AMI ID field states, "Cannot load detail for ami-xxxxx. You may not be permitted to view it." To determine which AMI was used to create the instance, you must open the system log. In the EC2 console, choose an instance, and from the context-menu (right-click) choose Instance Settings and then choose Get System Log. The date the AMI was created and the SKU are listed in the AMI Origin Version and AMI Origin Name fields.
I'm tasked with automating the creation of Azure VM's, and naturally I do a number of more or less broken iterations of trying to deploy a VM image. As part of this, I automatically allocate serial hostnames, but there's a strange reason it's not working:
The code in the link above works very well, but the contents of my ResourceGroup is not as expected. Every time I deploy (successfully or not), a new entry is created in whatever list is returned by Get-AzureRmResourceGroupDeployment; however, in the Azure web interface I can only see a few of these entries. If, for instance, I omit a parameter for the JSON file, Azure cannot even begin to deploy something -- but the hostname is somehow reserved anyway.
Where is this list? How can I clean up after broken deployments?
Currently, Get-AzureRmResourceGroupDeployment returns:
azure-w10-tfs13
azure-w10-tfs12
azure-w10-tfs11
azure-w10-tfs10
azure-w10-tfs09
azure-w10-tfs08
azure-w10-tfs07
azure-w10-tfs06
azure-w10-tfs05
azure-w10-tfs02
azure-w7-tfs01
azure-w10-tfs19
azure-w10-tfs1
although the web interface only lists:
azure-w10-tfs12
azure-w10-tfs13
azure-w10-tfs09
azure-w10-tfs05
azure-w10-tfs02
Solved using the code $siblings = (Get-AzureRmResource).Name | Where-Object{$_ -match "^$hostname\d+$"}
(PS. If you have tips for better tags, please feel free to edit this question!)
If you create a VM in Azure Resource Management mode, it will have a deployment attached to it. In fact if you create any resource at all, it will have a resource deployment attached.
If you delete the resource you will still have the deployment record there, because you still deployed it at some stage. Consider deployments as part of the audit trail of what has happened within the account.
You can delete deployment records with Remove-AzureRmResourceGroupDeployment but there is very little point, since deployments have no bearing upon the operation of Azure. There is no cost associated they are just historical records.
Querying deployments with Get-AzureRmResourceGroupDeployment will yield you the following fields.
DeploymentName
Mode
Outputs
OutputsString
Parameters
ParametersString
ProvisioningState
ResourceGroupName
TemplateLink
TemplateLinkString
Timestamp
So you can know whether the deployment was successful via ProvisioningState know the templates you used with TemplateLink and TemplateLinkString and check the outputs of the deployment etc. This can be useful to figure out what template worked and what didn't.
If you want to see actual resources, that you are potentially being charged for, you can use Get-AzureRmResource
If you just want to retrieve a list of the names of VMs that exist within an Azure subscription, you can use
(Get-AzureRmVM).Name
I am trying to connect a powershell script to an Auto Scaling Group via the .NET Amazon API.
I have checked the documentation here, but I am struggling to get an object that contains the IP addresses of the instances belonging to the Auto Scaling Group.
I am not sure which class to use, or which class contains my object.
I am currently using:
$request = New-Object -TypeName Amazon.AutoScaling.Model.DescribeAutoScalingInstancesRequest
Has anyone come across the same situation? Which class/object contains the IP addresses of the instances running in the AutoScaling group?
For anyone looking to do this via aws-cli
replace my autoscale group
aws --output text --query "Reservations[*].Instances[*].PublicIpAddress" ec2 describe-instances --instance-ids `aws --output text --query "AutoScalingGroups[0].Instances[*].InstanceId" autoscaling describe-auto-scaling-groups --auto-scaling-group-names "my autoscale group"`
Using the SDK approach you take the set of instance ids embedded in the response/result data returned from the DescribeAutoScalingInstances call and pass them to the DescribeInstances call for EC2 (using new-object again to get an EC2 client and request objects). This will net you a collection of Amazon.EC2.Model.Reservation objects (again inside the response/result data) from which the RunningInstance collection inside each reservation will get you the ip address(es) for the EC2 instance.
It is however much simpler to use the AWS Tools for Windows PowerShell like this:
Get-ASAutoScalingInstance | select -expandproperty InstanceId | Get-EC2Instance | select -expandproperty RunningInstance | ft InstanceId, IpAddress
Get-ASAutoScalingInstance maps to the request in your question; this yields the set of EC2 instances from which we extract the id of each instance with a select. We then request details for the instance using Get-EC2Instance; as noted above this yields an Amazon.EC2.Model.Reservation object, within with are the details of the instance (in the RunningInstance collection). We flatten this to pull out the instance id and associated ip address for the table.
The sample pipeline above assumes you've set credentials and region to use in the shell using Set-AWSCredentials and Set-DefaultAWSRegion.
The AWS Tools for Windows PowerShell are included with the download msi for the SDK and Visual Studio toolkit available here http://aws.amazon.com/net/.
Hope this helps.
Expanding on the answer above - you may want to include the name of the auto scaling group so you don't get every instance from every group. Also, if you're using VPC your instances may not have public IPs so you'll be after the private IPs like this
Get-ASAutoScalingInstance | ? {$_.AutoScalingGroupName -eq "web-autoscaler-group"} | select -ExpandProperty InstanceId | Get-EC2Instance | select -ExpandProperty RunningInstance | ft InstanceId, PrivateIpAddress