Running cloudformation template fails when using custom Transform in CodeBuild - aws-cloudformation

One of my build steps runs a cloudformation template which has a custom transform. The transform is available in us-east-1 which is the same region in which code build is running. This same cloudformation template works when I run it with my local cli. Could you provide how to get this transform working in the code build container.
My local cli is aws-cli/1.16.223 Python/3.6.0 Windows/10 botocore/1.12.213
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/awscli/clidriver.py", line 207, in main
return command_table[parsed_args.command](remaining, parsed_args)
File "/usr/local/lib/python3.6/dist-packages/awscli/clidriver.py", line 348, in __call__
return command_table[parsed_args.operation](remaining, parsed_globals)
File "/usr/local/lib/python3.6/dist-packages/awscli/customizations/commands.py", line 187, in __call__
return self._run_main(parsed_args, parsed_globals)
File "/usr/local/lib/python3.6/dist-packages/awscli/customizations/cloudformation/deploy.py", line 295, in _run_main
parsed_args.fail_on_empty_changeset)
File "/usr/local/lib/python3.6/dist-packages/awscli/customizations/cloudformation/deploy.py", line 310, in deploy
tags=tags
File "/usr/local/lib/python3.6/dist-packages/awscli/customizations/cloudformation/deployer.py", line 227, in create_and_wait_for_changeset
self.wait_for_changeset(result.changeset_id, stack_name)
File "/usr/local/lib/python3.6/dist-packages/awscli/customizations/cloudformation/deployer.py", line 178, in wait_for_changeset
.format(ex, status, reason))
RuntimeError: Failed to create the changeset: Waiter ChangeSetCreateComplete failed: Waiter encountered a terminal failure state Status: FAILED. Reason: Failed to execute transform REDACTED::ALKSify
2019-08-21 17:03:42,717 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255
Failed to create the changeset: Waiter ChangeSetCreateComplete failed: Waiter encountered a terminal failure state Status: FAILED. Reason: Failed to execute transform REDACTED::ALKSify

I just run into the same issue and could solve it as follows.
You need to add the 2 following permissions to the IAM Role of your CodeBuild project.
Allow "cloudformation:CreateChangeSet" on the macro resource itself !Sub ${AWS::AccountId}::ALKSify
Allow "lambda:InvokeFunction" on the lambda function that implements the macro.
So it would look like:
BuildProjectRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Principal:
Service: codebuild.amazonaws.com
Action:
- "sts:AssumeRole"
Policies:
- PolicyName: codebuild
PolicyDocument:
Version: "2012-10-17"
Statement:
- Sid: lambda
Effect: Allow
Action:
- "lambda:InvokeFunction"
Resource:
- !Sub "arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:<MACRO FUNCTION NAME>"
- Sid: macro
Effect: Allow
Action:
- "cloudformation:CreateChangeSet"
Resource:
- !Sub "${AWS::AccountId}::<MACRO NAME>"
.... <Other Permissions>
Cheers,
Stan

Related

CloudFormation Transform Section stops Rule Assertions from executing

I have the following (contrived) CloudFormation Template, without a Transform Section, which executes the Validation Assertion Rules properly (will show AssertDescription if BlueGreen and Lambda Param values are Equal, and won't let me click "Next" off the first page in the CF Console):
AWSTemplateFormatVersion: "2010-09-09"
Parameters:
BlueGreen:
Type: String
Default: Yes
AllowedValues:
- Yes
- No
Lambda:
Type: String
Default: No
AllowedValues:
- Yes
- No
Rules:
ValidateBlueGreenOrLambda:
Assertions:
- AssertDescription: "Select BlueGreen or Lambda"
Assert: !Not [!Equals [!Ref BlueGreen, !Ref Lambda]]
Resources:
MyBucket:
Type: AWS::S3::Bucket
However, if I add a Transform Section to the Template, the Rules are not Asserted, and I can continue on in the Console to build the Stack:
AWSTemplateFormatVersion: "2010-09-09"
Transform: "AWS::CodeDeployBlueGreen"
... Rest of Template from above here
Does anyone know why adding a Transform Section would stop the Rule Assertions from executing?

Specflow living docs generator error: Multiple files are matching the pattern

I have a solution with multiple feature files in one project called Axis.Tests
I am trying to integrate these tests into Azure pipelines.
Current state: I am able to build the project and run tests without any problem, but generating test results using living docs in Azure DevOps always fails.
Pipeline looks like this:
trigger: none
pool: "Default"
variables:
pathToSolution: tests/Axis.Tests.sln
solutionDisplayName: Axis.Tests.sln
testsFolder: "tests/Axis.Tests"
environment: "QA"
steps:
- template: templates/build-template.yml
parameters:
pathToSolutionOrCsproj: $(pathToSolution)
displayName: $(solutionDisplayName)
- task: DotNetCoreCLI#2
displayName: "Run Test"
env:
ENV: $(environment)
inputs:
command: "test"
projects: "$(testsFolder)/Axis.Tests.csproj"
arguments: '--filter "Category=pipelineTesting"'
- task: SpecFlowPlus#0
displayName: "LivingDoc with TestAssembly generatorSource"
inputs:
generatorSource: "FeatureFolder"
projectFilePath: $(testsFolder)
projectName: "Axis"
projectLanguage: en
testExecutionJson: "$(testsFolder)/**/TestExecution.json"
bindingAssemblies: |
$(testsFolder)/**/Axis.Tests.dll
continueOnError: true
condition: always()
timeoutInMinutes: 15
Output from LivingDocs generator:
Starting: LivingDoc with TestAssembly generatorSource
==============================================================================
Task : SpecFlow+LivingDoc
Description : Generate living documentation from your feature files or test assembly.
Version : 0.6.961
Author : techtalk
Help : https://docs.specflow.org/projects/specflow-livingdoc/en/latest/Generating/Adding-a-Build-Step.html
==============================================================================
Error: Multiple files are matching the pattern: tests/Axis.Tests/**/Axis.Tests.dll
##[error]Error: Command failed: dotnet "C:\agent\_work\_tasks\SpecFlowPlus_32f3fe66-8bfc-476e-8e2c-9b4b59432ffa\0.6.961\CLI\LivingDoc.CLI.dll" feature-folder "C:\agent\_work\34\s\tests\Axis.Tests" --output-type JSON --test-execution-json "tests/Axis.Tests/**/TestExecution.json" --binding-assemblies "tests/Axis.Tests/**/Axis.Tests.dll" --include-test-output "All" --output "C:\agent\_work\34\s\10117\FeatureData.json" --project-name "Axis" --project-language "en"
Error: Multiple files are matching the pattern: tests/Axis.Tests/**/Axis.Tests.dll
at checkExecSyncError (child_process.js:629:11)
at Object.execSync (child_process.js:666:13)
at C:\agent\_work\_tasks\SpecFlowPlus_32f3fe66-8bfc-476e-8e2c-9b4b59432ffa\0.6.961\index.js:138:46
at step (C:\agent\_work\_tasks\SpecFlowPlus_32f3fe66-8bfc-476e-8e2c-9b4b59432ffa\0.6.961\index.js:44:23)
at Object.next (C:\agent\_work\_tasks\SpecFlowPlus_32f3fe66-8bfc-476e-8e2c-9b4b59432ffa\0.6.961\index.js:25:53)
at C:\agent\_work\_tasks\SpecFlowPlus_32f3fe66-8bfc-476e-8e2c-9b4b59432ffa\0.6.961\index.js:19:71
at new Promise (<anonymous>)
at __awaiter (C:\agent\_work\_tasks\SpecFlowPlus_32f3fe66-8bfc-476e-8e2c-9b4b59432ffa\0.6.961\index.js:15:12)
at C:\agent\_work\_tasks\SpecFlowPlus_32f3fe66-8bfc-476e-8e2c-9b4b59432ffa\0.6.961\index.js:77:12
at Object.<anonymous> (C:\agent\_work\_tasks\SpecFlowPlus_32f3fe66-8bfc-476e-8e2c-9b4b5943
I checked bin folder in my computer and there is just one Axis.Tests.dll file
Anyway, Axis.Tests.dll is in both /bin and /obj folders. Is this a problem ?
Update:
I tried to specify bindingAssembly in this ways:
$(testsFolder)/bin/**/Axis.Tests.dll
$(testsFolder)/bin/Debug/netcoreapp3.1/Axis.Tests.dll
Both of them cause me following error:
Error Message:
System.AggregateException : One or more errors occurred. (The HTTP request to the remote WebDriver server for URL http://localhost:57379/session timed out after 60 seconds.) (The given key 'driver' was not present in the dictionary.)
---- OpenQA.Selenium.WebDriverException : The HTTP request to the remote WebDriver server for URL http://localhost:57379/session timed out after 60 seconds.
-------- System.Threading.Tasks.TaskCanceledException : The operation was canceled.
------------ System.IO.IOException : Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request..
---------------- System.Net.Sockets.SocketException : The I/O operation has been aborted because of either a thread exit or an application request.
---- System.Collections.Generic.KeyNotFoundException : The given key 'driver' was not present in the dictionary.
Stack trace omitted for brevity

Azure pipelines yml validation error; did not find expected '-' indicator; while calling template

I get an yml template error while referencing a template from azure-pipelines.yml.
I've validated the YML in several online editors and it is valid (per them and me). Moreover, when I move the template step into the azure-pipeline.yml it works like a charm.
This is my code:
# azure-pipeline.yml
resources:
repositories:
- repository: templates
type: git
name: azure-devops-reusable-tasks
steps:
- template: maven-package-owasp-scan.yml#templates
# maven-package-owasp-scan.yml#templates
stages:
- stage: Build
displayName: Build
jobs:
- job: Pacakge
steps:
- bash: mvn -e package
displayName: mvn package
- task: dependency-check-build-task#6
displayName: Owasp dependency check
inputs:
projectName: $(Build.Repository.Name)
scanPath: '**/*.*'
format: 'HTML'
At first I had just the steps sequence in the latter template. This gave the same error.
The error in question:
Encountered error(s) while parsing pipeline YAML:
/maven-package-owasp-scan.yml#templates: (Line: 3, Col: 5, Idx: 40) - (Line: 3, Col: 5, Idx: 40): While parsing a block collection, did not find expected '-' indicator.
The error happens when I try and run the pipeline. Once I select the resources tab, it tries to validate the yml and it produces the error.
Above work was done on a feature branch. There was an error in the master branch. When trying to run the pipeline and clicking resources, it scanned master branch by default. When fixing the indentation error on the master branch (which above error was referring to, it just didn't say which branch) the problem was solved.

trigger concourse job via CLI: "resource not found"

I am attempting to trigger a concourse job from the command line. My pipeline has one resource (a git repo) and one job, which uses that repo. I am seeing:
$ fly -t tutorial trigger-job -j my-pipeline/my-job -w
error: resource not found
However, when I go the web UI and manually trigger the job by pressing the "+" button in the top right, it works fine.
Here is the full pipeline:
resources:
- name: cruise-source
type: git
source:
uri: git#github.com:my-org/cruise.git
branch: develop
jobs:
- name: build-image
public: true
plan:
- get: cruise-source
- task: list-files
config:
platform: linux
image_resource:
type: docker-image
source: {repository: alpine}
inputs:
- name: cruise-source
run:
path: ls
args: [cruise-source]
How can I trigger this job from the CLI?
The "resource not found" you get has nothing to do with the git resource :-) it actually means that the pipeline or job name is wrong. Looking at your pipeline configuration, you should issue
fly -t tutorial trigger-job -j my-pipeline/build-image -w
or if your configuration is different from what you have posted, maybe you have a typo in the name of the pipeline or job.

ConcourseCI: Run task from mapped/renamed output of get resource

I have a repo of which I'm looking at various folders at and building different things in each repo.
Since a lot of the steps are similar I was trying to streamline things a bit and use output mapping to "rename" the dir to a common name, but it doesn't seem to behave. All I can get is an error: "unknown artifact source: repo"
(A snippet of) My pipeline is:
resources:
# I have more of these, one for each path I'm interested in but not shown here.
- name: repo-folder--11.1--common
type: git
source:
uri: git#github.com:myorg/project
branch: concourse-pipeline
private_key: {{github_private_key}}
paths:
- 11.1/common
jobs:
- name: common-image-build
plan:
- get: repo-folder--11.1--common
output_mapping:
repo-folder--11.1--common: repo
trigger: true
- get: centos-docker-image
- task: generate-tag
file: repo/task-generate-tag.yml
params:
prefix: "1.11-"
I was hoping that the output_mapping on my get would let me refer to that git repo via a simpler name ("repo") in this build plan, but it doesn't seem to.
Am I missing some way of achieving this or is this a bug/design decision?
No need to use output_mapping, resource get has its own way of "renaming", by specifying the resource.
resources:
- name: repo-folder--11.1--common
type: git
source:
uri: git#github.com:myorg/project
branch: concourse-pipeline
private_key: {{github_private_key}}
paths:
- 11.1/common
jobs:
- name: common-image-build
plan:
- get: repo
resource: repo-folder--11.1--common
trigger: true
- get: centos-docker-image
- task: generate-tag
file: repo/task-generate-tag.yml
params:
prefix: "1.11-"