resources:
- name: nightly_cron_trigger
type: CronTrigger
configuration:
interval: "30 03 * * *" # Every day at 03:30AM UTC
branches:
include: *serviceBranchRegexp
pipelines:
- name: commons_nightly
steps:
- name: prepare_nightly_run
type: Bash
configuration:
nodePool: ci_c5large
inputResources:
- name: nightly_cron_trigger
- name: commons_bitbucket
trigger: false
outputResources:
- name: commons_property_bag
environmentVariables:
GIT_REPO_PATH:
default: *serviceGitRepoPath
execution:
onStart:
- source
currently we have a pipeline (runs with cron each night) where each step triggers an embedded pipeline and each step does the same - only the resources and names are changing. So I thought maybe the cron can run the main pipeline a few times at night but every run will have different params.
cron resource does not support this, meaning you cant trigger a pipeline with predefined variables using cronTrigger resource.
But may be you can use property bag resource. May be you can configure like this:
input cronTrigger will trigger a pipeline step and that pipelineStep will update the output propertyBag resource with different parameters.
cronTrigger -> pipelineStep -> propertyBag
which this propertyBag resource can be a input to a different pipeline now.
Related
I have a pipeline stage that is using a template as follows:
# Deploy to AKS
- stage: DeployTEST
displayName: Test env for my-app
condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/master'))
variables:
- group: 'my-app-var-group-test'
- group: 'package-variables'
- template: templates/shared-template-vars.yml#templates
jobs:
- deployment: TestDeployment
displayName: Deploy to AKS - Test
pool:
vmImage: $(vmImageName)
environment: env-test
strategy:
runOnce:
deploy:
steps:
- template: ./aks/deployment-steps.yml
...and the content of the template deployment-steps.yml is:
steps:
- script: |
echo AzureSubscription: '$(azureSubscription)'
echo KubernetesServiceConnection: '$(kubernetesServiceConnection)' # this is working
- task: KubernetesManifest#0
displayName: Create imagePullSecret
inputs:
action: createSecret
secretName: $(imagePullSecret)
dockerRegistryEndpoint: $(dockerRegistryServiceConnection)
kubernetesServiceConnection: $(kubernetesServiceConnection) # this is causing an error
I get an error like this:
There was a resource authorization issue: "The pipeline is not valid. Job TestDeployment: Step input kubernetesServiceConnection references service connection $(kubernetesServiceConnection) which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz."
and like this when I try to select individual stages prior manual pipeline run:
Encountered error(s) while parsing pipeline YAML:
Job TestDeployment: Step input kubernetesServiceConnection references service connection $(kubernetesServiceConnection) which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz.
The errors above are misleading, because it is not an authorization issue:
the referenced K8s service connection is authorized
when I hardcode the value of the $(kubernetesServiceConnection) variable the pipeline runs just fine - no errors
variable group my-app-var-group-test is authorized - IMPORTANT: this is where the $(kubernetesServiceConnection) variable is defined
NOTE: The variable kubernetesServiceConnection is defined in the my-app-var-group-test variable group & when I comment out the KubernetesManifest task, the value of the $(kubernetesServiceConnection) variable is properly printed to the pipeline console output without any issues and the pipeline runs successfully!?
I know I could use parameters to pass values into the template, but this setup is already used by all other pipelines (variable group vars are used/references in templates) and this issue appeared on a newly created pipeline. I have used file comparison to compare the yaml of a working pipeline and this one and failed to spot anything...
I might be missing something obvious, but I spent hours on this failing to resolve the error...
We have more number of common upstream pipelines - pipleline-a, pipleline-b, pipeline-c, pipeline-d … each in its own repository - repository-a, repository-b, repository-c, repository-d…
My target pipeline, say pipeline-y in repository-y, has a dependency on these upstream pipelines artifacts and the target pipeline needs to build when there is a change to any of the upstream libraries and the corresponding upstream pipeline builds successfully.
In other words, target pipeline-y needs to be triggered if any of the upstream pipelines completed successfully due to changes in them (CI triggers for upstream libraries work fine in their own pipelines).
We currently achieved this, using the resources pipelines trigger in the target pipeline-y, as below:
Upstream Pipeline - pipeline-a.yml
trigger:
- repository-a*
steps
- task: Maven#3
inputs:
mavenPomFile: 'pom.xml'
publishJUnitResults: false
javaHomeOption: 'JDKVersion'
mavenVersionOption: 'Default'
mavenAuthenticateFeed: true
effectivePomSkip: false
sonarQubeRunAnalysis: false
goals: 'package deploy'
Target pipeline-y.yml resources section
resources:
pipelines:
- pipeline: pipeline-a
source: pipeline-a
trigger:
branches:
- 'pipeline-a-v1*'
- pipeline: pipeline-b
source: pipeline-b
trigger:
branches:
- 'pipeline-b-v1*'
- pipeline: pipeline-c
source: pipeline-c
trigger:
branches:
- 'pipeline-c-v1*'
- pipeline: pipeline-d
source: pipeline-d
trigger:
branches:
- 'pipeline-d-v1*'
- pipeline: pipeline-e
source: pipeline-e
trigger:
branches:
- 'pipeline-e-v1*'
This works fine.
My question is, as we add more upstream common libraries, we have to update the resources section in the target downstream. When there are new versions of upstream libraries, we have to modify the version in resources-pipelines-pipiline-trigger - branches from “pipeline-a-v1” to “pipeline-a-v2”.
Is there a better way to do this? Can a variable be used in the resources-pipelines-pipeline-trigger - branches - example pipeline-a-$(version) . Can version be derived using Build system variables as below:
I tried
variables:
version: $[replace(variables['Build.SourceBranchName'], variables['Build.Repository.Name'], '')]
It did not seem to work.
It's not possible to dynamically specify resources in YAML.
A suggestion could be to use REST API hooks when new pipelines are added. Then trigger a program that generates new YAML for pipeline-y.yml.
I'm using multi branch resourcing in a concourse pipeline like so:
resources:
- name: my-resource
type: git-multibranch
source:
uri: git#github.com.../my-resource
branches: 'feature/.*'
private_key: ...
ignore-branches: ''
How can I access the branch the resource is on at the time the job runs? like so:
jobs:
...
outputs:
- name: my-resource
params:
GIT_BRANCH: {BRANCH-GOES-HERE}
I'm looking to access it via something like my-resource.branch but haven't found any thing that works yet
Reading through this https://docs.aws.amazon.com/codepipeline/latest/userguide/approvals-action-add.html
it sounds like you can only create a manual approval step through the UI Console or through CLI BUT NOT through CloudFormation Template?
Edgar
Actually, CloudFormation does support this.
You just need to set Provider for resource ActionTypeId (Pipeline -> Stage -> Action -> ActionTypeId) as Manual and that's it. More info about provider type - here.
Examle:
DeliveryPipeline:
Properties:
...
Stages:
...
- Actions:
- ActionTypeId:
Category: Approval
Owner: AWS
Provider: Manual
Version: '1'
Configuration:
NotificationArn: <<arn>>
InputArtifacts: []
Name: TestApproval
RunOrder: 1
Name: Development_Approval
...
Type: AWS::CodePipeline::Pipeline
Enable / disable sections of a CloudFormation for CodePipeline using Conditionals:
This creates a manual notification action once staging has been built and passed Runscope tests:
- InputArtifacts: []
Name: !Join ["",[!Ref GitHubRepository, "-prd-approval"]]
ActionTypeId:
Category: Approval
Owner: AWS
Version: '1'
Provider: Manual
OutputArtifacts: []
Configuration:
NotificationArn: !GetAtt ["SNSApprovalNotification", "Outputs.SNSTopicArn"]
ExternalEntityLink: OutputTestUrl
RunOrder: 3
How to enable/disable this like other CloudFormation resources with a Condition: .
Action steps don't recognize Condition: param
I could make 2 copies of the whole pipeline code one with and one without and then toggle which pipeline I create but it seems like there should be a better way.
You should be able to accomplish this by conditionally inserting the AWS::CodePipeline::Pipeline Resource's Action into the Actions list using the Fn::If Intrinsic Function referencing your Conditions element, returning the Action when the Condition is true and AWS::NoValue (which removes the property, in this case removing the item from the list) when it is not true:
- !If
- IsProdCondition
- InputArtifacts: []
Name: !Join ["",[!Ref GitHubRepository, "-prd-approval"]]
ActionTypeId:
Category: Approval
Owner: AWS
Version: '1'
Provider: Manual
OutputArtifacts: []
Configuration:
NotificationArn: !GetAtt ["SNSApprovalNotification", "Outputs.SNSTopicArn"]
ExternalEntityLink: OutputTestUrl
RunOrder: 3
- !Ref AWS::NoValue