Overview:
I'm using GitHub Pages feature to host documentation. I'm working on a CI/CD process to automate the build so that when the source for the documentation is updated it automatically rebuilds the content and deploys to GitHub Pages.
Details:
So far, using AWS CodeBuild, I've implemented the following:
Pulls down source from GitHub Repo
Uses MkDocs to build and deploy to the special gh-pages branch using the "mkdocs gh-deploy" command.
This is done with command lines in the CodeBuild Buildspec. The reason I'm using commands is that I want to use GitHub Deploy Keys opposed to creating user account (used as a machine account) that my team would need to manage.
I have it all working except what triggers the build. If the process was using a user account to authenticate then I can use AWS CodePipeline which creates a Webhook within the GitHub repo, and then notifications are sent via the Webhook to say that the master branch was updated, which would trigger a new build.
I'd like to implement a similar process but using the GitHub repo's Deploy Key. Any suggestions?
Related
I created a classic CI azure pipeline for a .net application present in GitHub repo and enabled continuous integration in triggers and also added 2 branches in branch filters. But when my team members made some changes and make commits, my pipeline is not triggering and build is not created. I can't understand what might be the problem as I'm new to azure pipelines
Here is some troubleshooting advices, and you can click this document for more detailed information:
Are you using the GitHub app connection to connect the pipeline to GitHub? If you are using a GitHub app connection, follow these steps:
Is the mapping set up properly between GitHub and Azure DevOps? Open a
pull request in your GitHub repository, and make the comment /azp
where. This reports back the Azure DevOps organization that the
repository is mapped to.
If no organizations are set up to build this repository using the app,
go to
https://github.com/<org_name>/<repo_name>/settings/installations and
complete the configuration of the app.
If a different Azure DevOps organization is reported, then someone has
already established a pipeline for this repo in a different
organization. We currently have the limitation that we can only map a
GitHub repo to a single DevOps org. Only the pipelines in the first
Azure DevOps org can be automatically triggered. To change the
mapping, uninstall the app from the GitHub organization, and
re-install it. As you re-install it, make sure to select the correct
organization when you are redirected to Azure DevOps.
Are you using OAuth or PAT to connect the pipeline to GitHub? If you are using a GitHub connection, follow these steps:
OAuth and PAT connections rely on webhooks to communicate updates to
Azure Pipelines. In GitHub, navigate to the settings for your
repository, then to Webhooks. Verify that the webhooks exist. Usually
you should see three webhooks - push, pull_request, and issue_comment.
If you don't, then you must re-create the service connection and
update the pipeline to use the new service connection.
Select each of the webhooks in GitHub and verify that the payload that
corresponds to the user's commit exists and was sent successfully to
Azure DevOps. You may see an error here if the event could not be
communicated to Azure DevOps.
Is your pipeline paused or disabled? Open the editor for the pipeline, and then select Settings to check. If your pipeline is
paused or disabled, then triggers do not work.
Have you used variables in defining the trigger or the paths? That is not supported.
Have you excluded the branches or paths to which you pushed your changes? Test by pushing a change to an included path in an included
branch. Note that paths in triggers are case-sensitive. Make sure that
you use the same case as those of real folders when specifying the
paths in triggers.
Updates:
You don't need to change webhooks in github, what you need to do is that go to github Settings -> Webhooks, check whether there are "Recent Deliveries". If there are errors in recent deliveries, the cause of the question is indicated.
If Azure DevOps and GitHub are properly connected, GitHub will automatically generate Webhooks. As shown below:
The content of "Payload URL" is:
https://dev.azure.com/{organization}/_apis/public/hooks/externalEvents?publisherId=github&channelId={channelId}&api-version=6.1-preview
Please note that a GitHub repository can only connect to one Azure DevOps organization. If you have connected to more than one organizations, keep the only one you are using currently.
In DevOps click Pipelines on the left-hand side navigation
Select your pipeline.
Click Edit
Then in the new window pane (top left), you'll see a tab for triggers.
This is where you can configure the continuous integration settings to trigger builds on push / pull requests etc.
Here's an example
Continues integration is enabled for the development branch, thus any push to that branch will trigger a build.
I have many projects which uses Google Cloud Build + Github build pipeline setup. However, there is this one project, which I cannot create a webhook in Github for.
It used to work - but commits to the repository doesn't trigger the build process any more. I deleted the trigger and added it again - but the webhook in Github is not created automatically for this project.
When I run the trigger manually, it picks the wrong, but fixed commit which I did before an year.
Any clue?
Could you try delete a repository on Cloud Source Repositories and setup Google Cloud Build again ?
See:
https://cloud.google.com/cloud-build/docs/running-builds/automate-builds
Note: For external repositories, such as GitHub and Bitbucket, you must have owner-level permissions for the Cloud Platform project with which you're working. When you set up a build trigger with an external repository for the first time, you'll need to set up authorization with that repository.
After you've set up your external repository, Cloud Source Repository creates a mirror of your repository.
https://source.cloud.google.com
https://cloud.google.com/source-repositories/docs/deleting-a-repository
https://cloud.google.com/source-repositories/docs/mirroring-a-github-repository
I am experiencing the same issue. I can create a trigger for a repo, but I cannot connect the repo automatically to cloud build. We also have many projects, and this manual labor is sort of annoying.
Is there any (under the hood) github/gcloud api available in which I can connect a github repo to cloud build? I am aware that this can only be done by someone with admin privileges on a repo or organization in github.
After this, I will be able to run the command gcloud build triggers create github [NAME]
Basically i have the following scenario that i need to solve:
On a schedule basis, OctopusDeploy is pulling the latest releases from code repo using GitHub Feed and then perform some operation through the tentacle agent on another server.
Problem:
Currently we have to trigger a release in Github manually whenever there is changes to the repo so that Octopus Deploy can pick up new changes. Is there a way to trigger the release in GitHub automatically whenever there is a merged event to the master.
Thanks.
I’m going to guess that you don’t want to expose your Octopus Deploy instance to the internet... so how about use reoccurring scheduled deployments?
https://octopus.com/blog/octopus-release-2018.4#recurring-scheduled-deployments
Currently we have to trigger a release in GitHub manually whenever there is changes to the repo so that Octopus Deploy can pick up new changes.
Is there a way to trigger the release in GitHub automatically whenever there is a merged event to the master.
First, since March 2022 (4 years later):
Octopus Deploy is now a GitHub secret scanning partner
GitHub secret scanning protects users by searching repositories for known types of secrets.
By identifying and flagging these secrets we help protect users from data leaks and fraud associated with exposed data.
We have partnered with Octopus Deploy to scan for access tokens for their cloud-hosted product, Octopus Cloud.
Octopus API keys allow users to perform tasks like creating and deploying releases. We'll forward access tokens found in public repositories to Octopus Deploy, who will notify the affected user via email
In that new context:
no need for an external private Octopus Deploy
The check would be done for any new commit
But this is limited to secret scanning, so it might not do everything you were doing in your case.
It looks like the GitHubPullRequestBuilder is not compatible with Jenkins v2.0 pipeline jobs.
How do you configure a pipeline job to be triggered from a GitHub pull request event?
The documentation on this topic is sparse and I cannot find any examples of this. Or is it better to create a web-hook in GitHub to trigger the pipeline job on the PR event?
I had similar issue. Here’s what worked for me
Pre-req
Jenkins ver. 2+ (I was using Jenkins 2.60)
Github (or Githhub
enterprise) account
Your github and Jenkins must be able to talk to
each other.
On Github
create a github Personal Access Token (PAT) with relevant rights.
For your repo, create a webhook with
URL as YourJenkinsURL/github-webhook/
Choose ‘Let me select individual events’ and check ‘Pull Request’
Add a Jenkinsfile to the root folder of your repo. For testing purpose you could put content as a basic hello world like below
pipeline {
agent any
stages {
stage('Test') {
steps {
echo 'Hello World ...'
}
}
}
}
On Jenkins
Install GitHub Pull Request Builder plugin. (You also need “Github” plugin but that should normally be installed as part of Jenkins ver 2+)
Jenkins – Credentials
Add github Personal Access Token (PAT) as a ‘secret text’ credential.
Add github username-password as ‘username-password’ credential.
Manage Jenkins – Configure System
Github – Github Servers : This is part of the Github plugin. Add a github server. ‘API URL’ It will default to https://api.github.com. If you are using enterprise github, replace with enterprise github url followed by /api/v3. For credential select the PAT option. Test the connection. ‘Manage Hooks’ is checked.
GitHub Pull Request Builder : for ‘GitHub Server API URL’ use same url as specified in Github Server section. Leave ‘Shared Secret’ blank. For credentials use ‘username-password’ credential. Test credentials to ensure its working. In my settings, ‘Auto-manage webhooks’ was checked.
Pipeline Job
Create a new item using ‘Pipeline’ option. Note: This is the vanilla Pipeline job, not Multibranch Pipeline.
General Section: Check ‘Github Project’ – Project URL : Enter your github repo url
Build Triggers: Check ‘GitHub Pull Request Builder’
For ‘GitHub API credentials’ select option you set for GitHub pull request builder in ‘Manage Jenkins – Configure System’ screen
For admin list: add your username
Check Use github hooks for build triggering
Pipeline:
Select ‘Pipeline Script from SCM’. Note this assumes that the root folder of your repo will contain a ‘Jenkinsfile’
SCM: Select ‘Git’
Repositories – enter repo detail. For credentials use ‘username-password’ based credentials.
Click Advanced and add refspec as +refs/pull/*:refs/remotes/origin/pr/*
Branch – should be ${sha1}
Script Path: defaulted to Jenkinsfile, leave as is.
Lightweight Checkout - Uncheck this (https://github.com/jenkinsci/ghprb-plugin/issues/507)
That’s it. You are all set. Creating a PR on master branch of your repo should now trigger your Jenkins Pipeline job
Some observations
Redelivering the webhook payload of a PR from github does not trigger the pipeline but opening a new PR or even re-opening a closed PR on github, triggers the pipeline job
In Pipeline Job Configuration, if you choose “Pipeline Script” and paste your pipeline script in there, the job doesn't trigger !!!
The flow in a nutshell can go like this:
You create your pipeline as code and save it in a file called Jenkinsfile at the root dir of your project. That code should describe how your project will get built. See here for examples: https://jenkins.io/doc/pipeline/examples/
Then you should create a new "Multibranch Pipeline Project" item in your Jenkins. You should set this up so as to scan your repo of step 1.
Now whenever you get a new PR branch opened on your step 1 repo, the branch will be checked-out and will get built according to the Jenkinsfile instructions included with it. You can set up more conditions on what gets built and when if you want to.
Notes:
"Multibranch Pipeline Project" example documentation: https://jenkins.io/doc/book/pipeline-as-code/ (scroll down to Multibranch Pipeline Projects)
Keep in mind that getting the plugin to build a PR after commending on the PR does not work. There is a discussion analyzing this here: https://github.com/jenkinsci/gitlab-plugin/issues/298 There are workarounds (also described in that discussion) but they are quite messy to set up.
The most straightforward way to use Pipeline with GitHub pull requests is to put the script into your repository under the name Jenkinsfile and then install the GitHub Branch Source plugin. Documentation
Follow the Below Steps for Triggering Jenkins Job Automatically on Pull request generated on GitHub
Create a web hook on GitHub i.e. http:///generic-
webhook-trigger/invoke
Content Type :application/json
Select Pull Request as event Now github Configuration Part is done.
Jenkins Job Configuration
Download Generic Webhook Trigger in Jenkins
Git Hub Configuration On Jenkins
Git Hub Configuration
7.Select Generic Webhook Trigger on jenkins
8.Generic Webhook Trigger Configuration on Jenkinsand follow step 9
9.After doing step 7 Jenkins job will get trigger on PullRequest
10.Step 8 required to get information from Pull request Payload
11.Branch configuration inside Generic Webhook Trigger to get Branch details from Pull Request
Thanks
Using the portal:
Using the portal at https://manage.windowsazure.com/ this is easy. From a newly created azure web site:
Go to the dashboard
On the quick glance options on the right, click "Set up deployment from source control"
On the "where is your source code?" option, select Github
Authorize, then choose the repo name and branch.
BINGO: now pushes to that branch will trigger a deployment of that branch onto the azure website. Lovely.
Here's the problem:
I cannot for the life of me see how to do this from the command line (using the xpat-cli installed using npm: "npm install azure-cli -g")
It seems like I should be typing something like:
azure site deployment github --verbose --githubusername [username] --githubrepository [username/reponame] [sitename]
Because the help for "azure site deployment github" says it will: "Link a website to a github account for deployment"
Doesn't seem to work though...
If I then type:
azure site show [sitename]
The output says the source control is "LocalGit" instead of github - which is what it says if I do it the portal way above.
Source Control
data: --------------
data: Type: LocalGit
Any ideas?
Depending on your use case the solution will be different.
Assuming you have the same github source but want to deploy to multiple different Azure websites
Potential Solution: Make use of the Deploy to Azure Button See details here: https://azure.microsoft.com/blog/2014/11/13/deploy-to-azure-button-for-azure-websites-2/
Assuming you have various repositories that you want to deploy to Azure: Potential Solution: I am assuming that you will be the person deploying it. The workflow will be as follows:
A. Download files from Github to local folder See here
B. Publish your local folder into a new/existing Azure site. See here, or here. There are other github repositories to get some scripts right away
I understand this doesnt directly hook your system to do direct pull, but its one way to do it.
There is also:
http://www.imtraum.com/blog/streamline-git-with-powershell/
https://gallery.technet.microsoft.com/scriptcenter/a-GitHub-Repository-265c0b49
Both documents I found describing how to set up continuous deployment from GitHub say that you can configure the repo and branch with an Azure Resource Manager template, but you have to enter your credentials and complete the authorization process in the portal.
The script used in this tutorial will automatically configure
continuous publishing from your GitHub repository. This requires that
your GitHub credentials are already stored in Azure, otherwise the
scripted deployment will fail when attempting to configure source
control settings for the web apps.
To store your GitHub credentials in Azure, create a web app in the
Azure preview portal and configure GitHub deployment. You only need to
do this once.
I'd also suggest posting your xpat-cli problem as an issue on their GitHub repo.