I want to create a GitHub Actions workflow to automatically test web based software. I need to manually log in with my local web browser for verification. How do I map the runner's public IP and expose the correct ports (let's say 3000), so that I can connect directly to the service from my local machine and verify some functions in the browser?
Here is a conceptually related workflow that is similar enough to my situation:
name: Start NPM Server
on:
push:
branches: [ main ]
jobs:
start-server:
runs-on: ubuntu-latest
steps:
- name: Check IP
run: curl https://api.ipify.org
- name: Checkout code
uses: actions/checkout#v2
- name: Install dependencies
run: npm install
- name: Start server
run: |
npm start
sleep 10000
I have found the external IP, but I haven't found a way to actually create the connection and access it from my web browser. Is it even feasible for me to open the port 3000, connect to the external ip that I find from the "Check IP" step, and do that verification manually?
Related
GitLab has an extremely useful feature called Review Apps which allows you to start up an instance of the web app from every PR which has its own subdomain and is linked on the PR page. I have done some searching and I don't see anything quite like it for GitHub.
Are there any ways to achieve a similar thing on github? 3rd party services are fine if they can integrate in with github. The app has a docker compose config so it would be just starting up an instance on a VM and shutting it down later.
The closest would be Delivering deployments/Deployment API, as described in the article "Deploy your pull requests with GitHub Actions and GitHub Deployments" from Sander Knape.
You can see its workflow here.
But the point is: there is not a directly integrated "review" deployment process like GitLab: you need to write your own GitHub workflow in order to deploy on a GitHub-managed Azure-based server, starting with:
deploy:
runs-on: ubuntu-latest
needs: deploy-check
if: needs.deploy-check.outputs.triggered == 'true'
steps:
- name: get pull request ref
id: get_pull_request_ref
uses: octokit/request-action#v2.x
with:
route: GET /repos/:repository/pulls/:issue_id
repository: ${{ github.repository }}
issue_id: ${{ github.event.issue.number }}
env:
GITHUB_TOKEN: "${{ secrets.GITHUB_TOKEN }}"
Now I am using github actions as my project CI, I want to do some unit test when using github actions build my project. When I using unit test, the project must using database, now my database have a white list and only IP in white list could connect my database, but now when I run unit test in GitHub Actions, I did not know the GitHub Actions IP address. Is it possible to use a static ip or any other way to solve the problem? I am not want to any IP could connect my database, it may have a security problem. any suggestion?
This is currently only possible with a self-hosted runner on a VM you can control the IP address of.
See also:
About self-hosted runners.
Alternatively, your GitHub action workflow may be able to adjust the firewall settings as part of the run.
Or you could use something like SQL Server LocalDB or SQLLite to connect to the database locally on the runner. Or spin up a temporary DB in a cloud environment, open it up to the runner and throw it away afterwards.
Or you could use a VPN client to connect to actions runner to your environment. You can install anything you want on the runner.
You can dynamically retrieve the GitHub Actions runner's IP address during your workflow using the public-ip action and update your RDS instance's security group ingress rules before and after your unit test steps.
This will allow you to use GitHub's hosted runners with your workflow instead of hosting your own.
Note: You will need to also set AWS credentials on your runner with permissions to update the associated security group. Also, you need to make sure the RDS instance is in a public subnet with an Internet Gateway attached and security group attached to it.
Your workflow should look something like this:
deploy:
name: deploy
runs-on: ubuntu-latest
env:
AWS_INSTANCE_SG_ID: <your-rds-subnet-sg-id>
steps:
- name: configure aws credentials
uses: aws-actions/configure-aws-credentials#v1
with:
aws-access-key-id: <your-ci-aws-access-key>
aws-secret-access-key: <your-ci-aws-secret-key>
aws-region: <your-rds-aws-region>
- name: get runner ip addresses
id: ip
uses: haythem/public-ip#v1.2
- name: whitelist runner ip address
run: |
aws ec2 authorize-security-group-ingress \
--group-id $AWS_INSTANCE_SG_ID \
--protocol tcp \
--port 22 \
--cidr ${{ steps.ip.outputs.ipv4 }}/32
- name: connect to your rds instance and run tests
run: |
...run tests...
- name: revoke runner ip address
run: |
aws ec2 revoke-security-group-ingress \
--group-id $AWS_INSTANCE_SG_ID \
--protocol tcp \
--port 22 \
--cidr ${{ steps.ip.outputs.ipv4 }}/32
Ideally though you would run your integration tests in an EC2 within the same VPC as your RDS instance to avoid publicly exposing your RDS instance.
This is in beta (September 1, 2022) but it is possible to assign static IP address to runners:
Fixed IP ranges to provide access to runners via allow list services
Setup a fixed IP range for your machines by simply ticking a check box, this provides an IP range that can be allow listed in internal systems and in GitHub’s allow list to keep using Actions while making your GitHub environment more secure.
More details here
If your database happens to be Redis or PostgreSQL, GitHub Actions includes a built-in feature called Service Containers to spin up an ephemeral database in CI for testing purposes.
These databases are short-lived: after your job that uses it completes, the service container hosting the database is destroyed. You can either run the database in a container or directly on the virtual machine if desired.
For more info, see Creating PostgreSQL service containers in the GitHub Actions docs.
If you happen to be using another database, you can install do some more manual legwork to install and run it yourself.
I have a repository which contains deploy keys. I want a workflow job which periodically checks whether any of the deploy keys are reaching their maximum allowed age before they must be rotated. I tried writing a workflow like this, using the GITHUB_TOKEN, but it looks like it doesn't have the necessary privileges. My repository belongs to a GitHub Organization.
name: Check age of repository deploy key
# This workflow is triggered on pushes to the repository.
on:
push:
schedule:
# Runs 06:00 every day
- cron: '0 6 */1 * *'
jobs:
expiry_check:
env:
DEPLOY_KEY_METADATA_URL: https://api.github.com/repos/my_org/my_repo/keys
DEPLOY_KEY_MAX_AGE: 3600*24*365 # 1 year
# This job runs on Linux
runs-on: ubuntu-latest
steps:
# GitHub repository checkout
- name: GitHub repository checkout
uses: actions/checkout#v1
- name: Check if any deploy keys are approaching their expiry data
run: |
python3 -c "import requests;import sys;url=sys.argv[1];token=sys.argv[2];r=requests.get(url, headers={'Authorization': f'Bearer {token}'});print(r.text)" $DEPLOY_KEY_METADATA_URL ${{ secrets.GITHUB_TOKEN }}
The response to my API request has this error: {"message":"Resource not accessible by integration","documentation_url":"https://docs.github.com/rest/reference/repos#list-deploy-keys"}
Is there some other solution to this problem, besides personal access tokens and GitHub Apps? The first option is not feasible; business logic can't break when an employee leaves the GitHub Organization. I suppose I could make a GitHub App, but I'd rather avoid that too, if I can. I'm not an admin in my GitHub Organization.
i have a Github actions with Gcloud, this setup worked well, after 3 months (no changes) i ran another Deployment/Action but now there is a message:
Bucket is requester pays bucket but no user project provided.
What is the cause of this error?
I repeat, this Action worked well and no changes were made, just ran again after 3 months and failed.
My current Actions yaml is:
name: Deploy to GCloud
on:
push:
branches:
- master
- qa
env:
type: prod
PROJECT_ID: ${{ secrets.GCE_PROJECT }}
jobs:
setup-deploy:
name: Setup and Deploy
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout#v2
- name: Setup gcloud environment
uses: GoogleCloudPlatform/github-actions/setup-gcloud#master
with:
version: '290.0.1'
service_account_key: ${{ secrets.GCE_SA_KEY }}
project_id: ${{ secrets.GCE_PROJECT }}
- name: Auth helper
run: |-
gcloud --quiet auth configure-docker
- name: Deploy
run: |-
gcloud app deploy app.${GITHUB_REF#refs/heads/}.yaml
ScreenShot
As the link I shared in my comment says, the error you are getting is because the Requester pays is enabled in the bucket you are using to deploy. I tried at my end enabling this feature, using the below command, and I got the same error as you.
gsutil requesterpays set on gs://BUCKET_NAME
There are some considerations you must take into account to use and access to a bucket when a Requester pays feature is enabled.
In case this feature was enabled long before you got the error and you suddenly started getting the error, or if the feature is not enable in the bucket (you can execute this command to verify it gsutil requesterpays get gs://BUCKET_NAME and you should get something like this gs://my-bucket: Enabled) and you get the error, I would recommend you to open a Support case to the Google team takes a look at your issue.
I just created the action on my project and configured everything over there, but unfortunately I'm getting a message like this into the 'deploy file' section> ssh: connect to host ec2-MYIP.us-east-2.compute.amazonaws.com port 22: Operation timed out
Good thing is that I know what's happening. I have to allow as an Inbound Rule the following:
Type: SSH / Protocol: TCP / Post range: 22 / Source: ::/0;
As you can see here, it works fine without limiting the source IP >
But obviously I don't want to do that for security reasons, so I need to find out the source I need to put there.
I've tried a lot of Github IP addresses already, but all of them were unsuccessful.
Does anyone here know what's the right source for it to work in a protected way or how can I find it?
Action I am using > https://github.com/wlixcc/SFTP-Deploy-Action
The IP addresses of GitHub hosted runners are documented here: https://docs.github.com/en/free-pro-team#latest/actions/reference/specifications-for-github-hosted-runners#ip-addresses
Windows and Ubuntu runners are hosted in Azure and have the same IP address ranges as Azure Data centers.
[...]
Microsoft updates the Azure IP address ranges weekly in a JSON file that you can download from the Azure IP Ranges and Service Tags - Public Cloud website. You can use this range of IP addresses if you require an allow-list to prevent unauthorized access to your internal resources.
An improved answer over riQQ's: Dynamically retrieve the Github Action runner's IP address during your workflow using the public-ip action and
update your EC2 server's security group ingress rules before and after your SSH steps.
Your EC2 instance will never be exposed to public IP addresses on your SSH port.
Note: You will need to also set AWS credentials on your runner with permissions to update the associated EC2 security group.
Your workflow should look something like this:
deploy:
name: deploy
runs-on: ubuntu-latest
env:
AWS_INSTANCE_SG_ID: <your-ec2-security-group-id>
steps:
- name: configure aws credentials
uses: aws-actions/configure-aws-credentials#v1
with:
aws-access-key-id: <your-ci-aws-access-key>
aws-secret-access-key: <your-ci-aws-secret-key>
aws-region: <your-ec2-aws-region>
- name: get runner ip address
id: ip
uses: haythem/public-ip#v1.2
- name: whitelist runner ip address
run: |
aws ec2 authorize-security-group-ingress \
--group-id $AWS_INSTANCE_SG_ID \
--protocol tcp \
--port 22 \
--cidr ${{ steps.ip.outputs.ipv4 }}/32
- name: ssh into your ec2 and do whatever
run: |
...do whatever you need to do...
- name: revoke runner ip address
run: |
aws ec2 revoke-security-group-ingress \
--group-id $AWS_INSTANCE_SG_ID \
--protocol tcp \
--port 22 \
--cidr ${{ steps.ip.outputs.ipv4 }}/32