I run my tests on Github Workflow, those tests are using a PostgreSQL database as a service.
In case there is a failure of the test suite, I'd like to make a pg dump as an artifact of the build.
The question is simple: how to call pgdump in Github Actions considering PostgreSQL is a service.
It was not that complicated. I added the installation of pg_dump (which I use after my test in case of failure) and the artifact action:
steps:
- name: Install pg_dump
run: sudo apt-get install postgresql-client
- name: Run tests
run: make test
env:
DATABASE_URL: postgres://username:password#localhost:5432/database_name
- uses: actions/upload-artifact#v2
with:
name: my-artifact
path: var/test-artifact/
Related
I'm trying to use Serverless Compose to deploy multiple services to AWS via CircleCI. I have 3 test services for a POC, and so far deploying these to a personal AWS account from the terminal works just fine. However, when I configure it to go through CircleCI with a config.yml file, I get this error:
Could not find the Serverless Framework CLI installation. Ensure Serverless Framework is installed before continuing.
I'm puzzled because my config.yml file looks like this:
version: 2.1
orbs:
aws-cli: circleci/aws-cli#3.1.1
serverless-framework: circleci/serverless-framework#2.0.0
node: circleci/node#5.0.2
jobs:
deploy:
parameters:
stage:
type: string
executor: serverless-framework/default
steps:
- checkout
- aws-cli/install
- serverless-framework/setup
- run:
command: serverless config credentials --provider aws --key $AWS_ACCESS_KEY_ID --secret $AWS_SECRET_ACCESS_KEY
name: Configure serverless
- run:
command: npm install #serverless/compose
name: Install #serverless/compose
- run:
command: serverless deploy --stage << parameters.stage >>
name: Deploy staging
workflows:
deploy-staging:
jobs:
- node/test:
version: 17.3.0
- deploy:
context: aws-*******-developers
name: ******-sandbox-use1
stage: staging
The serverless framework is set up, the orb is present, but it says that it could not be found. All steps are successful until I get to deploy staging. I've been digging through documentation but I can't seem to find where it's going wrong with CircleCI. Does anyone know what I may be missing?
Turns out this required a weird fix, but it's best to remove the following:
The orb serverless-framework: circleci/serverless-framework#2.0.0
The setup step in the job - serverless-framework/setup
The Configure Serverless step
Once these are removed, modify the Install #serverless/compose step to run npm install and install all the packages. Then run npx serverless deploy instead of serverless deploy. This fixed the problem for me.
I am trying to substitue variable in app.yaml with a cloud build trigger.
I Added substitution variable in build trigger.
Add environment variables to app.yaml in a way they can be easily substituted with build trigger variables. Like this:
env_variables:
SECRET_KEY: %SECRET_KEY%
Add a step in cloudbuild.yaml to substitute all %XXX% variables inside app.yaml with their values from build trigger.
steps:
- name: node:10.15.1
entrypoint: npm
args: ["install"]
- name: 'gcr.io/cloud-builders/gcloud'
entrypoint: bash
args:
- '-c'
- |
sed -i 's/%SESSION_SECRET%/'${_SESSION_SECRET}'/g' app.yaml
timeout: "1600s"
The problem is that Gcloud Build throw an exception :
Already have image (with digest): gcr.io/cloud-builders/gcloud
bash: _L/g: No such file or directory
Why ? How can I make a substitution of my app.yaml ?
I have a app.yaml to the root of the project at the same level of the cloudbuild.yaml
UPDATED
I am trying to build and debug gcloud locally with this command:
sudo cloud-build-local --config=cloudbuild.yaml --write-workspace=../workspace --dryrun=false --substitutions=_SESSION_SECRET=test --push .
When I take a look into the app.yaml file, the substitution worked as expected and there is no exception at all.
What is the difference with the gcloud build environment ?
OK I finally decided to use github action instead of google cloud triggers.
Since Google cloud triggers aren't able to find its own app.yaml and manage the freaking environment variable by itself.
Here is how to do it:
My environment :
App engine,
standard (not flex),
Nodejs Express application,
a PostgreSQL CloudSql
First the setup :
1. Create a new Google Cloud Project (or select an existing project).
2. Initialize your App Engine app with your project.
[Create a Google Cloud service account][sa] or select an existing one.
3. Add the the following Cloud IAM roles to your service account:
App Engine Admin - allows for the creation of new App Engine apps
Service Account User - required to deploy to App Engine as service account
Storage Admin - allows upload of source code
Cloud Build Editor - allows building of source code
[Download a JSON service account key][create-key] for the service account.
4. Add the following [secrets to your repository's secrets][gh-secret]:
GCP_PROJECT: Google Cloud project ID
GCP_SA_KEY: the downloaded service account key
The app.yaml
runtime: nodejs14
env: standard
env_variables:
SESSION_SECRET: $SESSION_SECRET
beta_settings:
cloud_sql_instances: SQL_INSTANCE
Then the github action
name: Build and Deploy to GKE
on: push
env:
PROJECT_ID: ${{ secrets.GKE_PROJECT }}
DATABASE_URL: ${{ secrets.DATABASE_URL}}
jobs:
setup-build-publish-deploy:
name: Setup, Build, Publish, and Deploy
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- uses: actions/setup-node#v2
with:
node-version: '12'
- run: npm install
- uses: actions/checkout#v1
- uses: ikuanyshbekov/app-yaml-env-compiler#v1.0
env:
SESSION_SECRET: ${{ secrets.SESSION_SECRET }}
- shell: bash
run: |
sed -i 's/SQL_INSTANCE/'${{secrets.DATABASE_URL}}'/g' app.yaml
- uses: actions-hub/gcloud#master
env:
PROJECT_ID: ${{ secrets.GKE_PROJECT }}
APPLICATION_CREDENTIALS: ${{ secrets.GCLOUD_AUTH }}
CLOUDSDK_CORE_DISABLE_PROMPTS: 1
with:
args: app deploy app.yaml
To add secrets into git hub action you must go to : Settings/secrets
Take note that I could handle all the substitution with the bash script. So I would not depend on the github project "ikuanyshbekov/app-yaml-env-compiler#v1.0"
It's a shame that GAE doesn't offer an easiest way to handle environment variable for the app.yaml. I don't want to use KMS since I need to update the beta-settings/cloud sql instance.. I really needed to substitute everything into the app.yaml.
This way I can make a specific action for the right environment and manage the secrets.
The entrypoint should be an executable, use /bin/bash or /bin/sh.
How to inspect inside the image (in general):
$ docker pull gcr.io/cloud-builders/gcloud
Using default tag: latest
latest: Pulling from cloud-builders/gcloud
...
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
gcr.io/cloud-builders/gcloud latest 8499764c4ef6 About an hour ago 4.01GB
$ docker run -ti --entrypoint '/bin/bash' 8499764c4ef6
root#60354dfb588a:/#
You can test your commands from there to test without having to sending it to Cloud Build each time.
Is there any way to cache docker-compose so that it will not build again and again?
here is my action workflow file:
name: Github Action
on:
push:
branches:
- staging
jobs:
test:
runs-on: ubuntu-18.04
steps:
- uses: actions/checkout#v1
- name: Bootstrap app on Ubuntu
uses: actions/setup-node#v1
with:
node-version: '12'
- name: Install global packages
run: npm install -g yarn prisma
- name: Install project deps
if: steps.cache-yarn.outputs.cache-hit != 'true'
run: yarn
- name: Build docker-compose
run: docker-compose -f docker-compose.test.prisma.yml up --build -d
I want to cache the docker build step. I have tried using if: steps.cache-docker.outputs.cache-hit != 'true' then only build but didn't work.
What you are referring to is called "docker layer caching", and it is not yet natively supported in GitHub Actions.
This is discussed extensively in several places, like:
Cache docker image forum thread
Cache a Docker image built in workflow forum thread
Docker caching issue in actions/cache repository
As mentioned in the comments, there are some 3rd party actions that provide this functionality (like this one), but for such a core and fundamental feature, I would be cautious with anything that is not officially supported by GitHub itself.
For those arriving here via Google, this now "supported". Or at least it is working: https://github.community/t/use-docker-layer-caching-with-docker-compose-build-not-just-docker/156049.
The idea is to build the images using docker (and its cache) and then use docker compose to run (up) them.
If using docker/bake-action or docker/build-push-action & want to access a cached image in subsequent steps -
Use load:true to save the image
Use the same image name as the cached image across steps in order to skip rebuilds.
Example:
...
name: Build and push
uses: docker/bake-action#master
with:
push: false
load: true
set: |
web.cache-from=type=gha
web.cache-to=type=gha
-
name: Test via compose
command: docker compose run web tests
...
services:
web:
build:
context: .
image: username/imagename
command: echo "Test run successful!"
See the docker team's responses;
How to access the bake-action cached image in subsequent steps?
How to use this plugin for a docker-compose?
How to share layers with Docker Compose?`
Experiment on caching docker compose images in GitHub Actions
in my GitHub action yaml file, I have two commands at the end. The first one is yarn start (which starts the server) and the second one is for running a test file.
from my local server I usually run yarn start, then wait until frontend and backend port to be run, then only I run the test from another terminal
but from GitHub action, it runs the yarn start command then immediately runs the test script, so when the test file is run, server is not listening on port. thats why my test scripts are failed. how can I ensure that the test script will run after the yarn start is finished?
here is my action.yml file
name: "Github Actions Test"
on:
push:
branches:
- wip/checkout2
jobs:
test:
runs-on: ubuntu-latest
env:
PRISMA_ENDPOINT: ${{secrets.PRISMA_ENDPOINT}}
PRISMA_SECRET: ${{secrets.PRISMA_SECRET}}
steps:
- uses: actions/checkout#v1
- name: "Install Node"
uses: actions/setup-node#v1
with:
node-version: "12.x"
- name: "Install global packages"
run: npm install -g yarn prisma-cli concurrently mocha
- name: "Run docker Container"
run: docker-compose -f docker-compose.yml up --build -d
- name: "Install deps"
run: yarn install
- name: "prisma deploy"
run: yarn deploy:backend
- name: "Seed Backend"
run: yarn seed:backend
- name: "Build app"
run: yarn build
- name: "Start backend and frontend concurrently on background and run tests"
run: |
yarn start &
yarn test
You will need to do one of these:
Option 1: Wait for a few seconds before running tests:
run: |
yarn start &
sleep 10
yarn test
Option 2: Wait for the port to be open with some utility designed for this purpose. Maybe wait-port (untested)
Option 3: Wait for the port to be open using native linux tools - example 1, example 2.
Similar to what #DannyB is suggesting, you could test the server is running properly in the background by waiting some seconds and then using curl to test network connectivity.
For example:
- name: "Start backend and frontend concurrently on background and run tests"
run: |
yarn start &
sleep 10 &&
curl http://localhost:8000 &&
yarn test
This way you can check the workflow job's log and confirm the server was up and running before executing tests.
If the connection is established, curl http://localhost:<PORT> will return the content of the web page by default. You can also add a -I at the end of the command to make sure to return only the request header and check if it has a HTTP/1.0 200 OK status.
Another option is wait-on
./node_modules/.bin/wait-on tcp:3000
I have a django project using travis CI
when I check pep8 in travis CI
It has no problem
and I want testing functional_test in travis CI
I insert the code in .travis.yml
In .travis.yml(hggg is a role name in my django project database)
language: python
python:
- "3.5.1"
install:
- pip install -r requirement/development.txt
before_script:
- createuser hggg
# # command to run tests
script:
- pep8
- python wef/manage.py test functional_test
but when build start
travis server can't create database
The command "pep8" exited with 0.
$ python wef/manage.py test users
Creating test database for alias 'default'...
Got an error creating the test database: permission denied to create database
Type 'yes' if you would like to try deleting the test database 'test_bookconnect', or 'no' to cancel:
No output has been received in the last 10 minutes, this potentially indicates a stalled build or something wrong with the build itself.
The build has been terminated
I want create a database(postgresql) in trvis CI
Please give me some advise