Gitlab CI - flutter test running very slow - flutter

I've been managing a project for the last couple of months using Flutter and TDD on Gitlab and using the Gitlab CI for monitoring code quality and tests. With the end of the project tin sight we now have over 700 tests and our CI has reached glacial speeds. At first we would take 5-10min to run the whole pipe, now it can take as long as 58min, with the only noticeable difference being the number of tests.
After removing the --machine from our scripts I noticed that the flutter test --coverage is taking 11 times longer than when run locally.
LOCAL
PS C:\Users\lr\Documents\GitHub\hive-manager> flutter test --coverage
04:04 +707: All tests passed!
GITLAB CI
$ flutter test --coverage
45:30 +707: All tests passed!
It is definitely the test phase of the pipeline that is causing issue as when I look at each jobs time there is a noticeable difference:
code_quality - 00:05:24
test - 00:48:47
coverage - 00:02:06 (82.2%)
semantic-version - 00:01:20
I'm a little lost on what to do now as I still have more tests but the CI is starting to cost quite a bit in minutes while at the same time I don't want to have to rebuild the CI somewhere else. Is this just a limit of Gitlab CI or is something going wrong here. I've attached the .yaml below
stages: # List of stages for jobs, and their order of execution
- analyze
- test
- coverage
- semantic-version
default:
image: cirrusci/flutter:latest
cache:
paths:
- /flutter/bin/cache/dart-sdk
code_quality:
stage: analyze
before_script:
- pub global activate dart_code_metrics
- export PATH="$PATH":"$HOME/.pub-cache/bin"
script:
- flutter --version
- flutter analyze
- metrics lib -r codeclimate > gl-code-quality-report.json
rules:
- if: $CI_COMMIT_BRANCH == "main"
- if: $CI_COMMIT_BRANCH == "develop" && $CI_PIPELINE_SOURCE == "push"
- if: $CI_COMMIT_BRANCH == "develop" && $CI_PIPELINE_SOURCE == "merge_request_event"
allow_failure: true
artifacts:
reports:
codequality: gl-code-quality-report.json
test: # This job runs in the test stage.
stage: test # It only starts when the job in the build stage completes successfully.
script:
- flutter test --coverage
rules:
- if: $CI_COMMIT_BRANCH == "main"
- if: $CI_COMMIT_BRANCH == "develop" && $CI_PIPELINE_SOURCE == "push"
- if: $CI_COMMIT_BRANCH == "develop" && $CI_PIPELINE_SOURCE == "merge_request_event"
coverage: # This job runs in the test stage.
stage: coverage # It only starts when the job in the build stage completes successfully.
script:
- lcov --summary coverage/lcov.info
- lcov --remove coverage/lcov.info
"lib/config/*"
"lib/application/l10n/l10n.dart"
"lib/application/l10n/**/*"
"lib/domain/repositories/*"
"lib/injection.config.dart"
"lib/presentation/routes/*"
"lib/infrastructure/repositories/firebase_injectable_module.dart"
"**/mock_*.dart"
"**/*.g.dart"
"**/*.gr.dart"
"**/*.freezed.dart"
"**/*.mocks.dart"
"**/*.config.dart"
-o coverage/clean_lcov.info
- genhtml coverage/clean_lcov.info --output=coverage
- curl -Os https://uploader.codecov.io/latest/linux/codecov
- chmod +x codecov
- ./codecov -t $CODECOV_TOKEN
- mv coverage/ public/
rules:
- if: $CI_COMMIT_BRANCH == "main"
- if: $CI_COMMIT_BRANCH == "develop" && $CI_PIPELINE_SOURCE == "push"
- if: $CI_COMMIT_BRANCH == "develop" && $CI_PIPELINE_SOURCE == "merge_request_event"
coverage: '/lines\.*: \d+\.\d+\%/'
artifacts:
paths:
- public
semantic-version:
image: node:16
stage: semantic-version
only:
refs:
- main
- develop
script:
- touch CHANGELOG.md
- npm install #semantic-release/gitlab #semantic-release/changelog
- npx semantic-release
artifacts:
paths:
- CHANGELOG.md
ADDITIONAL INFO
Looking into the report.xml from GitLab I can also see that most tests take ~2sec to complete, but the total tests only take 135sec.
While the test job takes ~44min to complete.
I've also tried removing the --coverage to reduce the time which resulted in 8min30sec, but which is still a lot more than the 1min30sec that it takes locally.

Related

Github Actions Job being skipped

Using Github Actions for some CI/CD.
Currently I am experiencing strange behavior where my jobs are being skipped despite the conditions being met. deploy-api has two conditions, if code was pushed to master and test-api was a success. But even though we are meeting those conditions, it is still being skipped.
jobs:
test-api:
name: Run tests on API
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v1
- name: Get dependencies
run: npm install
working-directory: ./api
- name: Run tests
run: npm run test
working-directory: ./api
deploy-api:
needs: test-api # other job must finish
if: github.ref == 'refs/heads/master' && needs.test-api.status == 'success' #only run if it's a commit to master AND previous success
As seen in the picture the second job is being skipped despite the push being on the master branch (as seen on the top) AND the previous job being successful.
Am I missing something in the code? Does anyone know of a workaround that can be used?
It would be nice if the UI told the user why it was skipped!
Use needs.test-api.result == 'success' (there is no .status) in the if expression.
See https://docs.github.com/en/actions/reference/context-and-expression-syntax-for-github-actions#needs-context.

How to trigger pipelines in GitLab CI

I have the problem, that I want to trigger another pipeline (B) in antoher project (B), only when the deploy job in pipeline (A) is finished. But my configuration starts the second pipeline as soon as the deploy job in pipeline (A) starts. How can I do it, that the second pipeline is triggered, only when the deploy job in pipeline (A) in projet (A) is finished?
Here is my gitlab-ci.yml
workflow:
rules:
- if: '$CI_COMMIT_BRANCH'
before_script:
- gem install bundler
- bundle install
pages:
stage: deploy
script:
- bundle exec jekyll build -d public
artifacts:
paths:
- public
rules:
- if: '$CI_COMMIT_BRANCH == "master"'
staging:
variables:
ENVIRONMENT: staging
stage: build
trigger: example/example
test:
stage: test
script:
- bundle exec jekyll build -d test
artifacts:
paths:
- test
rules:
- if: '$CI_COMMIT_BRANCH != "master"'
You don't declare stages order, so gitlab pipeline don't know what order are expected.
At the beginning of .gitlab-ci.yaml file add something like this (or whatever order you want):
stages:
- deploy
- test
- build
# rest of you file...
Alternatively you can use needs to build jobs relation.

Compiling with multiple scala versions

I wanted to run travis build against two Scala versions (2.12, 2.13) i.e crossCompilation, so I created two jobs for it, as logs were huge and there is a log limit of 4 MB in travis. So I created two jobs for it.Here is my travis.yml file. I am not so good with travis-ci. So I am struggling to run two jobs with different scala versions. Here is my travis.yml file:
language: scala
jdk:
- openjdk11
if: tag IS blank
services:
- mysql
addons:
apt:
sources:
- mysql-5.7-xenial
packages:
- mysql-server
dist: bionic
sudo: required
before_install:
- echo -e "machine github.com\n login $GITHUB_AUTH_TOKEN" > ~/.netrc
- mysql -e 'CREATE DATABASE IF NOT EXISTS $ZZ_API_TEST_DB_NAME;'
- sudo mysql -e "use mysql; update user set authentication_string=PASSWORD('') where user='$ZZ_API_DB_USERNAME'; update user set plugin='mysql_native_password';FLUSH PRIVILEGES;"
- sudo mysql_upgrade -u $ZZ_API_DB_USERNAME
- sudo service mysql restart
git:
depth: false
env:
global:
- ZZ_API_DB_HOST="localhost:3306"
- ZZ_API_TEST_DB_NAME=issue_management_test
- ZZ_API_DB_USERNAME=root
- ZZ_API_DB_PASSWORD=""
- SCALA_2_12="2.12.8"
- SCALA_2_13="2.13.3"
before_cache:
- find $HOME/.ivy2 -name "ivydata-*.properties" -delete
- find $HOME/.sbt -name "*.lock" -delete
cache:
directories:
- $HOME/.sbt/boot/scala*
- $HOME/.sbt/cache
- $HOME/.sbt/launchers
- $HOME/.ivy2/cache
- $HOME/.coursier
stages
- version_2.12
- version_2.13
jobs:
include:
- stage: version_2.12
name: "2.12.8"
script:
- if [ "$TRAVIS_EVENT_TYPE" == "cron" ]; then sbt coverage $SCALA_2_12 test ; else sbt $SCALA_2_12 test; fi
after_success:
- sbt coverageReport coverageAggregate
deploy:
- provider: script
skip_cleanup: true
script: sbt publish
on:
all_branches: true
condition: $TRAVIS_BRANCH != master || $TRAVIS_BRANCH != develop
- provider: script
skip_cleanup: true
before_deploy:
- travis/before_deploy.sh
script: sbt publish
on:
branch: develop
- provider: script
skip_cleanup: true
script: travis/release.sh
on:
branch: master
- stage: version_2.13
name: "2.13.3"
script:
- if [ "$TRAVIS_EVENT_TYPE" == "cron" ]; then sbt coverage $SCALA_2_13 test ; else sbt $SCALA_2_13 test; fi
after_success:
- sbt coverageReport coverageAggregate
deploy:
- provider: script
skip_cleanup: true
script: sbt publish
on:
all_branches: true
condition: $TRAVIS_BRANCH != master || $TRAVIS_BRANCH != develop
- provider: script
skip_cleanup: true
before_deploy:
- travis/before_deploy.sh
script: sbt publish
on:
branch: develop
- provider: script
skip_cleanup: true
script: travis/release.sh
on:
branch: master
I am not much familiar with travis, somehow its not picking
- SCALA_2_12="2.12.8"
- SCALA_2_13="2.13.3"
and this command:
- if [ "$TRAVIS_EVENT_TYPE" == "cron" ]; then sbt coverage $SCALA_2_12 test ; else sbt $SCALA_2_12 test; fi
is failing in travis build.
How to specify two different scala versions for these two different task, someone please help on this
It worked finally, this change I did,
changed $SCALA_2_13 to ++$SCALA_2_13

How to run a github-actions step, even if the previous step fails, while still failing the job

I'm trying to follow an example Github has for testing my build with github actions, and then compressing the test results and uploading them as an artifact.
https://help.github.com/en/actions/automating-your-workflow-with-github-actions/persisting-workflow-data-using-artifacts#uploading-build-and-test-artifacts
I'm having trouble with what to do when my tests fail though. This is my action. When my tests pass everything works great, my results are zipped an exported as an artifact, but if my tests fail, it stops the rest of the steps in the job, so my results never get published.
I tried adding the continue-on-error: true https://help.github.com/en/actions/automating-your-workflow-with-github-actions/workflow-syntax-for-github-actions#jobsjob_idstepscontinue-on-error
This makes it continue after it fails and uploads my test results. but then the job is marked as passed, even though my test step failed. Is there some way to have it upload my artifact even if a step fails, while still marking the overall job as failed?
name: CI
on:
pull_request:
branches:
- master
push:
branches:
- master
jobs:
build-and-test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v1
- name: Test App
run: ./gradlew test
- name: Archive Rest Results
uses: actions/upload-artifact#v1
with:
name: test-results
path: app/build/reports/tests
You can add
if: always()
to your step to have it run even if a previous step fails
https://docs.github.com/en/actions/learn-github-actions/expressions#status-check-functions
so for a single step it would look like this:
steps:
- name: Build App
run: ./build.sh
- name: Archive Test Results
if: always()
uses: actions/upload-artifact#v1
with:
name: test-results
path: app/build
Or you can add it to a job:
jobs:
job1:
job2:
needs: job1
job3:
if: always()
needs: [job1, job2]
Additionally, as pointed out below, putting always() will cause the function to run even if the build is canceled.
If dont want the function to run when you manually cancel a job, you can instead put:
if: success() || failure()
Other way, you can add continue-on-error: true.
Look like
- name: Job fail
continue-on-error: true
run |
exit 1
- name: Next job
run |
echo Hello
Read more in here.
run a github-actions step, even if the previous step fails
If you only need to execute the step if it succeeds or fails, then:
steps:
- name: Build App
run: ./build.sh
- name: Archive Test Results
if: success() || failure()
uses: actions/upload-artifact#v1
with:
name: test-results
path: app/build
Why use success() || failure() instead of always()?
Reading the Status check functions documentation on Github:
always
Causes the step to always execute, and returns true, even when canceled. A job or step will not run when a critical failure prevents the task from running. For example, if getting sources failed.
Which means the job will run even when it gets cancelled, if that's what you want, then go ahead. Otherwise, success() || failure() would be more suitable.
Note -
The documentation made clear thanks to Vladimir Panteleev in which he submitted the following PR: Github Docs PR #8411
Addon: if you have following sitution. 2 steps i.e. build > deploy and in some cases i.e. workflow_dispatch with input parameters you might want to skip build and proceed with deploy. At the same time you might want deploy to be skipped, when build failed.
Logically that would be something like skipped or not failed as deploy conditional.
if: always() will not work, cause it will always trigger deploy, even if build failed.
Solution is pretty simple:
if: ${{ !failure() }}
Mind that you cannot skip brackets when negating in if:, cause it reports syntax error.
The other answers here are great and work, but you might want a little more granularity.
For instance, ./upload only if ./test ran, even if it failed.
However, if something else failed and prevented the tests from running, don't upload.
# ... Other steps
- run: ./test
id: test
- run: ./upload
if: success() || steps.test.conclusion == 'failure'
steps.*.conclusion will be success, failure, cancelled, or skipped.
success or failure indicate the step ran. cancelled or skipped means it didn't.
Note there is an important caveat that you must test at least one success() or failure() in if.
if: steps.test.conclusion == 'success' || steps.test.conclusion == 'failure' won't work as expected.
you can add || true to your command.
example:
jobs:
build-and-test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v1
- name: Test App
run: ./gradlew test || true

How to run tests on PR, deploy on merge with Travis CI

I'm automating the testing and deployment of a web app. I have the following stages that need to run:
Test
Deploy (dev)
Deploy (prod)
When a PR is opened, I'd like only stages 1. (test) and 2. (deploy, dev) to run. Then, when the PR is merged to master, I'd like only stage 3. (deploy, prod) to run. I don't want any CI/CD on regular commits/pushes.
Here's how I'm trying to define my .travis.yml:
# install and other things omitted for brevity
stages:
- name: dev-test
if: type = pull_request
branch:
except: master
- name: dev-deploy
if: type = pull_request
branch:
except: master
- name: prod-deploy
if: type = pull_request
branch:
only: master
jobs:
include:
- stage: dev-test
script: python scripts/dev_test.py
- stage: dev-deploy
script: python scripts/dev_deploy.py
- stage: prod-deploy
script: python scripts/prod_deploy.py
Right now all three stages run when a PR is opened (not just dev-test and dev-deploy). How should I set up my stages to accomplish what I want?
I solved this by not using build stages altogether. Instead, I just verify attributes of the GitHub PR in my script directive in .travis.yml, via Travis environment variables.
# install and other things omitted for brevity
script:
- if [[ $TRAVIS_EVENT_TYPE == "pull_request" ]] ; then python scripts/dev_test.py ; fi
- if [[ $TRAVIS_EVENT_TYPE == "pull_request" && $TRAVIS_PULL_REQUEST_BRANCH != "master" ]] ; then python scripts/dev_deploy.py ; fi
- if [[ $TRAVIS_EVENT_TYPE == "push" && $TRAVIS_BRANCH == "master" ]] ; then python scripts/prod_deploy.py ; fi