I am creating a parallel Flutter unit test on GitLab CI to avoid 60 minutes run-time limit.
The initial command was
flutter test --machine --coverage ./lib > test-report.jsonl
The process took around 59 minutes because we have a lot of unit tests.
In order to reduce the CI pipeline time, I modify the pipeline to be parallel using flutter shard and GitLab CI parallel feature.
The command is like this
flutter test \
--total-shards $CI_NODE_TOTAL \
--shard-index $( expr $CI_NODE_INDEX - 1 ) \
--machine \
--coverage \
--coverage-path coverage/lcov.info-$( expr $CI_NODE_INDEX - 1 ) \
./lib \
> test-report.jsonl-$( expr $CI_NODE_INDEX - 1 )
However, all the parallel jobs still run more than a 60-minute time limit.
What did I miss or how to debug it?
Related
I'm using ArgoWorkflow to automate our CI/CD chains.
In order to build images, and push them to our private registry we are faced between the choice of either buildah or kaniko. But I can't put my finger on the main difference between the two. Pros and cons wise, and also on how do these tools handle parallel builds and cache management. Can anyone clarify these points ? Or even suggest another tool that can maybe do the job in a more simple way.
Some clarifications on the subject would be really helpful.
Thanks in advance.
buildah will require either a privileged container with more then one UID or a container running with CAP_SETUID, CAP_SETGID to build container images.
It is not hacking on the file system like kanicko does to get around these requirements. It runs full contianers when building.
--isolation chroot, will make it a little easier to get buildah to work within kubernetes.
kaniko is very simple to setup and has some magic that let it work with no requirements in kubernetes :)
I also tried buildah but was unable to configure it and found it too complex to setup in a kubernetes environment.
You can use an internal Docker registry as cache management for kaniko, but a local storage can be configured instead (not tried yet). Just use the latest version of kaniko (v1.7.0), that fixes an important bug in the cached layers management.
These are some functions (declared in the file ci/libkaniko.sh) that I use in my GitLab CI pipelines, executed by a GitLab kubernetes runner. They should hopefully clarify setup and usage of kaniko.
function kaniko_config
{
local docker_auth="$(echo -n "$CI_REGISTRY_USER:$CI_REGISTRY_PASSWORD" | base64)"
mkdir -p $DOCKER_CONFIG
[ -e $DOCKER_CONFIG/config.json ] || \
cat <<JSON > $DOCKER_CONFIG/config.json
{
"auths": {
"$CI_REGISTRY": {
"auth": "$docker_auth"
}
}
}
JSON
}
# Usage example (.gitlab-ci.yml)
#
# build php:
# extends: .build
# variables:
# DOCKER_CONFIG: "$CI_PROJECT_DIR/php/.docker"
# DOCKER_IMAGE_PHP_DEVEL_BRANCH: &php-devel-image "${CI_REGISTRY_IMAGE}/php:${CI_COMMIT_REF_SLUG}-build"
# script:
# - kaniko_build
# --destination $DOCKER_IMAGE_PHP_DEVEL_BRANCH
# --dockerfile $CI_PROJECT_DIR/docker/images/php/Dockerfile
# --target devel
function kaniko_build
{
kaniko_config
echo "Kaniko cache enabled ($CI_REGISTRY_IMAGE/cache)"
/kaniko/executor \
--build-arg http_proxy="${HTTP_PROXY}" \
--build-arg https_proxy="${HTTPS_PROXY}" \
--build-arg no_proxy="${NO_PROXY}" \
--cache --cache-repo $CI_REGISTRY_IMAGE/cache \
--context "$CI_PROJECT_DIR" \
--digest-file=/dev/termination-log \
--label "ci.job.id=${CI_JOB_ID}" \
--label "ci.pipeline.id=${CI_PIPELINE_ID}" \
--verbosity info \
$#
[ -r /dev/termination-log ] && \
echo "Manifest digest: $(cat /dev/termination-log)"
}
With these functions a new image can be built with:
stages:
- build
build app:
stage: build
image:
name: gcr.io/kaniko-project/executor:v1.7.0-debug
entrypoint: [""]
variables:
DOCKER_CONFIG: "$CI_PROJECT_DIR/app/.docker"
DOCKER_IMAGE_APP_RELEASE_BRANCH: &app-devel-image "${CI_REGISTRY_IMAGE}/phelps:${CI_COMMIT_REF_SLUG}"
GIT_SUBMODULE_STRATEGY: recursive
before_script:
- source ci/libkaniko.sh
script:
- kaniko_build
--destination $DOCKER_IMAGE_APP_RELEASE_BRANCH
--digest-file $CI_PROJECT_DIR/docker-content-digest-app
--dockerfile $CI_PROJECT_DIR/docker/Dockerfile
artifacts:
paths:
- docker-content-digest-app
tags:
- k8s-runner
Note that you have to use the debug version of kaniko executor because this image tag provides a shell (and other busybox based binaries).
I have several pipelines in my project, and I need to schedule them in specific dates.
Is it possible to execute a pipeline every 11th day?
Is it possible to visualize all the pipelines in the project and see the next one to be executed or a table or something with all the pipelines schedules the way I'm working is the "classic view" not YAML.
I am sorry if the doubts seem dumb or something, but I'm searching and I can't find what I am looking for.
Scheduled triggers in YAML cause a pipeline to run on a schedule defined using cron syntax:
mm HH DD MM DW
\ \ \ \ \__ Days of week
\ \ \ \____ Months
\ \ \______ Days
\ \________ Hours
\__________ Minutes
So, if you want it will run every 11th of the month, you can achieve in this syntax:
0 0 11 * *
In your YAML:
schedules:
- cron: "0 0 11 * *" # replace with your schedule
You can use this site to check the syntax.
Regarding your second question, I'm afraid that impossible with YAML pipelines, there is no "table" or something :/
But, it's a good idea, I suggest you create a feature request here.
I want to execute multiple tests at once but am getting a nastygram:
pytest tests -k '1.1.18 or 4.1.12 or 6.1.11 or 6.1.12' ...
...
ERROR: Wrong expression passed to '-k': 1.1.18 or 4.1.12 or 6.1.11 or 6.1.12
pytest tests -k 1.1.18 ... works fine to execute one test. My team has structured our tests using suite files that are specific to OS distros and associate the test numbers with scripts & methods like so:
tests:
- test_function: "foo.py:test_bar1"
test_id: "1.1.1.1"
- test_function: "foo.py:test_bar2"
test_id: "1.2"
...
What am I doing wrong?
First pip install pytest-xdist
Then use pytest -n NUM where NUM is the integer of the number you want running in parrallel.
So to run 3 in parallel.
pytest -n3
Or
pytest -n 3
Documentation link:
https://docs.pytest.org/en/3.0.1/xdist.html
EDIT
-k takes test function names. You want them to be unique.
tests:
- test_function1: "foo.py:test_bar1"
test_id: "1.1.1.1"
- test_function2: "foo.py:test_bar2"
test_id: "1.2"
...
pytest -k "test_function1 or test_function1"
Note in windows cmd you must use " not '
I understand that dataproc workflow-templates is still in beta, but how do you pass parameters via the add-job into the executable sql? Here is a basic example:
#/bin/bash
DATE_PARTITION=$1
echo DatePartition: $DATE_PARTITION
# sample job
gcloud beta dataproc workflow-templates add-job hive \
--step-id=0_first-job \
--workflow-template=my-template \
--file='gs://mybucket/first-job.sql' \
--params="DATE_PARTITION=$DATE_PARTITION"
gcloud beta dataproc workflow-templates run $WORK_FLOW
gcloud beta dataproc workflow-templates remove-job $WORK_FLOW --step-
id=0_first-job
echo `date`
Here is my first-job.sql file called from the shell:
SET hive.input.format=org.apache.hadoop.hive.ql.io.CombineHiveInputFormat;
SET mapred.output.compress=true;
SET hive.exec.compress.output=true;
SET mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec;
SET io.compression.codecs=org.apache.hadoop.io.compress.GzipCodec;
USE mydb;
CREATE EXTERNAL TABLE if not exists data_raw (
field1 string,
field2 string
)
PARTITIONED BY (dt String)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
LOCATION 'gs://data/first-job/';
ALTER TABLE data_raw ADD IF NOT EXISTS PARTITION(dt="${hivevar:DATE_PARTITION}");
In the ALTER TABLE statement, what is the correct syntax? I’ve tried what feels like over 15 variations but nothing works. If I hard code it like this (ALTER TABLE data_raw ADD IF NOT EXISTS PARTITION(dt="2017-10-31");) the partition gets created, but unfortunately it needs to be parameterized.
BTW – The error I receive is consistently like this:
Error: Error while compiling statement: FAILED: ParseException line 1:48 cannot recognize input near '${DATE_PARTITION}' ')' '' in constant
I am probably close but not sure what I am missing.
TIA,
Melissa
Update: Dataproc now has workflow template parameterization, a beta feature:
https://cloud.google.com/dataproc/docs/concepts/workflows/workflow-parameters
For your specific case, you can do the following:
Create an empty template
gcloud beta dataproc workflow-templates create my-template
Add a job with a placeholder for the value you want to parameterize
gcloud beta dataproc workflow-templates add-job hive \
--step-id=0_first-job \
--workflow-template=my-template \
--file='gs://mybucket/first-job.sql' \
--params="DATE_PARTITION=PLACEHOLDER"
Export the template configuration to a file
gcloud beta dataproc workflow-templates export my-template \
--destination=hive-template.yaml
Edit the file to add a parameter
jobs:
- hiveJob:
queryFileUri: gs://mybucket/first-job.sql
scriptVariables:
DATE_PARTITION: PLACEHOLDER
stepId: 0_first-job
parameters:
- name: DATE_PARTITION
fields:
- jobs['0_first-job'].hiveJob.scriptVariables['DATE_PARTITION']
Import the changes
gcloud beta dataproc workflow-templates import my-template \
--source=hive-template.yaml
Add a managed cluster or cluster selector
gcloud beta dataproc workflow-templates set-managed-cluster my-template \
--cluster-name=my-cluster \
--zone=us-central1-a
Run your template with parameters
gcloud beta dataproc workflow-templates instantiate my-template \
--parameters="DATE_PARTITION=${DATE_PARTITION}"
Thanks for trying out Workflows! First-class support for parameterization is part of our roadmap. However for now your remove-job/add-job trick is the best way to go.
Regarding your specific question:
Values passed via params are accessed as ${hivevar:PARAM} (see [1]). Alternatively, you can set --properties which are accessed as ${PARAM}
The brackets around params are not needed. If it's intended to handle spaces in parameter values use quotations like: --params="FOO=a b c,BAR=X"
Finally, I noticed an errant space here DATE_PARTITION =$1 which probably results in empty DATE_PARTITION value
Hope this helps!
[1] How to use params/properties flag values when executing hive job on google dataproc
I have run robot suite with 3 tests, 2/3 was passed. After that I run same pybot command with option --rerun first_run_output.xml, 1 test was passed. Next I run rebot -o merged.xml --merge first_run_output.xml rerun_output.xml
In merged.xml the test witch was failed and rerun in merged.xml has starttime="N/A" and endtime="N/A", this value propagate up in xml (parents nodes has also "N/A" value in timestamps)
Reassuming:
- run pybot - A-pass B-pass C-failed -> generate first_run_output.xml
- run pybot --rerun - C-passed -> generate rerun_output.xml
- run rebot -o merged.xml --merge first_run_output.xml rerun_output.xml
- In first_run_output.xml and rerun_output.xml ale starttime and endtime has data value
- In merged.xml in Node <status> for C has starttime="N/A" endtime="N/A" and parents of suite C (thats aggregate A,B,C) also have starttime="N/A" endtime="N/A"
I ask same question on robotframe-user group,
answer is "This is by design"
details: https://groups.google.com/forum/#!topic/robotframework-users/TWvZMtbizrs