Is it possible to make the build stage parallel?
today the build stage builds and deploys all the images in a sequence, which takes quite a lot of time. it would save a lot of time if each image will be built in parallel to the others (same as the deploy stage).
The deploy stage does run in parallel, unless you opt to deploy them in order with the stages.deployments field in your pipeline manifest.
As for the build stage, you can make changes to your own pipeline's buildspec, specifically in this block:
for env in $pl_envs; do
tag=$(sed 's/:/-/g' <<<"${CODEBUILD_BUILD_ID##*:}-${env}" | rev | cut -c 1-128 | rev)
for svc in $svcs; do
./copilot-linux svc package -n $svc -e $env --output-dir './infrastructure' --tag $tag --upload-assets;
if [ $? -ne 0 ]; then
echo "Cloudformation stack and config files were not generated. Please check build logs to see if there was a manifest validation error." 1>&2;
exit 1;
fi
done;
for job in $jobs; do
./copilot-linux job package -n $job -e $env --output-dir './infrastructure' --tag $tag --upload-assets;
if [ $? -ne 0 ]; then
echo "Cloudformation stack and config files were not generated. Please check build logs to see if there was a manifest validation error." 1>&2;
exit 1;
fi
done;
done;
Related
Is there a way to use the github-cli or api to view the inputs of an action while it is running?
I want to allow Github actions to run concurrently. The resources they will manage are determined by the input stack_name. I want to make sure two pipelines cannot run at the same time with the same stack_name input. If this happens, then I want one of the pipeline actions to fail and stop immediately.
I am also taking the input and turning it into an environmental variable for one of my jobs. After the job finishes, the values are available in the logs and I can grep through the following output to get a pipelines stack_name:
$ gh run view $running_pipeline_id --repo=$GITHUB_SERVER_URL/$GITHUB_REPOSITORY --log
....
env-check env-check 2022-03-22T17:06:30.2615395Z STACK_NAME: foo
However, this is not available while a job is running and I instead get this error:
run 1234567890 is still in progress; logs will be available when it is complete
Here is my current attempt at a code block that can achieve this. I could also use suggestions on how to make better gh run list and/or gh run view calls that can avoid using grep and awk. Clean json output I can parse with jq is preferable.
set +e
running_pipeline_ids=$(gh run list --workflow=$SLEEVE --repo=$GITHUB_SERVER_URL/$GITHUB_REPOSITORY \
| grep 'in_progress' \
| awk '{print $((NF-2))}' \
| grep -v $GITHUB_RUN_ID)
set -e
for running_pipeline_id in $running_pipeline_ids; do
# get the stack name for all other running pipelines
running_pipeline_stack_name=$(gh run view $running_pipeline_id --repo=$GITHUB_SERVER_URL/$GITHUB_REPOSITORY --log \
| grep 'STACK_NAME:' | head -n 1 \
| awk -F "STACK_NAME:" '{print $2}' | awk '{print $1}')
# fail if we detect another pipeline running against the same stack
if [ "$running_pipeline_stack_name" == "$STACK_NAME" ]; then
echo "ERROR: concurrent pipeline detected. $GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$running_pipeline_id"
echo "Please try again after the running pipeline has completed."
exit 1
fi
done
Perhaps you could use the concurrency feature of GitHub Actions?
Now you cannot directly bake this into an action, but if it's possible for you to extract your action into a reusable workflow then you could make use of the concurrency feature.
It would look something like this:
# ./github/workflows/partial.yaml
on:
workflow_call:
inputs:
stack-name:
description: "name of the stack"
required: true
type: string
jobs:
greet:
runs-on: ubuntu-latest
concurrency:
group: ${{ inputs.stack-name }}
cancel-in-progress: true
steps:
- uses: my/other-action
with:
stack_name: ${{ inputs.stack-name }}
And then where you're using it:
jobs:
test:
uses: my/app-repo/.github/workflows/partial.yml#main
with:
stack-name: 'my-stack'
How to check if the spark job is succeeded or failed programmatically, while running spark-submit. Usually the unix exit code is used.
phase: Failed
container status:
container name: spark-kubernetes-driver
container image: <regstry>/spark-py:spark3.2.1
container state: terminated
container started at: 2022-03-25T19:10:51Z
container finished at: 2022-03-25T19:10:57Z
exit code: 1
termination reason: Error
2022-03-25 15:10:58,457 INFO submit.LoggingPodStatusWatcherImpl: Application Postgres-Minio-Kubernetes.py with submission ID spark:postgres-minio-kubernetes-py-b70d3f7fc27829ec-driver finished
2022-03-25 15:10:58,465 INFO util.ShutdownHookManager: Shutdown hook called
2022-03-25 15:10:58,466 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-3321e67c-73d5-422d-a26d-642a0235cf23
Process failed and when I get the exit code in unix by echo $? it return a zero error code!
$ echo $?
0
The pod generated is also random way. What's the way the spark-submit is handled apart from using sparkonk8operator?
If you are using bash, one way to to grep on the output. You might have to grep on stderr or stdout depending on where the log output is being sent.
Something like this:
OUTPUT=`spark-submit ...`
if echo "$OUTPUT" | grep -q "exit code: 1"; then
exit 1
fi
In addition to things which #Rico mentioned, I have also considered the deployment mode of cluster and client with changing spark-submit shell file in $SPARK_HOME/bin directory as follows.
#!/usr/bin/env bash
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
if [ -z "${SPARK_HOME}" ]; then
source "$(dirname "$0")"/find-spark-home
fi
# disable randomized hash for string in Python 3.3+
export PYTHONHASHSEED=0
# check deployment mode.
if echo "$#" | grep -q "\-\-deploy-mode cluster";
then
echo "cluster mode..";
# temp log file for spark job.
export TMP_LOG="/tmp/spark-job-log-$(date '+%Y-%m-%d-%H-%M-%S').log";
exec "${SPARK_HOME}"/bin/spark-class org.apache.spark.deploy.SparkSubmit "$#" |& tee ${TMP_LOG};
# when exit code 1 is contained in spark log, then return exit 1.
if cat ${TMP_LOG} | grep -q "exit code: 1";
then
echo "exit code: 1";
rm -rf ${TMP_LOG};
exit 1;
else
echo "job succeeded.";
rm -rf ${TMP_LOG};
exit 0;
fi
elif echo "$#" | grep -q "\-\-conf spark.submit.deployMode=cluster";
then
echo "cluster mode..";
# temp log file for spark job.
export TMP_LOG="/tmp/spark-job-log-$(date '+%Y-%m-%d-%H-%M-%S').log";
exec "${SPARK_HOME}"/bin/spark-class org.apache.spark.deploy.SparkSubmit "$#" |& tee ${TMP_LOG};
# when exit code 1 is contained in spark log, then return exit 1.
if cat ${TMP_LOG} | grep -q "exit code: 1";
then
echo "exit code: 1";
rm -rf ${TMP_LOG};
exit 1;
else
echo "job succeeded.";
rm -rf ${TMP_LOG};
exit 0;
fi
else
echo "client mode..";
exec "${SPARK_HOME}"/bin/spark-class org.apache.spark.deploy.SparkSubmit "$#"
fi
Then, I have built and pushed my spark docker image.
See the following link for more details:
https://itnext.io/things-to-consider-to-submit-spark-jobs-on-kubernetes-766402c21716
When deploying an application into dedicated Bluemix it uses DEA architecture by default. How can I force it to use DIEGO architecture instead?
You have to use more steps. Deploy without start, switch to diego, start.
cf push APPLICATION_NAME --no-start
cf disable-diego APPLICATION_NAME
cf start APPLICATION_NAME
Ref Deploying Apps
I built a bash exec to do this, which will use your existing manifest.yml file and pack all of this into a single request. The contents of the bash exec follow:
#!/bin/bash
filename="manifest.yml"
if [ -e $filename ];
then
echo "using manifest.yml file in this directory"
else
echo "no manifest.yml file found. exiting"
exit -2
fi
shopt -s nocasematch
string='name:'
targetName=""
echo "Retrieving name from manifest file"
while read -r line
do
name="$line"
variable=${name%%:*}
if [[ $variable == *"name"* ]]
then
inBound=${name#*:}
targetName="$(echo -e "${inBound}" | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//')"
fi
done < "$filename"
if [ "$targetName" == "" ];
then
echo "Could not find name of application in manifest.yml file. Cancelling build."
echo "application name is identified by the 'name: ' term in the manifest.yml file"
exit -1
else
echo "starting cf push for $targetName"
cf push --no-start
echo "cf enable-diego $targetName"
cf enable-diego $targetName
echo "cf start $targetName"
cf start $targetName
exit 0
fi
Just put this code into your editor as a new file and then make the file executable. I keep a copy of this exec in each of my repos in the root directory. After doing a copy-paste and executing this exec, you may get the following error:
/bin/bash^M: bad interpreter: No such file or directory
If you do, just run the dos2unix command and it will 'fix up' the line endings to match your os.
In a fastlane project that I am taking over everything is run by command line (e.g. not fastfile). In this project (using fastlane 1.7) there are aliases used for arguments. Where would I go to find out what each of the aliases map to as far as fastlane commands? For example:
def build(Myapp, skip_profile)
if skip_profile || download_provisioning_profiles(MyApp)
build_cmd = "gym -a -r -s #{MyApp.name} -o ./build -n #{MyApp.ipa_name} --use_legacy_build_api"
system(build_cmd)
else
puts "Was unable to install provisioning profiles"
exit 1
end
end
Looking at this I am pretty sure that -o is the output but where would I look to find out explicitly what -a and -r and -s and -o are?
Run
fastlane gym --help
to get a list of all available options for the gym tool.
Hi great people of stackoverflow,
Were hosting a docker container on EB with an nodejs based code running on it.
When redeploying our docker container we'd like the old one to do a graceful shutdown.
I've found help & guides on how our code could receive a sigterm signal produced by 'docker stop' command.
However further investigation into the EB machine running docker at:
/opt/elasticbeanstalk/hooks/appdeploy/enact/01flip.sh
shows that when "flipping" from current to the new staged container, the old one is killed with 'docker kill'
Is there any way to change this behaviour to docker stop?
Or in general a recommended approach to handling graceful shutdown of the old container?
Thanks!
Self answering as I've found a solution that works for us:
tl;dr: use .ebextensions scripts to run your script before 01flip, your script will make sure a graceful shutdown of whatevers inside the docker takes place
first,
your app (or whatever your'e running in docker) has to be able to catch a signal, SIGINT for example, and shutdown gracefully upon it.
this is totally unrelated to Docker, you can test it running wherever (locally for example)
There is a lot of info about getting this kind of behaviour done for different kind of apps on the net (be it ruby, node.js etc...)
Second,
your EB/Docker based project can have a .ebextensions folder that holds all kinda of scripts to execute while deploying.
we put 2 custom scripts into it, gracefulshutdown_01.config and gracefulshutdown_02.config file that looks something like this:
# gracefulshutdown_01.config
commands:
backup-original-flip-hook:
command: cp -f /opt/elasticbeanstalk/hooks/appdeploy/enact/01flip.sh /opt/elasticbeanstalk/hooks/appdeploy/01flip.sh.bak
test: '[ ! -f /opt/elasticbeanstalk/hooks/appdeploy/01flip.sh.bak ]'
cleanup-custom-hooks:
command: rm -f 05gracefulshutdown.sh
cwd: /opt/elasticbeanstalk/hooks/appdeploy/enact
ignoreErrors: true
and:
# gracefulshutdown_02.config
commands:
reorder-original-flip-hook:
command: mv /opt/elasticbeanstalk/hooks/appdeploy/enact/01flip.sh /opt/elasticbeanstalk/hooks/appdeploy/enact/10flip.sh
test: '[ -f /opt/elasticbeanstalk/hooks/appdeploy/enact/01flip.sh ]'
files:
"/opt/elasticbeanstalk/hooks/appdeploy/enact/05gracefulshutdown.sh":
mode: "000755"
owner: root
group: root
content: |
#!/bin/sh
# find currently running docker
EB_CONFIG_DOCKER_CURRENT_APP_FILE=$(/opt/elasticbeanstalk/bin/get-config container -k app_deploy_file)
EB_CONFIG_DOCKER_CURRENT_APP=""
if [ -f $EB_CONFIG_DOCKER_CURRENT_APP_FILE ]; then
EB_CONFIG_DOCKER_CURRENT_APP=`cat $EB_CONFIG_DOCKER_CURRENT_APP_FILE | cut -c 1-12`
echo "Graceful shutdown on app container: $EB_CONFIG_DOCKER_CURRENT_APP"
else
echo "NO CURRENT APP TO GRACEFUL SHUTDOWN FOUND"
exit 0
fi
# give graceful kill command to all running .js files (not stats!!)
docker exec $EB_CONFIG_DOCKER_CURRENT_APP sh -c "ps x -o pid,command | grep -E 'workers' | grep -v -E 'forever|grep' " | awk '{print $1}' | xargs docker exec $EB_CONFIG_DOCKER_CURRENT_APP kill -s SIGINT
echo "sent kill signals"
# wait (max 5 mins) until processes are done and terminate themselves
TRIES=100
until [ $TRIES -eq 0 ]; do
PIDS=`docker exec $EB_CONFIG_DOCKER_CURRENT_APP sh -c "ps x -o pid,command | grep -E 'workers' | grep -v -E 'forever|grep' " | awk '{print $1}' | cat`
echo TRIES $TRIES PIDS $PIDS
if [ -z "$PIDS" ]; then
echo "finished graceful shutdown of docker $EB_CONFIG_DOCKER_CURRENT_APP"
exit 0
else
let TRIES-=1
sleep 3
fi
done
echo "failed to graceful shutdown, please investigate manually"
exit 1
gracefulshutdown_01.config is a small util that backups the original flip01 and deletes (if exists) our custom script.
gracefulshutdown_02.config is where the magic happens.
it creates a 05gracefulshutdown enact script and makes sure flip will happen afterwards by renaming it to 10flip.
05gracefulshutdown, the custom script, does this basically:
find current running docker
find all processes that need to be sent a SIGINT (for us its processes with 'workers' in its name
send a sigint to the above processes
loop:
check if processes from before were killed
continue looping for an amount of tries
if tries are over, exit with status "1" and dont continue to 10flip, manual interference is needed.
this assumes you only have 1 docker running on the machine, and that you are able to manually hop on to check whats wrong in the case it fails (for us never happened yet).
I imagine it can also be improved in many ways, so have fun.