Cloudbuild wait for artifacts upload before specific step - google-cloud-storage

Question

I wrote a cloudbuild.yaml file that doing deploy for application to Compute Engine, the process take the code and build it with go build ..., then archive the binary file and upload to Cloud Storage, then create Compute Engine template that have startup-script which read the file from cloud storage and doing the deploy and initialization for each machine. These are the relevant steps:
- name: 'mirror.gcr.io/library/golang:1.18-buster'
id: 'build-app'
env: [
'GO111MODULE=on',
'GOPROXY=https://proxy.golang.org,direct',
'GOOS=linux',
'GOARCH=amd64'
]
args: ['go', 'build', '-o', 'deploy/usr/bin/app', './services/service-name/']
- name: 'debian'
id: 'tar-app-file'
args: [ 'tar', '-czf', '${_DEPLOY_FILENAME}', '-C', './deploy', '.' ]
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
id: 'move-startup-script'
args: [ 'gsutil', 'cp', './services/service-name/startup-script.sh', '${_STARTUP_SCRIPT_URL}' ]
- name: 'gcr.io/cloud-builders/gcloud'
id: 'create-template'
args: [ 'compute', 'instance-templates', 'create', 'MY_NICE_TEMPLATE',
....
'--metadata', 'app-location=${_DEPLOY_DIR}${_DEPLOY_FILENAME},startup-script-url=${_STARTUP_SCRIPT_URL}' ]
# ... more steps that replace that instance group template to the newly created one using "gcloud compute instance-groups managed rolling-action" command
substitutions:
_DEPLOY_DIR: 'gs://bucket-name/deploy/service-name/${COMMIT_SHA}/'
_DEPLOY_FILENAME: 'app.tar.gz'
_STARTUP_SCRIPT_URL: 'gs://bucket-name/deploy/service-name/startup-script.sh'
artifacts:
objects:
location: '${_DEPLOY_DIR}'
paths: ['${_DEPLOY_FILENAME}']
The startup script file:
#! /bin/sh
set -ex
APP_LOCATION=$(curl -s "http://metadata.google.internal/computeMetadata/v1/instance/attributes/app-location" -H "Metadata-Flavor: Google")
gsutil cp "$APP_LOCATION" app.tar.gz
tar -xzf app.tar.gz
# Start the service included in app.tar.gz.
service service-name start
The problem is that sometimes the startup script run before the build artifcate finish uploaded, so the file is not yet exist in Cloud Storage so I get this error
startup-script-url: CommandException: No URLs matched: gs://bucket-name/deploy/service-name/some-commit-sha-123/app.tar.gz
And the build is finished successfully, so eventullay there is an instance up and running that didn't start up properly.
How can I tell cloudbuild to wait for artifacts upload to finish before starting a new step?
How can I mark the build as failed in case the startup script failed? So the instance group won't update in this case (not necessarily related to the specific error above, but any error)?

Answer 1

This is expected because you're depending on the artifacts statement.
This statement will upload the artifacts only when all the steps are done so you're incurring in a race condition.
There is no way to say to Cloud Build to upload the artifacts before finishing the steps when using:
artifacts:
objects:
location: '${_DEPLOY_DIR}'
paths: ['${_DEPLOY_FILENAME}']
Then you may need to explicitly upload them in a step before updating your MIG:
...
- name: 'debian'
id: 'tar-app-file'
args: [ 'tar', '-czf', '${_DEPLOY_FILENAME}', '-C', './deploy', '.' ]
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
id: 'upload-artifacts'
args: [ 'gsutil', 'cp', '${_DEPLOY_FILENAME}', '${_DEPLOY_DIR}' ]
...