I'm positive that this is something easy but I'm not able to track down exactly what I'm looking for. I have a workflow that performs a build and creates an artifact. The artifact uses an environment variable in the filename. So far so good. Then when I try to pass this file name to S3 upload action, it isn't found because the environment variable isn't evaluated. Here is the relevant part of my workflow:
- name: Build project
run: ./build_project.sh
- run: ls -l "${GITHUB_WORKSPACE}/build/${FILE_NAME}.zip" # file exists in directory
- run: echo "${GITHUB_WORKSPACE}/build/${FILE_NAME}.zip" # echo returns the location properly
- uses: hkusu/s3-upload-action#v2
id: upload # specify some ID for use in subsequent steps
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: "eu-west-2"
aws-bucket: ${{ secrets.AWS_BUCKET }}
file-path: "${GITHUB_WORKSPACE}/build/${FILE_NAME}.zip" # Error: file does not exist
output-file-url: "true" # specify true
- name: Show URL
run: echo '${{ steps.upload.outputs.file-url }}' # use this output
My actual question is how to replace "${GITHUB_WORKSPACE}/build/${FILE_NAME}.zip" with the file path and name when it actually runs the workflow. Also, I have tried a few of different combinations of things - no quotes, no curly braces, neither, both.
Since these parameters are passed through to the s3-upload-action, the behaviour depends on whether the action expands shell parameters or not, but the input value will be literally
${GITHUB_WORKSPACE}/build/${FILE_NAME}.zip
i.e., unexpanded.
You can use expressions to work around this:
file-path: ${{ github.workspace }}/build/${{ env.FILE_NAME }}.zip
You maybe have assumed that environment variables expand everywhere as they do when evaluated by a shell, such as in a run: step, but they don't.
Related
I'm trying to use a part of current branch name as a tag for publishing docker image in CI pipeline. The problem is that in my org, there is a convention to name branches like feature/foo, and docker tags can't contain slashes. So I want to split the string on '/' symbol and take the last part.
The full branch name, with slashes, should be in variable github.ref_name. Here is the expression I'm trying to evaluate: ${{ github.ref_name.split('/')[-1] }}.
I get this error: Unable to interpolate expression 'format('{0}', github.ref_name.split('/')[-1])': Failed to parse: unexpected token "(" while parsing arguments of function call. expecting ",", ")"
What are the options for manipulating strings in github actions expression? I didn't find it in docs https://docs.github.com/en/actions/learn-github-actions/expressions
IMPROVED:
This technique doesn't need to bring in anything from the actions marketplace.
Add this to steps: in an action:
- name: Split branch name
env:
BRANCH: ${{ github.ref_name }}
id: split
run: echo "::set-output name=fragment::${BRANCH##*/}"
It captures the output of a shell command. The only necessary actions are setting github.ref_name to an environmental variable and using parameter expansion to get the part of the branch name you want.
## greedily removes the subsequent pattern off the front
*/ matches anything followed by a /
So, ##*/ removes everything up to and including the final forward slash.
Because id is split, and name=fragment, you then reference your split string with steps.split.outputs.fragment. For example,
# Another step
- name: Test variable
run: |
echo ${{ steps.split.outputs.fragment }}
Other parameter expansion features would be useful here, like #, %, and %% in addition to ##.
220829: Final update
Originally I spawned another process with $(echo ${BRANCH##*/}) but that is unnecessary. The variable can be directly referenced.
ORIGINAL:
I'm doing something essentially the same and using an action was the easiest temporary workaround, however I also would have preferred something built-in like a string-splitting expression. For now here is what worked.
- uses: jungwinter/split#master
id: branch
with:
msg: ${{ github.ref_name }}
separator: "/"
maxsplit: -1
From its repo:
Outputs
_0, _1, ..., _n: Each result of a split
According to metadata syntax of outputs, it has _ prefix
Currently, support only 100 splits
length: Length of the splits
So, then I can reference each portion of the string with steps.branch.outputs._n, with branch being the id and n being the split index.
I was looking for the first instead of the last value in the branch name, so I could set maxsplit to 1 and just take _0. In your case you might need to do some magic with checking length and choosing the index from there.
I ended up using bash to do the split and publishing it as job output. In the retrospect should have went for just installing python (bash syntax is way more confusing and less maintainable).
build-docker:
container: docker:cli
outputs:
docker-tag: ${{steps.docker-tag.outputs.DOCKER_TAG}}
steps:
- name: add bash
run: |
apk update
apk upgrade
apk add bash
- name: determine docker tag by splitting branch name on slash
id: docker-tag
shell: bash
run: |
split=(${GITHUB_REF_NAME//\// })
index=$((${#split[#]}-1))
DOCKER_TAG=${split[$index]}
echo DOCKER_TAG=$DOCKER_TAG >> $GITHUB_ENV
echo "::set-output name=DOCKER_TAG::$DOCKER_TAG"
- name: Build and push
uses: docker/build-push-action#v3
file: ./docker/ci/Dockerfile
push: true
tags: |
artifactory.com/repo/image:${{ env.DOCKER_TAG }}
test-job:
needs: build-docker
runs-on: [self-hosted, Linux, docker]
container:
image: artifactory.com/repo/image:${{ needs.build-docker.outputs.docker-tag }}
jobs:
my-name:
name: "My Name"
...
steps:
- name: Slack Notification
uses: my-action
with:
slack-msg: ${{ jobs.${{ env.GITHUB_JOB }}.name }}
I want that slack-msg to evaluate to "My Name". I'm using my-action in multiple jobs, and I always want to pass in the job name, but I don't know how to do that. When I tried the above, the job literally didn't run and I don't know how to troubleshoot why: the github workflow log for my-name literally doesn't exist.
How do I pass job-name into an input parameter?
As nested expression are not supported you can use a trick like below to obtaint the matrix job name.
jobs:
test:
env:
# to expose matrix job name to steps, which is not possible with expansions
JOB_NAME: ${{ matrix.name || format('{0} ({1})', matrix.tox-target, matrix.os) }}
name: ${{ matrix.name || format('{0} ({1})', matrix.tox-target, matrix.os) }}
Note that you cannot really access the matrix name, but you can ensure you save the same name into an environment variable and use that.
Here is a part of the workflow file:
env:
resourceGroupName: 'rg-${GITHUB_REF#refs/heads/}'
I am trying to create an environment variable that concats a string and another environment variable but in the logs all I see from an echo from either way I can see is the following:
echo "$resourceGroupName" -> rg-$***GITHUB_REF#refs/heads/***
echo "{{ env.resourceGroupName}}" -> *** env.resourceGroupName***
Instead of what I'd have expected such as:
rg-the-name-of-the-branch
The docs do not seem particularly good in this regard. It also seems to be trying and failing to mask the variables as it's placing erroneous extra asterisks.
You need to have ${{ github.ref }} for it to work with the GitHub Actions vars. The linux envs are also exposed, so you can also use $GITHUB_REF directly without the brackets.
Alternatively, you can also use the format option to combine strings if you like that more:
format("rb-{0}", ${{ github.ref }})
So I'm using github secrets in one of my github actions and I found a weird behavior. When I pass my secrets to a script like this:
- name: Run script
run: python script.py
env:
SPOTIPY_CLIENT_SECRET=: ${{ secrets.SPOTIPY_CLIENT_SECRET}}
SPOTIPY_CLIENT_ID=: ${{ secrets.SPOTIPY_CLIENT_ID}}
SPOTIPY_REDIRECT_URI=: ${{ secrets.SPOTIPY_REDIRECT_URI}}
SPOTIPY_CACHE=: "${{ secrets.SPOTIPY_CACHE}}"
MAIN_TOKEN=: '${{ secrets.MAIN_TOKEN}}'
If I print the values of this tokens they all have a "=" appended on the start of the secret value. So if secret for MAIN_TOKEN is 12345
print(os.environ["MAIN_TOKEN"])
Prints =12345
Is there any way to remove that = from being added to the secret value?
According to the documentation about environment variables, you shouldn't use = to set your variables based on your secrets in your workflow.
This is the correct syntaxe in your case:
- name: Run script
run: python script.py
env:
SPOTIPY_CLIENT_SECRET: ${{ secrets.SPOTIPY_CLIENT_SECRET}}
SPOTIPY_CLIENT_ID: ${{ secrets.SPOTIPY_CLIENT_ID}}
SPOTIPY_REDIRECT_URI: ${{ secrets.SPOTIPY_REDIRECT_URI}}
SPOTIPY_CACHE: ${{ secrets.SPOTIPY_CACHE}}
MAIN_TOKEN: ${{ secrets.MAIN_TOKEN}}
Note that you don't need to inform " or ' if you get the variables from your secrets.
For (mainly) pedagogical reasons, I'm trying to run this workflow in GitHub actions:
name: "We 🎔 Perl"
on:
issues:
types: [opened, edited, milestoned]
jobs:
seasonal_greetings:
runs-on: windows-latest
steps:
- name: Maybe greet
id: maybe-greet
env:
HEY: "Hey you!"
GREETING: "Merry Xmas to you too!"
BODY: ${{ github.event.issue.body }}
run: |
$output=(perl -e 'print ($ENV{BODY} =~ /Merry/)?$ENV{GREETING}:$ENV{HEY};')
Write-Output "::set-output name=GREET::$output"
produce_comment:
name: Respond to issue
runs-on: ubuntu-latest
steps:
- name: Dump job context
env:
JOB_CONTEXT: ${{ jobs.maybe-greet.steps.id }}
run: echo "$JOB_CONTEXT"
I need two different jobs, since they use different context (operating systems), but I need to get the output of a step in the first job to the second job. I am trying with several combinations of the jobs context as found here but there does not seem to be any way to do that. Apparently, jobs is just the name of a YAML variable that does not really have a context, and the context job contains just the success or failure. Any idea?
Check the "GitHub Actions: New workflow features" from April 2020, which could help in your case (to reference step outputs from previous jobs)
Job outputs
You can specify a set of outputs that you want to pass to subsequent jobs and then access those values from your needs context.
See documentation:
jobs.<jobs_id>.outputs
A map of outputs for a job.
Job outputs are available to all downstream jobs that depend on this job.
For more information on defining job dependencies, see jobs.<job_id>.needs.
Job outputs are strings, and job outputs containing expressions are evaluated on the runner at the end of each job. Outputs containing secrets are redacted on the runner and not sent to GitHub Actions.
To use job outputs in a dependent job, you can use the needs context.
For more information, see "Context and expression syntax for GitHub Actions."
To use job outputs in a dependent job, you can use the needs context.
Example
jobs:
job1:
runs-on: ubuntu-latest
# Map a step output to a job output
outputs:
output1: ${{ steps.step1.outputs.test }}
output2: ${{ steps.step2.outputs.test }}
steps:
- id: step1
run: echo "test=hello" >> $GITHUB_OUTPUT
- id: step2
run: echo "test=world" >> $GITHUB_OUTPUT
job2:
runs-on: ubuntu-latest
needs: job1
steps:
- run: echo ${{needs.job1.outputs.output1}} ${{needs.job1.outputs.output2}}
Note the use of $GITHUB_OUTPUT, instead of the older ::set-output now (Oct. 2022) deprecated.
To avoid untrusted logged data to use set-state and set-output workflow commands without the intention of the workflow author we have introduced a new set of environment files to manage state and output.
Jesse Adelman adds in the comments:
This seems to not work well for anything beyond a static string.
How, for example, would I take a multiline text output of step (say, I'm running a pytest or similar) and use that output in another job?
either write the multi-line text to a file (jschmitter's comment)
or base64-encode the output and then decode it in the next job (Nate Karasch's comment)
Update: It's now possible to set job outputs that can be used to transfer string values to downstream jobs. See this answer.
What follows is the original answer. These techniques might still be useful for some use cases.
Write the data to file and use actions/upload-artifact and actions/download-artifact. A bit awkward, but it works.
Create a repository dispatch event and send the data to a second workflow. I prefer this method personally, but the downside is that it needs a repo scoped PAT.
Here is an example of how the second way could work. It uses repository-dispatch action.
name: "We 🎔 Perl"
on:
issues:
types: [opened, edited, milestoned]
jobs:
seasonal_greetings:
runs-on: windows-latest
steps:
- name: Maybe greet
id: maybe-greet
env:
HEY: "Hey you!"
GREETING: "Merry Xmas to you too!"
BODY: ${{ github.event.issue.body }}
run: |
$output=(perl -e 'print ($ENV{BODY} =~ /Merry/)?$ENV{GREETING}:$ENV{HEY};')
Write-Output "::set-output name=GREET::$output"
- name: Repository Dispatch
uses: peter-evans/repository-dispatch#v1
with:
token: ${{ secrets.REPO_ACCESS_TOKEN }}
event-type: my-event
client-payload: '{"greet": "${{ steps.maybe-greet.outputs.GREET }}"}'
This triggers a repository dispatch workflow in the same repository.
name: Repository Dispatch
on:
repository_dispatch:
types: [my-event]
jobs:
myEvent:
runs-on: ubuntu-latest
steps:
- run: echo ${{ github.event.client_payload.greet }}
In my case I wanted to pass an entire build/artifact, not just a string:
name: Build something on Ubuntu then use it on MacOS
on:
workflow_dispatch:
# Allows for manual build trigger
jobs:
buildUbuntuProject:
name: Builds the project on Ubuntu (Put your stuff here)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- uses: some/compile-action#v99
- uses: actions/upload-artifact#v2
# Upload the artifact so the MacOS runner do something with it
with:
name: CompiledProject
path: pathToCompiledProject
doSomethingOnMacOS:
name: Runs the program on MacOS or something
runs-on: macos-latest
needs: buildUbuntuProject # Needed so the job waits for the Ubuntu job to finish
steps:
- uses: actions/download-artifact#master
with:
name: CompiledProject
path: somewhereToPutItOnMacOSRunner
- run: ls somewhereToPutItOnMacOSRunner # See the artifact on the MacOS runner
It is possible to capture the entire output (and return code) of a command within a run step, which I've written up here to hopefully save someone else the headache. Fair warning, it requires a lot of shell trickery and a multiline run to ensure everything happens within a single shell instance.
In my case, I needed to invoke a script and capture the entirety of its stdout for use in a later step, as well as preserve its outcome for error checking:
# capture stdout from script
SCRIPT_OUTPUT=$(./do-something.sh)
# capture exit code as well
SCRIPT_RC=$?
# FYI, this would get stdout AND stderr
SCRIPT_ALL_OUTPUT=$(./do-something.sh 2>&1)
Since Github's job outputs only seem to be able to capture a single line of text, I also had to escape any newlines for the output:
echo "::set-output name=stdout::${SCRIPT_OUTPUT//$'\n'/\\n}"
Additionally, I needed to ultimately return the script's exit code to correctly indicate whether it failed. The whole shebang ends up looking like this:
- name: A run step with stdout as a captured output
id: myscript
run: |
# run in subshell, capturiing stdout to var
SCRIPT_OUTPUT=$(./do-something.sh)
# capture exit code too
SCRIPT_RC=$?
# print a single line output for github
echo "::set-output name=stdout::${SCRIPT_OUTPUT//$'\n'/\\n}"
# exit with the script status
exit $SCRIPT_RC
continue-on-error: true
- name: Add above outcome and output as an issue comment
uses: actions/github-script#v5
env:
STEP_OUTPUT: ${{ steps.myscript.outputs.stdout }}
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
// indicates whather script succeeded or not
let comment = `Script finished with \`${{ steps.myscript.outcome }}\`\n`;
// adds stdout, unescaping newlines again to make it readable
comment += `<details><summary>Show Output</summary>
\`\`\`
${process.env.STEP_OUTPUT.replace(/\\n/g, '\n')}
\`\`\`
</details>`;
// add the whole damn thing as an issue comment
github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: comment
})
Edit: there is also an action to accomplish this with much less bootstrapping, which I only just found.
2022 October update: GitHub is deprecating set-output and recommends to use GITHUB_OUTPUT instead. The syntax for defining the outputs and referencing them in other steps, jobs.
An example from the docs:
- name: Set color
id: random-color-generator
run: echo "SELECTED_COLOR=green" >> $GITHUB_OUTPUT
- name: Get color
run: echo "The selected color is ${{ steps.random-color-generator.outputs.SELECTED_COLOR }}"