Introducing secret variables to dockerfile on Github Actions - github

I am trying to configure my etc/pip.conf file to download a private PyPi artifactory while using a secret variable on my dockerfile.
Dockerfile
FROM python
WORKDIR ./app
COPY . /app
RUN pip install --upgrade pip
RUN pip install -r pre-requirements.txt
RUN echo ${{ secrets.PIP }} > etc/pip.conf
RUN pip install -r post-requirements.txt
CMD ["python", "./simpleflask.py"]
docker-image.yml
name: CI
on:
push:
branches: [ "main" ]
pull_request:
branches: [ "main" ]
workflow_dispatch:
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Setup JFrog CLI
uses: jfrog/setup-jfrog-cli#v2
env:
JF_ARTIFACTORY_SERVER: ${{ secrets.JFROG_CLI }}
- name: Checkout
uses: actions/checkout#v3
- name: Build
run: |
docker build -t simple-flask .
docker tag simple-flask awakzdev.jfrog.io/docker-local/simple-flask:latest
docker push awakzdev.jfrog.io/docker-local/simple-flask:latest
pretty simple and straightfoward but my pipeline returns the following
Step 6/8 : RUN echo ${{ secrets.PIP }} > etc/pip.conf
---> Running in deb3e3f4167f
/bin/sh: 1: Bad substitution
The command '/bin/sh -c echo ${{ secrets.PIP }} > etc/pip.conf' returned a non-zero code: 2
Error: Process completed with exit code 2.
Edit :
Trying a slightly difference approach and went to install dependencies in the pipeline
my .yml looks like this now
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Setup JFrog CLI
uses: jfrog/setup-jfrog-cli#v2
env:
JF_ARTIFACTORY_SERVER: ${{ secrets.JFROG_CLI }}
- name: Checkout
uses: actions/checkout#v3
- name: install dependencies
run: |
pip config -v list
echo "${{ secrets.PIP }}" > /etc/pip.conf
pip install ganesha-experimental==2.0.1
- name: Build
run: |
docker build -t simple-flask .
docker tag simple-flask awakzdev.jfrog.io/docker-local/simple-flask:latest
docker push awakzdev.jfrog.io/docker-local/simple-flask:latest
but the following error is being returned:
1s
Run pip config -v list
For variant 'global', will try loading '/etc/xdg/pip/pip.conf'
For variant 'global', will try loading '/etc/pip.conf'
For variant 'user', will try loading '/home/runner/.pip/pip.conf'
For variant 'user', will try loading '/home/runner/.config/pip/pip.conf'
For variant 'site', will try loading '/usr/pip.conf'
/home/runner/work/_temp/09382b8f-ce09-4646-816f-fb337f40ad4b.sh: line 2: /etc/pip.conf: Permission denied
Error: Process completed with exit code 1.

I've placed the secret on my .yml file instead.
as for the broken pip permissions I used
sudo chown runner /etc/
echo ${{ secrets.PIP }} > /etc/pip.conf
which resulted in another error with the contents of the pip.conf file (it was configured correctly through secrets)
so I found you can specify the url like so
ganesha_experimental==5.0.0 --find-links=https://awakzdev.jfrog.io/artifactory/

Related

Jmeter upload test artifacts on GIT

Hello I want to upload the HTML file generated from the execution of my Jmeter, unfortunately I'm encountering an error upon executing my script. Your response is highly appreciated. Thank you
Here's my YAML file.
name: CI
on:
push:
branches: [ "main" ]
pull_request:
branches: [ "main" ]
workflow_dispatch:
inputs:
choice:
type: choice
description: Environment
options:
- test
- dev
- uat
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v3
- name: setup-jmeter
run: |
sudo apt-get update
sudo apt install curl -y
sudo apt install -y default-jdk
sudo curl -O https://archive.apache.org/dist/jmeter/binaries/apache-jmeter-5.3.tgz
sudo tar -xvf apache-jmeter-5.3.tgz
cd $GITHUB_WORKSPACE/apache-jmeter-5.3/lib && sudo curl -O https://repo1.maven.org/maven2/kg/apc/cmdrunner/2.2.1/cmdrunner-2.2.1.jar
cd $GITHUB_WORKSPACE/apache-jmeter-5.3/lib/ext && sudo curl -O https://repo1.maven.org/maven2/kg/apc/jmeter-plugins-manager/1.6/jmeter-plugins-manager-1.6.jar
cd $GITHUB_WORKSPACE/apache-jmeter-5.3/lib && sudo java -jar cmdrunner-2.2.1.jar --tool org.jmeterplugins.repository.PluginManagerCMD install-all-except jpgc-hadoop,jpgc-oauth,ulp-jmeter-autocorrelator-plugin,ulp-jmeter-videostreaming-plugin,ulp-jmeter-gwt-plugin,tilln-iso8583
- name: run-jmeter-test
run: |
echo "choice is ${{ github.event.inputs.choice }}" / ${{ inputs.choice }}
$GITHUB_WORKSPACE/apache-jmeter-5.3/bin/./jmeter.sh -n -t testGIT.jmx -Jchoice="${{ github.event.inputs.choice }}" -l result.jtl -e -o $GITHUB_WORKSPACE/html/test
- name: Upload Results
uses: actions/upload-artifact#v2
with:
name: jmeter-results
path: result.jtl
- name: Upload HTML
uses: actions/upload-artifact#v2
with:
name: jmeter-results-HTML
path: index.html
Expected Result:
I should able to see 2 entries for the result one for jmeter-results and the other one is jmeter-results-HTML.
Screenshot:
Note: the index.html generated from my local this is what I want to display from my execution
You're creating HTML Reporting Dashboard under html/test folder and trying to upload index.html file from the current folder. I believe you need to change the artifact path to
path: html/test/index.html
It doesn't make sense to archive index.html alone, it relies on the content and sbadmin2-1.0.7 folders so it's better to consider uploading the whole folder otherwise the dashboard will not be usable.
According to JMeter Best Practices you should always be using the latest version of JMeter so consider upgrading to JMeter 5.5 (or whatever is the latest stable version available at JMeter Downloads page)

How to limit pytest code coverage calculation to only specific files?

I am trying to setup continues integration in github using actions. I setup a action to get the code coverage automatically, this part works great, however it's calculating the code coverage for every file in my src directory, this is not preferred, I would like it to calculate the code coverage for only the files modified, how do I do that?
name: Pytest Coverage
on:
pull_request:
branches: [ main, dev ]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- name: Set up Python 3.10
uses: actions/setup-python#v2
with:
python-version: "3.10"
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install flake8 pytest pytest-cov
if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
- name: Changed Files Exporter
id: files
uses: umani/changed-files#v3.3.0
with:
repo-token: ${{ github.token }}
pattern: '^src.*\.(py)$'
result-encoding: 'string'
- name: Build coverage file
run: |
echo "${{ steps.files.outputs.files_updated }} ${{ steps.files.outputs.files_created }}"
pytest --cache-clear --cov-branch --cov=src > pytest-coverage.txt
- name: Comment coverage
uses: coroo/pytest-coverage-commentator#v1.0.2
- name: Get Coverage %
run: |
LAST_LINE=$(tail -4 pytest-coverage.txt)
LAST_LINE=$(head -n 1 <<< "$LAST_LINE")
COVERAGE=$(sed 's/.*[[:space:]]\([0-9]\+\)%/\1/' <<< "$LAST_LINE")
echo "Overall code coverage is $COVERAGE"
echo 'SCORE<<EOF' >> $GITHUB_ENV
echo "$SCORE" >> $GITHUB_ENV
echo 'EOF' >> $GITHUB_ENV
The important 2 lines are here:
echo "${{ steps.files.outputs.files_updated }} ${{ steps.files.outputs.files_created }}"
pytest --cache-clear --cov-branch --cov=src > pytest-coverage.txt
The echo print out the correct files:
src/scripts/testing_script.py src/server.py
How do I change this line so it ONLY calculate the total coverage of thoese 2 modified files?
pytest --cache-clear --cov-branch --cov=src > pytest-coverage.txt

I don't have access my codes in the runner in GitHub Actions

I created the following "main.yml" file.
name: Deploy
# Controls when the workflow will run
on:
# Triggers the workflow on push or pull request events but only for the main branch
push:
branches: [ main ]
pull_request:
branches: [ main ]
# Allows you to run this workflow manually from the Actions tab
workflow_dispatch:
# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
# This workflow contains a single job called "build"
build:
# The type of runner that the job will run on
runs-on: ubuntu-latest
# Steps represent a sequence of tasks that will be executed as part of the job
steps:
- name: test
run: ls -al && cd .. && ls -al
- name: Create SSH key
run: |
mkdir -p ~/.ssh/
echo "$SSH_PRIVATE_KEY" > ~/.ssh/private.key
sudo chmod 600 ~/.ssh/private.key
ssh-keyscan -H ${{secrets.SSH_HOST}} > ~/.ssh/known_hosts
echo "Host ${{secrets.SSH_HOST}}
User ${{secrets.SSH_USER}}
IdentityFile ~/.ssh/private.key" > ~/.ssh/config
cat ~/.ssh/config
shell: bash
env:
SSH_PRIVATE_KEY: ${{secrets.SSH_PRIVATE_KEY}}
- name: test-remote
run: rsync -r ${{secrets.SSH_USER}}#${{secrets.SSH_HOST}}:~/${{secrets.SSH_HOST}}
- name: Deploy with rsync
run: cd .. && ls -al && rsync -avz ./ ${{ secrets.SSH_USER }}#${{ secrets.SSH_HOST }}:/var/www/${{ secrets.SSH_HOST }}
However, I cannot access my codes in the github repository as seen in the following output in the runner.
Maybe I'm using the rsync command incorrectly, so I tried to output with ls and even to output from its parent directory. How do you think I can solve it?
Junior things... I forgot to checkout in the beginning. I added checkout to the beginning of the steps as below and the problem was solved.
- name: Checkout
uses: actions/checkout#main

github action azure/login#v1 not working on self hosted git runner?

Anyone familiar with this issue? The example from https://github.com/Azure/cli does not work on self-hosted github runner it seems as az is missing
gitaction.yml
name: auzure-deployment
on:
push:
branches: [ main ]
jobs:
myjob:
runs-on: [self-hosted, linux]
steps:
- uses: azure/login#v1
with:
creds: ${{ secrets.AZURE_CREDENTIALS }}
- uses: azure/CLI#v1
with:
azcliversion: 2.0.72
inlineScript: |
az account list
error
Runner group name: 'Default'
Machine name: '98de1add3979'
GITHUB_TOKEN Permissions
Prepare workflow directory
Prepare all required actions
Getting action download info
Download action repository 'azure/login#v1'
Download action repository 'azure/CLI#v1'
0s
Run azure/login#v1
Error: Az CLI Login failed. Please check the credentials. For more information refer https://aka.ms/create-secrets-for-GitHub-workflows
Error: Error: Unable to locate executable file: az. Please verify either the file path exists or the file can be found within a directory specified by the PATH environment variable. Also check the file mode to verify the file is executable.
I hacked a workaround to not have to use loging#v1 but its not elegant, as they secrets are printed to the git log prompt:
name: auzure-deployment
on:
push:
branches: [ main ]
jobs:
buildandpush:
runs-on: [self-hosted, linux]
env:
credentials: ${{ secrets.AZURE_CREDENTIALS }}
AZURE_CLIENT_ID: ${{ fromJSON(secrets.AZURE_CREDENTIALS)['clientId'] }}
AZURE_CLIENT_SECRET: ${{ fromJSON(secrets.AZURE_CREDENTIALS)['clientSecret'] }}
AZURE_TENANT_ID: ${{ fromJSON(secrets.AZURE_CREDENTIALS)['tenantId'] }}
- uses: azure/CLI#v1
with:
azcliversion: 2.0.72
inlineScript: |
az login --service-principal -u $AZURE_CLIENT_ID -p $AZURE_CLIENT_SECRET --tenant $AZURE_TENANT_ID
az account list
There's now an open issue to install the cli in the login action if it doesn't exist: https://github.com/Azure/login/issues/154
The work around on self hosted runners is to install the cli before the login action using another action like this https://github.com/elstudio/action-install-azure-cli
or to not be dependent on someone elses action, run the commands directly from the script in the above repo.
- name: Install Azure cli
run: |
sudo apt-get install ca-certificates curl apt-transport-https lsb-release gnupg
curl -sL https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/microsoft.gpg > /dev/null
AZ_REPO=$(lsb_release -cs)
echo "deb [arch=amd64] https://packages.microsoft.com/repos/azure-cli/ $AZ_REPO main" | sudo tee /etc/apt/sources.list.d/azure-cli.list
sudo apt-get update
sudo apt-get install azure-cli
- uses: azure/login#v1
with:
creds: ${{ secrets.AZURE_CREDENTIALS }}
- uses: azure/CLI#v1
with:
azcliversion: 2.0.72
inlineScript: |
az account list

/bin/sh: /packages/flutter_tools/bin/xcode_backend.sh: No such file or directory in github actions

when I build my project in github actions, shows this error:
/bin/sh: /packages/flutter_tools/bin/xcode_backend.sh: No such file or directory
in my local machine, it could build success. the is the log output:
+---------------+------------------------------+
| Build environment |
+---------------+------------------------------+
| xcode_path | /Applications/Xcode_11.7.app |
| gym_version | 2.172.0 |
| export_method | ad-hoc |
| sdk | iPhoneOS13.7.sdk |
+---------------+------------------------------+
[16:09:59]: ▸ export YACC=yacc
[16:09:59]: ▸ export arch=arm64
[16:09:59]: ▸ export variant=normal
[16:09:59]: ▸ /bin/sh -c /Users/runner/Library/Developer/Xcode/DerivedData/Runner-gzzbtgmsqethlzedjqlbspydxjjv/Build/Intermediates.noindex/ArchiveIntermediates/Runner/IntermediateBuildFilesPath/Runner.build/Release-iphoneos/Runner.build/Script-9740EEB61CF901F6004384FC.sh
[16:09:59]: ▸ /bin/sh: /packages/flutter_tools/bin/xcode_backend.sh: No such file or directory
[16:09:59]:
[16:09:59]: ⬆️ Check out the few lines of raw `xcodebuild` output above for potential hints on how to solve this error
[16:09:59]: 📋 For the complete and more detailed error log, check the full log at:
[16:09:59]: 📋 /Users/runner/Library/Logs/gym/Runner-Runner.log
[16:09:59]:
[16:09:59]: Looks like fastlane ran into a build/archive error with your project
[16:09:59]: It's hard to tell what's causing the error, so we wrote some guides on how
[16:09:59]: to troubleshoot build and signing issues: https://docs.fastlane.tools/codesigning/getting-started/
[16:09:59]: Before submitting an issue on GitHub, please follow the guide above and make
[16:09:59]: sure your project is set up correctly.
[16:09:59]: fastlane uses `xcodebuild` commands to generate your binary, you can see the
[16:09:59]: the full commands printed out in yellow in the above log.
[16:09:59]: Make sure to inspect the output above, as usually you'll find more error information there
and this is my github action script:
name: Cruise-CI
on:
push:
branches: [ main ]
jobs:
build:
runs-on: macos-10.15
steps:
- uses: actions/checkout#v2
- uses: actions/setup-java#v1
with:
java-version: '12.x'
- uses: subosito/flutter-action#v1
with:
flutter-version: '1.22.5'
- name: Select Xcode version
run: sudo xcode-select -s '/Applications/Xcode_11.7.app/Contents/Developer'
- name: Bundle install
run: cd ./ios && flutter clean && gem install cocoapods -v 1.10.0 && bundle install && bundle update fastlane
- name: Install tools
run: |
flutter precache
flutter pub get
cd ./ios && pod repo update && pod install
#- run: flutter pub get
#- run: flutter build apk
#- run: flutter build ios --release --no-codesign
- name: Setup SSH Keys and known_hosts for fastlane match
run: |
SSH_PATH="$HOME/.ssh"
mkdir -p "$SSH_PATH"
touch "$SSH_PATH/known_hosts"
echo "$PRIVATE_KEY" > "$SSH_PATH/id_rsa"
chmod 700 "$SSH_PATH"
ssh-keyscan github.com >> ~/.ssh/known_hosts
chmod 600 "$SSH_PATH/known_hosts"
chmod 600 "$SSH_PATH/id_rsa"
eval $(ssh-agent)
ssh-add "$SSH_PATH/id_rsa"
env:
PRIVATE_KEY: ${{ secrets.SSH_PRIVATE_KEY }}
- name: Deploy to TestFlight/PGY
run: |
cd ./ios && security unlock-keychain ${HOME}/Library/Keychains/login.keychain && flutter clean && bundle exec fastlane beta
env:
APPLE_ID: ${{ secrets.APPLE_ID }}
GIT_URL: ${{ secrets.GIT_URL }}
PGY_USER_KEY: ${{ secrets.PGY_USER_KEY }}
PGY_API_KEY: ${{ secrets.PGY_API_KEY }}
TEAM_ID: ${{ secrets.TEAM_ID }}
ITC_TEAM_ID: ${{ secrets.ITC_TEAM_ID }}
FASTLANE_USER: ${{ secrets.FASTLANE_USER }}
FASTLANE_PASSWORD: ${{ secrets.FASTLANE_PASSWORD }}
FASTLANE_APPLE_APPLICATION_SPECIFIC_PASSWORD: ${{ secrets.FASTLANE_APPLE_APPLICATION_SPECIFIC_PASSWORD }}
FASTLANE_SESSION: ${{ secrets.FASTLANE_SESSION }}
MATCH_PASSWORD: ${{ secrets.MATCH_PASSWORD }}
MATCH_KEYCHAIN_NAME: ${{ secrets.MATCH_KEYCHAIN_NAME }}
MATCH_KEYCHAIN_PASSWORD: ${{ secrets.MATCH_KEYCHAIN_PASSWORD }}
DELIVER_ITMSTRANSPORTER_ADDITIONAL_UPLOAD_PARAMETERS: ${{ secrets.DELIVER_ITMSTRANSPORTER_ADDITIONAL_UPLOAD_PARAMETERS }}
what should I do to fix it? I am searching from internet and tell me to set flutter_root,but I did not know how to set the variable in github action, seem it management by system and dit not need to set.