I'm using Github Actions to implement a CI pipeline in my project. Currently, I'm trying to use actions/cache#v2 to cache yarn cache dir to improve the pipeline time. Unfortunately, always that the actions/cache#v2 runs I'm getting an error in the post-job saying: /bin/tar: unrecognized option: posix. The complete log is:
Post job cleanup.
/usr/bin/docker exec 4decc52e7744d9ab2e81bb24c99a830acc848912515ef1e86fbb9b8d5049c9cf sh -c "cat /etc/*release | grep ^ID"
/bin/tar --posix -z -cf cache.tgz -P -C /__w/open-tuna-api/open-tuna-api --files-from manifest.txt
/bin/tar: unrecognized option: posix
BusyBox v1.31.1 () multi-call binary.
Usage: tar c|x|t [-ZzJjahmvokO] [-f TARFILE] [-C DIR] [-T FILE] [-X FILE] [--exclude PATTERN]... [FILE]...
Create, extract, or list files from a tar file
c Create
x Extract
t List
-f FILE Name of TARFILE ('-' for stdin/out)
-C DIR Change to DIR before operation
-v Verbose
-O Extract to stdout
-m Don't restore mtime
-o Don't restore user:group
-k Don't replace existing files
-Z (De)compress using compress
-z (De)compress using gzip
-J (De)compress using xz
-j (De)compress using bzip2
-a (De)compress using lzma
-h Follow symlinks
-T FILE File with names to include
-X FILE File with glob patterns to exclude
--exclude PATTERN Glob pattern to exclude
Warning: Tar failed with error: The process '/bin/tar' failed with exit code 1
I'm following the example of the official action cache repository. Here a snippet of my CI.yml
# Configure cache
- name: Get yarn cache directory path
id: yarn-cache-dir-path
run: echo "::set-output name=dir::$(yarn cache dir)"
- uses: actions/cache#v2
id: yarn-cache # use this to check for `cache-hit` (`steps.yarn-cache.outputs.cache-hit != 'true'`)
with:
path: ${{ steps.yarn-cache-dir-path.outputs.dir }}
key: ${{ runner.os }}-yarn-${{ hashFiles('yarn.lock') }}
restore-keys: |
${{ runner.os }}-yarn-
Because of the above error, the cache is not created and the pipeline time is not improved. I've tried changing the hasFiles expression and the entire key, but no success.
My question is: Am I making some mistake in the use of Action Cache? Can anyone help me with this issue? Thanks.
Your problem is that you're running inside an Alpine Linux-based container. Alpine Linux is designed for small size, and as a result it replaces many of the standard GNU utilities with those from busybox, a multi-call binary. Your version of tar is one of those.
The actions/cache#v2 action uses tar --posix, which tells tar to create a standard pax-format archive. pax archives are a form of tar archive that can handle arbitrarily long filenames, huge file sizes, and other types of metadata that tar archives cannot. This format is specified by POSIX and is a better choice than GNU tar-style archives because it works across a variety of systems and isn't specified by what one implementation does, in addition to being more featureful.
However, the version of tar shipped as part of busybox doesn't support the --posix option, and as a result this command fails. If you want to use the actions/cache#v2 GitHub Action, then you need to provide a version of GNU or BSD (libarchive) tar earlier in your PATH before running it so that that command can be used instead of busybox's.
Related
Suppose i've got a zip file available under some URL. I need to get its hash, which should be identical to the one output by nix-prefetch-url --unpack <URL>, but without a working Nix installation. How can one do it?
Seems there is no easy way, as nix-prefetch-url adds the file to the store. More details here: https://discourse.nixos.org/t/generate-a-file-hash-similar-to-the-one-output-by-nix-prefetch-url/19907 (many thanks to prompt and thorough community member's response)
Use Docker.
Demo:
$ nix-prefetch-url --unpack https://github.com/hraban/git-hly/archive/06ff628d5f2b02d1a883c94b01d58187d117f4f3.tar.gz
path is '/nix/store/gxx1pfp19s3a39j6gl0xw197b4409cmp-06ff628d5f2b02d1a883c94b01d58187d117f4f3.tar.gz'
164gyvpdm6l6rdvn2rwjz95j1jz0w2igcbk9shy862sdx2rdw9hn
$ # Or .zip: it's the same, because of --unpack:
$ nix-prefetch-url --unpack https://github.com/hraban/git-hly/archive/06ff628d5f2b02d1a883c94b01d58187d117f4f3.zip
path is '/nix/store/1bpjlzknnmq1x3hq213r44jwag1xkaqs-06ff628d5f2b02d1a883c94b01d58187d117f4f3.zip'
164gyvpdm6l6rdvn2rwjz95j1jz0w2igcbk9shy862sdx2rdw9hn
Download to a local directory
$ cd "$(mktemp -d)"
$ curl -sSL --fail https://github.com/hraban/git-hly/archive/06ff628d5f2b02d1a883c94b01d58187d117f4f3.tar.gz | tar xz
$ cd *
And test it:
$ # Using the modern nix command:
$ nix hash path --base32 .
164gyvpdm6l6rdvn2rwjz95j1jz0w2igcbk9shy862sdx2rdw9hn
$ # Or the same, using nix-hash:
$ nix-hash --type sha256 --base32 .
164gyvpdm6l6rdvn2rwjz95j1jz0w2igcbk9shy862sdx2rdw9hn
Same in Docker:
$ docker run --rm -v "$PWD":/data nixos/nix nix --extra-experimental-features nix-command hash path --base32 /data
164gyvpdm6l6rdvn2rwjz95j1jz0w2igcbk9shy862sdx2rdw9hn
$ docker run --rm -v "$PWD":/data nixos/nix nix-hash --type sha256 --base32 /data
164gyvpdm6l6rdvn2rwjz95j1jz0w2igcbk9shy862sdx2rdw9hn
P.S.: I'm not a huge fan of nix-prefetch-url's default output (base32). The default output of nix hash path is better, if you can use it:
$ nix hash path .
sha256-FibesuhNC4M81Gku9qLg4MsgS/qSZ2F3y4aa2u72j5g=
$ # Sanity check:
$ nix-hash --type sha256 --to-base32 $(<<<"FibesuhNC4M81Gku9qLg4MsgS/qSZ2F3y4aa2u72j5g=" base64 -d | hexdump -v -e '/1 "%02x"' )
164gyvpdm6l6rdvn2rwjz95j1jz0w2igcbk9shy862sdx2rdw9hn
I was trying to deploy a NextJS application to Elastic Beanstalk via eb deploy. But the source bundle failed to unzip during deployment as the source bundle contained some pre-built .next page which the file name is in UTF-8 encoding. The error is stated as below.
2022/xx/xx xx:xx:xx.xxxxxx [INFO] Executing instruction: StageApplication
2022/xx/xx xx:xx:xx.xxxxxx [INFO] extracting /opt/elasticbeanstalk/deployment/app_source_bundle to /var/app/staging/
2022/01/31 04:56:44.300483 [INFO] Running command /bin/sh -c /usr/bin/unzip -q -o /opt/elasticbeanstalk/deployment/app_source_bundle -d /var/app/staging/
2022/01/31 04:56:45.932820 [ERROR] An error occurred during execution of command [app-deploy] - [StageApplication]. Stop running the command. Error: Command /bin/sh -c /usr/bin/unzip -q -o /opt/elasticbeanstalk/deployment/app_source_bundle -d /var/app/staging/ failed with error exit status 50. Stderr:error: cannot create /var/app/staging/.next/server/pages/\u6e2c\u8a66/\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66.html
File name too long
error: cannot create /var/app/staging/.next/server/pages/\u6e2c\u8a66/\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66.json
File name too long
I was able to unzip the file with option -O UTF-8, is there any way I could add this flag to the eb deploy unzip process?
edit 1. I am working with the Platform 64bit Amazon Linux 2/5.4.9
Not sure if it is good practice, but I eventually added an ebextensions to overcome the original unzipping flow.
commands:
command backup original zip:
command: |
logger "backup zip" && cp /opt/elasticbeanstalk/deployment/app_source_bundle /tmp/app_source_bundle_bak &&
logger "rm existing zip .next folder" && zip -Ad /opt/elasticbeanstalk/deployment/app_source_bundle ".next/*"
cwd: /home/ec2-user
ignoreErrors: false
container_commands:
replace the original zip to staging:
command: |
logger "custom unzip" &&
unzip -O UTF-8 -q -o /tmp/app_source_bundle_bak -d /var/app/staging/
cwd: /home/ec2-user
ignoreErrors: false
I have a GitHub project with a Hugo-based web site in it. Whenever someone pushes something to the prod branch, I want to build the Hugo page (transform Markdown files to HTML) and upload it to my hosting provider. I have problems building the page.
I have this script in GitHub Actions:
name: Publish prod branch
on:
push:
branches:
- prod
jobs:
build:
name: Greeting
runs-on: ubuntu-latest
steps:
- name: Hello world
uses: actions/hello-world-javascript-action#v1
with:
who-to-greet: Dmitrii
id: hello
- name: Echo the greeting's time
run: echo 'The time was ${{ steps.hello.outputs.time }}.'
- name: Build Hugo
uses: srt32/hugo-action#master
It fails because it does not find the configuration file config.toml, even though it is there:
/usr/bin/docker run --name e87b520e21a5125f094485b4e030650bd57153_f8bc76 --label e87b52 --workdir /github/workspace --rm -e HOME -e GITHUB_REF -e GITHUB_SHA -e GITHUB_REPOSITORY -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_ACTOR -e GITHUB_WORKFLOW -e GITHUB_HEAD_REF -e GITHUB_BASE_REF -e GITHUB_EVENT_NAME -e GITHUB_WORKSPACE -e GITHUB_ACTION -e GITHUB_EVENT_PATH -e RUNNER_OS -e RUNNER_TOOL_CACHE -e RUNNER_TEMP -e RUNNER_WORKSPACE -e ACTIONS_RUNTIME_URL -e ACTIONS_RUNTIME_TOKEN -e ACTIONS_CACHE_URL -e GITHUB_ACTIONS=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/work/_temp/_github_home":"/github/home" -v "/home/runner/work/_temp/_github_workflow":"/github/workflow" -v "/home/runner/work/hugo-wp-site/hugo-wp-site":"/github/workspace" e87b52:0e21a5125f094485b4e030650bd57153
Error: Unable to locate config file or config directory. Perhaps you need to create a new site.
#################################################
Run `hugo help new` for details.
Starting the Hugo Action
Total in 0 ms
How can I fix it, i. e. make hugo see my config.toml file?
Update 1: I tried to find out the version of Hugo being used by modifying the script as follows:
name: Publish prod branch
on:
push:
branches:
- prod
jobs:
build:
name: Build and publish web site to hosting provider
runs-on: ubuntu-latest
steps:
- name: Hello world
uses: actions/hello-world-javascript-action#v1
with:
who-to-greet: Dmitrii
id: hello
- name: Echo the greeting's time
run: echo 'The time was ${{ steps.hello.outputs.time }}.'
- name: Output the version of Hugo
run: hugo version
- name: Build Hugo
uses: srt32/hugo-action#master
But when I run it, I get the following error:
hugo version
shell: /bin/bash -e {0}
/home/runner/work/_temp/9e57960c-2f2c-4f2a-870c-c1cbc41d820f.sh: line 1: hugo: command not found
##[error]Process completed with exit code 127.
Update 2: Found out the version of Hugo in the output:
(7/7) Installing hugo (0.61.0-r0)
Update 3: The earliest Hugo version that may have the issue 6794 fixed is v0.64.0 because that issue was merged on January 31st and v0.64.0 is the first version that came out after that day.
Update 4: It seems that in order to fix this error, I need to make sure that the Hugo action uses a more recent version of Hugo. To achieve this, I changed the Dockerfile so that version 0.65.3-r0 is installed (according to this answer):
RUN apk add --no-cache hugo=0.65.3-r0 bash
But when I run the script, Alpine Linux fails to install Hugo:
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/community/x86_64/APKINDEX.tar.gz
ERROR: unsatisfiable constraints:
hugo-0.61.0-r0:
breaks: world[hugo=0.65.3-r0]
The command '/bin/sh -c apk add --no-cache hugo=0.65.3-r0 bash' returned a non-zero code: 1
##[warning]Docker build failed with exit code 1, back off 9.558 seconds before retry.
/usr/bin/docker build -t e87b52:dfe904e1240c4dbea120e452e5568b51 "/home/runner/work/_actions/dpisarenko/hugo-action/master"
Sending build context to Docker daemon 7.168kB
Any help on how to fix this is highly appreciated.
Update 5: After changing the section for installation of Hugo to
RUN echo "http://dl-cdn.alpinelinux.org/alpine/edge/community" >> /etc/apk/repositories
RUN apk update
RUN apk add --no-cache hugo=0.65.3-r0 bash
the action installs a more recent version of Hugo:
Step 10/13 : RUN apk add --no-cache hugo=0.65.3-r0 bash
---> Running in 633b06ba9a65
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/community/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz
(1/7) Installing ncurses-terminfo-base (6.1_p20200118-r2)
(2/7) Installing ncurses-libs (6.1_p20200118-r2)
(3/7) Installing readline (8.0.1-r0)
(4/7) Installing bash (5.0.11-r1)
Executing bash-5.0.11-r1.post-install
(5/7) Installing libgcc (9.2.0-r3)
(6/7) Installing libstdc++ (9.2.0-r3)
(7/7) Installing hugo (0.65.3-r0)
But I still get the same error:
Run dpisarenko/hugo-action#master
/usr/bin/docker run --name e87b52fba2a6bbd65d4e86b03264ae4ae92e94_cbeaf6 --label e87b52 --workdir /github/workspace --rm -e HOME -e GITHUB_REF -e GITHUB_SHA -e GITHUB_REPOSITORY -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_ACTOR -e GITHUB_WORKFLOW -e GITHUB_HEAD_REF -e GITHUB_BASE_REF -e GITHUB_EVENT_NAME -e GITHUB_WORKSPACE -e GITHUB_ACTION -e GITHUB_EVENT_PATH -e RUNNER_OS -e RUNNER_TOOL_CACHE -e RUNNER_TEMP -e RUNNER_WORKSPACE -e ACTIONS_RUNTIME_URL -e ACTIONS_RUNTIME_TOKEN -e ACTIONS_CACHE_URL -e GITHUB_ACTIONS=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/work/_temp/_github_home":"/github/home" -v "/home/runner/work/_temp/_github_workflow":"/github/workflow" -v "/home/runner/work/hugo-wp-site/hugo-wp-site":"/github/workspace" e87b52:fba2a6bbd65d4e86b03264ae4ae92e94
#################################################
Starting the Hugo Action
Error: Unable to locate config file or config directory. Perhaps you need to create a new site.
Run `hugo help new` for details.
Update 6: I added the commands pwd and ls -al into the file entrypoint.sh in which hugo is being called:
echo "pwd:"
pwd
echo "ls -al:"
ls -al
hugo "$#"
Here is its output:
Starting the Hugo Action
pwd:
/github/workspace
ls -al:
total 8
drwxr-xr-x 2 1001 115 4096 Mar 15 17:39 .
drwxr-xr-x 5 root root 4096 Mar 15 17:39 ..
Error: Unable to locate config file or config directory. Perhaps you need to create a new site.
Run `hugo help new` for details.
It seems that the action tries to run hugo inside the directory /github/workspace which is empty.
My next step is to find out in which directory the contens of my git branch is located.
Update 7: I tried to output the contents of the directories
/home/runner/work/_temp/_github_home,
/github/home,
/home/runner/work/_temp/_github_workflow,
/github/workflow,
/home/runner/work/hugo-wp-site/hugo-wp-site, and
/github/workspace
in entrypoint.sh, but none of them contains my Hugo code.
Update 8: I added the following line to entrypoint.sh to find the directory with Hugo sources:
find / -name "*archetypes*"
All Hugo projects contain that directory.
But find did not find anything. It looks like Docker of the GitHub action is running in the wrong directory.
This is based on srt32/hugo-action which uses possibly an older version of Hugo.
Check first the hugo version, to see if issue 6794 applies (it was fixed in january 2020 with PR 6834.
It seems that the Hugo code was not checked out at all. Therefore the solution is to modify the GitHub action so that
git is installed in the Dockerized Linux and
the Hugo source code is checked out.
To do the former, the Dockerfile needs to be modified like shown below (see RUN apk add --no-cache git):
FROM alpine:latest
LABEL "com.github.actions.name"="Hugo Actions"
LABEL "com.github.actions.description"="Commands to help with building Hugo based static sites"
LABEL "com.github.actions.icon"="mic"
LABEL "com.github.actions.color"="yellow"
LABEL "repository"="http://github.com/dpisarenko/hugo-action"
LABEL "homepage"="http://github.com/dpisarenko/hugo-action"
RUN echo "http://dl-cdn.alpinelinux.org/alpine/edge/community" >> /etc/apk/repositories
RUN apk update
RUN apk add --no-cache hugo=0.65.3-r0 bash
RUN apk add --no-cache bash
RUN apk add --no-cache git
ADD entrypoint.sh /
RUN chmod +x /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]
Then we need to call git clone in entrypoint.sh:
#!/bin/bash
set -e
echo "#################################################"
echo "Starting the Hugo Action"
git clone --branch prod https://github.com/dpisarenko/hugo-wp-site.git /hugo
cd /hugo
hugo "$#"
echo "#################################################"
echo "Completed the Hugo Action"
I have the below .sh code which need to get converted to Ansible tasks.
#!/bin/sh
echo "Installing Sonar"
SONAR_HOME=/tui/hybris/sonar
if [ ! -d "$SONAR_HOME" ]; then
mkdir -p $SONAR_HOME
fi
cd $SONAR_HOME
wget https://s3-eu-west-1.amazonaws.com/tuiuk/source/sonarqube/sonarqube-5.4.zip
unzip sonarqube-5.4.zip
echo "Modifying Sonar config file"
cd sonarqube-5.4/conf
perl -p -i -e 's/#sonar.jdbc.username=/sonar.jdbc.username=sonar/g' sonar.properties
perl -p -i -e 's/#sonar.jdbc.password=/sonar.jdbc.password=sonar/g' sonar.properties
perl -p -i -e 's/#sonar.jdbc.url=jdbc:mysql/sonar.jdbc.url=jdbc:mysql/g' sonar.properties
cd $SONAR_HOME
echo "downloading and copying plugins"
wget https://s3-eu-west-1.amazonaws.com/tuiuk/source/sonarqube/sonarqube5.4_plugins.zip
unzip sonarqube5.4_plugins.zip
cp plugins/* sonarqube-5.4/extensions/plugins/
cd sonarqube-5.4/bin/linux-x86-64
echo "Starting Sonar"
./sonar.sh start
Below is my task.I got stuck where I need to execute perl script. Could any of you help me in proceeding further.
- hosts: docker_test
tasks:
- name: Creates directory
file: path=/tui/hybris/sonar state=directory mode=0777
sudo: yes
- name: Installing Sonar
get_url:
url: "https://s3-eu-west-1.amazonaws.com/tuiuk/source/sonarqube/sonarqube-5.4.zip"
dest: "/tui/hybris/sonar/sonarqube-5.4.zip"
register: get_solr
- debug:
msg: "solr was downloaded"
when: get_solr|changed
- name: Unzip SonarQube
unarchive: src=/tui/hybris/sonar/sonarqube-5.4.zip dest=/tui/hybris/sonar copy=no
I bet you don't need perl here, use lineinfile with regex option (if you need to modify a single line in the file) or replace module (if you need to modify all occurrences).
Just call perl with command or shell-module:
- task: Modifying Sonar config file
shell: cd sonarqube-5.4/conf && perl -p -i -e ...
I'm trying to build an RPM of a node.js project I'm working on. The project's code is entirely contained in one directory and its subdirectories. Here's part of my build script:
Put everything in the SOURCES directory, except the node_modules
echo "Copying sources..."
cd $DIR/dashboard
tar -zcf $DIR/rpmbuild/SOURCES/project-$VERSION.tgz . --exclude=node_modules > /dev/null
cd $DIR
# Create the spec file
touch $DIR/rpmbuild/SPECS/specfile
# write to it
cat >$DIR/rpmbuild/SPECS/specfile <<EOL
Summary: [Summary]
Name: [Name]
Version: $VERSION
Release: 1
License: GPL
Group: Applications/WebApp
Source: [Name]-$VERSION.tgz
Vendor: None
Requires: nodejs
%description
[Description]
%prep
echo "in prep"
echo $(ls -al)
%setup -c
npm install
yes | ./compile_all.sh
mv app/index.html app/index_big.html
mv app/index_small.html app/index.html
%build
echo "Nothing to build!"
%install
mkdir -p /opt/project
mv -r ./ /opt/project
ln -s /opt/project/server_start /usr/local/bin
ln -s /opt/project/server_stop /usr/local/bin
%files
$DIR/rpmbuild/SOURCES
$DIR/rpmbuild/SPECS
EOL
HOME=`pwd` rpmbuild -bs $DIR/rpmbuild/SPECS/specfile
The compile_all.sh script just minifies the html, css, and javascript and concatenates them. index_small.html uses the smaller versions. $DIR points to the location of the script, which is one directory above the project code.
The script does indeed generate the source rpm. I've looked at it using mc, and inside CONTENTS.cpio there's the specfile and the tgz archive of the source code. However, trying to install doesn't work. With rpm -Vp project.rpm, I can see that it lists the specfile and the tgz archive as missing. With verbose mode enabled, I see something similar to:
Directories not explicitly included in package:
/home/USERNAME/rpmbuild/SPECS/
/home/USERNAME/rpmbuild/SOURCES`
My rpmbuild directory isn't in my home directory, it's in the same directory as the build script. This leads to my actual question: how can I point rpmbuild at the correct location?