Prepare coursier artifact for offline use inside container - scala

I have an sbt project producing my artifact xyz.
I would like to put it along with all its dependencies in the docker container so it can be used using
coursier launch --mode offline xyz
The important part is that preparation should take use of local cursier cache from host.
I tried
executing sbt publishLocal,
then resolving my artifact dependencies (cursier resolve xyz),
then preparing to directories - local & cache - by copying resolved artifact into them
then copying those directories into docker container (as coursier cache and ivy local respectively).
This didn't work because coursier doesn't list .pom and .xml files in its output. I tried copying whole directories (abc/1.0.0 instead of abc/1.0.0/some.jar) but AFAIK there is no reliable way to know how many folders up one has to go because maven and ivy have different dir structures.

while my usecase is not quite identical to yours -- I figure I'd write up my findings and maybe my solution works for you as well!
here's my sample dockerfile, I used this to install scalafmt in an offline-compatible way
FROM ubuntu:jammy
RUN : \
&& apt-get update \
&& apt-get install -y --no-install-recommends \
ca-certificates \
curl \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/* # */ stackoverflow highlighting bug
ARG CS=v2.1.0-RC4
ARG CS_SHA256=176e92e08ab292531aa0c4993dbc9f2c99dec79578752f3b9285f54f306db572
ARG JDK_SHA256=aef49cc7aa606de2044302e757fa94c8e144818e93487081c4fd319ca858134b
ENV PATH=/opt/coursier/bin:$PATH
RUN : \
&& curl --location --silent --output /tmp/cs.gz "https://github.com/coursier/coursier/releases/download/${CS}/cs-x86_64-pc-linux.gz" \
&& echo "${CS_SHA256} /tmp/cs.gz" | sha256sum --check \
&& curl --location --silent --output /tmp/jdk.tgz "https://download.java.net/openjdk/jdk17/ri/openjdk-17+35_linux-x64_bin.tar.gz" \
&& echo "${JDK_SHA256} /tmp/jdk.tgz" | sha256sum --check \
&& mkdir -p /opt/coursier \
&& tar --strip-components=1 -C /opt/coursier -xf /tmp/jdk.tgz \
&& gunzip /tmp/cs.gz \
&& mv /tmp/cs /opt/coursier/bin \
&& chmod +x /opt/coursier/bin/cs \
&& rm /tmp/jdk.tgz
ENV COURSIER_CACHE=/opt/.cs-cache
RUN : \
&& cs fetch scalafmt:3.6.1 \
&& cs install scalafmt:3.6.1 --dir /opt/wd/bin
the key to offline execution for me was to use cs fetch and set COURSIER_CACHE
here's the offline execution succeeding:
$ docker run --net=none --rm -ti cs /opt/wd/bin/scalafmt --version
scalafmt 3.6.1

Related

Deploying Jenkins using skaffold via GitHub Action Runner

I am deploying Jenkins Using GitHub Action Runner using Skaffold.
While the Skaffold is installed over the default image of GitHub Runner
The pod is restarting due to crash loop back off error and causing it to restart.
I am not sure why it is happening.
When I am deploying runner over Google Kubernetes Engine my runner is failing because of following error:
'''A runner exists with the same name
√ Successfully replaced the runner
√ Runner connection is good
# Runner settings
√ Settings Saved.
√ Connected to GitHub
Current runner version: '2.294.0'
2022-12-01 06:03:57Z: Listening for Jobs
Runner update in progress, do not shutdown runner.
Downloading 2.299.1 runner
Waiting for current job finish running.
Generate and execute update script.
Runner will exit shortly for update, should be back online within 10 seconds.
Runner update process finished.
Runner listener exit because of updating, re-launch runner in 5 seconds
Restarting runner...
/home/docker/actions-runner/run-helper.sh: line 20: /home/docker/actions-runner/bin/Runner.Listener: No such file or directory
Exiting with unknown error code: 127
Exiting runner...
'''
Following is the Dockerfile used for runner :
'''FROM ubuntu:22.04
#instalIng skaffold
RUN apt-get update -y && apt-get upgrade -y sudo
RUN apt-get install -y curl
RUN curl -LO https://storage.googleapis.com/skaffold/releases/v2.0.2/skaffold-linux-amd64 \
&& sudo chmod +x skaffold-linux-amd64 \
&& sudo mv skaffold-linux-amd64 /usr/local/bin/skaffold
#install
ARG DEBIAN_FRONTEND=noninteractive
RUN apt-get update -y && apt-get upgrade -y && useradd -m docker
RUN apt-get install -y curl jq build-essential libssl-dev libffi-dev python3 python3-venv python3-dev ca-certificates gnupg2 iputils-ping software-properties-common apt-transport-https lsb-release git zip unzip postgresql-client python3-pip npm
RUN ln -sf /usr/bin/python3 /usr/bin/python
# set the github runner version
ARG RUNNER_VERSION="2.294.0"
# cd into the user directory, download and unzip the github actions runner
RUN cd /home/docker && mkdir actions-runner && cd actions-runner \
&& curl -O -L https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz \
&& tar xzf ./actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz
# install some additional dependencies
RUN chown -R docker ~docker && /home/docker/actions-runner/bin/installdependencies.sh
#Docker
RUN curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add - && \
add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable" && \
apt-get update && \
apt-get -y install docker-ce
# Downloading gcloud package
RUN curl https://dl.google.com/dl/cloudsdk/release/google-cloud-sdk.tar.gz > /tmp/google-cloud-sdk.tar.gz
# Installing the package
RUN mkdir -p /usr/local/gcloud \
&& tar -C /usr/local/gcloud -xvf /tmp/google-cloud-sdk.tar.gz \
&& /usr/local/gcloud/google-cloud-sdk/install.sh
# Adding the package path to local
ENV PATH $PATH:/usr/local/gcloud/google-cloud-sdk/bin
#Install Kubectl
RUN curl -LO "https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl" \
&& chmod +x ./kubectl \
&& mv ./kubectl /usr/local/bin/kubectl
# copy over the start.sh script
COPY start.sh start.sh
# make the script executable
RUN chmod +x start.sh && mv start.sh /home/docker
# since the config and run script for actions are not allowed to be run by root,
# set the user to "docker" so all subsequent commands are run as the docker user
USER docker
# set the entrypoint to the start.sh script
ENTRYPOINT ["/home/docker/start.sh"] '''
Below is the startup script :
#!/bin/bash
SNAPTIME=`date '+%Y%m%d%H%M%S'`
echo "Started $SNAPTIME"
ORGANIZATION=$ORGANIZATION
ACCESS_TOKEN=`cat /etc/pat/pat`
GH_PROJECT=$GH_PROJECT
RUNNER_NAME="${RUNNER_NAME:-RUN$SNAPTIME}"
RUNNER_LABELS="${RUNNER_LABELS:-simple}"
REG_TOKEN=$(curl -sX POST -H "Authorization: token ${ACCESS_TOKEN}" https://api.github.com/repos/${ORGANIZATION}/$GH_PROJECT/actions/runners/registration-token | jq .token --raw-output)
# gcloud auth activate-service-account --key-file=${GOOGLE_APPLICATION_CREDENTIALS}
cd /home/docker/actions-runner
./config.sh --name $RUNNER_NAME --labels ${RUNNER_LABELS} --url https://github.com/${ORGANIZATION}/${GH_PROJECT} --unattended --replace --token ${REG_TOKEN}
cleanup() {
echo "Removing runner..."
./config.sh remove --unattended --token ${REG_TOKEN}
}
trap 'cleanup; exit 130' INT
trap 'cleanup; exit 143' TERM
./run.sh & wait $!
The pod is restarting whenever the load is coming into it:
runner-automation-dev-docker-595f48c7dc-k2wbz 1/2 CrashLoopBackOff 7 (67s ago) 18m
I am not sure what exactly is causing this issue.

New Relic PHP Agent Kubernetes (GKE)

Could you please advise on setting up permissions with the docker file for the www-data user to start PHP agent within the docker container running on GKE. Please advise.
FROM php:7.4-fpm as test
RUN \
curl -L https://download.newrelic.com/php_agent/release/newrelic-php5-10.1.0.313-linux.tar.gz | tar -C /tmp -zx && \
export NR_INSTALL_USE_CP_NOT_LN=1 && \
export NR_INSTALL_SILENT=1 && \
/tmp/newrelic-php5-*/newrelic-install install && \
rm -rf /tmp/newrelic-php5-* /tmp/nrinstall* && \
sed -i \
-e 's/"REPLACE_WITH_REAL_KEY"/"My-Key"/' \
-e 's/newrelic.appname = "PHP Application"/newrelic.appname = "test"/' \
-e 's/;newrelic.daemon.app_connect_timeout =.*/newrelic.daemon.app_connect_timeout=15s/' \
-e 's/;newrelic.daemon.start_timeout =.*/newrelic.daemon.start_timeout=5s/' \
/usr/local/etc/php/conf.d/newrelic.ini
USER www
php app related build. etc....
Thank you very much.
In your docker file you are changing the user to USER www due to that it's not running.
As suggested in error it is expected to run by the root user so you can remove the USER www line from docker and try building a new docker image with --no-cache and it will start working with root.
Official ref : https://docs.newrelic.com/docs/apm/agents/php-agent/advanced-installation/docker-other-container-environments-install-php-agent/

Building postgres from source throws "'utils/errcodes.h' file not found" when called from other makefile

I am currently constructing a Makefile and one of the things it will be doing is downloading and building postgres from source. Before starting to write the makefile file I always used the following set of commands to do this:
curl -LJ https://github.com/postgres/postgres/archive/refs/tags/REL_13_3.zip -o postgres.zip
unzip postgres.zip
rm postgres.zip
cd postgres-REL_13_3
./configure --prefix "`pwd`" --without-readline --without-zlib
make
make install
Executing the above listed commands in the terminal results in the successful installation of postgres. Then I translated these into a makefile which looks as follows:
build:
curl -LJ https://github.com/postgres/postgres/archive/refs/tags/REL_13_3.zip -o postgres.zip
unzip postgres.zip
rm postgres.zip
cd postgres-REL_13_3 \
&& ./configure --prefix "`pwd`" --without-readline --without-zlib \
&& $(MAKE) \
&& $(MAKE) install
Running this Makefile results in the error:
../../src/include/utils/elog.h:71:10: fatal error: 'utils/errcodes.h' file not found
It seems that something about calling the make from another Makefile causes a referencing issue with the files during the build process, but I just can figure out for the life of me what I have to change to fix this.
It appears to be the influence of Makefile enviromental variables. I didn't discover the exact mechanism, but unsetting them helps.
build:
curl -LJ https://github.com/postgres/postgres/archive/refs/tags/REL_13_3.zip -o postgres.zip
unzip postgres.zip
rm postgres.zip
unset MAKELEVEL && unset MAKEFLAGS && unset MFLAGS && cd postgres-REL_13_3 \
&& ./configure --prefix "`pwd`" --without-readline --without-zlib \
&& $(MAKE) \
&& $(MAKE) install

why do cp command get error without noting add -e option?

Premise · What I want to realize
I've getting an error into writing a Dockerfile, so I'm worried about a getting error.
That command is cp.
testing environment
base container image centos:7
My laptop is MacBook Pro (it may not need info, isn't it?)
What I did
the Dockerfile is here,
FROM centos:7 # Official centos image. this is a comment for asking here.
ENV JAVA_HOME=/usr/lib/jvm/java-11-openjdk \
SCALA_HOME=/usr/local/scala \
SCALA_VERSION=scala-2.13.0
WORKDIR /usr/local/lib/
RUN : " *** nginx install ***" \
&& { \
echo '[nginx-stable]'; \
echo 'name=nginx stable repo'; \
echo 'baseurl=http://nginx.org/packages/centos/$releasever/$basearch/'; \
echo 'gpgcheck=1'; \
echo 'enabled=1'; \
echo 'gpgkey=https://nginx.org/keys/nginx_signing.key'; \
echo 'gpgkey=https://nginx.org/keys/nginx_signing.key'; \
} > /etc/yum.repos.d/nginx.repo \
&& yum install -y nginx \
&& yum install -y rsyslog \
&& rsyslogd \
&& cp /usr/lib/systemd/system/nginx.service /etc/systemd/system/ \
&& nginx -version \
&& : " *** JDK install ***" \
&& JAVA_HOME=${JAVA_HOME}/bin \
&& PATH=$PATH:${JAVA_HOME}/bin \
&& java -version \
&& javac -version \
&& : "*** Scala install ***" \
&& wget http://downloads.typesafe.com/scala/2.13.0/scala-2.13.0.tgz \
&& tar zxvf scala-2.13.0.tgz \
&& ln -s ${SCALA_VERSION} scala \
&& mkdir ${SCALA_HOME} \
&& mv ${SCALA_VERSION} SCALA_HOME \
&& SCALA_HOME=${SCALA_HOME}/bin >> /etc/profile.d/scala.sh \
&& PATH=$PATH:${SCALA_HOME}/bin >> /etc/profile.d/scala.sh \
&& source /etc/profile.d/scala.sh \
&& cd \
&& scala -version \
&& : "*** sbt install ***" \
&& curl https://bintray.com/sbt/rpm/rpm | tee /etc/yum.repos.d/bintray-sbt-rpm.repo \
&& yum install -y sbt \
&& sbt -version \
&& ln -sf /usr/share/zoneinfo/Asia/Tokyo /etc/localtime \
&& yum clean all \
&& mkdir -p /usr/share/app
WORKDIR /usr/share/app
EXPOSE 80
Problems occurring · Error messages
the getting error is,
cp: invalid option -- 'e'
Try 'cp --help' for more information.
umm.., I've completely understand the error.
I haven't cp alias too.
Could you please help me?
If you need more information, feel free asking me.
Regards,
K
i had the same error as you because i wrote the command as below
sudo find /home/usersdata -type f -user mark -exec cp -p -parents {} /official ;
the parameters parents should --parents instead of -parents
For some reasons the error does not provide explicitly what is wrong.
Hope it helps you

How to mount devcontainer to $HOME

I'm trying to setup a devcontainer which mounts the workspace in VSCode to /home/node and puts everything in the project into $HOME in docker.
the reason why I want to do this is so when the project mounts into docker I have everything contained in the individual projects I'm working on.
I get an error, when running with this configuration.
anyone have suggestions on how to achieve this?
this is my devcontainer.json
// For format details, see https://aka.ms/vscode-remote/devcontainer.json or the definition README at
// https://github.com/microsoft/vscode-dev-containers/tree/master/containers/typescript-node-lts
{
"name": "Node.js (latest LTS) & TypeScript",
"dockerFile": "Dockerfile",
// Use 'settings' to set *default* container specific settings.json values on container create.
// You can edit these settings after create using File > Preferences > Settings > Remote.
"settings": {
"terminal.integrated.shell.linux": "/bin/bash"
},
// Uncomment the next line if you want to publish any ports.
"appPort": [3000, 8000, 6006],
// Uncomment the next line to run commands after the container is created.
"postCreateCommand": "mkdir ~/.ssh && cp .ssh/* ~/.ssh && chmod 600 ~/.ssh/id_rsa",
// Uncomment the next line to use a non-root user. On Linux, this will prevent
// new files getting created as root, but you may need to update the USER_UID
// and USER_GID in .devcontainer/Dockerfile to match your user if not 1000.
"runArgs": ["-u", "node"],
// Add the IDs of extensions you want installed when the container is created in the array below.
"extensions": [
"dbaeumer.vscode-eslint",
"esbenp.prettier-vscode",
"Orta.vscode-jest"
],
"workspaceMount": "src=/home/node,dst=/home/node,type=volume,volume-driver=local",
"workspaceFolder": "/home/node"
}
and my Dockerfile
#-------------------------------------------------------------------------------------------------------------
# Copyright (c) Microsoft Corporation. All rights reserved.
# Licensed under the MIT License. See https://go.microsoft.com/fwlink/?linkid=2090316 for license information.
#-------------------------------------------------------------------------------------------------------------
FROM node:lts
# Avoid warnings by switching to noninteractive
ENV DEBIAN_FRONTEND=noninteractive
# The node image comes with a base non-root 'node' user which this Dockerfile
# gives sudo access. However, for Linux, this user's GID/UID must match your local
# user UID/GID to avoid permission issues with bind mounts. Update USER_UID / USER_GID
# if yours is not 1000. See https://aka.ms/vscode-remote/containers/non-root-user.
ARG USER_UID=1000
ARG USER_GID=$USER_UID
# Configure apt and install packages
RUN apt-get update \
&& apt-get -y install --no-install-recommends apt-utils dialog 2>&1 \
#
# Verify git and needed tools are installed
&& apt-get -y install git iproute2 procps \
#
# Remove outdated yarn from /opt and install via package
# so it can be easily updated via apt-get upgrade yarn
&& rm -rf /opt/yarn-* \
&& rm -f /usr/local/bin/yarn \
&& rm -f /usr/local/bin/yarnpkg \
&& apt-get install -y curl apt-transport-https lsb-release \
&& curl -sS https://dl.yarnpkg.com/$(lsb_release -is | tr '[:upper:]' '[:lower:]')/pubkey.gpg | apt-key add - 2>/dev/null \
&& echo "deb https://dl.yarnpkg.com/$(lsb_release -is | tr '[:upper:]' '[:lower:]')/ stable main" | tee /etc/apt/sources.list.d/yarn.list \
&& apt-get update \
&& apt-get -y install --no-install-recommends yarn \
#
# Install tools globally
&& npm install -g jest prettier eslint typescript localtunnel \
#
# [Optional] Update a non-root user to match UID/GID - see https://aka.ms/vscode-remote/containers/non-root-user.
&& if [ "$USER_GID" != "1000" ]; then groupmod node --gid $USER_GID; fi \
&& if [ "$USER_UID" != "1000" ]; then usermod --uid $USER_UID node; fi \
# [Optional] Add add sudo support for non-root user
&& apt-get install -y sudo \
&& echo node ALL=\(root\) NOPASSWD:ALL > /etc/sudoers.d/node \
&& chmod 0440 /etc/sudoers.d/node \
#
# Clean up
&& apt-get autoremove -y \
&& apt-get clean -y \
&& rm -rf /var/lib/apt/lists/*
# Switch back to dialog for any ad-hoc use of apt-get
ENV DEBIAN_FRONTEND=
not 100% sure, as you did not share the error you encoutered, but
it might be because you use:
"workspaceMount": "src=/home/node,dst=/home/node,type=volume,volume-driver=local",
when it's not a 'volume' in your case. The type should be 'bind' here.
see here for more information.