How to pass credentials to sbt build cli? - scala

I'm trying to Dockerize an old sbt build that needs access to a private nexus repo. I've previously been using a local credentials file referenced from the build.sbt, but that doesn't really fit my use currently since I want to bootstrap everything from a Dockerfile build. I rather not have to output stuff to a file and then copy it into my docker build container but rather just pass it as docker ARG's.
Under
https://www.scala-sbt.org/1.0/docs/Publishing.html
I read I can pass it like so:
credentials += Credentials("Some Nexus Repository Manager",
"my.artifact.repo.net", "admin", "admin123")
So therefore I figured I could do something like:
ARG REPO_USER
ARG REPO_PWD
RUN sbt ";credentials += Credentials(\"Some Nexus Repository Manager\", \"repo.host.com\", ${REPO_USER}, ${REPO_PWD}) ;package"
and then run
docker build . --build-arg REPO_USER=foobar --build-arg REPO_PWD=*****
in my Dockerfile but that didn't work. I still get:
Unable to find credentials for [Sonatype Nexus Repository Manager # repo.host.com]
Is there any nice way to pass repo credentials to sbt from cli?
Update:
I tried a a file approach but that didn't solve the problem so I guess I might be on the wrong track on what's actually wrong here.
RUN echo "realm=Sonatype Nexus Repository Manager" >> .credentials && \
echo "host=repo.host.se" >> .credentials && \
echo "user=$REPO_USER" >> .credentials && \
echo "password=$REPO_PWD" >> .credentials && \
export SBT_CREDENTIALS=.credentials && \
sbt package
Update 2
I think this is no longer a Docker question at all since I've debugged it in the docker container sbt simply wouldn't pick my creds up any way I passed it according to the sbt docs.
I'll answer my own question.

You can use environment variables. You can set them in dockerfile either straight or from args. Something like this:
ARG REPO_USR
ARG REPO_PWD
ENV REPO_USR = ${REPO_USR}
ENV REPO_PWD = ${REPO_PWD}
Then you can use the environment variables in sbt:
val repoUser = sys.env.get("REPO_USR").getOrElse("")
val repoPass = sys.env.get("REPO_PWS").getOrElse("")
credentials += Credentials("Repo Realm", "repo.url.com", repoUser, repoPass)
Then you can basically pass args to docker build and they will be passed on to sbt.

The sbt docs are misleading at best or simply just wrong. After debugging this to bits in the docker container I found that there was no way I could pass the creds cli so they got picked up. The SBT_CREDENTIALS variable just doesn't work either.
This comment finally saved me: SBT is unable to find credentials when attempting to download from an Artifactory virtual repo
The least intrusive way I've got working is to add an sbt config file in the docker image's home dir:
RUN mkdir .sbt/0.13/plugins && \
echo "credentials += Credentials(\"Sonatype Nexus Repository Manager\", \"repo.host.se\", \"$REPO_USER\", \"$REPO_PWD\")" >> .sbt/0.13/plugins/creds.sbt
RUN sbt package

Related

How to decide Quarkus application arguments in Kubernetes at run-time?

I've built a Quarkus 2.7.1 console application using picocli that includes several subcommands. I'd like to be able to run this application within a Kubernetes cluster and decide its arguments at run-time. This is so that I can use the same container image to run the application in different modes within the cluster.
To get things started I added the JIB extension and tried setting the arguments using a configuration value quarkus.jib.jvm-arguments. Unfortunately it seems like this configuration value is locked at build-time so I'm unable to update this at run-time.
Next I tried setting quarkus.args while using default settings for JIB. The configuration value documentation makes it sound general enough for the job but it doesn't seem to have an affect when the application is run in the container. Since most references to this configuration value in documentation are in the context of Dev Mode I'm wondering if this may be disabled outside of that.
How can I get this application running in a container image with its arguments decided at run-time?
You can set quarkus.jib.jvm-entrypoint to any container entrypoint command you want, including scripts. An example in the doc is quarkus.jib.jvm-entrypoint=/deployments/run-java.sh. You could make use of $CLI_ARGUMENTS in such a script. Even something like quarkus.jib.jvm-entrypoint=/bin/sh,-c,'/deployments/run-java.sh $CLI_ARGUMENTS' should work too, as long as you place the script run-java.sh at /deployments in the image. The possibility is limitless.
Also see this SO answer if there's an issue. (The OP in the link put a customer script at src/main/jib/docker/run-java.sh (src/main/jib is Jib's default "extra files directory") so that Jib places the script in the image at /docker/run-java.sh.
I was able to find a solution to the problem with a bit of experimenting this morning.
With the quarkus-container-image-docker extension (instead of quarkus.jib.jvm-arguments) I was able to take the template Dockerfile.jvm and extend it to pass through arguments to the CLI. The only line that needed changing was the ENTRYPOINT (details included in the snippet below). I changed the ENTRYPOINT form (from exec to shell) and added an environment variable as an argument to pass-through program arguments.
FROM registry.access.redhat.com/ubi8/ubi-minimal:8.3
ARG JAVA_PACKAGE=java-11-openjdk-headless
ARG RUN_JAVA_VERSION=1.3.8
ENV LANG='en_US.UTF-8' LANGUAGE='en_US:en'
# Install java and the run-java script
# Also set up permissions for user `1001`
RUN microdnf install curl ca-certificates ${JAVA_PACKAGE} \
&& microdnf update \
&& microdnf clean all \
&& mkdir /deployments \
&& chown 1001 /deployments \
&& chmod "g+rwX" /deployments \
&& chown 1001:root /deployments \
&& curl https://repo1.maven.org/maven2/io/fabric8/run-java-sh/${RUN_JAVA_VERSION}/run-java-sh-${RUN_JAVA_VERSION}-sh.sh -o /deployments/run-java.sh \
&& chown 1001 /deployments/run-java.sh \
&& chmod 540 /deployments/run-java.sh \
&& echo "securerandom.source=file:/dev/urandom" >> /etc/alternatives/jre/lib/security/java.security
# Configure the JAVA_OPTIONS, you can add -XshowSettings:vm to also display the heap size.
ENV JAVA_OPTIONS="-Dquarkus.http.host=0.0.0.0 -Djava.util.logging.manager=org.jboss.logmanager.LogManager"
# We make four distinct layers so if there are application changes the library layers can be re-used
COPY --chown=1001 target/quarkus-app/lib/ /deployments/lib/
COPY --chown=1001 target/quarkus-app/*.jar /deployments/
COPY --chown=1001 target/quarkus-app/app/ /deployments/app/
COPY --chown=1001 target/quarkus-app/quarkus/ /deployments/quarkus/
EXPOSE 8080
USER 1001
# [== BEFORE ==]
# ENTRYPOINT [ "/deployments/run-java.sh" ]
# [== AFTER ==]
ENTRYPOINT "/deployments/run-java.sh" $CLI_ARGUMENTS
I have tried the above approaches but they didn't work with the default quarkus JIB's ubi8/openjdk-17-runtime image. This is because this base image doesn't use /work as the WORKIR, but instead the /home/jboss.
Therefore, I created a custom start-up script and referenced it on the properties file as following. This approach works better if there's a need to set application params using environment variables:
File: application.properties
quarkus.jib.jvm-entrypoint=/bin/sh,run-java.sh
File: src/main/jib/home/jboss/run-java.sh
java \
-Djavax.net.ssl.trustStore=/deployments/truststore \
-Djavax.net.ssl.trustStorePassword="$TRUST_STORE_PASSWORD" \
-jar quarkus-run.jar

Is there a way to automatically create a container when starting Azurite?

For test purposes I create and run an Azurite docker image, in a test pipeline.
I would like to have the blob container automatically created though after Azurite is started, as it would simplify things.
Is there any good way to achieve this?
For the Postgres image we use, we can specify an init.sql which is run on startup. If something similar is available for Azurite, that would be awesome.
You can use the following Dockerfile to install the azure-storage-blob Python package on the Alpine based azurite image. The resulting image size is ~400MB compared to the ~1.2GB azure-cli image.
ARG AZURITE_VERSION="3.17.0"
FROM mcr.microsoft.com/azure-storage/azurite:${AZURITE_VERSION}
# Install azure-storage-blob python package
RUN apk update && \
apk --no-cache add py3-pip && \
apk add --virtual=build gcc libffi-dev musl-dev python3-dev && \
pip3 install --upgrade pip && \
pip3 install azure-storage-blob==12.12.0
# Copy init_azurite.py script
COPY ./init_azurite.py init_azurite.py
# Copy local blobs to azurite
COPY ./init_containers init_containers
# Run the blob emulator and initialize the blob containers
CMD python3 init_azurite.py --directory=init_containers & \
azurite-blob --blobHost 0.0.0.0 --blobPort 10000
The init_azurite.py script is a local Python script that uses the azure-storage-blob package to batch upload files and directories to the azurite blob storage emulator.
import argparse
import os
from time import sleep
from azure.core.exceptions import ResourceExistsError
from azure.storage.blob import BlobServiceClient, ContainerClient
def upload_file(container_client: ContainerClient, source: str, dest: str) -> None:
"""
Upload a single file to a path inside the container.
"""
print(f"Uploading {source} to {dest}")
with open(source, "rb") as data:
try:
container_client.upload_blob(name=dest, data=data)
except ResourceExistsError:
pass
def upload_dir(container_client: ContainerClient, source: str, dest: str) -> None:
"""
Upload a directory to a path inside the container.
"""
prefix = "" if dest == "" else dest + "/"
prefix += os.path.basename(source) + "/"
for root, dirs, files in os.walk(source):
for name in files:
dir_part = os.path.relpath(root, source)
dir_part = "" if dir_part == "." else dir_part + "/"
file_path = os.path.join(root, name)
blob_path = prefix + dir_part + name
upload_file(container_client, file_path, blob_path)
def init_containers(
service_client: BlobServiceClient, containers_directory: str
) -> None:
"""
Iterate on the containers directory and do the following:
1- create the container.
2- upload all folders and files to the container.
"""
for container_name in os.listdir(containers_directory):
container_path = os.path.join(containers_directory, container_name)
if os.path.isdir(container_path):
container_client = service_client.get_container_client(container_name)
try:
container_client.create_container()
except ResourceExistsError:
pass
for blob in os.listdir(container_path):
blob_path = os.path.join(container_path, blob)
if os.path.isdir(blob_path):
upload_dir(container_client, blob_path, "")
else:
upload_file(container_client, blob_path, blob)
if __name__ == "__main__":
parser = argparse.ArgumentParser(
description="Initialize azurite emulator containers."
)
parser.add_argument(
"--directory",
required=True,
help="""
Directory that contains subdirectories named after the
containers that we should create. Each subdirectory will contain the files
and directories of its container.
"""
)
args = parser.parse_args()
# Connect to the localhost emulator (after 5 secs to make sure it's up).
sleep(5)
blob_service_client = BlobServiceClient(
account_url="http://localhost:10000/devstoreaccount1",
credential={
"account_name": "devstoreaccount1",
"account_key": (
"Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq"
"/K1SZFPTOtr/KBHBeksoGMGw=="
)
}
)
# Only initialize if not already initialized.
if next(blob_service_client.list_containers(), None):
print("Emulator already has containers, will skip initialization.")
else:
init_containers(blob_service_client, args.directory)
This script will be copied to the azurite container and will populate the initial blob containers every time the azurite container is started unless some containers were already persisted using docker volumes. In that case, nothing will happen.
Following is an example docker-compose.yml file:
azurite:
build:
context: ./
dockerfile: Dockerfile
args:
AZURITE_VERSION: 3.17.0
restart: on-failure
ports:
- 10000:10000
volumes:
- azurite-data:/opt/azurite
volumes:
azurite-data:
Using such volumes will persist the emulator data until you destroy them (e.g. by using docker-compose down -v).
Finally, init_containers is a local directory that contains the containers and their folders/files. It will be copied to the azurite container when the image is built.
For example:
init_containers:
container-name-1:
dir-1:
file.txt
img.png
dir-2:
file.txt
container-name-2:
dir-1:
file.txt
img.png
I've solved the issue by creating a custom docker image and executing azure-cli tools from a health check. There could certainly be better solutions, and I will update the accepted answer if someone posts a better solution.
In more details
A solution to create the required data on startup is to run my own script. I chose to trigger the script from a health check I defined in docker-compose. What it does is use azure cli tools to create a container and then verify that it exists.
The script:
AZURE_STORAGE_CONNECTION_STRING="UseDevelopmentStorage=true"
export AZURE_STORAGE_CONNECTION_STRING
az storage container create -n images
az storage container show -n images
exit $?
However, the azurite image is based on alpine, which doesn't have apt, so installing azure cli was a bit tricky. So I did it the other way around, and based my image on mcr.microsoft.com/azure-cli:latest. With that done I installed Azurite like this:
RUN apk add npm
RUN npm install -g azurite --silent
All that's left is to actually run azurite, see the official azurite dockerfile for details.
It is possible to do this without azure-cli and use curl instead (and with that, not having to use the azure-cli docker image). However this was a bit complicated to get the authentication header working properly, so using azure-cli was easier.

Unable to set HomePath And config in grafana

I am new to grafana and I am getting this error while executing the grafana-server.exe
Grafana-server Init Failed: Could not find config defaults, make sure homepath command line parameter is set or working directory is homepath
Firstly, I am not clear about which path to specify as homepath and which to specify as config path.
Secondly, I have tried to set the homepath using this command:
grafana-cli admin reset-admin-password --homepath "c:\" mynewpassword
But getting this error :
"Incorrect Usage: flag provided but not defined: -homepath"
in grafana version 7.3.5 this is the help message.
NAME:
Grafana CLI - A new cli application
USAGE:
grafana-cli [global options] command [command options] [arguments...]
VERSION:
7.3.5
AUTHOR:
Grafana Project <hello#grafana.com>
COMMANDS:
plugins Manage plugins for grafana
admin Grafana admin commands
help, h Shows a list of commands or help for one command
GLOBAL OPTIONS:
--pluginsDir value Path to the Grafana plugin directory (default: "/var/lib/grafana/plugins") [$GF_PLUGIN_DIR]
--repo value URL to the plugin repository (default: "https://grafana.com/api/plugins") [$GF_PLUGIN_REPO]
--pluginUrl value Full url to the plugin zip file instead of downloading the plugin from grafana.com/api [$GF_PLUGIN_URL]
--insecure Skip TLS verification (insecure) (default: false)
--debug Enable debug logging (default: false)
--configOverrides value Configuration options to override defaults as a string. e.g. cfg:default.paths.log=/dev/null
--homepath value Path to Grafana install/home path, defaults to working directory
--config value Path to config file
--help, -h show help (default: false)
--version, -v print the version (default: false)
so you can set it by passing --homepath .
be careful, it seems its diffrent with grafana-server parametes. in that case you must set flag with only 1 hyphen (-homepath)
but come back to your problem there is 2 things to say.
first is to order your command correctly. i mean something like this
grafana-cli --homepath path ...
because homepath flag is for grafana-cli so it must come right after that or there will not be any guarantee of "what you want to do is what you write".
second is the homepath. consider this tree directory
.
|_LICENSE
|_NOTICE.md
|_README.md
|_VERSION
|_bin
|_conf
|_data
|_plugin-bundled
|_public
|_scripts
here is installation directory or homepath which you must set. more specifically exacly around bin(or conf or data or ...) directory.

Yocto: Nothing provides python-re-native

I'm running into an issue including python pyparted as a native dependency in one of my image creation bbclasses.
There is a python scrip that runs to create a partitioned image file, normally I run sudo apt install python-pyparted to have pyparted in the environment in ubuntu. But I'm not sure what I did (update??), the ubuntu environment is completely ignored now. I tried figuring out how to make sure the dependencies are correct in my sdimage bbclass.
do_image_sdimage[depends] = "parted-native:do_populate_sysroot \
dosfstools-native:do_populate_sysroot \
mtools-native:do_populate_sysroot \
virtual/kernel:do_deploy \
splash-images:do_deploy \
python3-native:do_populate_sysroot \
python3-pyparted-native:do_populate_sysroot \
${#d.getVar('IMAGE_BOOTLOADER', True) and d.getVar('IMAGE_BOOTLOADER', True) + ':do_deploy' or ''}"
I get an error showing
ERROR: Nothing PROVIDES 'python3-re-native' (but virtual:native:/home/dev/app/OS/sources/meta-openembedded/meta-python/recipes-extended/python-pyparted/python3-pyparted_3.10.7.bb DEPENDS on or otherwise requires it). Close matches:
python3-rpm-native
python3-native
python3-nose-native
python3-native RPROVIDES python3-re-native
ERROR: Required build target 'my-image-default' has no buildable providers.
Missing or unbuildable dependency chain was: ['my-image-default', 'python3-pyparted-native', 'python3-re-native']
based on this it looks like I should be able to do this, but the depency chain ignores python3-native's RPROVIDES?

How to configure a Jenkins plugin from a Dockerfile

I have a user that just has access to pull from github. In my Dockerfile I have added the plugins for Jenkins, such as github:1.22.4, but I want to configure the plugins as some of the people that will build the image won't know how to do the configuration, and don't care to learn.
So, I have some plugins for Jenkins and I want to be able to configure them using the Dockerfile. How can I do that?
My Dockerfile is pretty basic right now:
FROM jenkins
COPY plugins.txt /plugins.txt
RUN /usr/local/bin/plugins.sh /plugins.txt
and I have several plugins in plugins.txt, but the one I want to configure is to pull the code from github.
Did you check this git repository?
lets say you have plugins.txt like:
github:1.22.4
maven-plugin:2.7.1
ant:1.3
and Dockerfile like in your question.
You can take a look into example of plugins.sh and here is part for installing plugins. since you want do configure some plugins you can add configuration when you are installing plugin:
if ! grep -q "${plugin[0]}:${plugin[1]}" "$TEMP_ALREADY_INSTALLED"
then
echo "Downloading ${plugin[0]}:${plugin[1]}"
curl --retry 3 --retry-delay 5 -sSL -f "${JENKINS_UC_DOWNLOAD}/plugins/${plugin[0]}/${plugin[1]}/${plugin[0]}.hpi" -o "$REF/${plugin[0]}.jpi"
unzip -qqt "$REF/${plugin[0]}.jpi"
# if [ some plugin ] then
# here your configuration
# fi
(( COUNT_PLUGINS_INSTALLED += 1 ))
else
echo " ... skipping already installed: ${plugin[0]}:${plugin[1]}"
fi