Run tests using docker compose in scala - scala

Trying to execute integration tests using docker.
Below is the Dockerfile:
FROM openjdk:8u232
ARG SBT_VERSION=1.2.8
# Install sbt
RUN \
curl -L -o sbt-$SBT_VERSION.deb https://dl.bintray.com/sbt/debian/sbt-$SBT_VERSION.deb && \
dpkg -i sbt-$SBT_VERSION.deb && \
rm sbt-$SBT_VERSION.deb && \
apt-get update && \
apt-get install sbt && \
sbt sbtVersion
ENV WORK_DIR="/test"
USER root
COPY . ${WORK_DIR}/
RUN cd ${WORK_DIR}
RUN sbt clean it:test
Below is the docker-compose.yml:
version: '3.7'
services:
test:
build:
context: ../..
dockerfile: Dockerfile.test
restart: always
when I run docker-compose -f test.yaml up :
it gives me error:
Step 10/10 : RUN sbt clean it:test
---> Running in d9513c01439b
[warn] No sbt.version set in project/build.properties, base directory: /
[info] Set current project to root (in build file:/)
[success] Total time: 0 s, completed Jun 6, 2020 7:20:37 AM
[error] Expected ';'
[error] No such setting/task
[error] it:test
to check it i removed this line from docker file and built the image and exec into container and my tests ran fine.
The structure of my project is:
src
main
it
test
I tried executing the unit tests as well
but it gave me:
Step 10/10 : RUN sbt clean test
---> Running in c300fcf2bf53
[warn] No sbt.version set in project/build.properties, base directory: /
[info] Set current project to root (in build file:/)
[success] Total time: 0 s, completed Jun 6, 2020 7:19:13 AM
[success] Total time: 10 s, completed Jun 6, 2020 7:19:24 AM
Removing intermediate container c300fcf2bf53
---> c0215c62663d
Successfully built c0215c62663d
Successfully tagged it_it:latest
Creating it_it_1 ... done
Attaching to it_it_1
when i changed it to:
Step 10/10 : CMD sbt clean it:test
---> Running in 719b239fb7e4
Removing intermediate container 719b239fb7e4
---> f7adb19f2cb6
Successfully built f7adb19f2cb6
Successfully tagged it_it:latest
Creating it_it_1 ... done
Attaching to it_it_1
it_1 | [warn] No sbt.version set in project/build.properties, base directory: /
it_1 | [info] Set current project to root (in build file:/)
[success] Total time: 0 s, completed Jun 6, 2020 7:32:24 AM
[error] Expected ';'
it_1 | [error] No such setting/task
it_1 | [error] it:test
it_1 | [error]
How can i execute sbt it:test from dockerfile using compose?

If it was working inside the container then this must work:
ENTRYPOINT cd ${WORK_DIR} && \
sbt clean it:test

Change to next:
RUN cd ${WORK_DIR} && sbt clean it:test
Different RUN in dockerfile won't affect eachother.

Related

MariaDB fails to start in Github workflow

I'm building an app that needs MariaDB 10.3 for testing in a CI pipeline. I'm trying to set this up with Github actions but I'm having trouble getting MariaDB to start. The ubuntu runner includes MySQL 8.0 by default so my workflow removes that first and then installs MariaDB, but MariaDB fails to start. At first I saw this error:
2022-09-17T01:24:30.8867144Z + sudo cat /var/log/mysql/error.log
2022-09-17T01:24:30.8926847Z 2022-09-17 1:24:20 0 [ERROR] InnoDB: Invalid flags 0x4800 in ./ibdata1
2022-09-17T01:24:30.8927599Z 2022-09-17 1:24:20 0 [ERROR] InnoDB: Plugin initialization aborted with error Data structure corruption
2022-09-17T01:24:30.8928456Z 2022-09-17 1:24:21 0 [ERROR] Plugin 'InnoDB' init function returned error.
2022-09-17T01:24:30.8929215Z 2022-09-17 1:24:21 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2022-09-17T01:24:30.8930300Z 2022-09-17 1:24:21 0 [ERROR] Unknown/unsupported storage engine: InnoDB
2022-09-17T01:24:30.8930738Z 2022-09-17 1:24:21 0 [ERROR] Aborting
I think this is because MySQL 8.0 leaves behind some old files, so I added a step to remove /var/lib/mysql but now the action stalls after the MariaDB installation.
I made a copy in a new public repo to show the issue here: https://github.com/llamafilm/mariadb_test/actions/runs/3071684353/jobs/4962586417
The workflow is like this:
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v3
- name: Install dependencies
run: |
set -x
sudo apt update
sudo apt autoremove mysql*
- name: Look around
run: |
set -x
sudo ls -l /var/lib/mysql
sudo rm -rf /var/lib/mysql
- name: Install MariaDB
run: |
sudo apt install -y mariadb-server-10.3
- name: Validate DB
run: |
set -x
sudo cat /var/log/mysql/error.log
sudo ls -l /var/lib/mysql
mysql -e "SHOW STATUS"

Elastic Beanstalk failed to unzip source file with UTF-8 file name

I was trying to deploy a NextJS application to Elastic Beanstalk via eb deploy. But the source bundle failed to unzip during deployment as the source bundle contained some pre-built .next page which the file name is in UTF-8 encoding. The error is stated as below.
2022/xx/xx xx:xx:xx.xxxxxx [INFO] Executing instruction: StageApplication
2022/xx/xx xx:xx:xx.xxxxxx [INFO] extracting /opt/elasticbeanstalk/deployment/app_source_bundle to /var/app/staging/
2022/01/31 04:56:44.300483 [INFO] Running command /bin/sh -c /usr/bin/unzip -q -o /opt/elasticbeanstalk/deployment/app_source_bundle -d /var/app/staging/
2022/01/31 04:56:45.932820 [ERROR] An error occurred during execution of command [app-deploy] - [StageApplication]. Stop running the command. Error: Command /bin/sh -c /usr/bin/unzip -q -o /opt/elasticbeanstalk/deployment/app_source_bundle -d /var/app/staging/ failed with error exit status 50. Stderr:error: cannot create /var/app/staging/.next/server/pages/\u6e2c\u8a66/\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66.html
File name too long
error: cannot create /var/app/staging/.next/server/pages/\u6e2c\u8a66/\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66\u6e2c\u8a66.json
File name too long
I was able to unzip the file with option -O UTF-8, is there any way I could add this flag to the eb deploy unzip process?
edit 1. I am working with the Platform 64bit Amazon Linux 2/5.4.9
Not sure if it is good practice, but I eventually added an ebextensions to overcome the original unzipping flow.
commands:
command backup original zip:
command: |
logger "backup zip" && cp /opt/elasticbeanstalk/deployment/app_source_bundle /tmp/app_source_bundle_bak &&
logger "rm existing zip .next folder" && zip -Ad /opt/elasticbeanstalk/deployment/app_source_bundle ".next/*"
cwd: /home/ec2-user
ignoreErrors: false
container_commands:
replace the original zip to staging:
command: |
logger "custom unzip" &&
unzip -O UTF-8 -q -o /tmp/app_source_bundle_bak -d /var/app/staging/
cwd: /home/ec2-user
ignoreErrors: false

Docker compose missing yarn dependencies on build

Can't get node_modules folder when running yarn install in the Dockerfile
test-sof
├── docker-compose.yml
├── Dockerfile
├── package.json
└── yarn.lock
docker-compose.yml
version: '3'
services:
web:
build: .
volumes:
- .:/myapp
package.json
{
"name": "site",
"private": true,
"dependencies": {
"#rails/webpacker": "^3.2.1",
"babel-preset-react": "^6.24.1",
"prop-types": "^15.6.0",
"react": "^16.2.0",
"react-dom": "^16.2.0",
"reactjs": "^1.0.0",
"underscore": "^1.8.3"
},
"devDependencies": {
"webpack-dev-server": "^2.11.1"
}
}
Dockferfile
FROM ruby:2.5
RUN curl -sL https://deb.nodesource.com/setup_8.x | bash - && \
curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - && \
echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list && \
apt-get update && \
apt-get install -qq -y build-essential libpq-dev nodejs yarn
RUN mkdir /myapp
WORKDIR /myapp
ADD ./package.json /myapp/
RUN yarn install
output of the step RUN yarn install when docker-compose build:
Step 6/6 : RUN yarn install
---> Running in 3a0e7095ec81
yarn install v1.3.2
info No lockfile found.
[1/4] Resolving packages...
[2/4] Fetching packages...
info fsevents#1.1.3: The platform "linux" is incompatible with this module.
info "fsevents#1.1.3" is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] Linking dependencies...
warning "#rails/webpacker > postcss-cssnext#3.1.0" has unmet peer dependency "caniuse-lite#^1.0.30000697".
warning " > webpack-dev-server#2.11.1" has unmet peer dependency "webpack#^2.2.0 || ^3.0.0".
warning "webpack-dev-server > webpack-dev-middleware#1.12.2" has unmet peer dependency "webpack#^1.0.0 || ^2.0.0 || ^3.0.0".
[4/4] Building fresh packages...
success Saved lockfile.
Done in 21.11s.
Removing intermediate container 3a0e7095ec81
---> 5720579a0f2a
Successfully built 5720579a0f2a
Successfully tagged testsof_web:latest
Running command: docker-compose run web bash to get in the container
root#11af1818e494:/myapp# ls
Dockerfile docker-compose.yml package.json
no node_modules folder present, but later when running inside the container: yarn install output:
root#11af1818e494:/myapp# yarn install
yarn install v1.3.2
info No lockfile found.
[1/4] Resolving packages...
[2/4] Fetching packages...
info fsevents#1.1.3: The platform "linux" is incompatible with this module.
info "fsevents#1.1.3" is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] Linking dependencies...
warning "#rails/webpacker > postcss-cssnext#3.1.0" has unmet peer dependency "caniuse-lite#^1.0.30000697".
warning " > webpack-dev-server#2.11.1" has unmet peer dependency "webpack#^2.2.0 || ^3.0.0".
warning "webpack-dev-server > webpack-dev-middleware#1.12.2" has unmet peer dependency "webpack#^1.0.0 || ^2.0.0 || ^3.0.0".
[4/4] Building fresh packages...
success Saved lockfile.
Done in 13.03s.
then when listing:
root#11af1818e494:/myapp# ls
Dockerfile docker-compose.yml node_modules package.json yarn.lock
folder node_modules IT IS present. Why?
This part of Dockerfile installs yarn packages:
RUN mkdir /myapp
WORKDIR /myapp
ADD ./package.json /myapp/
RUN yarn install
Folder /myapp is created, package.json is copied to it and yarn packages are installed. Build is successful and, of course, node_modules folder is inside built image.
But after that you start built image with:
volumes:
- .:/myapp
which means that content of folder where docker-compose.yaml is is mounted to /myapp folder inside container, so it covers content of container's /myapp folder.
You don't need to mount current folder to container's folder to achieve what you want. Just delete it from your docker-compose.yaml:
version: '3'
services:
web:
build: .
Now you can:
$ docker-compose build
$ docker-compose run web bash
root#558d5b0c2ccb:/myapp# ls -la
total 268
drwxr-xr-x 3 root root 4096 Feb 23 22:25 .
drwxr-xr-x 65 root root 4096 Feb 23 22:36 ..
drwxr-xr-x 818 root root 36864 Feb 23 22:25 node_modules
-rw-rw-r-- 1 root root 333 Feb 23 22:07 package.json
-rw-r--r-- 1 root root 219075 Feb 23 22:25 yarn.lock
EDIT:
But what I want is when building the image, get these dependencies not
when spinning up the containers. Otherwise I have another container
which mounts de source code and needs this node_modules folder when
running the "docker-compose up" and I'd like to avoid some kind of
ugly sleep until the node_modules is finished. So I need present this
folder on my root host before up the containers somehow
If you want to achieve the above goal, you can use the following workaround:
1. You modify Dockerfile a little:
FROM ruby:2.5
RUN curl -sL https://deb.nodesource.com/setup_8.x | bash - && \
curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - && \
echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list && \
apt-get update && \
apt-get install -qq -y build-essential libpq-dev nodejs yarn
RUN mkdir /build && mkdir /myapp
WORKDIR /build
ADD ./package.json /build/
RUN yarn install
WORKDIR /myapp
CMD cp -a /build/node_modules/ /myapp/
That's means that yarn packages will be built in /build folder inside image and copied to /myapp folder once container is started.
2. You use the original docker-compose.yaml file:
version: '3'
services:
web:
build: .
volumes:
- .:/myapp
when you start web container:
docker-compose up web
folder node_modules is copied to mounted folder i.e. to . folder on your host machine.
3. Now you can start any container and it will be contain node_modules folder inside /myapp:
docker-compose run web bash
So, you will be able to achieve your goal the following way:
$ docker-compose build && docker-compose up web
$ docker-compose run web bash
root#4b38e60adfa3:/myapp# ls -la
total 64
drwxrwxr-x 3 1000 1000 4096 Feb 24 10:59 .
drwxr-xr-x 66 root root 4096 Feb 24 11:13 ..
-rw-rw-r-- 1 1000 1000 497 Feb 24 10:55 Dockerfile
-rw-rw-r-- 1 1000 1000 73 Feb 24 09:02 docker-compose.yaml
drwxr-xr-x 818 root root 40960 Feb 24 10:57 node_modules
-rw-rw-r-- 1 root root 333 Feb 23 22:07 package.json

java.lang.RuntimeException: No main class detected

I am trying to run Scala application using docker. I've created a sample project with the following structure.
build.sbt
name := "test"
version := "1.0"
scalaVersion := "2.12.2"
project/build.properties
sbt.version = 0.13.15
src/main/scala/HelloWorld.scala
object HelloWorld {
def main(args: Array[String]): Unit = {
println("Hello, world!")
}
}
Now if I do sbt run, everything works fine and I have the lovely hello-world greeting.
$ sbt run
...
[info] Set current project to test (in build file:/Users/yuchen/Documents/test/)
[info] Running HelloWorld
Hello, world!
[success] Total time: 1 s, completed 22-May-2017 4:30:28 PM
I aded a Dockerfile:
FROM openjdk:8
ENV SCALA_VERSION 2.12.2
ENV SBT_VERSION 0.13.15
RUN touch /usr/lib/jvm/java-8-openjdk-amd64/release
RUN \
curl -fsL http://downloads.typesafe.com/scala/$SCALA_VERSION/scala-$SCALA_VERSION.tgz | tar xfz - -C /root/ && \
echo >> /root/.bashrc && \
echo 'export PATH=~/scala-$SCALA_VERSION/bin:$PATH' >> /root/.bashrc
RUN \
curl -L -o sbt-$SBT_VERSION.deb http://dl.bintray.com/sbt/debian/sbt-$SBT_VERSION.deb && \
dpkg -i sbt-$SBT_VERSION.deb && \
rm sbt-$SBT_VERSION.deb && \
apt-get update && \
apt-get install sbt && \
sbt sbtVersion
WORKDIR /root
CMD sbt run
They are all copied from https://hub.docker.com/r/hseeberger/scala-sbt/~/dockerfile/ and with the additional last line CMD sbt run.
And I tried running docker commands:
docker build -t test .
docker tag test test/test:1.0
docker run test/test:1.0
However, I keep getting this error:
[info] Set current project to root (in build file:/root/)
[info] Updating {file:/root/}root...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
java.lang.RuntimeException: No main class detected.
at scala.sys.package$.error(package.scala:27)
[trace] Stack trace suppressed: run last compile:run for the full output.
[error] (compile:run) No main class detected.
[error] Total time: 0 s, completed May 23, 2017 4:00:19 AM
What does it mean and how do I resolve this?
FYI: I am aware of http://www.scala-sbt.org/sbt-native-packager/formats/docker.html, but just want to try writing Dockerfile as I am learning this.
I don't really know sbt. But this appears that you are not copying the project into the image (there is no COPY or ADD in the Dockerfile).
Try doing something like:
FROM openjdk:8
ENV SCALA_VERSION 2.12.2
ENV SBT_VERSION 0.13.15
RUN touch /usr/lib/jvm/java-8-openjdk-amd64/release
RUN \
curl -fsL http://downloads.typesafe.com/scala/$SCALA_VERSION/scala-$SCALA_VERSION.tgz | tar xfz - -C /root/ && \
echo >> /root/.bashrc && \
echo 'export PATH=~/scala-$SCALA_VERSION/bin:$PATH' >> /root/.bashrc
RUN \
curl -L -o sbt-$SBT_VERSION.deb http://dl.bintray.com/sbt/debian/sbt-$SBT_VERSION.deb && \
dpkg -i sbt-$SBT_VERSION.deb && \
rm sbt-$SBT_VERSION.deb && \
apt-get update && \
apt-get install sbt && \
sbt sbtVersion
WORKDIR /myapp
CMD sbt run
COPY . /myapp
The addition here is just a COPY to copy all the code to /myapp and to changing the working directory to this folder as well. This would copy the build context to /myapp and then the WORKDIR should run sbt run from the same folder.

Export usage error when doing example from sbt docs

The sbt tutorial shows this example.
$ mkdir hello
$ cd hello
$ echo 'object Hi { def main(args: Array[String]) = println("Hi!") }' > hw.scala
$ sbt
...
> run
...
Hi!
However, when I try it, I get this:
$ mkdir hello
$ cd hello
$ echo 'object Hi { def main(args: Array[String]) = println("Hi!") }' > hw.scala
$ sbt
[info] Set current project to hello (in build file:/Users/coryklein/playground/hello/)
[error] export usage:
[error] export <tasks>+ Executes tasks and displays the equivalent command lines.
[error]
[error] export
[error]
My sbt version
$ sbt --version
sbt launcher version 0.13.6
Am I doing something wrong here?
Here is the output of sh -x sbt
+ '[' -f /Users/coryklein/.sbtconfig ']'
+ exec /usr/local/Cellar/sbt/0.13.6/libexec/sbt
[info] Set current project to hello (in build file:/Users/coryklein/playground/hello/)
[error] export usage:
[error] export <tasks>+ Executes tasks and displays the equivalent command lines.
[error]
[error] export
[error]
I installed sbt via brew install sbt.
$ ll `which sbt`
lrwxr-xr-x 1 coryklein admin 28B Oct 14 16:03 /usr/local/bin/sbt -> ../Cellar/sbt/0.13.6/bin/sbt
I'm trying to look through the source code to figure out what the problem is, but it's difficult when a working sbt is required to build sbt from source. :(
I was unable to figure out exactly why it failed to work, but I resolved it by doing the manual install like so:
brew remove sbt
wget https://dl.bintray.com/sbt/native-packages/sbt/0.13.6/sbt-0.13.6.tgz
extract sbt-0.13.6.tgz
mv sbt/bin/sbt-launch.jar ~/bin
echo 'SBT_OPTS="-Xms512M -Xmx1536M -Xss1M -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256M"\n
java $SBT_OPTS -jar `dirname $0`/sbt-launch.jar "$#"' > ~/bin/sbt
chmod u+x ~/bin/sbt