stuck at command "sbt compile" in docker ubuntu - scala

I try to include sbt into docker images. However, it never works and always stuck at Getting org.scala-sbt sbt 0.13.7 ... Also, it is also not working for changing the sbt version.
Here is the snippet of docker file
FROM ubuntu:14.04
RUN echo "deb https://dl.bintray.com/sbt/debian /" | sudo tee -a /etc/apt/sources.list.d/sbt.list
RUN sudo apt-get update
RUN sudo apt-get install sbt //also I used force--yes
also, I try to install it on the container manually by using
wget http://dl.bintray.com/sbt/debian/sbt-0.13.5.deb
sudo apt-get update
sudo dpkg -i sbt-0.13.5.deb
When I run sbt compile, it also stuck trying to get Getting org.scala-sbt ...
but it is working for sbt --version
Basically, I dont know why sbt stucks trying to get Getting org.scala-sbt ...

You will need a Java Virtual Machine for SBT and so I think it's a good think you start from the java official docker image. Here is a basic docker file that uses the official ubuntu installation method:
FROM java
RUN echo "deb http://dl.bintray.com/sbt/debian /" | tee -a /etc/apt/sources.list.d/sbt.list
RUN apt-key update
RUN apt-get update
RUN apt-get -y --force-yes install sbt
NOTE: --force-yes has to be there because it is not an authenticated package. I tried adding RUN apt-key update, but that didn't make a difference so you can omit this line.
Then, build a test image: docker build -t test/sbt ., create an interactive container docker run -i -t test/sbt sbt and play with it.
This works for me, but I noticed the download times were slow for launching SBT, so be patient at this step.
This is because the SBT executable in itself is really light and will fetch a bunch of libraries on the first run to accomplish its task. It's also a way for SBT to support multiple projects using multiple SBT versions. If you are stuck at libraries resolution, check your networking configuration. For SBT errors, they are mostly printed on the command line, but you can configure logging if you want.
What's left for you to figure out is to add your project files and issue a compile command to test it.

sbt will try to download a higher version of itself if the project require a higher version than the currently installed sbt version. Usually there is a {projectFolder}/project/build.properties which specifies the desired sbt version for a sbt project. for example: sbt.version=0.13.7 requires version 0.13.7
You seem to get stuck at Getting org.scala-sbt sbt 0.13.7 .... But I believe sbt is actually trying to download sbt 0.13.7 to your local. As the package is not small, depending on your network speed, it may take a while.
It is also likely that there is a network connectivity issue that prevents sbt from downloading its package. So you can try to verify first your network connectivity is not a problem.
If your network is fine, another approach you can try is to go to sbt site to download 0.13.7 package manually to your docker and install it there by following instructions you can find from sbt site.
Hope this helps.

Sometimes sbt stuck when downloading files. You can periodically check size ~/.ivy2 folder and if size isn't grow kick sbt process and rerun sbt.
For me only after 5 kicks sbt download all files!!!

Related

Running sbt in a Docker Container

I am trying to use Github actions for my scala project and created a Docker workflow for it. Basically, I am trying to install sbt into my container and run the project.
Dockerfile looks like this:
FROM centos:centos8
ENV SCALA_VERSION 2.13.1
ENV SBT_VERSION 1.5.2
RUN yum install -y epel-release
RUN yum update -y && yum install -y wget
# INSTALL JAVA
RUN yum install -y java-11-openjdk
# INSTALL SBT
RUN wget http://dl.bintray.com/sbt/rpm/sbt-${SBT_VERSION}.rpm
RUN yum install -y sbt-${SBT_VERSION}.rpm
RUN wget -O /usr/local/bin/sbt-launch.jar http://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/$SBT_VERSION/sbt-launch.jar
WORKDIR /root
EXPOSE 8080
RUN sbt compile
CMD sbt run
But when I push anything, I get the following error:
The command '/bin/sh -c wget http://dl.bintray.com/sbt/rpm/sbt-${SBT_VERSION}.rpm' returned a non-zero code: 8
When I check the link manually (by setting the sbt version), I see indeed bintray responds with 403 forbidden error but status.bintray.com tells all systems are operational.
Am I doing something wrong or is something wrong with bintray?
Forbidden doesnt mean non operational.
I think that url is incorrect as its not hosted on bintray rather jfrog, please see section on Centos which states
remove old Bintray repo file
https://www.scala-sbt.org/1.x/docs/Installing-sbt-on-Linux.html

sbt is installed but not found

Fedora 25. sbt is installed and I have been using it, probably last time was a week ago. But today it is not found.
$ sbt
bash: sbt: command not found...
Install package 'sbt' to provide command 'sbt'? [N/y]
Tried a reinstall but dnf knows what I do
$ sudo dnf install sbt
[sudo] password for xxx:
Last metadata expiration check: 0:31:12 ago on Thu Apr 27 19:39:34 2017.
Package sbt-0.13.15.2-2.noarch is already installed, skipping.
Dependencies resolved.
Nothing to do.
Complete!
Checked for install location with which but no luck
$ which sbt
/usr/bin/which: no sbt in (/usr/lib64/qt-3.3/bin:/usr/lib64/ccache:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/home/xxx/.local/bin:/home/xxx/bin)
I would guess the problem is in my $PATH however I have not altered it, although I have installed some packages
$ echo $PATH
/usr/lib64/qt-3.3/bin:/usr/lib64/ccache:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/home/xxx/.local/bin:/home/xxx/bin
Finally i did find a jar named sbt-launch.jar in /usr/share/sbt/bin/ but I don't think /usr/share has ever been part of my $PATH.
Recommendations? thx
fixed it by running
$dnf reinstall sbt
Out of curiosity I checked $PATH again afterwards and it was the same. So it was something else evidently.
I have resolved it reinstalling:
First, uninstall sbt:
yum remove sbt
Then download the version you want from https://dl.bintray.com/sbt/debian/
wget http://dl.bintray.com/sbt/rpm/sbt-0.13.16.rpm
And install by yum
sudo yum install sbt-0.13.16.rpm
Check it with sbt about
sbt about
It must show you similar to
sbt about
[info] This is sbt 0.13.16
[info] sbt, sbt plugins, and build definitions are using Scala 2.10.6

How to build Spark from the sources from the Download Spark page?

I tried to install and build Spark 2.0.0 on Ubuntu VM with Ubuntu 16.04 as follows:
Install Java
sudo apt-add-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java8-installer
Install Scala
Go to their Downloads tab on their site: scala-lang.org/download/all.html
I used Scala 2.11.8.
sudo mkdir /usr/local/src/scala
sudo tar -xvf scala-2.11.8.tgz -C /usr/local/src/scala/
Modify the .bashrc file and include the path for scala:
export SCALA_HOME=/usr/local/src/scala/scala-2.11.8
export PATH=$SCALA_HOME/bin:$PATH
then type:
. .bashrc
Install git
sudo apt-get install git
Download and build spark
Go to: http://spark.apache.org/downloads.html
Download Spark 2.0.0 (Build from Source - for standalone mode).
tar -xvf spark-2.0.0.tgz
cd into the Spark folder (that has been extracted).
now type:
./build/sbt assembly
After its done Installing, I get the message:
[success] Total time: 1940 s, completed...
followed by date and time...
Run Spark shell
bin/spark-shell
That's when all hell breaks loose and I start getting the error. I go into the assembly folder to look for a folder called target. But there's no such folder there. The only things visible in assembly are: pom.xml, README, and src.
I looked it up online for quite a while and I haven't been able to find a single concrete solution that would help solve the error. Can someone please provide explicit step-by-step instructions as to how to go about solving this ?!? It's driving me nuts now... (T.T)
Screenshot of the error:
For some reason, Scala 2.11.8 is not working well while building but if I switch over to Scala 2.10.6 then it builds properly. I guess the reason I would need Scala in the first place is to get access to sbt to be able to build spark. Once its built, I need to direct myself to the spark folder and type:
build/sbt package
This will build the missing JAR files for me using Scala 2.11... kinda weird but that's how its working (I am assuming by looking at the logs).
Once spark builds again, type: bin/spark-shell (while being in the spark folder) and you'll have access to the spark shell.
type sbt package in spark directory not in build directory.
If your goal is really to build your custom Spark package from the sources you've downloaded from http://spark.apache.org/downloads.html, you should do the following instead:
./build/mvn -Phadoop-2.7,yarn,mesos,hive,hive-thriftserver -DskipTests clean install
You may want to read the official document Building Spark.
NB You don't have to install Scala and git packages to build Spark so you could have skipped "2. Install Scala" and "3. Install git" steps.

Cannot build sbt project due to launcher version

I have a SBT projec on Ubuntu 14.04. When I run sbt I get the following message:
Detected sbt version sbt.version = 0.13.8
Starting sbt: invoke with -help for other options
Cannot find sbt launcher sbt.version = 0.13.8
Please download:
From http://typesafe.artifactoryonline.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/sbt.version = 0.13.8/sbt-launch.jar
To /home/vagrant/.sbt/.lib/sbt.version/sbt-launch.jar
When I try and download from the above url I am met with a 404 response. In fact even the url http://typesafe.artifactoryonline.com is giving 404.
Any ideas how I can get the correct sbt version on my system?
One solution is not to use the package manager at all. Uninstall sbt, and simply use the sbt shell script by Paul Phillips: https://github.com/paulp/sbt-extras :
curl -s https://raw.githubusercontent.com/paulp/sbt-extras/master/sbt > ~/bin/sbt \
&& chmod 0755 ~/bin/sbt
This is quite up-to-date and should have the correct repositories for the launchers.
How did you install sbt?
If you're using sbt-extras you might be hitting #107. Try upgrading it to a newer version.
An alternative is to switch to the official version of sbt. See Installing sbt on Linux:
echo "deb https://dl.bintray.com/sbt/debian /" | sudo tee -a /etc/apt/sources.list.d/sbt.list
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 642AC823
sudo apt-get update
sudo apt-get install sbt
Your installed sbt version is probably a bit older and still refers to the old repository.
Typesafe moved to a bintray repository so the new url would be:
https://dl.bintray.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/0.13.8/sbt-launch.jar
or with alias
https://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/0.13.8/sbt-launch.jar
repo.typesafe.com may be replaced by repo.lightbend.com in the future ;)
Reinstalling the sbt run script would probably help, too.

Building spark-jobserver Using SBT and Scala

Can anyone suggest me a better documentation about spark-jobserver. I have gone through the url spark-jobserver but unable to follow the same. It will be great if some one explain step by step instruction on how to use spark-jobserver.
Tools used in building the project.
sbt launcher version 0.13.5
Scala code runner version 2.11.6
With the above mentioned tools I am getting errors while building the spark-jobserver.
The documentation provided in the jobserver repo is indeed confusing.
Here's the steps I followed to manually build and run Spark Job Server on a local machine.
1. git clone https://github.com/spark-jobserver/spark-jobserver
2. sudo mkdir -p /var/log/job-server
3. sudo chown user1:user1 /var/log/job-server
4. cd spark-jobserver
5. sbt job-server/assembly
6. cd config
7. cp local.sh.template abc.sh # Note that the same name 'abc' is used in steps 8 and 10 as well
8. cp ec2.conf.template abc.conf
9. cd .. # The jobserver root dir
10. ./bin/server_package.sh abc # This script copies the files and packages necessary to run job server into a single dir [ default - /tmp/job-server]
11. cd /tmp/job-server [This is where the files and packages necessary to run job server are published by default]
12. ./server_start.sh
13. Run ./server_stop.sh to stop the server
Hope this helps
Here are the steps that I used to install:
Clone the jobserver repo.
Get sbt using wget https://dl.bintray.com/sbt/native-packages/sbt/0.13.8/sbt-0.13.8.tgz
Move "sbt-launch.jar" in sbt/bin to /bin
Create a script /bin/sbt, contents found here, making sure to change the pointer to java if necessary
Make the above script executable
Now cd into the spark jobserver directory, and run sbt publish-local
Assuming the above was successful, run sbt in the same directory
Finally, use the command re-start, and if it succeeds the server is now running!