Importing tensorflow into Scala - scala

Question

I have installed Tensorflow for Scala on my OSX, and although everything seemed fine, I have a NoClassDefFoundError when trying to run simple example like that –
import org.platanios.tensorflow.api.Tensor
val tensor = Tensor( 1.2, 4.5)
which gives:
java.lang.NoClassDefFoundError: Could not initialize class org.platanios.tensorflow.api.package$
at #worksheet#.tensor$lzycompute(testone.sc:3)
at #worksheet#.tensor(testone.sc:3)
at #worksheet#.get$$instance$$tensor(testone.sc:3)
at A$A16$.main(testone.sc:17)
at A$A16.main(testone.sc)
at #worksheet#.#worksheet#(testone.sc)
Similar error both on a Jupyter Notebook and an IntelliJ worksheet. My build.sbt:
scalaVersion := "2.12.4"
resolvers += Resolver.sonatypeRepo("snapshots")
libraryDependencies += "org.platanios" %% "tensorflow" % "0.1.2-SNAPSHOT"
libraryDependencies += "org.platanios" %% "tensorflow" % "0.1.2-SNAPSHOT" classifier "darwin-cpu-x86_64"

Answer 1

The problem could be due to a missing dependency library of libtensorflow_jni.so contained by the scala_tensorflow jar
To find the missing library:
sbt console
then from the scala shell import the tensorflow api:
scala> import org.platanios.tensorflow.api._
scala> val tensor = Tensor.zeros(INT32, Shape(2, 5))
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/angelo/.ivy2/cache/org.slf4j/slf4j-log4j12/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/angelo/.ivy2/cache/ch.qos.logback/logback-classic/jars/logback-classic-1.2.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
log4j:WARN No appenders could be found for logger (TensorFlow Native).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
java.lang.UnsatisfiedLinkError: /tmp/tensorflow_scala_native_libraries3327494822622243889/libtensorflow_jni.so: libcusolver.so.9.0: cannot open shared object file: No such file or directory
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
at java.lang.Runtime.load0(Runtime.java:809)
at java.lang.System.load(System.java:1086)
at org.platanios.tensorflow.jni.TensorFlow$$anonfun$load$3.apply(TensorFlow.scala:95)
at org.platanios.tensorflow.jni.TensorFlow$$anonfun$load$3.apply(TensorFlow.scala:93)
at scala.Option.foreach(Option.scala:257)
at org.platanios.tensorflow.jni.TensorFlow$.load(TensorFlow.scala:93)
at org.platanios.tensorflow.jni.TensorFlow$.<init>(TensorFlow.scala:155)
at org.platanios.tensorflow.jni.TensorFlow$.<clinit>(TensorFlow.scala)
at org.platanios.tensorflow.jni.Tensor$.<init>(Tensor.scala:24)
at org.platanios.tensorflow.jni.Tensor$.<clinit>(Tensor.scala)
at org.platanios.tensorflow.api.tensors.Context$.apply(Context.scala:50)
at org.platanios.tensorflow.api.package$.<init>(package.scala:89)
at org.platanios.tensorflow.api.package$.<clinit>(package.scala)
... 40 elided
You can check all the missing libraries using ldd on another terminal (ldd in linux or otool -L on macosx):
ldd /tmp/tensorflow_scala_native_libraries3327494822622243889/libtensorflow_jni.so
/tmp/tensorflow_scala_native_libraries3327494822622243889/libtensorflow_jni.so: /usr/lib/libcublas.so.9.0: version `libcublas.so.9.0' not found (required by /tmp/tensorflow_scala_native_libraries3327494822622243889/libtensorflow.so)
/tmp/tensorflow_scala_native_libraries3327494822622243889/libtensorflow_jni.so: /usr/lib/libcublas.so.9.0: version `libcublas.so.9.0' not found (required by /tmp/tensorflow_scala_native_libraries3327494822622243889/libtensorflow_framework.so)
linux-vdso.so.1 => (0x00007ffc00d85000)
libdlfaker.so => /usr/lib/x86_64-linux-gnu/libdlfaker.so (0x00007f0cd05d8000)
librrfaker.so => /usr/lib/x86_64-linux-gnu/librrfaker.so (0x00007f0cd033d000)
libtensorflow.so => /tmp/tensorflow_scala_native_libraries3327494822622243889/libtensorflow.so (0x00007f0cc8023000)
libtensorflow_framework.so => /tmp/tensorflow_scala_native_libraries3327494822622243889/libtensorflow_framework.so (0x00007f0cc714b000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f0cc6dc9000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f0cc6bb3000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f0cc67e9000)
libGL.so.1 => /usr/lib/nvidia-390/libGL.so.1 (0x00007f0cc64ac000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f0cc62a8000)
libturbojpeg.so.0 => /usr/lib/x86_64-linux-gnu/libturbojpeg.so.0 (0x00007f0cc6047000)
libXv.so.1 => /usr/lib/x86_64-linux-gnu/libXv.so.1 (0x00007f0cc5e42000)
libX11.so.6 => /usr/lib/x86_64-linux-gnu/libX11.so.6 (0x00007f0cc5b08000)
libXext.so.6 => /usr/lib/x86_64-linux-gnu/libXext.so.6 (0x00007f0cc58f6000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f0cc56d9000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f0cc53d0000)
/lib64/ld-linux-x86-64.so.2 (0x00007f0cd0a66000)
libcublas.so.9.0 => /usr/lib/libcublas.so.9.0 (0x00007f0cc1caf000)
libcusolver.so.9.0 => not found
libcudart.so.9.0 => not found
libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f0cc1a8d000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f0cc1885000)
libcuda.so.1 => /usr/lib/x86_64-linux-gnu/libcuda.so.1 (0x00007f0cc0ce5000)
libcudnn.so.7 => /usr/local/cuda-9.1/targets/x86_64-linux/lib/libcudnn.so.7 (0x00007f0cafd54000)
libcufft.so.9.0 => not found
libcurand.so.9.0 => not found
libcudart.so.9.0 => not found
libnvidia-tls.so.390.30 => /usr/lib/nvidia-390/tls/libnvidia-tls.so.390.30 (0x00007f0cafb50000)
libnvidia-glcore.so.390.30 => /usr/lib/nvidia-390/libnvidia-glcore.so.390.30 (0x00007f0cadd50000)
libxcb.so.1 => /usr/lib/x86_64-linux-gnu/libxcb.so.1 (0x00007f0cadb2e000)
libnvidia-fatbinaryloader.so.390.30 => /usr/lib/nvidia-390/libnvidia-fatbinaryloader.so.390.30 (0x00007f0cad8e2000)
libXau.so.6 => /usr/lib/x86_64-linux-gnu/libXau.so.6 (0x00007f0cad6de000)
libXdmcp.so.6 => /usr/lib/x86_64-linux-gnu/libXdmcp.so.6 (0x00007f0cad4d8000)
On my computer the rutime linking process cannot resolve libcusolver.so.9.0 since I have installed cuda-9.1
To make it working I had to compile tensorflow as follows:
git clone https://github.com/tensorflow/tensorflow.git
cd tensorflow
./configure
bazel build --cxxopt="-D_GLIBCXX_USE_CXX11_ABI=0" --config=opt //tensorflow:libtensorflow.so
Copy the libraries into a path which is in LD_LIBRARY_PATH:
sudo cp bazel-bin/tensorflow/libtensorflow.so /usr/local/lib
sudo cp bazel-bin/tensorflow/libtensorflow_framework.so /usr/local/lib
Finally I was able to compile the tensorflow_scala project:
sbt compile
Now from the tensorflow_scala project I can run sbt console and it works:
scala> import org.platanios.tensorflow.api._
import org.platanios.tensorflow.api._
scala> val tensor = Tensor.zeros(INT32, Shape(2, 5))
2018-02-16 17:08:26.184 [run-main-0] INFO TensorFlow Native - Extracting the 'tensorflow_jni' native library to /tmp/tensorflow_scala_native_libraries8283851378265055495/libtensorflow_jni.so.
2018-02-16 17:08:26.188 [run-main-0] INFO TensorFlow Native - Copied 645872 bytes to /tmp/tensorflow_scala_native_libraries8283851378265055495/libtensorflow_jni.so.
2018-02-16 17:08:26.254 [run-main-0] INFO TensorFlow Native - Extracting the 'tensorflow_ops' native library to /tmp/tensorflow_scala_native_libraries8283851378265055495/libtensorflow_ops.so.
2018-02-16 17:08:26.254 [run-main-0] INFO TensorFlow Native - Copied 78232 bytes to /tmp/tensorflow_scala_native_libraries8283851378265055495/libtensorflow_ops.so.
2018-02-16 17:08:26.449239: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:898] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-02-16 17:08:26.449483: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1331] Found device 0 with properties:
name: GeForce GTX 1050 major: 6 minor: 1 memoryClockRate(GHz): 1.493
pciBusID: 0000:01:00.0
totalMemory: 3,95GiB freeMemory: 1,53GiB
2018-02-16 17:08:26.449498: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1410] Adding visible gpu devices: 0
2018-02-16 17:08:26.670568: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-02-16 17:08:26.670601: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-02-16 17:08:26.670610: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-02-16 17:08:26.670698: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1021] Creating TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1287 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1050, pci bus id: 0000:01:00.0, compute capability: 6.1)
2018-02-16 17:08:26.698355: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1410] Adding visible gpu devices: 0
2018-02-16 17:08:26.698399: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-02-16 17:08:26.698408: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-02-16 17:08:26.698414: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-02-16 17:08:26.698518: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1021] Creating TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 224 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1050, pci bus id: 0000:01:00.0, compute capability: 6.1)
tensor: org.platanios.tensorflow.api.tensors.Tensor = INT32[2, 5]
To use the locally built tensorflow_scala:
sbt
sbt:TensorFlow for Scala> + publishLocal
The jar will be placed into ~/.ivy2/local/org.platanios ; then you can add the jar to your sbt project e.g:
libraryDependencies ++= { Seq("org.platanios" %% "tensorflow" % "0.1.2-SNAPSHOT") }