I am trying to run the scala test from the command line without SBT and I am failing. I followed the documentation line-by-line.
import collection.mutable.Stack
import org.scalatest._
import flatspec._
import matchers._
class FirstSpec extends AnyFlatSpec with should.Matchers {
"A Stack" should "pop values in last-in-first-out order" in {
val stack = new Stack[Int]
stack.push(1)
stack.push(2)
stack.pop() should be (2)
stack.pop() should be (1)
}
}
Error message:
> scala -cp scalatest_2.13-3.2.5.jar org.scalatest.tools.Runner -R . -o -s FirstSpec.scala
No such file or class on classpath: org.scalatest.tools.Runner
Repository
ScalaTest has been modularised since 2.3.0 so just scalatest.jar artifact is not sufficient from raw shell. Build tools such as sbt would usually resolve all the transitive dependencies automatically, however if you are not using a build tool, then it is necessary to do that manually.
So download all the transitive dependencies and run something like
scala -cp scalatest_2.13-3.2.4.jar:scalatest-compatible-3.2.4.jar:scalatest-core_2.13-3.2.4.jar:scalactic_2.13-3.2.4.jar:scalatest-diagrams_2.13-3.2.4.jar:scalatest-matchers-core_2.13-3.2.4.jar:scalatest-shouldmatchers_2.13-3.2.4.jar:scalatest-flatspec_2.13-3.2.4.jar:scala-xml_2.13-1.3.0.jar org.scalatest.run ExampleSpec
or given all the transitive jars are in the same directory
scala -cp '*' org.scalatest.run ExampleSpec
or coursier can help you fetch and build the correct classpath
scala -cp "$(cs fetch --classpath org.scalatest:scalatest_2.13:3.2.4)" org.scalatest.run ExampleSpec
or use coursier to launch the main class from directory containing compiled tests
cs launch org.scalatest:scalatest_2.13:3.2.4 -M org.scalatest.run
or launch the default main runner which provides basic GUI by providing the run path -R
cs launch org.scalatest:scalatest_2.13:3.2.4 -- -R .
For record here are all the transitive dependencies of scalatest.jar
cs resolve org.scalatest:scalatest_2.13:3.2.5
org.scala-lang:scala-library:2.13.4:default
org.scala-lang:scala-reflect:2.13.4:default
org.scala-lang.modules:scala-xml_2.13:1.2.0:default
org.scalactic:scalactic_2.13:3.2.5:default
org.scalatest:scalatest-compatible:3.2.5:default
org.scalatest:scalatest-core_2.13:3.2.5:default
org.scalatest:scalatest-diagrams_2.13:3.2.5:default
org.scalatest:scalatest-featurespec_2.13:3.2.5:default
org.scalatest:scalatest-flatspec_2.13:3.2.5:default
org.scalatest:scalatest-freespec_2.13:3.2.5:default
org.scalatest:scalatest-funspec_2.13:3.2.5:default
org.scalatest:scalatest-funsuite_2.13:3.2.5:default
org.scalatest:scalatest-matchers-core_2.13:3.2.5:default
org.scalatest:scalatest-mustmatchers_2.13:3.2.5:default
org.scalatest:scalatest-propspec_2.13:3.2.5:default
org.scalatest:scalatest-refspec_2.13:3.2.5:default
org.scalatest:scalatest-shouldmatchers_2.13:3.2.5:default
org.scalatest:scalatest-wordspec_2.13:3.2.5:default
org.scalatest:scalatest_2.13:3.2.5:default
Related
I am trying to compile this simple code in scala to run it in spark:
import org.apache.spark.sql.SparkSession
object Main {
def main(args: Array[String]) {
if (args.length < 1) {
System.err.println("Usage: HDFStEST <file>")
System.exit(1)
}
val spark = SparkSession.builder.appName("TesteHelena").getOrCreate()
println("hellooo")
spark.stop() }
}
I don't know how to make scalac find the dependency org.apache.spark.sql.SparkSession
I try to set where are the jars files with the following command:
scalac main.scala -cp C:\Spark\spark-2.4.0-bin-hadoop2.7\jars -d main.jar
which returns me the error:
main.scala:1: error: object apache is not a member of package org
import org.apache.spark.sql.SparkSession
and if I just send every jar file with the command:
scalac main.scala -cp C:\Spark\spark-2.4.0-bin-hadoop2.7\jars\* org.apache.spark.sql.SparkSession -d main.jar
it returns me the error:
error: IO error while decoding C:\Spark\spark-2.4.0-bin-hadoop2.7\jars\aircompressor-0.10.jar with UTF-8
for every jar file.
The command:
scalac main.scala -cp org.apache.spark.sql.SparkSession -d main.jar
returns me:
main.scala:1: error: object apache is not a member of package org
import org.apache.spark.sql.SparkSession
So, is there a way to use the Spark dependency in scalac to compile a program? I cannot use any dependency builder, like sbt and gradle, because I don't have access to internet in my terminal, due to security issues of my job, and they call those dependencies in their repository.
I solved my issue with the command:
scalac -cp C:\Spark\spark-2.4.0-bin-hadoop2.7\jars\ -extdirs C:\Spark\spark-2.4.0-bin-hadoop2.7\jars\ main.scala -d main1.jar
so, I added the scalac's option "-extdirs", which overrides the location of installed extensions. And it worked!
I am trying to run scala code using java -jar <> i am getting below issue
ERROR:
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataOutputStream at com.cargill.finance.cdp.blackline.Ingest.main(Ingest.scala) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataOutputStream
The same code is running fine with spark-submit.
I am trying to write data to hdfs file.
I have imported below classes
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.FileSystem
import org.apache.hadoop.fs.Path
import org.apache.hadoop.fs.FSDataOutputStream
You need to add all dependencies (including transitive dependencies, i.e. dependencies of dependencies) to -cp argument. If you just look at direct dependencies of hadoop-core you'll see why you should never do this manually. Instead use a build system. If you followed e.g. https://spark.apache.org/docs/latest/quick-start.html it actually sets up SBT, so you can do sbt run to run the main class like java -cp <lots of libraries> -jar <jarfile> would). If you didn't, add build.sbt as described there.
I have my test compiled in a directory: samplesuite
It works when running one Suite using org.scalatest.run.
Nothing happens when I try to run the directory that contains several suites.
scala -cp "..\mytestframework\target\scala-2.12\mytestframework-assembly-0.1.jar;../../Downloads/scalactic_2.12-3.0.5.jar;..\..\Downloads\scalatest_2.12-3.0.5.jar" org.scalatest.run samplesuite.SomeSpec
But it won't run when trying:
scala -cp "..\mytestframework\target\scala-2.12\mytestframework-assembly-0.1.jar;../../Downloads/scalactic_2.12-3.0.5.jar;..\..\Downloads\scalatest_2.12-3.0.5.jar" org.scalatest.tools.Runner -R samplesuite
Assuming tests are defined in package samplesuite, for example
package samplesuite
class SomeSpec extends FlatSpec with Matchers
and current working directory . is one level above samplesuite, then discover and run all suites with
scala -cp "..\mytestframework\target\scala-2.12\mytestframework-assembly-0.1.jar;../../Downloads/scalactic_2.12-3.0.5.jar;..\..\Downloads\scalatest_2.12-3.0.5.jar" org.scalatest.tools.Runner -R .
In general, when specifying runpath from which tests will be discovered with -R point to the root directory where your packages start, or to the .jar file generated with sbt test:package:
scala -cp target/scala-2.12/classes:scalatest_2.12-3.0.5.jar:scalactic_2.12-3.0.5.jar org.scalatest.tools.Runner -R target/scala-2.12/test-classes
scala -cp target/scala-2.12/classes:scalatest_2.12-3.0.5.jar:scalactic_2.12-3.0.5.jar org.scalatest.tools.Runner -R target/scala-2.12/how-can-i-perform-an-or-with-scalatest_2.12-0.1.0-SNAPSHOT-tests.jar
I am new to java / scala and coming from python i don't know the compile phases...
I changed a scala example given with spark, but when I re-execute it, the changes are note taken into account. I guess it is because I am used not to compile my scripts :-/
What are the commands to compile a scala script ?
Namely : ~/spark/examples/src/main/scala/org/apache/spark/examples/streaming/FlumePollingEventCount.scala
scalac /Users/romain/spark/examples/src/main/scala/org/apache/spark/examples/streaming/FlumePollingEventCount2.scala
/Users/romain/spark/examples/src/main/scala/org/apache/spark/examples/streaming/FlumePollingEventCount2.scala:21:
error: object SparkConf is not a member of package org.apache.spark
import org.apache.spark.SparkConf
^
According to Running Build Targets For Individual Projects:
$ # sbt
$ build/sbt package
$ # Maven
$ build/mvn package -DskipTests -pl assembly
In your case it'd be build/sbt examples:package.
You may want to read up Building Spark to learn the build process in Spark.
I've created a simple app that generates PageView for later Spark tasks.
I've only one scala file taht use a simple MAP
When created a package with SBT I run my class with command:
java -cp .\target\scala-2.10\pageviewstream_2.10-1.0.0.jar "clickstream.PageViewGenerator"
but I receive this error:
Exception in thread "main" java.lang.NoClassDefFoundError: scala/collection/immutable/Map
What I am doing wrong?
Many thanks in advance
Roberto
To run it correctly you need to add Scala runtime library into your class path:
java -cp $SCALA_HOME/lib/scala-library.jar;.\target\scala-2.10\pageviewstream_2.10-1.0.0.jar "clickstream.PageViewGenerator"
But .. you can run your application also as:
scala -classpath .\target\scala-2.10\pageviewstream_2.10-1.0.0.jar "clickstream.PageViewGenerator"
when you have scala already in PATH
or use directly sbt as:
sbt "runMain clickstream.PageViewGenerator"
when clickstream.PageViewGenerator is you only application it is enough to run:
sbt run
or when you are in sbt interactive mode just type:
> runMain clickstream.PageViewGenerator
or when it is only application in your project it is enough to run:
> run