Mocking a partial (exception throwing) function - scala

I'm writing a unit test using Mockito and Specs 2 in Scala. I have a class Foo with a method bar which is partially defined in the domain of its input parameter x. If the result is not defined for the given input value an FooException is thrown:
public abstract class Foo {
public abstract int bar(String x) throws FooException;
}
public class FooException extends Exception {}
Now I'm trying to create a mock using Mockito. I wish to specify that bar only returns a result for x and y and throws an exception otherwise:
import org.specs2.mutable.Specification
import org.specs2.mock.Mockito
import org.mockito.Matchers._
class FooTest extends Specification with Mockito {
val foo = mock[Foo]
foo.bar("x") returns 1
foo.bar("y") returns 2
foo.bar(anyString) throws mock[FooException]
"Foo.bar()" should {
"return an integer for x" in {
foo.bar("x") must be equalTo(1)
}
"throw an exception for z" in {
foo.bar("z") must throwA[FooException]
}
}
}
Unfortunately, this test fails. How do I write my mock in such a way that my tests pass?
Update
The tests fail because an exception is thrown at runtime. The code above results in:
java.lang.NullPointerException
[error] Could not run test FooTest: java.lang.NullPointerException
If I change the order, as suggested in one of the answers, I get the following stack trace:
[error] Could not create an instance of FooTest
[error] caused by java.lang.Exception: Could not instantiate class FooTest: null
[error] org.specs2.reflect.Classes$class.tryToCreateObjectEither(Classes.scala:93)
[error] org.specs2.reflect.Classes$.tryToCreateObjectEither(Classes.scala:207)
[error] org.specs2.specification.SpecificationStructure$$anonfun$createSpecificationEither$2.apply(BaseSpecification.scala:119)
[error] org.specs2.specification.SpecificationStructure$$anonfun$createSpecificationEither$2.apply(BaseSpecification.scala:119)
[error] scala.Option.getOrElse(Option.scala:120)
[error] org.specs2.specification.SpecificationStructure$.createSpecificationEither(BaseSpecification.scala:119)
[error] org.specs2.runner.SbtRunner.org$specs2$runner$SbtRunner$$specificationRun(SbtRunner.scala:73)
[error] org.specs2.runner.SbtRunner$$anonfun$newTask$1$$anon$5.execute(SbtRunner.scala:59)
[error] sbt.TestRunner.runTest$1(TestFramework.scala:84)
[error] sbt.TestRunner.run(TestFramework.scala:94)
[error] sbt.TestFramework$$anon$2$$anonfun$$init$$1$$anonfun$apply$8.apply(TestFramework.scala:219)
[error] sbt.TestFramework$$anon$2$$anonfun$$init$$1$$anonfun$apply$8.apply(TestFramework.scala:219)
[error] sbt.TestFramework$.sbt$TestFramework$$withContextLoader(TestFramework.scala:207)
[error] sbt.TestFramework$$anon$2$$anonfun$$init$$1.apply(TestFramework.scala:219)
[error] sbt.TestFramework$$anon$2$$anonfun$$init$$1.apply(TestFramework.scala:219)
[error] sbt.TestFunction.apply(TestFramework.scala:224)
[error] sbt.Tests$$anonfun$7.apply(Tests.scala:196)
[error] sbt.Tests$$anonfun$7.apply(Tests.scala:196)
[error] sbt.std.Transform$$anon$3$$anonfun$apply$2.apply(System.scala:45)
[error] sbt.std.Transform$$anon$3$$anonfun$apply$2.apply(System.scala:45)
[error] sbt.std.Transform$$anon$4.work(System.scala:64)
[error] sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:237)
[error] sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:237)
[error] sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:18)
[error] sbt.Execute.work(Execute.scala:244)
[error] sbt.Execute$$anonfun$submit$1.apply(Execute.scala:237)
[error] sbt.Execute$$anonfun$submit$1.apply(Execute.scala:237)
[error] sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:160)
[error] sbt.CompletionService$$anon$2.call(CompletionService.scala:30)
[error] java.util.concurrent.FutureTask.run(FutureTask.java:262)
[error] java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[error] java.util.concurrent.FutureTask.run(FutureTask.java:262)
[error] java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[error] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[error] java.lang.Thread.run(Thread.java:744)
[error] caused by java.lang.NullPointerException

You need to change the order of your stubbing around. Stub the generalized behavior first (anyString) and then add more specific behaviors after that. So do this:
val foo = mock[Foo]
foo.bar(anyString) throws mock[FooException]
foo.bar("x") returns 1
foo.bar("y") returns 2

You could set up a callback on the mock:
val foo = mock[Foo]
foo.bar(anyString) answers (derp => {
derp match {
case "x" => 1
case "y" => 2
case _ => throw mock[FooException]
}
})
There are some more details in the specs2 documentation.

Related

AOP is failing while build sbt assembly

I am trying to build sbt code. I am using sbt assembly but it is breaking at Aop. Used following MergeStragegy for Aop
Following is the error log
<!DOCTYPE aspectj PUBLIC "-//AspectJ//DTD//EN" "http://www.eclipse.org/aspectj/dtd/aspectj.dtd">
/home/puneet/repo/target/streams/_global/assemblyOption/_global/streams/assembly/sbtMergeTarget-e2e021ed2f7893685f6d16c35a11a6d2dcda6205.tmp[error] org.xml.sax.SAXParseExceptionpublicId: -//AspectJ//DTD//EN; systemId: http://www.eclipse.org/aspectj/dtd/aspectj.dtd; lineNumber: 1; columnNumber: 2; The markup declarations contained or pointed to by the document type declaration must be well-formed.
[error] at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:203)
[error] at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
[error] at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:400)
[error] at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:327)
[error] at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1473)
[error] at com.sun.org.apache.xerces.internal.impl.XMLDTDScannerImpl.scanDecls(XMLDTDScannerImpl.java:2044)
[error] at com.sun.org.apache.xerces.internal.impl.XMLDTDScannerImpl.scanDTDExternalSubset(XMLDTDScannerImpl.java:307)
[error] at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$DTDDriver.dispatch(XMLDocumentScannerImpl.java:1174)
[error] at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$DTDDriver.next(XMLDocumentScannerImpl.java:1045)
[error] at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:959)
[error] at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:602)
[error] at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:505)
[error] at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:842)
[error] at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:771)
[error] at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
[error] at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
[error] at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:643)
[error] at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:327)
[error] at scala.xml.factory.XMLLoader.loadXML(XMLLoader.scala:41)
[error] at scala.xml.factory.XMLLoader.loadXML$(XMLLoader.scala:37)
[error] at scala.xml.XML$.loadXML(XML.scala:60)
[error] at scala.xml.factory.XMLLoader.loadFile(XMLLoader.scala:48)
[error] at scala.xml.factory.XMLLoader.loadFile$(XMLLoader.scala:48)
[error] at scala.xml.XML$.loadFile(XML.scala:60)
[error] at AopMerge$.$anonfun$apply$1(AopMerge.scala:17)
[error] at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
[error] at scala.collection.Iterator.foreach(Iterator.scala:941)
[error] at scala.collection.Iterator.foreach$(Iterator.scala:941)
[error] at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
[error] at scala.collection.IterableLike.foreach(IterableLike.scala:74)
[error] at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
[error] at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
[error] at scala.collection.TraversableLike.map(TraversableLike.scala:238)
[error] at scala.collection.TraversableLike.map$(TraversableLike.scala:231)
[error] at scala.collection.AbstractTraversable.map(Traversable.scala:108)
[error] at AopMerge$.apply(AopMerge.scala:17)
[error] at sbtassembly.MergeStrategy.apply(MergeStrategy.scala:20)
[error] at sbtassembly.Assembly$.applyStrategy$1(Assembly.scala:110)
[error] at sbtassembly.Assembly$.$anonfun$applyStrategies$11(Assembly.scala:135)
[error] at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
[error] at scala.collection.Iterator.foreach(Iterator.scala:941)
[error] at scala.collection.Iterator.foreach$(Iterator.scala:941)
[error] at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
[error] at scala.collection.IterableLike.foreach(IterableLike.scala:74)
[error] at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
[error] at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
[error] at scala.collection.TraversableLike.map(TraversableLike.scala:238)
[error] at scala.collection.TraversableLike.map$(TraversableLike.scala:231)
[error] at scala.collection.AbstractTraversable.map(Traversable.scala:108)
[error] at sbtassembly.Assembly$.applyStrategies(Assembly.scala:132)
[error] at sbtassembly.Assembly$.x$1$lzycompute$1(Assembly.scala:25)
[error] at sbtassembly.Assembly$.x$1$1(Assembly.scala:23)
[error] at sbtassembly.Assembly$.stratMapping$lzycompute$1(Assembly.scala:23)
[error] at sbtassembly.Assembly$.stratMapping$1(Assembly.scala:23)
[error] at sbtassembly.Assembly$.inputs$lzycompute$1(Assembly.scala:68)
[error] at sbtassembly.Assembly$.inputs$1(Assembly.scala:58)
[error] at sbtassembly.Assembly$.apply(Assembly.scala:85)
[error] at sbtassembly.Assembly$.$anonfun$assemblyTask$1(Assembly.scala:244)
[error] at scala.Function1.$anonfun$compose$1(Function1.scala:49)
[error] at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:62)
[error] at sbt.std.Transform$$anon$4.work(Transform.scala:67)
[error] at sbt.Execute.$anonfun$submit$2(Execute.scala:281)
[error] at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:19)
[error] at sbt.Execute.work(Execute.scala:290)
[error] at sbt.Execute.$anonfun$submit$1(Execute.scala:281)
[error] at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:178)
[error] at sbt.CompletionService$$anon$2.call(CompletionService.scala:37)
[error] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[error] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[error] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[error] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[error] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[error] at java.lang.Thread.run(Thread.java:748)
[error] (repo / assembly) org.xml.sax.SAXParseExceptionpublicId: -//AspectJ//DTD//EN; systemId: http://www.eclipse.org/aspectj/dtd/aspectj.dtd; lineNumber: 1; columnNumber: 2; The markup declarations contained or pointed to by the document type declaration must be well-formed.
Last week the same changes where working. But somehow it is throwing this error. What could be the reason?
sbt version -> 1.3.10
I have created a new MergeStrategy for aop.xml files (part of the Kamon dependencies).
If replacing http with https does not work, it might be worth trying disabling DTD validation as follows:
import java.io.FileInputStream
import org.xml.sax.InputSource
val parser = {
val factory = javax.xml.parsers.SAXParserFactory.newInstance()
factory.setFeature("http://apache.org/xml/features/nonvalidating/load-external-dtd", false)
factory.newSAXParser()
}
val xmls: Seq[Elem] = files.map(f => XML.loadXML(new InputSource(new FileInputStream(f)), parser))

scala spark rdd error : java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda

I am a beginner to Scala and Spark.
scala version : 2.12.10
spark version : 3.0.1
I'm trying a very simple spark rdd function in scala.
But I get an error.
(1) build.sbt
scalaVersion := "2.12.10"
name := "hello-world"
organization := "ch.epfl.scala"
version := "1.0"
libraryDependencies += "org.scala-lang.modules" %% "scala-parser-combinators" % "1.1.2"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "3.0.1"
libraryDependencies += "org.apache.spark" %% "spark-core" % "3.0.1"
(2) Main.scala
import org.apache.spark.sql.SparkSession
object Main extends App {
println("Hello, World!")
implicit val spark = SparkSession.builder()
.master("spark://centos-master:7077")
// .master("local[*]")
.appName("spark-api")
.getOrCreate()
val inputrdd = spark.sparkContext.parallelize(Seq(("arth",10), ("arth", 20), ("samuel", 60), ("jack", 65)))
println("inputrdd : ", inputrdd)
val mapped = inputrdd.mapValues(x => (x, 1))
println("mapped : ", mapped)
mapped.collect.foreach(println)
}
(3) When the error occurred
It seems that an error occurs in the mapped.collect.foreach(println) part.
(4) Error content
21/04/17 20:54:19 INFO DAGScheduler: Job 0 failed: collect at Main.scala:16, took 6.083947 s
[error] (run-main-0) org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 7, 192.168.0.220, executor 0):
java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda
to field org.apache.spark.rdd.MapPartitionsRDD.f of type scala.Function3 in
instance of org.apache.spark.rdd.MapPartitionsRDD
[error] at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2301)
[error] at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1431)
[error] at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2410)
[error] at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2328)
[error] at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2186)
[error] at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1666)
[error] at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2404)
[error] at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2328)
[error] at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2186)
[error] at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1666)
[error] at java.io.ObjectInputStream.readObject(ObjectInputStream.java:502)
[error] at java.io.ObjectInputStream.readObject(ObjectInputStream.java:460)
[error] at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
[error] at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
[error] at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:83)
[error] at org.apache.spark.scheduler.Task.run(Task.scala:127)
[error] at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
[error] at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
[error] at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
[error] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[error] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[error] at java.lang.Thread.run(Thread.java:748)
[error]
[error] Driver stacktrace:
21/04/17 20:54:19 INFO TaskSetManager: Lost task 1.3 in stage 0.0 (TID 6) on 192.168.0.220, executor 0: java.lang.ClassCastException (cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f of type scala.Function3 in instance of org.apache.spark.rdd.MapPartitionsRDD) [duplicate 7]
21/04/17 20:54:19 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
[error] org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 7, 192.168.0.220, executor 0): java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f of type scala.Function3 in instance of org.apache.spark.rdd.MapPartitionsRDD
[error] at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2301)
[error] at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1431)
[error] at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2410)
[error] at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2328)
[error] at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2186)
[error] at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1666)
[error] at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2404)
[error] at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2328)
[error] at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2186)
[error] at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1666)
[error] at java.io.ObjectInputStream.readObject(ObjectInputStream.java:502)
[error] at java.io.ObjectInputStream.readObject(ObjectInputStream.java:460)
[error] at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
[error] at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
[error] at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:83)
[error] at org.apache.spark.scheduler.Task.run(Task.scala:127)
[error] at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
[error] at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
[error] at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
[error] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[error] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[error] at java.lang.Thread.run(Thread.java:748)
[error]
[error] Driver stacktrace:
[error] at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2059)
[error] at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2008)
[error] at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2007)
[error] at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
[error] at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
[error] at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
[error] at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2007)
[error] at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:973)
[error] at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:973)
[error] at scala.Option.foreach(Option.scala:407)
[error] at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:973)
[error] at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2239)
[error] at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2188)
[error] at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2177)
[error] at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
[error] at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:775)
[error] at org.apache.spark.SparkContext.runJob(SparkContext.scala:2099)
[error] at org.apache.spark.SparkContext.runJob(SparkContext.scala:2120)
[error] at org.apache.spark.SparkContext.runJob(SparkContext.scala:2139)
[error] at org.apache.spark.SparkContext.runJob(SparkContext.scala:2164)
[error] at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1004)
[error] at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
[error] at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
[error] at org.apache.spark.rdd.RDD.withScope(RDD.scala:388)
[error] at org.apache.spark.rdd.RDD.collect(RDD.scala:1003)
[error] at Main$.delayedEndpoint$Main$1(Main.scala:16)
[error] at Main$delayedInit$body.apply(Main.scala:2)
[error] at scala.Function0.apply$mcV$sp(Function0.scala:39)
[error] at scala.Function0.apply$mcV$sp$(Function0.scala:39)
[error] at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
[error] at scala.App.$anonfun$main$1$adapted(App.scala:80)
[error] at scala.collection.immutable.List.foreach(List.scala:392)
[error] at scala.App.main(App.scala:80)
[error] at scala.App.main$(App.scala:78)
[error] at Main$.main(Main.scala:2)
[error] at Main.main(Main.scala)
[error] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[error] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[error] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[error] at java.lang.reflect.Method.invoke(Method.java:498)
[error] Caused by: java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f of type scala.Function3 in instance of org.apache.spark.rdd.MapPartitionsRDD
[error] at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2301)
[error] at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1431)
[error] at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2410)
[error] at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2328)
[error] at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2186)
[error] at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1666)
[error] at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2404)
[error] at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2328)
[error] at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2186)
[error] at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1666)
[error] at java.io.ObjectInputStream.readObject(ObjectInputStream.java:502)
[error] at java.io.ObjectInputStream.readObject(ObjectInputStream.java:460)
[error] at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
[error] at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
[error] at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:83)
[error] at org.apache.spark.scheduler.Task.run(Task.scala:127)
[error] at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
[error] at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
[error] at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
[error] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[error] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[error] at java.lang.Thread.run(Thread.java:748)
If I need more libraries or if the code is wrong (but it works fine in spark-shell)....
How to solve it?
You need to submit your jars to spark so that your code can run on there. spark-shell is hiding all of this from you behind the scenes.
This answer provides better detail https://stackoverflow.com/a/28367602/1810962 with the background.
You can use bin/spark-submit as workaround and provide your local classpath using --class, --jars, and --driver-class-path

java.lang.NoSuchFieldError when using ScalaTest

I'm getting a quite-hard-to-debug error when using ScalaTest. Oddly this seems to occur when my program has no Main object, but not when it does have a Main object. My code is really just using a typeclass with a polymorphic method to get a slice from a list, and looks like this (apologies for the slightly verbose example, I've reduced it down as much as I can):
package sportarray
object ArrayDefs {
abstract class IsArr[A, I0, DT] {
def getElem(self: A, i: Int): DT
def getSlice[R](self: A, r: R)(implicit sliceTc: IsSlice[R]): sliceTc.Out = sliceTc.getSlice(self, r)
trait IsSlice[R] {
type Out
def getSlice(self: A, ref: R): Out
}
object IsSlice {
implicit val iLocTCForInt = new IsSlice[Int] {
type Out = DT
def getSlice(self: A, ref: Int): Out = getElem(self, ref)
}
implicit val iLocTCForList = new IsSlice[List[Int]] {
type Out = List[DT]
def getSlice(self: A, ref: List[Int]): Out = ref.map(getElem(self, _))
}
}
}
object IsArrSyntax {
implicit class IsArrOps[A, I0, T](self: A)(implicit
tc1d: IsArr[A, I0, T]
) {
def getElem(i: Int) = tc1d.getElem(self, i)
def getSlice[R](r: R)(implicit sliceTc: tc1d.IsSlice[R]) = tc1d.getSlice(self, r)
}
}
}
object ListOfListsObj {
import ArrayDefs._
case class List1d[I0, T] (
indices: List[I0],
data: List[T],
)
implicit def list1dIs1dSpArr[A, I0, T] =
new IsArr[List1d[I0, T], I0, T] {
def getElem(self: List1d[I0, T], i: Int) = self.data(i)
}
}
My test is simple and looks like this, in its own file in the test directory:
package sportarray
import org.scalatest.flatspec.AnyFlatSpec
import org.scalatest.matchers.should.Matchers
class ArraySpec extends AnyFlatSpec with Matchers {
"List" should "act as an array" in {
import ArrayDefs._
import ArrayDefs.IsArrSyntax._
import ListOfListsObj._
val list1d = List1d[Int, Double](List(1,2,3), List(1.0, 2.0, 3.0))
assert(list1d.getSlice(1) == 2.0)
}
}
If I sbt test this code as it is, I get the following error:
java.lang.NoSuchFieldError: sportarray$ArrayDefs$IsArrSyntax$IsArrOps$$tc1d
at sportarray.ArraySpec.$anonfun$new$1(HelloSpec.scala:12)
at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
at org.scalatest.Transformer.apply(Transformer.scala:22)
at org.scalatest.Transformer.apply(Transformer.scala:20)
at org.scalatest.flatspec.AnyFlatSpecLike$$anon$5.apply(AnyFlatSpecLike.scala:1683)
I then tried adding a Main class to the program to see if this was a problem with the test, or a problem with the code itself:
object Main extends App {
import ArrayDefs._
import ArrayDefs.IsArrSyntax._
import ListOfListsObj._
val list1d = List1d[Int, Double](List(1,2,3), List(1.0, 2.0, 3.0))
assert(list1d.getSlice(1) == 2.0)
}
This works fine if I sbt run, and, oddly enough, also allows me to run sbt test without any problems. If I then delete this main class, sbt test once again fails. Is anybody able to shed any light on what is going on here?
Thanks for any help!
Actually I ran Scalatest from command line and it succeded
$ scalac -cp .:scalactic_2.13-3.2.2.jar:scalatest_2.13-3.2.2.jar:scalatest-core_2.13-3.2.2.jar:scalatest-flatspec_2.13-3.2.2.jar:scalatest-matchers-core_2.13-3.2.2.jar:scalatest-shouldmatchers_2.13-3.2.2.jar:scalatest-compatible-3.2.2.jar App.scala AppTest.scala
$ scala -cp .:scalactic_2.13-3.2.2.jar:scalatest_2.13-3.2.2.jar:scalatest-core_2.13-3.2.2.jar:scalatest-flatspec_2.13-3.2.2.jar:scalatest-matchers-core_2.13-3.2.2.jar:scalatest-shouldmatchers_2.13-3.2.2.jar:scalatest-compatible-3.2.2.jar:scala-xml_2.13-1.3.0.jar org.scalatest.run AppTest
Run starting. Expected test count is: 1
AppTest:
List
- should act as an array
Run completed in 151 milliseconds.
Total number of tests run: 1
Suites: completed 1, aborted 0
Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
All tests passed.
while in IntelliJ and sbt it failed:
(IntelliJ)
An exception or error caused a run to abort: App$ArrayDefs$IsArrSyntax$IsArrOps$$tc1d
java.lang.NoSuchFieldError: App$ArrayDefs$IsArrSyntax$IsArrOps$$tc1d
at AppTest.$anonfun$new$1(AppTest.scala:10)
at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
at org.scalatest.Transformer.apply(Transformer.scala:22)
at org.scalatest.Transformer.apply(Transformer.scala:20)
at org.scalatest.flatspec.AnyFlatSpecLike$$anon$5.apply(AnyFlatSpecLike.scala:1683)
at org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
at org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
at org.scalatest.flatspec.AnyFlatSpec.withFixture(AnyFlatSpec.scala:1685)
at org.scalatest.flatspec.AnyFlatSpecLike.invokeWithFixture$1(AnyFlatSpecLike.scala:1681)
at org.scalatest.flatspec.AnyFlatSpecLike.$anonfun$runTest$1(AnyFlatSpecLike.scala:1693)
at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
at org.scalatest.flatspec.AnyFlatSpecLike.runTest(AnyFlatSpecLike.scala:1693)
at org.scalatest.flatspec.AnyFlatSpecLike.runTest$(AnyFlatSpecLike.scala:1675)
at org.scalatest.flatspec.AnyFlatSpec.runTest(AnyFlatSpec.scala:1685)
at org.scalatest.flatspec.AnyFlatSpecLike.$anonfun$runTests$1(AnyFlatSpecLike.scala:1751)
at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
at scala.collection.immutable.List.foreach(List.scala:333)
at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:390)
at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:427)
at scala.collection.immutable.List.foreach(List.scala:333)
at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
at org.scalatest.flatspec.AnyFlatSpecLike.runTests(AnyFlatSpecLike.scala:1751)
at org.scalatest.flatspec.AnyFlatSpecLike.runTests$(AnyFlatSpecLike.scala:1750)
at org.scalatest.flatspec.AnyFlatSpec.runTests(AnyFlatSpec.scala:1685)
at org.scalatest.Suite.run(Suite.scala:1112)
at org.scalatest.Suite.run$(Suite.scala:1094)
at org.scalatest.flatspec.AnyFlatSpec.org$scalatest$flatspec$AnyFlatSpecLike$$super$run(AnyFlatSpec.scala:1685)
at org.scalatest.flatspec.AnyFlatSpecLike.$anonfun$run$1(AnyFlatSpecLike.scala:1796)
at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
at org.scalatest.flatspec.AnyFlatSpecLike.run(AnyFlatSpecLike.scala:1796)
at org.scalatest.flatspec.AnyFlatSpecLike.run$(AnyFlatSpecLike.scala:1794)
at org.scalatest.flatspec.AnyFlatSpec.run(AnyFlatSpec.scala:1685)
at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:45)
at org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13(Runner.scala:1320)
at org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13$adapted(Runner.scala:1314)
at scala.collection.immutable.List.foreach(List.scala:333)
at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1314)
at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24(Runner.scala:993)
at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24$adapted(Runner.scala:971)
at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:1480)
at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:971)
at org.scalatest.tools.Runner$.run(Runner.scala:798)
at org.scalatest.tools.Runner.run(Runner.scala)
at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.runScalaTest2or3(ScalaTestRunner.java:40)
at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.main(ScalaTestRunner.java:27)
(sbt)
[error] java.lang.NoSuchFieldError: App$ArrayDefs$IsArrSyntax$IsArrOps$$tc1d
[error] at AppTest.$anonfun$new$1(AppTest.scala:10)
[error] at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
[error] at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
[error] at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[error] at org.scalatest.Transformer.apply(Transformer.scala:22)
[error] at org.scalatest.Transformer.apply(Transformer.scala:20)
[error] at org.scalatest.flatspec.AnyFlatSpecLike$$anon$5.apply(AnyFlatSpecLike.scala:1683)
[error] at org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
[error] at org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
[error] at org.scalatest.flatspec.AnyFlatSpec.withFixture(AnyFlatSpec.scala:1685)
[error] at org.scalatest.flatspec.AnyFlatSpecLike.invokeWithFixture$1(AnyFlatSpecLike.scala:1681)
[error] at org.scalatest.flatspec.AnyFlatSpecLike.$anonfun$runTest$1(AnyFlatSpecLike.scala:1693)
[error] at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
[error] at org.scalatest.flatspec.AnyFlatSpecLike.runTest(AnyFlatSpecLike.scala:1693)
[error] at org.scalatest.flatspec.AnyFlatSpecLike.runTest$(AnyFlatSpecLike.scala:1675)
[error] at org.scalatest.flatspec.AnyFlatSpec.runTest(AnyFlatSpec.scala:1685)
[error] at org.scalatest.flatspec.AnyFlatSpecLike.$anonfun$runTests$1(AnyFlatSpecLike.scala:1751)
[error] at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
[error] at scala.collection.immutable.List.foreach(List.scala:333)
[error] at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
[error] at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:390)
[error] at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:427)
[error] at scala.collection.immutable.List.foreach(List.scala:333)
[error] at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
[error] at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
[error] at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
[error] at org.scalatest.flatspec.AnyFlatSpecLike.runTests(AnyFlatSpecLike.scala:1751)
[error] at org.scalatest.flatspec.AnyFlatSpecLike.runTests$(AnyFlatSpecLike.scala:1750)
[error] at org.scalatest.flatspec.AnyFlatSpec.runTests(AnyFlatSpec.scala:1685)
[error] at org.scalatest.Suite.run(Suite.scala:1112)
[error] at org.scalatest.Suite.run$(Suite.scala:1094)
[error] at org.scalatest.flatspec.AnyFlatSpec.org$scalatest$flatspec$AnyFlatSpecLike$$super$run(AnyFlatSpec.scala:1685)
[error] at org.scalatest.flatspec.AnyFlatSpecLike.$anonfun$run$1(AnyFlatSpecLike.scala:1796)
[error] at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
[error] at org.scalatest.flatspec.AnyFlatSpecLike.run(AnyFlatSpecLike.scala:1796)
[error] at org.scalatest.flatspec.AnyFlatSpecLike.run$(AnyFlatSpecLike.scala:1794)
[error] at org.scalatest.flatspec.AnyFlatSpec.run(AnyFlatSpec.scala:1685)
[error] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:318)
[error] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:513)
[error] at sbt.TestRunner.runTest$1(TestFramework.scala:139)
[error] at sbt.TestRunner.run(TestFramework.scala:154)
[error] at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.$anonfun$apply$1(TestFramework.scala:317)
[error] at sbt.TestFramework$.sbt$TestFramework$$withContextLoader(TestFramework.scala:277)
[error] at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:317)
[error] at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:317)
[error] at sbt.TestFunction.apply(TestFramework.scala:329)
[error] at sbt.Tests$.$anonfun$toTask$1(Tests.scala:311)
[error] at sbt.std.Transform$$anon$3.$anonfun$apply$2(Transform.scala:46)
[error] at sbt.std.Transform$$anon$4.work(Transform.scala:67)
[error] at sbt.Execute.$anonfun$submit$2(Execute.scala:281)
[error] at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:19)
[error] at sbt.Execute.work(Execute.scala:290)
[error] at sbt.Execute.$anonfun$submit$1(Execute.scala:281)
[error] at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:178)
[error] at sbt.CompletionService$$anon$2.call(CompletionService.scala:37)
[error] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[error] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[error] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[error] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[error] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[error] at java.lang.Thread.run(Thread.java:748)
[error] (Test / executeTests) java.lang.NoSuchFieldError: App$ArrayDefs$IsArrSyntax$IsArrOps$$tc1d
So the issue seems to be how IntelliJ and sbt run Scalatest.
NoSuchFieldError Java
Java NoSuchFieldError when using Reflection
https://github.com/unicredit/hbase-rdd/issues/21

For loop over list while filtering

I have 2 DataFrames called df1 and df2, where they both have the same column names. I wish to run a for loop over unique dates, from df1 and apply the same date filter to df2. I created a list of unique dates and then tried to iterate through that. However what I have is throwing errors.
Here is what I have:
val unique_weeks = df1.select(df1("date")).distinct
for( week <- unique_weeks) {
val df1_filtered = df1.filter($"date" === week)
val df2_filtered = df2.filter($"date" === week)
/// will run a join here and more code
}
I think <- this part may be incorrect - but not sure how I can filter the DataFrames using another method.
Here is the error:
[error] (run-main-0) org.apache.spark.SparkException: Job aborted due to stage failure: Task 35 in stage 3.0 failed 1 times, most recent failure: Lost task 35.0 in stage 3.0 (TID 399, localhost, executor driver): java.lang.RuntimeException: Unsupported literal type class org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema [1591772400000]
[error] at org.apache.spark.sql.catalyst.expressions.Literal$.apply(literals.scala:75)
[error] at org.apache.spark.sql.functions$.lit(functions.scala:101)
[error] at org.apache.spark.sql.Column.$eq$eq$eq(Column.scala:267)
[error] at spark_pkg.SparkMain$$anonfun$main$1.apply(SparkMain.scala:880)
[error] at spark_pkg.SparkMain$$anonfun$main$1.apply(SparkMain.scala:878)
[error] at scala.collection.Iterator$class.foreach(Iterator.scala:893)
[error] at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
[error] at org.apache.spark.rdd.RDD$$anonfun$foreach$1$$anonfun$apply$28.apply(RDD.scala:917)
[error] at org.apache.spark.rdd.RDD$$anonfun$foreach$1$$anonfun$apply$28.apply(RDD.scala:917)
[error] at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1944)
[error] at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1944)
[error] at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
[error] at org.apache.spark.scheduler.Task.run(Task.scala:99)
[error] at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
[error] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[error] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[error] at java.lang.Thread.run(Thread.java:748)
[error]
[error] Driver stacktrace:
[error] org.apache.spark.SparkException: Job aborted due to stage failure: Task 35 in stage 3.0 failed 1 times, most recent failure: Lost task 35.0 in stage 3.0 (TID 399, localhost, executor driver): java.lang.RuntimeException: Unsupported literal type class org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema [1591772400000]
[error] at org.apache.spark.sql.catalyst.expressions.Literal$.apply(literals.scala:75)
[error] at org.apache.spark.sql.functions$.lit(functions.scala:101)
[error] at org.apache.spark.sql.Column.$eq$eq$eq(Column.scala:267)
[error] at spark_pkg.SparkMain$$anonfun$main$1.apply(SparkMain.scala:880)
[error] at spark_pkg.SparkMain$$anonfun$main$1.apply(SparkMain.scala:878)
[error] at scala.collection.Iterator$class.foreach(Iterator.scala:893)
[error] at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
[error] at org.apache.spark.rdd.RDD$$anonfun$foreach$1$$anonfun$apply$28.apply(RDD.scala:917)
[error] at org.apache.spark.rdd.RDD$$anonfun$foreach$1$$anonfun$apply$28.apply(RDD.scala:917)
[error] at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1944)
[error] at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1944)
[error] at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
[error] at org.apache.spark.scheduler.Task.run(Task.scala:99)
[error] at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
[error] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[error] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[error] at java.lang.Thread.run(Thread.java:748)
[error]
[error] Driver stacktrace:
[error] at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1435)
[error] at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1423)
[error] at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1422)
[error] at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
[error] at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
[error] at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1422)
[error] at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802)
[error] at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802)
[error] at scala.Option.foreach(Option.scala:257)
[error] at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:802)
[error] at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1650)
[error] at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1605)
[error] at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1594)
[error] at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
[error] at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:628)
[error] at org.apache.spark.SparkContext.runJob(SparkContext.scala:1918)
[error] at org.apache.spark.SparkContext.runJob(SparkContext.scala:1931)
[error] at org.apache.spark.SparkContext.runJob(SparkContext.scala:1944)
[error] at org.apache.spark.SparkContext.runJob(SparkContext.scala:1958)
[error] at org.apache.spark.rdd.RDD$$anonfun$foreach$1.apply(RDD.scala:917)
[error] at org.apache.spark.rdd.RDD$$anonfun$foreach$1.apply(RDD.scala:915)
[error] at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
[error] at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
[error] at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)
[error] at org.apache.spark.rdd.RDD.foreach(RDD.scala:915)
[error] at org.apache.spark.sql.Dataset$$anonfun$foreach$1.apply$mcV$sp(Dataset.scala:2286)
[error] at org.apache.spark.sql.Dataset$$anonfun$foreach$1.apply(Dataset.scala:2286)
[error] at org.apache.spark.sql.Dataset$$anonfun$foreach$1.apply(Dataset.scala:2286)
[error] at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:57)
[error] at org.apache.spark.sql.Dataset.withNewExecutionId(Dataset.scala:2765)
[error] at org.apache.spark.sql.Dataset.foreach(Dataset.scala:2285)
[error] at spark_pkg.SparkMain$.main(SparkMain.scala:878)
[error] at spark_pkg.SparkMain.main(SparkMain.scala)
[error] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[error] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[error] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[error] at java.lang.reflect.Method.invoke(Method.java:498)
[error] Caused by: java.lang.RuntimeException: Unsupported literal type class org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema [1591772400000]
[error] at org.apache.spark.sql.catalyst.expressions.Literal$.apply(literals.scala:75)
[error] at org.apache.spark.sql.functions$.lit(functions.scala:101)
[error] at org.apache.spark.sql.Column.$eq$eq$eq(Column.scala:267)
[error] at spark_pkg.SparkMain$$anonfun$main$1.apply(SparkMain.scala:880)
[error] at spark_pkg.SparkMain$$anonfun$main$1.apply(SparkMain.scala:878)
[error] at scala.collection.Iterator$class.foreach(Iterator.scala:893)
[error] at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
[error] at org.apache.spark.rdd.RDD$$anonfun$foreach$1$$anonfun$apply$28.apply(RDD.scala:917)
[error] at org.apache.spark.rdd.RDD$$anonfun$foreach$1$$anonfun$apply$28.apply(RDD.scala:917)
[error] at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1944)
[error] at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1944)
[error] at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
[error] at org.apache.spark.scheduler.Task.run(Task.scala:99)
[error] at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
[error] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[error] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[error] at java.lang.Thread.run(Thread.java:748)
[error] stack trace is suppressed; run 'last Compile / bgRun' for the full output
[error] Nonzero exit code: 1
[error] (Compile / run) Nonzero exit code: 1
[error] Total time: 137 s (02:17), completed Aug 20, 2020 1:16:02 PM
A dataframe is not an iterator, and therefore, you cannot run a for loop over it. You can run something like this - but I don't think it will do what you're hoping to achieve based on your other code.
unique_weeks.foreachPartition{ weeks : Iterator[YourData] =>
for( week <- weeks) {
}
}
Your question suggests your mental model of what a dataframe is and how Spark works is not quite complete. Think of a Dataframe more as a List[List[YourData]], except each inner List[YourData] is located on an independent piece of a machine, and may not necessarily know or interact with any of the other Lists until you collect them back to the driver.

Unable to run the main method

I am currently working on a system where staff members in an organization can be informed regarding their pending tasks to be completed by their managers via devices. This is demonstrated via the following classes:
Staff
Manager (Extends Staff)
Device (Receives tasks and removes them once completed)
Task
System (Assigns tasks)
Main
I am currently stuck at a point where I am unable to determine why I cannot run my classes, as I receive a non-zero exit code of 1.
The problem currently seems to lie within the Main.scala file, at the moment:
object Main {
def main(args: Array[String]): Unit = {
var system = new System
var s1 = new Staff(1, "John", "johndoe#outlook.com", "Brazil")
system.addStaff(s1)
var s2 = new Manager(2, "Reese", "reesecups#gmail.com", "Japan")
system.addStaff(s2)
s2.assignTask(system, 1, "PLEASE WORK")
}
}
The code used to run as intended, however when I added the following line to the Main.scala file:
s2.assignTask(system, 1, "PLEASE WORK")
I received the following error:
[error] java.lang.RuntimeException: No main class detected.
[error] at scala.sys.package$.error(package.scala:30)
[error] at sbt.Defaults$.$anonfun$bgRunTask$4(Defaults.scala:1477)
[error] at scala.Option.getOrElse(Option.scala:189)
[error] at sbt.Defaults$.$anonfun$bgRunTask$3(Defaults.scala:1477)
[error] at scala.Function1.$anonfun$compose$1(Function1.scala:49)
[error] at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:62)
[error] at sbt.std.Transform$$anon$4.work(Transform.scala:67)
[error] at sbt.Execute.$anonfun$submit$2(Execute.scala:281)
[error] at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:19)
[error] at sbt.Execute.work(Execute.scala:290)
[error] at sbt.Execute.$anonfun$submit$1(Execute.scala:281)
[error] at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:178)
[error] at sbt.CompletionService$$anon$2.call(CompletionService.scala:37)
[error] at java.util.concurrent.FutureTask.run(Unknown Source)
[error] at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
[error] at java.util.concurrent.FutureTask.run(Unknown Source)
[error] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
[error] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[error] at java.lang.Thread.run(Unknown Source)
[error] (Compile / bgRun) No main class detected.
Is there any indication to what the problem may be?