I am upgrading a Spark 2.4 project to Spark 3.x. We are hitting a snag with some existing Spark-ml code:
var stringIndexers = Array[StringIndexer]()
for (featureColumn <- FEATURE_COLS) {
stringIndexers = stringIndexers :+ new StringIndexer().setInputCol(featureColumn).setOutputCol(featureColumn + "_index")
}
val pipeline = new Pipeline().setStages(stringIndexers)
val dfWithNumericalFeatures = pipeline.fit(decoratedDf).transform(decoratedDf)
Specifically, this line: val dfWithNumericalFeatures = pipeline.fit(decoratedDf).transform(decoratedDf) now results in this cryptic exception in Spark 3:
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 238.0 failed 1 times, most recent failure: Lost task 0.0 in stage 238.0 (TID 5589) (executor driver): com.esotericsoftware.kryo.KryoException: Unable to find class: org.apache.spark.util.collection.OpenHashMap$mcJ$sp$$Lambda$13346/2134122295
[info] Serialization trace:
[info] org$apache$spark$util$collection$OpenHashMap$$grow (org.apache.spark.util.collection.OpenHashMap$mcJ$sp)
[info] at com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:156)
[info] at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:133)
[info] at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:670)
[info] at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:118)
[info] at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
[info] at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
[info] at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:396)
[info] at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:307)
[info] at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)
[info] at org.apache.spark.serializer.KryoSerializerInstance.deserialize(KryoSerializer.scala:397)
[info] at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificSafeProjection.apply(Unknown Source)
[info] at org.apache.spark.sql.execution.aggregate.ComplexTypedAggregateExpression.deserialize(TypedAggregateExpression.scala:271)
[info] at org.apache.spark.sql.catalyst.expressions.aggregate.TypedImperativeAggregate.merge(interfaces.scala:568)
[info] at org.apache.spark.sql.execution.aggregate.AggregationIterator$$anonfun$1.$anonfun$applyOrElse$3(AggregationIterator.scala:199)
[info] at org.apache.spark.sql.execution.aggregate.AggregationIterator$$anonfun$1.$anonfun$applyOrElse$3$adapted(AggregationIterator.scala:199)
[info] at org.apache.spark.sql.execution.aggregate.AggregationIterator.$anonfun$generateProcessRow$7(AggregationIterator.scala:213)
[info] at org.apache.spark.sql.execution.aggregate.AggregationIterator.$anonfun$generateProcessRow$7$adapted(AggregationIterator.scala:207)
[info] at org.apache.spark.sql.execution.aggregate.ObjectAggregationIterator.processInputs(ObjectAggregationIterator.scala:151)
[info] at org.apache.spark.sql.execution.aggregate.ObjectAggregationIterator.<init>(ObjectAggregationIterator.scala:77)
[info] at org.apache.spark.sql.execution.aggregate.ObjectHashAggregateExec.$anonfun$doExecute$2(ObjectHashAggregateExec.scala:107)
[info] at org.apache.spark.sql.execution.aggregate.ObjectHashAggregateExec.$anonfun$doExecute$2$adapted(ObjectHashAggregateExec.scala:85)
[info] at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsWithIndexInternal$2(RDD.scala:885)
[info] at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsWithIndexInternal$2$adapted(RDD.scala:885)
[info] at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
[info] at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
[info] at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
[info] at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
[info] at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
[info] at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
[info] at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
[info] at org.apache.spark.scheduler.Task.run(Task.scala:131)
[info] at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)
[info] at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)
[info] at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)
[info] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[info] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[info] at java.lang.Thread.run(Thread.java:750)
[info] Caused by: java.lang.ClassNotFoundException: org.apache.spark.util.collection.OpenHashMap$mcJ$sp$$Lambda$13346/2134122295
[info] at java.lang.Class.forName0(Native Method)
[info] at java.lang.Class.forName(Class.java:348)
[info] at com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:154)
[info] ... 36 more
I have searched around and the only relevant issue I've found is this un-answered SO question with the same issue: Spark Kryo Serialization issue.
OpenHashMap is not used in my code, seems likely that there is a bug with the KryoSerializer during this Pipeline.fit() function. Any ideas how to get around this? Thanks!
EDIT: I also just attempted removing usage of the KryoSerializer during my unit tests:
spark = SparkSession
.builder
.master("local[*]")
.appName("UnitTest")
.config("spark.serializer", "org.apache.spark.serializer.JavaSerializer")
.config("spark.driver.bindAddress", "127.0.0.1")
.getOrCreate()
Confirmed that I am using the JavaSerializer:
println(spark.conf.get("spark.serializer")) outputs org.apache.spark.serializer.JavaSerializer. Still same issue however, even when not using the KryoSerializer.
I've got a Scala project which I'm building with sbt. When running sbt test, the tests themselves pass, but then the command fails with "Forked test harness failed: java.io.EOFException".
The build.sbt file does not specify fork in Test.
Example of the error it fails with after running sbt test:
[info] Run completed in 5 seconds, 494 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 0, aborted 0
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[error] Error during tests:
[error] Forked test harness failed: java.io.EOFException
[error] at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2959)
[error] at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1539)
[error] at java.io.ObjectInputStream.readObject(ObjectInputStream.java:430)
[error] at sbt.React.react(ForkTests.scala:177)
[error] at sbt.ForkTests$Acceptor$1$.run(ForkTests.scala:108)
[error] at java.lang.Thread.run(Thread.java:748)
[error] (serverTests / Test / test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 14 s, completed Mar 12, 2020 4:35:26 PM
Minimal example of test which fails:
package com.example
import akka.http.scaladsl.testkit.ScalatestRouteTest
import org.scalatest.FreeSpecLike
class ForkedTestHarnessFailedForNoReasonSpec extends FreeSpecLike with ScalatestRouteTest {
"This test" - {
"should not fail" in {
assert("Foo" == "Foo")
}
}
}
What does this error indicate and how should one resolve it?
The cause in my case was that AKKA is shutting down JVM on coordinated shutdown. Put this to your test config (src/test/resources/reference.conf in my case):
akka.coordinated-shutdown.exit-jvm = off
I need some help understanding errors that are being generated through Scala class for the RandomForestAlgorithm.scala (https://github.com/PredictionIO/PredictionIO/blob/develop/examples/scala-parallel-classification/custom-attributes/src/main/scala/RandomForestAlgorithm.scala).
I am building the project as is (custom-attributes for classification template) in PredictionIO and am getting a pio build error:
hduser#hduser-VirtualBox:~/PredictionIO/classTest$ pio build --verbose
[INFO] [Console$] Using existing engine manifest JSON at /home/hduser/PredictionIO/classTest/manifest.json
[INFO] [Console$] Using command '/home/hduser/PredictionIO/sbt/sbt' at the current working directory to build.
[INFO] [Console$] If the path above is incorrect, this process will fail.
[INFO] [Console$] Uber JAR disabled. Making sure lib/pio-assembly-0.9.5.jar is absent.
[INFO] [Console$] Going to run: /home/hduser/PredictionIO/sbt/sbt package assemblyPackageDependency
[INFO] [Console$] [info] Loading project definition from /home/hduser/PredictionIO/classTest/project
[INFO] [Console$] [info] Set current project to template-scala-parallel-classification (in build file:/home/hduser/PredictionIO/classTest/)
[INFO] [Console$] [info] Compiling 1 Scala source to /home/hduser/PredictionIO/classTest/target/scala-2.10/classes...
[INFO] [Console$] [error] /home/hduser/PredictionIO/classTest/src/main/scala/RandomForestAlgorithm.scala:28: class RandomForestAlgorithm **needs to be abstract**, since method train in class P2LAlgorithm of type (sc: org.apache.spark.SparkContext, pd: com.test1.PreparedData)com.test1.**PIORandomForestModel is not defined**
[INFO] [Console$] [error] class RandomForestAlgorithm(val ap: RandomForestAlgorithmParams) // CHANGED
[INFO] [Console$] [error] ^
[INFO] [Console$] [error] one error found
[INFO] [Console$] [error] (compile:compile) Compilation failed
[INFO] [Console$] [error] Total time: 6 s, completed Jun 8, 2016 4:37:36 PM
[ERROR] [Console$] Return code of previous step is 1. Aborting.
so when I address the line causing the error and make it an abstract object:
// extends P2LAlgorithm because the MLlib's RandomForestModel doesn't
// contain RDD.
abstract class RandomForestAlgorithm(val ap: RandomForestAlgorithmParams) // CHANGED
extends P2LAlgorithm[PreparedData, PIORandomForestModel, // CHANGED
Query, PredictedResult] {
def train(data: PreparedData): PIORandomForestModel = { // CHANGED
// CHANGED
// Empty categoricalFeaturesInfo indicates all features are continuous.
val categoricalFeaturesInfo = Map[Int, Int]()
val m = RandomForest.trainClassifier(
data.labeledPoints,
ap.numClasses,
categoricalFeaturesInfo,
ap.numTrees,
ap.featureSubsetStrategy,
ap.impurity,
ap.maxDepth,
ap.maxBins)
new PIORandomForestModel(
gendersMap = data.gendersMap,
educationMap = data.educationMap,
randomForestModel = m
)
}
pio build is successful but training fails because it can't instantiate the new assignments for the model:
[INFO] [Engine] Extracting datasource params...
[INFO] [WorkflowUtils$] No 'name' is found. Default empty String will be used.
[INFO] [Engine] Datasource params: (,DataSourceParams(6))
[INFO] [Engine] Extracting preparator params...
[INFO] [Engine] Preparator params: (,Empty)
[INFO] [Engine] Extracting serving params...
[INFO] [Engine] Serving params: (,Empty)
[WARN] [Utils] Your hostname, hduser-VirtualBox resolves to a loopback address: 127.0.1.1; using 10.0.2.15 instead (on interface eth0)
[WARN] [Utils] Set SPARK_LOCAL_IP if you need to bind to another address
[INFO] [Remoting] Starting remoting
[INFO] [Remoting] Remoting started; listening on addresses :[akka.tcp://sparkDriver#10.0.2.15:59444]
[WARN] [MetricsSystem] Using default name DAGScheduler for source because spark.app.id is not set.
**Exception in thread "main" java.lang.InstantiationException**
at sun.reflect.InstantiationExceptionConstructorAccessorImpl.newInstance(InstantiationExceptionConstructorAccessorImpl.java:48)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at io.prediction.core.Doer$.apply(AbstractDoer.scala:52)
at io.prediction.controller.Engine$$anonfun$1.apply(Engine.scala:171)
at io.prediction.controller.Engine$$anonfun$1.apply(Engine.scala:170)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.immutable.List.foreach(List.scala:318)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractTraversable.map(Traversable.scala:105)
at io.prediction.controller.Engine.train(Engine.scala:170)
at io.prediction.workflow.CoreWorkflow$.runTrain(CoreWorkflow.scala:65)
at io.prediction.workflow.CreateWorkflow$.main(CreateWorkflow.scala:247)
at io.prediction.workflow.CreateWorkflow.main(CreateWorkflow.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
So two questions:
1. Why is the following model not considered defined during building:
class PIORandomForestModel(
val gendersMap: Map[String, Double],
val educationMap: Map[String, Double],
val randomForestModel: RandomForestModel
) extends Serializable
How can I define PIORandomForestModel in a way that does not throw a pio build error and lets training re-assign attributes to the object?
I have posted this question in the PredictionIO Google group but have not gotten a response.
Thanks in advance for your help.
I'm using spray-json 1.3.0. I have a simple piece of Json that I am asking spray to parse, here it is
import org.scalatest.{FlatSpec, MustMatchers}
import spray.json._
class BlockCypherOutputMarshallerTest extends FlatSpec with MustMatchers {
val expectedOutput = """{"value":7454642,
|"script":"76a9148d5968ad26f9e277849ff9f8f39920f28944467388ac",
|"addresses":["mtQLgLiqmytKkgE9sVGwypAFsLvkxBQ6XX"],
|"script_type":"pay-to-pubkey-hash}""".stripMargin
val json = expectedOutput.parseJson
"BlockCypherOutputMarshaller" must "parse an output from the blockcypher api" in {
//test case
}
}
however I am getting an error message on the line that val json = expectedOutput.parseJson is called. Here is the error message
> last test:testOnly
[debug] Running TaskDef(com.blockcypher.api.marshallers.BlockCypherOutputMarshallerTest, org.scalatest.tools.Framework$$anon$1#4178c07b, false, [SuiteSelector])
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3332)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:622)
at java.lang.StringBuilder.append(StringBuilder.java:202)
at spray.json.JsonParser.appendSB(JsonParser.scala:179)
at spray.json.JsonParser.char(JsonParser.scala:138)
at spray.json.JsonParser.string(JsonParser.scala:129)
at spray.json.JsonParser.value(JsonParser.scala:62)
at spray.json.JsonParser.members$1(JsonParser.scala:80)
at spray.json.JsonParser.object(JsonParser.scala:84)
at spray.json.JsonParser.value(JsonParser.scala:59)
at spray.json.JsonParser.parseJsValue(JsonParser.scala:43)
at spray.json.JsonParser$.apply(JsonParser.scala:28)
at spray.json.PimpedString.parseJson(package.scala:45)
at com.blockcypher.api.marshallers.BlockCypherOutputMarshallerTest.<init>(BlockCypherOutputMarshallerTest.scala:16)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at java.lang.Class.newInstance(Class.java:442)
at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:641)
at sbt.TestRunner.runTest$1(TestFramework.scala:76)
at sbt.TestRunner.run(TestFramework.scala:85)
at sbt.TestFramework$$anon$2$$anonfun$$init$$1$$anonfun$apply$8.apply(TestFramework.scala:202)
at sbt.TestFramework$$anon$2$$anonfun$$init$$1$$anonfun$apply$8.apply(TestFramework.scala:202)
at sbt.TestFramework$.sbt$TestFramework$$withContextLoader(TestFramework.scala:185)
at sbt.TestFramework$$anon$2$$anonfun$$init$$1.apply(TestFramework.scala:202)
at sbt.TestFramework$$anon$2$$anonfun$$init$$1.apply(TestFramework.scala:202)
at sbt.TestFunction.apply(TestFramework.scala:207)
at sbt.Tests$$anonfun$9.apply(Tests.scala:216)
at sbt.Tests$$anonfun$9.apply(Tests.scala:216)
[error] Could not run test com.blockcypher.api.marshallers.BlockCypherOutputMarshallerTest: java.lang.OutOfMemoryError: Java heap space
[debug] Summary for ScalaCheck not available.
[debug] Summary for specs2 not available.
[info] ScalaTest
[info] Run completed in 1 second, 541 milliseconds.
[info] Total number of tests run: 0
[info] Suites: completed 0, aborted 0
[info] Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0
[info] No tests were executed.
[error] Error: Total 0, Failed 0, Errors 0, Passed 0
[error] Error during tests:
[error] com.blockcypher.api.marshallers.BlockCypherOutputMarshallerTest
[error] (test:testOnly) sbt.TestsFailedException: Tests unsuccessful
I'm unsure of why I am running out of heap space, it seems that the piece of json is simple enough and I am not having any issues on any other similar size json test cases.
You need to close quote in your last line "pay-to-pubkey-hash
"script_type":"pay-to-pubkey-hash}"""
should be
"script_type":"pay-to-pubkey-hash"}"""
sbt 0.12.4 on Windows.
Firstly, I move into the project directory, that is example - a Scala project under sbt. When I run sbt, I get the following errors:
C:\programs\example>sbt
[info] Loading project definition from C:\programs\example\project\project
[info] Updating {file:/C:/programs/example/project/project/}default-2ad7de...
[info] Resolving org.scala-sbt#sbt;0.12.4 ...
[info] Resolving org.scala-sbt#main;0.12.4 ...
[info] Resolving org.scala-sbt#actions;0.12.4 ...
[info] Resolving org.scala-sbt#classpath;0.12.4 ...
[info] Resolving org.scala-sbt#launcher-interface;0.12.4 ...
[info] Resolving org.scala-lang#scala-library;2.9.2 ...
[info] Resolving org.scala-sbt#interface;0.12.4 ...
[info] Resolving org.scala-sbt#io;0.12.4 ...
[info] Resolving org.scala-sbt#control;0.12.4 ...
[info] Resolving org.scala-lang#scala-compiler;2.9.2 ...
[info] Resolving org.scala-sbt#completion;0.12.4 ...
[info] Resolving org.scala-sbt#collections;0.12.4 ...
[info] Resolving jline#jline;1.0 ...
[info] Resolving org.scala-sbt#api;0.12.4 ...
[info] Resolving org.scala-sbt#compiler-integration;0.12.4 ...
[info] Resolving org.scala-sbt#incremental-compiler;0.12.4 ...
[info] Resolving org.scala-sbt#logging;0.12.4 ...
[info] Resolving org.scala-sbt#process;0.12.4 ...
[info] Resolving org.scala-sbt#compile;0.12.4 ...
[info] Resolving org.scala-sbt#persist;0.12.4 ...
[info] Resolving org.scala-tools.sbinary#sbinary_2.9.0;0.4.0 ...
[info] Resolving org.scala-sbt#classfile;0.12.4 ...
[info] Resolving org.scala-sbt#compiler-ivy-integration;0.12.4 ...
[info] Resolving org.scala-sbt#ivy;0.12.4 ...
[info] Resolving org.apache.ivy#ivy;2.3.0-rc1 ...
[info] Resolving com.jcraft#jsch;0.1.46 ...
[info] Resolving commons-httpclient#commons-httpclient;3.1 ...
[info] Resolving commons-logging#commons-logging;1.0.4 ...
[info] Resolving commons-codec#commons-codec;1.2 ...
[info] Resolving org.scala-sbt#run;0.12.4 ...
[info] Resolving org.scala-sbt#task-system;0.12.4 ...
[info] Resolving org.scala-sbt#tasks;0.12.4 ...
[info] Resolving org.scala-sbt#tracking;0.12.4 ...
[info] Resolving org.scala-sbt#cache;0.12.4 ...
[info] Resolving org.scala-sbt#testing;0.12.4 ...
[info] Resolving org.scala-sbt#test-agent;0.12.4 ...
[info] Resolving org.scala-tools.testing#test-interface;0.5 ...
[info] Resolving org.scala-sbt#command;0.12.4 ...
[info] Resolving org.scala-sbt#compiler-interface;0.12.4 ...
[info] Resolving org.scala-sbt#precompiled-2_8_2;0.12.4 ...
[info] Resolving org.scala-sbt#precompiled-2_9_3;0.12.4 ...
[info] Resolving org.scala-sbt#precompiled-2_10_1;0.12.4 ...
[info] Done updating.
[info] Loading project definition from C:\programs\example\project
[info] Updating {file:/C:/programs/example/project/}default-116b7c...
[info] Resolving net.databinder#dispatch-http_2.9.2;0.8.8 ...
[info] Resolving net.databinder#dispatch-core_2.9.2;0.8.8 ...
[info] Resolving org.scala-lang#scala-library;2.9.2 ...
[info] Resolving org.apache.httpcomponents#httpclient;4.1.3 ...
[info] Resolving org.apache.httpcomponents#httpcore;4.1.4 ...
[info] Resolving commons-logging#commons-logging;1.1.1 ...
[info] Resolving commons-codec#commons-codec;1.4 ...
[info] Resolving net.databinder#dispatch-futures_2.9.2;0.8.8 ...
//some mor like the above
[info] Done updating.
[info] Compiling 8 Scala sources to C:\programs\example\project\target\scala-2.9
.2\sbt-0.12\classes...
[error] C:\programs\example\project\ProgFunBuild.scala:190: object Test is not a
value
[error] (argTask, currentProject, baseDirectory, handoutFiles, submitProject
Name, target, projectDetailsMap, compile in Test) map { (args, currentProject, b
asedir, filesFinder, submitProject, targetDir, detailsMap, _) =>
[error]
^
[error] C:\programs\example\project\ProgFunBuild.scala:56: object Test is not a
value
[error] (unmanagedSourceDirectories in Test) <<= (scalaSource in Test)(Seq(_
)),
[error] ^
[error] C:\programs\example\project\ProgFunBuild.scala:265: object Test is not a
value
[error] (unmanagedSources in Test) <<= (unmanagedSources in Test, scalaSourc
e in Test, projectDetailsMap, currentProject, gradingTestPackages) map { (source
s, srcTestScalaDir, detailsMap, projectName, gradingSrcs) =>
[error] ^
[error] C:\programs\example\project\ProgFunBuild.scala:265: reassignment to val
[error] (unmanagedSources in Test) <<= (unmanagedSources in Test, scalaSourc
e in Test, projectDetailsMap, currentProject, gradingTestPackages) map { (source
s, srcTestScalaDir, detailsMap, projectName, gradingSrcs) =>
[error] ^
[error] C:\programs\example\project\ProgFunBuild.scala:288: object Test is not a
value
[error] val setTestPropertiesHook = (test in Test) <<= (test in Test).dependsO
n(setTestProperties)
[error] ^
[error] C:\programs\example\project\ProgFunBuild.scala:288: reassignment to val
[error] val setTestPropertiesHook = (test in Test) <<= (test in Test).dependsO
n(setTestProperties)
[error] ^
[error] C:\programs\example\project\ProgFunBuild.scala:304: object Test is not a
value
[error] compile in Test,
[error] ^
[error] C:\programs\example\project\ProgFunBuild.scala:512: object Test is not a
value
[error] val readTestCompileLog = (compile in Test) <<= (compile in Test) mapR
handleFailure(compileTestFailed)
[error] ^
[error] C:\programs\example\project\ProgFunBuild.scala:512: reassignment to val
[error] val readTestCompileLog = (compile in Test) <<= (compile in Test) mapR
handleFailure(compileTestFailed)
[error] ^
[error] C:\programs\example\project\ProgFunBuild.scala:553: object Test is not a
value
[error] (sourceDirectory in Test) <<= (sourceDirectory in (assignmentProject
, Test))
[error] ^
[error] C:\programs\example\project\ProgFunBuild.scala:553: reassignment to val
[error] (sourceDirectory in Test) <<= (sourceDirectory in (assignmentProject
, Test))
[error] ^
[error] C:\programs\example\project\ProgFunBuild.scala:561: object Test is not a
value
[error] (unmanagedSources in Test) <<= (unmanagedSources in Test, scalaSourc
e in (assignmentProject, Test), gradingTestPackages in assignmentProject, gradeP
rojectDetails) map { (sources, testSrcScalaDir, gradingSrcs, project) =>
[error] ^
[error] C:\programs\example\project\ProgFunBuild.scala:561: reassignment to val
[error] (unmanagedSources in Test) <<= (unmanagedSources in Test, scalaSourc
e in (assignmentProject, Test), gradingTestPackages in assignmentProject, gradeP
rojectDetails) map { (sources, testSrcScalaDir, gradingSrcs, project) =>
[error] ^
[error] C:\programs\example\project\ProgFunBuild.scala:570: object Test is not a
value
[error] compile in Test,
[error] ^
[error] 14 errors found
[error] (compile:compile) Compilation failed
Project loading failed: (r)etry, (q)uit, (l)ast, or (i)gnore?
UPDATE
When I press l at the commmand prompt, the error log is as follows:
[info] Loading project definition from C:\A\example\project\project
[debug] Running task... Cancelable: false, check cycles: false
[debug]
[debug] Initial source changes:
[debug] removed:Set()
[debug] added: Set()
[debug] modified: Set()
[debug] Removed products: Set()
[debug] Modified external sources: Set()
[debug] Modified binary dependencies: Set()
[debug] Initial directly invalidated sources: Set()
[debug]
[debug] Sources indirectly invalidated by:
[debug] product: Set()
[debug] binary dep: Set()
[debug] external source: Set()
[debug] All initially invalidated sources: Set()
[debug] Copy resource mappings:
[debug]
[info] Loading project definition from C:\A\example\project
[debug] Running task... Cancelable: false, check cycles: false
[debug]
[debug] Initial source changes:
[debug] removed:Set()
[debug] added: Set(C:\A\example\project\Settings.scala, C:\A\example\pro
ject\StyleChecker.scala, C:\A\example\project\GradingFeedback.scala, C:\A\exampl
e\project\CourseraHttp.scala, C:\A\example\project\ProgFunBuild.scala, C:\A\exam
ple\project\ScalaTestRunner.scala, C:\A\example\project\RichJsValue.scala, C:\A\
example\project\RecordingLogger.scala)
[debug] modified: Set()
[debug] Removed products: Set()
[debug] Modified external sources: Set()
[debug] Modified binary dependencies: Set()
[debug] Initial directly invalidated sources: Set(C:\A\example\project\Settings.
scala, C:\A\example\project\StyleChecker.scala, C:\A\example\project\GradingFeed
back.scala, C:\A\example\project\CourseraHttp.scala, C:\A\example\project\ProgFu
nBuild.scala, C:\A\example\project\ScalaTestRunner.scala, C:\A\example\project\R
ichJsValue.scala, C:\A\example\project\RecordingLogger.scala)
[debug]
[debug] Sources indirectly invalidated by:
[debug] product: Set()
[debug] binary dep: Set()
[debug] external source: Set()
[debug] All initially invalidated sources: Set(C:\A\example\project\Settings.sca
la, C:\A\example\project\StyleChecker.scala, C:\A\example\project\GradingFeedbac
k.scala, C:\A\example\project\CourseraHttp.scala, C:\A\example\project\ProgFunBu
ild.scala, C:\A\example\project\ScalaTestRunner.scala, C:\A\example\project\Rich
JsValue.scala, C:\A\example\project\RecordingLogger.scala)
[debug] Recompiling all 8 sources: invalidated sources (8) exceeded 50.0% of all
sources
[info] Compiling 8 Scala sources to C:\A\example\project\target\scala-2.9.2\sbt-
0.12\classes...
[debug] Getting compiler-interface from component compiler for Scala 2.9.2
[debug] Getting compiler-interface from component compiler for Scala 2.9.2
[debug] Running cached compiler b206e9, interfacing (CompilerInterface) with Sca
la compiler version 2.9.2
[debug] Calling Scala compiler with arguments (CompilerInterface):
[debug] -deprecation
[debug] -d
[debug] C:\A\example\project\target\scala-2.9.2\sbt-0.12\classes
[debug] -bootclasspath
[debug] C:\Program Files\Java\jre6\lib\resources.jar;C:\Program Files\Ja
va\jre6\lib\rt.jar;C:\Program Files\Java\jre6\lib\sunrsasign.jar;C:\Program File
s\Java\jre6\lib\jsse.jar;C:\Program Files\Java\jre6\lib\jce.jar;C:\Program Files
\Java\jre6\lib\charsets.jar;C:\Program Files\Java\jre6\lib\modules\jdk.boot.jar;
C:\Program Files\Java\jre6\classes;C:\Documents and Settings\User\.sbt\boot\scal
a-2.9.2\lib\scala-library.jar
[debug] -classpath
[debug] C:\A\example\project\target\scala-2.9.2\sbt-0.12\classes;C:\Docu
ments and Settings\User\.ivy2\cache\net.databinder\dispatch-http_2.9.2\jars\disp
atch-http_2.9.2-0.8.8.jar;C:\Documents and Settings\User\.ivy2\cache\net.databin
der\dispatch-core_2.9.2\jars\dispatch-core_2.9.2-0.8.8.jar;C:\Documents and Sett
ings\User\.ivy2\cache\org.apache.httpcomponents\httpclient\jars\httpclient-4.1.3
.jar;C:\Documents and Settings\User\.ivy2\cache\org.apache.httpcomponents\httpco
re\jars\httpcore-4.1.4.jar;C:\Documents and Settings\User\.ivy2\cache\commons-lo
gging\commons-logging\jars\commons-logging-1.1.1.jar;C:\Documents and Settings\U
ser\.ivy2\cache\commons-codec\commons-codec\jars\commons-codec-1.4.jar;C:\Docume
nts and Settings\User\.ivy2\cache\net.databinder\dispatch-futures_2.9.2\jars\dis
patch-futures_2.9.2-0.8.8.jar;C:\Documents and Settings\User\.ivy2\cache\org.sca
lastyle\scalastyle_2.9.1\jars\scalastyle_2.9.1-0.1.3-SNAPSHOT.jar;C:\Documents a
nd Settings\User\.ivy2\cache\org.scalariform\scalariform_2.9.1\jars\scalariform_
2.9.1-0.1.1.jar;C:\Documents and Settings\User\.ivy2\cache\com.github.scopt\scop
t_2.9.1\jars\scopt_2.9.1-2.0.0.jar;C:\Documents and Settings\User\.ivy2\cache\cc
.spray\spray-json_2.9.2\jars\spray-json_2.9.2-1.1.1.jar;C:\Documents and Setting
s\User\.ivy2\cache\org.parboiled\parboiled-scala\jars\parboiled-scala-1.0.2.jar;
C:\Documents and Settings\User\.ivy2\cache\org.parboiled\parboiled-core\jars\par
boiled-core-1.0.2.jar;C:\Documents and Settings\User\.ivy2\cache\org.scalatest\s
calatest_2.9.2\jars\scalatest_2.9.2-1.9.1.jar;C:\Documents and Settings\User\.iv
y2\cache\org.apache.commons\commons-lang3\jars\commons-lang3-3.1.jar;C:\Document
s and Settings\User\.ivy2\cache\scala_2.9.2\sbt_0.12\com.typesafe.sbteclipse\sbt
eclipse-plugin\jars\sbteclipse-plugin-2.1.0.jar;C:\Documents and Settings\User\.
ivy2\cache\scala_2.9.2\sbt_0.12\com.typesafe.sbteclipse\sbteclipse-core\jars\sbt
eclipse-core-2.1.0.jar;C:\Documents and Settings\User\.ivy2\cache\org.scalaz\sca
laz-core_2.9.2\jars\scalaz-core_2.9.2-6.0.4.jar;C:\Documents and Settings\User\.
ivy2\cache\org.scala-sbt\sbt\jars\sbt-0.12.4.jar;C:\Documents and Settings\User\
.ivy2\cache\org.scala-sbt\main\jars\main-0.12.4.jar;C:\Documents and Settings\Us
er\.ivy2\cache\org.scala-sbt\actions\jars\actions-0.12.4.jar;C:\Documents and Se
ttings\User\.ivy2\cache\org.scala-sbt\classpath\jars\classpath-0.12.4.jar;C:\Doc
uments and Settings\User\.ivy2\cache\org.scala-sbt\launcher-interface\jars\launc
her-interface-0.12.4.jar;C:\Documents and Settings\User\.ivy2\cache\org.scala-sb
t\interface\jars\interface-0.12.4.jar;C:\Documents and Settings\User\.ivy2\cache
\org.scala-sbt\io\jars\io-0.12.4.jar;C:\Documents and Settings\User\.ivy2\cache\
org.scala-sbt\control\jars\control-0.12.4.jar;C:\Documents and Settings\User\.sb
t\boot\scala-2.9.2\lib\scala-compiler.jar;C:\Documents and Settings\User\.ivy2\c
ache\org.scala-sbt\completion\jars\completion-0.12.4.jar;C:\Documents and Settin
gs\User\.ivy2\cache\org.scala-sbt\collections\jars\collections-0.12.4.jar;C:\Doc
uments and Settings\User\.ivy2\cache\jline\jline\jars\jline-1.0.jar;C:\Documents
and Settings\User\.ivy2\cache\org.scala-sbt\api\jars\api-0.12.4.jar;C:\Document
s and Settings\User\.ivy2\cache\org.scala-sbt\compiler-integration\jars\compiler
-integration-0.12.4.jar;C:\Documents and Settings\User\.ivy2\cache\org.scala-sbt
\incremental-compiler\jars\incremental-compiler-0.12.4.jar;C:\Documents and Sett
ings\User\.ivy2\cache\org.scala-sbt\logging\jars\logging-0.12.4.jar;C:\Documents
and Settings\User\.ivy2\cache\org.scala-sbt\process\jars\process-0.12.4.jar;C:\
Documents and Settings\User\.ivy2\cache\org.scala-sbt\compile\jars\compile-0.12.
4.jar;C:\Documents and Settings\User\.ivy2\cache\org.scala-sbt\persist\jars\pers
ist-0.12.4.jar;C:\Documents and Settings\User\.ivy2\cache\org.scala-tools.sbinar
y\sbinary_2.9.0\jars\sbinary_2.9.0-0.4.0.jar;C:\Documents and Settings\User\.ivy
2\cache\org.scala-sbt\classfile\jars\classfile-0.12.4.jar;C:\Documents and Setti
ngs\User\.ivy2\cache\org.scala-sbt\compiler-ivy-integration\jars\compiler-ivy-in
tegration-0.12.4.jar;C:\Documents and Settings\User\.ivy2\cache\org.scala-sbt\iv
y\jars\ivy-0.12.4.jar;C:\Documents and Settings\User\.ivy2\cache\org.apache.ivy\
ivy\jars\ivy-2.3.0-rc1.jar;C:\Documents and Settings\User\.ivy2\cache\com.jcraft
\jsch\jars\jsch-0.1.46.jar;C:\Documents and Settings\User\.ivy2\cache\commons-ht
tpclient\commons-httpclient\jars\commons-httpclient-3.1.jar;C:\Documents and Set
tings\User\.ivy2\cache\org.scala-sbt\run\jars\run-0.12.4.jar;C:\Documents and Se
ttings\User\.ivy2\cache\org.scala-sbt\task-system\jars\task-system-0.12.4.jar;C:
\Documents and Settings\User\.ivy2\cache\org.scala-sbt\tasks\jars\tasks-0.12.4.j
ar;C:\Documents and Settings\User\.ivy2\cache\org.scala-sbt\tracking\jars\tracki
ng-0.12.4.jar;C:\Documents and Settings\User\.ivy2\cache\org.scala-sbt\cache\jar
s\cache-0.12.4.jar;C:\Documents and Settings\User\.ivy2\cache\org.scala-sbt\test
ing\jars\testing-0.12.4.jar;C:\Documents and Settings\User\.ivy2\cache\org.scala
-sbt\test-agent\jars\test-agent-0.12.4.jar;C:\Documents and Settings\User\.ivy2\
cache\org.scala-tools.testing\test-interface\jars\test-interface-0.5.jar;C:\Docu
ments and Settings\User\.ivy2\cache\org.scala-sbt\command\jars\command-0.12.4.ja
r;C:\Documents and Settings\User\.ivy2\cache\org.scala-sbt\compiler-interface\ja
rs\compiler-interface-bin-0.12.4.jar;C:\Documents and Settings\User\.ivy2\cache\
org.scala-sbt\compiler-interface\jars\compiler-interface-src-0.12.4.jar;C:\Docum
ents and Settings\User\.ivy2\cache\org.scala-sbt\precompiled-2_8_2\jars\compiler
-interface-bin-0.12.4.jar;C:\Documents and Settings\User\.ivy2\cache\org.scala-s
bt\precompiled-2_9_3\jars\compiler-interface-bin-0.12.4.jar;C:\Documents and Set
tings\User\.ivy2\cache\org.scala-sbt\precompiled-2_10_1\jars\compiler-interface-
bin-0.12.4.jar
A pragmatic "solution" that may help to get you running for now: replace all those occurrences of in Test in ProgFunBuild.scala with in sbt.Test and then try again.
Still, something is broken in your environment.
What happens when you run sbt is that it is not compiling your sources but the build definition of sbt. The problem seems to be that Test is not found. I can only vaguely speculate...
Did you touch that file ProgFunBuild in any way?
Did you accidentally place your normal source code in the project sub directory? Perhaps you did that and have an object Test defined in there which shadows sbt's Test scope?
From what directory do you run sbt? You should be in the root folder of the example, not inside the project directory.
You should have the following layout:
project/ProgFunBuild.scala
project/build.properties (optional)
build.sbt (optional)
src/main/scala/... (your sources in there)