How can I migrate from avro4s 3.0.4 to 4.0.0-RC2?
I have the following compiling errors:
[error] /Users/nicolae.marasoiu/proj/data-availability-global-topic-conveyor/src/main/scala/com/ovoenergy/globaltopics/serdes/AvroFormatImplicits.scala:8:15: value const is not a member of object com.sksamuel.avro4s.SchemaFor
[error] SchemaFor.const(new Schema.Parser().parse(getClass.getResourceAsStream(hasSchema.resourcePath)))
[error] ^
[error] /Users/nicolae.marasoiu/proj/data-availability-global-topic-conveyor/src/main/scala/com/ovoenergy/globaltopics/serdes/AvroFormatImplicits.scala:11:26: not enough arguments for method apply: (implicit evidence$1: com.sksamuel.avro4s.Encoder[T], implicit evidence$2: com.sksamuel.avro4s.Decoder[T])com.sksamuel.avro4s.RecordFormat[T] in object RecordFormat.
[error] Unspecified value parameter evidence$2.
[error] RecordFormat.apply[T](AvroSchema[T](readSchema))
[error] ^
[error] /Users/nicolae.marasoiu/proj/data-availability-global-topic-conveyor/src/main/scala/com/ovoenergy/globaltopics/serdes/SerdeProvider.scala:29:37: org.apache.avro.Schema does not take parameters
[error] val schema = SchemaFor[T].schema(DefaultFieldMapper)
[error] ^
[error] /Users/nicolae.marasoiu/proj/data-availability-global-topic-conveyor/src/main/scala/com/ovoenergy/globaltopics/serdes/SerdeProvider.scala:37:70: no arguments allowed for nullary method build: ()com.sksamuel.avro4s.AvroOutputStream[T]
[error] val os = AvroOutputStream.binary[T].to(output).build(schema)
[error] ^
[error] four errors found
[error] (Compile / compileIncremental) Compilation failed
[error] Total time: 17 s, completed 28-Jul-2020 19:42:21
[IJ]sbt:global-topic-conveyor>
First of all, you can always follow the changes at avro4s github repository. Furthermore, you can see the specific changes made between the versions you specified.
You did not attach any source code, so I'll try to address your failures as I can understand them from your failures.
SchemaFor.const was removed at version 3.0.5 . You can see that in the diff between the versions. Assuming you had Schema s, and you initiated so far the SchemaFor.const(s), Now you'll need to initiate it: SchemaFor(s). The default fieldMapper will be used. You can see it here.
RecordFormat does not take arguments anymore. Therefore assuming you had a type T, that was applied at RecordFormat.apply[T](AvroSchema[T](readSchema)), you need to change it into: RecordFormat[T].
Very similar to number 1. SchemaFor.schema was removed. Instead, you can just do: SchemaFor[T].schema. Schema does not take parameters anymore, so you cannot call it with parentheses.
AvroOutputStream.build used to take a schema as a parameter. It doesn't take it anymore. You need to change AvroOutputStream.binary[T].to(output).build(schema) into: AvroOutputStream.binary[T].to(output).build()
I am working on Rocket Chip Generator, which is a SoC written in Chisel.
My objective is to extract the Floating-Point Unit, in order to synthesize it and study its power consumption/area ...etc. separately from the rest of the SoC.
So I cloned the project and tried to create a dedicated TestHarness where I only instanciate an FPU. This is an example code of what I did (very similar to TestHarness
package freechips.rocketchip.system
import Chisel._
import freechips.rocketchip.config.Parameters
import freechips.rocketchip.devices.debug.Debug
import freechips.rocketchip.diplomacy.LazyModule
import freechips.rocketchip.rocket._
// importing FPU class
import freechips.rocketchip.tile._
class FPUTestHarness() (implicit p: Parameters) extends Module {
val io = new Bundle {
val dummyIn = Bool(INPUT)
val dummyOut = Bool(OUTPUT)
}
// The parameters of the FPU
val fpu_params = FPUParams(
fLen = 32 // default is 64
divSqrt = false // default is true
// sfmaLatency = 3, // default
// dfmaLatency = 4 // default
)
val fpu = Module(new FPU(fpu_params)(p))
}
The problem is that the implicit parameter p (of type Parameters) doesn't contain something called TileKey, which is needed I don't know where. Apparently the implicit parameter p, passed to FPUTestHarness doesn't contain that information. How can I resolve that? Is it possible to hardcode that somewhere for example?
rocket-chip-new/vsim$ make verilog
mkdir -p /home/mylab/rocket-chip/vsim/generated-src/
cd /home/mylab/rocket-chip && java -Xmx2G -Xss8M -XX:MaxPermSize=256M -jar /home/mylab/rocket-chip/sbt-launch.jar "runMain freechips.rocketchip.system.Generator /home/mylab/rocket-chip/vsim/generated-src freechips.rocketchip.system FPUTestHarness freechips.rocketchip.system DefaultConfig"
OpenJDK 64-Bit Server VM warning: Ignoring option MaxPermSize; support was removed in 8.0
[info] Loading settings for project rocket-chip-new-build from plugins.sbt ...
[info] Loading project definition from /home/mylab/rocket-chip/project
[info] Loading settings for project rocketchip from build.sbt ...
[info] Loading settings for project hardfloat from build.sbt ...
[info] Loading settings for project chisel from build.sbt ...
Using addons:
[info] Set current project to rocketchip (in build file:/home/mylab/rocket-chip/)
[info] Compiling 1 Scala source to /home/mylab/rocket-chip/target/scala-2.12/classes ...
[warn] there were two feature warnings; re-run with -feature for details
[warn] one warning found
[info] Done compiling.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.protobuf.UnsafeUtil (file:/home/aitsaidn/.sbt/boot/scala-2.12.7/org.scala-sbt/sbt/1.2.7/protobuf-java-3.3.1.jar) to field java.nio.Buffer.address
WARNING: Please consider reporting this to the maintainers of com.google.protobuf.UnsafeUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[warn] Multiple main classes detected. Run 'show discoveredMainClasses' to see the list
[info] Packaging /home/mylab/rocket-chip/target/scala-2.12/rocketchip_2.12-1.2-SNAPSHOT.jar ...
[info] Done packaging.
[info] Running freechips.rocketchip.system.Generator /home/mylab/rocket-chip/vsim/generated-src freechips.rocketchip.system FPUTestHarness freechips.rocketchip.system DefaultConfig
[info] [0.003] Elaborating design...
List(RocketTileParams(RocketCoreParams(0,true,false,true,true,false,true,false,0,1,false,8,0,true,false,true,0,Some(0),true,true,false,false,false,0,538447876,Some(MulDivParams(8,1,true,true,1)),Some(FPUParams(64,true,3,4))),Some(ICacheParams(64,4,64,32,0,None,None,None,false,64,2,4)),Some(DCacheParams(64,4,64,32,None,None,1,0,17,16,1,64,false,false,false,false,None)),Some(BTBParams(28,14,6,6,Some(BHTParams(512,1,8,3)),false)),0,Some(tile),0,None,None,false))
[error] (run-main-0) java.lang.reflect.InvocationTargetException
[error] java.lang.reflect.InvocationTargetException
[error] at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
[error] at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
[error] at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
[error] at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
[error] at freechips.rocketchip.util.HasGeneratorUtilities.$anonfun$elaborate$1(GeneratorUtils.scala:54)
[error] at chisel3.core.Module$.do_apply(Module.scala:52)
[error] at chisel3.Driver$.$anonfun$elaborate$1(Driver.scala:95)
[error] at chisel3.internal.Builder$.$anonfun$build$2(Builder.scala:343)
[error] at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
[error] at chisel3.internal.Builder$.$anonfun$build$1(Builder.scala:341)
[error] at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
[error] at chisel3.internal.Builder$.build(Builder.scala:341)
[error] at chisel3.Driver$.elaborate(Driver.scala:95)
[error] at freechips.rocketchip.util.HasGeneratorUtilities.elaborate(GeneratorUtils.scala:59)
[error] Caused by: java.lang.IllegalArgumentException: requirement failed: Key TileKey is not defined in Parameters
[error] at scala.Predef$.require(Predef.scala:277)
[error] at freechips.rocketchip.config.View.apply(Config.scala:15)
[error] at freechips.rocketchip.config.View.apply(Config.scala:12)
[error] at freechips.rocketchip.tile.HasNonDiplomaticTileParameters.tileParams(BaseTile.scala:37)
[error] at freechips.rocketchip.tile.HasNonDiplomaticTileParameters.tileParams$(BaseTile.scala:37)
[error] at freechips.rocketchip.tile.CoreModule.tileParams(Core.scala:90)
[error] at freechips.rocketchip.tile.HasCoreParameters.$init$(Core.scala:51)
[error] at freechips.rocketchip.tile.CoreModule.<init>(Core.scala:90)
[error] at freechips.rocketchip.tile.FPUModule.<init>(FPU.scala:378)
[error] at freechips.rocketchip.tile.FPU.<init>(FPU.scala:661)
[error] at freechips.rocketchip.system.FPUTestHarness.$anonfun$fpu$1(FPUTestHarness.scala:71)
[error] at chisel3.core.Module$.do_apply(Module.scala:52)
[error] at freechips.rocketchip.system.FPUTestHarness.<init>(FPUTestHarness.scala:71)
[error] at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
[error] Nonzero exit code: 1
[error] (Compile / runMain) Nonzero exit code: 1
[error] Total time: 6 s, completed Jul 25, 2019, 7:59:01 PM
/home/mylab/rocket-chip/vsim/Makefrag-verilog:12: recipe for target '/home/mylab/rocket-chip/vsim/generated-src/freechips.rocketchip.system.DefaultConfig.fir' failed
make: *** [/home/mylab/rocket-chip/vsim/generated-src/freechips.rocketchip.system.DefaultConfig.fir] Error 1
It seems that you use the config as "DefaultConfig". Error log states that you did not specify TileKey parameter in this config. Adding TileKey parameter will resolve the issue. DefaultConfig is defined in rocket-chip/src/main/scala/system/Configs.scala
I have a scala Spark job. I want to compress the output using Gzip and then saveToTextFile.
compressedEvents.saveAsTextFile(outputDirectory, org.apache.hadoop.io.compress.GzipCodec)
But I get the following error:
[error] /var/lib/jenkins/workspace/producer-data-test/producer-data-test-build/src/main/scala/IpFromLogs.scala:46: object org.apache.hadoop.io.compress.GzipCodec is not a value
[error] compressedEvents.saveAsTextFile(outputDirectory, org.apache.hadoop.io.compress.GzipCodec)
[error] ^
[error] one error found
[error] (compile:compileIncremental) Compilation failed
I tried different variations of the same but none of them work. Please help!
Correct way of saving is
compressedEvents.saveAsTextFile(outputDirectory, classOf[GzipCodec])
Or
before you save set the configuration as
sc.hadoopConfiguration.setClass(FileOutputFormat.COMPRESS_CODEC, classOf[GzipCodec], classOf[CompressionCodec])
And save it as
compressedEvents.saveAsTextFile(outputDirectory)
After adding the following method to map case classes to Map or js.Dictionary - and I've tried now five or six variants of the following - my code compiles fine and without warnings, but then hits errors at the fastOptJS sjs linking stage.
The method
def ccToMap(cc: AnyRef) =
(Map[String, Any]() /: cc.getClass.getDeclaredFields) {
(a, f) =>
f.setAccessible(true)
a + (f.getName -> f.get(cc))
}
Note that all the variants I have tried do the same thing in a slightly different manner.
The error
[info] Fast optimizing /Users/justin/Desktop/arete/jt/client/target/scala-2.11/client-fastopt.js
[error] Referring to non-existent class java.lang.reflect.Field
[error] called from com.jshin47.jtdc.client.module.visualization.DiodeStateVizC$.ccToMap(java.lang.Object)scala.collection.immutable.Map
[error] called from com.jshin47.jtdc.client.module.visualization.DiodeStateVizC$.<init>()
[error] called from com.jshin47.jtdc.client.module.landing.LandingLocC$$anonfun$2.apply(japgolly.scalajs.react.extra.router.RouterCtl)japgolly.scalajs.react.ReactElement
[error] called from com.jshin47.jtdc.client.module.landing.LandingLocC$$anonfun$2.apply(java.lang.Object)java.lang.Object
[error] called from scala.collection.LinearSeqOptimized$class.foreach(scala.collection.LinearSeqOptimized,scala.Function1)scala.Unit
[error] called from scala.collection.mutable.MutableList.foreach(scala.Function1)scala.Unit
[error] called from scala.collection.TraversableLike$WithFilter.map(scala.Function1,scala.collection.generic.CanBuildFrom)java.lang.Object
[error] called from scala.collection.immutable.Stream$StreamWithFilter.map(scala.Function1,scala.collection.generic.CanBuildFrom)java.lang.Object
[error] called from org.scalajs.testinterface.internal.Slave.org$scalajs$testinterface$internal$Slave$$execute(scala.scalajs.js.Dynamic)scala.Unit
[error] called from org.scalajs.testinterface.internal.Slave.handleMsgImpl(java.lang.String,scala.Function0)scala.Unit
[error] called from org.scalajs.testinterface.internal.BridgeBase.handleMsg(java.lang.String)scala.Unit
[error] called from org.scalajs.testinterface.internal.BridgeBase.$$anonfun$1(java.lang.String)scala.Unit
[error] called from org.scalajs.testinterface.internal.BridgeBase.init()scala.Unit
[error] called from org.scalajs.testinterface.internal.BridgeBase.$$js$exported$meth$init()java.lang.Object
[error] called from org.scalajs.testinterface.internal.BridgeBase.init
[error] exported to JavaScript with #JSExport
[error] involving instantiated classes:
[error] com.jshin47.jtdc.client.module.visualization.DiodeStateVizC$
[error] com.jshin47.jtdc.client.module.landing.LandingLocC$$anonfun$2
[error] scala.collection.mutable.Queue
[error] scala.collection.mutable.MutableList
[error] scala.collection.TraversableLike$WithFilter
[error] scala.collection.immutable.Stream$StreamWithFilter
[error] org.scalajs.testinterface.internal.Slave
[error] org.scalajs.testinterface.internal.Master
[error] Referring to non-existent method java.lang.Class.getDeclaredFields() [java.lang.reflect.Field
[error] called from com.jshin47.jtdc.client.module.visualization.DiodeStateVizC$.ccToMap(java.lang.Object)scala.collection.immutable.Map
[error] called from com.jshin47.jtdc.client.module.visualization.DiodeStateVizC$.<init>()
[error] called from com.jshin47.jtdc.client.module.landing.LandingLocC$$anonfun$2.apply(japgolly.scalajs.react.extra.router.RouterCtl)japgolly.scalajs.react.ReactElement
[error] called from com.jshin47.jtdc.client.module.landing.LandingLocC$$anonfun$2.apply(java.lang.Object)java.lang.Object
[error] called from scala.collection.LinearSeqOptimized$class.foreach(scala.collection.LinearSeqOptimized,scala.Function1)scala.Unit
[error] called from scala.collection.mutable.MutableList.foreach(scala.Function1)scala.Unit
[error] called from scala.collection.TraversableLike$WithFilter.map(scala.Function1,scala.collection.generic.CanBuildFrom)java.lang.Object
[error] called from scala.collection.immutable.Stream$StreamWithFilter.map(scala.Function1,scala.collection.generic.CanBuildFrom)java.lang.Object
[error] called from org.scalajs.testinterface.internal.Slave.org$scalajs$testinterface$internal$Slave$$execute(scala.scalajs.js.Dynamic)scala.Unit
[error] called from org.scalajs.testinterface.internal.Slave.handleMsgImpl(java.lang.String,scala.Function0)scala.Unit
[error] called from org.scalajs.testinterface.internal.BridgeBase.handleMsg(java.lang.String)scala.Unit
[error] called from org.scalajs.testinterface.internal.BridgeBase.$$anonfun$1(java.lang.String)scala.Unit
[error] called from org.scalajs.testinterface.internal.BridgeBase.init()scala.Unit
[error] called from org.scalajs.testinterface.internal.BridgeBase.$$js$exported$meth$init()java.lang.Object
[error] called from org.scalajs.testinterface.internal.BridgeBase.init
[error] exported to JavaScript with #JSExport
[error] involving instantiated classes:
[error] com.jshin47.jtdc.client.module.visualization.DiodeStateVizC$
[error] com.jshin47.jtdc.client.module.landing.LandingLocC$$anonfun$2
[error] scala.collection.mutable.Queue
[error] scala.collection.mutable.MutableList
[error] scala.collection.TraversableLike$WithFilter
[error] scala.collection.immutable.Stream$StreamWithFilter
[error] org.scalajs.testinterface.internal.Slave
[error] org.scalajs.testinterface.internal.Master
[trace] Stack trace suppressed: run last client/compile:fastOptJS for the full output.
[error] (client/compile:fastOptJS) There were linking errors
[error] Total time: 36 s, completed May 10, 2016 2:01:07 AM
What I've tried
Being unfamiliar with the details of the linker I am only able to try some "obvious" diagnostics:
Whether I call the method or not, this error is thrown (it doesn't have to be in the code path, so this is getting thrown when the method itself is linked)
The Map type itself works just fine as an argument to the function where I am trying to invoke this
I know to a certainty that iff this method (or similar) is present, then I get the above linker error. (Without, no error.)
Any tips on how to proceed, debug, etc are appreciated.
Alternatively, any tips on how I may convert a case class to a Map without the above-style (reflection-based) function, please let me know
You simply can't use Java reflection in Scala.js:
Java reflection and, a fortiori, Scala reflection, are not supported. There is limited support for java.lang.Class, e.g., obj.getClass.getName will work for any Scala.js object (not for objects that come from JavaScript interop).
Use macros for this instead. See e.g. Scala Macros: Making a Map out of fields of a class in Scala
I'm getting this exception when running the scalatra specs2 example from the scalatra docs:
ThrowableException: org.eclipse.jetty.http.HttpGenerator.flushBuffer()I (FutureTask.java:138)
Here is the test code (starting on line 5, skipping imports):
class MyAppTest extends MutableScalatraSpec {
addServlet(classOf[MyApp], "/*")
"GET / on AdminApp" should {
"return status 200" in {
get("/") {
status must_== 200
}
}
}
}
Here is the app definition:
class MyApp extends ScalatraServlet {
get("/") {
"aloha"
}
}
I'm using scalatra-specs2 2.0.4 and scala 2.9.1. I'm running an embedded jetty server using xsbt-web-plugin 0.2.10 with sbt 0.11.2. The test was executed using sbt test.
Here is the full trace:
[info] GET / on AdminApp should
[error] ! Fragment evaluation error
[error] ThrowableException: org.eclipse.jetty.http.HttpGenerator.flushBuffer()I (FutureTask.java:138)
[error] org.eclipse.jetty.testing.HttpTester.generate(HttpTester.java:225)
[error] org.scalatra.test.ScalatraTests$class.submit(ScalatraTests.scala:46)
[error] com.example.MyAppTest.submit(MyAppTest.scala:5)
[error] org.scalatra.test.ScalatraTests$class.submit(ScalatraTests.scala:71)
[error] com.example.MyAppTest.submit(MyAppTest.scala:5)
[error] org.scalatra.test.ScalatraTests$class.get(ScalatraTests.scala:127)
[error] com.example.MyAppTest.get(MyAppTest.scala:5)
[error] com.example.MyAppTest$$anonfun$1$$anonfun$apply$3.apply(MyAppTest.scala:10)
[error] com.example.MyAppTest$$anonfun$1$$anonfun$apply$3.apply(MyAppTest.scala:10)
[error] org.eclipse.jetty.http.HttpGenerator.flushBuffer()I
[error] org.eclipse.jetty.testing.HttpTester.generate(HttpTester.java:225)
[error] org.scalatra.test.ScalatraTests$class.submit(ScalatraTests.scala:46)
[error] com.example.MyAppTest.submit(MyAppTest.scala:5)
[error] org.scalatra.test.ScalatraTests$class.submit(ScalatraTests.scala:71)
[error] com.example.MyAppTest.submit(MyAppTest.scala:5)
[error] org.scalatra.test.ScalatraTests$class.get(ScalatraTests.scala:127)
[error] com.example.MyAppTest.get(MyAppTest.scala:5)
[error] com.example.MyAppTest$$anonfun$1$$anonfun$apply$3.apply(MyAppTest.scala:10)
[error] com.example.MyAppTest$$anonfun$1$$anonfun$apply$3.apply(MyAppTest.scala:10)
This is the only search result that has turned up so far:
Fragment Evaluation Error.
Can someone point me in the right direction?
Thanks,
-f
Still unsure of the root cause, but the test executes successfully after rolling jetty-webapp back from 8.0.3.v20111011 to 7.6.0.v20120127.
You probably have a conflict in your dependencies, more specifically with the Jetty library version. Since the "flush" method on HttpGenerator has changed between Jetty 6 and Jetty 7, you might be getting a "NoSuchMethodFoundError" which explains the strange "org.eclipse.jetty.http.HttpGenerator.flushBuffer()I" signature in the exception message.
This also explains why you get a "fragment evaluation error" and not a regular failure as explained in the link you mentioned.
If you give a go at the latest specs2-1.10-SNAPSHOT, you will get a better message for "fragment evaluation error" showing 'NoSuchMethodError' when that happens. This will help you diagnosing the issue faster.