I am trying to use Record class to generate RegInit. This can help to dynamically create RegInit with other information in Module.
I am trying to use Record class to generate RegInit.
Here are my codes.
class MyReg(reg_info: List[(String, UInt)]) extends Record {
override def elements: ListMap[String, UInt] = ListMap(
reg_info.map(
x =>
x._1 -> x._2
):_*
)
override def cloneType: MyReg.this.type = new MyReg(reg_info).asInstanceOf[this.type]
}
class reg_test(reg_list: List[(String, String, List[(String, Int)])]) extends Module with RegSugar {
val io = IO(new Bundle() {
val a = Input(UInt(16.W))
val b = Input(UInt(16.W))
val d = Output(UInt(16.W))
val e = Output(UInt(16.W))
})
val myBundle = new Bundle() {
val a = UInt(16.W)
val b = UInt(16.W)
}
val reg_init_bundle = RegInit(myBundle, 0.U.asTypeOf(myBundle))
println(reg_init_bundle)
reg_init_bundle.a := io.a
reg_init_bundle.b := io.b
val list_b = List(("a", UInt(16.W)), ("b", UInt(16.W)))
val myRecord = new MyReg(list_b)
val reg_init = RegInit(myRecord, 0.U.asTypeOf(myRecord))
reg_init.elements("a") := io.a
reg_init.elements("b") := io.b
io.d := reg_init.elements("a") + reg_init.elements("b")
io.e := reg_init_bundle.a + reg_init_bundle.b
}
In my sight, Bundle extends Record and if these code work on Bundle, they should also work on Record.
However, the error is reported on the line:
val reg_init = RegInit(myRecord, 0.U.asTypeOf(myRecord))
Exception in thread "main" chisel3.package$RebindingException: Attempted reassignment of binding to reg_test.reg_init_?.a: Wire[UInt<16>], from: ChildBinding(reg_test.reg_init_?: Reg[MyReg])
at ... ()
at implicit_test.reg_test.$anonfun$reg_init$2(reg_test.scala:61)
at chisel3.internal.prefix$.apply(prefix.scala:48)
at implicit_test.reg_test.$anonfun$reg_init$1(reg_test.scala:61)
at chisel3.internal.plugin.package$.autoNameRecursively(package.scala:33)
at implicit_test.reg_test.<init>(reg_test.scala:61)
at implicit_test.reg_test$.$anonfun$verilogString$1(reg_test.scala:77)
at ... ()
at ... (Stack trace trimmed to user code only. Rerun with --full-stacktrace to see the full stack trace)
It seems that these codes can successfully generate RegInit with Bundle but failed to generate RegInit with Record.
Is that a way to generate RegInit with Record? or there is a compiler error?
https://github.com/chipsalliance/chisel3/issues/2998. Thanks, guys. Here is the explanation for this Question. Case closed.
Related
I'm trying to name my source processor using the Consumed.as() method (full code below):
val usersOrdersStreams: KStream[UserId, Order] = builder
.stream[UserId, Order](ordersByUserTopic)(Consumed.as("topic-name"))
However when I'm running the application I'm getting the following exception:
scalaorg.apache.kafka.common.config.ConfigException: Please specify a value serde or set one through StreamsConfig#DEFAULT_VALUE_SERDE_CLASS_CONFIG
When I looked at the definition of .as() I saw this:
public static <K, V> Consumed<K, V> as(final String processorName) {
return new Consumed<>(null, null, null, null, processorName);
}
So I guessed the issue was that the key/value serdes were set to null.
I tried to solve it by adding a call to withValueSerde():
val orderSerde = ...
val usersOrdersStreams: KStream[UserId, Order] = builder
.stream[UserId, Order](ordersByUserTopic)(Consumed.as("topic-name").withValueSerde(orderSerde))
But got the same error. What am I doing wrong?
Note: if I remove the Consumed.as() part the code works and the exception is not being thrown
Following is the full code (some imports were removed for readability reasons):
import org.apache.kafka.common.serialization.Serde
import org.apache.kafka.streams.kstream.{GlobalKTable, JoinWindows, TimeWindows, Windowed}
import org.apache.kafka.streams.scala.ImplicitConversions._
import org.apache.kafka.streams.scala.serialization.Serdes
import org.apache.kafka.streams.scala.serialization.Serdes._
import scala.concurrent.duration._
object KafkaStreamsApp {
implicit def serde[A >: Null : Decoder : Encoder]: Serde[A] = {
val serializer = (a: A) => a.asJson.noSpaces.getBytes
val deserializer = (aAsBytes: Array[Byte]) => {
val aAsString = new String(aAsBytes)
val aOrError = decode[A](aAsString)
aOrError match {
case Right(a) => Option(a)
case Left(error) =>
Option.empty
}
}
Serdes.fromFn[A](serializer, deserializer)
}
implicit val orderSerde: Serde[Order] = serde[Order]
// Topics
final val ordersByUserTopic = "orders-by-user"
final val filterOrders = "filter-low-orders"
final val applyMapValues = "mapValues-apply-discount"
final val payedOrdersTopic = "filtered-orders"
type UserId = String
case class Order(user: UserId, amount: Double)
val builder = new StreamsBuilder
val usersOrdersStreams: KStream[UserId, Order] =
builder.stream[UserId, Order](ordersByUserTopic)(Consumed.as("vvv").withValueSerde(orderSerde))
def paidOrdersTopology(): Unit = {
usersOrdersStreams
.filter((_, v) => v.amount > 1000.0, named = Named.as(filterOrders))
.mapValues(v => v.copy(amount = v.amount * 0.85), named = Named.as(applyMapValues))
.to(payedOrdersTopic)
}
def main(args: Array[String]): Unit = {
val props = new Properties
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "orders-application")
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092")
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.stringSerde.getClass)
paidOrdersTopology()
val topology: Topology = builder.build()
println(topology.describe())
val application: KafkaStreams = new KafkaStreams(topology, props)
application.start()
}
}
So... after some digging I managed to find the issue: the key serde was missing. The following code sets only the values serde, which creates a Consumed object with a null key serde:
val orderSerde = ...
val usersOrdersStreams: KStream[UserId, Order] = builder
.stream[UserId, Order](ordersByUserTopic)(Consumed.as("topic-name").withValueSerde(orderSerde))
When I added the key serde as well:
val orderSerde = ...
val consumed = Consumed.as("topic-name")
.withKeySerde(Serdes.stringSerde) // Missing key serde
.withValueSerde(orderSerde)
val usersOrdersStreams: KStream[UserId, Order] =
builder.stream[UserId, Order](ordersByUserTopic)(consumed)
The code started working.
The only thing I'm not sure about is why the error thrown stated that value serde was missing, when it's the key serde that's missing.
I'm writing a Wishbone Intercon module to make the address decoding automatically.
I have two Bundle classes that describe Wishbone master and Wishbone slave interface.
class WbMaster (val dwidth: Int,
val awidth: Int) extends Bundle {
val adr_o = Output(UInt(awidth.W))
//...
val cyc_o = Output(Bool())
}
// Wishbone slave interface
class WbSlave (val dwidth: Int,
val awidth: Int) extends Bundle {
val adr_i = Input(UInt(awidth.W))
//...
val cyc_i = Input(Bool())
}
I want to pass these Bundle as parameter to my module Wishbone like following:
class WbInterconOneMaster(val awbm: WbMaster,
val awbs: Vec(WbSlave)) extends Module {
val io = IO(new Bundle{
val wbm = Flipped(new WbMaster(awbm.dwidth, awbm.awidth))
val wbs = Vec(?)
})
}
The objective is to permit a variable number of wishbone slaves and let the module doing the plumbing. Like following:
val spi2Wb = Module(new Spi2Wb(dwidth, awidth))
val wbMdio1 = Module(new MdioWb(mainFreq, targetFreq))
val wbMdio2 = Module(new MdioWb(mainFreq, targetFreq))
val slavesVec = Vec(Seq(wbMdio1, wbMdio2))
val wbIntercon = Module(new WbIntercon(spi2Wb.io.wbm, slavesVec))
The question is multiple:
is it the right way to do it ?
How to declare the Vec() in module parameters ?
I tryied this but does not work:
// Wishbone Intercone with one master and several slaves
// data bus is same size as master
class WbInterconOneMaster(val awbm: WbMaster,
val awbs: Vec[Seq[WbSlave]]) extends Module {
val io = IO(new Bundle{
val wbm = Flipped(new WbMaster(awbm.dwidth, awbm.awidth))
val wbs = Vec.fill(awbs.size){awbs.map(_.cloneType())}
})
}
Try thinking about your parameters as generators of the types that you need. The following is a toy example of this idea. In this case the one constructor parameter bgen is a generator method that will return an instance of a Bundle. It shows the use of this generator as is and also as part of a Vec
class BundleX extends Bundle {
val a = UInt(8.W)
val b = UInt(8.W)
}
class ModuleX(bgen: () => BundleX, numInputs: Int) extends Module {
val io = IO(new Bundle{
val in1 = Input(Vec(numInputs, bgen()))
val out1 = Output(bgen())
})
// output fields a and b are the the sum of all the corresponding inputs
io.out1.a := io.in1.foldLeft(0.U) { case (res, value) => res +% value.a}
io.out1.b := io.in1.foldLeft(0.U) { case (res, value) => res +% value.b}
}
class BundleXSpec extends ChiselPropSpec {
property("testname") {
elaborate(new ModuleX(() => new BundleX, 4))
}
}
I found a solution with MixedVec (experimental) module. I simply pass a Seq of WbSlave Bundle as a module parameter and I made a MixedVec (WbSlave can have different parameters in fact):
class WbInterconOneMaster(val awbm: WbMaster,
val awbs: Seq[WbSlave]) extends Module {
val io = IO(new Bundle{
val wbm = Flipped(new WbMaster(awbm.dwidth, awbm.awidth))
val wbs = MixedVec(awbs.map{i => new WbSlave(i.dwidth, i.awidth)})
})
io.wbm.dat_i := 0.U
io.wbm.ack_i := 0.U
for(i <- 0 until io.wbs.size){
io.wbs(i).dat_o := 0.U
io.wbs(i).ack_o := 0.U
}
}
That compile in the testbench.
result.map { res =>
val totaldocs: Int = res.value
// do something with this number
}
//val totaldocs = 60
val totalpages:Int = (totaldocs/ipp)+1
Compilation error not found: value x but I declared it already, what is wrong with my implementation, sorry I am new to play framework and scala programming language.
I would say this line is the problem:
val totalpages:Int = (totaldocs/ipp)+1
because totaldocs is only defined inside the map scope
maybe you want something like:
private def getTotalPages(query:BSONDocument, ipp:Int) (implicit ec: ExecutionContext) = {
val key = collectionName + ":" + BSONDocument.pretty(query)
Logger.debug("Query key = "+key)
val command = Count(query)
val result: Future[CountResult] = collection.runCommand(command)
result.map { res =>
val totaldocs: Int = res.value
// do something with this number
val totalpages:Int = (totaldocs/ipp)+1
Logger.debug(s"Total docs $totaldocs, Total pages $totalpages, Items per page, $ipp")
totalpages
}
}
but now it will return a Future[Int] and you will have to deal with the future on the caller.
Note: this is just one solution, depending on your code it may not be the most adequate one
How does one access the parameters used to construct a Module from inside the Tester that is testing it?
In the test below I am passing the parameters explicitly both to the Module and to the Tester. I would prefer not to have to pass them to the Tester but instead extract them from the module that was also passed in.
Also I am new to scala/chisel so any tips on bad techniques I'm using would be appreciated :).
import Chisel._
import math.pow
class TestA(dataWidth: Int, arrayLength: Int) extends Module {
val dataType = Bits(INPUT, width = dataWidth)
val arrayType = Vec(gen = dataType, n = arrayLength)
val io = new Bundle {
val i_valid = Bool(INPUT)
val i_data = dataType
val i_array = arrayType
val o_valid = Bool(OUTPUT)
val o_data = dataType.flip
val o_array = arrayType.flip
}
io.o_valid := io.i_valid
io.o_data := io.i_data
io.o_array := io.i_array
}
class TestATests(c: TestA, dataWidth: Int, arrayLength: Int) extends Tester(c) {
val maxData = pow(2, dataWidth).toInt
for (t <- 0 until 16) {
val i_valid = rnd.nextInt(2)
val i_data = rnd.nextInt(maxData)
val i_array = List.fill(arrayLength)(rnd.nextInt(maxData))
poke(c.io.i_valid, i_valid)
poke(c.io.i_data, i_data)
(c.io.i_array, i_array).zipped foreach {
(element,value) => poke(element, value)
}
expect(c.io.o_valid, i_valid)
expect(c.io.o_data, i_data)
(c.io.o_array, i_array).zipped foreach {
(element,value) => poke(element, value)
}
step(1)
}
}
object TestAObject {
def main(args: Array[String]): Unit = {
val tutArgs = args.slice(0, args.length)
val dataWidth = 5
val arrayLength = 6
chiselMainTest(tutArgs, () => Module(
new TestA(dataWidth=dataWidth, arrayLength=arrayLength))){
c => new TestATests(c, dataWidth=dataWidth, arrayLength=arrayLength)
}
}
}
If you make the arguments dataWidth and arrayLength members of TestA you can just reference them. In Scala this can be accomplished by inserting val into the argument list:
class TestA(val dataWidth: Int, val arrayLength: Int) extends Module ...
Then you can reference them from the test as members with c.dataWidth or c.arrayLength
To embed Scala as a "scripting language", I need to be able to compile text fragments to simple objects, such as Function0[Unit] that can be serialised to and deserialised from disk and which can be loaded into the current runtime and executed.
How would I go about this?
Say for example, my text fragment is (purely hypothetical):
Document.current.elements.headOption.foreach(_.open())
This might be wrapped into the following complete text:
package myapp.userscripts
import myapp.DSL._
object UserFunction1234 extends Function0[Unit] {
def apply(): Unit = {
Document.current.elements.headOption.foreach(_.open())
}
}
What comes next? Should I use IMain to compile this code? I don't want to use the normal interpreter mode, because the compilation should be "context-free" and not accumulate requests.
What I need to get hold off from the compilation is I guess the binary class file? In that case, serialisation is straight forward (byte array). How would I then load that class into the runtime and invoke the apply method?
What happens if the code compiles to multiple auxiliary classes? The example above contains a closure _.open(). How do I make sure I "package" all those auxiliary things into one object to serialize and class-load?
Note: Given that Scala 2.11 is imminent and the compiler API probably changed, I am happy to receive hints as how to approach this problem on Scala 2.11
Here is one idea: use a regular Scala compiler instance. Unfortunately it seems to require the use of hard disk files both for input and output. So we use temporary files for that. The output will be zipped up in a JAR which will be stored as a byte array (that would go into the hypothetical serialization process). We need a special class loader to retrieve the class again from the extracted JAR.
The following assumes Scala 2.10.3 with the scala-compiler library on the class path:
import scala.tools.nsc
import java.io._
import scala.annotation.tailrec
Wrapping user provided code in a function class with a synthetic name that will be incremented for each new fragment:
val packageName = "myapp"
var userCount = 0
def mkFunName(): String = {
val c = userCount
userCount += 1
s"Fun$c"
}
def wrapSource(source: String): (String, String) = {
val fun = mkFunName()
val code = s"""package $packageName
|
|class $fun extends Function0[Unit] {
| def apply(): Unit = {
| $source
| }
|}
|""".stripMargin
(fun, code)
}
A function to compile a source fragment and return the byte array of the resulting jar:
/** Compiles a source code consisting of a body which is wrapped in a `Function0`
* apply method, and returns the function's class name (without package) and the
* raw jar file produced in the compilation.
*/
def compile(source: String): (String, Array[Byte]) = {
val set = new nsc.Settings
val d = File.createTempFile("temp", ".out")
d.delete(); d.mkdir()
set.d.value = d.getPath
set.usejavacp.value = true
val compiler = new nsc.Global(set)
val f = File.createTempFile("temp", ".scala")
val out = new BufferedOutputStream(new FileOutputStream(f))
val (fun, code) = wrapSource(source)
out.write(code.getBytes("UTF-8"))
out.flush(); out.close()
val run = new compiler.Run()
run.compile(List(f.getPath))
f.delete()
val bytes = packJar(d)
deleteDir(d)
(fun, bytes)
}
def deleteDir(base: File): Unit = {
base.listFiles().foreach { f =>
if (f.isFile) f.delete()
else deleteDir(f)
}
base.delete()
}
Note: Doesn't handle compiler errors yet!
The packJar method uses the compiler output directory and produces an in-memory jar file from it:
// cf. http://stackoverflow.com/questions/1281229
def packJar(base: File): Array[Byte] = {
import java.util.jar._
val mf = new Manifest
mf.getMainAttributes.put(Attributes.Name.MANIFEST_VERSION, "1.0")
val bs = new java.io.ByteArrayOutputStream
val out = new JarOutputStream(bs, mf)
def add(prefix: String, f: File): Unit = {
val name0 = prefix + f.getName
val name = if (f.isDirectory) name0 + "/" else name0
val entry = new JarEntry(name)
entry.setTime(f.lastModified())
out.putNextEntry(entry)
if (f.isFile) {
val in = new BufferedInputStream(new FileInputStream(f))
try {
val buf = new Array[Byte](1024)
#tailrec def loop(): Unit = {
val count = in.read(buf)
if (count >= 0) {
out.write(buf, 0, count)
loop()
}
}
loop()
} finally {
in.close()
}
}
out.closeEntry()
if (f.isDirectory) f.listFiles.foreach(add(name, _))
}
base.listFiles().foreach(add("", _))
out.close()
bs.toByteArray
}
A utility function that takes the byte array found in deserialization and creates a map from class names to class byte code:
def unpackJar(bytes: Array[Byte]): Map[String, Array[Byte]] = {
import java.util.jar._
import scala.annotation.tailrec
val in = new JarInputStream(new ByteArrayInputStream(bytes))
val b = Map.newBuilder[String, Array[Byte]]
#tailrec def loop(): Unit = {
val entry = in.getNextJarEntry
if (entry != null) {
if (!entry.isDirectory) {
val name = entry.getName
// cf. http://stackoverflow.com/questions/8909743
val bs = new ByteArrayOutputStream
var i = 0
while (i >= 0) {
i = in.read()
if (i >= 0) bs.write(i)
}
val bytes = bs.toByteArray
b += mkClassName(name) -> bytes
}
loop()
}
}
loop()
in.close()
b.result()
}
def mkClassName(path: String): String = {
require(path.endsWith(".class"))
path.substring(0, path.length - 6).replace("/", ".")
}
A suitable class loader:
class MemoryClassLoader(map: Map[String, Array[Byte]]) extends ClassLoader {
override protected def findClass(name: String): Class[_] =
map.get(name).map { bytes =>
println(s"defineClass($name, ...)")
defineClass(name, bytes, 0, bytes.length)
} .getOrElse(super.findClass(name)) // throws exception
}
And a test case which contains additional classes (closures):
val exampleSource =
"""val xs = List("hello", "world")
|println(xs.map(_.capitalize).mkString(" "))
|""".stripMargin
def test(fun: String, cl: ClassLoader): Unit = {
val clName = s"$packageName.$fun"
println(s"Resolving class '$clName'...")
val clazz = Class.forName(clName, true, cl)
println("Instantiating...")
val x = clazz.newInstance().asInstanceOf[() => Unit]
println("Invoking 'apply':")
x()
}
locally {
println("Compiling...")
val (fun, bytes) = compile(exampleSource)
val map = unpackJar(bytes)
println("Classes found:")
map.keys.foreach(k => println(s" '$k'"))
val cl = new MemoryClassLoader(map)
test(fun, cl) // should call `defineClass`
test(fun, cl) // should find cached class
}