Chisel: Access to Module Parameters from Tester - scala

How does one access the parameters used to construct a Module from inside the Tester that is testing it?
In the test below I am passing the parameters explicitly both to the Module and to the Tester. I would prefer not to have to pass them to the Tester but instead extract them from the module that was also passed in.
Also I am new to scala/chisel so any tips on bad techniques I'm using would be appreciated :).
import Chisel._
import math.pow
class TestA(dataWidth: Int, arrayLength: Int) extends Module {
val dataType = Bits(INPUT, width = dataWidth)
val arrayType = Vec(gen = dataType, n = arrayLength)
val io = new Bundle {
val i_valid = Bool(INPUT)
val i_data = dataType
val i_array = arrayType
val o_valid = Bool(OUTPUT)
val o_data = dataType.flip
val o_array = arrayType.flip
}
io.o_valid := io.i_valid
io.o_data := io.i_data
io.o_array := io.i_array
}
class TestATests(c: TestA, dataWidth: Int, arrayLength: Int) extends Tester(c) {
val maxData = pow(2, dataWidth).toInt
for (t <- 0 until 16) {
val i_valid = rnd.nextInt(2)
val i_data = rnd.nextInt(maxData)
val i_array = List.fill(arrayLength)(rnd.nextInt(maxData))
poke(c.io.i_valid, i_valid)
poke(c.io.i_data, i_data)
(c.io.i_array, i_array).zipped foreach {
(element,value) => poke(element, value)
}
expect(c.io.o_valid, i_valid)
expect(c.io.o_data, i_data)
(c.io.o_array, i_array).zipped foreach {
(element,value) => poke(element, value)
}
step(1)
}
}
object TestAObject {
def main(args: Array[String]): Unit = {
val tutArgs = args.slice(0, args.length)
val dataWidth = 5
val arrayLength = 6
chiselMainTest(tutArgs, () => Module(
new TestA(dataWidth=dataWidth, arrayLength=arrayLength))){
c => new TestATests(c, dataWidth=dataWidth, arrayLength=arrayLength)
}
}
}

If you make the arguments dataWidth and arrayLength members of TestA you can just reference them. In Scala this can be accomplished by inserting val into the argument list:
class TestA(val dataWidth: Int, val arrayLength: Int) extends Module ...
Then you can reference them from the test as members with c.dataWidth or c.arrayLength

Related

Vec of Bundle as a Module parameter

I'm writing a Wishbone Intercon module to make the address decoding automatically.
I have two Bundle classes that describe Wishbone master and Wishbone slave interface.
class WbMaster (val dwidth: Int,
val awidth: Int) extends Bundle {
val adr_o = Output(UInt(awidth.W))
//...
val cyc_o = Output(Bool())
}
// Wishbone slave interface
class WbSlave (val dwidth: Int,
val awidth: Int) extends Bundle {
val adr_i = Input(UInt(awidth.W))
//...
val cyc_i = Input(Bool())
}
I want to pass these Bundle as parameter to my module Wishbone like following:
class WbInterconOneMaster(val awbm: WbMaster,
val awbs: Vec(WbSlave)) extends Module {
val io = IO(new Bundle{
val wbm = Flipped(new WbMaster(awbm.dwidth, awbm.awidth))
val wbs = Vec(?)
})
}
The objective is to permit a variable number of wishbone slaves and let the module doing the plumbing. Like following:
val spi2Wb = Module(new Spi2Wb(dwidth, awidth))
val wbMdio1 = Module(new MdioWb(mainFreq, targetFreq))
val wbMdio2 = Module(new MdioWb(mainFreq, targetFreq))
val slavesVec = Vec(Seq(wbMdio1, wbMdio2))
val wbIntercon = Module(new WbIntercon(spi2Wb.io.wbm, slavesVec))
The question is multiple:
is it the right way to do it ?
How to declare the Vec() in module parameters ?
I tryied this but does not work:
// Wishbone Intercone with one master and several slaves
// data bus is same size as master
class WbInterconOneMaster(val awbm: WbMaster,
val awbs: Vec[Seq[WbSlave]]) extends Module {
val io = IO(new Bundle{
val wbm = Flipped(new WbMaster(awbm.dwidth, awbm.awidth))
val wbs = Vec.fill(awbs.size){awbs.map(_.cloneType())}
})
}
Try thinking about your parameters as generators of the types that you need. The following is a toy example of this idea. In this case the one constructor parameter bgen is a generator method that will return an instance of a Bundle. It shows the use of this generator as is and also as part of a Vec
class BundleX extends Bundle {
val a = UInt(8.W)
val b = UInt(8.W)
}
class ModuleX(bgen: () => BundleX, numInputs: Int) extends Module {
val io = IO(new Bundle{
val in1 = Input(Vec(numInputs, bgen()))
val out1 = Output(bgen())
})
// output fields a and b are the the sum of all the corresponding inputs
io.out1.a := io.in1.foldLeft(0.U) { case (res, value) => res +% value.a}
io.out1.b := io.in1.foldLeft(0.U) { case (res, value) => res +% value.b}
}
class BundleXSpec extends ChiselPropSpec {
property("testname") {
elaborate(new ModuleX(() => new BundleX, 4))
}
}
I found a solution with MixedVec (experimental) module. I simply pass a Seq of WbSlave Bundle as a module parameter and I made a MixedVec (WbSlave can have different parameters in fact):
class WbInterconOneMaster(val awbm: WbMaster,
val awbs: Seq[WbSlave]) extends Module {
val io = IO(new Bundle{
val wbm = Flipped(new WbMaster(awbm.dwidth, awbm.awidth))
val wbs = MixedVec(awbs.map{i => new WbSlave(i.dwidth, i.awidth)})
})
io.wbm.dat_i := 0.U
io.wbm.ack_i := 0.U
for(i <- 0 until io.wbs.size){
io.wbs(i).dat_o := 0.U
io.wbs(i).ack_o := 0.U
}
}
That compile in the testbench.

How to generate Function1 by parameter Class and return Class

I want to register a scala Object Method into spark udf. Now i get the MethodMirror by scala reflect and Parameter by java reflect. But i cannot generate a Function object used to register into spark.udf. Such as:
object ArrayUdfs {
def array2String(arr: Seq[Long])= {
arr.mkString(",")
}
}
I want to register method 'array2String' into spark udf.
First i get the MethodMirror:
def getObjectMethod(clazzPath:String, methodName:String) = {
import scala.reflect.runtime.universe
lazy val runtimeMirror = universe.runtimeMirror(getClass.getClassLoader)
lazy val module = runtimeMirror.staticModule(clazzPath)
lazy val obj = runtimeMirror.reflectModule(module)
lazy val objMirror = runtimeMirror.reflect(obj.instance)
lazy val method = obj.symbol.typeSignature.member(universe.TermName(methodName)).asMethod
lazy val methodObject = objMirror.reflectMethod(method)
methodObject
}
lazy val method = getObjectMethod(clazzPath, funName);
Second. I get Parameter.
def getObjectMethodParams(clazzPath:String, methodName:String): Array[Parameter] = {
val methods = Class.forName(clazzPath).getDeclaredMethods()
var params: Array[Parameter] = null;
methods.foreach(method => {
if (method.getName == methodName) params = method.getParameters
})
params
}
val params = getObjectMethodParams(clazzPath, funName)
Third, register it into spark.udf
val function1: Function1[Seq[String], String] = (arr) => {
method.apply(arr).asInstanceOf[String]
}
spark.udf.register("array2String", function1)
So i want to replace 'Seq[String]' in 'Function1[Seq[String], String]' by 'params' object. The 'String' in 'Function1[Seq[String], String]' is same.

How to use chisel dsptools with floats

I need to convert a Float32 into a Chisel FixedPoint, perform some computation and convert back FixedPoint to Float32.
For example, I need the following:
val a = 3.1F
val b = 2.2F
val res = a * b // REPL returns res: Float 6.82
Now, I do this:
import chisel3.experimental.FixedPoint
val fp_tpe = FixedPoint(6.W, 2.BP)
val a_fix = a.Something (fp_tpe) // convert a to FixPoint
val b_fix = b.Something (fp_tpe) // convert b to FixPoint
val res_fix = a_fix * b_fix
val res0 = res_fix.Something (fp_tpe) // convert back to Float
As a result, I'd expect the delta to be in a range of , e.g
val eps = 1e-4
assert ( abs(res - res0) < eps, "The error is too big")
Who can provide a working example for Chisel3 FixedPoint class for the pseudocode above?
Take a look at the following code:
import chisel3._
import chisel3.core.FixedPoint
import dsptools._
class FPMultiplier extends Module {
val io = IO(new Bundle {
val a = Input(FixedPoint(6.W, binaryPoint = 2.BP))
val b = Input(FixedPoint(6.W, binaryPoint = 2.BP))
val c = Output(FixedPoint(12.W, binaryPoint = 4.BP))
})
io.c := io.a * io.b
}
class FPMultiplierTester(c: FPMultiplier) extends DspTester(c) {
//
// This will PASS, there is sufficient precision to model the inputs
//
poke(c.io.a, 3.25)
poke(c.io.b, 2.5)
step(1)
expect(c.io.c, 8.125)
//
// This will FAIL, there is not sufficient precision to model the inputs
// But this is only caught on output, this is likely the right approach
// because you can't really pass in wrong precision data in hardware.
//
poke(c.io.a, 3.1)
poke(c.io.b, 2.2)
step(1)
expect(c.io.c, 6.82)
}
object FPMultiplierMain {
def main(args: Array[String]): Unit = {
iotesters.Driver.execute(Array("-fiv"), () => new FPMultiplier) { c =>
new FPMultiplierTester(c)
}
}
}
I'd also suggest looking at ParameterizedAdder in dsptools, that gives you a feel of how to write hardware modules that you pass different types. Generally you start with DspReals, confirm the model then start experimenting/calculating with FixedPoint sizes that return results with the desired precision.
For others benefit, I provide an improved solution from #Chick, rewritten in a more abstract Scala with variable DSP tolerances.
package my_pkg
import chisel3._
import chisel3.core.{FixedPoint => FP}
import dsptools.{DspTester, DspTesterOptions, DspTesterOptionsManager}
class FPGenericIO (inType:FP, outType:FP) extends Bundle {
val a = Input(inType)
val b = Input(inType)
val c = Output(outType)
}
class FPMul (inType:FP, outType:FP) extends Module {
val io = IO(new FPGenericIO(inType, outType))
io.c := io.a * io.b
}
class FPMulTester(c: FPMul) extends DspTester(c) {
val uut = c.io
// This will PASS, there is sufficient precision to model the inputs
poke(uut.a, 3.25)
poke(uut.b, 2.5)
step(1)
expect(uut.c, 3.25*2.5)
// This will FAIL, if you won't increase tolerance, which is eps = 0.0 by default
poke(uut.a, 3.1)
poke(uut.b, 2.2)
step(1)
expect(uut.c, 3.1*2.2)
}
object FPUMain extends App {
val fpInType = FP(8.W, 4.BP)
val fpOutType = FP(12.W, 6.BP)
// Update default DspTester options and increase tolerance
val opts = new DspTesterOptionsManager {
dspTesterOptions = DspTesterOptions(
fixTolLSBs = 2,
genVerilogTb = false,
isVerbose = true
)
}
dsptools.Driver.execute (() => new FPMul(fpInType, fpOutType), opts) {
c => new FPMulTester(c)
}
}
Here's my ultimate DSP multiplier implementation, which should support both FixedPoint and DspComplex numbers. #ChickMarkley, how do I update this class to implement a Complex multiplication?
package my_pkg
import chisel3._
import dsptools.numbers.{Ring,DspComplex}
import dsptools.numbers.implicits._
import dsptools.{DspContext}
import chisel3.core.{FixedPoint => FP}
import dsptools.{DspTester, DspTesterOptions, DspTesterOptionsManager}
class FPGenericIO[A <: Data:Ring, B <: Data:Ring] (inType:A, outType:B) extends Bundle {
val a = Input(inType.cloneType)
val b = Input(inType.cloneType)
val c = Output(outType.cloneType)
override def cloneType = (new FPGenericIO(inType, outType)).asInstanceOf[this.type]
}
class FPMul[A <: Data:Ring, B <: Data:Ring] (inType:A, outType:B) extends Module {
val io = IO(new FPGenericIO(inType, outType))
DspContext.withNumMulPipes(3) {
io.c := io.a * io.b
}
}
class FPMulTester[A <: Data:Ring, B <: Data:Ring](c: FPMul[A,B]) extends DspTester(c) {
val uut = c.io
//
// This will PASS, there is sufficient precision to model the inputs
//
poke(uut.a, 3.25)
poke(uut.b, 2.5)
step(1)
expect(uut.c, 3.25*2.5)
//
// This will FAIL, there is not sufficient precision to model the inputs
// But this is only caught on output, this is likely the right approach
// because you can't really pass in wrong precision data in hardware.
//
poke(uut.a, 3.1)
poke(uut.b, 2.2)
step(1)
expect(uut.c, 3.1*2.2)
}
object FPUMain extends App {
val fpInType = FP(8.W, 4.BP)
val fpOutType = FP(12.W, 6.BP)
//val comp = DspComplex[Double] // How to declare a complex DSP type ?
val opts = new DspTesterOptionsManager {
dspTesterOptions = DspTesterOptions(
fixTolLSBs = 0,
genVerilogTb = false,
isVerbose = true
)
}
dsptools.Driver.execute (() => new FPMul(fpInType, fpOutType), opts) {
//dsptools.Driver.execute (() => new FPMul(comp, comp), opts) { // <-- this won't compile
c => new FPMulTester(c)
}
}

How to inject quasi quotes array

I have an array of quasi quotes called definitions which I want to inject to the quasi quote tree. How do I go about doing this?
private def generateDaoComponent(files: Array[File]) = {
val file = createNewFile(compDirectory)
val definitions = files.map(f => {
val daoName = f.getName.replace(".java", "")
val daoType = TypeName(daoName)
val daoTerm = TermName(daoName)
q"""def $daoTerm = getValueOrInstantiate($daoName, () => new $daoType(configuration))
"""
})
val tree = q"""
package database.dao {
import org.jooq.SQLDialect
import org.jooq.impl.DefaultConfiguration
import utility.StrongHashMap
trait $componentType extends StrongHashMap[String, Dao] {
this: DaoManager =>
private lazy val configuration = new DefaultConfiguration().set(connection).set(SQLDialect.POSTGRES_9_4)
${definitions.foreach(f => q"${f}")}
}
}"""
writeToFile(file, tree)
}
This is some crazy late night coding, but I found it thanks to this website
3 approaches to Scala code generation
I noticed when it passed the $params array into the quasi quote it used two .. in front of the class constructor like this:
val params = schema.fields.map { field =>
val fieldName = newTermName(field.name)
val fieldType = newTypeName(field.valueType.fullName)
q"val $fieldName: $fieldType"
}
val json = TypeSchema.toJson(schema)
// rewrite the class definition
c.Expr(
q"""
case class $className(..$params) {
def schema = ${json}
}
"""
)
There are two steps to the code I posted in the question to make this working.
1) Convert $definitions.toList to List
2) Add the two .. in front
So the final code looks like this:
val definitions = files.map(f => {
val daoName = f.getName.replace(".java", "")
val daoType = TypeName(daoName)
val daoTerm = TermName(new StringBuilder("get").append(daoName).toString())
q"""def $daoTerm = getValueOrInstantiate($daoName, () => new $daoType(configuration))"""
}).toList <--- HERE!
val tree = q"""
package database.dao {
import org.jooq.SQLDialect
import org.jooq.impl.DefaultConfiguration
import utility.StrongHashMap
trait $componentType extends StrongHashMap[String, Dao] {
this: DaoManager =>
private lazy val configuration = new DefaultConfiguration().set(connection).set(SQLDialect.POSTGRES_9_4)
..$definitions <-- AND HERE!
}
}"""

Creating serializable objects from Scala source code at runtime

To embed Scala as a "scripting language", I need to be able to compile text fragments to simple objects, such as Function0[Unit] that can be serialised to and deserialised from disk and which can be loaded into the current runtime and executed.
How would I go about this?
Say for example, my text fragment is (purely hypothetical):
Document.current.elements.headOption.foreach(_.open())
This might be wrapped into the following complete text:
package myapp.userscripts
import myapp.DSL._
object UserFunction1234 extends Function0[Unit] {
def apply(): Unit = {
Document.current.elements.headOption.foreach(_.open())
}
}
What comes next? Should I use IMain to compile this code? I don't want to use the normal interpreter mode, because the compilation should be "context-free" and not accumulate requests.
What I need to get hold off from the compilation is I guess the binary class file? In that case, serialisation is straight forward (byte array). How would I then load that class into the runtime and invoke the apply method?
What happens if the code compiles to multiple auxiliary classes? The example above contains a closure _.open(). How do I make sure I "package" all those auxiliary things into one object to serialize and class-load?
Note: Given that Scala 2.11 is imminent and the compiler API probably changed, I am happy to receive hints as how to approach this problem on Scala 2.11
Here is one idea: use a regular Scala compiler instance. Unfortunately it seems to require the use of hard disk files both for input and output. So we use temporary files for that. The output will be zipped up in a JAR which will be stored as a byte array (that would go into the hypothetical serialization process). We need a special class loader to retrieve the class again from the extracted JAR.
The following assumes Scala 2.10.3 with the scala-compiler library on the class path:
import scala.tools.nsc
import java.io._
import scala.annotation.tailrec
Wrapping user provided code in a function class with a synthetic name that will be incremented for each new fragment:
val packageName = "myapp"
var userCount = 0
def mkFunName(): String = {
val c = userCount
userCount += 1
s"Fun$c"
}
def wrapSource(source: String): (String, String) = {
val fun = mkFunName()
val code = s"""package $packageName
|
|class $fun extends Function0[Unit] {
| def apply(): Unit = {
| $source
| }
|}
|""".stripMargin
(fun, code)
}
A function to compile a source fragment and return the byte array of the resulting jar:
/** Compiles a source code consisting of a body which is wrapped in a `Function0`
* apply method, and returns the function's class name (without package) and the
* raw jar file produced in the compilation.
*/
def compile(source: String): (String, Array[Byte]) = {
val set = new nsc.Settings
val d = File.createTempFile("temp", ".out")
d.delete(); d.mkdir()
set.d.value = d.getPath
set.usejavacp.value = true
val compiler = new nsc.Global(set)
val f = File.createTempFile("temp", ".scala")
val out = new BufferedOutputStream(new FileOutputStream(f))
val (fun, code) = wrapSource(source)
out.write(code.getBytes("UTF-8"))
out.flush(); out.close()
val run = new compiler.Run()
run.compile(List(f.getPath))
f.delete()
val bytes = packJar(d)
deleteDir(d)
(fun, bytes)
}
def deleteDir(base: File): Unit = {
base.listFiles().foreach { f =>
if (f.isFile) f.delete()
else deleteDir(f)
}
base.delete()
}
Note: Doesn't handle compiler errors yet!
The packJar method uses the compiler output directory and produces an in-memory jar file from it:
// cf. http://stackoverflow.com/questions/1281229
def packJar(base: File): Array[Byte] = {
import java.util.jar._
val mf = new Manifest
mf.getMainAttributes.put(Attributes.Name.MANIFEST_VERSION, "1.0")
val bs = new java.io.ByteArrayOutputStream
val out = new JarOutputStream(bs, mf)
def add(prefix: String, f: File): Unit = {
val name0 = prefix + f.getName
val name = if (f.isDirectory) name0 + "/" else name0
val entry = new JarEntry(name)
entry.setTime(f.lastModified())
out.putNextEntry(entry)
if (f.isFile) {
val in = new BufferedInputStream(new FileInputStream(f))
try {
val buf = new Array[Byte](1024)
#tailrec def loop(): Unit = {
val count = in.read(buf)
if (count >= 0) {
out.write(buf, 0, count)
loop()
}
}
loop()
} finally {
in.close()
}
}
out.closeEntry()
if (f.isDirectory) f.listFiles.foreach(add(name, _))
}
base.listFiles().foreach(add("", _))
out.close()
bs.toByteArray
}
A utility function that takes the byte array found in deserialization and creates a map from class names to class byte code:
def unpackJar(bytes: Array[Byte]): Map[String, Array[Byte]] = {
import java.util.jar._
import scala.annotation.tailrec
val in = new JarInputStream(new ByteArrayInputStream(bytes))
val b = Map.newBuilder[String, Array[Byte]]
#tailrec def loop(): Unit = {
val entry = in.getNextJarEntry
if (entry != null) {
if (!entry.isDirectory) {
val name = entry.getName
// cf. http://stackoverflow.com/questions/8909743
val bs = new ByteArrayOutputStream
var i = 0
while (i >= 0) {
i = in.read()
if (i >= 0) bs.write(i)
}
val bytes = bs.toByteArray
b += mkClassName(name) -> bytes
}
loop()
}
}
loop()
in.close()
b.result()
}
def mkClassName(path: String): String = {
require(path.endsWith(".class"))
path.substring(0, path.length - 6).replace("/", ".")
}
A suitable class loader:
class MemoryClassLoader(map: Map[String, Array[Byte]]) extends ClassLoader {
override protected def findClass(name: String): Class[_] =
map.get(name).map { bytes =>
println(s"defineClass($name, ...)")
defineClass(name, bytes, 0, bytes.length)
} .getOrElse(super.findClass(name)) // throws exception
}
And a test case which contains additional classes (closures):
val exampleSource =
"""val xs = List("hello", "world")
|println(xs.map(_.capitalize).mkString(" "))
|""".stripMargin
def test(fun: String, cl: ClassLoader): Unit = {
val clName = s"$packageName.$fun"
println(s"Resolving class '$clName'...")
val clazz = Class.forName(clName, true, cl)
println("Instantiating...")
val x = clazz.newInstance().asInstanceOf[() => Unit]
println("Invoking 'apply':")
x()
}
locally {
println("Compiling...")
val (fun, bytes) = compile(exampleSource)
val map = unpackJar(bytes)
println("Classes found:")
map.keys.foreach(k => println(s" '$k'"))
val cl = new MemoryClassLoader(map)
test(fun, cl) // should call `defineClass`
test(fun, cl) // should find cached class
}