Chisel: How to change module parameters from command line? - scala

Question

I have many modules with multiple parameters. Take as a toy example a modified version of the GCD in the template:
class GCD (len: Int = 16, validHigh: Boolean = true) extends Module {
val io = IO(new Bundle {
val value1 = Input(UInt(len.W))
val value2 = Input(UInt(len.W))
val loadingValues = Input(Bool())
val outputGCD = Output(UInt(len.W))
val outputValid = Output(Bool())
})
val x = Reg(UInt())
val y = Reg(UInt())
when(x > y) { x := x - y }
.otherwise { y := y - x }
when(io.loadingValues) {
x := io.value1
y := io.value2
}
io.outputGCD := x
if (validHigh) {
io.outputValid := (y === 0.U)
} else {
io.outputValid := (y =/= 0.U)
}
}
To test or synthesize many different designs, I want to change the values from the command line when I call the tester or the generator apps. Preferably, like this:
[generation or test command] --len 12 --validHigh false
but this or something similar would also be okay
[generation or test command] --param "len=12" --param "validHigh=false"
After some trial and error, I came up with a solution that looks like this:
gcd.scala
package gcd
import firrtl._
import chisel3._
case class GCDConfig(
len: Int = 16,
validHigh: Boolean = true
)
class GCD (val conf: GCDConfig = GCDConfig()) extends Module {
val io = IO(new Bundle {
val value1 = Input(UInt(conf.len.W))
val value2 = Input(UInt(conf.len.W))
val loadingValues = Input(Bool())
val outputGCD = Output(UInt(conf.len.W))
val outputValid = Output(Bool())
})
val x = Reg(UInt())
val y = Reg(UInt())
when(x > y) { x := x - y }
.otherwise { y := y - x }
when(io.loadingValues) {
x := io.value1
y := io.value2
}
io.outputGCD := x
if (conf.validHigh) {
io.outputValid := y === 0.U
} else {
io.outputValid := y =/= 0.U
}
}
trait HasParams {
self: ExecutionOptionsManager =>
var params: Map[String, String] = Map()
parser.note("Design Parameters")
parser.opt[Map[String, String]]('p', "params")
.valueName("k1=v1,k2=v2")
.foreach { v => params = v }
.text("Parameters of Design")
}
object GCD {
def apply(params: Map[String, String]): GCD = {
new GCD(params2conf(params))
}
def params2conf(params: Map[String, String]): GCDConfig = {
var conf = new GCDConfig
for ((k, v) <- params) {
(k, v) match {
case ("len", _) => conf = conf.copy(len = v.toInt)
case ("validHigh", _) => conf = conf.copy(validHigh = v.toBoolean)
case _ =>
}
}
conf
}
}
object GCDGen extends App {
val optionsManager = new ExecutionOptionsManager("gcdgen")
with HasChiselExecutionOptions with HasFirrtlOptions with HasParams
optionsManager.parse(args) match {
case true =>
chisel3.Driver.execute(optionsManager, () => GCD(optionsManager.params))
case _ =>
ChiselExecutionFailure("could not parse results")
}
}
and for tests
GCDSpec.scala
package gcd
import chisel3._
import firrtl._
import chisel3.tester._
import org.scalatest.FreeSpec
import chisel3.experimental.BundleLiterals._
import chiseltest.internal._
import chiseltest.experimental.TestOptionBuilder._
object GCDTest extends App {
val optionsManager = new ExecutionOptionsManager("gcdtest") with HasParams
optionsManager.parse(args) match {
case true =>
//println(optionsManager.commonOptions.programArgs)
(new GCDSpec(optionsManager.params)).execute()
case _ =>
ChiselExecutionFailure("could not parse results")
}
}
class GCDSpec(params: Map[String, String] = Map()) extends FreeSpec with ChiselScalatestTester {
"Gcd should calculate proper greatest common denominator" in {
test(GCD(params)) { dut =>
dut.io.value1.poke(95.U)
dut.io.value2.poke(10.U)
dut.io.loadingValues.poke(true.B)
dut.clock.step(1)
dut.io.loadingValues.poke(false.B)
while (dut.io.outputValid.peek().litToBoolean != dut.conf.validHigh) {
dut.clock.step(1)
}
dut.io.outputGCD.expect(5.U)
}
}
}
This way, I can generate different designs and test them with
sbt 'runMain gcd.GCDGen --params "len=12,validHigh=false"'
sbt 'test:runMain gcd.GCDTest --params "len=12,validHigh=false"'
But there are a couple of problems or annoyances with this solution:
It uses deprecated features (ExecutionOptionsManager and HasFirrtlOptions). I'm not sure if this solution is portable to the new FirrtlStage Infrastructure.
There is a lot of boilerplate involved. It becomes tedious to write new case classes and params2conf functions for every module and rewrite both when a parameter is added or removed.
Using conf.x instead of x all the time. But I guess, this is unavoidable because there is nothing like python's kwargs in Scala.
Is there a better way or one that is at least not deprecated?

Answer 1

Good Question.
I think you are you have pretty much everything right. I don't usually find that I need the command line to alter my tests, my development cycle usually is just poking values in the test params directly running. I use intelliJ which seems to make that easy (but may only work for my habits and the scale of projects I work on).
But I would like to offer you a suggestions that will get you away from ExecutionOptions style as that is going away fast.
In my example code below I offer basically two files here in line, in the first there a few library like tools that use the modern annotation idioms and, I believe, minimize boiler plate. They rely on stringy matching but that is fixable.
In the second, is your GCD, GCDSpec, slightly modified to pull out the params a bit differently. At the bottom of the second is some very minimal boiler plate that allows you to get the command line access you want.
Good luck, I hope this is mostly self explanatory.
First file:
import chisel3.stage.ChiselCli
import firrtl.AnnotationSeq
import firrtl.annotations.{Annotation, NoTargetAnnotation}
import firrtl.options.{HasShellOptions, Shell, ShellOption, Stage, Unserializable}
import firrtl.stage.FirrtlCli
trait TesterAnnotation {
this: Annotation =>
}
case class TestParams(params: Map[String, String] = Map.empty) {
val defaults: collection.mutable.HashMap[String, String] = new collection.mutable.HashMap()
def getInt(key: String): Int = params.getOrElse(key, defaults(key)).toInt
def getBoolean(key: String): Boolean = params.getOrElse(key, defaults(key)).toBoolean
def getString(key: String): String = params.getOrElse(key, defaults(key))
}
case class TesterParameterAnnotation(paramString: TestParams)
extends TesterAnnotation
with NoTargetAnnotation
with Unserializable
object TesterParameterAnnotation extends HasShellOptions {
val options = Seq(
new ShellOption[Map[String, String]](
longOption = "param-string",
toAnnotationSeq = (a: Map[String, String]) => Seq(TesterParameterAnnotation(TestParams(a))),
helpText = """a comma separated, space free list of additional paramters, e.g. --param-string "k1=7,k2=dog" """
)
)
}
trait TesterCli {
this: Shell =>
Seq(TesterParameterAnnotation).foreach(_.addOptions(parser))
}
class GenericTesterStage(thunk: (TestParams, AnnotationSeq) => Unit) extends Stage {
val shell: Shell = new Shell("chiseltest") with TesterCli with ChiselCli with FirrtlCli
def run(annotations: AnnotationSeq): AnnotationSeq = {
val params = annotations.collectFirst { case TesterParameterAnnotation(p) => p }.getOrElse(TestParams())
thunk(params, annotations)
annotations
}
}
Second File:
import chisel3._
import chisel3.tester._
import chiseltest.experimental.TestOptionBuilder._
import chiseltest.{ChiselScalatestTester, GenericTesterStage, TestParams}
import firrtl._
import firrtl.options.StageMain
import org.scalatest.freespec.AnyFreeSpec
case class GCD(testParams: TestParams) extends Module {
val bitWidth = testParams.getInt("len")
val validHigh = testParams.getBoolean("validHigh")
val io = IO(new Bundle {
val value1 = Input(UInt(bitWidth.W))
val value2 = Input(UInt(bitWidth.W))
val loadingValues = Input(Bool())
val outputGCD = Output(UInt(bitWidth.W))
val outputValid = Output(Bool())
})
val x = Reg(UInt())
val y = Reg(UInt())
when(x > y) { x := x - y }.otherwise { y := y - x }
when(io.loadingValues) {
x := io.value1
y := io.value2
}
io.outputGCD := x
if (validHigh) {
io.outputValid := y === 0.U
} else {
io.outputValid := y =/= 0.U
}
}
class GCDSpec(params: TestParams, annotations: AnnotationSeq = Seq()) extends AnyFreeSpec with ChiselScalatestTester {
"Gcd should calculate proper greatest common denominator" in {
test(GCD(params)).withAnnotations(annotations) { dut =>
dut.io.value1.poke(95.U)
dut.io.value2.poke(10.U)
dut.io.loadingValues.poke(true.B)
dut.clock.step(1)
dut.io.loadingValues.poke(false.B)
while (dut.io.outputValid.peek().litToBoolean != dut.validHigh) {
dut.clock.step(1)
}
dut.io.outputGCD.expect(5.U)
}
}
}
class GcdTesterStage
extends GenericTesterStage((params, annotations) => {
params.defaults ++= Seq("len" -> "16", "validHigh" -> "false")
(new GCDSpec(params, annotations)).execute()
})
object GcdTesterStage extends StageMain(new GcdTesterStage)

Answer 2

Based on http://blog.echo.sh/2013/11/04/exploring-scala-macros-map-to-case-class-conversion.html, I was able to find another way of removing the params2conf boilerplate using scala macros. I also extended Chick's answer with verilog generation since that was also part of the original question. A full repository of my solution can be found on github.
Basically there are three four files:
The macro that converts a map to a case class:
package mappable
import scala.language.experimental.macros
import scala.reflect.macros.whitebox.Context
trait Mappable[T] {
def toMap(t: T): Map[String, String]
def fromMap(map: Map[String, String]): T
}
object Mappable {
implicit def materializeMappable[T]: Mappable[T] = macro materializeMappableImpl[T]
def materializeMappableImpl[T: c.WeakTypeTag](c: Context): c.Expr[Mappable[T]] = {
import c.universe._
val tpe = weakTypeOf[T]
val companion = tpe.typeSymbol.companion
val fields = tpe.decls.collectFirst {
case m: MethodSymbol if m.isPrimaryConstructor => m
}.get.paramLists.head
val (toMapParams, fromMapParams) = fields.map { field =>
val name = field.name.toTermName
val decoded = name.decodedName.toString
val returnType = tpe.decl(name).typeSignature
val fromMapLine = returnType match {
case NullaryMethodType(res) if res =:= typeOf[Int] => q"map($decoded).toInt"
case NullaryMethodType(res) if res =:= typeOf[String] => q"map($decoded)"
case NullaryMethodType(res) if res =:= typeOf[Boolean] => q"map($decoded).toBoolean"
case _ => q""
}
(q"$decoded -> t.$name.toString", fromMapLine)
}.unzip
c.Expr[Mappable[T]] { q"""
new Mappable[$tpe] {
def toMap(t: $tpe): Map[String, String] = Map(..$toMapParams)
def fromMap(map: Map[String, String]): $tpe = $companion(..$fromMapParams)
}
""" }
}
}
Library like tools:
package cliparams
import chisel3.stage.{ChiselStage, ChiselGeneratorAnnotation, ChiselCli}
import firrtl.AnnotationSeq
import firrtl.annotations.{Annotation, NoTargetAnnotation}
import firrtl.options.{HasShellOptions, Shell, ShellOption, Stage, Unserializable, StageMain}
import firrtl.stage.FirrtlCli
import mappable._
trait SomeAnnotaion {
this: Annotation =>
}
case class ParameterAnnotation(map: Map[String, String])
extends SomeAnnotaion
with NoTargetAnnotation
with Unserializable
object ParameterAnnotation extends HasShellOptions {
val options = Seq(
new ShellOption[Map[String, String]](
longOption = "params",
toAnnotationSeq = (a: Map[String, String]) => Seq(ParameterAnnotation(a)),
helpText = """a comma separated, space free list of additional paramters, e.g. --param-string "k1=7,k2=dog" """
)
)
}
trait ParameterCli {
this: Shell =>
Seq(ParameterAnnotation).foreach(_.addOptions(parser))
}
class GenericParameterCliStage[P: Mappable](thunk: (P, AnnotationSeq) => Unit, default: P) extends Stage {
def mapify(p: P) = implicitly[Mappable[P]].toMap(p)
def materialize(map: Map[String, String]) = implicitly[Mappable[P]].fromMap(map)
val shell: Shell = new Shell("chiseltest") with ParameterCli with ChiselCli with FirrtlCli
def run(annotations: AnnotationSeq): AnnotationSeq = {
val params = annotations
.collectFirst {case ParameterAnnotation(map) => materialize(mapify(default) ++ map.toSeq)}
.getOrElse(default)
thunk(params, annotations)
annotations
}
}
The GCD source file
// See README.md for license details.
package gcd
import firrtl._
import chisel3._
import chisel3.stage.{ChiselStage, ChiselGeneratorAnnotation}
import firrtl.options.{StageMain}
// Both have to be imported
import mappable._
import cliparams._
case class GCDConfig(
len: Int = 16,
validHigh: Boolean = true
)
/**
* Compute GCD using subtraction method.
* Subtracts the smaller from the larger until register y is zero.
* value in register x is then the GCD
*/
class GCD (val conf: GCDConfig = GCDConfig()) extends Module {
val io = IO(new Bundle {
val value1 = Input(UInt(conf.len.W))
val value2 = Input(UInt(conf.len.W))
val loadingValues = Input(Bool())
val outputGCD = Output(UInt(conf.len.W))
val outputValid = Output(Bool())
})
val x = Reg(UInt())
val y = Reg(UInt())
when(x > y) { x := x - y }
.otherwise { y := y - x }
when(io.loadingValues) {
x := io.value1
y := io.value2
}
io.outputGCD := x
if (conf.validHigh) {
io.outputValid := y === 0.U
} else {
io.outputValid := y =/= 0.U
}
}
class GCDGenStage extends GenericParameterCliStage[GCDConfig]((params, annotations) => {
(new chisel3.stage.ChiselStage).execute(
Array("-X", "verilog"),
Seq(ChiselGeneratorAnnotation(() => new GCD(params))))}, GCDConfig())
object GCDGen extends StageMain(new GCDGenStage)
and the tests
// See README.md for license details.
package gcd
import chisel3._
import firrtl._
import chisel3.tester._
import org.scalatest.FreeSpec
import chisel3.experimental.BundleLiterals._
import chiseltest.internal._
import chiseltest.experimental.TestOptionBuilder._
import firrtl.options.{StageMain}
import mappable._
import cliparams._
class GCDSpec(params: GCDConfig, annotations: AnnotationSeq = Seq()) extends FreeSpec with ChiselScalatestTester {
"Gcd should calculate proper greatest common denominator" in {
test(new GCD(params)) { dut =>
dut.io.value1.poke(95.U)
dut.io.value2.poke(10.U)
dut.io.loadingValues.poke(true.B)
dut.clock.step(1)
dut.io.loadingValues.poke(false.B)
while (dut.io.outputValid.peek().litToBoolean != dut.conf.validHigh) {
dut.clock.step(1)
}
dut.io.outputGCD.expect(5.U)
}
}
}
class GCDTestStage extends GenericParameterCliStage[GCDConfig]((params, annotations) => {
(new GCDSpec(params, annotations)).execute()}, GCDConfig())
object GCDTest extends StageMain(new GCDTestStage)
Both, generation and tests can be parameterized via CLI as in the OQ:
sbt 'runMain gcd.GCDGen --params "len=12,validHigh=false"'
sbt 'test:runMain gcd.GCDTest --params "len=12,validHigh=false"'