I am new to chisel and I have designed basic circuits and a simple risc-v processor with it but I still can't figure out how the scope works in Scala/ Chisel. Considering the following simple counter:
package example
import chisel3._
class GenericCounter(n: Int) extends Module {
val io = IO(new Bundle {
val ld = Input(UInt(1.W))
val cld = Input(UInt(log2Ceil(n).W))
val cout = Output(UInt(log2Ceil(n).W))
})
val cnt = RegInit(0.asUInt(n.W))
when(io.ld === 1.U){
cnt := io.cld
} .otherwise{
cnt := Mux(cnt===100.U, 0.U, cnt + 1.U)
}
io.cout := cnt
}
While trying to compile the above code, the compiler give an error that log2ceil is not defined. But if I use util.log2ceil it works fine. This is true for all util functions such as Cat, isPow2 etc. I know that the import chisel3._ should have imported all the necessary functions but it seems that I am missing something here. Can someone help me out?
In Scala, importing all of the contents of a package does not import the contents of any subpackages. Thus if you wish to import the contents of chisel.util you should also write import chisel3.util._
Related
I'm trying to defer ScalaJS Linking to runtime, which allows multi-stage compilation to be more flexible and less dependent on sbt.
The setup looks like this:
Instead of using scalajs-sbt plugin, I chose to invoke scalajs-compiler directly as a scala compiler plugin:
scalaCompilerPlugins("org.scala-js:scalajs-compiler_${vs.scalaV}:${vs.scalaJSV}")
This can successfully generate the "sjsir" files under project output directory, but no further.
Use the solution in this post:
Build / Compile latest SalaJS (1.3+) using gradle on a windows machine?
"Linking scala.js yourself" to invoke the linker on all the compiled sjsir files to produce js files, this is my implementation:
in compile-time & runtime dependencies, add scalajs basics and scalajs-linker:
bothImpl("org.scala-js:scalajs-library_${vs.scalaBinaryV}:${vs.scalaJSV}")
bothImpl("org.scala-js:scalajs-linker_${vs.scalaBinaryV}:${vs.scalaJSV}")
bothImpl("org.scala-js:scalajs-dom_${vs.scalaJSSuffix}:2.1.0")
Write the following code:
import org.scalajs.linker.interface.{Report, StandardConfig}
import org.scalajs.linker.{PathIRContainer, PathOutputDirectory, StandardImpl}
import org.scalajs.logging.{Level, ScalaConsoleLogger}
import java.nio.file.{Path, Paths}
import java.util.Collections
import scala.concurrent.duration.Duration
import scala.concurrent.{Await, ExecutionContext}
object JSLinker {
implicit def gec = ExecutionContext.global
def link(classpath: Seq[Path], outputDir: Path): Report = {
val logger = new ScalaConsoleLogger(Level.Warn)
val linkerConfig = StandardConfig() // look at the API of this, lots of options.
val linker = StandardImpl.linker(linkerConfig)
// Same as scalaJSModuleInitializers in sbt, add if needed.
val moduleInitializers = Seq()
val cache = StandardImpl.irFileCache().newCache
val result = PathIRContainer
.fromClasspath(classpath)
.map(_._1)
.flatMap(cache.cached _)
.flatMap(linker.link(_, moduleInitializers, PathOutputDirectory(outputDir), logger))
Await.result(result, Duration.Inf)
}
def linkClasses(outputDir: Path = Paths.get("./")): Report = {
import scala.jdk.CollectionConverters._
val cl = Thread.currentThread().getContextClassLoader
val resources = cl.getResources("")
val rList = Collections.list(resources).asScala.toSeq.map { v =>
Paths.get(v.toURI)
}
link(rList, outputDir)
}
lazy val linkOnce = {
linkClasses()
}
}
The resources detection was successful, all roots containing sjsir are detected:
rList = {$colon$colon#1629} "::" size = 4
0 = {UnixPath#1917} "/home/peng/git-scaffold/scaffold-gradle-kts/build/classes/scala/test"
1 = {UnixPath#1918} "/home/peng/git-scaffold/scaffold-gradle-kts/build/classes/scala/testFixtures"
2 = {UnixPath#1919} "/home/peng/git-scaffold/scaffold-gradle-kts/build/classes/scala/main"
3 = {UnixPath#1920} "/home/peng/git-scaffold/scaffold-gradle-kts/build/resources/main"
But linking still fails:
Fatal error: java.lang.Object is missing
called from core module analyzer
There were linking errors
org.scalajs.linker.interface.LinkingException: There were linking errors
at org.scalajs.linker.frontend.BaseLinker.reportErrors$1(BaseLinker.scala:91)
at org.scalajs.linker.frontend.BaseLinker.$anonfun$analyze$5(BaseLinker.scala:100)
at scala.concurrent.impl.Promise$Transformation.run$$$capture(Promise.scala:467)
at scala.concurrent.impl.Promise$Transformation.run(Promise.scala)
at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402)
at java.util.concurrent.ForkJoinTask.doExec$$$capture(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
I wonder what this error message entails. Clearly java.lang.Object is not compiled into sjsir. Does this error message make sense? How do I fix it?
Thanks to #sjrd I now have the correct runtime compilation stack. There are 2 problems in my old settings:
It turns out that cl.getResources("") is indeed not able to infer all classpath, so I switch to system property java.class.path, which contains classpaths of all dependencies
moduleInitializers has to be manually set to point to a main method, which will be invoked when the js function is called.
After correcting them, the compilation class becomes:
import org.scalajs.linker.interface.{ModuleInitializer, Report, StandardConfig}
import org.scalajs.linker.{PathIRContainer, PathOutputDirectory, StandardImpl}
import org.scalajs.logging.{Level, ScalaConsoleLogger}
import java.nio.file.{Files, Path, Paths}
import scala.concurrent.duration.Duration
import scala.concurrent.{Await, ExecutionContext, ExecutionContextExecutor}
object JSLinker {
implicit def gec: ExecutionContextExecutor = ExecutionContext.global
val logger = new ScalaConsoleLogger(Level.Info) // TODO: cannot be lazy val, why?
lazy val linkerConf: StandardConfig = {
StandardConfig()
} // look at the API of this, lots of options.
def link(classpath: Seq[Path], outputDir: Path): Report = {
val linker = StandardImpl.linker(linkerConf)
// Same as scalaJSModuleInitializers in sbt, add if needed.
val moduleInitializers = Seq(
ModuleInitializer.mainMethodWithArgs(SlinkyHelloWorld.getClass.getName.stripSuffix("$"), "main")
)
Files.createDirectories(outputDir)
val cache = StandardImpl.irFileCache().newCache
val result = PathIRContainer
.fromClasspath(classpath)
.map(_._1)
.flatMap(cache.cached _)
.flatMap { v =>
linker.link(v, moduleInitializers, PathOutputDirectory(outputDir), logger)
}
Await.result(result, Duration.Inf)
}
def linkClasses(outputDir: Path = Paths.get("./ui/build/js")): Report = {
val rList = getClassPaths
link(rList, outputDir)
}
def getClassPaths: Seq[Path] = {
val str = System.getProperty("java.class.path")
val paths = str.split(':').map { v =>
Paths.get(v)
}
paths
}
lazy val linkOnce: Report = {
val report = linkClasses()
logger.info(
s"""
|=== [Linked] ===
|${report.toString()}
|""".stripMargin
)
report
}
}
This is all it takes to convert sjsir artefacts to a single main.js file.
Is there a way to list the tags associated with a Task in sbt?
inspect and show don’t seem to have anything there.
The cool and powerful aspect of sbt is that the build definition it generates is a regular Scala application meaning we can inspect its objects, like we would in any other Scala application, by simply invoking member methods to query their state. Executing sbt starts the REPL for the special DSL build language, however we can drop to a lower level by executing
sbt consoleProject
to start true Scala REPL:
starts the Scala interpreter with access to your project definition
and to sbt... consoleProject can be useful for creating and
modifying your build in the same way that the Scala interpreter is
normally used to explore writing code. Note that this gives you raw
access to your build.
There exists a public tags method
final case class Task[T](info: Info[T], work: Action[T]) {
...
def tags: TagMap = info get tagsKey getOrElse TagMap.empty
}
so there must be a way to invoke it (even if there might not be a ready top level command for it such as inspect). Say we have the following tagged task definition in build.sbt
lazy val hello = taskKey[Unit]("Vulcan greeting")
hello := Def.task(println("Live long and prosper")).tag(Tags.CPU, Tags.Compile).value
After executing consoleProject our build definition is imported
scala> import _root_.scala.xml.{TopScope=>$scope}
import _root_.sbt._
import _root_.sbt.Keys._
import _root_.sbt.nio.Keys._
import _root_.sbt.ScriptedPlugin.autoImport._
import _root_.sbt.plugins.IvyPlugin
import _root_.sbt.plugins.JvmPlugin
import _root_.sbt.plugins.CorePlugin
import _root_.sbt.ScriptedPlugin
import _root_.sbt.plugins.SbtPlugin
import _root_.sbt.plugins.SemanticdbPlugin
import _root_.sbt.plugins.JUnitXmlReportPlugin
import _root_.sbt.plugins.Giter8TemplatePlugin
import $d408b7d79eabe42459a4.root
import currentState._
import extracted._
import cpHelpers._
Now we can make use of Extracted#get to get the TaskKey and explore it like so
scala> extracted.get(hello).tags
res1: sbt.ConcurrentRestrictions.TagMap = Map(Tag(cpu) -> 1, Tag(compile) -> 1)
Furthermore, note the import $d408b7d79eabe42459a4. We can use this object to access regular val/def members, for example, say we had defined in build.sbt
def helloTask = Def.task { println("Live long and prosper") } tag(Tags.CPU, Tags.Compile)
then we could access helloTask like so
scala> $d408b7d79eabe42459a4.helloTask.evaluate(structure.data).tags
res0: sbt.ConcurrentRestrictions.TagMap = Map(Tag(cpu) -> 1, Tag(compile) -> 1)
Both approaches show the required Map(Tag(cpu) -> 1, Tag(compile) -> 1).
Addressing the comment compileTask does not seem to be tagged thus
scala> get(Compile/compile).tags
res8: sbt.ConcurrentRestrictions.TagMap = Map()
however, for example, updateFull task is indeed tagged
updateFull := (updateTask tag (Tags.Update, Tags.Network)).value
hence
scala> get(updateFull).tags
res9: sbt.ConcurrentRestrictions.TagMap = Map(Tag(update) -> 1, Tag(network) -> 1)
//package com.jsonReader
import play.api.libs.json._
import play.api.libs.json._
import play.api.libs.json.Reads._
import play.api.libs.json.Json.JsValueWrapper
import org.apache.spark._
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.SQLContext
//import org.apache.spark.implicits._
//import sqlContext.implicits._
object json {
def flatten(js: JsValue, prefix: String = ""): JsObject = js.as[JsObject].fields.foldLeft(Json.obj()) {
case (acc, (k, v: JsObject)) => {
val nk = if(prefix.isEmpty) k else s"$prefix.$k"
acc.deepMerge(flatten(v, nk))
}
case (acc, (k, v: JsArray)) => {
val nk = if(prefix.isEmpty) k else s"$prefix.$k"
val arr = flattenArray(v, nk).foldLeft(Json.obj())(_++_)
acc.deepMerge(arr)
}
case (acc, (k, v)) => {
val nk = if(prefix.isEmpty) k else s"$prefix.$k"
acc + (nk -> v)
}
}
def flattenArray(a: JsArray, k: String = ""): Seq[JsObject] = {
flattenSeq(a.value.zipWithIndex.map {
case (o: JsObject, i: Int) =>
flatten(o, s"$k[$i]")
case (o: JsArray, i: Int) =>
flattenArray(o, s"$k[$i]")
case a =>
Json.obj(s"$k[${a._2}]" -> a._1)
})
}
def flattenSeq(s: Seq[Any], b: Seq[JsObject] = Seq()): Seq[JsObject] = {
s.foldLeft[Seq[JsObject]](b){
case (acc, v: JsObject) =>
acc:+v
case (acc, v: Seq[Any]) =>
flattenSeq(v, acc)
}
}
def main(args: Array[String]) {
val appName = "Stream example 1"
val conf = new SparkConf().setAppName(appName).setMaster("local[*]")
//val spark = new SparkContext(conf)
val sc = new SparkContext(conf)
//val sqlContext = new SQLContext(sc)
val sqlContext=new SQLContext(sc);
//val spark=sqlContext.sparkSession
val spark = SparkSession.builder().appName("json Reader")
val df = sqlContext.read.json("C://Users//ashda//Desktop//test.json")
val set = df.select($"user",$"status",$"reason",explode($"dates")).show()
val read = flatten(df)
read.printSchema()
df.show()
}
}
I'm trying to use this code to flatten a higly nested json. For this I created a project and converted it to a maven project. I edited the pom.xml and included the libraries I needed but when I run program it says "Error: Could not find or load main class".
I tried converting the code to sbt project and then run but I get the same error. I tried packaging the code and run through spark-submit which gives me same error. Please let me know what am I missing here. I have tried I could for this.
Thanks
Hard to say, but maybe you have many classes that qualify as main so the build tool does not know which one to choose. Maybe try to clean the project first sbt clean.
Anyway in scala the preferred way to define a main class is to extend the App -trait.
object SomeApp extends App
Then the whole object body will become your main method.
You can also define in your build.sbt the main class. This is necessary if you have many objects that extend the App -trait.
mainClass in (Compile, run) := Some("io.example.SomeApp")
I am answering this question for sbt configurations. I also got the same issues which I resolved recently and made some basic mistakes which I would like you to note :
1. Configure your sbt file
go to build.sbt file and see that the scala version you are using is compatible with spark.As per version 2.4.0 of spark https://spark.apache.org/docs/latest/ ,scala version required is 2.11.x and not 2.12.x . So, even though your IDE (Eclipse/IntelliJ) shows the latest version of scala or the version you downloaded, change it to compatible version. Also, include this line of code
libraryDependencies += "org.scala-lang" % "scala-library" % "2.11.6"
2.11.x is your scala version
2. File Hierarchy
Make sure your Scala file is under /src/main/scala package only
3. Terminal
If your IDE allows you to launch terminal within it, launch it(IntelliJ allows, Not sure of Eclipse or any other) OR Go to terminal and change directory to your project directory
then run :
sbt clean
This will clear any libraries loaded previously or folders created after compilation.
sbt package
This will pack your files into a single jar file under target/scala-/ package
Then submit to spark :
spark-submit target/scala-<version>/<.jar file> --class "<ClassName>(In your case , com.jsonReader.json)" --jars target/scala-<version>/<.jar file> --master local[*]
Note here that -- if specified in a program isnt required here
As a Slick noob I do not understand why I have to specify my model twice, first in Scala and then in 1.sql to create the tables. That does not look DRY. Is there an easy way to generate 1.sql (and 2..n.sql) from the model during development?
I created my own sbt task to easily generate the 1.sql from model using the code generation:
in build.sbt file:
val generate_schema = taskKey[Unit]("Schema Generator")
generate_schema <<= (fullClasspath in Runtime) map {classpath =>
val loader: ClassLoader = ClasspathUtilities.toLoader(classpath.map(_.data).map(_.getAbsoluteFile))
val schemaGenerator = loader.loadClass("misc.SchemaGenerator").newInstance().asInstanceOf[Runnable]
schemaGenerator.run
}
the misc.SchemaGenerator class:
package misc
import models.Article
import models.Category
import play.api.Logger
import slick.driver.PostgresDriver.api._
import scala.concurrent._
import ExecutionContext.Implicits.global
import scala.reflect.io.File
class SchemaGenerator extends Runnable {
def run = {
println("---------------------GENERATING SCHEMA.....")
val categories = TableQuery[Category]
val articles = TableQuery[Article]
val file = File("/home/pedro/NetBeansProjects/play-scala-angular-sample/my-blog-server/conf/evolutions/default/1.sql")
val sb = new StringBuilder("# --- !Ups \n\n")
categories.schema.create.statements.foreach(st => sb.append(st.toString + ";\n"))
sb.append("\n\n")
articles.schema.create.statements.foreach(st => sb.append(st.toString + ";\n"))
sb.append("\n\n")
sb.append("# --- !Downs")
sb.append("\n\n")
categories.schema.drop.statements.foreach(st => sb.append(st.toString + ";\n"))
sb.append("\n\n")
articles.schema.drop.statements.foreach(st => sb.append(st.toString + ";\n"))
// Logger.info("value: [" + sb + "] sb")
file.writeAll(sb.toString)
Logger.info("----------------------FINISHED GENERATING SCHEMA--------------------------")
}
}
You can execute the task from the activator console: generate_schema.
Hope it helps.
...ddl.create is no longer supported since slick 3.0. A motivation can be found at the bottom of the following page: https://www.playframework.com/documentation/2.4.x/PlaySlickMigrationGuide
So I have to hand edit my schemas or use code generation.
Is it possible to create a custom Arbitrary Generator in a ScalaTest (which mixins Checkers for ScalaCheck property) which is testing Java code? for e.g. following are the required steps for each test within forAll
val fund = new Fund()
val fundAccount = new Account(Account.RETIREMENT)
val consumer = new Consumer("John")
.createAccount(fundAccount)
fund.addConsumer(consumer)
fundAccount.deposit(amount)
above is a prep code before asserting results etc.
You sure can. This should get you started.
import org.scalacheck._
import Arbitrary._
import Prop._
case class Consumer(name:String)
object ConsumerChecks extends Properties("Consumer") {
lazy val genConsumer:Gen[Consumer] = for {
name <- arbitrary[String]
} yield Consumer(name)
implicit lazy val arbConsumer:Arbitrary[Consumer] = Arbitrary(genConsumer)
property("some prop") = forAll { c:Consumer =>
// Check stuff
true
}
}