(scala)
Files.walk(Paths.get("")).forEach(x => log.info(x.toString))
gives
Error:(21, 16) missing parameter type
.forEach(x => log.info(x.toString))
^
and (java8)
Files.walk(Paths.get("")).forEach(x -> System.out.println(x.toString()));
works fine
What's wrong?
stream.forEach(x -> foo()) in java is syntactic sugar for
stream.forEach(
new Consumer<Path> { public void accept(Path x) { foo(); } }
)
This is not at all the same as x => ... in scala, which is an instance of Function[Path,Unit].
Try this;
Files.walk(Paths.get(""))
.forEach(new Consumer[Path] { def accept(s: Path) = println(s) })
An alternative route: instead of converting your scala function to a java consumer, you can convert the java stream to a scala stream and use normal scala functions.
scala> import scala.collection.JavaConverters._
scala> import java.nio.file._
scala> val files = Files.walk(Paths.get("/tmp")).iterator.asScala.toStream
files: scala.collection.immutable.Stream[java.nio.file.Path] = Stream(/tmp, ?)
files.foreach(println(_))
Scala 2.12 comes with much better Java 8 interoperability, check the Scala 2.12 announcement; as a result, your code as written compiles just fine in 2.12:
Files.walk(Paths.get("")).forEach(x => System.out.println(x.toString))
If you need this to work in 2.11, use scala-java8-compat. Here's the dependency
libraryDependencies += "org.scala-lang.modules" %% "scala-java8-compat" % "0.8.0"
and in this case you can use it like this:
import scala.compat.java8.FunctionConverters._
Files.walk(Paths.get("")).forEach( asJavaConsumer { x => println(x.toString) } )
Related
I have some code that works in Scala 2.{10,11,12,13} that I'm now trying to convert to Scala 3. Scala 3 does Enumeration differently than Scala 2. I'm trying to figure out how to convert the following code that interacts with play-json so that it will work with Scala 3. Any tips or pointers to code from projects that have already crossed this bridge?
// Scala 2.x style code in EnumUtils.scala
import play.api.libs.json._
import scala.language.implicitConversions
// see: http://perevillega.com/enums-to-json-in-scala
object EnumUtils {
def enumReads[E <: Enumeration](enum: E): Reads[E#Value] =
new Reads[E#Value] {
def reads(json: JsValue): JsResult[E#Value] = json match {
case JsString(s) => {
try {
JsSuccess(enum.withName(s))
} catch {
case _: NoSuchElementException =>
JsError(s"Enumeration expected of type: '${enum.getClass}', but it does not appear to contain the value: '$s'")
}
}
case _ => JsError("String value expected")
}
}
implicit def enumWrites[E <: Enumeration]: Writes[E#Value] = new Writes[E#Value] {
def writes(v: E#Value): JsValue = JsString(v.toString)
}
implicit def enumFormat[E <: Enumeration](enum: E): Format[E#Value] = {
Format(EnumUtils.enumReads(enum), EnumUtils.enumWrites)
}
}
// ----------------------------------------------------------------------------------
// Scala 2.x style code in Xyz.scala
import play.api.libs.json.{Reads, Writes}
object Xyz extends Enumeration {
type Xyz = Value
val name, link, unknown = Value
implicit val enumReads: Reads[Xyz] = EnumUtils.enumReads(Xyz)
implicit def enumWrites: Writes[Xyz] = EnumUtils.enumWrites
}
As an option you can switch to jsoniter-scala.
It supports enums for Scala 2 and Scala 3 out of the box.
Also it has handy derivation of safe and efficient JSON codecs.
Just need to add required libraries to your dependencies:
libraryDependencies ++= Seq(
// Use the %%% operator instead of %% for Scala.js and Scala Native
"com.github.plokhotnyuk.jsoniter-scala" %% "jsoniter-scala-core" % "2.13.5",
// Use the "provided" scope instead when the "compile-internal" scope is not supported
"com.github.plokhotnyuk.jsoniter-scala" %% "jsoniter-scala-macros" % "2.13.5" % "compile-internal"
)
And then derive a codec and use it:
import com.github.plokhotnyuk.jsoniter_scala.core._
import com.github.plokhotnyuk.jsoniter_scala.macros._
implicit val codec: JsonValueCodec[Xyz.Xyz] = JsonCodecMaker.make
println(readFromString[Xyz.Xyz]("\"name\""))
BTW, you can run the full code on Scastie: https://scastie.scala-lang.org/Evj718q6TcCZow9lRhKaPw
My question is how do I access DownloadFiles.fileURLList property in sbt console (Scala REPL)?
I created a SBT Scala project and have this code at src/main/scala/DownloadFiles.scala
import sys.process._
import biz.neumann.url.NiceURLCodecs._
import java.net._
import java.io._
object DownloadFiles extends App {
val fileURLList = Array(
"https://www.example.com/file1.txt",
"https://www.example.com/file2.txt",
"https://www.example.com/file3.txt"
)
fileURLList.foreach(url => {
val fileName = "downloaded-files/" + new File(new URI(url).getPath).getName;
println(url.decode.encode + ": " + fileName)
new URL(url.decode.encode) #> new File(fileName) !!
})
}
I opened Scala REPL by using sbt console and when I access fileURLList property of DownloadFiles object I get null value (unevaluated) as shown below.
scala> DownloadFiles.fileURLList
res0: Array[String] = null
What I instead need is evaluated value of the property of that object.
Here's the build.sbt if it helps:
scalaVersion := "2.12.10"
name := "image-downloader"
organization := "in.bhargav.inc"
version := "1.0"
libraryDependencies += "org.scala-lang.modules" %% "scala-parser-combinators" % "1.1.2"
libraryDependencies += "biz.neumann" % "nice-url-encode-decode_2.12" % "1.5"
This is a consequence of Hello extending App which indirectly extends DalayedInit which changes the initialisation semantics:
#nowarn("""cat=deprecation&origin=scala\.DelayedInit""")
trait App extends DelayedInit
Quick fix is to make fileURLList lazy
➜ scala
Welcome to Scala 2.13.5 (OpenJDK 64-Bit Server VM, Java 1.8.0_282).
Type in expressions for evaluation. Or try :help.
scala> object DownloadFiles extends App {
| val fileURLList = Array("https://www.example.com/file1.txt")
| }
object DownloadFiles
scala> DownloadFiles.fileURLList
val res1: Array[String] = null
scala> object DownloadFiles extends App {
| lazy val fileURLList = Array("https://www.example.com/file1.txt")
| }
object DownloadFiles
scala> DownloadFiles.fileURLList
val res2: Array[String] = Array(https://www.example.com/file1.txt)
This behaviour changed in Scala 3 which drops DelayedInit semantics
➜ scala3-repl
scala> object DownloadFiles extends App {
| val fileURLList = Array("https://www.example.com/file1.txt")
| }
// defined object DownloadFiles
scala> DownloadFiles.fileURLList
val res0: Array[String] = Array(https://www.example.com/file1.txt)
The previous functionality of App, which relied on the "magic"
DelayedInit trait, is no longer available. App still exists in limited
form for now, but it does not support command line arguments and will
be deprecated in the future. If programs need to cross-build between
Scala 2 and Scala 3, it is recommended to use an explicit main method
with an Array[String] argument instead.
The first part of your code uses a variable called fileURLList. The second part of your code and your SBT command use imageURLList. imageURLList is never declared as a variable, therefore it is null. Find and replace your code from fileURLList to imageURLList and I bet it will do what you were expecting.
I am a little surprised you didn't get other errors though.
//package com.jsonReader
import play.api.libs.json._
import play.api.libs.json._
import play.api.libs.json.Reads._
import play.api.libs.json.Json.JsValueWrapper
import org.apache.spark._
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.SQLContext
//import org.apache.spark.implicits._
//import sqlContext.implicits._
object json {
def flatten(js: JsValue, prefix: String = ""): JsObject = js.as[JsObject].fields.foldLeft(Json.obj()) {
case (acc, (k, v: JsObject)) => {
val nk = if(prefix.isEmpty) k else s"$prefix.$k"
acc.deepMerge(flatten(v, nk))
}
case (acc, (k, v: JsArray)) => {
val nk = if(prefix.isEmpty) k else s"$prefix.$k"
val arr = flattenArray(v, nk).foldLeft(Json.obj())(_++_)
acc.deepMerge(arr)
}
case (acc, (k, v)) => {
val nk = if(prefix.isEmpty) k else s"$prefix.$k"
acc + (nk -> v)
}
}
def flattenArray(a: JsArray, k: String = ""): Seq[JsObject] = {
flattenSeq(a.value.zipWithIndex.map {
case (o: JsObject, i: Int) =>
flatten(o, s"$k[$i]")
case (o: JsArray, i: Int) =>
flattenArray(o, s"$k[$i]")
case a =>
Json.obj(s"$k[${a._2}]" -> a._1)
})
}
def flattenSeq(s: Seq[Any], b: Seq[JsObject] = Seq()): Seq[JsObject] = {
s.foldLeft[Seq[JsObject]](b){
case (acc, v: JsObject) =>
acc:+v
case (acc, v: Seq[Any]) =>
flattenSeq(v, acc)
}
}
def main(args: Array[String]) {
val appName = "Stream example 1"
val conf = new SparkConf().setAppName(appName).setMaster("local[*]")
//val spark = new SparkContext(conf)
val sc = new SparkContext(conf)
//val sqlContext = new SQLContext(sc)
val sqlContext=new SQLContext(sc);
//val spark=sqlContext.sparkSession
val spark = SparkSession.builder().appName("json Reader")
val df = sqlContext.read.json("C://Users//ashda//Desktop//test.json")
val set = df.select($"user",$"status",$"reason",explode($"dates")).show()
val read = flatten(df)
read.printSchema()
df.show()
}
}
I'm trying to use this code to flatten a higly nested json. For this I created a project and converted it to a maven project. I edited the pom.xml and included the libraries I needed but when I run program it says "Error: Could not find or load main class".
I tried converting the code to sbt project and then run but I get the same error. I tried packaging the code and run through spark-submit which gives me same error. Please let me know what am I missing here. I have tried I could for this.
Thanks
Hard to say, but maybe you have many classes that qualify as main so the build tool does not know which one to choose. Maybe try to clean the project first sbt clean.
Anyway in scala the preferred way to define a main class is to extend the App -trait.
object SomeApp extends App
Then the whole object body will become your main method.
You can also define in your build.sbt the main class. This is necessary if you have many objects that extend the App -trait.
mainClass in (Compile, run) := Some("io.example.SomeApp")
I am answering this question for sbt configurations. I also got the same issues which I resolved recently and made some basic mistakes which I would like you to note :
1. Configure your sbt file
go to build.sbt file and see that the scala version you are using is compatible with spark.As per version 2.4.0 of spark https://spark.apache.org/docs/latest/ ,scala version required is 2.11.x and not 2.12.x . So, even though your IDE (Eclipse/IntelliJ) shows the latest version of scala or the version you downloaded, change it to compatible version. Also, include this line of code
libraryDependencies += "org.scala-lang" % "scala-library" % "2.11.6"
2.11.x is your scala version
2. File Hierarchy
Make sure your Scala file is under /src/main/scala package only
3. Terminal
If your IDE allows you to launch terminal within it, launch it(IntelliJ allows, Not sure of Eclipse or any other) OR Go to terminal and change directory to your project directory
then run :
sbt clean
This will clear any libraries loaded previously or folders created after compilation.
sbt package
This will pack your files into a single jar file under target/scala-/ package
Then submit to spark :
spark-submit target/scala-<version>/<.jar file> --class "<ClassName>(In your case , com.jsonReader.json)" --jars target/scala-<version>/<.jar file> --master local[*]
Note here that -- if specified in a program isnt required here
I have found an example code for a Scala runtime scripting in answer to Generating a class from string and instantiating it in Scala 2.10, however the code seems to be obsolete for 2.11 - I cannot find any function corresponding to build.setTypeSignature. Even if it worked, the code seems hard to read and follow to me.
How can Scala scripts be compiled and executed in Scala 2.11?
Let us assume I want following:
define several variables (names and values)
compile script
(optional improvement) change variable values
execute script
For simplicity consider following example:
I want to define following variables (programmatically, from the code, not from the script text):
val a = 1
val s = "String"
I want a following script to be compiled and on execution a String value "a is 1, s is String" returned from it:
s"a is $a, s is $s"
How should my functions look like?
def setupVariables() = ???
def compile() = ???
def changeVariables() = ???
def execute() : String = ???
Scala 2.11 adds a JSR-223 scripting engine. It should give you the functionality you are looking for. Just as a reminder, as with all of these sorts of dynamic things, including the example listed in the description above, you will lose type safety. You can see below that the return type is always Object.
Scala REPL Example:
scala> import javax.script.ScriptEngineManager
import javax.script.ScriptEngineManager
scala> val e = new ScriptEngineManager().getEngineByName("scala")
e: javax.script.ScriptEngine = scala.tools.nsc.interpreter.IMain#566776ad
scala> e.put("a", 1)
a: Object = 1
scala> e.put("s", "String")
s: Object = String
scala> e.eval("""s"a is $a, s is $s"""")
res6: Object = a is 1, s is String`
An addition example as an application running under scala 2.11.6:
import javax.script.ScriptEngineManager
object EvalTest{
def main(args: Array[String]){
val e = new ScriptEngineManager().getEngineByName("scala")
e.put("a", 1)
e.put("s", "String")
println(e.eval("""s"a is $a, s is $s""""))
}
}
For this application to work make sure to include the library dependency.
libraryDependencies += "org.scala-lang" % "scala-compiler" % scalaVersion.value
Does TypeSafe Slick work on Scala 2.9.3? I get
[ERROR] exception when typing query.list
exception when typing query.listscala.tools.nsc.symtab.Types$TypeError: class file needed by StatementInvoker is missing.
[INFO] class file needed by StatementInvoker is missing.
[INFO] reference type Either of object package refers to nonexisting symbol.
which goes away when I use Scala 2.10.x, but I'm too new to Scala to understand why.
import slick.session.Database
import scala.slick.jdbc.StaticQuery
import Database.threadLocalSession
import com.typesafe.config.ConfigFactory
object PostgresDao {
protected val conf = ConfigFactory.load
def findFoo(a: Int, b: String): Option[Int] = {
Database.forURL("jdbc:postgresql://localhost/bar", driver = "org.postgresql.Driver") withSession {
val query = StaticQuery.query[(Int, String), Int](
"""
select some_int
from some_table t
where t.a = ? and t.b = ?
""".stripMargin)
val list: List[Int] = query.list((a, b))
if (list.isEmpty) {
None
}
else {
Some(list.head)
}
}
}
I shamelessly copy-past official doc here:
Slick requires Scala
2.10. (For Scala 2.9 please use ScalaQuery, the
predecessor of Slick).