I am new to scala. Please be gentle.
The import below imports everything (every class, trait and object) under ml.
import org.apache.spark.ml._
but NOT ParamMap, which is under
import org.apache.spark.ml.param._
In other words, for the code below, if I do:
import org.apache.spark.ml.param._
import org.apache.spark.ml._
class Kmeans extends Transformer {
def copy(extra: ParamMap): Unit = {
defaultCopy(extra)
}}
Then I have no import errors, but if I comment import org.apache.spark.ml.param._:
//import org.apache.spark.ml.param._
import org.apache.spark.ml._
class Kmeans extends Transformer {
def copy(extra: ParamMap): Unit = {
defaultCopy(extra)
}}
It gives an import error on ParamMap.
Question
why isn't this import org.apache.spark.ml.param.ParamMap included import org.apache.spark.ml.param._
Scala imports are not recursive - import org.apache.spark.ml._ means import all classes and fields directly under ml package but not the ones under its sub-packages.
Since ParamMap is under one of ml's sub-packages (ml.param), you'll have to import that package or ParamMap class directly.
Related
I'm working on scala streaming unit test but while reading from csv file getting ArrayOutOfBoundsException
Code :
import org.scalatest.matchers.should.Matchers
import org.scalatest.wordspec.AnyWordSpecLike
import org.apache.spark.sql.SparkSession
class StreamingTest extends AnyWordSpecLike with Matchers {
val sparkses = SparkSession.builder.appName("MyApp").config("spark.master","local").getOrCreate()
val df = sparkses.read.format("csv").load("file.csv")
df.printSchema()
}
The code works fine without extending AnyWordSpecLike with Matchers, but we need it to work with EmbeddedKafka.
Any guidance would be helpful.
I am using Play 2.8.7, Scala 2.13.4.
import play.api.mvc._
import play.api.mvc.Results._
import play.mvc.Controller
import play.api.i18n.{I18nSupport, MessagesApi}
import javax.inject._
class Application #Inject() (
val messagesApi: MessagesApi
) extends Controller with I18nSupport {
def greeting = Action { implicit request =>
Ok("hello")
}
}
1. I want to import play.api.mvc.Results.Ok — why is it only imported when I do import play.api.mvc.Results._ but not when I only do import play.api.mvc._?
The latter used to work when I used Play 2.4.3 (Scala 2.11.11).
2. The compiler cannot resolve symbol "Action". Why is that...? I did import play.api.mvc._
UPDATE:
There was a suggestion to import play.mvc.BaseController.
It seems not to exist in Play 2.8.7.
"Controller" is for Java, so you should use "play.mvc.BaseController" in Scala.
Q1 & Q2 both would be resolved if you are using "play.mvc.BaseController". " import play.api.mvc.Results._" is also unnecessary.
import play.mvc.Controller and play.mvc.Result and extends your class to Controller.
Example:
import play.mvc.Controller;
import play.mvc.Result;
public class ClassName extends Controller {
//code
}
I have the following code snippet:
package org.test.test.datahelper
import org.apache.spark.rdd.RDD
import org.apache.spark.sql._
class WeatherHelper(sparkSession: SparkSession, weather: DataFrame) {
def prepareRRRColumn: DataFrame = {
import org.apache.spark.sql.functions
weather.withColumn("Year", year(col("DateTime")))
weather
}
}
The problem is that Scala (or probably IntelliJ IDEA) does not see method year just like col (Cannot resolve symbol year and col respectfully), despite the necessary import is just one line above (however, it doesn't work even if the import is global). Following to the source code of org.apache.spark.sql.functions I found the following lines:
def col(colName : scala.Predef.String) : org.apache.spark.sql.Column = { /* compiled code */ }
def year(e : org.apache.spark.sql.Column) : org.apache.spark.sql.Column = { /* compiled code */ }
i.e. both methods do exist. What am I doing wrong?
This is more of a scala import syntax issue.
To import the methods(col,year) within the class/package function you have to use.
import org.apache.spark.sql.functions._
// Or import only specific functions
import org.apache.spark.sql.functions.{col, year}
Instead of
import org.apache.spark.sql.functions
I am trying to build custom database column converters for a new Slick 3 project. It's pretty easy to make these using the MappedColumnType, but you have to have imported the driver api. For a one-off type in a single DAO class, this is straight forward. But I would like to use my custom column types across all my DAO objects. I have been unable to construct my import in a way that the compiler can recognize the implicits.
Here is an example of the type of library I would like to construct. It has a single converter, very similar to the ubiquitous Joda date converter seen in many Slick 2 examples.
package dao
import java.sql.Date
import data.Timestamp
import play.api.db.slick.{DatabaseConfigProvider, HasDatabaseConfigProvider}
import slick.driver.JdbcProfile
case class StandardConversions(protected val dbConfigProvider: DatabaseConfigProvider)
extends HasDatabaseConfigProvider[JdbcProfile] {
import driver.api._
implicit val timestampColumnType = MappedColumnType.base[Timestamp, Date](
{ data => new Date(data.value) },
{ sql => Timestamp(sql.getTime) }
)
}
In the DAO class I try doing the import like this:
val conversions = StandardConversions(dbConfigProvider)
import conversions._
The compiler error is the familiar:
could not find implicit value for parameter tt: slick.ast.TypedType[data.Timestamp]
I'm basically stuck in dependency injection, implicit hell. Has anybody come up with a good way to maintain their custom conversions in Slick 3? Please share.
This is where traits come in handy:
package dao
import java.sql.Date
import data.Timestamp
import play.api.db.slick.HasDatabaseConfig
import slick.driver.JdbcProfile
trait StandardConversions extends HasDatabaseConfigProvider[JdbcProfile] {
import driver.api._
implicit val timestampColumnType = MappedColumnType.base[Timestamp, Date](
{ data => new Date(data.value) },
{ sql => Timestamp(sql.getTime) }
)
}
And then simply extend from this trait in your DAOs:
class SomeDAO #Inject()(protected val dbConfigProvider: DatabaseConfigProvider)
extends HasDatabaseConfigProvider[JdbcProfile]
with StandardConversions {
import driver.api._
// all implicits of StandardConversions are in scope here
}
In combination with Roman's solution, you should probably add the following import:
import play.api.libs.concurrent.Execution.Implicits.defaultContext
I need to compile a query in Slick with Play and PostgreSQL
val bioMaterialTypes: TableQuery[Tables.BioMaterialType] = Tables.BioMaterialType
def getAllBmts() = for{ bmt <- bioMaterialTypes } yield bmt
val queryCompiled = Compiled(getAllBmts _)
but in Scala IDE I get this error in the Apply of Compiled
Multiple markers at this line
- Computation of type () => scala.slick.lifted.Query[models.Tables.BioMaterialType,models.Tables.BioMaterialTypeRow,Seq]
cannot be compiled (as type C)
- not enough arguments for method apply: (implicit compilable: scala.slick.lifted.Compilable[() =>
scala.slick.lifted.Query[models.Tables.BioMaterialType,models.Tables.BioMaterialTypeRow,Seq],C], implicit driver:
scala.slick.profile.BasicProfile)C in object Compiled. Unspecified value parameters compilable, driver.
This are my imports:
import scala.concurrent.Future
import scala.slick.jdbc.StaticQuery.staticQueryToInvoker
import scala.slick.lifted.Compiled
import scala.slick.driver.PostgresDriver
import javax.inject.Inject
import javax.inject.Singleton
import models.BioMaterialType
import models.Tables
import play.api.Application
import play.api.db.slick.Config.driver.simple.TableQuery
import play.api.db.slick.Config.driver.simple.columnExtensionMethods
import play.api.db.slick.Config.driver.simple.longColumnType
import play.api.db.slick.Config.driver.simple.queryToAppliedQueryInvoker
import play.api.db.slick.Config.driver.simple.queryToInsertInvoker
import play.api.db.slick.Config.driver.simple.stringColumnExtensionMethods
import play.api.db.slick.Config.driver.simple.stringColumnType
import play.api.db.slick.Config.driver.simple.valueToConstColumn
import play.api.db.slick.DB
import play.api.db.slick.DBAction
You can simply do
val queryCompiled = Compiled(bioMaterialTypes)