I am trying to use HBase API in my Scala project, but I'm getting an error when I try:
import org.apache.hadoop.hbase
The error is "object hbase is not a member of package org.apache.hadoop"
I am using sbt 1.3.12 to build my project, this is a part of the build.sbt:
scalaVersion := "2.11.8"
libraryDependencies ++= Seq(
"org.apache.hadoop" % "hadoop-common" % "2.7.3",
"org.apache.hadoop" % "hadoop-client" % "2.7.3",
"org.apache.hbase" % "hbase-common" % "1.2.1",
"org.apache.hbase" % "hbase-client" % "1.2.1",
"org.apache.hbase" % "hbase-protocol" % "1.2.1",
"org.apache.hbase" % "hbase-server" % "1.2.1"
)
Do you know how to solve the issue?
It is probably the val hbaseVersion = that is in the middle of your build.sbt. Try to remove it:
scalaVersion := "2.11.8"
libraryDependencies ++= Seq(
"org.apache.hadoop" % "hadoop-common" % "2.7.3",
"org.apache.hadoop" % "hadoop-client" % "2.7.3",
"org.apache.hbase" % "hbase-common" % "1.2.1",
"org.apache.hbase" % "hbase-client" % "1.2.1",
"org.apache.hbase" % "hbase-protocol" % "1.2.1",
"org.apache.hbase" % "hbase-server" % "1.2.1"
)
Code run at Scastie.
Related
I want to load data from HBase by Spark SQL, I use the hbase-spark official example and it raise NullPointerException
My build.sbt file is:
name := "proj_1"
version := "0.1"
scalaVersion := "2.11.12"
libraryDependencies ++= Seq(
"org.apache.spark" % "spark-core_2.11" % "2.3.1",
"org.apache.spark" % "spark-sql_2.11" % "2.3.1",
"org.apache.spark" % "spark-mllib_2.11" % "2.3.1",
"org.apache.spark" % "spark-streaming_2.11" % "2.3.1",
"org.apache.spark" % "spark-hive_2.11" % "2.3.1",
"org.elasticsearch" % "elasticsearch-hadoop" % "6.4.0",
"org.apache.hadoop" % "hadoop-core" % "2.6.0-mr1-cdh5.15.1",
"org.apache.hbase" % "hbase" % "2.1.0",
"org.apache.hbase" % "hbase-server" % "2.1.0",
"org.apache.hbase" % "hbase-common" % "2.1.0",
"org.apache.hbase" % "hbase-client" % "2.1.0",
"org.apache.hbase" % "hbase-spark" % "2.1.0-cdh6.x-SNAPSHOT"
)
resolvers += "Cloudera Repository" at "https://repository.cloudera.com/artifactory/cloudera-repos/"
resolvers += "clojars" at "https://clojars.org/repo"
resolvers += "conjars" at "http://conjars.org/repo"
resolvers += "Apache HBase" at "https://repository.apache.org/content/repositories/releases"
The wrong code is:
def withCatalog(cat: String): DataFrame = {
sqlContext
.read
.options(Map(HBaseTableCatalog.tableCatalog->cat))
.format("org.apache.hadoop.hbase.spark")
.option("zkUrl", "127.0.0.1:2181/chen_test")
.load()
}
val df = withCatalog(catalog)
Exception info is:
Exception in thread "main" java.lang.NullPointerException
at org.apache.hadoop.hbase.spark.HBaseRelation.<init> (DefaultSource.scala:139)
at org.apache.hadoop.hbase.spark.DefaultSource.createRelation(DefaultSource.scala:70)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:340)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:239)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:227)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:164)
at hbase_test$.withCatalog$1(hbase_test.scala:57)
at hbase_test$.main(hbase_test.scala:59)
at hbase_test.main(hbase_test.scala)
How do I fix it? Can you help me?
Ran into this problem recently. Suggest you try this:
import org.apache.hadoop.fs.Path
val conf = HBaseConfiguration.create()
conf.addResource(new Path("/path/to/hbase-site.xml"))
new HBaseContext(sc, conf) // "sc" is the SparkContext you created earlier.
The last expression is introducing a stable value into the environment; found this quite accidentally by scanning Hbase's codebase.
Hope it helps.
I'm trying to build a Scala project based on the framework Play using the sbt. When I try to launch "sbt compile" from my terminal I get several compilation errors like the following one:
[error] <project.dir>/app/controllers/Application.scala:229:
object database is not a member of package views.html
[error] Ok(views.html.database(t2, JsonHelpers.TotalTip.langsForm))
[error] ^
This is my build.sbt file:
import play.sbt.PlayImport._
import com.typesafe.sbt.less.Import.LessKeys
name := """<app_name>"""
lazy val root = (project in file("."))
.enablePlugins(PlayScala)
.enablePlugins(GatlingPlugin)
.enablePlugins(BuildInfoPlugin, GitVersioning, GitBranchPrompt)
.settings(
buildInfoKeys := Seq[BuildInfoKey](name, version, scalaVersion, sbtVersion),
buildInfoPackage := "app")
https://github.com/puffnfresh/wartremover/issues/182
wartremoverWarnings ++= Warts.unsafe.filter(_ != Wart.Throw)
git.useGitDescribe := true
routesGenerator := InjectedRoutesGenerator
fork in run := true
scalaVersion := "2.11.8"
libraryDependencies += "com.typesafe.play" %% "play-json" % "2.5.4"
libraryDependencies ++= Seq(
jdbc,
cache,
ws,
filters,
specs2 % Test,
"com.typesafe.play" % "anorm_2.11" % "2.5.0",
"org.webjars" % "bootstrap" % "3.3.2",
"org.webjars" % "metisMenu" % "1.1.3",
"org.webjars" % "morrisjs" % "0.5.1",
"org.webjars" % "font-awesome" % "4.3.0",
"org.webjars" % "jquery" % "2.1.3",
"org.webjars" % "flot" % "0.8.3",
"org.webjars" % "datatables" % "1.10.5",
"org.webjars" % "datatables-plugins" % "1.10.5",
"com.newrelic.agent.java" % "newrelic-agent" % "3.14.0",
"com.newrelic.agent.java" % "newrelic-api" % "3.14.0",
"org.pac4j" % "play-pac4j" % "2.6.2",
"org.pac4j" % "pac4j-saml" % "1.9.5",
"org.pac4j" % "pac4j-sql" % "1.9.5",
"org.pac4j" % "pac4j-oidc" % "1.9.5" exclude("commons-io" , "commons-io"),
"org.pac4j" % "pac4j-openid" % "1.9.5" exclude("xml-apis" , "xml-apis"),
"org.pac4j" % "pac4j-jwt" % "1.9.5" exclude("commons-io" , "commons-io"),
"org.pac4j" % "pac4j-mongo" % "1.9.5",
"org.pac4j" % "pac4j-http" % "1.9.5",
"org.pac4j" % "pac4j-oauth" % "1.9.5",
"org.pac4j" % "pac4j-stormpath" % "1.9.5",
"com.typesafe.play" % "play-cache_2.11" % "2.5.4",
"org.pac4j" % "pac4j-http" % "1.9.5",
"org.pac4j" % "pac4j-cas" % "1.9.5",
"commons-io" % "commons-io" % "2.5"
)
includeFilter in (Assets, LessKeys.less) := "sb-admin-2.less" | "tables.less"
libraryDependencies += "io.gatling.highcharts" % "gatling-charts-highcharts" % "2.1.7"
libraryDependencies += "io.gatling" % "gatling-test-framework" % "2.1.7"
libraryDependencies += "org.scalaz" %% "scalaz-core" % "7.1.5"
libraryDependencies += "org.reactivemongo" %% "reactivemongo" % "0.11.10"
libraryDependencies += "io.swagger" %% "swagger-play2" % "1.6.0"
libraryDependencies += "com.github.tototoshi" %% "play-json-naming" % "1.0.0"
libraryDependencies += "com.github.tototoshi" %% "scala-csv" % "1.3.4"
libraryDependencies += "com.fasterxml.jackson.module" %% "jackson-module-scala" % "2.8.7"
libraryDependencies += "com.github.nscala-time" %% "nscala-time" % "2.16.0"
resolvers ++= Seq(Resolver.mavenLocal, "scalaz-bintray" at "http://dl.bintray.com/scalaz/releases", "Sonatype snapshots repository" at "https://oss.sonatype.org/content/repositories/snapshots/", "<internal repo>" at "<repo_url>")
Is it a dependencies issue? Does anyone have an idea why I get these errors?
Thanks
sbt clean compile
sometime that helps
if not have a look at https://www.playframework.com/documentation/2.5.x/IDE
I've imported Specs2 and everything looks good but some things that are not imported, among them the PlaySpecification trait.
I've tried to reload in sbt, to invalidate caches in Intellij ... But this trait is missing!
My built.sbt
name := """web2"""
version := "1.0-SNAPSHOT"
scalaVersion := "2.12.2"
lazy val root = (project in file(".")).enablePlugins(PlayScala, LauncherJarPlugin)
pipelineStages := Seq(digest)
libraryDependencies ++= Seq(
evolutions,
jdbc,
ehcache,
ws,
"com.softwaremill.macwire" %% "macros" % "2.3.0" % "provided",
"org.postgresql" % "postgresql" % "42.1.1",
"org.scalikejdbc" %% "scalikejdbc" % "3.0.0",
"org.scalikejdbc" %% "scalikejdbc-config" % "3.0.0",
"ch.qos.logback" % "logback-classic" % "1.2.3",
"de.svenkubiak" % "jBCrypt" % "0.4.1",
//"org.scalatestplus.play" %% "scalatestplus-play" % "3.0.+" % "test",
"org.mockito" % "mockito-core" % "2.7.22" % "test",
"org.specs2" %% "specs2-core" % "3.9.+" % "test"
)
resolvers += "scalaz-bintray" at "http://dl.bintray.com/scalaz/releases"
plugins.sbt
// The Play plugin
addSbtPlugin("com.typesafe.play" % "sbt-plugin" % "2.6.3")
// Web plugins
addSbtPlugin("com.typesafe.sbt" % "sbt-digest" % "1.1.2")
According to the documentation, replace "org.specs2" %% "specs2-core" % "3.9.+" % "test" with specs2 % Test in your dependencies:
libraryDependencies ++= Seq(
evolutions,
jdbc,
ehcache,
...
"org.mockito" % "mockito-core" % "2.7.22" % "test",
specs2 % Test
)
I have a project tree consisting of three projects A, B and C
B depends on A, and C depends on both A and B.
A and B are checked out in C's lib/ and both build fine using sbt compile
However, when I compile C, the build of B fails, complaining that it cannot find certain types/packages:
import org.scalatra.sbt._
import sbt.Keys._
import sbt._
object NwbApiBuild extends Build {
val Organization = "org.nwb"
val Name = "NWB API"
val Version = "0.1.0-SNAPSHOT"
val ScalaVersion = "2.10.3"
val ScalatraVersion = "2.3.0"
lazy val active_slick= Project (
"active-slick",
base = file("lib/active-slick")
)
lazy val slick_auth= Project (
"slick-auth",
base = file("lib/slick-auth")
)
lazy val project = Project (
"root",
file("."),
settings = Defaults.defaultSettings ++ ScalatraPlugin.scalatraWithJRebel ++ Seq(
organization := Organization,
name := Name,
version := Version,
scalaVersion := ScalaVersion,
resolvers += Classpaths.typesafeReleases,
libraryDependencies ++= Seq(
"org.scalatra" %% "scalatra" % ScalatraVersion,
"org.scalatra" %% "scalatra-specs2" % ScalatraVersion % "test",
"ch.qos.logback" % "logback-classic" % "1.0.6" % "runtime",
"org.eclipse.jetty" % "jetty-webapp" % "8.1.8.v20121106" % "container",
"org.eclipse.jetty.orbit" % "javax.servlet" % "3.0.0.v201112011016" % "container;provided;test" artifacts (Artifact("javax.servlet", "jar", "jar")),
"com.typesafe.slick" %% "slick" % "2.0.2",
"mysql" % "mysql-connector-java" % "5.1.31",
"joda-time" % "joda-time" % "2.3",
"org.joda" % "joda-convert" % "1.5",
"com.github.tototoshi" %% "slick-joda-mapper" % "1.1.0",
"org.json4s" %% "json4s-native" % "3.2.10",
"org.json4s" %% "json4s-jackson" % "3.2.7",
"c3p0" % "c3p0" % "0.9.1.2"
)
)
) aggregate(active_slick, slick_auth) dependsOn(active_slick, slick_auth)
}
where slick auth has build file
import org.scalatra.sbt._
name := "slick-auth"
version := "0.0.1-SNAPSHOT"
scalaVersion := "2.10.3"
val ScalatraVersion = "2.3.0"
lazy val active_slick = Project(
"active-slick",
base = file("lib/active-slick")
)
lazy val root = Project(
"root",
file("."),
settings = Defaults.defaultSettings ++ ScalatraPlugin.scalatraSettings ++ Seq(
libraryDependencies ++= Seq(
"com.typesafe.slick" %% "slick" % "2.0.2",
"org.slf4j" % "slf4j-nop" % "1.6.4",
"org.scalatest" %% "scalatest" % "2.2.0" % "test",
"org.scalatra" %% "scalatra" % ScalatraVersion,
"org.scalatra" %% "scalatra-specs2" % ScalatraVersion % "test",
"ch.qos.logback" % "logback-classic" % "1.0.6" % "runtime",
"org.eclipse.jetty" % "jetty-webapp" % "8.1.8.v20121106" % "container",
"org.eclipse.jetty.orbit" % "javax.servlet" % "3.0.0.v201112011016" % "container;provided;test" artifacts (Artifact("javax.servlet", "jar", "jar")),
"com.typesafe.slick" %% "slick" % "2.0.2",
"joda-time" % "joda-time" % "2.3",
"org.joda" % "joda-convert" % "1.5",
"com.github.tototoshi" %% "slick-joda-mapper" % "1.1.0",
"org.json4s" %% "json4s-native" % "3.2.10",
"org.json4s" %% "json4s-jackson" % "3.2.7",
"c3p0" % "c3p0" % "0.9.1.2"
)
)
).aggregate(active_slick).dependsOn(active_slick)
and active_slick:
name := "active-slick"
version := "0.0.1-SNAPSHOT"
scalaVersion := "2.10.3"
libraryDependencies ++= Seq(
"com.typesafe.slick" %% "slick" % "2.0.2",
"org.slf4j" % "slf4j-nop" % "1.6.4",
"org.scalatest" %% "scalatest" % "2.2.0" % "test",
"com.h2database" % "h2" % "1.3.166" % "test"
)
If you want to use another project as a dependency (rather than its binary version) you can use project references. There are two types of references, ProjectRef or a simpler version of the ProjectRef, which is RootProject.
You should change your build definition to reference slick_auth as
lazy val slick_auth = RootProject(file("lib/slick-auth"))
and active_slick as
lazy val active_slick = RootProject(file("lib/active-slick"))
This question is partially related to sbt-scalariform plugin - can't resolve settings. I managed to run scalariform from command line as SBT task.
Now the problem is with IDEA. When I open my build.sbt, which looks like this:
import scalariform.formatter.preferences._
name := """scheduling-backend"""
version := "1.0"
scalaVersion := "2.10.2"
resolvers += "spray repo" at "http://repo.spray.io"
resolvers += "spray nightlies" at "http://nightlies.spray.io"
resolvers += "SpringSource Milestone Repository" at "http://repo.springsource.org/milestone"
resolvers += "Neo4j Cypher DSL Repository" at "http://m2.neo4j.org/content/repositories/releases"
libraryDependencies ++= Seq(
"com.typesafe.akka" %% "akka-actor" % "2.3.0",
"com.typesafe.akka" %% "akka-slf4j" % "2.3.0",
"com.typesafe.akka" %% "akka-testkit" % "2.3.0" % "test",
"com.typesafe.akka" %% "akka-persistence-experimental" % "2.3.0",
"io.spray" % "spray-can" % "1.3.0",
"io.spray" % "spray-routing" % "1.3.0",
"io.spray" % "spray-testkit" % "1.3.0" % "test",
"io.spray" %% "spray-json" % "1.2.5",
"ch.qos.logback" % "logback-classic" % "1.0.13",
"org.specs2" %% "specs2" % "1.14" % "test",
"org.springframework.scala" % "spring-scala" % "1.0.0.M2",
"org.springframework.data" % "spring-data-neo4j" % "3.0.0.RELEASE",
"org.springframework.data" % "spring-data-neo4j-rest" % "3.0.0.RELEASE",
"javax.validation" % "validation-api" % "1.1.0.Final",
"com.github.nscala-time" %% "nscala-time" % "0.8.0",
"org.neo4j" % "neo4j-kernel" % "2.0.1" % "test" classifier "tests",
"com.sun.jersey" % "jersey-core" % "1.9",
"org.mockito" % "mockito-all" % "1.9.5"
)
scalacOptions ++= Seq(
"-unchecked",
"-deprecation",
"-Xlint",
"-Ywarn-dead-code",
"-language:_",
"-target:jvm-1.7",
"-encoding", "UTF-8"
)
org.scalastyle.sbt.ScalastylePlugin.Settings
scalariformSettings
ScalariformKeys.preferences := ScalariformKeys.preferences.value
.setPreference(AlignParameters, true)
.setPreference(CompactControlReadability, true)
IDEA reports problems with my file.
I am getting Cannot resolve symbol scalariformSettings and Cannot resolve symbol ScalariformKeyseven if I everything works from terminal.
adding import com.typesafe.sbt.SbtScalariform._ to build.sbt seems to fix the error on 13.1.1 with scala plugin 0.33.403, but I have to admit it ignored the import at first and then randomly started to see it.