I've been playing around with Redis and Scala separately, and thought it'd be neat to combine them in a simple Lift app.
I've done a fair bit of Googling and can't find any examples of a Lift app that uses Redis. Is there a reason for this?
What drivers/APIs do you recommend for using Redis w/Lift? I'm currently working with Jedis (https://github.com/xetorthio/jedis).
I use scalatra with jedis as the connector to redis, works fine as well. Java data types will be converted to equivalents in scala implicitly when scala.collection.JavaConversions._ is imported (in scala 2.8 or later). To use jedis, simply add this line to your project definition file in sbt 0.7.x:
val jedis = "redis.clients" % "jedis" % "2.0.0"
or this in sbt 0.10.x:
libraryDependencies += "redis.clients" % "jedis" % "2.0.0"
I have tested a couple of scala redis connectors - settled on https://github.com/debasishg/scala-redis for further testing.
Simply
val scalaredis = "net.debasishg" % "redisclient_2.9.0" % "2.3.1"
in SBT
According to http://mvnrepository.com/artifact/net.debasishg/redisclient_2.9.1,
libraryDependencies += "net.debasishg" %% "redisclient" % "2.7"
Related
I want to use Gatling in my Scala 3 / sbt Project.
The problem is that Gatling packages its library without Version-Postfix. So I think you have the same problem for any Scala library that does that.
I tried a few things, for example:
Adding the dependency according to the documentation:
libraryDependencies += "io.gatling" % "gatling-test-framework" % "3.7.2" % "test"
Gives:
Modules were resolved with conflicting cross-version suffixes in ProjectRef(uri("file:.../"), "api"):
com.softwaremill.quicklens:quicklens _3, _2.13
And
According to the Scala 3 Documentation:
libraryDependencies += ("io.gatling" % "gatling-test-framework" % "3.7.2" % "test").cross(CrossVersion.for3Use2_13)
Gives:
not found: https://repo1.maven.org/maven2/io/gatling/gatling-test-framework_2.13/3.7.2/gatling-test-framework_2.13-3.7.2.pom
Is there a way?
Not sure why but gatling-test-framework is not published with the version postfix as you said.
This means that you don't need/can't use the for3Use2_13 as there is no 2.13 version nor 3 version: there's just a single version without postfix.
Looking at its dependencies, version 3.7.2 targets Scala 2.13: https://mvnrepository.com/artifact/io.gatling/gatling-test-framework/3.7.2. As Scala 3 is compatible with Scala 2.13, it should be just fine with your first attempt.
Not sure where the conflict with quicklens comes from but if it comes from Gatling dependency, you can probably exclude the _2.13 version from Gatling (or even globally) as you are pulling the _3 version yourself:
libraryDependencies += "io.gatling" % "gatling-test-framework" % "3.7.2" % "test" exclude("com.softwaremill.quicklens" % "quicklens")
I am trying to list all objects in a bucket, and then read some or all of them as CSV. I have spent two days now, trying to do both, but I can only get one working at a time if I'm using Google's libraries.
I think the problem lies in an incompatibility between Google's own libraries, but I'm not entirely sure. First, I think I should show how I'm doing each thing.
This is how I'm reading a single file. In my version of Scala, you can use a gs:// url with spark.read.csv:
val jsonKeyFile = "my-local-keyfile.json"
ss.sparkContext.hadoopConfiguration.set("google.cloud.auth.service.account.json.keyfile", jsonKeyFile)
spark.read
.option("header", "true")
.option("sep", ",")
.option("inferSchema", "false")
.option("mode", "FAILFAST")
.csv(gcsFile)
This actually works alone, and I get a working DF from it. Then the problem arises when I try to add Google's Storage library:
libraryDependencies += "com.google.cloud" % "google-cloud-storage" % "1.70.0"
If I try to run the same code again, I get this bad boy from the .csv call:
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
19/05/14 16:38:00 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
An exception or error caused a run to abort: Class com.google.common.base.Suppliers$SupplierOfInstance does not implement the requested interface java.util.function.Supplier
java.lang.IncompatibleClassChangeError: Class com.google.common.base.Suppliers$SupplierOfInstance does not implement the requested interface java.util.function.Supplier
at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.getGcsFs(GoogleHadoopFileSystemBase.java:1488)
at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.configure(GoogleHadoopFileSystemBase.java:1659)
at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.initialize(GoogleHadoopFileSystemBase.java:683)
at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.initialize(GoogleHadoopFileSystemBase.java:646)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3303)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
...(lots more trace, probably irrelevant)
Then, you might ask, why don't you just not use the library? Well... This is the code that lists the objects in a bucket:
StorageOptions
.newBuilder()
.setCredentials(ServiceAccountCredentials.fromStream(
File(jsonKeyFile).inputStream()))
.build()
.getService
.list(bucket)
.getValues
.asScala
.map(irrelevant)
.toSeq
.toDF("irrelevant")
And I have not yet found a way to do this easily without the specified library.
I found out what caused the problem. Guava:27.1-android was a dependency of some library at some point, I don't know which and how it got there, but it was in use. In this version of Guava, the Supplier interface does not extend the Java Supplier interface.
I fixed it by adding Guava 27.1-jre to my dependencies. I don't know if the order matters, but I don't dare touch anything at this point. Here is where I placed it:
libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.5" % "test"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.4.1" % "provided"
libraryDependencies += "com.google.guava" % "guava" % "27.1-jre"
libraryDependencies += "com.google.cloud" % "google-cloud-storage" % "1.70.0"
//BQ samples as of 27feb2019 use hadoop2 but hadoop3 seems to work fine and are recommended elsewhere
libraryDependencies += "com.google.cloud.bigdataoss" % "bigquery-connector" % "hadoop3-0.13.16" % "provided"
libraryDependencies += "com.google.cloud.bigdataoss" % "gcs-connector" % "hadoop3-1.9.16" % "provided"
Hope this prevents some other poor soul from spending 2 days on this bs.
#Andy That's a life savior! after spending time investigating I've found that some of the libraries I'm using use different versions of guava so you should load this 27.1-jre version into your environment.
In my case using Gradle I had to do something like the following:
// This is here *the first line* to support GCS files [Do not remove or move!]
implementation group: 'com.google.guava', name: 'guava', version: '27.1-jre'
api(project(':other.project')) {
exclude group: 'com.google.guava', module: 'guava'
}
implementation(group: 'other.library', name: 'library.name', version: 'library.version') {
exclude group: 'com.google.guava', module: 'guava'
}
I have a text file in S3 that I would like to load into an RDD with spark-shell.
I have downloaded Spark 2.3.0 for Hadoop. Naively, I would expect that I just need to set the hadoop settings and I'd be set.
val inFile = "s3a://some/path"
val accessKey = "some-access-key"
val secretKey = "some-secret-key"
sc.hadoopConfiguration.set("fs.s3a.access.key", accessKey)
sc.hadoopConfiguration.set("fs.s3a.secret.key", secretKey)
sc.textFile(inFile).count()
println(run())
Invoking the final line returns:
Failure(java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found)
This seems to be asking that I provide the library which includes S3AFileSystem. No problem - I download the appropriate jar and add this line to the beginning of the script.
:require C:\{path-to-jar}\hadoop-aws-3.1.0.jar
Now, running the script fails at the final line with a variety of errors similar to this:
error: error while loading Partition, class file 'C:\spark\spark-2.3.0-bin-hadoop2.7\jars\spark-core_2.11-2.3.0.jar(org/apache/spark/Partition.class)' has location not matching its contents: contains class Partition
I'm lost at this point - clearly, it had no issue defining the run method before.
I can access the Partition class myself directly, but something is happening above that prevents the code from accessing Partition.
scala> new org.apache.spark.Partition {def index = 3}
res6: org.apache.spark.Partition = $anon$1#3
Curiously, running the final line of the script yields a different error in subsequent invocations.
scala> sc.textFile(inFile).count()
java.lang.NoClassDefFoundError: org/apache/hadoop/fs/StreamCapabilities
at java.lang.ClassLoader.defineClass1(Native Method)
...
The documentation claims this is part of hadoop 3.1.0, which I'm using, but when exploring hadoop-aws-3.1.0.jar I see no trace of StreamCapabilities.
Is there a different jar I should be using? Am I trying to solve this problem incorrectly? Or, have I fallen into the XY problem trap?
Answers I tried
The official docs assume I'm running the script on a cluster. But I'm running spark-shell locally.
This other StackOverflow question is for an older problem. I'm using s3a as a result, but am encountering a different problem.
I also tried using every jar of Hadoop from 2.6 to 3.1, to no avail.
org.apache.hadoop.fs.StreamCapabilities is in hadoop-common-3.1.jar
You are probably mixing version of Hadoop JARs, which, as coved in the s3a troubleshooting docs is doomed.
Spark shell works fine with the right JARs in. But ASF Spark releases don't work with Hadoop 3.x yet, due to some outstanding issues. Stick to Hadoop 2.8.x and you'll get good S3 performance without so much pain.
I found a path that fixed the issue, but I have no idea why.
Create an SBT IntelliJ project
Include the below dependencies and overrides
Run the script (sans require statement) from sbt console
scalaVersion := "2.11.12"
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.3.0"
libraryDependencies += "org.apache.hadoop" % "hadoop-aws" % "3.1.0"
libraryDependencies += "org.apache.hadoop" % "hadoop-client" % "3.1.0"
dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-core" % "2.8.7"
dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-databind" % "2.8.7"
dependencyOverrides += "com.fasterxml.jackson.module" % "jackson-module-scala_2.11" % "2.8.7"
The key part, naturally, is overriding the jackson dependencies.
In sbt, we define dependencies for a project:
libraryDependencies ++= Seq(
"com.beachape" %% "enumeratum" % "1.3.2",
"org.scalatest" %% "scalatest" % "2.2.4" % "test"
)
Is it possible for the Scala application thus compiled to get access to this data, somehow?
I am making a modular system of Play 2.4 APIs and would like the "umbrella" to be able to list which APIs it's carrying.
I will probably get this done using sbt-buildinfo that I found via this question.
Other suggestions are of course welcome.
A simple solution is checking maven repositories. For example, the link below shows all libraries com.beachape" %% "enumeratum" % "1.3.2" depends on.
http://mvnrepository.com/artifact/com.beachape/enumeratum_2.11/1.3.2
I want to develop a rather simple web application in Scala, with Lift or Play framework, using Eclipse as an environment and some simple webserver like Jetty. However, to my inexpressibly great surprise, I cannot setup the whole thing to work together. I also could not find any sensible simple and clear guide on how to do this. After a half day of searching, I came to an opinion that everyone around seem to use a mix of sbt/maven and feel it ok migrating from one to another and writing project config files manually just to get a simple blank project to begin with.
There is no plain simple way to create even an empty project. With Java I remember that it was a couple of clicks - to integrate a webserver into Eclipse, create a simple web app project and run it right from there. Where had gone the power and simplicity of Scala in this case? And that's only if I want to try Lift. What if I would like to try Play also, should I travel the same path again?
Is there anywhere a simple and complete guide that describes how to setup the environment so that it is possible to start developing the apps right away?
UPDATE: I have reached a successful Play project integration with Eclipse, with all the capabilities that Play has out-of-the-box, thanks to the advice of Peter Gwiazda. I am using this setup for developing right now. However, my question of interest remains: what are other ways to acheive similar functionality with other frameworks such as Lift, Scalatra and others?
With Playframework 2.0 it's pretty simple. Just follow the tutorial
http://www.playframework.org/documentation/2.0/ScalaTodoList
With Play you don't need anything else - Play already contains a server.
IMHO Play is way easier to work with than Lift.
EDIT
OK, you asked for it ;-)
Here's a bleeding edge setup for Scalatra with SBT Coffeescript & LESS (see HERE for SBT-Eclipse dependency management)
1) eclipsify a test project
2) in project root create "build.sbt" file:
import AssemblyKeys._
import Keys._
name := "your project name"
version := "1.0"
scalaVersion := "2.9.1"
fork in run := true
resolvers ++= Seq(
"Sonatype OSS Snapshots" at "http://oss.sonatype.org/content/repositories/snapshots/",
"Typesafe repository" at "http://typesafe.artifactoryonline.com/typesafe/ivy-releases/"
)
seq(webSettings :_*)
seq(assemblySettings: _*)
seq(coffeeSettings: _*)
seq(lessSettings:_*)
(LessKeys.mini in (Compile, LessKeys.less)) := false
libraryDependencies ++= Seq(
"org.scalatra" %% "scalatra" % "2.1.0-SNAPSHOT",
"org.scalatra" %% "scalatra-scalate" % "2.1.0-SNAPSHOT",
"org.scalatra" %% "scalatra-lift-json" % "2.1.0-SNAPSHOT",
"org.scalatra" %% "scalatra-anti-xml" % "2.1.0-SNAPSHOT",
"org.scalatra" %% "scalatra-fileupload" % "2.1.0-SNAPSHOT",
"org.eclipse.jetty" % "jetty-webapp" % "8.1.0.RC2" % "test;container;provided",
"javax.servlet" % "javax.servlet-api" % "3.0.1" % "provided"
)
unmanagedBase <<= baseDirectory { base => base / "/src/main/webapp/WEB-INF/lib/" }
3) create folder "project" in root with plugins.sbt file:
libraryDependencies <+= sbtVersion(v => "com.github.siasia" %% "xsbt-web-plugin" % (v+"-0.2.10"))
resolvers += Resolver.url("sbt-plugin-releases", new URL("http://scalasbt.artifactoryonline.com/scalasbt/sbt-plugin-releases/"))(Resolver.ivyStylePatterns)
addSbtPlugin("com.typesafe.sbteclipse" % "sbteclipse-plugin" % "2.0.0-M3")
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.7.2")
addSbtPlugin("me.lessis" % "coffeescripted-sbt" % "0.2.2")
addSbtPlugin("me.lessis" % "less-sbt" % "0.1.9")
4) from terminal/command prompt start sbt & enable code reloading:
# sbt
> ~;container:start;container:reload /
Open up the Scalatra Book and start hacking ;-)
ORIGINAL
Have to mention it, but a micro framework a la Scalatra, Spray, or Unfiltered might be of interest as well.
That is, if you're not looking for the kitchen sink that Lift and Play provide; if you are looking for the kitchen sink and want to get rolling quickly, Play 2.0 does indeed look quite nice.
Disclaimer: I'm a member of the Vaadin team.
You could also try out Vaadin which works perfectly with Scala, HOWTO here. You can also use Maven or SBT if you want. You should also check out Scaladin, the semi-official Scala wrapper for Vaadin.
Vaadin is a component based library (just one JAR with no dependencies) and it allows you to create your Ajax and HTML5 enabled UI in pure Scala without any HTML templates, RPC, XML or JavaScript.
You could use my Maven prototype for Scalatra, then simply import the maven project into Eclipse. Quite nice and you're not forced to use SBT.
https://github.com/fancellu/scalatra-maven-prototype
You can have a look on my Github repo where I have a project that uses Lift and Jetty (as an embedded server). It's not well documented yet but is small enough to grasp how it's working
P4G Server Repo
The server starts from com.p4g.Server object (which is called within com.p4g.Main Application object )
My Lift boostrap object is located in boostrap.liftweb package
BTW, I'm also using ScalaQuery and ScalaZ