How can use Solr from within Scala/ Play? Specifically how do I add/ update documents?
Update: see my newer answer refer https://stackoverflow.com/a/17315047/604511
Here is code I wrote which uses Play's JSON library and Dispatch HTTP client. Its not perfect, but it should help you get started.
package controllers
import play.api._
import play.api.mvc._
import play.api.libs.json.Json
import play.api.libs.json.Json.toJson
import dispatch._
object Application extends Controller {
def index = Action {
val addDocument = Json.toJson(
Map(
"add" ->
Seq(
//a document
Map(
"id" -> toJson("123"),
"subject" -> toJson("you have been served")
)
)
))
val toSend = Json.stringify( addDocument)
val params = Map( "commit" -> "true", "wt" -> "json")
val headers = Map( "Content-type" -> "application/json")
val solr = host( "127.0.0.1", 8983)
val req = solr / "solr" / "update" / "json" <<?
params <:< headers << toSend
val response = Http(req)()
Ok( toSend + response.getResponseBody)
//Redirect(routes.Application.tasks)
}
def tasks = TODO
def newTask = TODO
def deleteTask(id: Long) = TODO
}
You might consider using the SolrJ Java Lib, which uses a binary protocol to communicate with the Solr Server which performs better than using the XML way.
Adding a document to the index is done this:
http://wiki.apache.org/solr/Solrj#Adding_Data_to_Solr
Not directly related to updating documents, but a nice query DSL for Solr in Scala build by foursquare is described in their engineering blog article: http://engineering.foursquare.com/2011/08/29/slashem-a-rogue-like-type-safe-scala-dsl-for-querying-solr/
Related
I want to inserting a new entry in embedded field in existing document in mongodb using spark.
Example
Blog {
id:"001"
title:"This is a test blog",
content:"...."
comments:[{title:"comment1",content:".."},{title:"comment2",content:"..."}]
}
What I want to do
db.blogs.update({id:"001"}, {$push:{comments:{title:"commentX",content:".."}}});
Is it possible currently in this library? If not, can you please point me to the right direction.
Thanks in Advance
I was able to do the operation I wished using the Casbah Library for Spark-mongoDb.
import java.sql.Timestamp
import java.util.Date
import com.mongodb.casbah.MongoClient
import com.mongodb.casbah.commons.MongoDBObject
import com.mongodb.casbah.query.Imports._
object TestCasbah {
def main(args: Array[String]) {
val mongoClient = MongoClient("172.18.96.45", 27017)
val db = mongoClient("agentCallRecord")
val coll = db("CallDetails")
val query = MongoDBObject("agentId" -> "agent_1")
val callRatingMongoObject = MongoDBObject("audioId" -> 12351,"startTime" -> new Timestamp(new Date().getTime).toString, "endTime" -> new Timestamp(new Date().getTime).toString, "totalScore" -> 1, "sentiment" -> "NEGATIVE")
val update = $push("callRating" -> callRatingMongoObject)
coll.update(query, update)
}
}
I would like to manually run my evolution script at the beginning of each test file. I'm working with Play! 2.4 and Slick 3.
According to the documentation, the way to go seems to be:
Evolutions.applyEvolutions(database)
but I don't manage to get an instance of my database. In the documentation play.api.db.Databases is imported in order to get a database instance but if I try to import it, I get this error: object Databases is not a member of package play.api.db
How can I get an instance of my database in order to run the evolution script?
Edit: as asked in the comments, here is the entire source code giving the error:
import models._
import org.scalatest.concurrent.ScalaFutures._
import org.scalatest.time.{Seconds, Span}
import org.scalatestplus.play._
import play.api.db.evolutions.Evolutions
import play.api.db.Databases
class TestAddressModel extends PlaySpec with OneAppPerSuite {
lazy val appBuilder = new GuiceApplicationBuilder()
lazy val injector = appBuilder.injector()
lazy val dbConfProvider = injector.instanceOf[DatabaseConfigProvider]
def beforeAll() = {
//val database: Database = ???
//Evolutions.applyEvolutions(database)
}
"test" must {
"test" in { }
}
}
I finally found this solution. I inject with Guice:
lazy val appBuilder = new GuiceApplicationBuilder()
lazy val injector = appBuilder.injector()
lazy val databaseApi = injector.instanceOf[DBApi] //here is the important line
(You have to import play.api.db.DBApi.)
And in my tests, I simply do the following (actually I use an other database for my tests):
override def beforeAll() = {
Evolutions.applyEvolutions(databaseApi.database("default"))
}
override def afterAll() = {
Evolutions.cleanupEvolutions(databaseApi.database("default"))
}
Considering that you are using Play 2.4, where evolutions were moved into a separate module, you have to add evolutions to your project dependencies.
libraryDependencies += evolutions
Source: Evolutions
Relevant commit: Split play-jdbc into three different modules
To have access to play.api.db.Databases, you must add jdbc to your dependencies :
libraryDependencies += jdbc
Hope it helps some people passing here.
EDIT: the code would then look like this :
import play.api.db.Databases
val database = Databases(
driver = "com.mysql.jdbc.Driver",
url = "jdbc:mysql://localhost/test",
name = "mydatabase",
config = Map(
"user" -> "test",
"password" -> "secret"
)
)
You now have an instance of the DB, and can execute queries on it :
val statement = database.getConnection().createStatement()
val resultSet = statement.executeQuery("some_sql_query")
You can see more from the docs
EDIT: typo
I find the easiest way to run tests with evolutions applied is to use FakeApplication, and input the connection info for the DB manually.
def withDB[T](code: => T): T =
// Create application to run database evolutions
running(FakeApplication(additionalConfiguration = Map(
"db.default.driver" -> "<my-driver-class>",
"db.default.url" -> "<my-db-url>",
"db.default.user" -> "<my-db>",
"db.default.password" -> "<my-password>",
"evolutionplugin" -> "enabled"
))) {
// Start a db session
withSession(code)
}
Use it like this:
"test" in withDB { }
This allows you, for example, to use an in-memory database for speeding up your unit tests.
You can access the DB instance as play.api.db.DB if you need it. You'll also need to import play.api.Play.current.
Use FakeApplication to read your DB configuration and provide a DB instance.
def withDB[T](code: => T): T =
// Create application to run database evolutions
running(FakeApplication(additionalConfiguration = Map(
"evolutionplugin" -> "disabled"))) {
import play.api.Play.current
val database = play.api.db.DB
Evolutions.applyEvolutions(database)
withSession(code)
Evolutions.cleanupEvolutions(database)
}
Use it like this:
"test" in withDB { }
I’m getting started with a Finagle server (twitter/finagle):
import com.twitter.finagle.{Http, Service}
import com.twitter.util.{Await, Future}
import java.net.InetSocketAddress
import org.jboss.netty.handler.codec.http._
object Server extends App {
val service = new Service[HttpRequest, HttpResponse] {
def apply(req: HttpRequest): Future[HttpResponse] =
Future.value(new DefaultHttpResponse(
req.getProtocolVersion, HttpResponseStatus.OK))
}
val server = Http.serve(":8080", service)
Await.ready(server)
}
Client (twitter/finagle):
import com.twitter.finagle.{Http, Service}
import com.twitter.util.{Await, Future}
import java.net.InetSocketAddress
import org.jboss.netty.handler.codec.http._
object Client extends App {
val client: Service[HttpRequest, HttpResponse] =
Http.newService("localhost:8080")
val request = new DefaultHttpRequest(
HttpVersion.HTTP_1_1, HttpMethod.GET, "/")
val response: Future[HttpResponse] = client(request)
response onSuccess { resp: HttpResponse =>
println("GET success: " + resp)
}
Await.ready(response)
}
How do I send data like Map("data_id" -> 5) from the client to the server? And where in the server do I receive it? Do I have to add a callback to the server?
I haven’t found it by searching. If you can give me a link with an example, that will be enough.
Finagle is a very thin library. That means that you'll have to handle most of the "magic" by yourself.
To make the request with parameters from the Client, I use these helper methods:
def buildUri(base: String, path: String, params: Map[String, String] = Map.empty): String = {
val p = if (params.isEmpty) ""
else params map { case (k,v) => urlEncode(k) + "=" + urlEncode(v) } mkString ("?", "&", "")
base + path + p
}
def urlEncode(url: String): String = URLEncoder.encode(url, "UTF-8")
And then I call it like this:
val url = buildUri(baseAddress, path, defaultParams ++ params)
val req = RequestBuilder().url(url).setHeader("Accept", "*/*").buildGet
client(req)
As for the server you have to do basically the same thing and parse the parameters by hand. Either using java.net.URI or even org.jboss.netty.handler.codec.http.QueryStringDecoder.
Of course you can also use URI and QueryStringEncoder to encode as well, instead of using my helper methods.
That said, if you want to do that on higher level, you can use one of these libraries above Finagle:
https://github.com/fwbrasil/zoot
http://finatra.info/ (this is for the server part only)
I am currently reading 'Play for Scala' by Peter Hilton. I have just got the the end of the first example Play app where you build a paperclip directory.
However upon compiling I get a compilation error telling me that the value 'flash' has not been found. Usually this is a simple mistake I have made but given that I am just following the guide in the book I can't identify a fix.
The error is in lines 52 and 53 in the 'NewProduct' function
Here is the code:
package controllers
import play.api.mvc.{Action, Controller}
import models.Product
import play.api.data.Form
import play.api.data.Forms.{mapping, longNumber, nonEmptyText}
import play.api.i18n.Messages
import play.api.mvc.Flash
object Products extends Controller {
private val productForm: Form[Product] = Form(
mapping(
"ean" -> longNumber.verifying(
"validation.ean.duplicate", Product.findByEan(_).isEmpty),
"name" -> nonEmptyText,
"description" -> nonEmptyText
)(Product.apply)(Product.unapply)
)
def list = Action {implicit request =>
val products = Product.findAll
Ok(views.html.products.list(products))
}
def show(ean: Long) = Action {implicit request =>
Product.findByEan(ean).map {product =>
Ok(views.html.products.details(product))
}.getOrElse(NotFound)
}
def save = Action { implicit request =>
val newProductForm = productForm.bindFromRequest()
newProductForm.fold(
hasErrors = {form =>
Redirect(routes.Products.newProduct()).
flashing(Flash(form.data) + ("error" -> Messages("validation.errors")))
},
success = {newProduct =>
Product.add(newProduct)
val message = Messages("products.new.success", newProduct.name)
Redirect(routes.Products.show(newProduct.ean)).
flashing("success" -> message)
}
)
}
def newProduct = Action { implicit request =>
val form = if(flash.get("error").isDefined)
productForm.bind(flash.data)
else
productForm
Ok(views.html.products.editProduct(form))
}
}
Example is working with Play < 2.3, you may want to check which version you are currently using. With Play > 2.3, request.flash must be used instead. In both case you can use request.flash (which is more explicit).
I'm using gatling in linux terminal. When I pass parameter like described in github I get error:
value users is not a member of Integer
This is my code:
package mypackage
import io.gatling.core.Predef._
import io.gatling.http.Predef._
import io.gatling.jdbc.Predef._
import io.gatling.http.Headers.Names._
import scala.concurrent.duration._
import bootstrap._
import assertions._
import util.Random
class MySimulation extends Simulation {
val usersCount = Integer.getInteger("users", 1)
val links = csv("links.csv").random
val httpProtocol = http
.baseURL("http://mywebsite.com:8080/")
.acceptCharsetHeader("ISO-8859-1,utf-8;q=0.7,*;q=0.7")
.acceptHeader("text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8")
.acceptEncodingHeader("gzip, deflate")
.acceptLanguageHeader("fr,fr-fr;q=0.8,en-us;q=0.5,en;q=0.3")
.disableFollowRedirect
val headers_1 = Map(
"Keep-Alive" -> "115")
val headers_3 = Map(
"Keep-Alive" -> "115",
"Content-Type" -> "application/x-www-form-urlencoded")
val scn = scenario("big project benchmark")
.repeat(50) {
feed(links)
.exec(
http("request_1")
.get("${pageUri}")
.headers(headers_1)).pause(1000 millisecond)
}
setUp(scn.inject(ramp(usersCount users) over (30 seconds)))
.protocols(httpProtocol)
.assertions(global.successfulRequests.percent.is(100), details("request_1").responseTime.max.lessThan(1000))
I start this in terminal using:
JAVA_OPTS="-Dusers=300" ./gatling.sh -s mypackage.mySimulation -on testing -sd test1
Please, be patient, because I'm totally new to scala and gatling. Thanks for any help.
The problem comes from the usersCount users part of the setUp.
In Scala, this is interpreted as usersCount.users which, in our case does not exist since Integer does not have a users method.
I think (but I'm not sure since I cannot test it right now), that you should make usersCount an Int like so: val usersCount: Int = Integer.getInteger("users", 1).toInt.
Hope this helps!
PS: The reason you should convert Integer to Int is because of implicit conversions. This is a really powerful feature of Scala.
PPS: The wiki documentation was valid for Gatling 1.X, it will be updated accordingly for Gatling 2.X