Is there support for compression in ReactiveMongo? - mongodb

I am using ReactiveMongo as the connector for an Akka-Http, Akka-Streams project. I am creating the MongoConnection as shown below, but the data in the database is compressed using Snappy. No matter where I look, I can't find any mention of compression support in the ReactiveMongo documentation. When I try to connect to the Mongo database using a URL with the compressors=snappy flag, it returns an exception.
I looked through the source code and indeed it appears to have no mention of compression support at all. At this point I'm willing to accept a hack work around.
Can anyone help me please?
MongoConnection.fromString("mongodb://localhost:27017?compressors=snappy").flatMap(uri => driver.connect(uri))
Exception:
23:09:15.311 [default-akka.actor.default-dispatcher-6] ERROR akka.actor.ActorSystemImpl - Error during processing of request: 'The connection URI contains unsupported options: compressors'. Completing with 500 Internal Server Error response. To change default exception handling behavior, provide a custom ExceptionHandler.
java.lang.IllegalArgumentException: The connection URI contains unsupported options: compressors
at reactivemongo.api.AsyncDriver.connect(AsyncDriver.scala:227)
at reactivemongo.api.AsyncDriver.connect(AsyncDriver.scala:203)
at reactivemongo.api.AsyncDriver.connect(AsyncDriver.scala:252)
If you need a workable example, you can try this:
(You don't actually need a MongoDB container running locally for the error to be thrown)
object ReactiveMongoCompressorIssue extends App {
import scala.concurrent.Await
import scala.concurrent.duration._
implicit val actorSystem = ActorSystem("ReactiveMongoCompressorIssue")
implicit val dispatcher: ExecutionContextExecutor = actorSystem.dispatcher
final val driver = AsyncDriver()
val url = "mongodb://localhost:27017/?compressors=snappy"
val connection = Await.result(MongoConnection.fromString(url).flatMap(uri => driver.connect(uri)), 3.seconds)
assert(connection.active)
}

Thanks to what #cchantep said about how compression in MongoDB is handled on the server side (see the MongoDB docs here) I went back through the ReactiveMongo source code to see if there was a way to either bypass the check or remove the flag from the URL myself and connect without it.
Indeed, I found that there is a boolean flag called strictMode which determines whether ignoredOptions such as the compressors flag should cause an exception to be thrown or not. So now my connection looks like this:
MongoConnection.fromString(url).flatMap(uri => driver.connect(uri, None, strictMode = false))
The None refers to a name of a connection pool, but the other connect method I was using before doesn't use one either so this works fine.
Thank you for the help!

Related

Geode: Serialization Exception & Could not create an instance of a class on server

I have the client side working, looks like everything is alright. I can save the object and retrieve the object back from the client side.
However, if i am trying to use gfsh or data browser to view the data, i got this exception on gfsh
Message : Could not create an instance of a class com.room.backend.soa.spring.cache.geode.test.domain.Book
Result : false
and on data browser
javalangException- Query could not be executed due to - orgapachegeodepdxPdxSerializationException- Could not create an instance of a class comroombackendsoaspringcachegeodetestdomainBook
My code is like this
ClientCacheFactoryBean gemfireCache = new ClientCacheFactoryBean();
gemfireCache.setClose(true);
gemfireCache.setProperties(gemfireProperties);
gemfireCache.setPdxSerializer(pdxSerializer);
pdxSerializer is
ReflectionBasedAutoSerializer serializer =
new ReflectionBasedAutoSerializer(
"com.room.backend.soa.spring.cache.geode.test.domain.*",
"com.room.backend.soa.spring.cache.geode.test.domain.Book",
"com.room.backend.soa.spring.cache.geode.test.domain.Book#identity=id.*",
"com.room.backend.soa.spring.cache.geode.test.domain.#identity=id.*"
);
You need to run the gfsh>configure pdx --read-serialized=true command. This should be done after starting the locator, but before starting the servers. Please refer to this docs page for details.

Multipart Form Errors with Lagom

Most of our Lagom entrypoints don't use multipart form requests, but one does. Since Lagom doesn't currently support multipart requests natively, the general suggestion I have seen is to call the underlying Play API, using the PlayServiceCall mechanism.
We have done that, and it works--most of the time. But we experience intermittent errors, especially when submitting large files. These are always cases of java.util.zip.ZipException (of various kinds), looking as if not an entire file has been received for processing.
Here's how the entrypoint looks in the code; in particular, the Play wrapping mechanism:
def upload = PlayServiceCall[NotUsed, UUID] {
wrapCall => Action.async(multipartFormData) {
request => wrapCall(ServiceCall { _ =>
val upload = request.body.file("upload")
val input = new FileInputStream(upload.get.ref.file)
val filename = upload.get.filename
// ...
// other code to actually process the file
// ...
})(request).run
}
}
Here are just two examples of exceptions we're seeing:
Caused by: java.util.zip.ZipException: invalid code lengths set
at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:164)
at java.util.zip.ZipInputStream.read(ZipInputStream.java:194)
at org.apache.poi.openxml4j.util.ZipSecureFile$ThresholdInputStream.read(ZipSecureFile.java:214)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
etc.
Caused by: java.util.zip.ZipException: invalid distance too far back
at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:164)
at java.util.zip.ZipInputStream.read(ZipInputStream.java:194)
at org.apache.poi.openxml4j.util.ZipSecureFile$ThresholdInputStream.read(ZipSecureFile.java:214)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
etc.
We use Lagom 1.3.8, in Scala. Any suggestions, please?
Try using the new service gateway based on Akka HTTP.
You can enable this by adding the following to your build.sbt:
lagomServiceGatewayImpl in ThisBuild := "akka-http"
The new service gateway is still disabled by default in Lagom 1.3.8, but Lagom users that have experienced this problem have reported that it is resolved by enabling the akka-http gateway. This will become the default implementation in Lagom 1.4.0.

Polling with Akka-Http stream

I have found an [example][1] where akka-http is used with Source.single to make a request. Now I'd like to use Source.tick to implement polling requests which are execute every X seconds like this:
import akka.http.scaladsl.model._
import scala.concurrent.duration._
val request: HttpRequest = RequestBuilding.Get(Uri("http://api.someSite.com"))
val source: Source[HttpRequest, Cancellable] = Source.tick(1.seconds, 1.seconds, request)
val sourceWithDest = source.via(Http().superPool())
However, I get a compile error in the last line which I cant resolve(Type mismatch). Any ideas on what I am doing wrong or suggestions for alternatives?
[1]: https://gist.github.com/steinybot/a1f79fe9a67693722164
As per the docs:
The Flow returned by Http().superPool(...) is very similar to the one
from the Host-Level Client-Side API, so the Using a Host Connection
Pool section also applies here.
And then
The “pool client flow” returned by
Http().cachedHostConnectionPool(...) has the following type:
Flow[(HttpRequest, T), (Try[HttpResponse], T), HostConnectionPool]
This is to give client-side code the possibility to implement some logic to match the original requests to the corresponding response. Assuming you don't need this kind of behaviour in your case, you can always proceed by appending NotUsed to your request before feeding it to the pool flow. E.g.
val sourceWithDest: Source[Try[HttpResponse], Cancellable] =
source.map(req ⇒ (req, NotUsed)).via(Http().superPool[NotUsed]()).map(_._1)

Exceeded configured max-open-requests

recently I started to build some small web processing service using akka streams. It's quite simple, I'm pulling urls from redis, then I'm downloading those urls(they are images) later I'm processing images, and pushing them to s3 and some json to redis.
I'm downloading lot of different kinds of images from multiple sites, I'm getting whole bunch of errors like 404, Unexpected disconnect , Response Content-Length 17951202 exceeds the configured limit of 8388608, EntityStreamException: Entity stream truncation and redirects. With redirects I'm invoking requestWithRedirects with address founded in location header of response.
Part responsible for downloading is pretty much like this:
override lazy val http: HttpExt = Http()
def requestWithRedirects(request: HttpRequest, retries: Int = 10)(implicit akkaSystem: ActorSystem, materializer: FlowMaterializer): Future[HttpResponse] = {
TimeoutFuture(timeout, msg = "Download timed out!") {
http.singleRequest(request)
}.flatMap {
response => handleResponse(request, response, retries)
}.recoverWith {
case e: Exception if retries > 0 =>
requestWithRedirects(request, retries = retries - 1)
}
}
TimeoutFuture is quite simple it takes future and timeout. If future takes longer than timeout it returns other future with timeout exception.
The problem I'm having is: after some time I'm getting an error:
Message: RuntimeException: Exceeded configured max-open-requests value of [128] akka.http.impl.engine.client.PoolInterfaceActor$$anonfun$receive$1.applyOrElse in PoolInterfaceActor.scala::109
akka.actor.Actor$class.aroundReceive in Actor.scala::467
akka.http.impl.engine.client.PoolInterfaceActor.akka$stream$actor$ActorSubscriber$$super$aroundReceive in PoolInterfaceActor.scala::46
akka.stream.actor.ActorSubscriber$class.aroundReceive in ActorSubscriber.scala::208
akka.http.impl.engine.client.PoolInterfaceActor.akka$stream$actor$ActorPublisher$$super$aroundReceive in PoolInterfaceActor.scala::46
akka.stream.actor.ActorPublisher$class.aroundReceive in ActorPublisher.scala::317
akka.http.impl.engine.client.PoolInterfaceActor.aroundReceive in PoolInterfaceActor.scala::46
akka.actor.ActorCell.receiveMessage in ActorCell.scala::516
akka.actor.ActorCell.invoke in ActorCell.scala::487
akka.dispatch.Mailbox.processMailbox in Mailbox.scala::238
akka.dispatch.Mailbox.run in Mailbox.scala::220
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec in AbstractDispatcher.scala::397
scala.concurrent.forkjoin.ForkJoinTask.doExec in ForkJoinTask.java::260
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask in ForkJoinPool.java::1339
scala.concurrent.forkjoin.ForkJoinPool.runWorker in ForkJoinPool.java::1979
scala.concurrent.forkjoin.ForkJoinWorkerThread.run in ForkJoinWorkerThread.java::107
I'm not sure what could be the problem but I think I have some downloads that were not finished properly and they stay in some global pool of connections after a while causing mentioned error. Any ideas what could be causing the problem? Or how to try find root of the problem: I already tested 404 responses, and Response Content-Length exceeds... errors, and they doesn't seem to be my troublemakers.
EDIT:
Most likely the problem is with my TimeoutFuture. I'm filling it with error as described here https://stackoverflow.com/a/29330010/2963977 but in my opinion future that is actually downloading an image never completes and it's taking my connection pool resources.
I wonder why those settings doesn't have any impact in my case :
akka.http.client.connecting-timeout = 1 s
akka.http.client.idle-timeout = 1 s
akka.http.host-connection-pool.idle-timeout = 1 s
EDIT2:
Apparently timeouts are not supported yet. Here is my bug report
https://github.com/akka/akka/issues/17732#issuecomment-112315953

Reliably working out if your db is down with Scala Slick

I am using Scala slick to work with my Mysql db.
I am wrapping all the calls using scala.util.Try
I would like to have different behaviour based on the problem
If the DB is down, ultimately I want my webapp to return a 503
If a strange query gets through to my db layer and there is a bug with my code then I want to return a 500
After some googling, it seems like you can get a wide array of different exceptions with error codes and I'm unsure what to look for.
With slick i am using the com.mysql.jdbc.driver
Thanks
Slick/MySQL will throw MySQLSyntaxErrorException for bad syntax and CommunicationsException when it's unable to reach the database.
Here's a quick example that will catch both of these types of exceptions:
try {
Database.forURL("jdbc:mysql://some-host:3306/db-name",
driver = "com.mysql.jdbc.Driver",
user="",
password="") withSession { session: Session =>
implicit val s = session
...
}
} catch {
case e: MySQLSyntaxErrorException =>
... handle the syntax error ...
// You may want to define your own Exception that wraps the MySQL one
// and adds more context
case e: CommunicationsException =>
... handle the connection error ...
}
Then, in your webapp code, you'll want to catch your custom exceptions (see the comment in the code) and return the HTTP codes accordingly.