I am using Play (2.6) with Slick, my database connection times out every second or third try. The only way to get the connection up again is by restarting the app with sbt run. It's driving me crazy, any help appreciated.
To be clear, I'm using Slick with a local MySQL database for very lightweight usage.
build.sbt
libraryDependencies += "mysql" % "mysql-connector-java" % "8.0.11"
libraryDependencies += "com.typesafe.slick" %% "slick" % "3.2.3"
libraryDependencies += "com.typesafe.slick" %% "slick-hikaricp" % "3.2.3"
application.conf
# db connections = ((physical_core_count * 2) + effective_spindle_count)
fixedConnectionPool = 5
repository.dispatcher {
executor = "thread-pool-executor"
throughput = 1
thread-pool-executor {
fixed-pool-size = ${fixedConnectionPool}
}
}
Error message:
java.sql.SQLTransientConnectionException: <dbconfig> - Connection is not available, request timed out after 1001ms.
Stack Trace:
com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:548)
com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:186)
com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:145)
com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:83)
slick.jdbc.hikaricp.HikariCPJdbcDataSource.createConnection(HikariCPJdbcDataSource.scala:14)
slick.jdbc.JdbcBackend$BaseSession.<init>(JdbcBackend.scala:453)
slick.jdbc.JdbcBackend$DatabaseDef.createSession(JdbcBackend.scala:46)
slick.jdbc.JdbcBackend$DatabaseDef.createSession(JdbcBackend.scala:37)
slick.basic.BasicBackend$DatabaseDef.acquireSession(BasicBackend.scala:249)
slick.basic.BasicBackend$DatabaseDef.acquireSession$(BasicBackend.scala:248)
slick.jdbc.JdbcBackend$DatabaseDef.acquireSession(JdbcBackend.scala:37)
slick.basic.BasicBackend$DatabaseDef$$anon$2.run(BasicBackend.scala:274)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
Update
I didn't include the specific db config in application.conf:
<name> {
slick.driver = scala.slick.driver.MySQLDriver
driver = "com.mysql.cj.jdbc.Driver"
url = __
user = __
password = __
}
I've added to the config:
keepAliveConnection = true
connectionPool = disabled
And it's working fine.
Related
Short story:
In a 2-node Kubernetes AKKA clustering setup, each node runs a piece of code that calls a Kubeless function as follows:
log.trace("Calling URL [{}] with bytes response.", address)
Http().singleRequest(
request = HttpRequest(
method = HttpMethods.POST,
uri = address,
entity = HttpEntity(contentType, data)
),
connectionContext = noCertificateCheckContext
)
The above code generates the following log on the good node:
Calling URL [http://10.233.54.59:8080] with bytes response.
[t-dispatcher-22] [36mr$$anonfun$receive$1[m: Creating pool.
[t-dispatcher-22] [36mr$$anonfun$receive$1[m: Dispatching request [POST Strict(14160 bytes)] to pool
[t-dispatcher-21] [36mr$$anonfun$receive$1[m: [0 (Unconnected)]Dispatching request [POST / Strict(14160 bytes)]
[t-dispatcher-21] [36mr$$anonfun$receive$1[m: [0 (Unconnected)]Before event [onNewRequest] In state [Unconnected] for [0 ms]
[t-dispatcher-21] [36mr$$anonfun$receive$1[m: [0 (Unconnected)]Establishing connection
[t-dispatcher-21] [36mr$$anonfun$receive$1[m: [0 (Connecting)]After event [onNewRequest] State change [Unconnected] -> [Connecting]
[t-dispatcher-14] [36mr$$anonfun$receive$1[m: Resolving 10.233.54.59 before connecting
[...]
...and each of these calls succeeds.
The above code generates the following code on the bad node:
Calling URL [http://10.233.54.59:8080] with bytes response.
...and literally nothing after that. The bad node can accept incoming requests through its HTTP API written in AKKA and can do all things, but not Http().singleRequest, which is used to call Kubeless functions.
The only difference between the good and the bad node is that the bad node first starts an AKKA Cluster object for a short amount of time, does some stuff with it, then tears it down.
system.terminate()
Await.result(system.whenTerminated, timeout)
...and then it starts it again. The cluster can then be formed between the two nodes. Both nodes report that they are in a cluster and can gossip properly.
The bad node can not perform HTTP requests even from the first one. HTTP entities are properly subscribed using response.entity.toStrict(20 seconds) or response.discardEntityBytes().
Hints or ideas on how to fix/debug the problem further are much appreciated!
More details:
I start one AKKA cluster on Kubernetes using the following dependencies:
"com.typesafe.akka" %% "akka-actor" % "2.6.16",
"com.typesafe.akka" %% "akka-slf4j" % "2.6.16",
"com.typesafe.akka" %% "akka-stream" % "2.6.16",
"com.typesafe.akka" %% "akka-cluster" % "2.6.16",
"com.typesafe.akka" %% "akka-cluster-tools" % "2.6.16",
"com.typesafe.akka" %% "akka-cluster-metrics" % "2.6.16",
"com.typesafe.akka" %% "akka-stream-typed" % "2.6.16",
"com.typesafe.akka" %% "akka-actor-typed" % "2.6.16",
"com.typesafe.akka" %% "akka-cluster-typed" % "2.6.16",
"com.typesafe.akka" %% "akka-cluster-sharding-typed" % "2.6.16",
"com.typesafe.akka" %% "akka-stream-typed" % "2.6.16",
"com.typesafe.akka" %% "akka-discovery" % "2.6.16",
"com.lightbend.akka.management" %% "akka-management-cluster-bootstrap" % "1.1.1",
"com.lightbend.akka.discovery" %% "akka-discovery-kubernetes-api" % "1.1.1",
"com.typesafe.akka" %% "akka-stream-testkit" % "2.6.16",
"com.typesafe.akka" %% "akka-stream-testkit" % "2.6.16" % "test",
"com.typesafe.akka" %% "akka-http" % "10.2.6",
"com.typesafe.akka" %% "akka-http-core" % "10.2.6",
"com.typesafe.akka" %% "akka-http-testkit" % "10.2.6",
"com.typesafe.akka" %% "akka-http-spray-json" % "10.2.6",
"com.typesafe.akka" %% "akka-http-testkit" % "10.2.6" % "test",
Below is the default configuration.
akka {
test {
timefactor = 3.0
}
loglevel = "DEBUG"
stdout-loglevel = "DEBUG"
loggers = ["akka.event.slf4j.Slf4jLogger"]
logging-filter = "akka.event.slf4j.Slf4jLoggingFilter"
log-config-on-start = on
http {
parsing {
uri-parsing-mode = strict
error-logging-verbosity = full
}
server {
server-header = ""
verbose-error-messages = on
idle-timeout = 60s
max-connections = 2048
request-timeout = 30s
websocket.periodic-keep-alive-mode = pong
}
host-connection-pool {
max-connections = 32
max-open-requests = 64
max-connection-lifetime = 10 min
keep-alive-timeout = 10 min
response-entity-subscription-timeout = 2.second
}
routing {
verbose-error-messages = on
}
}
discovery {
method = config
}
management {
cluster.bootstrap {
contact-point-discovery {
service-name = "local-cluster"
discovery-method = config
required-contact-point-nr = 1
}
}
}
actor {
allow-java-serialization = on
warn-about-java-serializer-usage = off
provider = "cluster"
blocking-dispatcher {
type = "Dispatcher"
executor = "fork-join-executor"
fork-join-executor {
parallelism-min = 32
parallelism-max = 128
}
throughput = 1
}
}
cluster {
min-nr-of-members = 1
log-info-verbose = off
sharding {
passivate-idle-entity-after = off
}
}
remote {
log-remote-lifecycle-events = off
artery {
transport = tcp
canonical.port = 0
bind.port = 0
bind.bind-timeout = 10s
}
}
extensions = [
"akka.cluster.metrics.ClusterMetricsExtension",
"akka.cluster.pubsub.DistributedPubSub"
]
}
On Kubernetes, the configuration is overwritten with the following:
akka {
discovery {
kubernetes-api {
pod-label-selector = "app=my-app"
pod-domain = "cluster.local"
}
}
management {
cluster.bootstrap {
contact-point-discovery {
discovery-method = kubernetes-api
service-name = ${?AKKA_CLUSTER_BOOTSTRAP_SERVICE_NAME}
service-namespace = "staging.svc.cluster.local"
interval = 10 seconds
required-contact-point-nr = 0
}
}
}
cluster {
configuration-compatibility-check.enforce-on-join = off
shutdown-after-unsuccessful-join-seed-nodes = 120s
downing-provider-class = "akka.cluster.sbr.SplitBrainResolverProvider"
sharding {
use-lease = "akka.coordination.lease.kubernetes"
}
split-brain-resolver {
active-strategy = "lease-majority"
lease-majority {
lease-implementation = "akka.coordination.lease.kubernetes"
}
}
}
}
I am streaming data from mysql using Slick 3 and Akka Streams.
This is how I build my source
import slick.jdbc.MySQLProfile.api._
val enableJdbcStreaming: (java.sql.Statement) => Unit = {statement =>
if (statement.isWrapperFor(classOf[com.mysql.cj.jdbc.StatementImpl])) {
statement.unwrap(classOf[com.mysql.cj.jdbc.StatementImpl]).enableStreamingResults()
}
}
val query = Tables.Foo.filter(r => r.isActive === true)
.map(r => r.id).result.withStatementParameters(statementInit = enableJdbcStreaming)
Source.fromPublisher(db.stream(query))
My application runs for like 20 minutes and then shuts down with the following error
[error] Exception in thread "abhipool network timeout executor" java.lang.NullPointerException
[info] 15:31:46 INFO [HikariPool] - abhipool - Close initiated...
[error] at com.mysql.cj.mysqla.io.MysqlaProtocol.setSocketTimeout(MysqlaProtocol.java:1397)
[error] at com.mysql.cj.mysqla.MysqlaSession$1.run(MysqlaSession.java:401)
[error] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[error] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[error] at java.lang.Thread.run(Thread.java:745)
I have a feeling that because my query is running for a very long time there is some kind of timeout occurring which is initiating this shutdown.
My connection
mysql {
profile = "slick.jdbc.MySQLProfile$"
dataSourceClass = "slick.jdbc.DatabaseUrlDataSource"
properties {
driver = "com.mysql.cj.jdbc.Driver"
url = "jdbc:mysql://foo:3306/bar?useLegacyDatetimeCode=false&serverTimezone=America/Chicago"
user = "foo"
password = "bar"
}
connectionTimeout = 0
idleTimeout = 0
maxLifetime = 0
maxConnections = 40
minConnections = 10
poolName = "abhipool"
numThreads = 10
}
Dependencies
"com.typesafe.slick" %% "slick" % "3.2.1",
"com.typesafe.slick" %% "slick-hikaricp" % "3.2.1",
"mysql" % "mysql-connector-java" % "6.0.6",
How can I configure my application database connections so that even if my streaming application streams data for several days... it keeps running.
There is an extremely lengthy conversation about this same issue here but it doesn't tell me how to really fix this issue. This issues makes it totally impossible to write long running streaming tasks which use Mysql as a source.
You can configure the MySQL driver by adding parameters in the URL
url = "jdbc:mysql://foo:3306/bar?useLegacyDatetimeCode=false&serverTimezone=America/Chicago&socketTimeout=30000"
I put 30000 for the sake of the example, put the right value that fits your need
Hi I am trying to connect to HBase from Scala application in Play framework. I am following this link to establish the connection. My application is not running properly. I am accessing Hbase remotely through putty. I am having this play application in my local windows machine. Where & how to mention the HBase server connection details in the application?
conf/application.conf:
e# This is the main configuration file for the application.
# ~~~~~
# Secret key
# ~~~~~
# The secret key is used to secure cryptographics functions.
# If you deploy your application to several instances be sure to use the same key!
application.secret="?#3Y^s/S>oCNuO7If3Mq8]U285PqOG[bh/;^WVjZ#p5=`KljrbDrg4tBG6clCPuN"
# The application languages
# ~~~~~
application.langs="en"
# Global object class
# ~~~~~
# Define the Global object class for this application.
# Default to Global in the root package.
# application.global=Global
# Router
# ~~~~~
# Define the Router object to use for this application.
# This router will be looked up first when the application is starting up,
# so make sure this is the entry point.
# Furthermore, it's assumed your route file is named properly.
# So for an application router like `conf/my.application.Router`,
# you may need to define a router file `my.application.routes`.
# Default to Routes in the root package (and `conf/routes`)
# application.router=my.application.Routes
# Database configuration
# ~~~~~
# You can declare as many datasources as you want.
# By convention, the default datasource is named `default`
#
# db.default.driver=org.h2.Driver
# db.default.url="jdbc:h2:mem:play"
# db.default.user=sa
# db.default.password=""
#
# You can expose this datasource via JNDI if needed (Useful for JPA)
# db.default.jndiName=DefaultDS
# Evolutions
# ~~~~~
# You can disable evolutions if needed
# evolutionplugin=disabled
# Ebean configuration
# ~~~~~
# You can declare as many Ebean servers as you want.
# By convention, the default server is named `default`
#
# ebean.default="models.*"
# Logger
# ~~~~~
# You can also configure logback (http://logback.qos.ch/), by providing a logger.xml file in the conf directory .
# Root logger:
logger.root=ERROR
# Logger used by the framework:
logger.play=INFO
# Logger provided to your application:
logger.application=DEBUG
ERROR:
http://localhost:9000 gives me the web page with one form and add button. When I click that add button it redirects me to http://localhost:9000/bars url and gives below error on the web page itself
Bad request
For request 'POST /bars' [Expecting xml body]
There is no error log on the console.
My \play-hbase\app\controllers\Application.scala looks like this:
package controllers
import play.api.mvc.{Action, Controller}
import org.apache.hadoop.hbase.{HColumnDescriptor, HTableDescriptor, HBaseConfiguration}
import org.apache.hadoop.hbase.client._
import org.apache.hadoop.hbase.util.Bytes
import play.api.Logger
import play.api.libs.json.Json
import java.util.UUID
import scala.collection.JavaConversions._
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.hbase.client.{ConnectionFactory,HTable,Put,HBaseAdmin}
object Application extends Controller {
val barsTableName = "bars"
val family = Bytes.toBytes("all")
val qualifier = Bytes.toBytes("json")
lazy val hbaseConfig = {
println("Hi .... hbaseConfig ... START")
val conf:Configuration = HBaseConfiguration.create()
conf.set("hbase.zookeeper.quorum", "xxx.xxx.xxx.xxx") // xxx.xxx.xxx.xxx IP address of my Cloudera virtual machine.
conf.set("hbase.zookeeper.property.clientPort", "2181")
val hbaseAdmin = new HBaseAdmin(conf)
// create a table in HBase if it doesn't exist
if (!hbaseAdmin.tableExists(barsTableName)) {
val desc = new HTableDescriptor(barsTableName)
desc.addFamily(new HColumnDescriptor(family))
hbaseAdmin.createTable(desc)
Logger.info("bars table created")
}
// return the HBase config
println("Hi .... hbaseConfig ... END")
conf
}
def index = Action {
// return the server-side generated webpage from app/views/index.scala.html
println("Hi .... index ... START")
Ok(views.html.index("Play Framework + HBase"))
}
def addBar() = Action(parse.json) { request =>
// create a new row in the table that contains the JSON sent from the client
println("Hi .... addBar ... START")
val table = new HTable(hbaseConfig, barsTableName)
val put1 = new Put(Bytes.toBytes(UUID.randomUUID().toString))
put1.add(family, qualifier, Bytes.toBytes(request.body.toString()))
table.put(put1)
table.close()
println("Hi .... addBar ... END")
Ok
}
def getBars = Action {
// query the table and return a JSON list of the bars in the table
val table = new HTable(hbaseConfig, barsTableName)
val scan = new Scan()
scan.addColumn(family, qualifier)
val scanner = table.getScanner(scan)
val results = try {
scanner.toList.map {result =>
Json.parse(result.getValue(family, qualifier))
}
} finally {
scanner.close()
table.close()
}
Ok(Json.toJson(results))
}
}
My \play-hbase\conf\routes file looks like this:
# Routes
# This file defines all application routes (Higher priority routes first)
# ~~~~
GET / controllers.Application.index
GET /bars controllers.Application.getBars
POST /bars controllers.Application.addBar
# Map static resources from the /public folder to the /assets URL path
GET /assets/*file controllers.Assets.at(path="/public", file)
GET /webjars/*file controllers.WebJarAssets.at(file)
I added println() statements in my Application.scala file to check the flow. It is just printing :
Hi .... index ... START
Hi .... index ... START
My \play-hbase\app\views\index.scala file looks like this:
#(title: String)
<!DOCTYPE html>
<html>
<head>
<title>#title</title>
<link rel='shortcut icon' type='image/png' href='#routes.Assets.at("images/favicon.png")'>
<link rel='stylesheet' href='#routes.WebJarAssets.at(WebJarAssets.locate("bootstrap.min.css"))'>
<link rel='stylesheet' href='#routes.Assets.at("stylesheets/index.css")'>
<script type='text/javascript' src='#routes.WebJarAssets.at(WebJarAssets.locate("jquery.min.js"))'></script>
<script type='text/javascript' src='#routes.Assets.at("javascripts/index.min.js")'></script>
</head>
<body>
<div class="navbar navbar-fixed-top">
<div class="navbar-inner">
<div class="container-fluid">
<a id="titleLink" class="brand" href="/">#title</a>
</div>
</div>
</div>
<div class="container">
<div class="well">
<h3>Bars</h3>
<ul id="bars"></ul>
<hr>
<h3>Add a Bar</h3>
<form id="addBar" action="#routes.Application.addBar()" method="post">
<input id="barName" placeholder="Name">
<button>Add Bar</button>
</form>
</div>
</div>
</body>
</html>
My \play-hbase\build.sbt looks like this:
name := "play-hbase"
version := "1.0-SNAPSHOT"
libraryDependencies ++= Seq(
// Select Play modules
//jdbc, // The JDBC connection pool and the play.api.db API
//anorm, // Scala RDBMS Library
//javaJdbc, // Java database API
//javaEbean, // Java Ebean plugin
//javaJpa, // Java JPA plugin
//filters, // A set of built-in filters
//javaCore, // The core Java API
// WebJars pull in client-side web libraries
"org.webjars" %% "webjars-play" % "2.2.0-RC1-1",
"org.webjars" % "bootstrap" % "2.3.1",
// HBase
//"org.apache.hadoop" % "hadoop-core" % "1.2.1",
//"org.apache.hbase" % "hbase" % "0.94.11",
"org.apache.hadoop" % "hadoop-common" % "2.6.0",
"org.apache.hadoop" % "hadoop-client" % "2.6.0",
"org.apache.hbase" % "hbase" % "1.2.0",
"org.apache.hbase" % "hbase-client" % "1.2.0",
"org.apache.hbase" % "hbase-common" % "1.2.0",
"org.apache.hbase" % "hbase-server" % "1.2.0",
"org.slf4j" % "slf4j-log4j12" % "1.7.5"
// Add your own project dependencies in the form:
// "group" % "artifact" % "version"
)
play.Project.playScalaSettings
My \play-hbase\project\plugins.sbt looks like this:
// Comment to get more information during initialization
logLevel := Level.Warn
// The Typesafe repository
resolvers += "Typesafe repository" at "http://repo.typesafe.com/typesafe/releases/"
// Use the Play sbt plugin for Play projects
addSbtPlugin("com.typesafe.play" % "sbt-plugin" % "2.2.0-RC1")
My \play-hbase\project\build.properties looks like this:
#Activator-generated Properties
#Sat Oct 22 14:55:10 UTC 2016
template.uuid=148fc4a0-928a-42a0-81c8-98d83d1a656d
sbt.version=0.13.0
Thanks.
I'm trying to create a client for ES in SCALA [ school project ] .
but when I want to import Elastic search I got some problems
I've written a sbt file :
libraryDependencies += "org.elasticsearch" %% "elasticsearch" % "1.4.2"
libraryDependencies += "org.apache.lucene" % "lucene-core" % "4.10.2"
with other lucene
and when I try to use it :
import org.elasticsearch.node.Nodebuilder.*
object Setup {
Node node = nodeBuilder().node();
Client client = node.client();
}
it does recognize org.elasticsearch.node. but not .Nodebuilder.
Anyone has an idea ?
solved
import org.elasticsearch.node.NodeBuilder.nodeBuilder
val node = nodeBuilder().node()
val client = node.client()
I would suggest you to use the following library: https://github.com/sksamuel/elastic4s
I have been playing around with Akka Persistence and have written the following program to test my understanding. The problem is that I get different results each time I run this program. The correct answer is 49995000 but I don't always get that. I have cleaned out the journal directory between each run but it does not make any difference. Can anyone see what's going wrong? The program simply sums all the numbers from 1 to n (where n is 9999 in the code below).
The correct answer is : (n * (n+1)) / 2. For n=9999 that's 49995000.
EDIT: Seems to work more consistently with JDK 8 than with JDK 7. Should I be using JDK 8 only?
package io.github.ourkid.akka.aggregator.guaranteed
import akka.actor.Actor
import akka.actor.ActorPath
import akka.actor.ActorSystem
import akka.actor.Props
import akka.actor.actorRef2Scala
import akka.persistence.AtLeastOnceDelivery
import akka.persistence.PersistentActor
case class ExternalRequest(updateAmount : Int)
case class CountCommand(deliveryId : Long, updateAmount : Int)
case class Confirm(deliveryId : Long)
sealed trait Evt
case class CountEvent(updateAmount : Int) extends Evt
case class ConfirmEvent(deliveryId : Long) extends Evt
class TestGuaranteedDeliveryActor(counter : ActorPath) extends PersistentActor with AtLeastOnceDelivery {
override def persistenceId = "persistent-actor-ref-1"
override def receiveCommand : Receive = {
case ExternalRequest(updateAmount) => persist(CountEvent(updateAmount))(updateState)
case Confirm(deliveryId) => persist(ConfirmEvent(deliveryId)) (updateState)
}
override def receiveRecover : Receive = {
case evt : Evt => updateState(evt)
}
def updateState(evt:Evt) = evt match {
case CountEvent(updateAmount) => deliver(counter, id => CountCommand(id, updateAmount))
case ConfirmEvent(deliveryId) => confirmDelivery(deliveryId)
}
}
class FactorialActor extends Actor {
var count = 0
def receive = {
case CountCommand(deliveryId : Long, updateAmount:Int) => {
count = count + updateAmount
sender() ! Confirm(deliveryId)
}
case "print" => println(count)
}
}
object GuaranteedDeliveryTest extends App {
val system = ActorSystem()
val factorial = system.actorOf(Props[FactorialActor])
val delActor = system.actorOf(Props(classOf[TestGuaranteedDeliveryActor], factorial.path))
import system.dispatcher
system.scheduler.schedule(0 seconds, 2 seconds) { factorial ! "print" }
for (i <- 1 to 9999)
delActor ! ExternalRequest(i)
}
SBT file
name := "akka_aggregator"
organization := "io.github.ourkid"
version := "0.0.1-SNAPSHOT"
scalaVersion := "2.11.4"
scalacOptions ++= Seq("-unchecked", "-deprecation")
resolvers ++= Seq(
"Typesafe Repository" at "http://repo.typesafe.com/typesafe/releases/"
)
val Akka = "2.3.7"
val Spray = "1.3.2"
libraryDependencies ++= Seq(
// Core Akka
"com.typesafe.akka" %% "akka-actor" % Akka,
"com.typesafe.akka" %% "akka-cluster" % Akka,
"com.typesafe.akka" %% "akka-persistence-experimental" % Akka,
"org.iq80.leveldb" % "leveldb" % "0.7",
"org.fusesource.leveldbjni" % "leveldbjni-all" % "1.8",
// For future REST API
"io.spray" %% "spray-httpx" % Spray,
"io.spray" %% "spray-can" % Spray,
"io.spray" %% "spray-routing" % Spray,
"org.typelevel" %% "scodec-core" % "1.3.0",
// CSV reader
"net.sf.opencsv" % "opencsv" % "2.3",
// Logging
"com.typesafe.akka" %% "akka-slf4j" % Akka,
"ch.qos.logback" % "logback-classic" % "1.0.13",
// Testing
"org.scalatest" %% "scalatest" % "2.2.1" % "test",
"com.typesafe.akka" %% "akka-testkit" % Akka % "test",
"io.spray" %% "spray-testkit" % Spray % "test",
"org.scalacheck" %% "scalacheck" % "1.11.6" % "test"
)
fork := true
mainClass in assembly := Some("io.github.ourkid.akka.aggregator.TestGuaranteedDeliveryActor")
application.conf file
##########################################
# Akka Persistence Reference Config File #
##########################################
akka {
# Loggers to register at boot time (akka.event.Logging$DefaultLogger logs
# to STDOUT)
loggers = ["akka.event.slf4j.Slf4jLogger"]
# Log level used by the configured loggers (see "loggers") as soon
# as they have been started; before that, see "stdout-loglevel"
# Options: OFF, ERROR, WARNING, INFO, DEBUG
loglevel = "DEBUG"
# Log level for the very basic logger activated during ActorSystem startup.
# This logger prints the log messages to stdout (System.out).
# Options: OFF, ERROR, WARNING, INFO, DEBUG
stdout-loglevel = "INFO"
# Filter of log events that is used by the LoggingAdapter before
# publishing log events to the eventStream.
logging-filter = "akka.event.slf4j.Slf4jLoggingFilter"
# Protobuf serialization for persistent messages
actor {
serializers {
akka-persistence-snapshot = "akka.persistence.serialization.SnapshotSerializer"
akka-persistence-message = "akka.persistence.serialization.MessageSerializer"
}
serialization-bindings {
"akka.persistence.serialization.Snapshot" = akka-persistence-snapshot
"akka.persistence.serialization.Message" = akka-persistence-message
}
}
persistence {
journal {
# Maximum size of a persistent message batch written to the journal.
max-message-batch-size = 200
# Maximum size of a deletion batch written to the journal.
max-deletion-batch-size = 10000
# Path to the journal plugin to be used
plugin = "akka.persistence.journal.leveldb"
# In-memory journal plugin.
inmem {
# Class name of the plugin.
class = "akka.persistence.journal.inmem.InmemJournal"
# Dispatcher for the plugin actor.
plugin-dispatcher = "akka.actor.default-dispatcher"
}
# LevelDB journal plugin.
leveldb {
# Class name of the plugin.
class = "akka.persistence.journal.leveldb.LeveldbJournal"
# Dispatcher for the plugin actor.
plugin-dispatcher = "akka.persistence.dispatchers.default-plugin-dispatcher"
# Dispatcher for message replay.
replay-dispatcher = "akka.persistence.dispatchers.default-replay-dispatcher"
# Storage location of LevelDB files.
dir = "journal"
# Use fsync on write
fsync = on
# Verify checksum on read.
checksum = off
# Native LevelDB (via JNI) or LevelDB Java port
native = on
# native = off
}
# Shared LevelDB journal plugin (for testing only).
leveldb-shared {
# Class name of the plugin.
class = "akka.persistence.journal.leveldb.SharedLeveldbJournal"
# Dispatcher for the plugin actor.
plugin-dispatcher = "akka.actor.default-dispatcher"
# timeout for async journal operations
timeout = 10s
store {
# Dispatcher for shared store actor.
store-dispatcher = "akka.persistence.dispatchers.default-plugin-dispatcher"
# Dispatcher for message replay.
replay-dispatcher = "akka.persistence.dispatchers.default-plugin-dispatcher"
# Storage location of LevelDB files.
dir = "journal"
# Use fsync on write
fsync = on
# Verify checksum on read.
checksum = off
# Native LevelDB (via JNI) or LevelDB Java port
native = on
}
}
}
snapshot-store {
# Path to the snapshot store plugin to be used
plugin = "akka.persistence.snapshot-store.local"
# Local filesystem snapshot store plugin.
local {
# Class name of the plugin.
class = "akka.persistence.snapshot.local.LocalSnapshotStore"
# Dispatcher for the plugin actor.
plugin-dispatcher = "akka.persistence.dispatchers.default-plugin-dispatcher"
# Dispatcher for streaming snapshot IO.
stream-dispatcher = "akka.persistence.dispatchers.default-stream-dispatcher"
# Storage location of snapshot files.
dir = "snapshots"
}
}
view {
# Automated incremental view update.
auto-update = on
# Interval between incremental updates
auto-update-interval = 5s
# Maximum number of messages to replay per incremental view update. Set to
# -1 for no upper limit.
auto-update-replay-max = -1
}
at-least-once-delivery {
# Interval between redelivery attempts
redeliver-interval = 5s
# Maximum number of unconfirmed messages that will be sent in one redelivery burst
redelivery-burst-limit = 10000
# After this number of delivery attempts a `ReliableRedelivery.UnconfirmedWarning`
# message will be sent to the actor.
warn-after-number-of-unconfirmed-attempts = 5
# Maximum number of unconfirmed messages that an actor with AtLeastOnceDelivery is
# allowed to hold in memory.
max-unconfirmed-messages = 100000
}
dispatchers {
default-plugin-dispatcher {
type = PinnedDispatcher
executor = "thread-pool-executor"
}
default-replay-dispatcher {
type = Dispatcher
executor = "fork-join-executor"
fork-join-executor {
parallelism-min = 2
parallelism-max = 8
}
}
default-stream-dispatcher {
type = Dispatcher
executor = "fork-join-executor"
fork-join-executor {
parallelism-min = 2
parallelism-max = 8
}
}
}
}
}
Correct output:
18:02:36.684 [default-akka.actor.default-dispatcher-3] INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
18:02:36.684 [default-akka.actor.default-dispatcher-3] DEBUG akka.event.EventStream - logger log1-Slf4jLogger started
18:02:36.684 [default-akka.actor.default-dispatcher-3] DEBUG akka.event.EventStream - Default Loggers started
0
18:02:36.951 [default-akka.actor.default-dispatcher-14] DEBUG a.s.Serialization(akka://default) - Using serializer[akka.persistence.serialization.MessageSerializer] for message [akka.persistence.PersistentImpl]
18:02:36.966 [default-akka.actor.default-dispatcher-3] DEBUG a.s.Serialization(akka://default) - Using serializer[akka.serialization.JavaSerializer] for message [io.github.ourkid.akka.aggregator.guaranteed.CountEvent]
3974790
24064453
18:02:42.313 [default-akka.actor.default-dispatcher-11] DEBUG a.s.Serialization(akka://default) - Using serializer[akka.serialization.JavaSerializer] for message [io.github.ourkid.akka.aggregator.guaranteed.ConfirmEvent]
49995000
49995000
49995000
49995000
Incorrect run:
17:56:22.493 [default-akka.actor.default-dispatcher-4] INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
17:56:22.508 [default-akka.actor.default-dispatcher-4] DEBUG akka.event.EventStream - logger log1-Slf4jLogger started
17:56:22.508 [default-akka.actor.default-dispatcher-4] DEBUG akka.event.EventStream - Default Loggers started
0
17:56:22.750 [default-akka.actor.default-dispatcher-2] DEBUG a.s.Serialization(akka://default) - Using serializer[akka.persistence.serialization.MessageSerializer] for message [akka.persistence.PersistentImpl]
17:56:22.765 [default-akka.actor.default-dispatcher-7] DEBUG a.s.Serialization(akka://default) - Using serializer[akka.serialization.JavaSerializer] for message [io.github.ourkid.akka.aggregator.guaranteed.CountEvent]
3727815
22167811
17:56:28.391 [default-akka.actor.default-dispatcher-3] DEBUG a.s.Serialization(akka://default) - Using serializer[akka.serialization.JavaSerializer] for message [io.github.ourkid.akka.aggregator.guaranteed.ConfirmEvent]
49995000
51084018
51084018
52316760
52316760
52316760
52316760
52316760
Another incorrect run:
17:59:12.122 [default-akka.actor.default-dispatcher-3] INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
17:59:12.137 [default-akka.actor.default-dispatcher-3] DEBUG akka.event.EventStream - logger log1-Slf4jLogger started
17:59:12.137 [default-akka.actor.default-dispatcher-3] DEBUG akka.event.EventStream - Default Loggers started
0
17:59:12.387 [default-akka.actor.default-dispatcher-7] DEBUG a.s.Serialization(akka://default) - Using serializer[akka.persistence.serialization.MessageSerializer] for message [akka.persistence.PersistentImpl]
17:59:12.402 [default-akka.actor.default-dispatcher-13] DEBUG a.s.Serialization(akka://default) - Using serializer[akka.serialization.JavaSerializer] for message [io.github.ourkid.akka.aggregator.guaranteed.CountEvent]
2982903
17710176
49347145
17:59:18.204 [default-akka.actor.default-dispatcher-13] DEBUG a.s.Serialization(akka://default) - Using serializer[akka.serialization.JavaSerializer] for message [io.github.ourkid.akka.aggregator.guaranteed.ConfirmEvent]
51704199
51704199
55107844
55107844
55107844
55107844
You're using AtLeastOnceDelivery semantics. As it said here:
Note At-least-once delivery implies that original message send order
is not always preserved and the destination may receive duplicate
messages. That means that the semantics do not match those of a normal
ActorRef send operation:
it is not at-most-once delivery message order for the same
sender–receiver pair is not preserved due to possible resends after a
crash and restart of the destination messages are still delivered—to
the new actor incarnation These semantics is similar to what an
ActorPath represents (see Actor Lifecycle), therefore you need to
supply a path and not a reference when delivering messages. The
messages are sent to the path with an actor selection.
So some numbers may be received more than once. You can just ignore duplicate numbers inside FactorialActor or don't use this semantic.