Why is Akka creating so many dispatchers? - scala

Question

I'm using Akka for several different Actors. The work done by these Actors is non-blocking. I noticed something odd - the number of dispatchers scales with the number of Actors I'm creating. If I create hundreds of actors, I find myself with hundreds of dispatchers, sometimes over 1000.
This is, even though, most of the dispatchers look like this:
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000003d503de50> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
(basically, doing nothing most of the time)
I initialize dispatchers with calls like this:
ActorMaterializer(ActorMaterializerSettings(system).withDispatcher(s"akka.dispatchers.$dispatcherName"))(system))
My dispatcher configuration is below (we have different dispatchers for different actors):
dispatchers {
connector-actor-dispatcher {
type = Dispatcher
executor = "thread-pool-executor"
thread-pool-executor {
fixed-pool-size = 200
}
throughput = 1
}
http-actor-dispatcher {
type = Dispatcher
executor = "fork-join-executor"
fork-join-executor {
parallelism-min = 1
parallelism-factor = 1.0
parallelism-max = 64
task-peeking-mode = "FIFO"
}
throughput = 1
}
commands-dispatcher {
type = Dispatcher
executor = "fork-join-executor"
fork-join-executor {
parallelism-min = 1
parallelism-factor = 1.0
parallelism-max = 64
task-peeking-mode = "FIFO"
}
throughput = 1
}
http-server-dispatcher {
type = Dispatcher
executor = "thread-pool-executor"
thread-pool-executor {
core-pool-size-factor = 1
}
throughput = 1
}
http-client-dispatcher-low {
type = Dispatcher
executor = "thread-pool-executor"
thread-pool-executor {
core-pool-size-factor = 1
}
throughput = 1
}
http-client-dispatcher-high {
type = Dispatcher
executor = "thread-pool-executor"
thread-pool-executor {
core-pool-size-factor = 1
}
throughput = 1
}
http-client-dispatcher-parser {
type = Dispatcher
executor = "thread-pool-executor"
thread-pool-executor {
fixed-pool-size = 200
}
throughput = 1
}
}
}
What am I missing?

Answer 1

It seems that this mostly answered in the comments above, but to collate them into an "Answer": it appears that "so many dispatchers" are getting created because you are creating them explicitly in your config.
Also, when you give an example of a "dispatcher" you are actually showing a thread stack trace. So you might be confusing threads with dispatchers. And many of the the dispatchers you are creating have large thread counts. As #Tim says "If you are creating hundreds of actors and each actor has its own dispatcher with many threads, you are going to get a lot of threads!"
But that's really a tuning question. The answer to the direct question is that generally (with the exception of the system dispatcher), dispatchers are generally only created when you specifically ask for them to be created. And it appears that you are creating many of them. And that you also have enormous thread counts for each dispatcher.
As discussed, the general best practice is to have one dispatcher for non-blocking actors and another dispatcher with blocking actors. Each dispatcher will also generally only need a small number of threads. There are some edge cases where you might want additional dedicated dispatchers for particularly sensitive or badly behaving actors, but it depends on your actors and your application.