I am load testing a jboss messaging install with 5 producers producing 100,000 100k messages. I am seeing significant bottlenecking. When I monitor the profiler, I see there are 15 threads named WorkerThread#. These threads are allocated 100% with no waits. I think they may be related. Does anyone know what function these threads service and if there is a threadpool setting. I am using a supp
JBoss Enterprise Application Server 4.3 CP08
JBoss Enterprise Service Bus 4.4 CP04
JBoss Transactions 4.2.3._CP07
JBoss Messaging 1.4.0.SP3-CP09
JBoss Rules 4.0.7
JBoss jBPM 3.2.9
JBoss Web Services 2.0.1.SP2_CP07
I've figured it out. Its not a pool of threads. In the jboss-messaging.sar/remoting-bisocket.xml file that defines the remoting connector for Jboss Messaging, you see a couple of values mainly clientMaxPool, maxPoolSize, numAcceptThreads.
In remoting, when a socket is established threads are created to monitor that socket up to the value of "numAcceptThreads". All this thread does is read data from the socket and hand it off to a thread in the client pool(governed by maxPoolSize).
The threads called workerThread#[] refer to the accept threads. The reason that I see more when I create more producers is because for the bisocket transport for Jboss Messaging there apparently are three sockets created. Initially there are 3, but when I create 5 producers that number is increased to 15(or 5*3 for those not mathematically inclined :)). The reason they are 100% allocated is because when I am sending all those messages the threads read from the socket, hand off to Server Thread, go back to reading from the socket(where this is always data)
So the short answer is there is no pool to govern these threads. You can have more than 1 accept thread, but It would almost never make sense. This because its job is so minimal read the data, hand it off, read the data... So have more threads would just add synchronization overhead.
This is from http://download.oracle.com/javase/tutorial/uiswing/concurrency/worker.html; hope it helps.
When a Swing program needs to execute a long-running task, it usually uses one of the worker threads, also known as the background threads. Each task running on a worker thread is represented by an instance of javax.swing.SwingWorker. SwingWorker itself is an abstract class; you must define a subclass in order to create a SwingWorker object; anonymous inner classes are often useful for creating very simple SwingWorker objects.
Related
I'm just going through vert.x documentation and got confused by the part about standard verticles:
No more worrying about synchronized and volatile any more, and you also avoid many other cases of race conditions and deadlock so prevalent when doing hand-rolled 'traditional' multi-threaded application development.
This is the link to it: https://vertx.io/docs/vertx-core/java/#_standard_verticles
Is this statement true only if I deploy only one instance of standard verticle, and if my vert.x application isn't clustered?
only if I deploy only one instance of standard verticle, and if my vert.x application isn't clustered?
Each verticle deployed is single threaded. So if you have 3 instances - each of them individually are single threaded.
vert.x application isn't clustered?
Not related. Clustered is across processes/machines - here we are talking about threads
In our architecture, we are assuming to run three jvm processes on one machine (approx.) and each jvm machine can host upto 15 kafka-stream apps.
And if I am not wrong each kafka-stream app spawns one java thread. So, this seems like an awkward architecture to have with around 45 kafka-stream apps running on a single machine.
So, I have question in three parts
1) Is my understanding correct that each kafka-stream app spawns one java thread ? Also, each kafka-stream starts a new tcp connection with kafka-broker ?
2) Is there a way to share one tcp connection for multiple kafka-streams ?
3) Is is difficult(not recommended) to run 45 streams on single machine ?
The answer to this is definitely NO unless there is a real use case in production.
Multiple answers:
a KafkaStreams instance start one processing thread by default (you
can configure more processing threads, too)
internally, KafkaStreams uses two KafkaConsumers and one KafkaProducer
(if you turn on EOS, it uses even more KafkaProducers): a KafkaConsumer
starts a background heartbeat thread and a KafkaProducer starts a
background sender thread => you get 4 threads in total (processing, 2x
heartbeat, sender) -- if you configure two processing threads, you end
up with 8 threads in total, etc)
there is more than one TCP connection as the consumer and the producer
(and the restore consumer, if you enable StandbyTasks) connect to the
cluster
it's not possible to share any TPC connections atm (this would require
a mayor rewrite of consumers and producers)
how many threads you can efficient run depends on your hardware and
workload... monitor you CPU utilization and see how buys your machine is...
Each Kafka stream job spawns a single thread.If the thread number is
set as n numbers it will provide parallelism in processing n number
of Kafka partitions.
If a single machine does not have the capacity to run large number of
threads, parallelism can be achieved by submitting the Streams
applications job with same application name in another machine
in the same cluster. The job will be identified by Kafka
streams and handled in background.
is is difficult(not recommended) to run 45 streams on single machine
? The answer to this is definitely NO unless there is a real use
case in production.--unless your system has these many cores
or the input has 45 partition this is not necessary
I frequently see queues in software architecture, especially those called "scalable" with prominent representative of Actor from Akka.io multi-actor platform. However, how can queue be scalable, if we have to synchronize placing messages in queue (and therefore operate in single thread vs multi thread) and again synchronize taking out messages from queue (to assure, that message it taken exactly once)? It get's even more complicated, when those messages can change state of (actor) system - in this case even after taking out message from queue, it cannot be load balanced, but still processed in single thread.
Is it correct, that putting messages in queue must be synchronized?
Is it correct, that putting messages out of queue must be synchronized?
If 1 or 2 is correct, then how is queue scalable? Doesn't synchronization to single thread immediately create bottleneck?
How can (actor) system be scalable, if it is statefull?
Does statefull actor/bean mean, that I have to process messages in single thread and in order?
Does statefullness mean, that I have to have single copy of bean/actor per entire system?
If 6 is false, then how do I share this state between instances?
When I am trying to connect my new P2P node to netowrk, I believe I have to have some "server" that will tell me, who are other peers, is that correct? When I am trying to download torrent, I have to connect to tracker - if there is "server" then we do we call it P2P? If this tracker will go down, then I cannot connect to peers, is that correct?
Is synchronization and statefullness destroying scalability?
Is it correct, that putting messages in queue must be synchronized?
Is it correct, that putting messages out of queue must be synchronized?
No.
Assuming we're talking about the synchronized java keyword then that is a reenetrant mutual exclusion lock on the object. Even multiple threads accessing that lock can be fast as long as contention is low. And each object has its own lock so there are many locks, each which only needs to be taken for a short time, i.e. it is fine-grained locking.
But even if it did, queues need not be implemented via mutual exclusion locks. Lock-free and even wait-free queue data structures exist. Which means the mere presence of locks does not automatically imply single-threaded execution.
The rest of your questions should be asked separately because they are not about message queuing.
Of course you are correct in that a single queue is not scalable. The point of the Actor Model is that you can have millions of Actors and therefore distribute the load over millions of queues—if you have so many cores in your cluster. Always remember what Carl Hewitt said:
One Actor is no actor. Actors come in systems.
Each single actor is a fully sequential and single-threaded unit of computation. The whole model is constructed such that it is perfectly suited to describe distribution, though; this means that you create as many actors as you need.
Frustratingly, HawtDispatch's website describes it as "thread pooling and NIO event notification framework API."
Let's take the 'thread pooling' part first. Most of the Executors provided by Java are also basically thread pools. How is HawtDispatch different?
It is also apparently an "NIO event notification framework API." I'm assuming it is a thin layer on top NIO which takes incoming data and passes to its notion of 'thread pool,' and passes it to a consumer when the thread pool scheduler finds the time. Correct? (Any improvement over NIO is welcomed). Has anyone done any performance analysis of netty vs HD?
HawtDispatch is designed to be a single system wide fixed size thread pool. It provides implements 2 flavors of Java Executors:
Global Dispatch Queue: Submitted Runnable objects are executed concurrently (You get the same effect using a Executors.newFixedThreadPool(n) executor)
Serial Dispatch Queue: Submitted Runnable objects are executed serially (You get the same effect using a Executors.newSingleThreadExecutor() executor)
Unlike the java executor model all global and serial dispatch queues share a single fixed size thread pool. You can use thousands of serial dispatch queues without increasing your thread count. Serial dispatch queues can be used like Erlang mailboxes to drive reactive actor style applications.
Since HawtDispatch is using a fixed size thread pool for processing all global and serial queue executions, all Runnable tasks it executes must be non-blocking. In a way this is similar to the NodeJS architecture except it using multiple threads instead of just one.
In comparison to Netty, HawtDispatch is not a framework for actually processing socket data. It does not provide a framework for how encode/decode, buffer and process the socket data. All it does is execute a user configured Runnable when data can be read or written on the non-blocking socket. It's up to you application then to actually read/write the socket data.
Is there a way to monitor how many threads are actually alive and running my scala actors ?
The only way to properly do this is to inject your own executor for the actors subsystem as, by default, the actor threads do not have actor- or scala-specific names (they may just be called Thread-N or pool-N-thread-M depending on which version of Scala you are using.
Philip Haller has given instructions on using your own executor, where you can monitor thread usage if you wish, or at the very least name the threads so created. If you override thread naming you could then use the standard Java system MBeans (i.e. ThreadMXBean) to monitor the threads programmatically (or via the JConsole/JVisualVM).
Note that you can control the default mechanism using the system properties:
actors.minPoolSize
actors.maxPoolSize
actors.corePoolSize
You might try the VisualVM tool (available free from Sun). Among other things, it can monitor threads in running JVMs.