Drools performance - drools

Question

I have an issue regarding performance of Drools on different Machines.
I made very simple JMH Benchmark test:
package ge.magticom.rules.benchmark;
import ge.magticom.rules.benchmark.Subscriber
rule "bali.free.smsparty"
activation-group "main"
salience 4492
when
$subs:Subscriber(chargingProfileID == 2)
then
$subs.setResult(15);
end
rule "bali.free.smsparty5"
activation-group "main"
salience 4492
when
$subs:Subscriber(chargingProfileID == 3)
then
$subs.setResult(14);
end
#Benchmark
public Subscriber send() throws Exception {
Subscriber subscriber = new Subscriber();
subscriber.setChargingProfileID(5);
StatelessKieSession session = ruleBase.newStatelessKieSession();
ArrayList<Object> objs = new ArrayList<Object>();
objs.add(subscriber);
session.execute(objs);
return subscriber;
}
On Home development machine
NAME="Ubuntu"
VERSION="20.04.2 LTS (Focal Fossa)"
(Intel(R) Core(TM) i7-8700 CPU # 3.20GHz 12 Threads ) 64 GB Memory with JDK 11 and have very great performance:
With 7 threads a have nearly 2M operation per second(Stateless)
Benchmark Mode Cnt Score Error Units
RulesBenchmark.send thrpt 5 2154292.750 ± 149405.498 ops/s
But on preproduction server Which is Intel(R) Xeon(R) Gold 6258R CPU # 2.70GHz with 112 Threads and 1 TB RAM I have half of performance (Even increasing threads)
NAME="Oracle Linux Server"
VERSION="8.4"
Benchmark Mode Cnt Score Error Units
RulesBenchmark.send thrpt 5 1084939.195 ± 107897.663 ops/s
I'm trying to test our billing system with java 11 and Drolls 7.54.0.Final.
Our system was based on Jrockit realtime 1.6 and drools version 4.0.3. We are moving system from Sun Solaris SPARK to Intel base system.
Running same rules with Jrockit 1.6 I got even worth performance issue with Home and Preproduction environment:
Home test benchmark:
Benchmark Mode Cnt Score Error Units
RulesBenchmark.send thrpt 20 692054.563 ± 3507.519 ops/s
Preproduction benchmark:
Benchmark Mode Cnt Score Error Units
RulesBenchmark.send thrpt 20 382283.288 ± 6405.953 ops/s
As you can see, it's nearly half performance of very simple rules.
But for real rules, such as our online charging system, it's even bad performance :
On home environment I got
Benchmark Mode Cnt Score Error Units
WorkerBenchmark.send thrpt 5 152.846 ± 87.076 ops/s
this means 1 message contains nearly 100 iterations
so in 00:01:49 benchmark processed 16287 sessions with 430590 events of rule calls. single rule call is about 2.33 millisecond in average, which is not very great, but not as bad as on preproduction
On Preproduction server
Benchmark Mode Cnt Score Error Units
WorkerBenchmark.send thrpt 5 35.013 ± 9.565 ops/s
in 00:01:54 I got only 3723 sessions which contains wholly 98571 events of rule calls. Each call is 10.7299 msc in average.
During running all these benchmark nothing was running on preproduction system. But on home environment there is a lot of development tools, was running tests from Intellij IDEA
Can you suggest anything, which may cause such difference in performance. I tried different java versions and vendors. These results are based on oracle-jdk-11.0.8.
Here are kernel params of Preproduction server:
fs.file-max = 6815744
kernel.sem = 2250 32000 100 128
kernel.shmmni = 4096
kernel.shmall = 1073741824
kernel.shmmax = 4398046511104
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
net.ipv4.conf.all.rp_filter = 2
net.ipv4.conf.default.rp_filter = 2
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 9000 65500

Answer 1

This is just a very wild guess since I definitively don't have enough information, but are the 2 environments using the same garbage collectors configured in the same way? Maybe you're using ParallelGC (which in my experience is better for pure throughput as you're measuring) on one side and G1 on the other?

Answer 2

Thanks for answer.
I used several GC configuration, none of them were ParallelGC. I think GC is not a problem. I used ZGC in final tests and GC pause times are not above 5 msc (tested also with java 16 and pause times are below 100 microsecond ). :
#Fork(value = 2, jvmArgs = {"--illegal-access=permit", "-Xms10G", "-XX:+UnlockDiagnosticVMOptions", "-XX:+DebugNonSafepoints",
"-Xmx10G","-XX:+UnlockExperimentalVMOptions", "-XX:ConcGCThreads=5", "-XX:ParallelGCThreads=10", "-XX:+UseZGC", "-XX:+UsePerfData", "-XX:MaxMetaspaceSize=10G", "-XX:MetaspaceSize=256M"}
java -version
java version "11.0.8" 2020-07-14 LTS
Java(TM) SE Runtime Environment 18.9 (build 11.0.8+10-LTS)
Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.8+10-LTS, mixed mode)
Here is Flame graph generated with AsyncProfilers
As you can see, at home enviroment Java process is using 95% of whole time, but on server only 65%. the time difference is also obviouse :
RulesBenchmark.send thrpt 5 1612318.098 ± 64712.672 ops/s
Home Result FlameGraph.html
RulesBenchmark.send thrpt 5 775498.081 ± 72237.890 ops/s
Server Flame Graph.html