collectd process plugin too slow - centos

Question

We have quite a lot of Centos 7 and a few Centos 6 servers in our simulation farm, and installed collectd for monitoring them. There is a problem with the processes plugin that only manifests on the Centos 7 machines. When I look in the log file I get the following
# grep "read-function of the" /var/log/collectd.log | grep 2022-07-07
[2022-07-07 10:07:31] plugin_read_thread: read-function of the `processes' plugin took 3.130 seconds, which is above its read interval (3.000 seconds). You might want to adjust the `Interval' or `ReadThreads' settings.
[2022-07-07 10:07:49] plugin_read_thread: read-function of the `processes' plugin took 6.216 seconds, which is above its read interval (3.000 seconds). You might want to adjust the `Interval' or `ReadThreads' settings.
[2022-07-07 11:36:32] plugin_read_thread: read-function of the `processes' plugin took 45.422 seconds, which is above its read interval (3.000 seconds). You might want to adjust the `Interval' or `ReadThreads' settings.
[2022-07-07 11:38:36] plugin_read_thread: read-function of the `processes' plugin took 46.150 seconds, which is above its read interval (3.000 seconds). You might want to adjust the `Interval' or `ReadThreads' settings.
[2022-07-07 11:42:47] plugin_read_thread: read-function of the `processes' plugin took 49.933 seconds, which is above its read interval (3.000 seconds). You might want to adjust the `Interval' or `ReadThreads' settings.
[2022-07-07 11:44:28] plugin_read_thread: read-function of the `processes' plugin took 46.702 seconds, which is above its read interval (3.000 seconds). You might want to adjust the `Interval' or `ReadThreads' settings.
[2022-07-07 11:48:42] plugin_read_thread: read-function of the `processes' plugin took 50.039 seconds, which is above its read interval (3.000 seconds). You might want to adjust the `Interval' or `ReadThreads' settings.
[2022-07-07 11:49:28] plugin_read_thread: read-function of the `processes' plugin took 40.787 seconds, which is above its read interval (3.000 seconds). You might want to adjust the `Interval' or `ReadThreads' settings.
[2022-07-07 13:22:15] plugin_read_thread: read-function of the `processes' plugin took 19.180 seconds, which is above its read interval (3.000 seconds). You might want to adjust the `Interval' or `ReadThreads' settings.
This is a sample from a typical day. The strange thing is that although I have a 3 second interval, this happens only in no more than 20 measurements in a day (28.8K measurements per 24 hours). The delay in the messages can reach 150 seconds or more in rare cases. Since I am monitoring the servers, I cannot see particularly high cpu load, which never exceeds 50% cpu utilisation from any category (user,system,wait,interrupt) when those issues occur. Also free memory is never a problem. This particular machine is a reasonably powerful 2 socket (24 cores each) Xeon Gold 6248R with 762GB RAM
The main configuration
# Global settings
Interval 3
CheckThresholds true
CollectInternalStats true
Timeout 10
ReadThreads 10
WriteThreads 10
WriteQueueLimitHigh 4000
WriteQueueLimitLow 4000
FQDNLookup False
# Plugin loading
LoadPlugin logfile
LoadPlugin threshold
# Plugin Configuration
<Plugin "logfile">
LogLevel "info"
File "/var/log/collectd.log"
Timestamp true
</Plugin>
Include "/etc/collectd.d/*.conf"
The processes configuration is the following
LoadPlugin processes
<Plugin "processes">
ProcessMatch "lsf" "lsf.*lim"
ProcessMatch "collectd" "collectd"
</Plugin>
<Plugin "threshold">
<Plugin "processes">
Instance "lsf"
<Type "ps_count">
DataSource "processes"
FailureMin 1
</Type>
Instance "collectd"
<Type "ps_count">
DataSource "processes"
FailureMin 1
</Type>
</Plugin>
</Plugin>
In total I use between 11 to 13 read plugins (cpu, df, interface, load, memory, processes, swap, uptime, users, logfile, threshold and python or perl in some systems), and one write module (write_riemann). The messages appear only for the processes plugin
My hope was that if I could speed up the execution of the plugin, the issue will go away, but start to realise this is not enough. Initially I used 5 read threads, but have now increased it to 10. Also I started with the EPEL provided packages of collectd 5.8, but later recompiled from to 5.12 with the -O3 and newer compilers (gcc11). Finally I start the collectd service with nice -n -20 to give maximum priority. There has been an improvement with those changes, but not enough to make the messages disappear.
The reason I need the delay to be kept at minimum is because if I do not receive an update on the processes I monitor for 60 seconds, I generate an alert. I thought of increasing the downtime before an alert to 2 minutes, but I still get false positives at least 1 per day from all 300 servers. Rising it further will improve the situation but I am wondering if this is the right approach.
Any suggestions on how to debug this issue are more than welcome