Data streaming in Apache-Ignite through Apache-Kafka consumes high CPU
I am using Ignite Source Connector (Single Kafka connector node) to export the Ignite Events from my ignite cluster(2 Nodes) to Kafka broker nodes (2 Nodes) with single topic and 29 partitions.
I have processed 0.1 Million events (PUT, DELETE) per minute with the 1k size messages(in Avg).
My connector node consumes 90% of the CPU (you can see it below).
My Connector node machine config.
RAM --> 30 GB
ROM --> 1TB
Configured Heap --> 15GB
No of CPU Core --> 40 (20 x 2)
#connector
name=my-ignite-source-connector
connector.class=class of my source connector
tasks.max=1
topicNames=ignite-data
#cache
cacheName=DOCIDS
cacheAllowOverwrite=true
cacheEvts=put , removed
evtBatchSize=100
numberOfPartitions=29
igniteCfg=/myconfig/ignite-config.xml
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:util="http://www.springframework.org/schema/util"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/util
http://www.springframework.org/schema/util/spring-util.xsd">
<bean abstract="true" id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
<!-- Set to true to enable distributed class loading for examples, default is false. -->
<property name="communicationSpi"><bean class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi"><property name="socketWriteTimeout" value="60000"/></bean></property><property name="peerClassLoadingEnabled" value="true"/><property name="clientFailureDetectionTimeout" value="10000"/><property name="dataStorageConfiguration"><bean class="org.apache.ignite.configuration.DataStorageConfiguration"><property name="walPath" value="wal_data/"/><property name="walArchivePath" value="wal_data/"/><property name="defaultDataRegionConfiguration"><bean class="org.apache.ignite.configuration.DataRegionConfiguration"><property name="persistenceEnabled" value="true"/><property name="maxSize" value="#{15L * 1024 * 1024 * 1024}"/></bean></property></bean></property>
<!-- Enable task execution events for examples. -->
<property name="includeEventTypes">
<list>
<!--Task execution events-->
<util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_STARTED"/>
<util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_FINISHED"/>
<util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_FAILED"/>
<util:constant static-field="org.apache.ignite.events.EventType.EVT_CACHE_OBJECT_PUT"/>
<util:constant static-field="org.apache.ignite.events.EventType.EVT_CACHE_OBJECT_REMOVED"/>
<!--Cache events-->
</list>
</property>
<!-- Explicitly configure TCP discovery SPI to provide list of initial nodes. -->
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<!--
Ignite provides several options for automatic discovery that can be used
instead os static IP based discovery. For information on all options refer
to our documentation: http://apacheignite.readme.io/docs/cluster-config
-->
<!-- Uncomment static IP finder to enable static-based discovery of initial nodes. -->
<!--<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">-->
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
<property name="addresses">
<list>
<!-- In distributed environment, replace with actual host IP address. -->
<value>ignite-server-node1:47500..47509</value><value>ignite-server-node2:47500..47509</value><value>kafka-conect-node:47500..47509</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
</bean>
</beans>
The data transfer process is working as intended only for a few hours because of this high CPU consumption, after that the Kafka connector node will be OFFLINE in ignite server topology.
Any pointers will help.
Thanks in advance.
Related
Hazelcast 3.12.4 is deployed in Kubernetes 1.21 along with other Applications.
Whenever any application instance updates the cache in the central Hazelcast, an event is triggered and sent to all the Application instances which have stored that data as in-memory Near Cache.
The application invalidates the Near Cache data upon receiving this event from central Hazelcast so that it can fetch the latest data from the central Hazelcast cache.
It is observed that sometimes Hazelcast events are not received or not processed by a few application instances and hence the application's cached data does not get updated.
Because of this issue, few application instances have latest data in their Near Cache while some application instances have stale data in their Near Cache.
This issue is randomly occurring and no specific steps are observed to reproduce this issue.
Following is the hazelcast-client xml:
<hazelcast-client xmlns="http://www.hazelcast.com/schema/client-config"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.hazelcast.com/schema/client-config
http://www.hazelcast.com/schema/client-config/hazelcast-client-config-5.1.xsd">
<cluster-name>dev</cluster-name>
<instance-name>UPC</instance-name>
<properties>
<property name="hazelcast.client.shuffle.member.list">true</property>
<property name="hazelcast.client.heartbeat.timeout">600000</property>
<property name="hazelcast.client.heartbeat.interval">180000</property>
<property name="hazelcast.client.event.queue.capacity">1000000</property>
<property name="hazelcast.client.invocation.timeout.seconds">120</property>
<property name="hazelcast.client.statistics.enabled">false</property>
<property name="hazelcast.discovery.enabled">false</property>
<property name="hazelcast.invalidation.reconciliation.interval.seconds">31</property>
<property name="hazelcast.invalidation.max.tolerated.miss.count">1</property>
<property name="hazelcast.map.invalidation.batch.enabled">true</property>
<property name="hazelcast.map.invalidation.batch.size">5</property>
<property name="hazelcast.map.invalidation.batchfrequency.seconds">5</property>
</properties>
<network>
<cluster-members>
<address>10.151.32.200:31050</address>
</cluster-members>
<discovery-strategies>
<discovery-strategy enabled="false"
class="com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy">
<properties>
<!-- configure discovery service API lookup -->
<property name="service-name">hazelcast</property>
<property name="namespace">default</property>
</properties>
</discovery-strategy>
</discovery-strategies>
<smart-routing>true</smart-routing>
<redo-operation>true</redo-operation>
<connection-timeout>90000</connection-timeout>
</network>
<near-cache name="default">
<in-memory-format>OBJECT</in-memory-format>
<serialize-keys>true</serialize-keys>
<invalidate-on-change>true</invalidate-on-change>
<eviction eviction-policy="NONE" max-size-policy="ENTRY_COUNT"/>
</near-cache>
<connection-strategy async-start="false" reconnect-mode="ASYNC">
<connection-retry>
<initial-backoff-millis>2000</initial-backoff-millis>
<max-backoff-millis>60000</max-backoff-millis>
<multiplier>3</multiplier>
<cluster-connect-timeout-millis>5000</cluster-connect-timeout-millis>
<jitter>0.5</jitter>
</connection-retry>
</connection-strategy>
I have a very simple setup with 2 nodes (connected to each other).
I have a unit test that produces 10 messages on a queue, then consumes all the messages from the queue and then check that it received 10 messages.
Here is the producer and consumer setup
<bean id="queueListener" class="SessionAwareMessageListener<TextMessage>" />
<jms:listener-container container-type="default" destination-type="queue" connection-factory="brokerPooledConnectionFactory" acknowledge="auto">
<jms:listener destination="queue.private.mb.sanity.V4" ref="queueListener" />
</jms:listener-container>
<bean id="queuePublisher" class="org.springframework.jms.core.JmsTemplate">
<property name="connectionFactory" ref="brokerPooledConnectionFactory" />
<property name="defaultDestinationName" value="queue.private.mb.sanity.V4" />
</bean>
And the connection setup
<amq:connectionFactory
brokerURL="failover:(tcp://${broker1.host}:${broker1.port},tcp://${broker2.host}:${broker2.port})?maxReconnectAttempts=1&startupMaxReconnectAttempts=1"
closeTimeout="100" id="brokerConnectionFactory" />
<!-- Pools of connections -->
<bean id="brokerPooledConnectionFactory" class="org.apache.activemq.pool.PooledConnectionFactory" init-method="start" destroy-method="stop">
<property name="maxConnections" value="5" />
<property name="idleTimeout" value="0" />
<property name="connectionFactory" ref="brokerConnectionFactory" />
<property name="maximumActiveSessionPerConnection" value="100" />
<property name="timeBetweenExpirationCheckMillis" value="60000" />
<property name="expiryTimeout" value="600000" />
</bean>
What I see is that message are dispatched on the 2 nodes.
Message received on the 2nd node are all consumed.
Some messages received on the 1st node are routed to the 2nd node and consumed from the 2nd node.
The other messages received on the 1st node are neither routed nor consumed.
If I launch a consumer connecting directly to the 1st node, it is able to consume the messages.
The same test with a topic instead of a queue works fine.
Any idea why I'm facing that behavior?
Thanks
Nicolas
Indeed, it was a setup issue. The message-load-balancing was set to STRICT instead of ON_DEMAND.
I am using the below configuration for integrating activemq with kafka. I receive message from activemq and forwards it to kafka. However, i am noticing that messages are getting dequeued from JMS Queue but messages are not going to kafka.
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:jms="http://www.springframework.org/schema/integration/jms"
xmlns:integration="http://www.springframework.org/schema/integration"
xmlns:int-kafka="http://www.springframework.org/schema/integration/kafka"
xmlns:task="http://www.springframework.org/schema/task"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/integration/jms
http://www.springframework.org/schema/integration/jms/spring-integration-jms.xsd
http://www.springframework.org/schema/integration/kafka
http://www.springframework.org/schema/integration/kafka/spring-integration-kafka.xsd">
<jms:message-driven-channel-adapter
id="helloJMSAdapater" destination="helloJMSQueue" connection-factory="jmsConnectionfactory"
channel="helloChannel" extract-payload="true" />
<integration:channel id="helloChannel" />
<integration:service-activator id="sayHelloServiceActivator"
input-channel="helloChannel" ref="sayHelloService" method="sayHello" />
<int-kafka:outbound-channel-adapter
id="kafkaOutboundChannelAdapter" kafka-template="template"
auto-startup="false" sync="true" channel="helloChannel" topic="test1234"
>
</int-kafka:outbound-channel-adapter>
<bean id="template" class="org.springframework.kafka.core.KafkaTemplate">
<constructor-arg>
<bean class="org.springframework.kafka.core.DefaultKafkaProducerFactory">
<constructor-arg>
<map>
<entry key="bootstrap.servers" value="localhost:9092" />
<!--entry key="retries" value="5" /> <entry key="batch.size" value="16384"
/> <entry key="linger.ms" value="1" /> <entry key="buffer.memory" value="33554432"
/> < entry key="key.serializer" value="org.apache.kafka.common.serialization.StringSerializer"
/> <entry key="value.serializer" value="org.apache.kafka.common.serialization.StringSerializer"
/ -->
</map>
</constructor-arg>
</bean>
</constructor-arg>
</bean>
</beans>
Also, in case there is any issue from Kafka, it is not even reporting any exception stack trace.
Did i miss anything ?
Your messages are consumed by sayHelloServiceActivator.
So change your helloChannel channel type to
<publish-subscribe-channel id="helloChannel"/>
Default is DirectChannel
The DirectChannel has point-to-point semantics but otherwise is more
similar to the PublishSubscribeChannel than any of the queue-based
channel implementations described above. It implements the
SubscribableChannel interface instead of the PollableChannel
interface, so it dispatches Messages directly to a subscriber. As a
point-to-point channel, however, it differs from the
PublishSubscribeChannel in that it will only send each Message to a
single subscribed MessageHandler.
As #Hassen Bennour says, if you want to send a message to two consumers, you need a publish/subscribe channel.
That said, you have auto-startup="false" on the kafka adapter, so it won't even be subscribed to the channel.
If it was started, with your current configuration messages would be sent round-robin alternately to the service activator and adapter.
How do I use Apache Camel's ActiveMQ component to connect to a remote instance of ActiveMQ?
You can define an endpoint for each of your ActiveMQ instances:
<bean id="instance1"
class="org.apache.activemq.camel.component.ActiveMQComponent">
<property name="brokerURL" value="tcp://server1:61616"/>
</bean>
...
<bean id="instance2"
class="org.apache.activemq.camel.component.ActiveMQComponent">
<property name="brokerURL" value="tcp://server2:61616"/>
</bean>
and then use it in routes as
<from uri="instanceX:myQueue"/> or <to uri="instanceX:myQueue"/>
I am new to Spring Integration.
I've configured a Spring file inbound-channel-adapter, e.g.
<file:inbound-channel-adapter channel="channel1" directory="${location}" prevent-duplicates="true" filename-pattern="*.csv">
<si:poller>
<si:interval-trigger interval="1000"/>
</si:poller>
</file:inbound-channel-adapter>
<si:service-activator input-channel="channel1" output-channel="channel2" ref="filenameGenerator" method="generate"/>
Now this is working fine.
But this needs to be deployed in a clustered environment. I want to make sure that multiple instances in the cluster do not attempt to read the same file. So will this work in such environment?
If no, can I use Quartz scheduler like this:
<file:inbound-channel-adapter channel="channel1" directory="${location}" prevent-duplicates="true" filename-pattern="*.csv">
<si:poller task-executor="taskExecutor" fixed-rate="1000"/>
</file:inbound-channel-adapter>
<si:service-activator input-channel="channel1" output-channel="channel2" ref="filenameGenerator" method="generate"/>
<bean id="taskExecutor" class="org.springframework.scheduling.quartz.SimpleThreadPoolTaskExecutor">
<property name="threadCount" value="20"/>
<property name="threadNamePrefix" value="consumer"/>
</bean>
Will this work and solve my problem??
Or do I have to use Transaction?
I hope the question is clear.
Thanks,
Adi
When multiple processes are reading from the same directory it can be
desirable to lock files to prevent them from being picked up
concurrently. To do this you can use a FileLocker
Check out the documentation around file lockers here. It seems that you can do soemthing like this:
<file:inbound-channel-adapter ... >
<file:nio-locker/>
</file:inbound-channel-adapter>
When multiple processes are reading from the same directory it can be
desirable to lock files to prevent them from being picked up
concurrently. To do this you can use a FileLocker
To ensure that a quartz-scheduled job executes once and only once within a cluster, configure a persistent, clustered quartz job schedule. Here's a sample config, for Quartz 1.6.6:
<bean id="scheduler" class="org.springframework.scheduling.quartz.SchedulerFactoryBean">
<!-- Set whether any jobs defined on this SchedulerFactoryBean should
overwrite existing job definitions.
-->
<property name="overwriteExistingJobs" value="true" />
<property name="dataSource" ref="myTransactionalDataSource" />
<!-- nonTransactionalDataSource is only necessary with clustered Quartz with an XA DataSource.
-->
<property name="nonTransactionalDataSource" ref="myNonTransactionalDataSource" />
<property name="quartzProperties">
<props>
<prop key="org.quartz.jobStore.selectWithLockSQL">SELECT * FROM {0}LOCKS WITH(UPDLOCK,HOLDLOCK) WHERE LOCK_NAME = ?</prop>
<!--
Run in cluster. Quartz ensures persisted jobs are executed once within the
cluster
-->
<prop key="org.quartz.jobStore.isClustered">true</prop>
<!-- Each node in the cluster must have a unique instance id.
-->
<prop key="org.quartz.scheduler.instanceId">AUTO</prop>
<!-- Default clusterCheckinInterval is 15000
-->
<!-- <prop key="org.quartz.jobStore.clusterCheckinInterval">20000</prop>
-->
</props>
</property>
<property name="transactionManager" ref="transactionManager" />
- <!--
In Quartz 1.6.6, Quartz's ThreadPool interface is used when firing job triggers,
in org.quartz.core.QuartzSchedulerThread.
Quartz 1.x still starts some unmanaged threads, notably org.quartz.impl.jdbcjobstore.JobStoreSupport's
ClusterManager which is used when clustered=true. Quartz 2.0 should correct this problem.
-->
<property name="taskExecutor" ref="myTaskExecutor" />
</bean>