AffinityKeyMapped not working with Ignite 2.4/2.5/2.6 and Scala - scala

Question

Using Scala 2.11.7/Java 1.8.0_161/RHEL7.
We have two caches whose elements share the same affinity key. The affinity key is defined as follows:
case class IgniteTradePayloadKey(
#(AffinityKeyMapped#field)
tradeKey: TradeKey,
... other fields
) extends Serializable
case class IgniteDealPayloadKey(
#(AffinityKeyMapped#field)
tradeKey: TradeKey,
child: Int,
... other fields
) extends Serializable
Those are used as key to two ignite caches (Trades and Deals). We want instances of Trades and Deals to be collocated, as we perform computations using both. Think of them as having a parent/child relationship, and we would like to keep the parent and its children in the same node because our calculations require both. Parents are uniquely identified by TradeKey, so we use that in both caches to control affinity. Note that they are also used as part of the Ignite key itself. They are not part of the value.
This worked beautifully with Ignite 1.7; we then tried to upgrade to a more recent version of Ignite (we tried 2.4, 2.5 and 2.6), and without any code change whatsoever, there are children that are no longer collocated with their parents. Reverted back to 1.7 to be sure, and collocation works. We tried to override the affinity function with something simple (just a hash on the TradeKey), and again, it works with 1.7, but not with any of the 2.X versions listed above.
What are we missing?
Configuration as requested (apologies for the massive file). We tried with and without defining our own affinity function, with the same results.
<beans xmlns="http://www.springframework.org/schema/beans"
<bean id="propertyConfigurer"
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="systemPropertiesModeName" value="SYSTEM_PROPERTIES_MODE_FALLBACK"/>
<property name="searchSystemEnvironment" value="true"/>
</bean>
<bean id="CLIENT_MODE" class="java.lang.String">
<constructor-arg value="${IGNITE_CLIENT_MODE:false}" />
</bean>
<!-- Ignite common configuration -->
<bean abstract="true" id="common.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="gridName" value="MTS Trades Cache Grid" />
<property name="failureDetectionTimeout" value="60000"/>
<property name="clientFailureDetectionTimeout" value="60000"/>
<property name="peerClassLoadingEnabled" value="true"/>
<property name="clientMode" ref="CLIENT_MODE"/>
<property name="rebalanceThreadPoolSize" value="4"/>
<property name="deploymentMode" value="CONTINUOUS"/>
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="localPort" value="47700"/>
<property name="localPortRange" value="20"/>
<!-- Setting up IP finder for this cluster -->
<property name="ipFinder">
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
<property name="addresses">
<list>
<value>127.0.0.1:47700..47720</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
<property name="communicationSpi">
<bean class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
<property name="localPort" value="49100"/>
<property name="sharedMemoryPort" value="-1" />
<property name="messageQueueLimit" value="1024"/>
</bean>
</property>
<!-- Cache configuration -->
<property name="cacheConfiguration">
<list>
<!-- deals -->
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="dealPayloads" />
<property name="cacheMode" value="PARTITIONED" />
<property name="backups" value="0" />
<property name="OnheapCacheEnabled" value="true"/>
<property name="queryEntities">
<list>
<bean class="org.apache.ignite.cache.QueryEntity">
<!-- setting indexed type's key class -->
<property name="keyType" value="com.company.ignite.IgniteDealPayloadKey" />
<!-- setting indexed type's value class -->
<property name="valueType" value="com.company.ignite.IgniteDealPayload" />
</bean>
</list>
</property>
<property name="affinity">
<bean class="com.company.ignite.affinity.IgniteAffinityFunction">
<property name="partitions" value="1024"/>
</bean>
</property>
<property name="atomicityMode" value="ATOMIC" />
<property name="rebalanceMode" value="ASYNC" />
<property name="copyOnRead" value="false" />
<!-- Set rebalance batch size to 8 MB. -->
<property name="rebalanceBatchSize" value="#{8 * 1024 * 1024}"/>
<!-- Explicitly disable rebalance throttling. -->
<property name="rebalanceThrottle" value="0"/>
<!-- Set 4 threads for rebalancing. -->
<property name="rebalanceThreadPoolSize" value="4"/>
</bean>
<!-- trade versions -->
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="tradePayloads" />
<property name="cacheMode" value="PARTITIONED" />
<property name="backups" value="0" />
<property name="OnheapCacheEnabled" value="true"/>
<property name="queryEntities">
<list>
<bean class="org.apache.ignite.cache.QueryEntity">
<!-- setting indexed type's key class -->
<property name="keyType" value="com.company.ignite.IgniteTradePayloadKey" />
<!-- setting indexed type's value class -->
<property name="valueType" value="com.company.ignite.IgniteTradePayload" />
</bean>
</list>
</property>
<property name="affinity">
<bean class="com.company.ignite.affinity.IgniteAffinityFunction">
<property name="partitions" value="1024"/>
</bean>
</property>
<property name="atomicityMode" value="ATOMIC" />
<property name="rebalanceMode" value="ASYNC" />
<property name="copyOnRead" value="false" />
<!-- Set rebalance batch size to 8 MB. -->
<property name="rebalanceBatchSize" value="#{8 * 1024 * 1024}"/>
<!-- Explicitly disable rebalance throttling. -->
<property name="rebalanceThrottle" value="0"/>
<!-- Set 4 threads for rebalancing. -->
<property name="rebalanceThreadPoolSize" value="4"/>
</bean>
</list>
</property>
</bean>
</beans>
In addition, this is the relevant exception:
[22:28:24,418][INFO][grid-timeout-worker-#23%MTS Trades Cache Grid%][IgniteKernal%MTS Trades Cache Grid] FreeList [name=MTS Trades Cache Grid, buckets=256, dataPages=9658, reusePages=0]
[22:28:57,335][INFO][pub-#314%MTS Trades Cache Grid%][GridDeploymentLocalStore] Class locally deployed: class com.company.pt.tradesrouter.routing.ComputeJob
[22:28:57,705][SEVERE][pub-#314%MTS Trades Cache Grid%][GridJobWorker] Failed to execute job [jobId=48df049c461-b3ba568d-6a39-4296-b03f-0c046e7cf3f7, ses=GridJobSessionImpl [ses=GridTaskSessionImpl [taskName=com.company.pt.tradesrouter.routing.ComputeJob, dep=GridDeployment [ts=1532382444003, depMode=CONTINUOUS, clsLdr=sun.misc.Launcher$AppClassLoader#764c12b6, clsLdrId=a717c19c461-e5241c47-40d4-4085-a7fa-4f1916275b2e, userVer=0, loc=true, sampleClsName=o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionFullMap, pendingUndeploy=false, undeployed=false, usage=2], taskClsName=com.company.pt.tradesrouter.routing.ComputeJob, sesId=38df049c461-b3ba568d-6a39-4296-b03f-0c046e7cf3f7, startTime=1532384937062, endTime=1532388537300, taskNodeId=b3ba568d-6a39-4296-b03f-0c046e7cf3f7, clsLdr=sun.misc.Launcher$AppClassLoader#764c12b6, closed=false, cpSpi=null, failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=false, topPred=null, subjId=b3ba568d-6a39-4296-b03f-0c046e7cf3f7, mapFut=IgniteFuture [orig=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1635946755]], execName=null], jobId=48df049c461-b3ba568d-6a39-4296-b03f-0c046e7cf3f7]]
class org.apache.ignite.IgniteException: null
at org.apache.ignite.internal.processors.closure.GridClosureProcessor$C2.execute(GridClosureProcessor.java:1858)
at org.apache.ignite.internal.processors.job.GridJobWorker$2.call(GridJobWorker.java:566)
at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6623)
at org.apache.ignite.internal.processors.job.GridJobWorker.execute0(GridJobWorker.java:560)
at org.apache.ignite.internal.processors.job.GridJobWorker.body(GridJobWorker.java:489)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1189)
at org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:1921)
at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
at com.company.pt.tradesrouter.routing.ComputeJob$$anonfun$1$$anonfun$2.apply(ComputeJob.scala:49)
at com.company.pt.tradesrouter.routing.ComputeJob$$anonfun$1$$anonfun$2.apply(ComputeJob.scala:47)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
at scala.collection.immutable.Set$Set1.foreach(Set.scala:79)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
at scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)
at scala.collection.SetLike$class.map(SetLike.scala:92)
at scala.collection.AbstractSet.map(Set.scala:47)
at com.company.pt.tradesrouter.routing.ComputeJob$$anonfun$1.apply(ComputeJob.scala:47)
at com.company.pt.tradesrouter.routing.ComputeJob$$anonfun$1.apply(ComputeJob.scala:44)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
at scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:221)
at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428)
at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at com.company.pt.tradesrouter.routing.ComputeJob.call(ComputeJob.scala:44)
at com.company.pt.tradesrouter.routing.ComputeJob.call(ComputeJob.scala:21)
at org.apache.ignite.internal.processors.closure.GridClosureProcessor$C2.execute(GridClosureProcessor.java:1855)
... 14 more

Answer 1

Most probably, you hit a bug, that was introduced in Ignite 2.0: https://issues.apache.org/jira/browse/IGNITE-5795
Because of this bug #AffinityKeyMapped annotation is ignored in classes, that are used in query entity configuration. As far as I can see, this is exactly your case.
It's going to be fixed in Ignite 2.7.
There is a workaround for this problem: you can list the problematic classes in BinaryConfiguration#classNames configuration property. Binary configuration should be specified as IgniteConfiguration#binaryConfiguration. This configuration should be the same on all nodes. You may also need to configure CacheConfiguration#keyConfiguration for your cache.