Is Kafka streaming to Ignite running in Transactional mode ACID? - apache-kafka

Question

I have a distributed database in Apache Ignite and a Apache Kafka streaming service that streams data to the Ignite cluster. The Kafka streamer works as following
Create ignite node to find cluster
Start kafka streamer singleton as a service in the cluster
Shut down the ignite node
The Ignite cluster is in Transactional mode, however I am unsure if this guarantees ACID or only enables it. Could this streaming service to Ignite be considered ACID?
Here is the code for the kafka streamer:
public class IgniteKafkaStreamerService implements Service {
private static final long serialVersionUID = 1L;
#IgniteInstanceResource
private Ignite ignite;
private KafkaStreamer<String, JSONObject> kafkaStreamer = new KafkaStreamer<>();
private IgniteLogger logger;
public static void main(String[] args) throws InterruptedException {
TcpDiscoverySpi spi = new TcpDiscoverySpi();
TcpDiscoveryMulticastIpFinder ipFinder = new TcpDiscoveryMulticastIpFinder();
// Set Multicast group.
//ipFinder.setMulticastGroup("228.10.10.157");
// Set initial IP addresses.
// Note that you can optionally specify a port or a port range.
ipFinder.setAddresses(Arrays.asList("127.0.0.1:47500..47509"));
spi.setIpFinder(ipFinder);
IgniteConfiguration cfg = new IgniteConfiguration();
// Override default discovery SPI.
cfg.setDiscoverySpi(spi);
Ignite ignite = Ignition.getOrStart(cfg);
// Deploy data streamer service on the server nodes.
ClusterGroup forServers = ignite.cluster().forServers();
IgniteKafkaStreamerService streamer = new IgniteKafkaStreamerService();
ignite.services(forServers).deployClusterSingleton("KafkaService", streamer);
ignite.close();
}
#Override
public void init(ServiceContext ctx) {
logger = ignite.log();
IgniteDataStreamer<String, JSONObject> stmr = ignite.dataStreamer("my_cache");
stmr.allowOverwrite(true);
stmr.autoFlushFrequency(1000);
List<String> topics = new ArrayList<>();
topics.add(0,"IoTData");
kafkaStreamer.setIgnite(ignite);
kafkaStreamer.setStreamer(stmr);
kafkaStreamer.setThreads(4);
kafkaStreamer.setTopic(topics);
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "NiFi-consumer");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "192.168.1.242:9092");
props.put("key.deserializer", StringDeserializer.class.getName());
props.put("value.deserializer", StringDeserializer.class.getName());
props.put("group.id", "hello");
kafkaStreamer.setConsumerConfig(props);
kafkaStreamer.setSingleTupleExtractor(msg -> {
JSONObject jsonObj = new JSONObject(msg.value().toString());
String key = jsonObj.getString("id") + "," + new Date(msg.timestamp());
JSONObject value = jsonObj.accumulate("date", new Date(msg.timestamp()));
return new AbstractMap.SimpleEntry<>(key, value);
});
}
#Override
public void execute(ServiceContext ctx) {
kafkaStreamer.start();
logger.info("KafkaStreamer started.");
}
#Override
public void cancel(ServiceContext ctx) {
kafkaStreamer.stop();
logger.info("KafkaStreamer stopped.");
}
}

Answer 1

KafkaStreamer uses IgniteDataStreamer implementations under the hood. IgniteDataStreamer isn't transactional by nature so there are no any transactional guarantees.