Unique Transactional IDs for Kafka Producer in distributed running mode - apache-kafka

Question

I have a big data application that is based on the process Consume -> Process -> Produce. I am using Kafka in my ingestion pipeline and I am using the transactional producer for producing messages. All pieces of my application run fine, however there is a small problem in generating the IDs for the Transactional Producer. Scenario:
Say my application is running on one machine, I instantiate 2 consumer which have their own producers, so for e.g. lets say
Producer 1 has the transactional ID -> Consumer-0-Producer
Producer 2 has the transactional ID -> Consumer-1-Producer
now transactions initiated by these two producers will not interfere with each other, and this is what I desire. Pseudo code looks something like this:
ExecutorService executorService// responsible for starting my consumers
for (int i = 0; i < 2; i++) {
prod_trans_id = "consumer-" + str(i) + "-producer"
Custom_Consumer consumer = new Custom_Consumer(prod_trans_id)
executorService.submit(consumer)
}
This works perfectly fine if my application works on a single machine, however, this is not the case as the application needs to be run on multiple machines so when the same code is run on machine 2 the producers instantiated by the consumers on machine 2 will have same transactional ID as on machine 1. I want transactional IDs to be produced in a way that they don't conflict with one another as well as they are reproducible, which means in case if a application crashes/stops (say someone does service application stop and then service application start) and when it comes back online, then it should use the same Transactional IDs that were being used previously. I thought of UUIDs based approach, however, UUIDs are random and will not be the same when the application on one machine dies and comes back up online.

Answer 1

private final static String HOSTNAME_COMMAND = "hostname";
public static String getHostName() {
BufferedReader inputStreamReader = null;
BufferedReader errorStreamReader = null;
try {
Process process = Runtime.getRuntime().exec(HOSTNAME_COMMAND);
inputStreamReader = new BufferedReader(new InputStreamReader(process.getInputStream()));
errorStreamReader = new BufferedReader(new InputStreamReader(process.getErrorStream()));
if (errorStreamReader.readLine() != null) {
throw new RuntimeException(String.format("Failed to get the hostname, exception message: %s",
errorStreamReader.readLine()));
}
return inputStreamReader.readLine();
} catch (IOException e) {
try {
if (inputStreamReader != null) {
inputStreamReader.close();
}
if (errorStreamReader != null) {
errorStreamReader.close();
}
} catch (IOException e1) {
LogExceptionTrace.logExceptionStackTrace(e1);
throw new RuntimeException(e1);
}
LogExceptionTrace.logExceptionStackTrace(e);
throw new RuntimeException(e);
}
}
And then use the hostname as follows:
final String producerTransactionalID = String.format("%s_producer", this.consumerName);
Where consumer name is set as follows:
for (int i = 0; i < NUMBER_OF_CONSUMERS; i++) {
String consumerName = String.format("%s-worker-%d", hostName, i);
Executor executor = new Executor(
Configuration, consumerName
);
Executors.add(executor);
futures.add(executorService.submit(executor));
}