The Flink consumer application I am developing reads from multiple Kafka topics. The messages published in the different topics adhere to the same schema (formatted as Avro). For schema management, I am using the Confluent Schema Registry.
I have been using the following snippet for the KafkaSource and it works just fine.
KafkaSource<MyObject> source = KafkaSource.<MyObject>builder()
.setBootstrapServers(BOOTSTRAP_SERVERS)
.setTopics(TOPIC-1, TOPIC-2)
.setGroupId(GROUP_ID)
.setStartingOffsets(OffsetsInitializer.earliest())
.setValueOnlyDeserializer(ConfluentRegistryAvroDeserializationSchema.forSpecific(MyObject.class, SCHEMA_REGISTRY_URL))
.build();
Now, I want to determine the topic-name for each message that I process. Since the current deserializer is ValueOnly, I started looking into the setDeserializer() method which I felt would give me access to the whole ConsumerRecord object and I can fetch the topic-name from that.
However, I am unable to figure out how to use that implementation. Should I implement my own deserializer? If so, how does the Schema registry fit into that implementation?
You can use the setDeserializer method with a KafkaRecordDeserializationSchema that might look something like this:
public class KafkaUsageRecordDeserializationSchema
implements KafkaRecordDeserializationSchema<UsageRecord> {
private static final long serialVersionUID = 1L;
private transient ObjectMapper objectMapper;
#Override
public void open(DeserializationSchema.InitializationContext context) throws Exception {
KafkaRecordDeserializationSchema.super.open(context);
objectMapper = JsonMapper.builder().build();
}
#Override
public void deserialize(
ConsumerRecord<byte[], byte[]> consumerRecord,
Collector<UsageRecord> collector) throws IOException {
collector.collect(objectMapper.readValue(consumerRecord.value(), UsageRecord.class));
}
#Override
public TypeInformation<UsageRecord> getProducedType() {
return TypeInformation.of(UsageRecord.class);
}
}
Then you can use the ConsumerRecord to access the topic and other metadata.
I took inspiration from the above answer (by David) and added the following custom deserializer -
KafkaSource<MyObject> source = KafkaSource.<MyObject>builder()
.setBootstrapServers(BOOTSTRAP_SERVERS)
.setTopics(TOPIC-1, TOPIC-2)
.setGroupId(GROUP_ID)
.setStartingOffsets(OffsetsInitializer.earliest())
.setDeserializer(KafkaRecordDeserializationSchema.of(new KafkaDeserializationSchema<Event>{
DeserializationSchema deserialzationSchema = ConfluentRegistryAvroDeserializationSchema.forSpecific(MyObject.class, SCHEMA_REGISTRY_URL);
#Override
public boolean isEndOfStream(Event nextElement) {
return false;
}
#Override
public String deserialize(ConsumerRecord<byte[], byte[]> consumerRecord) throws Exception {
Event event = new Event();
event.setTopicName(record.topic());
event.setMyObject((MyObject) deserializationSchema.deserialize(record.value()));
return event;
}
#Override
public TypeInformation<String> getProducedType() {
return TypeInformation.of(Event.class);
}
})).build();
The Event class is a wrapper over the MyObject class with additional field for storing the topic name.
Related
I'm trying to read JSON events from Kafka, aggregate it on a eventId and its category and write them to a different kafka topic through flink. The program is able to read messages from kafka, but KafkaSink is not writing the data back to the other kafka topic. I'm not sure on the mistake I'm doing. Can someone please check and let me know, where I'm wrong. Here is the code I'm using.
KafkaSource<EventMessage> source = KafkaSource.<EventMessage>builder()
.setBootstrapServers(LOCAL_KAFKA_BROKER)
.setTopics(INPUT_KAFKA_TOPIC)
.setGroupId(LOCAL_GROUP)
.setStartingOffsets(OffsetsInitializer.earliest())
.setValueOnlyDeserializer(new InputDeserializationSchema())
.build();
WindowAssigner<Object, TimeWindow> windowAssigner = TumblingEventTimeWindows.of(WINDOW_SIZE);
DataStream<EventMessage> eventStream = env.fromSource(source, WatermarkStrategy.noWatermarks(), "Event Source");
DataStream<EventSummary> events =
eventStream
.keyBy(eventMessage -> eventMessage.getCategory() + eventMessage.getEventId())
.window(windowAssigner)
.aggregate(new EventAggregator())
.name("EventAggregator test >> ");
KafkaSink<EventSummary> sink = KafkaSink.<EventSummary>builder()
.setBootstrapServers(LOCAL_KAFKA_BROKER)
.setRecordSerializer(KafkaRecordSerializationSchema.builder()
.setTopic(OUTPUT_KAFKA_TOPIC)
.setValueSerializationSchema(new OutputSummarySerializationSchema())
.build())
.setDeliverGuarantee(DeliveryGuarantee.AT_LEAST_ONCE)
.build();
events.sinkTo(sink);
These are the POJO's I've created for input message and output.
# EventMessage POJO
public class EventMessage implements Serializable {
private Long timestamp;
private int eventValue;
private String eventId;
private String category;
public EventMessage() { }
public EventMessage(Long timestamp, int eventValue, String eventId, String category) {
this.timestamp = timestamp;
this.eventValue = eventValue;
this.eventId = eventId;
this.category = category;
}
.....
}
# EventSummary POJO
public class EventSummary {
public EventMessage eventMessage;
public int sum;
public int count;
public EventSummary() { }
....
}
These are the deserialization and serialization schemas I'm using.
public class InputDeserializationSchema implements DeserializationSchema<EventMessage> {
static ObjectMapper objectMapper = new ObjectMapper();
#Override
public EventMessage deserialize(byte[] bytes) throws IOException {
return objectMapper.readValue(bytes, EventMessage.class);
}
#Override
public boolean isEndOfStream(EventMessage inputMessage) {
return false;
}
#Override
public TypeInformation<EventMessage> getProducedType() {
return TypeInformation.of(EventMessage.class);
}
}
public class OutputSummarySerializationSchema implements SerializationSchema<EventSummary> {
static ObjectMapper objectMapper = new ObjectMapper();
Logger logger = LoggerFactory.getLogger(OutputSummarySerializationSchema.class);
#Override
public byte[] serialize(EventSummary eventSummary) {
if (objectMapper == null) {
objectMapper.setVisibility(PropertyAccessor.FIELD, JsonAutoDetect.Visibility.ANY);
objectMapper = new ObjectMapper();
}
try {
String json = objectMapper.writeValueAsString(eventSummary);
return json.getBytes();
} catch (com.fasterxml.jackson.core.JsonProcessingException e) {
logger.error("Failed to parse JSON", e);
}
return new byte[0];
}
}
I'm using this aggregator for aggregating the JSON messages.
public class EventAggregator implements AggregateFunction<EventMessage, EventSummary, EventSummary> {
private static final Logger log = LoggerFactory.getLogger(EventAggregator.class);
#Override
public EventSummary createAccumulator() {
return new EventSummary();
}
#Override
public EventSummary add(EventMessage eventMessage, EventSummary eventSummary) {
eventSummary.eventMessage = eventMessage;
eventSummary.count += 1;
eventSummary.sum += eventMessage.getEventValue();
return eventSummary;
}
#Override
public EventSummary getResult(EventSummary eventSummary) {
return eventSummary;
}
#Override
public EventSummary merge(EventSummary summary1, EventSummary summary2) {
return new EventSummary(null,
summary1.sum + summary2.sum,
summary1.count + summary2.count);
}
}
Can someone help me on this?
Thanks in advance.
In order for event time windowing to work, you must specify a proper WatermarkStrategy. Otherwise, the windows will never close, and no results will be produced.
The role that watermarks play is to mark a place in a stream, and indicate that the stream is, at that point, complete through some specific timestamp. Until receiving this indicator of stream completeness, windows continue to wait for more events to be assigned to them.
To simply the debugging the watermarks, you might switch to a PrintSink until you get the watermarking working properly. Or to simplify debugging the KafkaSink, you could switch to using processing time windows until the sink is working.
I had an issue with injecting a custom ObjectMapper into Spring Kafka serializer which I have resolved with this answer, LocalDateTime are getting serialized with right pattern.
#Configuration
public class KafkaCustomizer implements DefaultKafkaProducerFactoryCustomizer {
#Bean
public ObjectMapper objectMapper() {
var mapper = new ObjectMapper();
var module = new JavaTimeModule();
var serializer = new LocalDateTimeSerializer(
DateTimeFormatter.ofPattern(DateConstants.DATETIME_FORMAT_PATTERN));
module.addSerializer(LocalDateTime.class, serializer);
mapper.registerModule(module);
mapper.disable(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS);
return mapper;
}
#Override
public void customize(DefaultKafkaProducerFactory<?, ?> producerFactory) {
producerFactory.setValueSerializer(new JsonSerializer<>(objectMapper()));
}
}
But now I face another problem, the spring.kafka.producer.properties.spring.json.type.mapping property is being ignored.
The __TypeId__ header of my record is set with FQCN and not with the token I have put in spring.json.type.mapping property : foo > com.foo.package.Foo
When I did debbug it seems that the configure method of org.springframework.kafka.support.serializer.JsonSerializer class is not being invoked :
#Override
public void configure(Map<String, ?> configs, boolean isKey) {
...
if (configs.containsKey(TYPE_MAPPINGS) && !this.typeMapperExplicitlySet
&& this.typeMapper instanceof AbstractJavaTypeMapper) {
((AbstractJavaTypeMapper) this.typeMapper)
.setIdClassMapping(createMappings((String) configs.get(TYPE_MAPPINGS)));
}
}
But when I disable the customization
#Override
public void customize(DefaultKafkaProducerFactory<?, ?> producerFactory) {
// producerFactory.setValueSerializer(new JsonSerializer<>(objectMapper()));
}
Then the __TypeId__ header is set with right token But as expected I loose the date format with my custom ObjectMapper
So how to handle this whole situation ?
If you do your own new JsonSerializer<>, you are on your own to feed it with appropriate producer configs. When the instance of serialized is not controlled by Kafka Client, that configure() is not called.
I would say it is possible to do it like this in your case:
public void customize(DefaultKafkaProducerFactory<?, ?> producerFactory) {
JsonSerializer<Object> jsonSerializer = new JsonSerializer<>(objectMapper());
jsonSerializer.configure(producerFactory.getConfigurationProperties(), false);
producerFactory.setValueSerializer(jsonSerializer);
}
There is some info in docs: https://docs.spring.io/spring-kafka/docs/current/reference/html/#tip-json, but probably we need to extend it for the programmatic configuration case...
I've a requirement to add track and span id to Flink jobs running in cluster, the request flows something like below
User --> Rest API -> Kafka-topic-1 --> FlinkJob-1 --> Kafka-topic-2 --> FlinkJob-2 --> Consumer --> DB
I'm using Spring boot to create my rest APIs and using Spring Sleuth to add track and span id to generated logs, the track and span id is added when rest API is invoked and when message is put over Kakfa-topic-1 as well but I'm not able to figure out how to add track and span id while consuming message at FlinkJob-1 and FLinkJob-2 since they are out of spring context.
One way is to make track and span Id to kafka message headers and have Kafka Consumer/Producer interceptor to extract and log track and span Id, I tried this but my interceptors are not invoked as Flink APIs use Flink version of Kafka-client.
Couldn't get my custom KafkaDeserializationSchema invoked
public class MyDeserializationSchema implements KafkaDeserializationSchema<String> {
private static final Logger LOGGER = LoggerFactory.getLogger(MyDeserializationSchema.class);
#Override
public TypeInformation<String> getProducedType() {
System.out.println("************** Invoked 1");
LOGGER.debug("************** Invoked 1");
return null;
}
#Override
public boolean isEndOfStream(String nextElement) {
System.out.println("************** Invoked 2");
LOGGER.debug("************** Invoked 2");
return true;
}
#Override
public String deserialize(ConsumerRecord<byte[], byte[]> record) throws Exception {
System.out.println("************** Invoked 3");
LOGGER.debug("************** Invoked 3");
return record.toString();
}
}
Can someone please suggest me how to achieve same.
You can use KafkaDeserializationSchema in order to get the Header as well
For accessing the key, value and metadata of the Kafka message, the
KafkaDeserializationSchema has the following deserialize method T
deserialize(ConsumerRecord record).
public class Bla implements KafkaDeserializationSchema {
#Override
public boolean isEndOfStream(Object dcEvents) {
return false;
}
#Override
public Object deserialize(ConsumerRecord consumerRecord) throws Exception {
return null;
}
#Override
public TypeInformation<DCEvents> getProducedType() {
return null;
}
You are using a Simple String here and in serialize byte to String can be done something like the below code.
public class MyDeserializationSchema implements KafkaDeserializationSchema<String> {
#Override
public boolean isEndOfStream(String nextElement) {
return false;
}
#Override
public String deserialize(ConsumerRecord<byte[], byte[]> record) throws Exception {
return new String(record.value(), "UTF-8");
}
#Override
public TypeInformation<String> getProducedType() {
return BasicTypeInfo.STRING_TYPE_INFO;
}
}
As of now, I'm able to connect to Cassandra via the following code:
import com.datastax.driver.core.Cluster;
import com.datastax.driver.core.Session;
public static Session connection() {
Cluster cluster = Cluster.builder()
.addContactPoints("IP1", "IP2")
.withCredentials("user", "password")
.withSSL()
.build();
Session session = null;
try {
session = cluster.connect("database_name");
session.execute("CQL Statement");
} finally {
IOUtils.closeQuietly(session);
IOUtils.closeQuietly(cluster);
}
return session;
}
The problem is that I need to write to Cassandra in a Spring Batch project. Most of the starter kits seem to use a JdbcBatchItemWriter to write to a mySQL database from a chunk. Is this possible? It seems that a JdbcBatchItemWriter cannot connect to a Cassandra database.
The current itemwriter code is below:
#Bean
public JdbcBatchItemWriter<Person> writer() {
JdbcBatchItemWriter<Person> writer = new JdbcBatchItemWriter<Person>();
writer.setItemSqlParameterSourceProvider(new
BeanPropertyItemSqlParameterSourceProvider<Person>());
writer.setSql("INSERT INTO people (first_name, last_name) VALUES
(:firstName, :lastName)");
writer.setDataSource(dataSource);
return writer;
}
Spring Data Cassandra provides repository abstractions for Cassandra that you should be able to use in conjunction with the RepositoryItemWriter to write to Cassandra from Spring Batch.
It is possible to extend Spring Batch to support Cassandra by customising ItemReader and ItemWriter.
ItemWriter example:
public class CassandraBatchItemWriter<Company> implements ItemWriter<Company>, InitializingBean {
protected static final Log logger = LogFactory.getLog(CassandraBatchItemWriter.class);
private final Class<Company> aClass;
#Autowired
private CassandraTemplate cassandraTemplate;
#Override
public void afterPropertiesSet() throws Exception { }
public CassandraBatchItemWriter(final Class<Company> aClass) {
this.aClass = aClass;
}
#Override
public void write(final List<? extends Company> items) throws Exception {
logger.debug("Write operations is performing, the size is {}" + items.size());
if (!items.isEmpty()) {
logger.info("Deleting in a batch performing...");
cassandraTemplate.deleteAll(aClass);
logger.info("Inserting in a batch performing...");
cassandraTemplate.insert(items);
}
logger.debug("Items is null...");
}
}
Then you can inject it as a #Bean through #Configuration
#Bean
public ItemWriter<Company> writer(final DataSource dataSource) {
final CassandraBatchItemWriter<Company> writer = new CassandraBatchItemWriter<Company>(Company.class);
return writer;
}
Full source code can be found in Github repo: Spring-Batch-with-Cassandra
I'm using Spring Kafka integration and I've my own value generic serializer/deserializer as shown below
Serializer:
public class KafkaSerializer<T> implements Serializer<T> {
private ObjectMapper mapper;
#Override
public void close() {
}
#Override
public void configure(final Map<String, ?> settings, final boolean isKey) {
mapper = new ObjectMapper();
}
#Override
public byte[] serialize(final String topic, final T object) {
try {
return mapper.writeValueAsBytes(object);
} catch (final JsonProcessingException e) {
throw new IllegalArgumentException(e);
}
}
}
Deserializer:
public class KafkaDeserializer<T> implements Deserializer<T> {
private ObjectMapper mapper;
#Override
public void close() {
}
#Override
public void configure(final Map<String, ?> settings, final boolean isKey) {
mapper = new ObjectMapper();
}
#Override
public T deserialize(final String topic, final byte[] bytes) {
try {
return mapper.readValue(bytes, new TypeReference<T>() {
});
} catch (final IOException e) {
throw new IllegalArgumentException(e);
}
}
}
The serializer is working perfectly but when it comes to deserialization of values while consuming message I get a LinkedHashMap instead of desired object, please enlighten me where I'm mistaking, thanks in advance.
Some situations need be confirmed:
your Serializer is works
the Deserializer is just works but it returned a LinkedHashMap instead of a object that you expected, right? and you can't convert that LinkedHashMap to your object.
I find the question transfers to How to Convert/Cast a LinkedHashMap to a Object, and you used ObjectMapper. If all situations can be confirmed, I found here a good post may be answer your question Casting LinkedHashMap to Complex Object
mapper.convertValue(desiredObject, new TypeReference<type-of-desiredObject>() { })
ObjectMapper's API at [here](https://fasterxml.github.io/jackson-databind/javadoc/2.3.0/com/fasterxml/jackson/databind/ObjectMapper.html#convertValue(java.lang.Object, com.fasterxml.jackson.core.type.TypeReference))
And I hopes I don't missing your intention, and you can complement necessary situations, so someone or me can improve this answer.