I'm trying to pull object from MongoDb and ADD it to my current payload and save it in another database:
#Override
public void configure() throws Exception
{
from(kafkaEndpoint)
.convertBodyTo(DBObject.class)
.enrich("mongodb:mongoDb?database=myDbName1&collection=UserColl&operation=findOneByQuery",
(original, external) -> {
DBObject originalBody = original.getIn().getBody(DBObject.class);
DBObject externalBody = external.getIn().getBody(DBObject.class);
Map<String, DBObject> map = new HashMap<String, DBObject>();
map.put("original", originalBody);
map.put("external", externalBody);
original.getIn().setBody(map);
return original;
})
.to("mongodb:mongoDb?database=myDbName2&collection=UserColl&operation=insert");
}
The problem that enrich fetch the query from the In.body that holds my original object...
So how can I pass query ({"entity.id": ""}) to enrich(mongoldb:...) and preserve original object to merge it with results?
Thanks.
#Override
public void configure() throws Exception
{
from(kafkaEndpoint)
.convertBodyTo(DBObject.class)
.enrich("direct:findOneByQuery", // <-------
(original, external) -> {
DBObject originalBody = original.getIn().getBody(DBObject.class);
DBObject externalBody = external.getIn().getBody(DBObject.class);
Map<String, DBObject> map = new HashMap<String, DBObject>();
map.put("original", originalBody);
map.put("external", externalBody);
original.getIn().setBody(map);
return original;
})
.to("mongodb:mongoDb?database=myDbName2&collection=UserColl&operation=insert");
}
from("direct:findOneByQuery")
.process(new Processor()
{
#Override
public void process(Exchange exchange) throws Exception
{
DBObject body = exchange.getIn().getBody(DBObject.class);
DBObject query = BasicDBObjectBuilder.start()
.append("entity._id", body.get("_id"))
.get();
exchange.getIn().setBody(query);
}
})
.to("mongodb:mongoDb?database=myDbName1&collection=UserColl&operation=findOneByQuery");
//
Related
I am trying to find a way to log the offset when an exception occurs.
Here is what I am trying to achieve:
void createTopology(StreamsBuilder builder) {
builder.stream(topic, Consumed.with(Serdes.String(), new JsonSerde()))
.filter(...)
.mapValues(value -> {
Map<String, Object> output;
try {
output = decode(value.get("data"));
} catch (DecodingException e) {
LOGGER.error(e.getMessage());
// TODO: LOG OFFSET FOR FAILED DECODE HERE
return new ArrayList<>();
}
...
return output;
})
.filter((k, v) -> !(v instanceof List && ((List<?>) v).isEmpty()))
.to(sink_topic);
}
I found this: https://docs.confluent.io/platform/current/streams/developer-guide/dsl-api.html#streams-developer-guide-dsl-transformations-stateful
and it is in my understanding that I need to use the Processor API but still haven't found a solution for my issue.
A ValueTransfomer can also access the offset via the ProcessorContext passed via init, and I believe it's much easier.
Here is the solution, as suggested by IUSR: https://stackoverflow.com/a/73465691/14945779 (thank you):
static class InjectOffsetTransformer implements ValueTransformer<JsonObject, JsonObject> {
private ProcessorContext context;
#Override
public void init(ProcessorContext context) {
this.context = context;
}
#Override
public JsonObject transform(JsonObject value) {
value.addProperty("offset", context.offset());
return value;
}
#Override
public void close() {
}
}
void createTopology(StreamsBuilder builder) {
builder.stream(topic, Consumed.with(Serdes.String(), new JsonSerde()))
.filter(...)
.transformValues(InjectOffsetTransformer::new)
.mapValues(value -> {
Map<String, Object> output;
try {
output = decode(value.get("data"));
} catch (DecodingException e) {
LOGGER.warn(String.format("Error reading from topic %s. Last read offset %s:", topic, lastReadOffset), e);
return new ArrayList<>();
}
lastReadOffset = value.get("offset").getAsLong();
return output;
})
.filter((k, v) -> !(v instanceof List && ((List<?>) v).isEmpty()))
.to(sink_topic);
}
I have the code like below
#Bean
public JdbcCursorItemReader<Map<String, Object>> itemReader() {
return new JdbcCursorItemReader<Map<String, Object>>() {
private JobParameters jobParameter;
String sql = "select EMPLOYEE_ID as empId, EMPLOYEE_NAME as empName EMPLOYEE_AGE as age from EMPLOYEE EMPLOYEE_DEPT =:empDept and EMPLOYEE_SAL > :empSal";
Map<String, Object> namedParameters = null;
#PostConstruct
public void initialize() throws Exception
{
setDataSource(dataSource);
setSql("select 1 from dual");
setRowMapper(new ColumnMapRowMapper());
}
#BeforeStep
public void retrieveExecutionContext(StepExecution stepExecution)
{
jobParameter = stepExecution.getJobParameters();
namedParameters = new HashMap<String, Object>() {
{
put("bstd", jobParameter.getString("empDept"));
put("bwtn", jobParameter.getString("empSal"));
}
};
jobParameter.getParameters().forEach((k, v) -> System.out.println("key =" + k + ", Value:" + v));
}
#Override
public void afterPropertiesSet() throws Exception {
setSql(NamedParameterUtils.substituteNamedParameters(sql, new MapSqlParameterSource(namedParameters)));
setPreparedStatementSetter(new ListPreparedStatementSetter(
Arrays.asList(NamedParameterUtils.buildValueArray(sql, namedParameters))));
setRowMapper(new ColumnMapRowMapper());
setDataSource(dataSource);
super.afterPropertiesSet();
}
};
}
Tried using calling afterPropertiesSet, but still seeing below exception
Caused by: org.springframework.dao.InvalidDataAccessApiUsageException: No value supplied for the SQL parameter 'empDept': No value registered for key 'empDept'
at org.springframework.jdbc.core.namedparam.NamedParameterUtils.buildValueArray(NamedParameterUtils.java:361) ~[spring-jdbc-5.3.22.jar:5.3.22]
at org.springframework.jdbc.core.namedparam.NamedParameterUtils.buildValueArray(NamedParameterUtils.java:485) ~[spring-jdbc-5.3.22.jar:5.3.22]
Requirement is dynamic query, so don't have control of the Select query and the where conditions.
Thanks in advance,
You can use a SpEL expression to inject and use job parameters in your item reader bean definition as follows:
#Bean
#StepScope
public JdbcCursorItemReader<Map<String, Object>> itemReader(#Value("#{jobParameters['empDept']}") String empDept, #Value("#{jobParameters['empSal']}") String empSal) {
JdbcCursorItemReader<Map<String, Object>> itemReader = new JdbcCursorItemReader<>();
// use parameters 'empDept' and 'empSal' in your sql query as needed
return itemReader;
}
Note that the item reader should be step-scoped for that to work. For more details, please refer to the documentation: Late Binding of Job and Step Attributes.
I am trying to build a custom state store which stores key to map of values.
Stream & Store configuration
final Serde<HashMap<String, ?>> userSessionsSerde = Serdes.serdeFrom(new HashMapSerializer(), new HashMapDeserializer());
StoreBuilder sessionStoreBuilder = Stores.keyValueStoreBuilder(Stores.persistentKeyValueStore(storeName),
Serdes.String(),
userSessionsSerde);
builder.addStateStore(sessionStoreBuilder);
builder.stream("connection-events", Consumed.with(Serdes.String(), wsSerde))
.transform(wsEventTransformerSupplier, storeName)
.to("status-changes", Produced.with(Serdes.String(), Serdes.String()));
KafkaStreams streams = new KafkaStreams(builder.build(), properties);
streams.start();
Transformer
public class WSEventProcessor implements Transformer<String, ConnectionEvent, KeyValue<String, String>> {
private String storeName = "user-sessions";
private KeyValueStore<String, Map<String, ConnectionEvent>> stateStore;
final Serde<HashMap<String, ?>> userSessionsSerde = Serdes.serdeFrom(new HashMapSerializer(), new HashMapDeserializer());
#SuppressWarnings("unchecked")
#Override
public void init(ProcessorContext context) {
this.context = context;
stateStore = (KeyValueStore<String, Map<String, ConnectionEvent>>) context.getStateStore(storeName);
}
#Override
public void close() {
}
#Override
public KeyValue<String, String> transform(String key, ConnectionEvent value) {
boolean sendUpdate = false;
//Send null if there are no updates to be sent to downstream processors
if(value.getState() == WebSocketConnection.CONNECTED) {
if(stateStore.get(key) == null) {
stateStore.put(key, new HashMap<>());
sendUpdate = true;
}
stateStore.get(key).put(value.getSessionId(), value);
return sendUpdate ? KeyValue.pair(key, "Online") : null;
}
else {
stateStore.get(key).remove(value.getSessionId());
int size = stateStore.get(key).size();
return stateStore.get(key).isEmpty() ? KeyValue.pair(key, "Offline") : null;
}
}
}
The state store always has 0 size map for each key irrespective of connected and disconnected events. Am I doing something wrong?
value object that you stored into stateStore.put(key, value) and stateStore.get(key) are different objects (as it serialized and then deserialized).
Your issue is related to modification of object returned from state store:
stateStore.get(key).put(value.getSessionId(), value) and stateStore.get(key).remove(value.getSessionId()). when you update object stateStore.get(key), it's actually not persisted to state store, only changes that object.
So, to fix your issue, calculate required value (in your case HashMap), and only after that apply stateStore.put(key, calculated_value). If you need to remove key-value from state store, use stateStore.put(key, null). Your transform method should look approximately like:
public KeyValue<String, String> transform(String key, ConnectionEvent value) {
Map<String, Object> valueFromStateStore = stateStore.get(key);
Map<String, Object> valueToUpdate = ofNullable(valueFromStateStore).orElseGet(Collections::emptyMap);
KeyValue<String, String> resultKeyValue = null;
//Send null if there are no updates to be sent to downstream processors
if(value.getState() == WebSocketConnection.CONNECTED) {
if(valueToUpdate.isEmpty()) {
resultKeyValue = KeyValue.pair(key, "Online");
}
valueToUpdate.put(value.getSessionId(), value);
}
else {
valueToUpdate.remove(value.getSessionId());
if (valueToUpdate.isEmpty()) {
resultKeyValue = KeyValue.pair(key, "Offline");
}
}
stateStore.put(key, valueToUpdate);
return resultKeyValue;
}
I have a data coming on kafka topic as (key:id, {id:1, body:...})
means key for the message is same as id. however there can be multiple messages with the same id but different body.
so I am getting the kstream <String, String>
Now I want to get all the messages having same id (key) and club all the values as a list and return as
Kstream<String, List<String>>
Any sugessions?
//Create a Stream with a state store
StreamsBuilder builder = new StreamsBuilder();
StoreBuilder<KeyValueStore<String, List<String>>> logTracerStateStore = Stores.keyValueStoreBuilder(
Stores.persistentKeyValueStore(LOG_TRACE_STATE_STORE), Serdes.String(),
new ListSerde<String>(Serdes.String()));
//add this to stream builder
builder.addStateStore(logTracerStateStore);
KStream<String, String> kafkaStream = builder.stream(TOPIC);
splitProcessor(kafkaStream);
logger.info("creating stream for topic {} ..", TOPIC);
final Topology topology = builder.build();
return new KafkaStreams(topology, streamConfiguration(bootstrapServers));
// Stream List Serde
public class ListSerde<T> implements Serde<List<T>> {
private final Serde<List<T>> inner;
public ListSerde( final Serde<T> avroSerde) {
inner = Serdes.serdeFrom(new ListSerializer<>( avroSerde.serializer()),
new ListDeserializer<>( avroSerde.deserializer()));
}
#Override
public Serializer<List<T>> serializer() {
return inner.serializer();
}
#Override
public Deserializer<List<T>> deserializer() {
return inner.deserializer();
}
#Override
public void configure(final Map<String, ?> configs, final boolean isKey) {
inner.serializer().configure(configs, isKey);
inner.deserializer().configure(configs, isKey);
}
#Override
public void close() {
inner.serializer().close();
inner.deserializer().close();
}
}
// Serializer & deserializers
public class ListSerializer<T> implements Serializer<List<T>> {
// private final Comparator<T> comparator;
private final Serializer<T> valueSerializer;
public ListSerializer( final Serializer<T> valueSerializer) {
// this.comparator = comparator;
this.valueSerializer = valueSerializer;
}
#Override
public void configure(final Map<String, ?> configs, final boolean isKey) {
// do nothing
}
#Override
public byte[] serialize(final String topic, final List<T> list) {
final int size = list.size();
final ByteArrayOutputStream baos = new ByteArrayOutputStream();
final DataOutputStream out = new DataOutputStream(baos);
final Iterator<T> iterator = list.iterator();
try {
out.writeInt(size);
while (iterator.hasNext()) {
final byte[] bytes = valueSerializer.serialize(topic, iterator.next());
out.writeInt(bytes.length);
out.write(bytes);
}
out.close();
} catch (final IOException e) {
throw new RuntimeException("unable to serialize List", e);
}
return baos.toByteArray();
}
#Override
public void close() {
}
}
//------------
public class ListDeserializer<T> implements Deserializer<List<T>> {
// private final Comparator<T> comparator;
private final Deserializer<T> valueDeserializer;
public ListDeserializer(final Deserializer<T> valueDeserializer) {
// this.comparator = comparator;
this.valueDeserializer = valueDeserializer;
}
#Override
public void configure(final Map<String, ?> configs, final boolean isKey) {
// do nothing
}
#Override
public List<T> deserialize(final String s, final byte[] bytes) {
if (bytes == null || bytes.length == 0) {
return null;
}
final List<T> list = new ArrayList<>();
final DataInputStream dataInputStream = new DataInputStream(new ByteArrayInputStream(bytes));
try {
final int records = dataInputStream.readInt();
for (int i = 0; i < records; i++) {
final byte[] valueBytes = new byte[dataInputStream.readInt()];
dataInputStream.read(valueBytes);
list.add(valueDeserializer.deserialize(s, valueBytes));
}
// dataInputStream.close();
} catch (final IOException e) {
throw new RuntimeException("Unable to deserialize PriorityQueue", e);
}finally {
try {
dataInputStream.close();
} catch (Exception e2) {
// TODO: handle exception
}
}
return list;
}
#Override
public void close() {
}
}
/// Now create Stream Processors
public class LogTraceStreamStateProcessor implements Processor<String, String>{
private static final Logger logger = Logger.getLogger(LogTraceStreamStateProcessor.class);
IStore stateStore;
/**
* Initialize the transformer.
*/
#Override
public void init(ProcessorContext context) {
logger.info("initializing processor and looking for monitoring store");
stateStore = MonitoringStateStoreFactory.getInstance().getStore();
logger.debug("found the monitoring store - {} ", stateStore);
stateStore.initLogTraceStoreProcess(context);
logger.debug("initalizing monitoring store.");
}
#Override
public void process(String key, String value) {
logger.debug("Storing the value for logtrace storage - {} ", value);
stateStore.storeLogTrace(value);
logger.debug("finished Storing the value for logtrace storage - {} ", value);
}
#Override
public void close() {
// TODO Auto-generated method stub
}
}
// access the key value state store like below
KeyValueStore<String, List<String>> stateStore = (KeyValueStore<String, List<String>>) traceStreamContext.getStateStore(EXEID_REQ_REL_STORE);
//Now add a list to new key for a new message and if the key exists then add a new message in the list
public void storeTraceData(String traceData) {
try {
TraceEvent tracer = new TraceEvent();
logger.debug("Received the Trace value - {}", traceData);
tracer = mapper.readValue(traceData, TraceEvent.class);
logger.debug("trace unmarshelling has been completed successfully !!!");
String key = tracer.getExecutionId();
List<String> listEvents = stateStore.get(key);
if (listEvents != null && !listEvents.isEmpty()) {
logger.debug("event is already in store so storing in the list for execution id - {}", key);
listEvents.add(requestId);
stateStore.put(key, listEvents);
} else {
logger.debug(
"event is not present in the store so creating a new list and adding into store for execution id - {}",
key);
List<String> list = new ArrayList<>();
list.add(requestId);
stateStore.put(key, list);
}
} catch (Throwable e) {
logger.error("exception while processing the trace event .. ", e);
} finally {
try {
traceStreamContext.commit();
} catch (Exception e2) {
e2.printStackTrace();
}
}
}
/// now this is how you can access the message from state store
public ReadOnlyKeyValueStore<String, List<String>> tracerStore() {
return waitUntilStoreIsQueryable(KEY_NAME);
}
I am using spring data mongo-db 1.4.1.RELEASE.
My entity 'Event' has a getter method which is calculated based on other properties:
public int getStatus() {
return (getMainEventId() == null) ? (elapseTimeInMin() < MINIMUM_TIME ? CANDIDATE :
VALID) : POINTER;
}
I wanted the property 'status' to be persisted only through the getter ,so I wrote converters:
#WritingConverter
public class EventWriteConverter implements Converter<Event ,BasicDBObject > {
static final Logger logger = LoggerFactory.getLogger(EventWriteConverter.class.getCanonicalName());
public BasicDBObject convert(Event event) {
logger.info("converting " +event );
if (event.getMainEventId() != null)
return new BasicDBObject("mainEventId", event.getMainEventId() );
BasicDBObject doc = new BasicDBObject("status",event.getStatus()).
append("updated_date",new Date()).
append("start",event.getS0()).
append("end",event.getS1()).
append("location",event.getLocation()).
;
BasicDBList list = new BasicDBList();
doc.append("access_points",event.getHotPoints());
return doc;
}
#ReadingConverter
public class EventReadConverter implements Converter<BasicDBObject, Event> {
#Inject
HotPointRepositry hotRepositry;
static final Logger logger = LoggerFactory.getLogger(EventReadConverter.class.getCanonicalName());
public Event convert(BasicDBObject doc) {
logger.info(" converting ");
Event event = new Event();
event.setId(doc.getObjectId("_id"));
event.setS0(doc.getDate("start"));
event.setS1(doc.getDate("end"));
BasicDBList dblist = (BasicDBList) doc.get("hot_points");
if (dblist != null) {
for (Object obj : dblist) {
ObjectId hotspotId = ((BasicDBObject) obj).getObjectId("_id");
event.addHot(hotRepositry.findOne(hotId));
}
}
dblist = (BasicDBList) doc.get("devices");
if (dblist != null) {
for (Object obj : dblist)
event.addDevice(obj.toString());
}
event.setMainEventId(doc.getObjectId("mainEventId"));
return event;
}
}
My test mongo configuration is
#Profile("test")
#Configuration
#EnableMongoRepositories(basePackages = "com.echo.spring.data.mongo")
#ComponentScan(basePackages = "com.echo.spring.data.mongo" )
public class MongoDbTestConfig extends AbstractMongoConfiguration {
static final Logger logger = LoggerFactory.getLogger(MongoDbTestConfig.class.getCanonicalName());
#Override
protected String getDatabaseName() {
return "echo";
}
#Override
public Mongo mongo() {
return new Fongo("echo-test").getMongo();
}
#Override
protected String getMappingBasePackage() {
return "com.echo.spring.data.mongo";
}
#Bean
#Override
public CustomConversions customConversions() {
logger.info("loading custom converters");
List<Converter<?, ?>> converterList = new ArrayList<Converter<?, ?>>();
converterList.add(new EventReadConverter());
converterList.add(new EventWriteConverter());
CustomConversions cus = new CustomConversions(converterList);
return new CustomConversions(converterList);
}
}
And my test (using fongo) is
ActiveProfiles("test")
#RunWith(SpringJUnit4ClassRunner.class)
#ContextConfiguration(classes = MongoDbTestConfig.class )
public class SampleMongoApplicationTests {
#Test
#ShouldMatchDataSet(location = "/MongoJsonData/events.json")
public void shouldSaveEvent() throws IOException {
URL url = Resources.getResource("MongoJsonData/events.json");
List<String> lines = Resources.readLines(url,Charsets.UTF_8);
for (String line : lines) {
Event event = objectMapper.readValue(line.getBytes(),Event.class);
eventRepository.save(event);
}
}
I can see the converters are loaded when the configuration customConversions() is called
I added logging and breakpoints in the convert methods but they do not seems to be
called when I run or debug, though they are loaded .
What am I doing wrong ?
I had a similar situation, I followed Spring -Mongodb storing/retrieving enums as int not string
and I need both the converter AND converterFactory wired to get it working.