I want something like redis set and ttl, such as
public void put(K key, V value, int timeToLive, TimeUnit timeUnit);
public long ttl(K key);
and one more operation that redis doesn't provide
public void putWithIdle(K key, V value, int timeToIdle, TimeUnit timeUnit);
There is no direct API for this, but after puting the value, you can change the expiry time as described here https://cache2k.org/docs/1.0/user-guide.html#entry-processor
cache.invoke("key",
e -> e.setValue("value").setExpiry(System.currentTimeMillis() +
TimeUnit.MINUTES.toMillis(120)));
Consider creating a feature request at https://github.com/cache2k/cache2k/issues
Related
I am using following program to produce records in kafka:
import java.io.IOException;
import java.security.SecureRandom;
public class SensorStatusProducer {
private final static String TOPIC = "SENSOR_STATUS_DETAILS";
private final static String PRODUCER_URI = "http://xxx.xxx.xxx.xxx:8082/topics/" + TOPIC;
private final static SecureRandom randomNumber = new SecureRandom();
private final static SensorDetails sensorDetails = new SensorDetails();
public static void main(String[] args) {
int[] sensorid = sensorDetails.getSensorid(); //this will return [1001,1002,1003,1004,1005]
try {
HttpRestProxyUtil rest = new HttpRestProxyUtil(); //this is defined in another class
for (int sid : sensorid) {
rest.produceRecords(PRODUCER_URI, String.format("{\"records\":[{\"key\": %d," +
"\"value\":{" +
"\"sensorid\":%d," +
"\"status\":%s," +
"\"lastconnectedtime\":%s}}]}", sid, sid, "\"CONNECTED\"", String.format("\"%s\"", sensorDetails.currentTimestamp()))); //currentTimestamp() function in defined in another class
}
} catch (InterruptedException | IOException me) {
me.printStackTrace();
}
}
}
The key has format specifier as %d but the record produced has key of STRING type.
This is evident by following:
When trying to make table:
CREATE TABLE STATUS_IB_TABLE (ROWKEY INT KEY,
sensorid INTEGER,
status VARCHAR,
lastconnectedtime STRING)
WITH (TIMESTAMP='lastconnectedtime', TIMESTAMP_FORMAT='yyyy-MM-dd HH:mm:ss', KAFKA_TOPIC='SENSOR_STATUS_DETAILS', VALUE_FORMAT='JSON', KEY='sensorid');
The KEY is serialized as STRING as pointed out by #Andrew Coates
I don't know how's that possible.
can someone please clarify this for me, what am I doing wrong?
PS:
=> this is a follow up question for my earlier question ksqlDB not taking rowkey properly
=> Confluent Platform version: 5.5
=> This is the main class of the program.
The REST Proxy supports various content types, but not including the primitive type to write a serialized 32-bit integer.
Your code is thus producing data to the topic with a string key. For an example of how to produce an INT see the example here which uses kafkacat.
Since you're using Java, you could use the native Java Producer API to control exactly how the data is produced to Kafka (which is also more performant and flexible than the REST API).
I'm trying to write a simple Kafka Streams application (targeting Kafka 2.2/Confluent 5.2) to transform an input topic with at-least-once semantics into an exactly-once output stream. I'd like to encode the following logic:
For each message with a given key:
Read a message timestamp from a string field in the message value
Retrieve the greatest timestamp we've previously seen for this key from a local state store
If the message timestamp is less than or equal to the timestamp in the state store, don't emit anything
If the timestamp is greater than the timestamp in the state store, or the key doesn't exist in the state store, emit the message and update the state store with the message's key/timestamp
(This is guaranteed to provide correct results based on ordering guarantees that we get from the upstream system; I'm not trying to do anything magical here.)
At first I thought I could do this with the Kafka Streams flatMapValues operator, which lets you map each input message to zero or more output messages with the same key. However, that documentation explicitly warns:
This is a stateless record-by-record operation (cf. transformValues(ValueTransformerSupplier, String...) for stateful value transformation).
That sounds promising, but the transformValues documentation doesn't make it clear how to emit zero or one output messages per input message. Unless that's what the // or null aside in the example is trying to say?
flatTransform also looked somewhat promising, but I don't need to manipulate the key, and if possible I'd like to avoid repartitioning.
Anyone know how to properly perform this kind of filtering?
you could use Transformer for implementing stateful operations as you described above. In order to not propagate a message downstream, you need to return null from transform method, this mentioned in Transformer java doc. And you could manage propagation via processorContext.forward(key, value). Simplified example provided below
kStream.transform(() -> new DemoTransformer(stateStoreName), stateStoreName)
public class DemoTransformer implements Transformer<String, String, KeyValue<String, String>> {
private ProcessorContext processorContext;
private String stateStoreName;
private KeyValueStore<String, String> keyValueStore;
public DemoTransformer(String stateStoreName) {
this.stateStoreName = stateStoreName;
}
#Override
public void init(ProcessorContext processorContext) {
this.processorContext = processorContext;
this.keyValueStore = (KeyValueStore) processorContext.getStateStore(stateStoreName);
}
#Override
public KeyValue<String, String> transform(String key, String value) {
String existingValue = keyValueStore.get(key);
if (/* your condition */) {
processorContext.forward(key, value);
keyValueStore.put(key, value);
}
return null;
}
#Override
public void close() {
}
}
I have a column defined like this:
expiry timestamp(0) without time zone not null
With Postgres, I can issue SQL like:
insert into my_table(expiry) values ('infinity')
I've been digging through the JOOQ doco, but couldn't find any example of dealing with this.
Can I do that with JOOQ - and what would it look like?
Additionally, is it possible using an UpdatableRecord? Is there some kind of infinity "flag" instance of Timestamp I can use?
Ok, found a way to do it directly.
MyRecord r = db.insertInto(
MY_RECORD,
MY_RECORD.ID,
MY_RECORD.CREATED,
MY_RECORD.EXPIRY
).values(
val(id),
currentTimestamp(),
val("infinity").cast(Timestamp.class)
).returning().fetchOne();
But that feels more like a workaround than the right way to do it. Casting a string to a timestamp seems a little bit round-about to me, so I wrote a CustomField to make using it and querying easier:
public class TimestampLiteral extends CustomField<Timestamp> {
public static final TimestampLiteral INFINITY =
new TimestampLiteral("'infinity'");
public static final TimestampLiteral NEGATIVE_INFINITY =
new TimestampLiteral("'-infinity'");
public static final TimestampLiteral TODAY =
new TimestampLiteral("'today'");
private String literalValue;
public TimestampLiteral(String literalValue){
super("timestamp_literal", SQLDataType.TIMESTAMP);
this.literalValue = literalValue;
}
#Override
public void accept(Context<?> context){
context.visit(delegate(context.configuration()));
}
private QueryPart delegate(Configuration configuration){
switch( configuration.dialect().family() ){
case POSTGRES:
return DSL.field(literalValue);
default:
throw new UnsupportedOperationException(
"Dialect not supported because I don't know how/if this works in other databases.");
}
}
}
Then the query is:
MyRecord r = db.insertInto(
MY_RECORD,
MY_RECORD.ID,
MY_RECORD.CREATED,
MY_RECORD.EXPIRY
).values(
val(id),
TimestampLiteral.TODAY,
TimestampLiteral.INFINITY
).returning().fetchOne();
Don't know if this is necessarily the "right" way to do this, but it seems to work for the moment.
Still interested to hear if there's a way to do this with an UpdatableRecord.
I create a java.sql.Timestamp passing org.postgresql.PGStatement.DATE_POSITIVE_INFINITY to its constructor.
create.insertInto(
MY_RECORD,
MY_RECORD.ID,
MY_RECORD.CREATED,
MY_RECORD.EXPIRY
).values(
1,
new Timestamp(System.currentTimeMillis()),
new Timestamp(PGStatement.DATE_POSITIVE_INFINITY)
).execute();
I'm using Kafka Streams version 0.10.0.1, and trying to find the min value in a stream.
The incoming messages come from a topic called kafka-streams-topic and have a key and the value is a JSON payload that looks like this:
{"value":2334}
This is a simple payload but I want to find the min value of this JSON.
The outgoing message is just a number:
2334
and the key is also part of the message.
So if the incoming topic got:
key=1, value={"value":1000}
outgoing topic, named min-topic, would get
key=1,value=1000
another message comes through:
key=1, value={"value":100}
because this is the same key I would like to now produce a message with key=1 value=100 since this is now smaller than the first message
Now lets say we got:
key=2 value=99
A new message would be produced where:
key=2 and value=99 but the key=1 and associated value shouldn't change.
Additionally if we got the message:
key=1 value=2000
No message should be produced since this message is larger than the current value of 100
This works but I'm wondering if this adheres to the intent of the API:
public class MinProcessor implements Processor<String,String> {
private ProcessorContext context;
private KeyValueStore<String, Long> kvStore;
private Gson gson = new Gson();
#Override
public void init(ProcessorContext context) {
this.context = context;
this.context.schedule(1000);
kvStore = (KeyValueStore) context.getStateStore("Counts");
}
#Override
public void process(String key, String value) {
Long incomingPotentialMin = ((Double)gson.fromJson(value, Map.class).get("value")).longValue();
Long minForKey = kvStore.get(key);
System.out.printf("key: %s incomingPotentialMin: %s minForKey: %s \n", key, incomingPotentialMin, minForKey);
if (minForKey == null || incomingPotentialMin < minForKey) {
kvStore.put(key, incomingPotentialMin);
context.forward(key, incomingPotentialMin.toString());
context.commit();
}
}
#Override
public void punctuate(long timestamp) {}
#Override
public void close() {
kvStore.close();
}
}
Here is the code that actually runs the processor:
public class MinLauncher {
public static void main(String[] args) {
TopologyBuilder builder = new TopologyBuilder();
StateStoreSupplier countStore = Stores.create("Counts")
.withKeys(Serdes.String())
.withValues(Serdes.Long())
.persistent()
.build();
builder.addSource("source", "kafka-streams-topic")
.addProcessor("process", () -> new MinProcessor(), "source")
.addStateStore(countStore, "process")
.addSink("sink", "min-topic", "process");
KafkaStreams streams = new KafkaStreams(builder, KafkaStreamsProperties.properties("kafka-streams-min-poc"));
streams.cleanUp();
streams.start();
Runtime.getRuntime().addShutdownHook(new Thread(streams::close));
}
}
Not sure what your exact input data and result is (maybe you can update you question with this information: what are your input records? what is your output? What "EXTRA messages [] are produced [] that [you] don't expect"?).
However, a few general clarifications (can refine this answer later on if required).
You do your computation based in keys, so you should expect a result for each key (not sure if you have multiple different keys in your input).
You emit data in punctuate() which is called periodically (base in the internally tracked stream-time -- i.e., based on the timestamp values extracted from your input records via TimestampExtractor). Hence, you will write the current min value of each key written to the topic when punctuate() gets called and therefore, you can have multiple updates per key that are all appended to your result topic. (Topics are append only and if you write two messages with the same key, you see both -- there is no overwrite.)
I am new to MongoDB and JDO after mostly doing development with Hibernate in the past. I am trying to persist a simple object and leverage the generated "_id" from MongoDB as the primary key for the persisted object. Unfortunately, it looks like DataNucleus is generating an "IDENTITY" field as well as Mongo generating an "_id" field in the persisted document. So, every object is persisted with two unique identifiers. How can I enforce DataNucleus to simply use the generated Mongo ObjectId? My persistent class is below.
#PersistentCapable(identityType=IdentityType.DATASTORE)
public class HistoricalPrice {
private String ticker;
private Date day;
private double open;
private double close;
private double high;
private double low;
private long volume;
public HistoricalPrice(String ticker, Date day, double open, double close, double high, double low, long volume) {
super();
this.ticker = ticker;
this.day = day;
this.open = open;
this.close = close;
this.high = high;
this.low = low;
this.volume = volume;
}
Define the datastore identity "strategy" to be IDENTITY (as opposed to the default of NATIVE).
#DatastoreIdentity(strategy=IdGeneratorStrategy.IDENTITY)
i.e consistent with what would be needed on an RDBMS to use some inbuilt mechanism.