How to execute Vertx Future in sequential order - vert.x

In my sample project, I tried to do some initialization work when the application is started.
Java 16
Vertx 4.1.0
Check the complete project codes.
log.info("Data initialization is starting...");
var deleteComments = this.comments.deleteAll().onSuccess(event -> log.info("deleted comments: {}", event));
var deletePosts = this.posts.deleteAll().onSuccess(event -> log.info("deleted posts: {}", event));
var deleteUsers = this.authors.deleteAll().onSuccess(event -> log.info("deleted users: {}", event));
//log.info("deleted rows: authors: {}, comments: {}, posts: {}", authorsDel, commentsDel, postDel);
var insertData = this.authors.create("user", "user#example.com").onSuccess(
authorId -> {
IntStream.range(1, 5)
.forEach(
i -> {
this.posts.create("Dgs post #" + i, "test content of #" + i, PostStatus.DRAFT.name(), authorId).onSuccess(
postId -> {
IntStream.range(1, new Random().nextInt(5) + 1)
.forEach(c -> this.comments.create("comment #" + c, postId));
}
);
}
);
}
);
var printPosts = this.posts.findAll().onSuccess(p -> log.info("post: {}", p));
var printComments = this.comments.findAll().onSuccess(p -> log.info("comment: {}", p));
var printAuthors = this.authors.findAll().onSuccess(p -> log.info("author: {}", p));
deleteComments
.flatMap(integer -> deletePosts)
.flatMap(integer -> deleteUsers)
.flatMap(integer -> insertData)
.flatMap( uuid -> printPosts)
.flatMap(postEntities -> printComments)
.flatMap(commentEntities -> printAuthors)
.onSuccess(event -> log.info("done"));
log.info("done data initialization...");
But it does not work as expected.
There is no a then method like Reactor.
And the Future.result() will return an exception instead of a blocking call(how to make execution in a blocking way, I thought it is a blocking invocation).
I can not find an effective way to execute them in sequential order.
Update:
I changed the codes as suggested.
this.comments.deleteAll().onSuccess(event -> log.info("deleted comments: {}", event))
.flatMap(r -> this.posts.deleteAll().onSuccess(event -> log.info("deleted posts: {}", event)))
.flatMap(r -> this.authors.deleteAll().onSuccess(event -> log.info("deleted users: {}", event)))
.flatMap(r -> this.authors.create("user", "user#example.com")
.onSuccess(
authorId -> {
log.info("inserted user: {}", authorId);
IntStream.range(1, 5)
.forEach(
i -> {
this.posts.create("Dgs post #" + i, "test content of #" + i, PostStatus.DRAFT.name(), authorId).onSuccess(
postId -> {
log.info("inserted post: {}", postId);
IntStream.range(1, new Random().nextInt(5) + 1)
.forEach(c -> this.comments.create("comment #" + c, postId)
.onSuccess(id -> log.info("inserted comment: {}", id))
);
}
);
}
);
}
)
)
.flatMap(r -> this.posts.findAll().onSuccess(p -> log.info("saved posts: {}", p)))
.flatMap(r -> this.comments.findAll().onSuccess(p -> log.info("saved comments: {}", p)))
.flatMap(r -> this.authors.findAll().onSuccess(p -> log.info("saved authors: {}", p)))
.onSuccess(event -> log.info("done data initialization..."));
And got the following log in the console.
10:20:46.699 [vert.x-eventloop-thread-1] INFO com.example.demo.DataInitializer - Data initialization is starting...
10:20:47.768 [vert.x-eventloop-thread-1] INFO com.example.demo.MainVerticle - HTTP server started on port 8080
10:20:47.768 [vert.x-eventloop-thread-0] INFO i.v.c.i.l.c.VertxIsolatedDeployer - Succeeded in deploying verticle
10:20:47.918 [vert.x-eventloop-thread-1] INFO com.example.demo.DataInitializer - deleted comments: 10
10:20:47.942 [vert.x-eventloop-thread-1] INFO com.example.demo.DataInitializer - deleted posts: 4
10:20:47.960 [vert.x-eventloop-thread-1] INFO com.example.demo.DataInitializer - deleted users: 1
10:20:48.022 [vert.x-eventloop-thread-1] INFO com.example.demo.DataInitializer - inserted user: 260fee93-cf60-408d-a3e9-f7b08ed7545a
10:20:48.047 [vert.x-eventloop-thread-1] INFO com.example.demo.DataInitializer - saved posts: []
10:20:48.052 [vert.x-eventloop-thread-1] INFO com.example.demo.DataInitializer - inserted post: 57a936e2-9611-43b5-94e2-1b01069cd327
10:20:48.056 [vert.x-eventloop-thread-1] INFO com.example.demo.DataInitializer - saved comments: []
10:20:48.067 [vert.x-eventloop-thread-1] INFO com.example.demo.DataInitializer - saved authors: [AuthorEntity[id=260fee93-cf60-408d-a3e9-f7b08ed7545a, name=user, email=user#example.co
m, createdAt=null]]
10:20:48.092 [vert.x-eventloop-thread-1] INFO com.example.demo.DataInitializer - done data initialization...
10:20:48.112 [vert.x-eventloop-thread-1] INFO com.example.demo.DataInitializer - inserted post: 534b99ae-4847-4cea-84c1-88cea87f20b7
10:20:48.117 [vert.x-eventloop-thread-1] INFO com.example.demo.DataInitializer - inserted post: f7dca2eb-0954-4368-b9b6-68a00f6748f2
10:20:48.120 [vert.x-eventloop-thread-1] INFO com.example.demo.DataInitializer - inserted post: 02e8f84f-25dd-4d69-b7f1-8c125ce35bc1
10:20:48.131 [vert.x-eventloop-thread-1] INFO com.example.demo.DataInitializer - inserted comment: b9bf618a-f579-48f5-96cc-51f962ca041c
10:20:48.134 [vert.x-eventloop-thread-1] INFO com.example.demo.DataInitializer - inserted comment: 052cfde8-627d-4cb3-8812-5543673a20ea
10:20:48.143 [vert.x-eventloop-thread-1] INFO com.example.demo.DataInitializer - inserted comment: 69a5dc2b-30f8-4008-90a1-e8c724336dcd
10:20:48.149 [vert.x-eventloop-thread-1] INFO com.example.demo.DataInitializer - inserted comment: a19c2626-1546-4ba0-a663-823df08d836f
How to make sure all futures in the insert block are completed before entering the print result.

It's most likely because your deleteAll and findAll operations are not lazy. Because of this, they are running as soon as you call them. Instead of doing:
deleteComments
.flatMap(integer -> deletePosts)
...
You want to be doing this:
deleteComments
.flatMap(integer -> this.posts.deleteAll())
...
This will make sure that they will run sequentially. If you want something that's more similar to Reactor, you can take a look at the RxJava3 extensions or mutiny bindings for Vert.x.

Related

LibrdKafkaError: Broker: Unknown member after running around about 2 hours randomly

Right now, i want to implement node-rdkafka into our service, but i faced this error many times Broker: Unknown member.
The same issue on github was https://github.com/confluentinc/confluent-kafka-dotnet/issues/1464. they say our consumer using same group id to retry or delay. but i didn't find any retry and delay on my code.
or https://github.com/confluentinc/confluent-kafka-python/issues/1004, but i have recheck all consumer group id and it was unique.
The config of node-rdkafka producer as follows:
this.producer = new Producer({
"client.id": this.cliendID,
"metadata.broker.list": this.brokerList,
'compression.codec': "lz4",
'retry.backoff.ms': 200,
'socket.keepalive.enable': true,
'queue.buffering.max.messages': 100000,
'queue.buffering.max.ms': 1000,
'batch.num.messages': 1000000,
"transaction.timeout.ms": 2000,
"enable.idempotence": false,
"max.in.flight.requests.per.connection": 1,
"debug": this.debug,
'dr_cb': true,
"retries": 0,
"log_cb": (_: any) => console.log(`log_cb =>`, _),
"sasl.username": this.saslUsername,
"sasl.password": this.saslPassword,
"sasl.mechanism": this.saslMechanism,
"security.protocol": this.securityProtocol
}, {
"acks": -1
})
The config of node-rdkafka consumer as follows:
this.consumer = new KafkaConsumer({
'group.id': this.groupID,
'metadata.broker.list': this.brokerList,
"sasl.username": this.saslUsername,
"sasl.password": this.saslPassword,
"enable.auto.commit": false,
"auto.commit.interval.ms": 2000,
"session.timeout.ms": 45000,
"max.poll.interval.ms": 300000,
"heartbeat.interval.ms": 3000,
"api.version.request.timeout.ms": 10000,
"max.in.flight.requests.per.connection": 1,
"debug": this.debug,
"sasl.mechanism": this.saslMechanism,
"log.connection.close": true,
"log.queue": true,
"log_level": 7,
"log.thread.name": true,
"isolation.level": "read_committed",
"ssl.ca.location": "/etc/ssl/certs/",
"log_cb": (_: any) => console.log(`log_cb =>`, _),
"security.protocol": this.securityProtocol
}, {})
await new Promise(resolve => {
this.consumer?.connect()
this.consumer?.on('ready', () => {
try {
this.consumer?.subscribe(subscriptions)
this.consumer?.consume()
console.log('[SUCCESS] Subscribe Event => all event')
} catch (err) {
console.log('[FAILED] Subscribe => all event')
console.log(err)
}
resolve(this.consumer)
}).on('data', async (data) => {
this.topicFunctionMap[data.topic]({
partition: data.partition,
topic: data.topic,
message: {
key: data.key,
offset: data.offset.toString(),
size: data.size,
value: data.value,
timestamp: data.timestamp?.toString()
}
} as ISubsCallbackParam)
this.consumer?.commitSync({
topic: data.topic,
offset: data.offset,
partition: data.partition
})
})
})
Using those configuration, the consumer is able to receive event but its not last for long. around 2hours more it randomly gives those error.
I am not sure if it because manual commit or our function tooks long because i have tried both async and sync commit so the commitSync its not depend on our function.
Let says it because the our function tooks long, and it make our cosumer kicked from the group. maybe its the suspect after i found additional error Broker: Specified group generation id is not valid
source: https://github.com/confluentinc/confluent-kafka-dotnet/issues/1155
Its says i need to increase the session time out, then i tried to increase it to "session.timeout.ms": 300000 or 5min, and the heartbeat "heartbeat.interval.ms":3000, i found in github issue, that the heartbeat should less than = (timeout/3). so i think 3sec will fine.
Using "session.timeout.ms": 300000 and "heartbeat.interval.ms":3000
the consumer is able to consume and last for long but the problems is:
first time using those config, its fine running around 0-2sec to receive
after a while, its received, but tooks 1-10sec to receive the message
The detail errors:
received event => onCustomerServiceRegister
[COMMIT_ERR] LibrdKafkaError: Broker: Unknown member
at Function.createLibrdkafkaError [as create] (/src/app/node_modules/node-rdkafka/lib/error.js:454:10)
at KafkaConsumer.Client._errorWrap (/src/app/node_modules/node-rdkafka/lib/client.js:481:29)
at KafkaConsumer.commitSync (/src/app/node_modules/node-rdkafka/lib/kafka-consumer.js:560:8)
at KafkaRDConnect.<anonymous> (/src/app/dist/events/connectors/kafkaRD.js:240:110)
at step (/src/app/dist/events/connectors/kafkaRD.js:53:23)
at Object.next (/src/app/dist/events/connectors/kafkaRD.js:34:53)
at /src/app/dist/events/connectors/kafkaRD.js:28:71
at new Promise (<anonymous>)
at __awaiter (/src/app/dist/events/connectors/kafkaRD.js:24:12)
at KafkaConsumer.<anonymous> (/src/app/dist/events/connectors/kafkaRD.js:213:72)
at KafkaConsumer.emit (node:events:376:20)
at KafkaConsumer.EventEmitter.emit (node:domain:470:12)
at /src/app/node_modules/node-rdkafka/lib/kafka-consumer.js:488:12 {

Path variables in Apache Camel rest route

I would like to use path variables in my REST route URL in the following way:
rest().post("/companies/{companyId}/branches/branchId={branchId}");
The companyId can be properly retrieved as header, however, "branchId={branchId}" is treated as literal string. Therefore request to:
/companies/100/branches/branchId=200 - will return 404 not found.
but
/companies/100/branches/branchId={branchId} - will enter the route.
I would like to use branchId as header, in the same way as companyId, without having to change the structure of the URL.
Every path or query parameter can be received through the headers in Apache Camel. You can retrieve the headers with using the simple expression;
${header.your-header}
So you can retrieve the parameters as below;
parameter
simple expression
companyId
${header.companyId}
branchId
${header.branchId}
Here is a working Route example that I've made for you;
package com.bzdgn.camelso.route;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.model.rest.RestBindingMode;
public class RestEndpointRoute extends RouteBuilder {
#Override
public void configure() throws Exception {
rest()
.post("/companies/{companyId}/branches/{branchId}")
.consumes("text/plain")
.produces("text/plain")
.to("direct:handle-post");
from("direct:handle-post")
.id("post-route")
.log("Received Body: ${body}")
.log("Company Id: ${header.companyId}")
.log("Branch Id: ${header.branchId}")
.setBody(simple("CompanyId = ${header.companyId} \nBranchId = ${header.branchId}"))
.convertBodyTo(String.class);
restConfiguration()
.component("netty-http")
.host("localhost")
.port("8080")
.bindingMode(RestBindingMode.auto);
}
}
The logs will be as below if you send a HTTP Post Request to the designated URL;
2021-05-28 21:13:30 INFO HttpServerBootstrapFactory:53 - BootstrapFactory on port 8080 is using bootstrap configuration: [NettyServerBootstrapConfiguration{protocol='http', host='localhost', port=8080, broadcast=false, sendBufferSize=65536, receiveBufferSize=65536, receiveBufferSizePredictor=0, workerCount=0, bossCount=1, keepAlive=true, tcpNoDelay=true, reuseAddress=true, connectTimeout=10000, backlog=0, serverInitializerFactory=org.apache.camel.component.netty.http.HttpServerInitializerFactory#ae3540e, nettyServerBootstrapFactory=null, options=null, ssl=false, sslHandler=null, sslContextParameters='null', needClientAuth=false, enabledProtocols='TLSv1,TLSv1.1,TLSv1.2, keyStoreFile=null, trustStoreFile=null, keyStoreResource='null', trustStoreResource='null', keyStoreFormat='JKS', securityProvider='SunX509', passphrase='null', bossGroup=null, workerGroup=null, networkInterface='null', reconnect='true', reconnectInterval='10000'}]
2021-05-28 21:13:30 INFO NettyComponent:164 - Creating shared NettyConsumerExecutorGroup with 17 threads
2021-05-28 21:13:31 INFO SingleTCPNettyServerBootstrapFactory:182 - ServerBootstrap binding to localhost:8080
2021-05-28 21:13:33 INFO NettyConsumer:77 - Netty consumer bound to: localhost:8080
2021-05-28 21:13:33 INFO AbstractCamelContext:2983 - Routes startup summary (total:2 started:2)
2021-05-28 21:13:33 INFO AbstractCamelContext:2988 - Started post-route (direct://handle-post)
2021-05-28 21:13:33 INFO AbstractCamelContext:2988 - Started route1 (rest://post:/companies/%7BcompanyId%7D/branches/%7BbranchId%7D)
2021-05-28 21:13:33 INFO AbstractCamelContext:3000 - Apache Camel 3.10.0 (camel-1) started in 3s337ms (build:93ms init:274ms start:2s970ms)
2021-05-28 21:13:35 INFO post-route:166 - Received Body: Example text body here
2021-05-28 21:13:35 INFO post-route:166 - Company Id: my-custom-company
2021-05-28 21:13:35 INFO post-route:166 - Branch Id: my-custom-branch-id
And you can test it via postman like this. The request, and the headers listed in the below images, and do not forget to add "Content Type: text/plain", because that's defined in the route that it will consume and produce "text/plain".

PutIfExists fails if a record has been recently added

In ScalarDB, the library that add ACID functionality to Cassandra, I am getting the following error
2020-09-24 18:51:33,607 [WARN] from com.scalar.db.transaction.consensuscommit.CommitHandler in ScalaTest-run-running-AllRepositorySpecs - preparing records failed
com.scalar.db.exception.storage.NoMutationException: no mutation was applied.
I am running a test case in which I get a record to check that it doesn't exist, then I add the record, then I fetch it to see it was successfully added, then I update it and then I get it again to see that the value was updated.
"update an answer if the answer exists" in {
beforeEach()
embeddedCassandraManager.executeStatements(cqlStartupStatements)
val cassandraConnectionService = CassandraConnectionManagementService()
val (cassandraSession, cluster) = cassandraConnectionService.connectWithCassandra("cassandra://localhost:9042/codingjedi", "codingJediCluster")
//TODOM - pick the database and keyspace names from config file.
cassandraConnectionService.initKeySpace(cassandraSession.get, "codingjedi")
val transactionService = cassandraConnectionService.connectWithCassandraWithTransactionSupport("localhost", "9042", "codingJediCluster" /*,dbUsername,dbPassword*/)
val repository = new AnswersTransactionRepository("codingjedi", "answer_by_user_id_and_question_id")
val answerKey = AnswerKeys(repoTestEnv.answerTestEnv.answerOfAPracticeQuestion.answer_id.get,
repoTestEnv.answerTestEnv.answerOfAPracticeQuestion.question_id,
Some(repoTestEnv.answerTestEnv.answerOfAPracticeQuestion.answer_id.get))
logger.trace(s"checking if answer already exists")
val distributedTransactionBefore = transactionService.get.start()
val resultBefore = repository.get(distributedTransactionBefore, answerKey) //answer should not exist
distributedTransactionBefore.commit()
resultBefore.isLeft mustBe true
resultBefore.left.get.isInstanceOf[AnswerNotFoundException] mustBe true
logger.trace(s"no answer found. adding answer")
val distributedTransactionDuring = transactionService.get.start()
repository.add(distributedTransactionDuring, repoTestEnv.answerTestEnv.answerOfAPracticeQuestion)//add answer
distributedTransactionDuring.commit()
logger.trace(s"answer added")
val distributedTransactionAfter = transactionService.get.start()
val result = repository.get(distributedTransactionAfter, answerKey) //now answer should exist
distributedTransactionAfter.commit()
result mustBe (Right(repoTestEnv.answerTestEnv.answerOfAPracticeQuestion))
logger.trace(s"got answer from repo ${result}")
val updatedNotes = if(repoTestEnv.answerTestEnv.answerOfAPracticeQuestion.notes.isDefined)
Some(repoTestEnv.answerTestEnv.answerOfAPracticeQuestion.notes.get+"updated") else Some("updated notes")
val updatedAnswer = repoTestEnv.answerTestEnv.answerOfAPracticeQuestion.copy(notes=updatedNotes) //updated answer
logger.trace(s"old notes ${repoTestEnv.answerTestEnv.answerOfAPracticeQuestion.notes} vs new notes ${updatedNotes}")
logger.trace(s"updated answer ${updatedAnswer}")
val distributedTransactionForUpdate = transactionService.get.start()
val resultOfupdate = repository.update(distributedTransactionForUpdate,updatedAnswer) //update answer
distributedTransactionForUpdate.commit() //fails here
logger.trace(s"update done. getting answer again")
val distributedTransactionAfterUpdate = transactionService.get.start()
val resultAfterUpdate = repository.get(distributedTransactionAfterUpdate, answerKey)
distributedTransactionForUpdate.commit()
resultAfterUpdate mustBe (Right(updatedAnswer))
logger.trace(s"got result after update ${resultAfterUpdate}")
afterEach()
}
The update method calls add with putIfExists condition
def update(transaction:DistributedTransaction, answer:AnswerOfAPracticeQuestion) = {
logger.trace(s"updating answer value ${answer}")
//checktest-update an answer if the answer exists
add(transaction,answer, new PutIfExists)
}
def add(transaction:DistributedTransaction,answer:AnswerOfAPracticeQuestion,mutationCondition:MutationCondition = new PutIfNotExists()) = {
logger.trace(s"adding answer ${answer} with mutation state ${mutationCondition}")
val pAnswerKey = new Key(new TextValue("answered_by_user", answer.answeredBy.get.answerer_id.toString),
new TextValue("question_id",answer.question_id.toString))
//to check duplication, both partition and clustering keys need to be present
//val cAnswerKey = new Key(new TextValue("answer_id",answer.answer_id.toString))
//logger.trace(s"created keys. ${pAnswerKey}, ${cAnswerKey}")
val imageData = answer.image.map(imageList=>imageList).getOrElse(List())
logger.trace(s"will check in ${keyspaceName},${tablename}")
val putAnswer: Put = new Put(pAnswerKey/*,cAnswerKey*/)
.forNamespace(keyspaceName)
.forTable(tablename)
.withCondition(mutationCondition)
.withValue(new TextValue("answer_id", answer.answer_id.get.toString))
.withValue(new TextValue("image", convertImageToString(imageData)))
.withValue(new TextValue("answer", convertAnswersFromModelToString(answer.answer)))
.withValue(new BigIntValue("creation_year", answer.creationYear.getOrElse(0)))
.withValue(new BigIntValue("creation_month", answer.creationMonth.getOrElse(0)))
.withValue(new TextValue("notes", answer.notes.getOrElse("")))
logger.trace(s"putting answer ${putAnswer}")
//checktest-add answer to respository
//checktest-not add answer to respository if duplicate
transaction.put(putAnswer)
}
Why am I getting the error even though the notes field is changed between the existing answer and the updated answer
The error trace is (note that it says IF NOT EXISTS!). Shouldn't it be IF EXISTS? There is also a trace there was a hit in the statement cache for [INSERT INTO codingjedi.answer_by_user_id_and_question_id (answered_by_user,question_id,tx_id,tx_state,tx_prepared_at,answer_id,image,answer,creation_year,creation_month,notes,tx_version) VALUES (?,?,?,?,?,?,?,?,?,?,?,?) IF NOT EXISTS;]. Does it mean that the previous put is still in cache and is that causing the conflict?
2020-09-24 18:51:33,593 [DEBUG] from com.scalar.db.storage.cassandra.StatementHandler in ScalaTest-run-running-AllRepositorySpecs - query to prepare : [INSERT INTO codingjedi.answer_by_user_id_and_question_id (answered_by_user,question_id,tx_id,tx_state,tx_prepared_at,answer_id,image,answer,creation_year,creation_month,notes,tx_version) VALUES (?,?,?,?,?,?,?,?,?,?,?,?) IF NOT EXISTS;].
2020-09-24 18:51:33,593 [DEBUG] from com.scalar.db.storage.cassandra.StatementHandler in ScalaTest-run-running-AllRepositorySpecs - there was a hit in the statement cache for [INSERT INTO codingjedi.answer_by_user_id_and_question_id (answered_by_user,question_id,tx_id,tx_state,tx_prepared_at,answer_id,image,answer,creation_year,creation_month,notes,tx_version) VALUES (?,?,?,?,?,?,?,?,?,?,?,?) IF NOT EXISTS;].
2020-09-24 18:51:33,593 [DEBUG] from com.scalar.db.storage.cassandra.ValueBinder in ScalaTest-run-running-AllRepositorySpecs - Optional[11111111-1111-1111-1111-111111111111] is bound to 0
2020-09-24 18:51:33,593 [DEBUG] from com.scalar.db.storage.cassandra.ValueBinder in ScalaTest-run-running-AllRepositorySpecs - Optional[11111111-1111-1111-1111-111111111111] is bound to 1
2020-09-24 18:51:33,593 [DEBUG] from com.scalar.db.storage.cassandra.ValueBinder in ScalaTest-run-running-AllRepositorySpecs - Optional[468492df-0960-4160-8391-27fe7fa626c5] is bound to 2
2020-09-24 18:51:33,593 [DEBUG] from com.scalar.db.storage.cassandra.ValueBinder in ScalaTest-run-running-AllRepositorySpecs - 1 is bound to 3
2020-09-24 18:51:33,593 [DEBUG] from com.scalar.db.storage.cassandra.ValueBinder in ScalaTest-run-running-AllRepositorySpecs - 1600969893592 is bound to 4
2020-09-24 18:51:33,593 [DEBUG] from com.scalar.db.storage.cassandra.ValueBinder in ScalaTest-run-running-AllRepositorySpecs - Optional[11111111-1111-1111-1111-111111111111] is bound to 5
2020-09-24 18:51:33,593 [DEBUG] from com.scalar.db.storage.cassandra.ValueBinder in ScalaTest-run-running-AllRepositorySpecs - Optional[{"image":["image1binarydata","image2binarydata"]}] is bound to 6
2020-09-24 18:51:33,593 [DEBUG] from com.scalar.db.storage.cassandra.ValueBinder in ScalaTest-run-running-AllRepositorySpecs - Optional[{"answer":[{"filename":"c.js","answer":"some answer"}]}] is bound to 7
2020-09-24 18:51:33,593 [DEBUG] from com.scalar.db.storage.cassandra.ValueBinder in ScalaTest-run-running-AllRepositorySpecs - 2019 is bound to 8
2020-09-24 18:51:33,593 [DEBUG] from com.scalar.db.storage.cassandra.ValueBinder in ScalaTest-run-running-AllRepositorySpecs - 12 is bound to 9
2020-09-24 18:51:33,593 [DEBUG] from com.scalar.db.storage.cassandra.ValueBinder in ScalaTest-run-running-AllRepositorySpecs - Optional[some notesupdated] is bound to 10
2020-09-24 18:51:33,593 [DEBUG] from com.scalar.db.storage.cassandra.ValueBinder in ScalaTest-run-running-AllRepositorySpecs - 1 is bound to 11
2020-09-24 18:51:33,607 [WARN] from com.scalar.db.transaction.consensuscommit.CommitHandler in ScalaTest-run-running-AllRepositorySpecs - preparing records failed
com.scalar.db.exception.storage.NoMutationException: no mutation was applied.
UPDATE
Traces
For the 1st put, putAnswer is
putting answer Put{namespace=Optional[codingjedi], table=Optional[answer_by_user_id_and_question_id], partitionKey=Key{TextValue{name=answered_by_user, value=Optional[11111111-1111-1111-1111-111111111111]}, TextValue{name=question_id, value=Optional[11111111-1111-1111-1111-111111111111]}}, clusteringKey=Optional.empty, values={answer_id=TextValue{na
me=answer_id, value=Optional[11111111-1111-1111-1111-111111111111]}, image=TextValue{name=image, value=Optional[{"image":["image1binarydata","image2binarydata"]}]}, answer=TextValue{name=answer, value=Optional[{"answer":[{"filename":"c.j
s","answer":"some answer"}]}]}, creation_year=BigIntValue{name=creation_year, value=2019}, creation_month=BigIntValue{name=creation_month, value=12}, notes=TextValue{name=notes, value=Optional[some notes]}}, consistency=SEQUENTIAL, condi
tion=Optional[com.scalar.db.api.PutIfNotExists#21bf308]}
For the 2nd put, putAnswer is
putting answer Put{namespace=Optional[codingjedi], table=Optional[answer_by_user_id_and_question_id], partitionKey=Key{TextValue{name=answered_by_user, value=Optional[11111111-1111-1111-1111-111111111111]}, TextValue{name=question_id, value=Optional[11111111-1111-1111-1111-111111111111]}}, clusteringKey=Optional.empty, values={answer_id=TextValue{name=answer_id, value=Optional[11111111-1111-1111-1111-111111111111]}, image=TextValue{name=image, value=Optional[{"image":["image1binarydata","image2binarydata"]}]}, answer=TextValue{name=answer, value=Optional[{"answer":[{"filename":"c.js","answer":"some answer"}]}]}, creation_year=BigIntValue{name=creation_year, value=2019}, creation_month=BigIntValue{name=creation_month, value=12}, notes=TextValue{name=notes, value=Optional[some notesupdated]}}, consistency=SEQUENTIAL, condition=Optional[com.scalar.db.api.PutIfExists#2e057637]}
notes field has changed from notes=TextValue{name=notes, value=Optional[some notes]}}, to notes=TextValue{name=notes, value=Optional[some notesupdated]}}
When the 2nd put is executed, I can see that the mutation condition used is IfNotExists
2020-09-25 12:35:34,188 [DEBUG] from com.scalar.db.storage.cassandra.Cassandra in ScalaTest-run-running-AllRepositorySpecs - executing put operation with Put{namespace=Optional[codingjedi], table=Optional[answer_by_user_id_and_question_id], partitionKey=Key{TextValue{name=answered_by_user, value=Optional[11111111-1111-1111-1111-111111111111]}, TextValue{name=question_id, value=Optional[11111111-1111-1111-1111-111111111111]}}, clusteringKey=Optional.empty, values={tx_id=TextValue{name=tx_id, value=Optional[c6bc39e9-656a-440c-8f68-af6005f37f7c]}, tx_state=IntValue{name=tx_state, value=1}, tx_prepared_at=BigIntValue{name=tx_prepared_at, value=1601033734188}, answer_id=TextValue{name=answer_id, value=Optional[11111111-1111-1111-1111-111111111111]}, image=TextValue{name=image, value=Optional[{"image":["image1binarydata","image2binarydata"]}]}, answer=TextValue{name=answer, value=Optional[{"answer":[{"filename":"c.js","answer":"some answer"}]}]}, creation_year=BigIntValue{name=creation_year, value=2019}, creation_month=BigIntValue{name=creation_month, value=12}, notes=TextValue{name=notes, value=Optional[**some notesupdated**]}, tx_version=IntValue{name=tx_version, value=1}}, consistency=LINEARIZABLE, condition=Optional[com.scalar.db.api.**PutIfNotExists**#21bf308]}
Scalar DB doesn't allow a blind write for the existing record. It looks there is no get before the update.
I think this process should check the current values and update the values in a transaction. In this code, there is no guarantee for atomicity between the get and the update.
Since the protocol is based on Snapshot Isolation, it needs to get a record in a snapshot first to update the record.
After Yuji's comment below, I changed the update method to add a get before put and the operation is successful now.
def update(transaction:DistributedTransaction, answer:AnswerOfAPracticeQuestion) = {
logger.trace(s"updating answer value ${answer}")
val key = AnswerKeys(answer.answeredBy.get.answerer_id,answer.question_id,answer.answer_id)
val result = get(transaction,key)
if(result.isLeft) throw result.left.get else add(transaction,answer, new PutIfExists())
}
I however don't understand why a get is required before update or delete in Scalardb

mongoDB inserting twice when called on different Threads

Basically I am consuming Messages from spring cloud stream kafka and inserting it into the MongoDB
My code works fine if my mongo cluster is up
I have 2 problems In case My Mongo Instance is down
auto commit of cloud stream is disabled (autoCommitOffset set to false) then also re-polling is not happening even if it hasn't Acknowledged the message yet
While Checking For Mongo Connection it takes some time and in that time period if it receive two meesages with same ID and after that if i start the instance of mongo it duplicates the messages which in normal case is working fine
Do we have any solution for these?
Here is my code,
interface ResourceInventorySink {
companion object {
const val INPUT = "resourceInventoryInput"
}
#Input(INPUT)
fun input(): SubscribableChannel
}
#EnableBinding(ResourceInventorySink::class)
class InventoryEventListeners {
val logger = LoggerFactory.getLogger(javaClass)
#Autowired
lateinit var resourceInventoryService : ResourceInventoryService
#StreamListener(ResourceInventorySink.INPUT, condition = OperationConstants.INSERT)
fun receiveInsert(event : Message<ResourceInventoryEvent>) {
logger.info("received Insert message {}", event.payload.toString())
val success = resourceInventoryService.insert(event.payload)
success.subscribe({
logger.info("Data Inserted", event.payload.toString())
event.headers.get(KafkaHeaders.ACKNOWLEDGMENT, Acknowledgment::class.java)?.acknowledge()
},{
if(it !is DataAccessResourceFailureException) {
logger.error("Exception Occured {} {}", it.message , it.cause.toString())
event.headers.get(KafkaHeaders.ACKNOWLEDGMENT, Acknowledgment::class.java)?.acknowledge()
}
else {
logger.error("Error Inserting in Mongo DB {}", it.cause)
}
})
}
Here is my service class
#Service
class ResourceInventoryService() {
val logger = LoggerFactory.getLogger(javaClass)
#Autowired
lateinit var resourceInventoryRepository: ResourceInventoryRepository
fun insert(newResource: ResourceInventoryEvent) = resourceInventoryRepository
.findByProductId(newResource.productId)
.switchIfEmpty(newResource.convertTODocument().toMono())
.flatMap { resourceInventoryRepository.save(it) }
.onErrorResume { Mono.error(it) }
this is my application.yml
spring:
cloud:
stream:
default:
consumer:
useNativeEncoding: true
kafka:
binder:
brokers:
- localhost:9092
consumer-properties:
key.deserializer : org.apache.kafka.common.serialization.StringDeserializer
value.deserializer: io.confluent.kafka.serializers.KafkaAvroDeserializer
schema.registry.url: http://localhost:8081
enable.auto.commit: false
specific.avro.reader: true
bindings:
resourceInventoryInput:
consumer:
autoOffsetCommit: false
default-binder: kafka
bindings:
resourceInventoryInput:
binder: kafka
destination: ${application.messaging.topic}
content-type: application/*+avro
group: ${application.messaging.group}
EDIT 1. Acknowledgment is null

Spring Cloud Stream Kafka application not generating messages with the correct Avro schema

I have an application (spring-boot-shipping-service) with a KStream that gets OrderCreatedEvent messages generated by an external producer (spring-boot-order-service). This producer uses the following schema:
order-created-event.avsc
{
"namespace" : "com.codependent.statetransfer.order",
"type" : "record",
"name" : "OrderCreatedEvent",
"fields" : [
{"name":"id","type":"int"},
{"name":"productId","type":"int"},
{"name":"customerId","type":"int"}
]
}
My KStream<Int, OrderCreatedEvent> is joined with a KTable<Int, Customer> and publishes to the order topic a new kind of message: OrderShippedEvent.
order-shipped-event.avsc
{
"namespace" : "com.codependent.statetransfer.order",
"type" : "record",
"name" : "OrderShippedEvent",
"fields" : [
{"name":"id","type":"int"},
{"name":"productId","type":"int"},
{"name":"customerName","type":"string"},
{"name":"customerAddress","type":"string"}
]
}
For some reason the new OrderShippedEvent messages aren't generated with a header application/vnd.ordershippedevent.v1+avro but application/vnd.ordercreatedevent.v1+avro.
This is the original OrderCreatedEvent in the order topic:
Key (4 bytes): +
Value (4 bytes): V?
Timestamp: 1555943926163
Partition: 0
Offset: 34
Headers: contentType="application/vnd.ordercreatedevent.v1+avro",spring_json_header_types={"contentType":"java.lang.String"}
And the produced OrderShippedEvent with the incorrect schema:
Key (4 bytes): +
Value (26 bytes): V?
JamesHill Street
Timestamp: 1555943926163
Partition: 0
Offset: 35
Headers: contentType="application/vnd.ordercreatedevent.v1+avro",spring_json_header_types={"contentType":"java.lang.String"}
I've checked the Confluent Schema Registry contents, and the order-shipped-event.avsc schema is there:
Why isn't it using the correct shema in the generated message?
Below you can see the full configuration and code of the example, which is also available on Github (https://github.com/codependent/event-carried-state-transfer/tree/avro)
In order to test it just start a Confluent Platform (v5.2.1), spring-boot-customer-service, spring-boot-order-service, spring-boot-shipping-service and execute the following curl commands:
curl -X POST http://localhost:8080/customers -d '{"id":1,"name":"James","address":"Hill Street"}' -H "content-type: application/json"
curl -X POST http://localhost:8084/orders -H "content-type: application/json" -d '{"id":1,"productId":1001,"/customerId":1}'
application.yml
server:
port: 8085
spring:
application:
name: spring-boot-shipping-service
cloud:
stream:
kafka:
streams:
binder:
configuration:
default:
key:
serde: org.apache.kafka.common.serialization.Serdes$IntegerSerde
bindings:
input:
destination: customer
contentType: application/*+avro
order:
destination: order
contentType: application/*+avro
output:
destination: order
contentType: application/*+avro
schema-registry-client:
endpoint: http://localhost:8081
ShippingKStreamProcessor
interface ShippingKStreamProcessor {
#Input("input")
fun input(): KStream<Int, Customer>
#Input("order")
fun order(): KStream<String, OrderCreatedEvent>
#Output("output")
fun output(): KStream<String, OrderShippedEvent>
ShippingKStreamConfiguration
#StreamListener
#SendTo("output")
fun process(#Input("input") input: KStream<Int, Customer>, #Input("order") orderEvent: KStream<Int, OrderCreatedEvent>): KStream<Int, OrderShippedEvent> {
val serdeConfig = mapOf(
AbstractKafkaAvroSerDeConfig.SCHEMA_REGISTRY_URL_CONFIG to "http://localhost:8081")
val intSerde = Serdes.IntegerSerde()
val customerSerde = SpecificAvroSerde<Customer>()
customerSerde.configure(serdeConfig, true)
val orderCreatedSerde = SpecificAvroSerde<OrderCreatedEvent>()
orderCreatedSerde.configure(serdeConfig, true)
val orderShippedSerde = SpecificAvroSerde<OrderShippedEvent>()
orderShippedSerde.configure(serdeConfig, true)
val stateStore: Materialized<Int, Customer, KeyValueStore<Bytes, ByteArray>> =
Materialized.`as`<Int, Customer, KeyValueStore<Bytes, ByteArray>>("customer-store")
.withKeySerde(intSerde)
.withValueSerde(customerSerde)
val customerTable: KTable<Int, Customer> = input.groupByKey(Serialized.with(intSerde, customerSerde))
.reduce({ _, y -> y }, stateStore)
return (orderEvent.filter { _, value -> value is OrderCreatedEvent && value.id != 0 }
.selectKey { _, value -> value.customerId } as KStream<Int, OrderCreatedEvent>)
.join(customerTable, { orderIt, customer ->
OrderShippedEvent(orderIt.id, orderIt.productId, customer.name, customer.address)
}, Joined.with(intSerde, orderCreatedSerde, customerSerde))
.selectKey { _, value -> value.id }
}
UPDATE: I've set trace logging level for org.springframework.messaging and apparently it looks ok:
2019-04-22 23:40:39.953 DEBUG 46039 --- [-StreamThread-1] o.s.web.client.RestTemplate : HTTP GET http://localhost:8081/subjects/ordercreatedevent/versions/1
2019-04-22 23:40:39.971 DEBUG 46039 --- [-StreamThread-1] o.s.web.client.RestTemplate : Accept=[application/json, application/*+json]
2019-04-22 23:40:39.972 DEBUG 46039 --- [-StreamThread-1] o.s.web.client.RestTemplate : Writing [] as "application/vnd.schemaregistry.v1+json"
2019-04-22 23:40:39.984 DEBUG 46039 --- [-StreamThread-1] o.s.web.client.RestTemplate : Response 200 OK
2019-04-22 23:40:39.985 DEBUG 46039 --- [-StreamThread-1] o.s.web.client.RestTemplate : Reading to [java.util.Map<?, ?>]
2019-04-22 23:40:40.186 INFO 46039 --- [read-1-producer] org.apache.kafka.clients.Metadata : Cluster ID: 5Sw6sBD0TFOaximF3Or-dQ
2019-04-22 23:40:40.318 DEBUG 46039 --- [-StreamThread-1] AvroSchemaRegistryClientMessageConverter : Obtaining schema for class class com.codependent.statetransfer.order.OrderShippedEvent
2019-04-22 23:40:40.318 DEBUG 46039 --- [-StreamThread-1] AvroSchemaRegistryClientMessageConverter : Avro type detected, using schema from object
2019-04-22 23:40:40.342 DEBUG 46039 --- [-StreamThread-1] o.s.web.client.RestTemplate : HTTP POST http://localhost:8081/subjects/ordershippedevent/versions
2019-04-22 23:40:40.342 DEBUG 46039 --- [-StreamThread-1] o.s.web.client.RestTemplate : Accept=[application/json, application/*+json]
2019-04-22 23:40:40.342 DEBUG 46039 --- [-StreamThread-1] o.s.web.client.RestTemplate : Writing [{"schema":"{\"type\":\"record\",\"name\":\"OrderShippedEvent\",\"namespace\":\"com.codependent.statetransfer.order\",\"fields\":[{\"name\":\"id\",\"type\":\"int\"},{\"name\":\"productId\",\"type\":\"int\"},{\"name\":\"customerName\",\"type\":{\"type\":\"string\",\"avro.java.string\":\"String\"}},{\"name\":\"customerAddress\",\"type\":{\"type\":\"string\",\"avro.java.string\":\"String\"}}]}"}] as "application/json"
2019-04-22 23:40:40.348 DEBUG 46039 --- [-StreamThread-1] o.s.web.client.RestTemplate : Response 200 OK
2019-04-22 23:40:40.348 DEBUG 46039 --- [-StreamThread-1] o.s.web.client.RestTemplate : Reading to [java.util.Map<?, ?>]
2019-04-22 23:40:40.349 DEBUG 46039 --- [-StreamThread-1] o.s.web.client.RestTemplate : HTTP POST http://localhost:8081/subjects/ordershippedevent
2019-04-22 23:40:40.349 DEBUG 46039 --- [-StreamThread-1] o.s.web.client.RestTemplate : Accept=[application/json, application/*+json]
2019-04-22 23:40:40.349 DEBUG 46039 --- [-StreamThread-1] o.s.web.client.RestTemplate : Writing [{"schema":"{\"type\":\"record\",\"name\":\"OrderShippedEvent\",\"namespace\":\"com.codependent.statetransfer.order\",\"fields\":[{\"name\":\"id\",\"type\":\"int\"},{\"name\":\"productId\",\"type\":\"int\"},{\"name\":\"customerName\",\"type\":{\"type\":\"string\",\"avro.java.string\":\"String\"}},{\"name\":\"customerAddress\",\"type\":{\"type\":\"string\",\"avro.java.string\":\"String\"}}]}"}] as "application/json"
2019-04-22 23:40:40.361 DEBUG 46039 --- [-StreamThread-1] o.s.web.client.RestTemplate : Response 200 OK
2019-04-22 23:40:40.362 DEBUG 46039 --- [-StreamThread-1] o.s.web.client.RestTemplate : Reading to [java.util.Map<?, ?>]
2019-04-22 23:40:40.362 DEBUG 46039 --- [-StreamThread-1] AvroSchemaRegistryClientMessageConverter : Finding correct DatumWriter for type com.codependent.statetransfer.order.OrderShippedEvent
How come the message is written with an incorrect content type header then?
UPDATE 2:
I've kept digging into the source code and found this:
KafkaStreamsMessageConversionDelegate correctly converts and determines the right header values, as seen in the logs above.
However in the serializeOnOutbound method we can find that it returns to the Kafka API only the payload, so the headers aren't taken into account:
return
messageConverter.toMessage(message.getPayload(),
messageHeaders).getPayload();
Moving forward in the record processing org.apache.kafka.streams.processor.internals.SinkNode.process() accesses the headers present in the context, which incorrectly contain application/vnd.ordercreatedevent.v1+avro instead of application/vnd.ordershippedevent.v1+avro (?):
collector.send(topic, key, value, context.headers(), timestamp, keySerializer, valSerializer, partitioner);
UPDATE 3:
Steps to reproduce:
Download and start Confluent 5.2.1
confluent start
Start the applications spring-boot-order-service, spring-boot-customer-service, spring-boot-shipping-service
Create a customer curl -X POST http://localhost:8080/customers -d '{"id":1,"name":"John","address":"Some Street"}' -H "content-type: application/json"
Create an order that will be joined with the customer: curl -X POST http://localhost:8084/orders -H "content-type: application/json" -d '{"id":1,"productId":1,"customerId":1}'
ShippingKStreamConfiguration's process() will create a KTable for the Customer and a state store (customer-store). Besides, it will join the order stream with the customer KTable to transform an OrderCreatedEvent into an OrderShippedEvent.
You can check that the newly created OrderShippedEvent message added to the order topic has an incorrect header. This can be seen either in the Confluent Control Center (localhost:9092 -> topics -> order) or running kafkacat:
$> kafkacat -b localhost:9092 -t order -C \
-f '\nKey (%K bytes): %k
Value (%S bytes): %s
Timestamp: %T
Partition: %p
Offset: %o
Headers: %h\n'
#codependent It is indeed an issue that we need to address in the binder which we will fix soon. In the meantime, as a workaround can you make your processor not return a KStream, but rather do the sending in the method itself. You can call to(TopicNameExtractor) on the currently returned KStream. TopicNameExtractor will give you access to the record context using which you can manually set the content type.