Ok, I realise this sounds unlikely and I'm prepared to be shot down on this one but here goes...
I have a gremlin server running against titanDB and dynamoDB (local). I'm running some unit tests that keep failing due to
tx 0x705eafda280e already locked key-column ( 8- 0- 0- 0- 0- 0- 0-128, 80-160) when tx 0x70629e1d56bf tried to lock
I'm running the following commands in the gremlin client console against a clean, completely empty DB (recreated between test runs using docker images). The aim of this work is to support database upgrade scripts. The original actual steps where more complete than the below but this is the minimum to reproduce the issue.
(Connect to local 'remote')
:remote connect tinkerpop.server conf/remote.yaml
(Add a unique constraint on a 'databaseMetadata' label which has a single 'version' property)
:> mgmt = graph.openManagement();if (!mgmt.getGraphIndex("bydatabaseMetadataversion")) {graph.tx().rollback();int size = graph.getOpenTransactions().size();for (i = 0; i < size; i++) { try { graph.getOpenTransactions().getAt(0).rollback();} catch(Throwable ex) { }; }; mgmt = graph.openManagement();propertyKey = (!mgmt.containsPropertyKey("version")) ? mgmt.makePropertyKey("version").dataType(String.class).cardinality(Cardinality.SINGLE).make():mgmt.getPropertyKey("version");labelObj = (!mgmt.containsVertexLabel("databaseMetadata")) ? mgmt.makeVertexLabel("databaseMetadata").make():mgmt.getVertexLabel("databaseMetadata");index = mgmt.buildIndex("bydatabaseMetadataversion", Vertex.class).addKey(propertyKey).unique().indexOnly(labelObj).buildCompositeIndex();mgmt.setConsistency(propertyKey, ConsistencyModifier.LOCK);mgmt.setConsistency(index, ConsistencyModifier.LOCK);mgmt.commit();mgmt = graph.openManagement();index = mgmt.getGraphIndex("bydatabaseMetadataversion");propertyKey = mgmt.getPropertyKey("version");if (index.getIndexStatus(propertyKey) == SchemaStatus.INSTALLED) {mgmt.awaitGraphIndexStatus(graph, "bydatabaseMetadataversion").status(SchemaStatus.REGISTERED).timeout(10, java.time.temporal.ChronoUnit.MINUTES).call();}; mgmt.commit();mgmt = graph.openManagement();index = mgmt.getGraphIndex("bydatabaseMetadataversion");propertyKey = mgmt.getPropertyKey("version");if (index.getIndexStatus(propertyKey) != SchemaStatus.ENABLED) {mgmt.commit();mgmt = graph.openManagement();mgmt.updateIndex(mgmt.getGraphIndex("bydatabaseMetadataversion"), SchemaAction.ENABLE_INDEX).get();mgmt.commit();mgmt = graph.openManagement();mgmt.awaitGraphIndexStatus(graph, "bydatabaseMetadataversion").status(SchemaStatus.ENABLED).timeout(10, java.time.temporal.ChronoUnit.MINUTES).call();}; mgmt.commit();} else {index = mgmt.getGraphIndex("bydatabaseMetadataversion");propertyKey = mgmt.getPropertyKey("version");if (index.getIndexStatus(propertyKey) != SchemaStatus.ENABLED) {mgmt.awaitGraphIndexStatus(graph, "bydatabaseMetadataversion").status(SchemaStatus.ENABLED).timeout(10, java.time.temporal.ChronoUnit.MINUTES).call();}; mgmt.commit();};
(Add the metadata vertex with initial version '0.0.1')
:> graph.addVertex(label, "databaseMetadata").property("version", "0.0.1");graph.tx().commit();
(Update the metadata vertex with the next version - 0.0.2)
:> g.V().hasLabel("databaseMetadata").has("version", "0.0.1").property("version", "0.0.2").next();g.tx().commit();
(THIS FAILS - Update the metadata vertex with the next version - 0.0.3)
:> g.V().hasLabel("databaseMetadata").has("version", "0.0.2").property("version", "0.0.3").next();g.tx().commit();
tx 0x705eafda280e already locked key-column ( 8- 0- 0- 0- 0- 0- 0-128, 80-160) when tx 0x70629e1d56bf tried to lock
Previously I had looked through the titan-dynamodb source and I saw that the commits/rollbacks etc of the transactions are logged, so I had changed the log level to get further information (full log file available).
When the 0.0.1 -> 0.0.2 update was executed the following locks were acquired:
[33mtitan_server_1 |[0m 120479 [gremlin-server-exec-3] TRACE com.amazon.titan.diskstorage.dynamodb.AbstractDynamoDBStore - acquiring lock on ( 8- 0- 0- 0- 0- 0- 0-128, 80-160) at 123552624951495
[33mtitan_server_1 |[0m 120489 [gremlin-server-exec-3] TRACE com.amazon.titan.diskstorage.dynamodb.AbstractDynamoDBStore - acquiring lock on ( 6-137-160- 48- 46- 48- 46-177, 0) at 123552635424334
[33mtitan_server_1 |[0m 120489 [gremlin-server-exec-3] TRACE com.amazon.titan.diskstorage.dynamodb.AbstractDynamoDBStore - acquiring lock on ( 6-137-160- 48- 46- 48- 46-178, 0) at 123552635704705
When that transaction was commited only TWO locks where released.
[33mtitan_server_1 |[0m 120722 [gremlin-server-exec-3] DEBUG com.amazon.titan.diskstorage.dynamodb.DynamoDBStoreTransaction - commit id:0x705eafda280e
[33mtitan_server_1 |[0m 120722 [gremlin-server-exec-3] TRACE com.amazon.titan.diskstorage.dynamodb.AbstractDynamoDBStore - Expiring ( 6-137-160- 48- 46- 48- 46-177, 0) in tx 0x705eafda280e because of EXPLICIT
[33mtitan_server_1 |[0m 120722 [gremlin-server-exec-3] TRACE com.amazon.titan.diskstorage.dynamodb.AbstractDynamoDBStore - Expiring ( 6-137-160- 48- 46- 48- 46-178, 0) in tx 0x705eafda280e because of EXPLICIT
[33mtitan_server_1 |[0m 120722 [gremlin-server-exec-3] DEBUG org.apache.tinkerpop.gremlin.server.op.AbstractEvalOpProcessor - Preparing to iterate results from - RequestMessage{, requestId=09f27811-dcc3-4e53-a749-22828d34997f, op='eval', processor='', args={gremlin=g.V().hasLabel("databaseMetadata").has("version", "0.0.1").property("version", "0.0.2").next();g.tx().commit();, batchSize=64}} - in thread [gremlin-server-exec-3]
The remaining lock ends up expiring after a few minutes, but in the mean time every other update fails as reported.
So, why does that lock not get removed? I suspect it's related to the unique index that's created, so I've either setup the index wrong (a good possibility) or this is a bug.
For ease of consumption, the (slightly shortened) index setup is below:
mgmt = graph.openManagement()
propertyKey = (!mgmt.containsPropertyKey("version")) ? mgmt.makePropertyKey("version").dataType(String.class).cardinality(Cardinality.SINGLE).make():mgmt.getPropertyKey("version")
labelObj = (!mgmt.containsVertexLabel("databaseMetadata")) ? mgmt.makeVertexLabel("databaseMetadata").make():mgmt.getVertexLabel("databaseMetadata")
index = mgmt.buildIndex("bydatabaseMetadataversion", Vertex.class).addKey(propertyKey).unique().indexOnly(labelObj).buildCompositeIndex()
mgmt.setConsistency(propertyKey, ConsistencyModifier.LOCK)
mgmt.setConsistency(index, ConsistencyModifier.LOCK)
mgmt.commit()
mgmt = graph.openManagement()
index = mgmt.getGraphIndex("bydatabaseMetadataversion")
propertyKey = mgmt.getPropertyKey("version")
if (index.getIndexStatus(propertyKey) == SchemaStatus.INSTALLED) {
mgmt.awaitGraphIndexStatus(graph, "bydatabaseMetadataversion").status(SchemaStatus.REGISTERED).timeout(10, java.time.temporal.ChronoUnit.MINUTES).call()
}
mgmt.commit()
mgmt = graph.openManagement()
index = mgmt.getGraphIndex("bydatabaseMetadataversion")
propertyKey = mgmt.getPropertyKey("version")
if (index.getIndexStatus(propertyKey) != SchemaStatus.ENABLED) {
mgmt.commit()
mgmt = graph.openManagement()
mgmt.updateIndex(mgmt.getGraphIndex("bydatabaseMetadataversion"), SchemaAction.ENABLE_INDEX).get()
mgmt.commit()
mgmt = graph.openManagement()
mgmt.awaitGraphIndexStatus(graph, "bydatabaseMetadataversion").status(SchemaStatus.ENABLED).timeout(10, java.time.temporal.ChronoUnit.MINUTES).call()
}
mgmt.commit()
I know this is a LONG issue description, but any help would be gratefully received!
(I should also say that I tried this against the local and cloud based dynamoDb instances and had the same issue on both, so came back to the local and turned on the logging.)
I'm using titan 1.0.0 and tinkerpop 3 as set in dynamo-titan on github.
I got repro and and found your issue. Basically, the LRU cache pulls its expiry time from storage.lock.expiry-time config. Default is 5 minutes, so if you try to make changes before 5 minutes pass, yes, the AbstractDynamoDBStore.keyColumnLocalLocks LRU cache will not let you make the second change. By reducing expiry-time and Thread.sleep() before making the second change, you allow the second change to claim the lock again and succeed.
//default lock expiry time is 300*1000 ms = 5 minutes. Set to 100ms.
config.setProperty("storage.lock.expiry-time", 100);
F.y.i. I have run all of your above code in Java using a Berkeley storage backend.
TitanGraph graph = ...;
TitanManagement mgmt = graph.openManagement();
PropertyKey propertyKey = (!mgmt.containsPropertyKey("version"))
? mgmt.makePropertyKey("version").dataType(String.class).cardinality(Cardinality.SINGLE).make()
: mgmt.getPropertyKey("version");
VertexLabel labelObj = (!mgmt.containsVertexLabel("databaseMetadata"))
? mgmt.makeVertexLabel("databaseMetadata").make()
: mgmt.getVertexLabel("databaseMetadata");
TitanGraphIndex index = mgmt.buildIndex("bydatabaseMetadataversion", Vertex.class).addKey(propertyKey).unique()
.indexOnly(labelObj).buildCompositeIndex();
mgmt.setConsistency(propertyKey, ConsistencyModifier.LOCK);
mgmt.setConsistency(index, ConsistencyModifier.LOCK);
mgmt.commit();
mgmt = graph.openManagement();
index = mgmt.getGraphIndex("bydatabaseMetadataversion");
propertyKey = mgmt.getPropertyKey("version");
if (index.getIndexStatus(propertyKey) == SchemaStatus.INSTALLED) {
try {
ManagementSystem.awaitGraphIndexStatus(graph,"bydatabaseMetadataversion").status(SchemaStatus.REGISTERED).timeout(10, java.time.temporal.ChronoUnit.MINUTES).call();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
mgmt.commit();
mgmt = graph.openManagement();
index = mgmt.getGraphIndex("bydatabaseMetadataversion");
propertyKey = mgmt.getPropertyKey("version");
if (index.getIndexStatus(propertyKey) != SchemaStatus.ENABLED) {
mgmt.commit();
mgmt = graph.openManagement();
try {
mgmt.updateIndex(mgmt.getGraphIndex("bydatabaseMetadataversion"), SchemaAction.ENABLE_INDEX).get();
} catch (InterruptedException | ExecutionException e) {
e.printStackTrace();
}
mgmt.commit();
mgmt = graph.openManagement();
try {
ManagementSystem.awaitGraphIndexStatus(graph, "bydatabaseMetadataversion").status(SchemaStatus.ENABLED)
.timeout(10, java.time.temporal.ChronoUnit.MINUTES).call();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
mgmt.commit();
Then the operations on the graph;
GraphTraversalSource g = graph.traversal();
graph.addVertex("databaseMetadata").property("version", "0.0.1");
graph.tx().commit();
g.V().hasLabel("databaseMetadata").has("version", "0.0.1").property("version", "0.0.2").iterate();
g.tx().commit();
g.V().hasLabel("databaseMetadata").has("version", "0.0.1").property("version", "0.0.2").iterate();
g.tx().commit();
g.V().hasLabel("databaseMetadata").has("version", "0.0.2").property("version", "0.0.3").iterate();
g.tx().commit();
g.V().hasLabel("databaseMetadata").has("version").properties("version").forEachRemaining(prop -> {
System.out.println("Version: " + prop.value());
});
The result was:
Version: 0.0.3
Sadly, the iterate() alteration of your query only applies to Java. Your scripts should work as they are. Because of the result of my experiment I strongly suspect that the DynamoDB backend is causing trouble.
Related
I tried to implement a simple read-preferred read-write lock using 2 mutexes (using redis.lock.Lock), like what is described in this link (https://en.wikipedia.org/wiki/Readers%E2%80%93writer_lock)
In the [End Read] steps, I encountered this problem:
If b = 0, unlock g #(a write lock).
As, this READ process is not the one that acquired the lock, so the system throws an error. I think it has some token stored somewhere, and I can get it to use for the lock release, but I am not sure.
Could someone give me a hint? Thanks.
from enum import Enum
from redis import StrictRedis, lock
# data in Redis cache:
# "read_counter_lock_name" : 0
# "read_lock_lock_name" -> read lock, protect "read_counter_lock_name"
# "write_lock_lock_name" -> write lock, protect write data section
class Prefix(Enum):
READ = 'read_lock_'
WRITE = 'write_lock_'
READ_COUNTER = 'read_counter_'
class RedisLockParams(Enum):
TIMEOUT = 60 # maximum life for the lock in seconds = 60 seconds
SLEEP_TIME = 0.1 # the amount of time to sleep in seconds per loop iteration
# in redis lock's acquire() - sleep then retry
BLOCKING = True # acquire() should block until the lock has been acquired
BLOCKING_TIMEOUT = None # maximum amount of time in seconds to spend trying
# to acquire the lock
class ReadWriteLock:
def __init__(self, lock_name: str, redis_host: str, redis_port: int, redis_key: str):
self.__read_lock_name = Prefix.READ.value + lock_name
self.__write_lock_name = Prefix.WRITE.value + lock_name
self.__read_counter_key = Prefix.READ_COUNTER.value + lock_name
# self.__redis_host = redis_host
# self.__redis_port = redis_port
# self.__redis_key = redis_key
self.__cache = StrictRedis(host = redis_host,
port = redis_port,
db=0, # up to 16 logical database
password = redis_key,
ssl=True)
print(f'ping return:{self.__cache.ping()}')
# set the read counter to 0, if it does not exist.
self.__cache.setnx(self.__read_counter_key, 0)
# init the read lock
self.__read_lock = lock.Lock(self.__cache,
self.__read_lock_name,
RedisLockParams.TIMEOUT.value,
RedisLockParams.SLEEP_TIME.value,
RedisLockParams.BLOCKING.value,
RedisLockParams.BLOCKING_TIMEOUT.value)
# init the write lock
self.__write_lock = lock.Lock(self.__cache,
self.__write_lock_name,
RedisLockParams.TIMEOUT.value,
RedisLockParams.SLEEP_TIME.value,
RedisLockParams.BLOCKING.value,
RedisLockParams.BLOCKING_TIMEOUT.value)
def acquire_read_lock(self) -> bool:
write_lock_acquired = False
self.__read_lock.acquire()
read_counter = self.__cache.incr(self.__read_counter_key)
if (read_counter == 1):
write_lock_acquired = self.__write_lock.acquire() # acquire write lock
self.__read_lock.release()
return write_lock_acquired
def release_read_lock(self):
read_lock_acquired = self.__read_lock.acquire()
read_counter = self.__cache.decr(self.__read_counter_key)
if read_counter == 0 and read_lock_acquired:
self.__write_lock.release() # release the write lock-> issue!!!
self.__read_lock.release()
def acquire_write_lock(self) -> bool:
return self.__write_lock.acquire()
def release_write_lock(self):
self.__write_lock.release()
I am having the same issue.
"LockNotOwnedError("Cannot release a lock"\nredis.exceptions.LockNotOwnedError: Cannot release a lock that's no longer owned"
redis = Redis.from_url(redis_url)
try:
with redis.lock(name, timeout=timeout, blocking_timeout=blocking_timeout) as redis_lock:
yield redis_lock
except RedisLockError as e:
logger.warning("Cannot acquire lock", name=name, timeout=timeout, blocking_timeout=blocking_timeout)
raise LockError(f"Cannot acquire lock {name}") from e
Can someone add some hints?
I have figured out how to release the not-owned redis lock by taking a look at the redis' python library source code. Below is the modified version of the multiread-single-write lock class.
# read_write_lock.py
from enum import Enum
from redis import StrictRedis, lock
# data in Redis cache:
# "read_counter_lock_name" : 0
# "read_lock_lock_name" -> read lock, protect "read_counter_lock_name"
# "write_lock_lock_name" -> write lock, protect write data section
class Prefix(Enum):
READ = 'read_lock_'
WRITE = 'write_lock_'
READ_COUNTER = 'read_counter_'
class RedisLockParams(Enum):
TIMEOUT = 60 # maximum life for the lock in seconds = 60 seconds
SLEEP_TIME = 0.1 # the amount of time to sleep in seconds per loop iteration
# in redis lock's acquire() - sleep then retry
BLOCKING = True # acquire() should block until the lock has been acquired
BLOCKING_TIMEOUT = None # maximum amount of time in seconds to spend trying
# to acquire the lock
class ReadWriteLock:
def __init__(self, lock_name: str, redis_host: str, redis_port: int, redis_key: str):
self.__read_lock_name = Prefix.READ.value + lock_name
self.__write_lock_name = Prefix.WRITE.value + lock_name
self.__read_counter_key = Prefix.READ_COUNTER.value + lock_name
self.__cache = StrictRedis(host = redis_host,
port = redis_port,
db=0, # up to 16 logical database
password = redis_key,
ssl=True)
# set the read counter to 0, if it does not exist.
self.__cache.setnx(self.__read_counter_key, 0)
# init the read lock
self.__read_lock = lock.Lock(self.__cache,
self.__read_lock_name,
RedisLockParams.TIMEOUT.value,
RedisLockParams.SLEEP_TIME.value,
RedisLockParams.BLOCKING.value,
RedisLockParams.BLOCKING_TIMEOUT.value)
# init the write lock
self.__write_lock = lock.Lock(self.__cache,
self.__write_lock_name,
RedisLockParams.TIMEOUT.value,
RedisLockParams.SLEEP_TIME.value,
RedisLockParams.BLOCKING.value,
RedisLockParams.BLOCKING_TIMEOUT.value)
def acquire_read_lock(self) -> bool:
write_lock_acquired = False
self.__read_lock.acquire()
read_counter = self.__cache.incr(self.__read_counter_key)
if (read_counter == 1):
write_lock_acquired = self.__write_lock.acquire() # acquire write lock
self.__read_lock.release()
return write_lock_acquired
def release_read_lock(self):
read_lock_acquired = self.__read_lock.acquire()
if read_lock_acquired:
read_counter = self.__cache.decr(self.__read_counter_key)
if read_counter == 0:
if self.__write_lock.owned():
self.__write_lock.release()
else: # if the lock was not owned, just take its token and override
write_lock_token = self.__cache.get(self.__write_lock_name)
self.__write_lock.local.token = write_lock_token
self.__write_lock.release()
self.__read_lock.release()
def acquire_write_lock(self) -> bool:
return self.__write_lock.acquire()
def release_write_lock(self) -> None:
self.__write_lock.release()
Using VxWorks 653 2.5.0.2 for P2020RDB-PC target, using BSP1.0/4
I have a very simple test application
void usrAppInit (void)
{
RETURN_CODE_TYPE errCode;
printf("\n I am alive!");
PROCESS_ATTRIBUTE_TYPE processAttributes;
PROCESS_ID_TYPE thandle;
processAttributes.BASE_PRIORITY = 10;
processAttributes.DEADLINE = SOFT;
processAttributes.ENTRY_POINT = (SYSTEM_ADDRESS_TYPE)task;
strncpy(processAttributes.NAME, "TASK", MAX_NAME_LENGTH);
processAttributes.PERIOD = INFINITE_TIME_VALUE;
processAttributes.STACK_SIZE = 1024;
processAttributes.TIME_CAPACITY = INFINITE_TIME_VALUE;
CREATE_PROCESS(&processAttributes, &thandle, &errCode);
if(errCode != NO_ERROR)
{
printf("Just had an error creating the task: %d", errCode);
}
else
{
START(thandle, &errCode);
if(errCode != NO_ERROR)
{
printf("Just had an error starting the task: %d", errCode);
}
}
SET_PARTITION_MODE (NORMAL, &errCode);
if (errCode != NO_ERROR){
printf("\nError changing partition mode: %d", errCode);
}
while(1);
}
void task()
{
printf("\nI am a process.");
while(1);
}
When the program gets to the START line, it reboots. If I comment out the START line, it executes till the end of the main and obviously does nothing. I have tried to increase partition memory and I am adding the APEX components to the makefile.
What could be causing this?
PS: system output
VxWorks 653 System Boot
Copyright (c) 1984-2016 Wind River Systems, Inc.
CPU: Freescale P2020E - Security Engine
Version: 2.5.0.2
BSP version: 1.0/4
Creation date: Apr 29 2020, 15:00:10
Press any key to stop auto-boot...
0
auto-booting...
boot device : mottsec
unit number : 0
processor number : 0
host name : felipe
file name : D:\Projects\WindRiver\helloWorld\boot.txt
inet on ethernet (e) : 192.168.1.172
host inet (h) : 192.168.1.75
gateway inet (g) : 192.168.1.1
user (u) : felipe
ftp password (pw) : pass
flags (f) : 0x0
target name (tn) : board
Attached TCP/IP interface to mottsec0.
Warning: netmask value is 0.
Attaching interface lo0...done
Loading D:\Projects\WindRiver\helloWorld\boot.txt
sm0=D:\Projects\WindRiver\helloWorld\configRecord.reloc
0x00001a00 + (0x000fe600)
sm1=D:\Projects\WindRiver\helloWorld\coreOS.sm
0x00050e08 + 0x00007130 + 0x00006084 + 0x00015cac
sm2=D:\Projects\WindRiver\helloWorld\vxSysLib.sm
0x00031078 + 0x00004b20 + 0x00000918 + 0x00001d90
sm3=D:\Projects\WindRiver\helloWorld\fsl_p2020_rdb_part1.sm
0x000027c8 + 0x000000d0 + 0x00000010 + 0x00000008
Starting at 0x100000...
After this, it returns to the beggining and repeats the process forever
So, in case anyone else drops here with similar issue, in my case there were two main things affecting the functionality.
First, in the usrAppInit() function, one cannot define the while(1) loop in the end of the function. On opposite to other ARINC-653 systems, where the partition main is the same as the user main, for VxWorks it does not seems to be the case. This way, after SET_PARTITION_MODE, nothing else can be defined.
Second, the stack size for the process was too small. That fit perfectly for different targets and ARINC RTOS (in-house OS, executing on ARMv7 target), but for VxWorks on the P2020 target it requires a bigger stack. In this case, I've used 4096.
Here is the complete example code, now functional, in case anyone needs this.
void usrAppInit (void)
{
RETURN_CODE_TYPE errCode;
printf("\n I am alive!");
PROCESS_ATTRIBUTE_TYPE processAttributes;
PROCESS_ID_TYPE thandle;
processAttributes.BASE_PRIORITY = 10;
processAttributes.DEADLINE = SOFT;
processAttributes.ENTRY_POINT = (SYSTEM_ADDRESS_TYPE)task;
strncpy(processAttributes.NAME, "TASK", MAX_NAME_LENGTH);
processAttributes.PERIOD = INFINITE_TIME_VALUE;
processAttributes.STACK_SIZE = 4096;
processAttributes.TIME_CAPACITY = INFINITE_TIME_VALUE;
CREATE_PROCESS(&processAttributes, &thandle, &errCode);
if(errCode != NO_ERROR)
{
printf("Just had an error creating the task: %d", errCode);
}
else
{
START(thandle, &errCode);
if(errCode != NO_ERROR)
{
printf("Just had an error starting the task: %d", errCode);
}
}
SET_PARTITION_MODE (NORMAL, &errCode);
if (errCode != NO_ERROR){
printf("\nError changing partition mode: %d", errCode);
}
}
void task()
{
printf("\nI am a process.");
while(1);
}
I am trying to publish in a transaction a message on 16 Kafka partitions on 7 brokers.
The flow is like this:
open transaction
write a message to 16 partitions
commit transaction
sleep 25 ms
repeat
Sometimes the transaction takes over 1 second to complete, with an average of 50 ms.
After enabling trace logging on producer's side, I noticed the following error:
TRACE internals.TransactionManager [kafka-producer-network-thread | producer-1] - [Producer clientId=producer-1, transactionalId=cma-2]
Received transactional response AddPartitionsToTxnResponse(errors={modelapp-ecb-0=CONCURRENT_TRANSACTIONS, modelapp-ecb-9=CONCURRENT_TRANSACTIONS, modelapp-ecb-10=CONCURRENT_TRANSACTIONS, modelapp-ecb-11=CONCURRENT_TRANSACTIONS, modelapp-ecb-12=CONCURRENT_TRANSACTIONS, modelapp-ecb-13=CONCURRENT_TRANSACTIONS, modelapp-ecb-14=CONCURRENT_TRANSACTIONS, modelapp-ecb-15=CONCURRENT_TRANSACTIONS, modelapp-ecb-1=CONCURRENT_TRANSACTIONS, modelapp-ecb-2=CONCURRENT_TRANSACTIONS, modelapp-ecb-3=CONCURRENT_TRANSACTIONS, modelapp-ecb-4=CONCURRENT_TRANSACTIONS, modelapp-ecb-5=CONCURRENT_TRANSACTIONS, modelapp-ecb-6=CONCURRENT_TRANSACTIONS, modelapp-ecb-=CONCURRENT_TRANSACTIONS, modelapp-ecb-8=CONCURRENT_TRANSACTIONS}, throttleTimeMs=0)
for request (type=AddPartitionsToTxnRequest, transactionalId=cma-2, producerId=59003, producerEpoch=0, partitions=[modelapp-ecb-0, modelapp-ecb-9, modelapp-ecb-10, modelapp-ecb-11, modelapp-ecb-12, modelapp-ecb-13, modelapp-ecb-14, modelapp-ecb-15, modelapp-ecb-1, modelapp-ecb-2, modelapp-ecb-3, modelapp-ecb-4, modelapp-ecb-5, modelapp-ecb-6, modelapp-ecb-7, modelapp-ecb-8])
The Kafka producer retries sending AddPartitionsToTxnRequest(s) several times until it succeeds, but this leads to delays.
The code looks like this:
Properties producerProperties = PropertiesUtil.readPropertyFile(_producerPropertiesFile);
_producer = new KafkaProducer<>(producerProperties);
_producer.initTransactions();
_producerService = Executors.newSingleThreadExecutor(new NamedThreadFactory(getClass().getSimpleName()));
_producerService.submit(() -> {
while (!Thread.currentThread().isInterrupted()) {
try {
_producer.beginTransaction();
for (int partition = 0; partition < _numberOfPartitions; partition++)
_producer.send(new ProducerRecord<>(_producerTopic, partition, KafkaRecordKeyFormatter.formatControlMessageKey(_messageNumber, token), EMPTY_BYTE_ARRAY));
_producer.commitTransaction();
_messageNumber++;
Thread.sleep(_timeBetweenProducedMessagesInMillis);
} catch (ProducerFencedException | OutOfOrderSequenceException | AuthorizationException | UnsupportedVersionException e) {
closeProducer();
break;
} catch (KafkaException e) {
_producer.abortTransaction();
} catch (InterruptedException e) {...}
}
});
Looking to broker's code, it seems there are 2 cases when this error is thrown, but I cannot tell why I get there
object TransactionCoordinator {
...
def handleAddPartitionsToTransaction(...): Unit = {
...
if (txnMetadata.pendingTransitionInProgress) {
// return a retriable exception to let the client backoff and retry
Left(Errors.CONCURRENT_TRANSACTIONS)
} else if (txnMetadata.state == PrepareCommit || txnMetadata.state == PrepareAbort) {
Left(Errors.CONCURRENT_TRANSACTIONS)
}
...
}
...
}
Thanks in advance for help!
Later edit:
Enabling trace logging on broker we were able to see that broker sends to the producer END_TXN response before transaction reaches state CompleteCommit. The producer is able to start a new transaction, which is rejected by the broker while it is still in the transition PrepareCommit -> CompleteCommit.
I am uisng akka-persistence with mongodb using this https://github.com/ironfish/akka-persistence-mongo/ mongodb plugins. when i am running my code, i am getting following error:
[ERROR] [11/19/2016 16:47:29.355] [transaction-system-akka.actor.default-dispatcher-5] [akka://transaction-system/user/$a] Persistence failure when replaying events for persistenceId [balanceTransactions]. Last known sequence number [0] (akka.persistence.RecoveryTimedOut)
I am not getting, what is the meaning of this error and how can i resolve this error. Following is my reference.conf file:
akka {
persistence {
journal {
plugin = "casbah-snapshot"
}
snapshot-store {
plugin = "casbah-snapshot"
}
}
}
casbah-snapshot {
mongo-url = "mongodb://localhost:27017/user.events"
woption = 1
wtimeout = 10000
load-attempts = 5
}
After changing my reference.conf file, my example works successfully. Below is valid reference.conf file.
akka {
stdout-loglevel = off // defaults to WARNING can be disabled with off. The stdout-loglevel is only in effect during system startup and shutdown
log-dead-letters-during-shutdown = off
loglevel = info
log-dead-letters = off
log-config-on-start = off // Log the complete configuration at INFO level when the actor system is started
loggers = ["akka.event.slf4j.Slf4jLogger"]
logging-filter = "akka.event.slf4j.Slf4jLoggingFilter"
persistence {
journal {
plugin = "casbah-journal"
}
}
}
casbah-journal {
mongo-url = "mongodb://localhost:27017/transaction.events"
woption = 1
wtimeout = 10000
load-attempts = 5
}
I have successfully created a standalone Scalatra / Jetty server, using the official instructions from Scalatra ( http://www.scalatra.org/2.3/guides/deployment/standalone.html )
I am debugging it under Ensime, and would like to limit the number of threads handling messages to a single one - so that single-stepping through the servlet methods will be easier.
I used this code to achieve it:
package ...
import org.eclipse.jetty.server.Server
import org.eclipse.jetty.servlet.{DefaultServlet, ServletContextHandler}
import org.eclipse.jetty.webapp.WebAppContext
import org.scalatra.servlet.ScalatraListener
import org.eclipse.jetty.util.thread.QueuedThreadPool
import org.eclipse.jetty.server.ServerConnector
object JettyLauncher {
def main(args: Array[String]) {
val port =
if (System.getenv("PORT") != null)
System.getenv("PORT").toInt
else
4080
// DEBUGGING MODE BEGINS
val threadPool = new QueuedThreadPool()
threadPool.setMaxThreads(8)
val server = new Server(threadPool)
val connector = new ServerConnector(server)
connector.setPort(port)
server.setConnectors(Array(connector))
// DEBUGGING MODE ENDS
val context = new WebAppContext()
context setContextPath "/"
context.setResourceBase("src/main/webapp")
context.addEventListener(new ScalatraListener)
context.addServlet(classOf[DefaultServlet], "/")
server.setHandler(context)
server.start
server.join
}
}
It works fine - except for one minor detail...
I can't tell Jetty to use 1 thread - the minimum value is 8!
If I do, this is what happens:
$ sbt assembly
...
$ java -jar ./target/scala-2.11/CurrentVersions-assembly-0.1.0-SNAPSHOT.jar
18:13:27.059 [main] INFO org.eclipse.jetty.util.log - Logging initialized #41ms
18:13:27.206 [main] INFO org.eclipse.jetty.server.Server - jetty-9.1.z-SNAPSHOT
18:13:27.220 [main] WARN o.e.j.u.component.AbstractLifeCycle - FAILED org.eclipse.jetty.server.Server#1ac539f: java.lang.IllegalStateException: Insufficient max threads in ThreadPool: max=1 < needed=8
java.lang.IllegalStateException: Insufficient max threads in ThreadPool: max=1 < needed=8
...which is why you see setMaxThreads(8) instead of setMaxThreads(1) in my code above.
Any ideas why this happens?
The reason is that the size of the threadpool also depends on th enumber of connectors you've got defined. If you look at the source code of the jetty server you'll see this:
// check size of thread pool
SizedThreadPool pool = getBean(SizedThreadPool.class);
int max=pool==null?-1:pool.getMaxThreads();
int selectors=0;
int acceptors=0;
if (mex.size()==0)
{
for (Connector connector : _connectors)
{
if (connector instanceof AbstractConnector)
acceptors+=((AbstractConnector)connector).getAcceptors();
if (connector instanceof ServerConnector)
selectors+=((ServerConnector)connector).getSelectorManager().getSelectorCount();
}
}
int needed=1+selectors+acceptors;
if (max>0 && needed>max)
throw new IllegalStateException(String.format("Insufficient threads: max=%d < needed(acceptors=%d + selectors=%d + request=1)",max,acceptors,selectors));
So the minimum with a single serverconnector is 2. It looks like you've got a couple of other default connectors or selectors running.