Hyperledger Fabric - Channel hasn't started yet - apache-kafka

Question

After an unfortunate docker outage, one of our test channels has stopped working properly. This is the output of a previously working “peer chaincode invoke” command:
Error: error sending transaction for invoke: got unexpected status: SERVICE_UNAVAILABLE -- will not enqueue, consenter for this channel hasn't started yet - proposal response: version:1 response:<status:200 payload:"be85bda14845a33cd07db9825d2e473dc65902e6986fdfccea30d8c32385f758" > payload:"\n \364\242+\t\222\216\361\020\024}d7\203\277WY04\233\225vA\376u\330r\2045\312\206\304\333\022\211\001\n3\022\024\n\004lscc\022\014\n\n\n\004strs\022\002\010\004\022\033\n\004strs\022\023\032\021\n\tkeepalive\032\004ping\032E\010\310\001\032#be85bda14845a33cd07db9825d2e473dc65902e6986fdfccea30d8c32385f758\"\013\022\004strs\032\0031.0" endorsement:<endorser:"\n\013BackboneMSP\022\203\007-----BEGIN CERTIFICATE----- etc. more output removed from here
I find this in the orderer's log:
2019-07-29 14:46:50.930 UTC [orderer/consensus/kafka] try -> DEBU 3c10 [channel: steel] Connecting to the Kafka cluster
2019-07-29 14:46:50.931 UTC [orderer/consensus/kafka] try -> DEBU 3c11 [channel: steel] Need to retry because process failed = kafka server: The requested offset is outside the range of offsets maintained by the server for the given topic/partition.
2019-07-29 14:46:56.967 UTC [common/deliver] Handle -> WARN 3c12 Error reading from 10.0.0.4:32800: rpc error: code = Canceled desc = context canceled
2019-07-29 14:46:56.967 UTC [orderer/common/server] func1 -> DEBU 3c13 Closing Deliver stream
2019-07-29 14:46:56.972 UTC [orderer/common/server] Deliver -> DEBU 3c14 Starting new Deliver handler
2019-07-29 14:46:56.972 UTC [common/deliver] Handle -> DEBU 3c15 Starting new deliver loop for 10.0.0.4:32802
2019-07-29 14:46:56.973 UTC [common/deliver] Handle -> DEBU 3c16 Attempting to read seek info message from 10.0.0.4:32802
2019-07-29 14:46:56.973 UTC [common/deliver] deliverBlocks -> WARN 3c17 [channel: steel] Rejecting deliver request for 10.0.0.4:32802 because of consenter error
2019-07-29 14:46:56.973 UTC [common/deliver] Handle -> DEBU 3c18 Waiting for new SeekInfo from 10.0.0.4:32802
2019-07-29 14:46:56.973 UTC [common/deliver] Handle -> DEBU 3c19 Attempting to read seek info message from 10.0.0.4:32802
2019-07-29 14:46:56.995 UTC [common/deliver] Handle -> WARN 3c1a Error reading from 10.0.0.23:49844: rpc error: code = Canceled desc = context canceled
2019-07-29 14:46:56.995 UTC [orderer/common/server] func1 -> DEBU 3c1b Closing Deliver stream
And this is from the endorser peer’s log:
2019-07-29 15:14:17.829 UTC [ConnProducer] DisableEndpoint -> WARN 3d6 Only 1 endpoint remained, will not black-list it
2019-07-29 15:14:17.834 UTC [blocksProvider] DeliverBlocks -> WARN 3d7 [steel] Got error &{SERVICE_UNAVAILABLE}
2019-07-29 15:14:27.839 UTC [blocksProvider] DeliverBlocks -> WARN 3d8 [steel] Got error &{SERVICE_UNAVAILABLE}
I use these docker images:
hyperledger/fabric-kafka:0.4.10
hyperledger/fabric-orderer:1.2.0
hyperledger/fabric-peer:1.2.0
Based on the above, I assume that the consistency between the orderer and the corresponding kafka topic is broken. It also doesn't help if I redirect requests to another orderer or force to change the kafka topic leader. Is it correct that if KAFKA_LOG_RETENTION_MS=-1 had been set, this error would probably have been prevented?
After reviewing the archives, I found that it is not possible to fix this error. As I see it, I can't shutdown only one channel, and I even have to stop all the peers subscribed to the channel if I don't want continuous error messages in the orderer logs. What is the best practice in cases like mine?
Regards;
Sandor

Answer 1

consenter error
which means before connection made b/w kafka & orderer you are trying
to do some operations.
Which means there is an error present in between kafka and orderer
Note: Probably you might have set up a connection which is not stable
Try to check the logs of orderer it should have a message posted
successfully.
whenever Kafka, orderer try to connect, orderer will post a message if it
successfully posted to a topic which means you have configured correctly
Make sure connection b/w kafka and orderer are configured correctly
2019-07-29 14:46:50.931 UTC [orderer/consensus/kafka] try -> DEBU 3c11 [channel: steel] Need to retry because process failed = kafka server: The requested offset is outside the range of offsets maintained by the server for the given topic/partition.
By above clue, it is complete with Kafka nothing todo with the orderer
try check this