Use postgresql as flink sink, And the connection can't serialize PGConnection by kyro? - postgresql

Question

I write a postgresql sink :
class PGTwoPhaseCommitSinkFunction
extends TwoPhaseCommmitSinkFunction[Row,PgConnection,Void](new
KryoSerializer[PgConnection](classOf[PgConnection]),new ExecutionConfig),
VoidSerializer.INSTANCE)
but, when i use it, find the PgConnection Can't serialize,
and the exception is :
com.esotericsoftware.kryo.KryoException: Error constructing instance of class: sun.nio.cs.UTF_8"
how can i handle it ? thanks

Answer 1

The transaction object specified as a second generic parameter should be first of all Serializable and I don't think it's right to use the PgConnection for it. Instead, it should be some custom lightweight transaction state object, holding transaction metadata like id - for example, below is the example of FlinkKafkaProducer transaction state:
/**
* State for handling transactions.
*/
#VisibleForTesting
#Internal
static class KafkaTransactionState {
private final transient FlinkKafkaInternalProducer<byte[], byte[]> producer;
#Nullable
final String transactionalId;
final long producerId;
final short epoch;
KafkaTransactionState(String transactionalId, FlinkKafkaInternalProducer<byte[], byte[]> producer) {
this(transactionalId, producer.getProducerId(), producer.getEpoch(), producer);
}
KafkaTransactionState(FlinkKafkaInternalProducer<byte[], byte[]> producer) {
this(null, -1, (short) -1, producer);
}
KafkaTransactionState(
#Nullable String transactionalId,
long producerId,
short epoch,
FlinkKafkaInternalProducer<byte[], byte[]> producer) {
this.transactionalId = transactionalId;
this.producerId = producerId;
this.epoch = epoch;
this.producer = producer;
}
boolean isTransactional() {
return transactionalId != null;
}
...
However, as it's good pointed in this thread, writing and maintaining a two-phase commit sink is a tricky and very difficult task and it's better to use the table API with the JDBC connector plus postgres driver.
Here is an example of the pipeline writing data into the postgres.