Class not found when trying to return List from consumeEvent in Quarkus - vert.x

When I try to return a Uni with a typed java.util.List in Quarkus in dev mode, i get a ClassNotFound exception. I have read about Quarkus using different class loaders in different profiles, but I don't see that I do anything fancy.
Here's the sender
#Query("offers")
public Uni<List<OfferResponse>> getOffers(#PathParam("category") Integer categoryId) {
OfferRequest event = new OfferRequest();
event.setCategoryId(categoryId);
Uni<List<OfferResponse>> offers = bus.<List<OfferResponse>>request(OfferRequest.ADDRESS, event).onItem().transform(Message::body);
return offers;
}
And here's the consumer
#ConsumeEvent(OfferRequest.ADDRESS)
public Uni<List<OfferResponse>> onOfferQuery(OfferRequest request) {
List<KelkooOffer> offers = getOffers(request.getCategoryId());
List<OfferResponse> responses = new ArrayList<OfferResponse>();
for (KelkooOffer offer : offers) {
responses.add(offer.getEventResponse());
}
return Uni.createFrom().item(responses);
}
The bean I'm trying to return is just a POJO
and the error message
2021-08-18 11:11:16,186 ERROR [io.qua.run.Application] (Quarkus Main Thread) Failed to start application (with profile dev): java.lang.ClassNotFoundException: java.util.List<se.bryderi.events.OfferResponse>
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass(QuarkusClassLoader.java:414)
at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass(QuarkusClassLoader.java:405)
at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass(QuarkusClassLoader.java:414)
at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass(QuarkusClassLoader.java:405)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:398)
at io.quarkus.deployment.steps.VertxProcessor$build609260703.deploy_0(VertxProcessor$build609260703.zig:142)
at io.quarkus.deployment.steps.VertxProcessor$build609260703.deploy(VertxProcessor$build609260703.zig:40)
at io.quarkus.runner.ApplicationImpl.doStart(ApplicationImpl.zig:784)
at io.quarkus.runtime.Application.start(Application.java:101)
at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:101)
at io.quarkus.runtime.Quarkus.run(Quarkus.java:66)
at io.quarkus.runtime.Quarkus.run(Quarkus.java:42)
at io.quarkus.runtime.Quarkus.run(Quarkus.java:119)
at io.quarkus.runner.GeneratedMain.main(GeneratedMain.zig:29)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at io.quarkus.runner.bootstrap.StartupActionImpl$1.run(StartupActionImpl.java:98)
at java.base/java.lang.Thread.run(Thread.java:829)
I get the same result if I run the dev profile or if I run the packaged fast-jar.
Happy for any hint that will point me in the right direction

There is a limitation of the Vert.x EventBus which prevents to encode/decode lists or sets directly.
You could create a wrapper for your OfferResponse list:
import java.util.AbstractList;
import java.util.List;
public class OfferResponseList extends AbstractList<OfferResponse> {
private List<OfferResponse> wrapped;
private OfferResponseList(List<OfferResponse> wrapped) {
this.wrapped = wrapped;
}
public static OfferResponseList wrap(List<OfferResponse> list) {
return new OfferResponseList(list);
}
#Override
public OfferResponse get(int index) {
return wrapped.get(index);
}
#Override
public int size() {
return wrapped.size();
}
}
Then transform your consumer to:
#ConsumeEvent(OfferRequest.ADDRESS)
public Uni<OfferResponseList> onOfferQuery(OfferRequest request) {
List<KelkooOffer> offers = getOffers(request.getCategoryId());
List<OfferResponse> responses = new ArrayList<OfferResponse>();
for (KelkooOffer offer : offers) {
responses.add(offer.getEventResponse());
}
// Notice the call to OfferResponseList.wrap here
return Uni.createFrom().item(OfferResponseList.wrap(responses));
}
That's it. Quarkus will register a codec for OfferResponseList automatically.
On the client side, you do not need to change your code:
#Query("offers")
public Uni<List<OfferResponse>> getOffers(#PathParam("category") Integer categoryId) {
OfferRequest event = new OfferRequest();
event.setCategoryId(categoryId);
// You can use Uni<List<OfferResponse>> or Uni<OfferResponseList>
Uni<List<OfferResponse>> offers = bus.<List<OfferResponse>>request(OfferRequest.ADDRESS, event).onItem().transform(Message::body);
return offers;
}

Related

Task not serializable : Spark

My spark job is throwing Task not serializable at runtime. Can anyone tell me if what i am doing wrong here?
#Component("loader")
#Slf4j
public class LoaderSpark implements SparkJob {
private static final int MAX_VERSIONS = 1;
private final AppProperties props;
public LoaderSpark(
final AppProperties props
) {
this.props = props;
}
#Override
public void run(SparkSession spark, final String... args) throws IOException {
HBaseUtil hBaseUtil = new HBaseUtil(props);
byte[][] prefixes = new byte[][]{toBytes("document"),
toBytes("dataSource"),
toBytes("hold:")};
Filter filter = new MultipleColumnPrefixFilter(prefixes);
Scan scan = new Scan();
scan.addFamily(toBytes("data"));
scan.setCaching(100000);
scan.setMaxVersions(MAX_VERSIONS);
scan.setFilter(filter);
JavaRDD<TestMethod> mapFileJavaRDD
= hBaseUtil.createScanRdd(spark, "TestTable", scan).mapPartitions(tuple -> {
return StreamUtils.asStream(tuple)
.map(this::extractResult)
.filter(Objects::nonNull)
.iterator();
});
Dataset<TestMethod> testDataset = spark.createDataset(mapFileJavaRDD.rdd(), bean(TestMethod.class));
testDataset.limit(100);
}
private TestMethod extractResult(Tuple2<ImmutableBytesWritable, Result> resultTuple) {
TestMethod.TestMethodBuilder testBuilder = TestMethod.builder();
Result result;
result = resultTuple._2();
CdfParser cdfParser = new CdfParser();
List<String> holdingId = new ArrayList<>();
testBuilder.dataSource(Bytes.toString(result.getValue(Bytes.toBytes("data"),
Bytes.toBytes("dataSource"))));
testBuilder.document(cdfParser.fromXml(result.getValue(Bytes.toBytes("data"),
Bytes.toBytes("document"))));
NavigableMap<byte[], byte[]> familyMap = result.getFamilyMap(Bytes.toBytes("data"));
for (byte[] bQunitifer : familyMap.keySet()) {
if (Bytes.toString(bQunitifer).contains("hold:")) {
LOG.info(Bytes.toString(bQunitifer));
holdingId.add(Bytes.toString(bQunitifer));
}
}
testBuilder.holding(holdingId);
return testBuilder.build();
}
}
HERE is the stacktrace:
2020-04-29 12:48:59,837 INFO [Thread-4]o.a.s.d.y.ApplicationMaster: Unregistering ApplicationMaster with FAILED (diag message: User class threw exception: java.lang.IllegalStateException: Failed to execute CommandLineRunner
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:787)
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:768)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:322)
at org.oclc.googlelinks.spark.SpringSparkJob.main(SpringSparkJob.java:56)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:694)
Caused by: org.apache.spark.SparkException: Task not serializable
at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:403)
at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:393)
at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:162)
at org.apache.spark.SparkContext.clean(SparkContext.scala:2326)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1.apply(RDD.scala:798)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1.apply(RDD.scala:797)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
at org.apache.spark.rdd.RDD.mapPartitions(RDD.scala:797)
at org.apache.spark.api.java.JavaRDDLike$class.mapPartitions(JavaRDDLike.scala:155)
at org.apache.spark.api.java.AbstractJavaRDDLike.mapPartitions(JavaRDDLike.scala:45)
at org.oclc.googlelinks.spark.job.LoaderSpark.run(LoaderSpark.java:79)
at org.oclc.googlelinks.spark.SpringSparkJob.run(SpringSparkJob.java:79)
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:784)
... 8 more
try adding getter and setters for props
public void setProps(AppProperties props) {
this.props = props;
}
public AppProperties getProps() {
return props;
}
Just make the function extractResult static . In order to call a static method, you don’t need to serialize the class, you need the declaring class to be reachable by the classloader (and it is the case, as the jar archives can be shared among driver and workers).
Thanks to https://www.nicolaferraro.me/2016/02/22/using-non-serializable-objects-in-apache-spark/

RabbitTransactionManager cannot commit transaction at ChainedTransactionManager

I'm trying to use one transaction manager for Rabbit and Kafka. First I'm getting a message from a Rabbit callback, then sending it to Kafka topic. But I always get an exception indicating that Rabbit cannot finish the transaction correctly:
2019-11-14 16:02:46.572 ERROR 15640 --- [cTaskExecutor-1] o.s.a.r.c.CachingConnectionFactory : Could not configure the channel to receive publisher confirms java.io.IOException: null
at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:126)
at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:122)
at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:144)
at com.rabbitmq.client.impl.ChannelN.confirmSelect(ChannelN.java:1552)
at com.rabbitmq.client.impl.ChannelN.confirmSelect(ChannelN.java:52)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.doCreateBareChannel(CachingConnectionFactory.java:602)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createBareChannel(CachingConnectionFactory.java:582)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.access$600(CachingConnectionFactory.java:99)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$CachedChannelInvocationHandler.invoke(CachingConnectionFactory.java:1053)
at com.sun.proxy.$Proxy124.txCommit(Unknown Source)
at org.springframework.amqp.rabbit.connection.RabbitResourceHolder.commitAll(RabbitResourceHolder.java:164)
at org.springframework.amqp.rabbit.transaction.RabbitTransactionManager.doCommit(RabbitTransactionManager.java:187)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:746)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:714)
at org.springframework.data.transaction.MultiTransactionStatus.commit(MultiTransactionStatus.java:74)
at org.springframework.data.transaction.ChainedTransactionManager.commit(ChainedTransactionManager.java:150)
at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:532)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:304)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)
at com.listener.ExecutionCallbackListener$$EnhancerBySpringCGLIB$$9b575a95.receiveCallback(<generated>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:181)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:114)
at org.springframework.amqp.rabbit.listener.adapter.HandlerAdapter.invoke(HandlerAdapter.java:51)
at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:188)
at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.onMessage(MessagingMessageListenerAdapter.java:126)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:1445)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.actualInvokeListener(AbstractMessageListenerContainer.java:1368)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:1355)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:1334)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:817)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:801)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$700(SimpleMessageListenerContainer.java:77)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1042)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED - cannot switch from tx to confirm mode, class-id=85, method-id=10)
at com.rabbitmq.utility.ValueOrException.getValue(ValueOrException.java:66)
at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:36)
at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:494)
at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:288)
at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:138)
... 37 common frames omitted
Here is the method where problem occurs in Rabbit listener:
#RabbitListener(queues = ["\${queue}"])
#Transactional("chainedTransactionManager")
fun receiveCallback(message: Message<List<CallbackMessage>>) {
traceMessage(message)
val callbacks = message.payload
callbacks.forEach { callback ->
kafkaService.sendAfterCallBack(Object())
}
}
And method in KafkaService:
#Transactional("chainedTransactionManager")
fun sendAfterCallBack(object: Object) {
convertAndSend(kafkaServiceProperties.topics.name, object)
}
Here is TransactionManager configuration:
#Configuration
class TransactionManagerConfiguration {
#Bean
fun chainedTransactionManager(
rabbitTransactionManager: RabbitTransactionManager,
kafkaTransactionManager: KafkaTransactionManager<*, *>
): ChainedTransactionManager {
return ChainedTransactionManager(kafkaTransactionManager, rabbitTransactionManager)
}
}
Rabbit configuration:
#Configuration
#EnableRabbit
#Import(RabbitAutoCreationConfiguration::class)
class RabbitConfiguration(
private val integrationProperties: IntegrationProperties,
private var clientProperties: RabbitClientProperties,
private val jacksonObjectMapper: ObjectMapper
) : RabbitListenerConfigurer {
#Bean
fun rabbitListenerContainerFactory(connectionFactory: ConnectionFactory): SimpleRabbitListenerContainerFactory {
val factory = SimpleRabbitListenerContainerFactory()
factory.setConnectionFactory(connectionFactory)
factory.setErrorHandler { t -> throw AmqpRejectAndDontRequeueException(t) }
return factory
}
#Bean
fun messageConverter(): MessageConverter {
val messageConverter = MappingJackson2MessageConverter()
messageConverter.objectMapper = jacksonObjectMapper
return messageConverter
}
#Bean
fun messageHandlerFactory(): MessageHandlerMethodFactory {
val factory = DefaultMessageHandlerMethodFactory()
factory.setMessageConverter(messageConverter())
return factory
}
#Bean
#ConditionalOnBean(CachingConnectionFactory::class)
fun rabbitConnectionFactoryCustomizer(factory: CachingConnectionFactory): SmartInitializingSingleton {
return SmartInitializingSingleton {
factory.rabbitConnectionFactory.clientProperties.apply {
clientProperties.copyright?.let { put("copyright", it) }
put("os", System.getProperty("os.name"))
put("host", InetAddress.getLocalHost().hostName)
clientProperties.platform?.let { put("platform", it) }
clientProperties.product?.let { put("product", it) }
clientProperties.service?.let { put("service", it) }
}
}
}
override fun configureRabbitListeners(registrar: RabbitListenerEndpointRegistrar?) {
registrar!!.messageHandlerMethodFactory = messageHandlerFactory()
}
#Bean
fun rabbitTemplate(
connectionFactory: ConnectionFactory,
jsonObjectMapper: ObjectMapper
): RabbitTemplate {
val rabbitTemplate = RabbitTemplate(connectionFactory)
val retryTemplate = RetryTemplate()
retryTemplate.setRetryPolicy(SimpleRetryPolicy(integrationProperties.callbackRetry))
rabbitTemplate.setRetryTemplate(retryTemplate)
rabbitTemplate.isChannelTransacted = true
return rabbitTemplate
}
#Bean
fun rabbitTransactionManager(connectionFactory: ConnectionFactory): RabbitTransactionManager {
val rtm = RabbitTransactionManager(connectionFactory)
rtm.transactionSynchronization = AbstractPlatformTransactionManager.SYNCHRONIZATION_ON_ACTUAL_TRANSACTION
return rtm
}
}
Kafka configuration:
#Configuration
#EnableKafka
class KafkaConfiguration(
#Qualifier("kafkaExchangeMessageConverter")
private val messageConverter: MessagingMessageConverter
) {
#Bean
fun kafkaListenerContainerFactory(
configurer: ConcurrentKafkaListenerContainerFactoryConfigurer,
consumerFactory: ConsumerFactory<Any, Any>
): ConcurrentKafkaListenerContainerFactory<Any, Any> {
val factory = ConcurrentKafkaListenerContainerFactory<Any, Any>()
factory.setMessageConverter(messageConverter)
configurer.configure(factory, consumerFactory)
return factory
}
#Bean
fun adminClient(kafkaAdmin: KafkaAdmin): AdminClient = AdminClient.create(kafkaAdmin.config)
#Bean
fun kafkaTransactionManager(
producerFactory: ProducerFactory<*, *>
): KafkaTransactionManager<*, *> {
val ktm = KafkaTransactionManager(producerFactory)
ktm.transactionSynchronization = AbstractPlatformTransactionManager.SYNCHRONIZATION_ON_ACTUAL_TRANSACTION
return ktm
}
}
Did I miss something in RabbitConfiguration or problem is in something else?
reply-text=PRECONDITION_FAILED - cannot switch from tx to confirm mode,
You cannot use publisher confirms and transactions on the same channel. Turn off publisher confirms.
Also, it's better to inject the chained transaction manager into the listener container rather than using #Transactional.

Apache beam IOException in decoder

I have a simple pipeline that reads from Kafka by KafkaIO reader and transforms next into pipeline. In the end, it writes down to GCP in avro format. So when I run the pipeline in DataFlow it works perfectly but when the runner is DirectRunner it reads all data from topics and throws the exception.
java.lang.IllegalArgumentException: Forbidden IOException when reading from InputStream
at org.apache.beam.sdk.util.CoderUtils.decodeFromSafeStream(CoderUtils.java:118)
at org.apache.beam.sdk.util.CoderUtils.decodeFromByteArray(CoderUtils.java:98)
at org.apache.beam.sdk.util.CoderUtils.decodeFromByteArray(CoderUtils.java:92)
at org.apache.beam.sdk.util.CoderUtils.clone(CoderUtils.java:141)
at org.apache.beam.runners.direct.CloningBundleFactory$CloningBundle.add(CloningBundleFactory.java:84)
at org.apache.beam.runners.direct.GroupAlsoByWindowEvaluatorFactory$OutputWindowedValueToBundle.outputWindowedValue(GroupAlsoByWindowEvaluatorFactory.java:251)
at org.apache.beam.runners.direct.GroupAlsoByWindowEvaluatorFactory$OutputWindowedValueToBundle.outputWindowedValue(GroupAlsoByWindowEvaluatorFactory.java:237)
at org.apache.beam.repackaged.direct_java.runners.core.ReduceFnRunner.lambda$onTrigger$1(ReduceFnRunner.java:1057)
at org.apache.beam.repackaged.direct_java.runners.core.ReduceFnContextFactory$OnTriggerContextImpl.output(ReduceFnContextFactory.java:438)
at org.apache.beam.repackaged.direct_java.runners.core.SystemReduceFn.onTrigger(SystemReduceFn.java:125)
at org.apache.beam.repackaged.direct_java.runners.core.ReduceFnRunner.onTrigger(ReduceFnRunner.java:1060)
at org.apache.beam.repackaged.direct_java.runners.core.ReduceFnRunner.onTimers(ReduceFnRunner.java:768)
at org.apache.beam.runners.direct.GroupAlsoByWindowEvaluatorFactory$GroupAlsoByWindowEvaluator.processElement(GroupAlsoByWindowEvaluatorFactory.java:185)
at org.apache.beam.runners.direct.DirectTransformExecutor.processElements(DirectTransformExecutor.java:160)
at org.apache.beam.runners.direct.DirectTransformExecutor.run(DirectTransformExecutor.java:124)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.EOFException
at org.apache.beam.sdk.util.VarInt.decodeLong(VarInt.java:73)
at org.apache.beam.sdk.coders.IterableLikeCoder.decode(IterableLikeCoder.java:136)
at org.apache.beam.sdk.coders.IterableLikeCoder.decode(IterableLikeCoder.java:60)
at org.apache.beam.sdk.coders.Coder.decode(Coder.java:159)
at org.apache.beam.sdk.coders.KvCoder.decode(KvCoder.java:82)
at org.apache.beam.sdk.coders.KvCoder.decode(KvCoder.java:36)
at org.apache.beam.sdk.util.CoderUtils.decodeFromSafeStream(CoderUtils.java:115)
... 19 more
I use custom serializator and deserializator for reading avro and getting paylod.
Kafka Reader
private PTransform<PBegin, PCollection<KV<String, AvroGenericRecord>>> createKafkaRead(Map<String, Object> configUpdates) {
return KafkaIO.<String, AvroGenericRecord>read()
.withBootstrapServers(bootstrapServers)
.withConsumerConfigUpdates(configUpdates)
.withTopics(inputTopics)
.withKeyDeserializer(StringDeserializer.class)
.withValueDeserializerAndCoder(BeamKafkaAvroGenericDeserializer.class, AvroGenericCoder.of(serDeConfig()))
.withMaxNumRecords(maxNumRecords)
.commitOffsetsInFinalize()
.withoutMetadata();
}
AvroGenericCoder
public class AvroGenericCoder extends CustomCoder<AvroGenericRecord> {
private final Map<String, Object> config;
private transient BeamKafkaAvroGenericDeserializer deserializer;
private transient BeamKafkaAvroGenericSerializer serializer;
public static AvroGenericCoder of(Map<String, Object> config) {
return new AvroGenericCoder(config);
}
protected AvroGenericCoder(Map<String, Object> config) {
this.config = config;
}
private BeamKafkaAvroGenericDeserializer getDeserializer() {
if (deserializer == null) {
BeamKafkaAvroGenericDeserializer d = new BeamKafkaAvroGenericDeserializer();
d.configure(config, false);
deserializer = d;
}
return deserializer;
}
private BeamKafkaAvroGenericSerializer getSerializer() {
if (serializer == null) {
serializer = new BeamKafkaAvroGenericSerializer();
}
return serializer;
}
#Override
public void encode(AvroGenericRecord record, OutputStream outStream) {
getSerializer().serialize(record, outStream);
}
#Override
public AvroGenericRecord decode(InputStream inStream) {
try {
return getDeserializer().deserialize(null, IOUtils.toByteArray(inStream));
} catch (IOException e) {
throw new RuntimeException("Error translating into bytes ", e);
}
}
#Override
public void verifyDeterministic() {
}
#Override
public Object structuralValue(AvroGenericRecord value) {
return super.structuralValue(value);
}
#Override
public int hashCode() {
return HashCodeBuilder.reflectionHashCode(this);
}
#Override
public boolean equals(Object obj) {
return EqualsBuilder.reflectionEquals(this, obj);
}
}
This is main pipeline
PCollection<AvroGenericRecord> records = p.apply(readKafkaTr)
.apply(Window.<AvroGenericRecord>into(FixedWindows.of(Duration.standardMinutes(options.getWindowInMinutes())))
.triggering(AfterWatermark.pastEndOfWindow()
.withEarlyFirings(AfterProcessingTime.pastFirstElementInPane()
.plusDelayOf(Duration.standardMinutes(options.getWindowInMinutes())))
.withLateFirings(AfterPane.elementCountAtLeast(options.getElementsCountToWaitAfterWatermark())))
.withAllowedLateness(Duration.standardDays(options.getAfterWatermarkInDays()))
.discardingFiredPanes()
);
records.apply(Filter.by((ProcessFunction<AvroGenericRecord, Boolean>) Objects::nonNull))
.apply(new WriteAvroFilesTr(options.getBasePath(), options.getNumberOfShards()));
Yes, I think #RyanSkraba is right - DirectRunner does many things that not all other runners do (because initial goal of DirectRunner was to be used for testing, so it performs many additional checks comparing to other runners).
Btw, why would not use Beam AvroCoder in this case? Simple example how to use it with KafkaIO:
https://github.com/aromanenko-dev/beam-issues/blob/master/kafka-io/src/main/java/KafkaAvro.java

Use JMockit test the RESTful API,RestTemplate,error missing invocation

I am learning to use the JMockit to test my Restful API.The missing invocation error is abusing me.
The Restful API:
#RestController
public class AgentInfoRest {
#Autowired
AgentInfoRepository agentInfoRepository;
private Logger logger = LoggerFactory.getLogger(AgentInfoRest.class);
#RequestMapping("/allagentInfo")
RestResponse getAllAgentInfo()
{
RestResponse restResponse = new RestResponse();
logger.info("getAllAgentInfo has no request parameter");
Collection<AgentInfo> result = agentInfoRepository.getAllAgents();
if(result.isEmpty())
{
restResponse.setRetCode(-1);
restResponse.setRetMsg("Request success,no such record in db");
}
restResponse.setResult(result);
return restResponse;
}
}
The Test Class:
public class AgentInfoRestTest {
String host = "http://127.0.0.1:8080";
#Test
public void getAllAgentInfoTest(#Mocked final AgentInfoRepository agentInfoRepository) {
RestTemplate restTemplate = new RestTemplate();
RestResponse restResponse = new RestResponse();
AgentInfoRest agentInfoRest = new AgentInfoRest();
new StrictExpectations() {
{
agentInfoRepository.getAllAgents(); result = null; times = 1;
}
};
String url = "/allagentInfo";
ResponseEntity<RestResponse> result = restTemplate.getForEntity(host + url, RestResponse.class);
new FullVerifications() {
{
}
};
}
}
The Error:
mockit.internal.MissingInvocation: Missing invocation of:
com.egoo.dao.repository.freelink.AgentInfoRepository#getAllAgents()
on mock instance: com.egoo.dao.repository.freelink.$Impl_AgentInfoRepository#50a7bc6e
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:117)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:253)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:84)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
Caused by: Missing invocation
at com.egoo.dao.rest.freelink.AgentInfoRestTest$1.(AgentInfoRestTest.java:33)
at com.egoo.dao.rest.freelink.AgentInfoRestTest.getAllAgentInfoTest(AgentInfoRestTest.java:31)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.lang.reflect.Method.invoke(Method.java:498)
... 7 more

Connection between Bluemix's Liberty for Java and Cloudant throw RuntimeError everyday

I create an application by using Bluemix's Liberty for Java to connect Cloudant. I found RuntimeException everyday while connecting to my RESTful API. However, I have to restart my application everyday to solve this problem.
An error is shown as follow:
Exception thrown by application class 'org.apache.cxf.interceptor.AbstractFaultChainInitiatorObserver.onMessage:116'
java.lang.RuntimeException: org.apache.cxf.interceptor.Fault: Error retrieving server response
at org.apache.cxf.interceptor.AbstractFaultChainInitiatorObserver.onMessage(AbstractFaultChainInitiatorObserver.java:116)
at [internal classes]
Caused by: org.apache.cxf.interceptor.Fault: Error retrieving server response
at org.apache.cxf.service.invoker.AbstractInvoker.createFault(AbstractInvoker.java:163)
... 1 more
Caused by: com.cloudant.client.org.lightcouch.CouchDbException: Error retrieving server response
at com.cloudant.client.org.lightcouch.CouchDbClient.execute(CouchDbClient.java:501)
at com.cloudant.client.org.lightcouch.CouchDbClient.executeToInputStream(CouchDbClient.java:515)
at com.cloudant.client.api.Database.findByIndex(Database.java:361)
at com.cloudant.client.api.Database.findByIndex(Database.java:321)
at th.co.gosoft.rest.TopicService.getHotTopicList(TopicService.java:82)
at sun.reflect.GeneratedMethodAccessor34.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at com.ibm.ws.jaxrs20.server.LibertyJaxRsServerFactoryBean.performInvocation(LibertyJaxRsServerFactoryBean.java:636)
... 1 more
Caused by: java.net.ProtocolException: Server rejected operation
at sun.net.www.protocol.http.HttpURLConnection.expect100Continue(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(Unknown Source)
at com.ibm.net.ssl.www2.protocol.https.b.getOutputStream(Unknown Source)
at com.cloudant.http.HttpConnection.execute(HttpConnection.java:231)
at com.cloudant.client.org.lightcouch.CouchDbClient.execute(CouchDbClient.java:466)
... 9 more
I use this CloudantClientMgr as follow:
package th.co.gosoft.util;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.Map.Entry;
import java.util.Set;
import com.cloudant.client.api.ClientBuilder;
import com.cloudant.client.api.CloudantClient;
import com.cloudant.client.api.Database;
import com.google.gson.JsonArray;
import com.google.gson.JsonElement;
import com.google.gson.JsonObject;
import com.google.gson.JsonParser;
public class CloudantClientMgr {
private static CloudantClient cloudant = null;
private static Database db = null;
private static String databaseName = "go10_db";
private static String url = "https://xxxxxxxxxx-bluemix.cloudant.com";
private static String user = "xxxxxxxxxx-bluemix";
private static String password = "password";
private static void initClient() {
if (cloudant == null) {
synchronized (CloudantClientMgr.class) {
if (cloudant != null) {
return;
}
try {
cloudant = createClient();
} catch (MalformedURLException e) {
throw new RuntimeException(e.getMessage(), e);
}
System.out.println("cloudant : " + cloudant.serverVersion());
}
}
}
private static CloudantClient createClient() throws MalformedURLException {
String VCAP_SERVICES = System.getenv("VCAP_SERVICES");
String serviceName = null;
CloudantClient client;
if (VCAP_SERVICES != null) {
System.out.println("VCAP_SERVICE");
JsonObject obj = (JsonObject) new JsonParser().parse(VCAP_SERVICES);
Entry<String, JsonElement> dbEntry = null;
Set<Entry<String, JsonElement>> entries = obj.entrySet();
for (Entry<String, JsonElement> eachEntry : entries) {
if (eachEntry.getKey().toLowerCase().contains("cloudant")) {
dbEntry = eachEntry;
break;
}
}
if (dbEntry == null) {
throw new RuntimeException("Could not find cloudantNoSQLDB key in VCAP_SERVICES env variable");
}
obj = (JsonObject) ((JsonArray) dbEntry.getValue()).get(0);
serviceName = (String) dbEntry.getKey();
System.out.println("Service Name - " + serviceName);
obj = (JsonObject) obj.get("credentials");
user = obj.get("username").getAsString();
password = obj.get("password").getAsString();
client = ClientBuilder.account(user)
.username(user)
.password(password)
.build();
} else {
System.out.println("LOCAL");
client = ClientBuilder.url(new URL(url))
.username(user)
.password(password)
.build();
}
return client;
}
public static Database getDB() {
if (cloudant == null) {
initClient();
}
if (db == null) {
try {
db = cloudant.database(databaseName, true);
} catch (Exception e) {
throw new RuntimeException("DB Not found", e);
}
}
return db;
}
private CloudantClientMgr() {
}
}
I use this code to connect CloudantClientMgr as follow:
#Path("topic")
public class TopicService {
#POST
#Path("/post")
#Consumes(MediaType.APPLICATION_JSON + ";charset=utf-8")
public Response createTopic(TopicModel topicModel) {
Database db = CloudantClientMgr.getDB();
com.cloudant.client.api.model.Response response = db.save(topicModel);
String result = response.getId();
return Response.status(201).entity(result).build();
}
}
The Cloudant version is 2.4.2
<dependency>
<groupId>com.cloudant</groupId>
<artifactId>cloudant-client</artifactId>
<version>2.4.2</version>
</dependency>
If anyone used to found this problem, please let me know the better solution than I have to restart my application every day.
The problem you describe sounds identical to one that was resolved in version 2.4.1.
I would check to make sure you don't have multiple versions of the cloudant-client included in your application or classpath and that your application is really using version 2.4.2. The line numbers in the stack trace you provide do not match with the source for 2.4.2.