The issue is pretty much the same as https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/455 but for the Functional approach
#Bean
public Consumer<Message<Foo>> fooConsumer() {
return message -> {
String key = // From message header - KafkaHeaders.RECEIVED_MESSAGE_KEY
Foo payLoad = message.getPayload();
if (payLoad == null) {
deleteSomething(key);
} else {
addSomething(key, payLoad);
}
};
}
When a proper 'Foo' (json) is produced the fooConsumer is invoked. But when a 'null' payload/tombstone is produced the consumer function is never called.
Workaround I tried with Custom 'Convertor' approach which is working:
#Component
public class KafkaNullConverter extends MappingJackson2MessageConverter {
#Override
protected Object convertFromInternal(Message<?> message, #NonNull Class<?> targetClass, Object conversionHint) {
if (message.getPayload() instanceof KafkaNull) {
return Foo.builder().isDeleted(true).build(); // Note: new `isDeleted` field added to `Foo`
}
return super.convertFromInternal(message, targetClass, conversionHint);
}
}
The fooConsumer then can check payload.isDeleted() to handle null case.
But this is verbose, pollutes pojo/model classes and have to repeat for every consumers.
I understand that spring-integration cannot working with null. But are there any better/standard approach to handle the 'tombstone' use-case with Functional approach?
Version: 3.0.4.RELEASE
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-binder-kafka</artifactId>
<version>3.0.4.RELEASE</version>
</dependency>
This was recently fixed on the main branch https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/pull/936 and https://github.com/spring-cloud/spring-cloud-function/issues/557
Related
I've sucessfully generated service proxies for service having methods returning Future<T>,
but just after I changed those methods to return Uni<T> according to API Translation - Smallrye Mutiny Vert.x bindings,
when I try to execute mvn clean compile it always tells me this error message :
Could not generate model for com.example.beers.BarmanService#giveMeAStaticBeer(java.lang.String): Proxy methods must have void or Fluent returns
I would need your help to enlighten me how to fix it.
I put those codes on GitHub, and these are some critical ones:
//BarmanService.java
#VertxGen
#ProxyGen
public interface BarmanService {
Uni<Beer> giveMeAStaticBeer(String customerName);
Uni<Integer> getMyBill(String customerName);
Uni<Void> payMyBill(String customerName);
static BarmanService createProxy(Vertx vertx, String address) {
return new BarmanServiceVertxEBProxy(vertx, address);
}
}
//BarmanServiceImpl.java
public class BarmanServiceImpl implements BarmanService {
Map<String, Integer> bills;
public BarmanServiceImpl() {
this.bills = new HashMap<>();
}
#Override
public Uni<Beer> giveMeAStaticBeer(String customerName) {
Beer beer = new Beer("Workshop River Stout", "English Stout", 5);
return Uni.createFrom().item(() -> beer);
}
#Override
public Uni<Integer> getMyBill(String customerName) {
return Uni.createFrom().item(() -> bills.get(customerName));
}
#Override
public Uni<Void> payMyBill(String customerName) {
bills.remove(customerName);
System.out.println("Removed debt of " + customerName);
return Uni.createFrom().voidItem();
}
}
//package-info.java
#ModuleGen(groupPackage = "com.example", name = "beers")
package com.example.beers;
import io.vertx.codegen.annotations.ModuleGen;
<!-- //pom.xml -->
<dependencies>
<!-- // ... -->
<dependency>
<groupId>io.vertx</groupId>
<artifactId>vertx-codegen</artifactId>
<classifier>processor</classifier>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>io.vertx</groupId>
<artifactId>vertx-service-proxy</artifactId>
</dependency>
<!-- // ... -->
<dependency>
<groupId>io.smallrye.reactive</groupId>
<artifactId>smallrye-mutiny-vertx-core</artifactId>
<version>2.30.1</version>
</dependency>
<dependency>
<groupId>io.smallrye.reactive</groupId>
<artifactId>vertx-mutiny-generator</artifactId>
<version>2.30.1</version>
</dependency>
<!-- // ... -->
</dependencies>
PS In the beginning, when I generated service proxies for service having methods returning Future<T>, there is a generated class returning Uni<T>, but I have no idea how to use it:
package com.example.mutiny.beers;
#io.smallrye.mutiny.vertx.MutinyGen(com.example.beers.BarmanService.class)
public class BarmanService {
public static final io.smallrye.mutiny.vertx.TypeArg<BarmanService> __TYPE_ARG = new io.smallrye.mutiny.vertx.TypeArg<>( obj -> new BarmanService((com.example.beers.BarmanService) obj),
BarmanService::getDelegate
);
private final com.example.beers.BarmanService delegate;
public BarmanService(com.example.beers.BarmanService delegate) {
this.delegate = delegate;
}
public BarmanService(Object delegate) {
this.delegate = (com.example.beers.BarmanService)delegate;
}
/**
* Empty constructor used by CDI, do not use this constructor directly.
**/
BarmanService() {
this.delegate = null;
}
public com.example.beers.BarmanService getDelegate() {
return delegate;
}
#CheckReturnValue
public io.smallrye.mutiny.Uni<com.example.beers.Beer> giveMeAStaticBeer(String customerName) {
return io.smallrye.mutiny.vertx.UniHelper.toUni(delegate.giveMeAStaticBeer(customerName));}
public com.example.beers.Beer giveMeAStaticBeerAndAwait(String customerName) {
return giveMeAStaticBeer(customerName).await().indefinitely();
}
public void giveMeAStaticBeerAndForget(String customerName) {
giveMeAStaticBeer(customerName).subscribe().with(io.smallrye.mutiny.vertx.UniHelper.NOOP);
}
// ...
public static com.example.mutiny.beers.BarmanService createProxy(io.vertx.mutiny.core.Vertx vertx, String address) {
com.example.mutiny.beers.BarmanService ret = com.example.mutiny.beers.BarmanService.newInstance((com.example.beers.BarmanService)com.example.beers.BarmanService.createProxy(vertx.getDelegate(), address));
return ret;
}
public static BarmanService newInstance(com.example.beers.BarmanService arg) {
return arg != null ? new BarmanService(arg) : null;
}
}
I just figured it out by myself. About to change service methods from returning Future<T> to Uni<T>,
The wrong apporach I did:
Edit package-info to remove useFutures = true
Edit service interfaces and change returning types
Edit service implimentations and change returning types, also change logic
Edit verticles to handle Uni<T> returned from service
And it turned out that the first three steps I did is unnecessary,
the suitable approach is:
Wrap vertx:
io.vertx.mutiny.core.Vertx mutinyVertx = new io.vertx.mutiny.core.Vertx(vertx);
Change the use of service interface to the generated one
import com.example.mutiny.beers.BarmanService;
Use the wrapped vertx:
BarmanService barmanService = BarmanService.createProxy(mutinyVertx, "beers.services.myapplication");
Edit verticles to handle Uni<T> returned from service
My problem has been solved, but I am not sure is it a good apporach to manually wrap the vertx on the MainVerticle launched by io.vertx.core.Launcher: io.vertx.mutiny.core.Vertx mutinyVertx = new io.vertx.mutiny.core.Vertx(vertx);, any suggestions guys?
I have a simple Solr query component as follows:
public class QueryPreprocessingComponent extends QueryComponent implements PluginInfoInitialized {
private static final Logger LOG = LoggerFactory.getLogger( QueryPreprocessingComponent.class );
private ExactMatchQueryProcessor exactMatchQueryProcessor;
public void init( PluginInfo info ) {
initializeProcessors(info);
}
private void initializeProcessors(PluginInfo info) {
List<PluginInfo> queryPreProcessors = info.getChildren("queryPreProcessors")
.get(0).getChildren("queryPreProcessor");
for (PluginInfo queryProcessor : queryPreProcessors) {
initializeProcessor(queryProcessor);
}
}
private void initializeProcessor(PluginInfo queryProcessor) {
QueryProcessorParam processorName = QueryProcessorParam.valueOf(queryProcessor.name);
switch(processorName) {
case ExactMatchQueryProcessor:
exactMatchQueryProcessor = new ExactMatchQueryProcessor(queryProcessor.initArgs);
LOG.info("ExactMatchQueryProcessor initialized...");
break;
default: throw new AssertionError();
}
}
#Override
public void prepare( ResponseBuilder rb ) throws IOException
{
if (exactMatchQueryProcessor != null) {
exactMatchQueryProcessor.modifyForExactMatch(rb);
}
}
#Override
public void process(ResponseBuilder rb) throws IOException
{
// do nothing - needed so we don't execute the query here.
return;
}
}
This works as expected functionally except when I use this in a distributed request, it has an issue with facets counts returned. It doubles the facet counts.
Note that I am not doing anything related to faceting in plugin. exactMatchQueryProcessor.modifyForExactMatch(rb); does a very minimal processing if the query is quoted otherwise it does nothing. Even if the incoming query is not quoted, facet count issue is there. Even if I comment everything inside prepare function, issue persists.
Note that this component is declared in as first-components in solrconfig.xml.
I resolved this issue by extending the class to SearchComponent instead of QueryComponent. It seems that SearchComponent sits at higher level of abstraction than QueryComponent and is useful when you want to work on a layer above shards.
I'm struggling to convert the following snippet, which is converting the resultHandler to an observer and subscribing to the Single, to RxJava2, which doesn't support null.
#Override
public WikiDatabaseService createPage(String title, String markdown, Handler<AsyncResult<Void>> resultHandler) {
dbClient.rxUpdateWithParams(sqlQueries.get(SqlQuery.CREATE_PAGE), new JsonArray().add(title).add(markdown))
.map(res -> (Void) null)
.subscribe(SingleHelper.toObserver(resultHandler));
return this;
}
Is anyone able to offer any advice?
Use toCompletable with CompletableHelper.toObserver:
#Override
public WikiDatabaseService createPage(String title, String markdown, Handler<AsyncResult<Void>> resultHandler) {
dbClient.rxUpdateWithParams(sqlQueries.get(SqlQuery.CREATE_PAGE), new JsonArray().add(title).add(markdown))
.toCompletable()
.subscribe(CompletableHelper.toObserver(resultHandler));
return this;
}
Make sure your project dependencies include the Vert.x RxJava2 module:
<dependency>
<groupId>io.vertx</groupId>
<artifactId>vertx-rx-java2</artifactId>
</dependency>
I'm having trouble finding an example of how to make a custom operator with RxJava 2. I've considered a few approaches:
Using Observable.create, and then flatMaping on it from the source observable. I can get this working, but it doesn't quite feel right. I end up creating a static function which I provide the source Observable, and then flatMap on the source. In the OnSubscribe, I then instantiate an object that I pass the emitter to, which handles and manages the Observable / Emitter (as it's not trivial, and I want everything as encapsulated as possible).
Creating an ObservableOperator and providing it to Observable.lift. I can't find any examples of this for RxJava 2. I had to debug into my own example to make sure my understanding of upstream and downstream were correct. Because I can't find any examples or documentation on this for RxJava 2 I'm a little worried I might accidentally do something I'm not supposed to.
Create my own Observable type. This seems to be how the underlying operators work, many of which extend AbstractObservableWithUpstream. There is a lot going on here though, and it seems easy to miss something or do something I shouldn't. I'm not sure if I'm supposed to take an approach like this or not. I stepped myself through the mental process, and it seems like it can get hairy pretty quickly.
I'm going to proceed forward with option #2, but thought it worthwhile to ask what the supported method for doing this was in RxJava2 and also find out if there was any documentation or examples for this.
Writing operators is not recommended for beginners and many desired flow patterns can be achieved via existing operators.
Have you looked at RxJava's wiki about writing operators for 2.x? I suggest reading it from top to bottom.
using create() is possible but most people use it to emit the elements of a List with a for-each loop, not recognizing that Flowable.fromIterable does that.
We kept this extension point although RxJava 2 operators don't use lift() themselves. If you want to avoid some boilerplate with option 3. then you may try this route.
This is how RxJava 2 operators are implemented. AbstractObservableWithUpstream is a small convenience and not necessary for external implementors.
This may help you. I implement operator RxJava2 to handle APiError. I used lift operator.
See the example.
public final class ApiClient implements ApiClientInterface {
...
#NonNull
#Override
public Observable<ActivateResponse> activate(String email, EmailData emailLinkData) {
return myApiService.activate(email, emailData)
.lift(getApiErrorTransformer())
.subscribeOn(Schedulers.io());
}
private <T>ApiErrorOperator<T> getApiErrorTransformer() {
return new ApiErrorOperator<>(gson, networkService);
}
}
And then you can find custom operator
public final class ApiErrorOperator<T> implements ObservableOperator<T, T> {
private static final String TAG = "ApiErrorOperator";
private final Gson gson;
private final NetworkService networkService;
public ApiErrorOperator(#NonNull Gson gson, #NonNull NetworkService networkService) {
this.gson = gson;
this.networkService = networkService;
}
#Override
public Observer<? super T> apply(Observer<? super T> observer) throws Exception {
return new Observer<T>() {
#Override
public void onSubscribe(Disposable d) {
observer.onSubscribe(d);
}
#Override
public void onNext(T value) {
observer.onNext(value);
}
#Override
public void onError(Throwable e) {
Log.e(TAG, "onError", e);
if (e instanceof HttpException) {
try {
HttpException error = (HttpException) e;
Response response = error.response();
String errorBody = response.errorBody().string();
ErrorResponse errorResponse = gson.fromJson(errorBody.trim(), ErrorResponse.class);
ApiException exception = new ApiException(errorResponse, response);
observer.onError(exception);
} catch (IOException exception) {
observer.onError(exception);
}
} else if (!networkService.isNetworkAvailable()) {
observer.onError(new NetworkException(ErrorResponse.builder()
.setErrorCode("")
.setDescription("No Network Connection Error")
.build()));
} else {
observer.onError(e);
}
}
#Override
public void onComplete() {
observer.onComplete();
}
};
}
}
My problem is that in the main class I have some osgi references that work just fine when the class is call. But after that all the references became null. When I close the main windows and call shutdown method, the hubService reference returns null. What do I do wrong here?
private void shutdown() {
if(hubService == null) {
throw new NullPointerException();
}
hubService.shutdownHub(); // why is hubService null?
}
// bind hub service
public synchronized void setHubService(IHubService service) {
hubService = service;
try {
hubService.startHub(PORT, authenticationHandler);
} catch (Exception e) {
JOptionPane.showMessageDialog(mainFrame, e.toString(), "Server", JOptionPane.ERROR_MESSAGE);
System.exit(0);
}
}
// remove hub service
public synchronized void unsetHubService(IHubService service) {
hubService.shutdownHub();
hubService = null;
}
If a field can be read and written by multiple threads, you must protect access to read as well as write. Your first method, shutdown, does not protect the read of hubService so that the value of hubService can change between the first read and the second read. You don't show the declaration of the hubService field. You could make it volatile or only read when synchronized (on the same object used to synchronized when writing the field). Then your shutdown implementation could look like:
private volatile IHubService hubService;
private void shutdown() {
IHubService service = hubService; // make a copy of the field in a local variable
if (service != null) // use local var from now on since the field could have changed
service.shutdownHub();
}
I assume your shutdown method is the DS deactivate method? If so, why do you shutdown in the unset method as well in the shutdown method?
Overall the design does not seem very sound. The IHubService is used as a factory and should return some object that is then closed in the deactivate method. You made the IHubService effectively a singleton. Since it must come from another bundle, it should handle its life cycle itself.
Since you also do not use annotations, it is not clear if your set/unset methods are static/dynamic and/or single/multiple. The following code should not have your problems (exammple code with bnd annotations):
#Component public class MyImpl {
IHubService hub;
#Activate
void activate() {
hubService.startHub(PORT, authenticationHandler);
}
#DeActivate
void deactivate() {
hubService.shutdown();
}
#Reference
void setHub(IHubService hub) { this.hub = hub; }
}