I'm using the embedded Kafka server in my test described here: https://micronaut-projects.github.io/micronaut-kafka/latest/guide/#kafkaEmbedded. The problem is I'm getting this io.micronaut.context.exceptions.BeanContextException: Error processing bean [Definition: org.app.messaging.TestConsumer] method definition [void receive(String msg)]: Failed to inject value for parameter [testService] of method [setTestService] of class: org.app.messaging.TestConsumer when I run the test. Any ideas how to fix this?
Here's what the test looks like:
void "test run kafka embedded server"() {
given:
ApplicationContext applicationContext = ApplicationContext.run(
Collections.singletonMap(
AbstractKafkaConfiguration.EMBEDDED, true
)
)
when:
AbstractKafkaConsumerConfiguration config = applicationContext.getBean(AbstractKafkaConsumerConfiguration)
Properties props = config.getConfig()
then:
props[ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG] == 9091
when:
KafkaEmbedded kafkaEmbedded = applicationContext.getBean(KafkaEmbedded)
then:
kafkaEmbedded.kafkaServer.isPresent()
kafkaEmbedded.zkPort.isPresent()
cleanup:
applicationContext.close()
}
Placing a test anywhere other than the root package seem to be causing multiple "bean definition not found" issues. There's no ComponentScan support in the framework so the only thing that worked for me was to move the test file to the root package. There's some ideas here: https://github.com/micronaut-projects/micronaut-core/issues/511 if you're experiencing similar issues with a CLI app. However, it didn't work when using the embedded server and embedded kafka.
Related
Compiler error when using example provided in Flink documentation. The Flink documentation provides sample Scala code to set the REST client factory parameters when talking to Elasticsearch, https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/elasticsearch.html.
When trying out this code i get a compiler error in IntelliJ which says "Cannot resolve symbol restClientBuilder".
I found the following SO which is EXACTLY my problem except that it is in Java and i am doing this in Scala.
Apache Flink (v1.6.0) authenticate Elasticsearch Sink (v6.4)
I tried copy pasting the solution code provided in the above SO into IntelliJ, the auto-converted code also has compiler errors.
// provide a RestClientFactory for custom configuration on the internally created REST client
// i only show the setMaxRetryTimeoutMillis for illustration purposes, the actual code will use HTTP cutom callback
esSinkBuilder.setRestClientFactory(
restClientBuilder -> {
restClientBuilder.setMaxRetryTimeoutMillis(10)
}
)
Then i tried (auto generated Java to Scala code by IntelliJ)
// provide a RestClientFactory for custom configuration on the internally created REST client// provide a RestClientFactory for custom configuration on the internally created REST client
import org.apache.http.auth.AuthScope
import org.apache.http.auth.UsernamePasswordCredentials
import org.apache.http.client.CredentialsProvider
import org.apache.http.impl.client.BasicCredentialsProvider
import org.apache.http.impl.nio.client.HttpAsyncClientBuilder
import org.elasticsearch.client.RestClientBuilder
// provide a RestClientFactory for custom configuration on the internally created REST client// provide a RestClientFactory for custom configuration on the internally created REST client
esSinkBuilder.setRestClientFactory((restClientBuilder) => {
def foo(restClientBuilder) = restClientBuilder.setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() {
override def customizeHttpClient(httpClientBuilder: HttpAsyncClientBuilder): HttpAsyncClientBuilder = { // elasticsearch username and password
val credentialsProvider = new BasicCredentialsProvider
credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(es_user, es_password))
httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider)
}
})
foo(restClientBuilder)
})
The original code snippet produces the error "cannot resolve RestClientFactory" and then Java to Scala shows several other errors.
So basically i need to find a Scala version of the solution described in Apache Flink (v1.6.0) authenticate Elasticsearch Sink (v6.4)
Update 1: I was able to make some progress with some help from IntelliJ. The following code compiles and runs but there is another problem.
esSinkBuilder.setRestClientFactory(
new RestClientFactory {
override def configureRestClientBuilder(restClientBuilder: RestClientBuilder): Unit = {
restClientBuilder.setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() {
override def customizeHttpClient(httpClientBuilder: HttpAsyncClientBuilder): HttpAsyncClientBuilder = {
// elasticsearch username and password
val credentialsProvider = new BasicCredentialsProvider
credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(es_user, es_password))
httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider)
httpClientBuilder.setSSLContext(trustfulSslContext)
}
})
}
}
The problem is that i am not sure if i should be doing a new of the RestClientFactory object. What happens is that the application connects to the elasticsearch cluster but then discovers that the SSL CERT is not valid, so i had to put the trustfullSslContext (as described here https://gist.github.com/iRevive/4a3c7cb96374da5da80d4538f3da17cb), this got me past the SSL issue but now the ES REST Client does a ping test and the ping fails, it throws an exception and the app shutsdown. I am suspecting that the ping fails because of the SSL error and maybe it is not using the trustfulSslContext i setup as part of new RestClientFactory and this makes me suspect that i should not have done the new, there should be a simple way to update the existing RestclientFactory object and basically this is all happening because of my lack of Scala knowledge.
Happy to report that this is resolved. The code i posted in Update 1 is correct. The ping to ECE was not working for two reasons:
The certificate needs to include the complete chain including the root CA, the intermediate CA and the cert for the ECE. This helped get rid of the whole trustfulSslContext stuff.
The ECE was sitting behind an ha-proxy and the proxy did the mapping for the hostname in the HTTP request to the actual deployment cluster name in ECE. this mapping logic did not take into account that the Java REST High Level client uses the org.apache.httphost class which creates the hostname as hostname:port_number even when the port number is 443. Since it did not find the mapping because of the 443 therefore the ECE returned a 404 error instead of 200 ok (only way to find this was to look at unencrypted packets at the ha-proxy). Once the mapping logic in ha-proxy was fixed, the mapping was found and the pings are now successfull.
I'm attempting to create a custom exception handler for my Spring Cloud Dataflow stream to route some errors to be requeued and others to be DLQ'd.
To do this I'm utilizing the global Spring Integration "errorChannel" and routing based on exception type.
This is the code for the Spring Integration error router:
package com.acme.error.router;
import com.acme.exceptions.DlqException;
import org.springframework.cloud.stream.annotation.EnableBinding;
import org.springframework.integration.annotation.MessageEndpoint;
import org.springframework.integration.annotation.Router;
import org.springframework.integration.transformer.MessageTransformationException;
import org.springframework.messaging.Message;
#MessageEndpoint
#EnableBinding({ ErrorMessageChannels.class })
public class ErrorMessageMappingRouter {
private static final Logger LOGGER = LoggerFactory.getLogger(ErrorMessageMappingRouter.class);
public static final String ERROR_CHANNEL = "errorChannel";
#Router(inputChannel = ERROR_CHANNEL)
public String onError(Message<Object> message) {
LOGGER.debug("ERROR ROUTER - onError");
if(message.getPayload() instanceof MessageTransformationException) {
MessageTransformationException exception = (MessageTransformationException) message.getPayload();
Message<?> failedMessage = exception.getFailedMessage();
if(exceptionChainContainsDlq(exception)) {
return ErrorMessageChannels.DLQ_QUEUE_NAME;
}
return ErrorMessageChannels.REQUEUE_CHANNEL;
}
return ErrorMessageChannels.DLQ_QUEUE_NAME;
}
...
}
The error router is picked up by each of the stream apps through a package scan on the Spring Boot App for each:
#ComponentScan(basePackages = { "com.acme.error.router" }
#SpringBootApplication
public class StreamApp {}
When this is deployed and run with the local Spring Cloud Dataflow server (version 1.5.0-RELEASE), and a DlqException is thrown, the message is successfully routed to the onError method in the errorRouter and then placed into the dlq topic.
However, when this is deployed as a docker container with SCDF Kubernetes server (also version 1.5.0-RELEASE), the onError method is never hit. (The log statement at the beginning of the router is never output)
In the startup logs for the stream apps, it looks like the bean is picked up correctly and registers as a listener for the errorChannel, but for some reason, when exceptions are thrown they do not get handled by the onError method in our router.
Startup Logs:
o.s.i.endpoint.EventDrivenConsumer : Adding {router:errorMessageMappingRouter.onError.router} as a subscriber to the 'errorChannel' channel
o.s.i.channel.PublishSubscribeChannel : Channel 'errorChannel' has 1 subscriber(s).
o.s.i.endpoint.EventDrivenConsumer : started errorMessageMappingRouter.onError.router
We are using all default settings for the spring cloud stream and kafka binder configurations:
spring.cloud:
stream:
binders:
kafka:
type: kafka
environment.spring.cloud.stream.kafka.binder.brokers=brokerlist
environment.spring.cloud.stream.kafka.binder.zkNodes=zklist
Edit: Added pod args from kubectl describe <pod>
Args:
--spring.cloud.stream.bindings.input.group=delivery-stream
--spring.cloud.stream.bindings.output.producer.requiredGroups=delivery-stream
--spring.cloud.stream.bindings.output.destination=delivery-stream.enricher
--spring.cloud.stream.binders.xdkafka.environment.spring.cloud.stream.kafka.binder.zkNodes=<zkNodes>
--spring.cloud.stream.binders.xdkafka.type=kafka
--spring.cloud.stream.binders.xdkafka.defaultCandidate=true
--spring.cloud.stream.binders.xdkafka.environment.spring.cloud.stream.kafka.binder.brokers=<brokers>
--spring.cloud.stream.bindings.input.destination=delivery-stream.config-enricher
One other idea we attempted was trying to use the Spring Cloud Stream - spring integration error channel support to send to a broker topic on errors, but since messages don't seem to be landing in the global Spring Integration errorChannel at all, that didn't work either.
Is there anything special we need to do in SCDF Kubernetes to enable the global Spring Integration errorChannel?
What am I missing here?
Update with solution from the comments:
After reviewing your configuration I am now pretty sure I know what
the issue is. You have a multi-binder configuration scenario. Even if
you only deal with a single binder instance the existence of
spring.cloud.stream.binders.... is what's going to make framework
treat it as multi-binder. Basically this a bug -
github.com/spring-cloud/spring-cloud-stream/issues/1384. As you can
see it was fixed but you need to upgrade to Elmhurst.SR2 or grab the
latest snapshot (we're in RC2 and 2.1.0.RELEASE is in few weeks
anyway) – Oleg Zhurakousky
This was indeed the problem with our setup. Instead of upgrading, we just eliminated our multi-binder usage for now and the issue was resolved.
Update with solution from the comments:
After reviewing your configuration I am now pretty sure I know what
the issue is. You have a multi-binder configuration scenario. Even if
you only deal with a single binder instance the existence of
spring.cloud.stream.binders.... is what's going to make framework
treat it as multi-binder. Basically this a bug -
github.com/spring-cloud/spring-cloud-stream/issues/1384. As you can
see it was fixed but you need to upgrade to Elmhurst.SR2 or grab the
latest snapshot (we're in RC2 and 2.1.0.RELEASE is in few weeks
anyway) – Oleg Zhurakousky
This was indeed the problem with our setup. Instead of upgrading, we just eliminated our multi-binder usage for now and the issue was resolved.
I have a Spring Boot project, built using Maven, where I intend to use embedded mongo db. I am using Eclipse on Windows 7.
I am behind a proxy that uses automatic configuration script, as I have observed in the Connection tab of Internet Options.
I am getting the following exception when I try to run the application.
java.io.IOException: Could not open inputStream for https://downloads.mongodb.org/win32/mongodb-win32-i386-3.2.2.zip
at de.flapdoodle.embed.process.store.Downloader.downloadInputStream(Downloader.java:131) ~[de.flapdoodle.embed.process-2.0.1.jar:na]
at de.flapdoodle.embed.process.store.Downloader.download(Downloader.java:69) ~[de.flapdoodle.embed.process-2.0.1.jar:na]
....
MongoDB gets downloaded just fine, when I hit the following URL in my web browser:
https://downloads.mongodb.org/win32/mongodb-win32-i386-3.2.2.zip
This leads me to believe that probably I'm missing some configuration in my Eclipse or may be the maven project itself.
Please help me to find the right configuration.
What worked for me on a windows machine:
Download the zip file (https://downloads.mongodb.org/win32/mongodb-win32-i386-3.2.2.zip)
manually and put it (not unpack) into this folder:
C:\Users\<Username>\.embedmongo\win32\
Indeed the problem is about your proxy (a corporate one I guess).
If the proxy do not require authentication, you can solve your problem easily just by adding the appropriate -Dhttp.proxyHost=... and -Dhttp.proxyPort=... (or/and the same with "https.[...]") as JVM arguments in your eclipse junit Runner, as suggested here : https://github.com/learning-spring-boot/learning-spring-boot-2nd-edition-code/issues/2
One solution to your problem is to do the following.
Download MongoDB and place it on a ftp server which is inside your corporate network (for which you would not need proxy).
Then write a configuration in your project like this
#Bean
#ConditionalOnProperty("mongo.proxy")
public IRuntimeConfig embeddedMongoRuntimeConfig() {
final Command command = Command.MongoD;
final IRuntimeConfig runtimeConfig = new RuntimeConfigBuilder()
.defaults(command)
.artifactStore(new ExtractedArtifactStoreBuilder()
.defaults(command)
.download(new DownloadConfigBuilder()
.defaultsForCommand(command)
.downloadPath("your-ftp-path")
.build())
.build())
.build();
return runtimeConfig;
}
With the property mongo.proxy you can control whether Spring Boot downloads MongoDB from your ftp server or from outside. If it is set to true then it downloads from the ftp server. If not then it tries to download from the internet.
The easiest way seems to me to customize the default configuration:
#Bean
DownloadConfigBuilderCustomizer mongoProxyCustomizer() {
return configBuilder -> {
configBuilder.proxyFactory(new HttpProxyFactory(host, port));
};
}
Got the same issue (with Spring Boot 2.6.1 the spring.mongodb.embedded.version property is mandatory).
To configure the proxy, I've added the configuration bean by myself:
#Value("${spring.mongodb.embedded.proxy.domain}")
private String proxyDomain;
#Value("${spring.mongodb.embedded.proxy.port}")
private Integer proxyPort;
#Bean
RuntimeConfig embeddedMongoRuntimeConfig(ObjectProvider<DownloadConfigBuilderCustomizer> downloadConfigBuilderCustomizers) {
Logger logger = LoggerFactory.getLogger(this.getClass().getPackage().getName() + ".EmbeddedMongo");
ProcessOutput processOutput = new ProcessOutput(Processors.logTo(logger, Slf4jLevel.INFO), Processors.logTo(logger, Slf4jLevel.ERROR), Processors.named("[console>]", Processors.logTo(logger, Slf4jLevel.DEBUG)));
return Defaults.runtimeConfigFor(Command.MongoD, logger).processOutput(processOutput).artifactStore(this.getArtifactStore(logger, downloadConfigBuilderCustomizers.orderedStream())).isDaemonProcess(false).build();
}
private ExtractedArtifactStore getArtifactStore(Logger logger, Stream<DownloadConfigBuilderCustomizer> downloadConfigBuilderCustomizers) {
de.flapdoodle.embed.process.config.store.ImmutableDownloadConfig.Builder downloadConfigBuilder = Defaults.downloadConfigFor(Command.MongoD);
downloadConfigBuilder.progressListener(new Slf4jProgressListener(logger));
downloadConfigBuilderCustomizers.forEach((customizer) -> {
customizer.customize(downloadConfigBuilder);
});
DownloadConfig downloadConfig = downloadConfigBuilder
.proxyFactory(new HttpProxyFactory(proxyDomain, proxyPort)) // <--- HERE
.build();
return Defaults.extractedArtifactStoreFor(Command.MongoD).withDownloadConfig(downloadConfig);
}
In my case, I had to add the HTTPS corporate proxy to Intellij Run Configuration.
Https because it was trying to download:
https://downloads.mongodb.org/win32/mongodb-win32-x86_64-4.0.2.zip
application.properties:
spring.data.mongodb.database=test
spring.data.mongodb.port=27017
spring.mongodb.embedded.version=4.0.2
Please keep in mind this is a (DEV) setup.
I know that embedded Elasticsearch is not recommended. I'm just trying it for testing purposes.
I'm trying to start an embedded Elasticsearch node giving the configuration from the following elasticsearch.yml
# Name for the cluster
cluster.name: elasticsearch
# Name for the embedded node
node.name: EmbeddedESNode
# Path to log files:
path.logs: logs
discovery.zen.ping.unicast.hosts: []
# Disable dynamic scripting
script.inline: false
script.stored: false
script.file: false
transport.type: local
http.type: netty3
I'm using es 5.1.1 and my code to start embedded node is as follows.
try {
Settings elasticsearchSetting = Settings.builder()
// Value for path.home is required for es but will not be used as long as other properties
// path.logs, path.data and path.conf is set.
.put(ES_PROPERTY_PATH_HOME, "nullpath")
.put(ES_PROPERTY_PATH_CONF, confDir)
.build();
Node node = new Node(elasticsearchSetting).start();
logger.info("Embedded Elasticsearch server successfully started ...");
} catch (Exception e) {
throw e;
}
I get the following trace.
java.lang.IllegalStateException: Unsupported http.type [netty3]
at org.elasticsearch.common.network.NetworkModule.getHttpServerTransportSupplier(NetworkModule.java:194) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.node.Node.<init>(Node.java:396) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.node.Node.<init>(Node.java:229) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.node.Node.<init>(Node.java:225) ~[elasticsearch-5.1.1.jar:5.1.1]
... 18 more
I've tried with http.type: netty4 as well but no luck so far. It works when http.enabled:false is set but I want to use http rest api for testing.
P.S:
I've been using this elasticsearch hadoop class for reference in implementing embedded es and unfortunately I couldn't find any docs on http.type.
Can't I start an embedded node with http now in ES 5.x ?
What am i doing wrong here ?
Any help is highly appreciated.
As mentioned by #Bastian the issue was that transport module not being in the classpath. The solution was already there in es-hadoop embedded es implementation.
private static class PluginConfigurableNode extends Node {
public PluginConfigurableNode(Settings settings, Collection<Class<? extends Plugin>> classpathPlugins) {
super(InternalSettingsPreparer.prepareEnvironment(settings, null), classpathPlugins);
}
}
We can give netty3 as a plugin as follows. Then everything works well.
Collection<Class<? extends Plugin>> plugins = Arrays.asList(Netty3Plugin.class);
Node node = new PluginConfigurableNode(elasticsearchSetting, plugins).start();
You need to add the module transport-netty4 to your classpath:
<dependency>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>transport-netty4-client</artifactId>
<version>5.1.1</version>
<scope>test</scope>
</dependency>
See also my answer here
I have service in which i am accessing few configuration properties from grailsApplication
I am injecting it like this
class MyWebService{
def grailsApplication
WebService webService = new WebService()
def getProxy(url, flag){
return webService.getClient(url)
}
def getResponse(){
def proxy = getProxy(grailsApplication.config.grails.wsdlURL, true)
def response = proxy.getItem(ItemType)
return response
}
}
When i call getProxy() method, i see this in tomcat logs
No signature of method: org.example.MyWebService.getProxy() is applicable for argument types: (groovy.util.ConfigObject, java.lang.Boolean) values: [[:], true]
Possible solutions: getProxy(), getProxy(java.lang.String, boolean), setProxy(java.lang.Object)
which means grailsApplication is not getting injected into the service, is there any alternate way to access configuration object ? according to burtbeckwith's post configurationholder has been deprecated, can't think of anything else.
Interestingly the very same service works fine in my local IDE(GGTS 3.1.0), that means locally grailsApplication is getting injected, but when i create a war to deploy to a standalone tomcat, it stops getting injected.
I seem to have figured out the problem, actually grailsApplication is getting injected properly otherwise it would have thrown a null pointer exception, i feel the configuration properties are not getting added. Actually the scenario is like, i have a separate custom configuration file which holds configuration data for different environments, my application listens to the environement type(a variable which is set from tomcat) and based on that merges the corresponding config properties from my custom configuration file. i think those propreties are probably not getting added