I am getting below error while using HttpClient. Can you let me know how to use HttpClient exactly. I am new with elastic4s.
I want to connect scala with ssl configured elasticsearch. I also want to know how I can pass SSL details with link such as keystore path, truststore path and user name , password.
scala> import com.sksamuel.elastic4s.http.{HttpClient, HttpResponse}
import com.sksamuel.elastic4s.http.{HttpClient, HttpResponse}
scala> import com.sksamuel.elastic4s.http.ElasticDsl._
import com.sksamuel.elastic4s.http.ElasticDsl._
scala> val client = HttpClient(ElasticsearchClientUri(uri))
<console>:39: error: not found: value HttpClient
val client = HttpClient(ElasticsearchClientUri(uri))
HttpClient appears to be a trait in the codebase.You seem to be using the same as an object. You can check the implementation Here. For your use case i think the better approach would be to use ElasticClient. Code would look something like this
import com.sksamuel.elastic4s.http._
import com.sksamuel.elastic4s.{ElasticClient, ElasticDsl, ElasticsearchClientUri}
val client = elastic4s.ElasticClient(ElasticsearchClientUri(uri))
I got the same problem, i.e. in my setup I got errors (not found) when trying to use HttpClient (elastic4s-core,elastic4s-http-streams and elastic4s-client-esjava version 7.3.1 on scala 2.12.10).
The solution: you should be able to find and use JavaClient, an implementation of HttpClient that wraps the Elasticsearch Java Rest Client.
An example of how to use the JavaClient can be found here.
Thus, your code should look like the following:
import com.sksamuel.elastic4s.http.JavaClient
import com.sksamuel.elastic4s.{ElasticClient, ElasticDsl, ElasticProperties}
...
val client = ElasticClient(JavaClient(ElasticProperties(uri)))
Compiler error when using example provided in Flink documentation. The Flink documentation provides sample Scala code to set the REST client factory parameters when talking to Elasticsearch, https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/elasticsearch.html.
When trying out this code i get a compiler error in IntelliJ which says "Cannot resolve symbol restClientBuilder".
I found the following SO which is EXACTLY my problem except that it is in Java and i am doing this in Scala.
Apache Flink (v1.6.0) authenticate Elasticsearch Sink (v6.4)
I tried copy pasting the solution code provided in the above SO into IntelliJ, the auto-converted code also has compiler errors.
// provide a RestClientFactory for custom configuration on the internally created REST client
// i only show the setMaxRetryTimeoutMillis for illustration purposes, the actual code will use HTTP cutom callback
esSinkBuilder.setRestClientFactory(
restClientBuilder -> {
restClientBuilder.setMaxRetryTimeoutMillis(10)
}
)
Then i tried (auto generated Java to Scala code by IntelliJ)
// provide a RestClientFactory for custom configuration on the internally created REST client// provide a RestClientFactory for custom configuration on the internally created REST client
import org.apache.http.auth.AuthScope
import org.apache.http.auth.UsernamePasswordCredentials
import org.apache.http.client.CredentialsProvider
import org.apache.http.impl.client.BasicCredentialsProvider
import org.apache.http.impl.nio.client.HttpAsyncClientBuilder
import org.elasticsearch.client.RestClientBuilder
// provide a RestClientFactory for custom configuration on the internally created REST client// provide a RestClientFactory for custom configuration on the internally created REST client
esSinkBuilder.setRestClientFactory((restClientBuilder) => {
def foo(restClientBuilder) = restClientBuilder.setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() {
override def customizeHttpClient(httpClientBuilder: HttpAsyncClientBuilder): HttpAsyncClientBuilder = { // elasticsearch username and password
val credentialsProvider = new BasicCredentialsProvider
credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(es_user, es_password))
httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider)
}
})
foo(restClientBuilder)
})
The original code snippet produces the error "cannot resolve RestClientFactory" and then Java to Scala shows several other errors.
So basically i need to find a Scala version of the solution described in Apache Flink (v1.6.0) authenticate Elasticsearch Sink (v6.4)
Update 1: I was able to make some progress with some help from IntelliJ. The following code compiles and runs but there is another problem.
esSinkBuilder.setRestClientFactory(
new RestClientFactory {
override def configureRestClientBuilder(restClientBuilder: RestClientBuilder): Unit = {
restClientBuilder.setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() {
override def customizeHttpClient(httpClientBuilder: HttpAsyncClientBuilder): HttpAsyncClientBuilder = {
// elasticsearch username and password
val credentialsProvider = new BasicCredentialsProvider
credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(es_user, es_password))
httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider)
httpClientBuilder.setSSLContext(trustfulSslContext)
}
})
}
}
The problem is that i am not sure if i should be doing a new of the RestClientFactory object. What happens is that the application connects to the elasticsearch cluster but then discovers that the SSL CERT is not valid, so i had to put the trustfullSslContext (as described here https://gist.github.com/iRevive/4a3c7cb96374da5da80d4538f3da17cb), this got me past the SSL issue but now the ES REST Client does a ping test and the ping fails, it throws an exception and the app shutsdown. I am suspecting that the ping fails because of the SSL error and maybe it is not using the trustfulSslContext i setup as part of new RestClientFactory and this makes me suspect that i should not have done the new, there should be a simple way to update the existing RestclientFactory object and basically this is all happening because of my lack of Scala knowledge.
Happy to report that this is resolved. The code i posted in Update 1 is correct. The ping to ECE was not working for two reasons:
The certificate needs to include the complete chain including the root CA, the intermediate CA and the cert for the ECE. This helped get rid of the whole trustfulSslContext stuff.
The ECE was sitting behind an ha-proxy and the proxy did the mapping for the hostname in the HTTP request to the actual deployment cluster name in ECE. this mapping logic did not take into account that the Java REST High Level client uses the org.apache.httphost class which creates the hostname as hostname:port_number even when the port number is 443. Since it did not find the mapping because of the 443 therefore the ECE returned a 404 error instead of 200 ok (only way to find this was to look at unencrypted packets at the ha-proxy). Once the mapping logic in ha-proxy was fixed, the mapping was found and the pings are now successfull.
I'm currently trying to expose restful webservice using OSGI remote services, but it seems that i'm missing something from ECF tutorial.
below the details here is the impl class:
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import org.example.api.HelloWorldService;
import org.osgi.service.component.annotations.Component;
#Path("/helloworld")
#Component(property = { "service.exported.interfaces=HTTP",
"service.exported.configs=ecf.jaxrs.jersey.server",
"ecf.jaxrs.jersey.server.alias=/jersey" })
public class HelloWorldResource implements HelloWorldService {
#GET
#Produces("text/plain")
public String getMessage() {
// Return some textual content
return "Hello World";
}}
I followed this link to know how things can be together but i wasn't able to get HTTP 200 ok message based on the jax-rs path annotation
https://wiki.eclipse.org/Tutorial:_Using_REST_and_OSGi_Standards_for_Micro_Services
Please note that all bundle are resolved correctly.
I've created a bndtools4 bndrun for something like your hello jaxrs service. First, you need to create a new workspace using the ECF bndtools.workspace template. Just follow the instructions here:
https://wiki.eclipse.org/Bndtools_Support_for_Remote_Services_Development
for creating a workspace from the ECF bndtools.workspace template. Just today I've added the JaxRS bundles to this workspace template.
Then I created an org.example.api project with this interface
package org.example.api;
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
#Path("/helloworld")
public interface HelloWorldService {
#GET
#Path("/hello")
#Produces("text/plain")
String getMessage();
}
Note that it differs slightly from yours because it has a #PATH annotation for the getMessage() method.
Then in another project...named org.example.impl there is this class:
package org.example.impl;
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import org.example.api.HelloWorldService;
import org.osgi.service.component.annotations.Component;
#Path("/helloworld")
#Component(property = { "service.exported.interfaces=*",
"service.intents=jaxrs" })
public class HelloWorldResource implements HelloWorldService {
#GET
#Path("/hello")
#Produces("text/plain")
public String getMessage() {
// Return some textual content
return "Hello World";
}}
The jaxrs annotations are the same as with the interface (as they should be). There are fewer
And here's the bndrun (which I've called jettyserver.bndrun):
-runrequires: \
bnd.identity;id='org.eclipse.ecf.provider.jersey.server',\
osgi.identity;filter:='(&(osgi.identity=javax.validation.api)(version>=1.1.0))',\
osgi.identity;filter:='(&(osgi.identity=org.apache.felix.gogo.command)(version>=1.0.2))',\
osgi.identity;filter:='(&(osgi.identity=org.apache.felix.gogo.runtime)(version>=1.0.10))',\
osgi.identity;filter:='(&(osgi.identity=org.slf4j.api)(version>=1.7.2))',\
bnd.identity;version='latest';id='org.example.impl',\
bnd.identity;id='org.apache.felix.gogo.command',\
bnd.identity;id='org.apache.felix.gogo.runtime',\
bnd.identity;id='org.apache.felix.gogo.shell',\
bnd.identity;id='org.eclipse.ecf.osgi.services.remoteserviceadmin.console',\
bnd.identity;id='org.apache.felix.scr',\
bnd.identity;id='org.eclipse.ecf.osgi.services.distribution',\
bnd.identity;id='org.eclipse.ecf.provider.jersey.client'
-runfw: org.apache.felix.framework;version='[5.6.10,5.6.10]'
-runee: JavaSE-1.8
-runbundles: \
com.fasterxml.jackson.core.jackson-annotations;version='[2.9.2,2.9.3)',\
com.fasterxml.jackson.core.jackson-core;version='[2.9.2,2.9.3)',\
com.fasterxml.jackson.core.jackson-databind;version='[2.9.2,2.9.3)',\
com.fasterxml.jackson.jaxrs.jackson-jaxrs-base;version='[2.9.2,2.9.3)',\
com.fasterxml.jackson.jaxrs.jackson-jaxrs-json-provider;version='[2.9.2,2.9.3)',\
javax.annotation;version='[1.2.0,1.2.1)',\
javax.inject;version='[1.0.0,1.0.1)',\
javax.persistence;version='[2.2.0,2.2.1)',\
javax.validation.api;version='[1.1.0,1.1.1)',\
javax.ws.rs;version='[2.0.1,2.0.2)',\
org.aopalliance;version='[1.0.0,1.0.1)',\
org.apache.felix.configadmin;version='[1.8.16,1.8.17)',\
org.apache.felix.eventadmin;version='[1.4.10,1.4.11)',\
org.apache.felix.gogo.command;version='[1.0.2,1.0.3)',\
org.apache.felix.gogo.runtime;version='[1.0.10,1.0.11)',\
org.apache.felix.http.jetty;version='[3.4.8,3.4.9)',\
org.apache.felix.http.servlet-api;version='[1.1.2,1.1.3)',\
org.apache.felix.scr;version='[2.0.14,2.0.15)',\
org.eclipse.core.jobs;version='[3.9.3,3.9.4)',\
org.eclipse.ecf;version='[3.9.0,3.9.1)',\
org.eclipse.ecf.discovery;version='[5.0.300,5.0.301)',\
org.eclipse.ecf.identity;version='[3.9.0,3.9.1)',\
org.eclipse.ecf.osgi.services.remoteserviceadmin;version='[4.6.800,4.6.801)',\
org.eclipse.ecf.osgi.services.remoteserviceadmin.proxy;version='[1.0.100,1.0.101)',\
org.eclipse.ecf.provider.jaxrs;version='[1.3.0,1.3.1)',\
org.eclipse.ecf.provider.jaxrs.server;version='[1.4.0,1.4.1)',\
org.eclipse.ecf.provider.jersey.server;version='[1.3.0,1.3.1)',\
org.eclipse.ecf.remoteservice;version='[8.13.0,8.13.1)',\
org.eclipse.ecf.remoteservice.asyncproxy;version='[2.1.0,2.1.1)',\
org.eclipse.equinox.common;version='[3.9.0,3.9.1)',\
org.eclipse.equinox.concurrent;version='[1.1.0,1.1.1)',\
org.eclipse.equinox.supplement;version='[1.7.0,1.7.1)',\
org.eclipse.osgi.services.remoteserviceadmin;version='[1.6.200,1.6.201)',\
org.glassfish.hk2.api;version='[2.5.0,2.5.1)',\
org.glassfish.hk2.osgi-resource-locator;version='[2.5.0,2.5.1)',\
org.glassfish.hk2.utils;version='[2.5.0,2.5.1)',\
org.glassfish.jersey.bundles.repackaged.jersey-guava;version='[2.22.1,2.22.2)',\
org.glassfish.jersey.containers.servlet.core;version='[2.22.1,2.22.2)',\
org.glassfish.jersey.core.jersey-common;version='[2.22.1,2.22.2)',\
org.glassfish.jersey.core.jersey-server;version='[2.22.1,2.22.2)',\
org.glassfish.jersey.ext.entityfiltering;version='[2.22.1,2.22.2)',\
org.glassfish.jersey.media.jersey-media-json-jackson;version='[2.22.1,2.22.2)',\
org.slf4j.api;version='[1.7.2,1.7.3)',\
org.example.api;version=snapshot,\
org.example.impl;version=snapshot,\
org.apache.felix.gogo.shell;version='[1.0.0,1.0.1)',\
org.eclipse.ecf.console;version='[1.1.0,1.1.1)',\
org.eclipse.ecf.osgi.services.remoteserviceadmin.console;version='[1.0.0,1.0.1)',\
org.eclipse.ecf.osgi.services.distribution;version='[2.1.200,2.1.201)',\
javassist;version='[3.13.0,3.13.1)',\
org.eclipse.ecf.provider.jaxrs.client;version='[1.3.0,1.3.1)',\
org.eclipse.ecf.provider.jersey.client;version='[1.3.0,1.3.1)',\
org.glassfish.hk2.locator;version='[2.5.0,2.5.1)',\
org.glassfish.jersey.core.jersey-client;version='[2.22.1,2.22.2)'
With your permission, I would like to add bndtools project templates for both jaxrs hello api and impl projects to the ECF bndtools.workspace repo, along with a jersey and cxf bndrun server and client templates (once I complete them). I'll create the bndruns and test over the next few days.
The exported interfaces property should not be "HTTP". Instead it must be an interface fqname or Star. So try this:
"service.exported.interfaces=*"
If you are not using already, please update to ECF 3.14.0. Further, after 3.14.0/Photon (last few weeks) there have been significant changes to the JaxRS providers to support OSGi R7 Async Remote Services so please make sure you have the latest from JaxRSProviders as well. The required remote service properties have changed based upon the R7 changes. Please see the service properties of the hello examples and this short tutorial for running the hello example on Karaf:
https://wiki.eclipse.org/Tutorial:_JaxRS_Remote_Services_on_Karaf
In short, it's no longer necessary to specify ecf.jaxrs.jersey.server.alias but there are other (newly standardize) properties to set.
I would like to create a bndtools4 JaxRS run descriptor template, and I think this would make it significantly easier for you. I suspect your problem might be that do not have all necessary jersey bundles, as jersey has quite a number of dependencies...all of which have to be present to successfully export a remote service. For ref, here's the setup info for using the project and bndrun templates that are currently there:
https://wiki.eclipse.org/Bndtools_Support_for_Remote_Services_Development
I will create a bndtools4 bndrun template for the CXF and Jersey distribution providers, but have not had enough time since the JaxRSProvider changes. If you would like to express your desire for bndrun template and/or help with contributions, please open an issue at https://github.com/ECF/JaxRSProviders/issues and that will help track.
In the mean time, the examples list of bundles is in the product files here:
https://github.com/ECF/JaxRSProviders/tree/master/examples/com.mycorp.examples.student.remoteservice.host/launch
For info:
I've added project templates for JaxRS API, Impl, Consumer projects (based upon the HelloWorldService) and I've added bndrun templates for Jersey server and client, as well as CXF server and client.
See https://github.com/ECF/JaxRSProviders/issues/6
I am currently in the process of moving all our Rest tests to the CI server and have noticed that all tests are failing due to the an SSL handshake, now I have successfully disabled this with the TrustManager with our Java test suite, but am unsure how to do it with Scala dispatch library, and havent been able to find many examples that could apply in this scenario.
val JSONstr = "{samplekey:samplevalue}"
val response:String = Http(url("https://www.host.com/path/to/post")
<< (checkInJSONstr, "application/json") as_str)
The following exception is occuring as expected:
javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated at com.sun.net.ssl.internal.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:352)
...
Is there a way to do it cleanly syntactically ignore SSL with the dispatch library?
import dispatch._
val http = new Http with HttpsLeniency
val response:String = http(url("https://www.host.com/path/to/post")
<< (checkInJSONstr, "application/json") as_str)
viktortnk's answer doesn't work anymore with the newest version of Dispatch (0.13.2). For the newest version, you can use the following to create an http client that accepts any certificate:
val myHttp = Http.withConfiguration(config => config.setAcceptAnyCertificate(true))
Then you can use it for GET requests like this:
myHttp(url("https://www.host.com/path").GET OK as.String)
I found this out here: Why does dispatch throw "java.net.ConnectException: General SSLEngine ..." and "unexpected status" exceptions for a particular URL?