I am currently in the process of moving all our Rest tests to the CI server and have noticed that all tests are failing due to the an SSL handshake, now I have successfully disabled this with the TrustManager with our Java test suite, but am unsure how to do it with Scala dispatch library, and havent been able to find many examples that could apply in this scenario.
val JSONstr = "{samplekey:samplevalue}"
val response:String = Http(url("https://www.host.com/path/to/post")
<< (checkInJSONstr, "application/json") as_str)
The following exception is occuring as expected:
javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated at com.sun.net.ssl.internal.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:352)
...
Is there a way to do it cleanly syntactically ignore SSL with the dispatch library?
import dispatch._
val http = new Http with HttpsLeniency
val response:String = http(url("https://www.host.com/path/to/post")
<< (checkInJSONstr, "application/json") as_str)
viktortnk's answer doesn't work anymore with the newest version of Dispatch (0.13.2). For the newest version, you can use the following to create an http client that accepts any certificate:
val myHttp = Http.withConfiguration(config => config.setAcceptAnyCertificate(true))
Then you can use it for GET requests like this:
myHttp(url("https://www.host.com/path").GET OK as.String)
I found this out here: Why does dispatch throw "java.net.ConnectException: General SSLEngine ..." and "unexpected status" exceptions for a particular URL?
Related
I am using flink 1.8.0 and I am trying to query my job state.
val descriptor = new ValueStateDescriptor("myState", Types.CASE_CLASS[Foo])
descriptor.setQueryable("my-queryable-State")
I used port 9067 which is the default port according to this, my client:
val client = new QueryableStateClient("127.0.0.1", 9067)
val jobId = JobID.fromHexString("d48a6c980d1a147e0622565700158d9e")
val execConfig = new ExecutionConfig
val descriptor = new ValueStateDescriptor("my-queryable-State", Types.CASE_CLASS[Foo])
val res: Future[ValueState[Foo]] = client.getKvState(jobId, "my-queryable-State","a", BasicTypeInfo.STRING_TYPE_INFO, descriptor)
res.map(_.toString).pipeTo(sender)
but I am getting :
[ERROR] [06/25/2019 20:37:05.499] [bvAkkaHttpServer-akka.actor.default-dispatcher-5] [akka.actor.ActorSystemImpl(bvAkkaHttpServer)] Error during processing of request: 'org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /127.0.0.1:9067'. Completing with 500 Internal Server Error response. To change default exception handling behavior, provide a custom ExceptionHandler.
java.util.concurrent.CompletionException: org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /127.0.0.1:9067
what am I doing wrong ?
how and where should I define QueryableStateOptions
So If You want to use the QueryableState You need to add the proper Jar to Your flink. The jar is flink-queryable-state-runtime, it can be found in the opt folder in Your flink distribution and You should move it to the lib folder.
As for the second question the QueryableStateOption is just a class that is used to create static ConfigOption definitions. The definitions are then used to read the configurations from flink-conf.yaml file. So currently the only option to configure the QueryableState is to use the flink-conf file in the flink distribution.
EDIT: Also, try reading this]1 it provides more info on how does Queryable State works. You shouldn't really connect directly to the server port but rather You should use the proxy port which by default is 9069.
Compiler error when using example provided in Flink documentation. The Flink documentation provides sample Scala code to set the REST client factory parameters when talking to Elasticsearch, https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/elasticsearch.html.
When trying out this code i get a compiler error in IntelliJ which says "Cannot resolve symbol restClientBuilder".
I found the following SO which is EXACTLY my problem except that it is in Java and i am doing this in Scala.
Apache Flink (v1.6.0) authenticate Elasticsearch Sink (v6.4)
I tried copy pasting the solution code provided in the above SO into IntelliJ, the auto-converted code also has compiler errors.
// provide a RestClientFactory for custom configuration on the internally created REST client
// i only show the setMaxRetryTimeoutMillis for illustration purposes, the actual code will use HTTP cutom callback
esSinkBuilder.setRestClientFactory(
restClientBuilder -> {
restClientBuilder.setMaxRetryTimeoutMillis(10)
}
)
Then i tried (auto generated Java to Scala code by IntelliJ)
// provide a RestClientFactory for custom configuration on the internally created REST client// provide a RestClientFactory for custom configuration on the internally created REST client
import org.apache.http.auth.AuthScope
import org.apache.http.auth.UsernamePasswordCredentials
import org.apache.http.client.CredentialsProvider
import org.apache.http.impl.client.BasicCredentialsProvider
import org.apache.http.impl.nio.client.HttpAsyncClientBuilder
import org.elasticsearch.client.RestClientBuilder
// provide a RestClientFactory for custom configuration on the internally created REST client// provide a RestClientFactory for custom configuration on the internally created REST client
esSinkBuilder.setRestClientFactory((restClientBuilder) => {
def foo(restClientBuilder) = restClientBuilder.setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() {
override def customizeHttpClient(httpClientBuilder: HttpAsyncClientBuilder): HttpAsyncClientBuilder = { // elasticsearch username and password
val credentialsProvider = new BasicCredentialsProvider
credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(es_user, es_password))
httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider)
}
})
foo(restClientBuilder)
})
The original code snippet produces the error "cannot resolve RestClientFactory" and then Java to Scala shows several other errors.
So basically i need to find a Scala version of the solution described in Apache Flink (v1.6.0) authenticate Elasticsearch Sink (v6.4)
Update 1: I was able to make some progress with some help from IntelliJ. The following code compiles and runs but there is another problem.
esSinkBuilder.setRestClientFactory(
new RestClientFactory {
override def configureRestClientBuilder(restClientBuilder: RestClientBuilder): Unit = {
restClientBuilder.setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() {
override def customizeHttpClient(httpClientBuilder: HttpAsyncClientBuilder): HttpAsyncClientBuilder = {
// elasticsearch username and password
val credentialsProvider = new BasicCredentialsProvider
credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(es_user, es_password))
httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider)
httpClientBuilder.setSSLContext(trustfulSslContext)
}
})
}
}
The problem is that i am not sure if i should be doing a new of the RestClientFactory object. What happens is that the application connects to the elasticsearch cluster but then discovers that the SSL CERT is not valid, so i had to put the trustfullSslContext (as described here https://gist.github.com/iRevive/4a3c7cb96374da5da80d4538f3da17cb), this got me past the SSL issue but now the ES REST Client does a ping test and the ping fails, it throws an exception and the app shutsdown. I am suspecting that the ping fails because of the SSL error and maybe it is not using the trustfulSslContext i setup as part of new RestClientFactory and this makes me suspect that i should not have done the new, there should be a simple way to update the existing RestclientFactory object and basically this is all happening because of my lack of Scala knowledge.
Happy to report that this is resolved. The code i posted in Update 1 is correct. The ping to ECE was not working for two reasons:
The certificate needs to include the complete chain including the root CA, the intermediate CA and the cert for the ECE. This helped get rid of the whole trustfulSslContext stuff.
The ECE was sitting behind an ha-proxy and the proxy did the mapping for the hostname in the HTTP request to the actual deployment cluster name in ECE. this mapping logic did not take into account that the Java REST High Level client uses the org.apache.httphost class which creates the hostname as hostname:port_number even when the port number is 443. Since it did not find the mapping because of the 443 therefore the ECE returned a 404 error instead of 200 ok (only way to find this was to look at unencrypted packets at the ha-proxy). Once the mapping logic in ha-proxy was fixed, the mapping was found and the pings are now successfull.
I am trying out Gatling first time.
But when I try to add .inferHtmlResources() i got following error:
value inferHtmlResources is not a member of com.excilys.ebi.gatling.http.config.HttpProtocolConfigurationBuilder
The same mattes when I get to execute part for resources:
value resources is not a member of com.excilys.ebi.gatling.http.request.builder.PostHttpRequestBuilder
Pseodo part of what I am doing
Part 1:
val httpConf = httpConfig.baseURL(urlBase).inferHtmlResources()
Part 2:
.exec(
http("Login")
.post(loginUrl+"/security/signin")
.headers(actionPostHeaders)
.param("""username""", """${loginName}""")
.param("""key""", """${key}""")
.check(status.is(200))
.check(headerRegex("Location", """.*/loggedIn"""))
.resources(http("request_5")
.get(tipUrl + "/authentication/authenticated")
)
).exitHereIfFailed
...
setUp(scn.users(numUsers).ramp(rampUp(numUsers)).protocolConfig(httpConf))
I am not used to Scala. But can it be the version of gatling?
Gatling HTTP protocol settings are created using http method, fe.: http.baseURL("http://localhost").inferHtmlResources(). Moreover I see that you are using some com.excilys.ebi.gatling classes, I think that those are extremely old packages from times when Gatling was just an internal project. Please make sure you are using newest version of gatling from io.gatling package.
I'm using Akka HTTP 10.0.9, but struggling to get my unit tests to have a working Remote Address.
eg unit test:
Get("/").withHeaders(
RawHeader("Remote-Address", "192.168.1.1"), RawHeader("X-Forwarded-For", "192.168.1.1")
) ~> route ~> check {
status must_== StatusCodes.OK
}
And in the web server code:
extractClientIP{ clientAddr =>
complete(s"$clientAddr")
}
When running the app via the command line, the client address is returned correctly. But when run via unit tests, the client address always comes back as "Unknown"
What am I doing wrong?
This is caused by akka http can't handle the RawHeader in test. You can solve it by use the Remote-Address object for set the IP for test:
import akka.http.scaladsl.model.headers.`Remote-Address`
Get().withHeaders(
`Remote-Address`(RemoteAddress(new InetSocketAddress("192.168.1.1", 23)))
)
I have a service using Akka HTTP that I have been doing some load testing on. Under stress, I've found that my service will ocassionally run into StreamTcpException when calling other service endpoints.
I create one flow for each endpoint which is shared by all of my actors. I am using something like this:
//this is done only once
val connectionFlow = Http(sys).outgoingConnection("host_name")
...
//each actor does this
val response = Source.single(HttpRequest(...)).via(connectionFlow).runWith(Sink.head)
I use Apache JMeter to load test my service, and with 40 threads, it typically takes 2000-4000 requests before I see my first error message. With 10 threads, it took me 9000 requests before I saw it.
The message looks like:
akka.stream.StreamTcpException: Tcp command [Connect(<host_here>/<ip_here>,None,List(),Some(10 seconds),true)] failed
I actually have 4 separate flows for 4 different endpoints my service relies on. I usually see StreamTcpException from all four if my service fails.
Anyone have any ideas why this is happening? Thanks in advance.
I face same issue previously, for me it is working as expected.
val httpClient: Flow[HttpRequest, HttpResponse, Future[Http.OutgoingConnection]] = Http().outgoingConnection(host,port,None,connSetts)
Problem place 👍 Source.single(request).via(httpClient)
It is unable to connect my host so I use proxy server for this.
val httpsProxyTransport : ClientTransport = ClientTransport.httpsProxy(InetSocketAddress.createUnresolved("proxyservicesHost", 90))
val connSetts: ClientConnectionSettings = ClientConnectionSettings(system).withTransport(httpsProxyTransport).withIdleTimeout(Duration(180, "second"))
val httpClient: Flow[HttpRequest, HttpResponse, Future[Http.OutgoingConnection]] = Http().outgoingConnection(host,port,None,connSetts)
Then I pass the httpclient in .via(httpclient), it works.