I am trying to create a Scala JDBC program where a connection to Hive is being made. To do this, I wrote the below code.
var HIVECON: Connection = null
def hiveConnection(): Connection = {
val conf = new Configuration()
conf.set("hadoop.security.authentication", "Kerberos")
// DEV System Properties
System.setProperty("java.security.krb5.kdc", "ip-address.ec2.internal");
System.setProperty("java.security.krb5.realm", "DEV.COM");
// DEV System Properties
// DEV loginUserFromKeytab
UserGroupInformation.loginUserFromKeytab("username#DEV.COM", "/home/username/username.keytab");
// DEV loginUserFromKeytab
try {
Class.forName("org.apache.hive.jdbc.HiveDriver")
if(HIVECON == null || HIVECON.isClosed)
HIVECON = DriverManager.getConnection("jdbc:hive2://ip-address.ec2.internal:10500/dbname;principal=hive/ip-address.ec2.internal#DEV.COM", "username","password")
else HIVECON
} catch {
case s:SQLException => s.printStackTrace()
case e:Exception => e.printStackTrace()
}
}
But the code gives a compilation error at these lines:
With the way I wrote, the catch statements are returning UNIT where my method is trying to return CONNECTION. Is there any way to handle the exception better ?
I would handle the Exception in a functional way.
If you do not care about specific Exception use Option:
var HIVECON: Option[Connection] = None
def hiveConnection(): Option[Connection] = {
...
try {
Class.forName("org.apache.hive.jdbc.HiveDriver")
if(HIVECON == None || HIVECON.get.isClosed)
HIVECON = Some(DriverManager.getConnection("jdbc:hive2://ip-address.ec2.internal:10500/dbname;principal=hive/ip-address.ec2.internal#DEV.COM", "username","password"))
HIVECON // return Some(Connection)
} catch {
case s:Exception =>
s.printStackTrace()
None
}
If you care for the Exception use Try:
var HIVECON: Connection = null
def hiveConnection(): Try[Connection] = {
...
Try {
Class.forName("org.apache.hive.jdbc.HiveDriver")
if(HIVECON == null || HIVECON.isClosed)
HIVECON = DriverManager.getConnection("jdbc:hive2://ip-address.ec2.internal:10500/dbname;principal=hive/ip-address.ec2.internal#DEV.COM", "username","password")
HIVECON // return Success(Connection)
}
In case of a Failure it returns Failure(Exception).
See here the Docs: https://docs.scala-lang.org/overviews/scala-book/functional-error-handling.html
Related
My Jenkins pipeline currently successfully invokes Bitbucket REST API by invoking curl in a shell, as in the code below:
// Jenkinsfile
#Library('my-sandbox-libs#dev') my_lib
pipeline {
agent any
stages {
stage( "1" ) { steps { script { echo "hello" } } }
stage( "2" ) {
steps {
script {
log = new org.log.Log()
def cred_id = "bitbucket_cred_id"
def url_base = "https://bitbucket.company.com"
def commit = "76136485c45df256a62cbc2c3c5f1f3efcc86258"
def status =
//"INPROGRESS",
//"SUCCESSFUL",
"FAILED"
def viz_url = "https://path/to/nowhere"
try {
my_lib.notifyBitbucketBuildStatus(cred_id,
url_base,
commit,
status,
"foo",
42,
viz_url,
log)
}
}
}
}
stage( "3" ) { steps { script { echo "world" } } }
}
post { always { script { echo log.asJsonString() } } }
}
import groovy.json.JsonOutput
def notifyBitbucketBuildStatus(cred_id,
url_base,
commit,
build_state,
build_info_name,
build_info_number,
viz_url,
log) {
def rest_path = "rest/build-status/1.0/commits"
def dict = [:]
dict.state = build_state
dict.key = "${build_info_name}_${build_info_number}"
dict.url = viz_url
withCredentials([string(credentialsId: cred_id,
variable: 'auth_token')]) {
def cmd = "curl -f -L " +
"-H \"Authorization: Bearer ${auth_token}\" " +
"-H \"Content-Type:application/json\" " +
"-X POST ${url_base}/${rest_path}/${commit} " +
"-d \'${JsonOutput.toJson(dict)}\'"
if ( 0 != sh(script: cmd, returnStatus: true) ) {
log.warn("Failed updating build status with Bitbucket")
}
}
}
I would like to refactor function notifyBitbucketBuildStatus() to use a "native" Groovy-language solution, rather than invoking curl in a shell. I read the following on this topic:
https://www.baeldung.com/groovy-web-services
Groovy built-in REST/HTTP client?
...based on which I thought the refactored function would look like this:
def notifyBitbucketBuildStatus(cred_id,
url_base,
commit,
build_state,
build_info_name,
build_info_number,
viz_url,
log) {
def rest_path = "rest/build-status/1.0/commits"
def dict = [:]
dict.state = build_state
dict.key = "${build_info_name}_${build_info_number}"
dict.url = viz_url
def req = new URL("${url_base}/${rest_path}/${commit}").openConnection()
req.setRequestMethod("POST")
req.setDoOutput(true)
req.setRequestProperty("Content-Type", "application/json")
withCredentials([string(credentialsId: cred_id,
variable: 'auth_token')]) {
req.setRequestProperty("Authorization", "Bearer ${auth_token}")
}
def msg = JsonOutput.toJson(dict)
req.getOutputStream().write(msg.getBytes("UTF-8"));
if ( 200 != req.getResponseCode() ) {
log.warn("Failed updating build status with Bitbucket")
}
}
...but this generates the exception java.io.NotSerializableException: sun.net.www.protocol.https.HttpsURLConnectionImpl
That "not serializable" made me think the error had something to do with a failure to transform something to a string, so I also tried this, but it did not change the error:
def msg = JsonOutput.toJson(dict).toString()
What is wrong with the refactored code that uses class URL, and what is the right way to use it to invoke the REST API?
For the life of me, I can't see what's different between the above and the linked Stack Overflow Q&A, and my inexperience with the language is such that I rely largely on adapting existing example.
Solution
I would highly suggest you use the HTTP Request and the Pipeline Steps Utility plugin for this. You can then use those steps in a Groovy script as follows
node('master') {
withCredentials([string(credentialsId: cred_id, variable: 'auth_token')]) {
def response = httpRequest url: "https://jsonplaceholder.typicode.com/todos", customHeaders: [[name: 'Authorization', value: "Bearer ${auth_token}"]]
}
if( response.status != 200 ) {
error("Service returned a ${response.status}")
}
def json = readJSON text: response.content
println "The User ID is ${json[0]['userId']}"
println "The follow json obj is ${json}"
}
Obviously you can modify the code if you want to build a method, and you will need to update with the appropriate URL.
I found a sucky and unsatisfying answer - but an answer nonetheless - that I posted here: https://stackoverflow.com/a/69486890/5437543
I hate that solution because it would appear to demonstrate that the Jenkins/Groovy language itself imposes an artificial contrivance on how I can organize my code. Effectively, I am prevented from doing
// Jenkinsfile
#Library('my-sandbox-libs#dev') my_lib
pipeline {
agent any
stages {
stage( "1" ) { steps { script { my_lib.func() } } }
}
}
// vars/my_lib.groovy
def func() {
def post = new URL("https://whatever").openConnection();
...
withCredentials([string(credentialsId: cred_id,
variable: 'auth_token')]) {
req.setRequestProperty("Authorization", "Bearer ${auth_token}")
}
...
}
...and I am forced to do
// Jenkinsfile
#Library('my-sandbox-libs#dev') my_lib
pipeline {
agent any
stages {
stage( "1" ) { steps { script { my_lib.func(my_lib.getCred()) } } }
}
}
// vars/my_lib.groovy
def getCred() {
withCredentials([string(credentialsId: cred_id,
variable: 'auth_token')]) {
return auth_token
}
}
def func(auth_token) {
def post = new URL("https://whatever").openConnection();
...
req.setRequestProperty("Authorization", "Bearer ${auth_token}")
...
}
Extremely dissatisfying conclusion. I hope another answerer can point out a solution that doesn't rely on this contrived code organization.
This question is a follow-up to Groovy/Jenkins: how to refactor sh(script:"curl ...") to URL?.
Though I intend to try invoking REST API with HTTP Request - as the one answerer has suggested - I'd like to also specifically learn about/understand the original problem RE: non-serializability.
I've reduced the code in the linked Q&A to more minimally demonstrate the problem.
This code:
#Library('my-sandbox-libs#dev') sandbox_lib
pipeline {
agent any
stages {
stage( "1" ) { steps { script { echo "hello" } } }
stage( "2" ) {
steps {
script {
try {
my_lib.v4()
}
catch(Exception e) {
echo "Jenkinsfile: ${e.toString()}"
throw e
}
}
}
}
stage( "3" ) { steps { script { echo "world" } } }
}
}
// vars/my_lib.groovy
import groovy.json.JsonOutput
def v4() {
def post = new URL("https://bitbucket.company.com/rest/build-status/1.0/commits/86c36485c0cbf956a62cbc1c370f1f3eecc8665d").openConnection();
def dict = [:]
dict.state = "INPROGRESS"
dict.key = "foo_42"
dict.url = "http://url/to/nowhere"
def message = JsonOutput.toJson(dict).toString()
post.setRequestMethod("POST")
post.setDoOutput(true)
post.setRequestProperty("Content-Type", "application/json")
post.getOutputStream().write(message.getBytes("UTF-8"));
def postRC = post.getResponseCode();
println(postRC);
if (postRC.equals(200)) {
println(post.getInputStream().getText());
}
}
...generates HTTP error 401. This is expected, because it invokes a Bitbucket REST API without necessary authentication.
What's missing is the "Bearer: xyz" secret-text. In the past, I've gotten this secret-text using Jenkins'/Groovy's withCredentials function, as in the modified function v4() below:
// vars/my_lib.groovy
import groovy.json.JsonOutput
def v4() {
def post = new URL("https://bitbucket.company.com/rest/build-status/1.0/commits/86c36485c0cbf956a62cbc1c370f1f3eecc8665d").openConnection();
def dict = [:]
dict.state = "INPROGRESS"
dict.key = "foo_42"
dict.url = "http://url/to/nowhere"
def message = JsonOutput.toJson(dict).toString()
post.setRequestMethod("POST")
post.setDoOutput(true)
post.setRequestProperty("Content-Type", "application/json")
withCredentials([string(credentialsId: 'bitbucket_cred_id',
variable: 'auth_token')]) {
post.setRequestProperty("Authorization", "Bearer " + auth_token)
}
post.getOutputStream().write(message.getBytes("UTF-8"));
def postRC = post.getResponseCode();
println(postRC);
if (postRC.equals(200)) {
println(post.getInputStream().getText());
}
}
...but I've discovered that the addition of that withCredentials-block, specifically, introduces a java.io.NotSerializableException: sun.net.www.protocol.https.HttpsURLConnectionImpl runtime exception.
At the linked Stack Overflow Q&A, one commenter has informed me that the problem has to do with serializable vs. non-serializable objects in this function. But I'm having a hard time understanding what non-serializable object is introduced by use of the withCredentials-block.
As a point of reference, use of the same withCredentials-block works just fine when I invoke the same REST API using curl instead of "native" Jenkins/Groovy functions. I.e. the following code works fine:
def v1() {
def dict = [:]
dict.state = "INPROGRESS"
dict.key = "foo_42"
dict.url = "http://url/to/nowhere"
withCredentials([string(credentialsId: 'bitbucket_cred_id',
variable: 'auth_token')]) {
def cmd = "curl -f -L " +
"-H \"Authorization: Bearer ${auth_token}\" " +
"-H \"Content-Type:application/json\" " +
"-X POST https://bitbucket.company.com/rest/build-status/1.0/commits/86c36485c0cbf956a62cbc1c370f1f3eecc8665d " +
"-d \'${JsonOutput.toJson(dict)}\'"
sh(script: cmd, returnStatus: true)
}
}
So, in summary, this question is why does withCredentials() introduce non-serializable objects (and what is the non-serializable object), which causes NonSerializableException exceptions with use of URL and/or HttpURLConnection, and does one work around this?
I'm not unwilling to use a different solution, such as httpRequest objects, but this question is about learning the nature of this specific problem, and how to work around it with the existing objects, i.e. URL and HttpURLConnection objects.
Update: As it turns out, I'm unable to use the suggested alternate solution using httpRequest objects, because the HTTP Request plugin is only available to Jenkins versions 2.222.4 or newer, which our Jenkins does not meet. It's outside my privilege to update our Jenkins version, and basically I'll need to assume an inability to upgrade Jenkins.
Our Jenkins version is 2.190.3, which has Groovy version 2.4.12.
When having such issues I usually try to put the "low level" code that calls down to Groovy/Java API, into #NonCPS functions. Objects in such functions don't need to be serializable, so we can freely use any Groovy/Java API.
Background reading: Pipeline CPS Method Mismatches
Make sure you don't call any Jenkins pipeline steps from #NonCPS functions (nor any other function that is not marked #NonCPS) - such code could silently fail or do unexpected stuff! (There are some "safe" functions like echo though.)
As withCredentials is a pipeline step, it has to be called from a "regular" function (not marked as #NonCPS), one level up the call chain.
Note that I'm passing auth_token as an argument to v4_internal. If you need other Jenkins variables in the code, these should also be passed as arguments.
// vars/my_lib.groovy
import groovy.json.JsonOutput
def v4() {
withCredentials([string(credentialsId: 'bitbucket_cred_id',
variable: 'auth_token')]) {
v4_internal(auth_token)
}
}
#NonCPS
def v4_internal( def auth_token ) {
def post = new URL("https://bitbucket.company.com/rest/build-status/1.0/commits/86c36485c0cbf956a62cbc1c370f1f3eecc8665d").openConnection();
def dict = [:]
dict.state = "INPROGRESS"
dict.key = "foo_42"
dict.url = "http://url/to/nowhere"
def message = JsonOutput.toJson(dict).toString()
post.setRequestMethod("POST")
post.setDoOutput(true)
post.setRequestProperty("Content-Type", "application/json")
post.setRequestProperty("Authorization", "Bearer " + auth_token)
post.getOutputStream().write(message.getBytes("UTF-8"));
def postRC = post.getResponseCode();
println(postRC);
if (postRC.equals(200)) {
println(post.getInputStream().getText());
}
}
This sucks - and I hope someone comes along with a better answer - but it looks like the only way I can pull this off is as follows:
#Library('my-sandbox-libs#dev') sandbox_lib
pipeline {
agent any
stages {
stage( "1" ) { steps { script { echo "hello" } } }
stage( "2" ) {
steps {
script {
try {
my_lib.v5(my_lib.getBitbucketCred())
}
catch(Exception e) {
echo "Jenkinsfile: ${e.toString()}"
throw e
}
}
}
}
stage( "3" ) { steps { script { echo "world" } } }
}
}
// vars/my_lib.groovy
import groovy.json.JsonOutput
def getBitbucketCred() {
withCredentials([string(credentialsId: 'bitbucket_cred_id',
variable: 'auth_token')]) {
return auth_token
}
}
def v5(auth_token) {
def post = new URL("https://bitbucket.company.com/rest/build-status/1.0/commits/86c36485c0cbf956a62cbc1c370f1f3eecc8665d").openConnection();
def dict = [:]
dict.state = "INPROGRESS"
dict.key = "foo_42"
dict.url = "http://url/to/nowhere"
def message = JsonOutput.toJson(dict).toString()
post.setRequestMethod("POST")
post.setDoOutput(true)
post.setRequestProperty("Content-Type", "application/json")
req.setRequestProperty("Authorization", "Bearer " + auth_token)
post.getOutputStream().write(message.getBytes("UTF-8"));
def postRC = post.getResponseCode();
println(postRC);
if (postRC.equals(200)) {
println(post.getInputStream().getText());
}
}
Specifically, I must invoke withCredentials() completely separate from the scope of the function that uses URL and/or HttpURLConnection.
I don't know if this is considered acceptable in Jenkins/Groovy, but I'm dissatisfied by the inability to call withCredentials() from the v5() function itself. I'm also unable to call a withCredentials()-wrapper function from v5().
When I was trying to call withCredentials() either directly in v5() or from a wrapper function called by v5(), I tried every combination of #NonCPS between v5() and the wrapper function, and that didn't work. I also tried explicitly setting the URL and HttpURLConnection objects to null before the end of the function (as suggested at Jenkins/Groovy: Why does withCredentials() introduce a NotSerializableException exception?), and neither did that work.
I'd be disappointed in Jenkins/Groovy if this is the only solution. This feels like an artificial limit on how one can choose to organize his code.
Updating with more detail in response to #daggett:
RE: calling withCredentials() directly from my_lib.v5() or calling it from a wrapper function, let's start with mylib.groovy set up as follows (let me also take the opportunity to give the functions better names):
def withCredWrapper() {
withCredentials([string(credentialsId: 'bitbucket_cred_id',
variable: 'auth_token')]) {
return auth_token
}
}
def callRestFunc() {
def post = new URL("https://bitbucket.company.com/rest/build-status/1.0/commits/86c36485c0cbf956a62cbc1c370f1f3eecc8665d").openConnection();
def dict = [:]
dict.state = "INPROGRESS"
dict.key = "foo_42"
dict.url = "http://url/to/nowhere"
def message = JsonOutput.toJson(dict).toString()
post.setRequestMethod("POST")
post.setDoOutput(true)
post.setRequestProperty("Content-Type", "application/json")
// version 1:
withCredentials([string(credentialsId: 'bb_auth_bearer_token_cred_id',
variable: 'auth_token')]) {
post.setRequestProperty("Authorization", "Bearer " + auth_token)
}
// version 2:
//post.setRequestProperty("Authorization", "Bearer " + withCredWrapper())
post.getOutputStream().write(message.getBytes("UTF-8"));
def postRC = post.getResponseCode();
println(postRC);
if (postRC.equals(200)) {
println(post.getInputStream().getText());
}
}
With the above code, function callRestFunc() can either call withCredentials() directly, as above, or indirectly by the wrapper function withCredWrapper(), i.e.:
...
// version 1:
//withCredentials([string(credentialsId: 'bb_auth_bearer_token_cred_id',
// variable: 'auth_token')]) {
// post.setRequestProperty("Authorization", "Bearer " + auth_token)
//}
// version 2:
post.setRequestProperty("Authorization", "Bearer " + withCredWrapper())
...
Further, #NonCPS can be applied to one of withCredWrapper() or callRestFunc(), both, or neither.
Below are the specific failures with all 8 combinations thereof:
1.
def withCredWrapper() {
...
}
def callRestFunc() {
...
// version 1:
withCredentials(...)
...
}
Failure: Jenkinsfile: java.io.NotSerializableException: sun.net.www.protocol.https.HttpsURLConnectionImpl
2.
def withCredWrapper() {
...
}
#NonCPS
def callRestFunc() {
...
// version 1:
withCredentials(...)
...
}
Failure: expected to call my_lib.callRestFunc but wound up catching withCredentials; see: https://jenkins.io/redirect/pipeline-cps-method-mismatches/ Masking supported pattern matches of $auth_token, Jenkinsfile: java.io.NotSerializableException: sun.net.www.protocol.https.HttpsURLConnectionImpl
3.
#NonCPS
def withCredWrapper() {
...
}
def callRestFunc() {
...
// version 1:
withCredentials(...)
...
}
Failure: Jenkinsfile: java.io.NotSerializableException: sun.net.www.protocol.https.HttpsURLConnectionImpl
4.
#NonCPS
def withCredWrapper() {
...
}
#NonCPS
def callRestFunc() {
...
// version 1:
withCredentials(...)
...
}
Failure: expected to call my_lib.callRestFunc but wound up catching withCredentials; see: https://jenkins.io/redirect/pipeline-cps-method-mismatches/ Masking supported pattern matches of $auth_token, Jenkinsfile: java.io.NotSerializableException: sun.net.www.protocol.https.HttpsURLConnectionImpl
5.
def withCredWrapper() {
...
}
def callRestFunc() {
...
// version 2:
post.setRequestProperty("Authorization", "Bearer " + withCredWrapper())
...
}
Failure: Jenkinsfile: java.io.NotSerializableException: sun.net.www.protocol.https.HttpsURLConnectionImpl
6.
def withCredWrapper() {
...
}
#NonCPS
def callRestFunc() {
...
// version 2:
post.setRequestProperty("Authorization", "Bearer " + withCredWrapper())
...
}
Failure: expected to call my_lib.callRestFunc but wound up catching my_lib.withCredWrapper; see: https://jenkins.io/redirect/pipeline-cps-method-mismatches/
7.
#NonCPS
def withCredWrapper() {
...
}
def callRestFunc() {
...
// version 2:
post.setRequestProperty("Authorization", "Bearer " + withCredWrapper())
...
}
Failure: expected to call my_lib.withCredWrapper but wound up catching withCredentials; see: https://jenkins.io/redirect/pipeline-cps-method-mismatches/ Masking supported pattern matches of $auth_token, Jenkinsfile: java.io.NotSerializableException: sun.net.www.protocol.https.HttpsURLConnectionImpl
8.
#NonCPS
def withCredWrapper() {
...
}
#NonCPS
def callRestFunc() {
...
// version 2:
post.setRequestProperty("Authorization", "Bearer " + withCredWrapper())
...
}
Failure: expected to call my_lib.callRestFunc but wound up catching withCredentials; see: https://jenkins.io/redirect/pipeline-cps-method-mismatches/
My question is how to use a Buffered stream when using Scalaj-Http.
I have written the following code which is a complete working example that will download a file from Hadoop HDFS using HttpFS. My goal is to handle very large files and this will require using a buffered approach with multiple I/O writes to a local file.
I have not been able to find documentation on how to use a stream with the ScalaJ-Http interface. I am interested in an example for both download and upload that can handle large multi GB files. My code below uses in memory buffering which is appropriate for only prototyping.
import scalaj.http._
import ujson.Js
import java.text.SimpleDateFormat
import java.net.SocketTimeoutException
import java.io.InputStream
import java.io.BufferedOutputStream
import java.io.FileOutputStream
import java.io.FileNotFoundException
object CopyFileFromHdfs {
def main(args: Array[String]) {
val host = "hadoop.example.com"
val user = "root"
var dstFile = ""
var srcFile = ""
val operation = "OPEN"
val port = 14000
System.setProperty("sun.net.http.allowRestrictedHeaders", "true")
if (args.length != 2)
{
println("Error: Missing or too many arguments")
println("Usage: CopyFileFromHdfs <srcfile> <dstfile>")
System.exit(1)
}
srcFile = args(0)
dstFile = args(1)
// ********************************************************************************
// Create the URL string that we will use to connect to Hadoop HttpFS
//
// The string will look like this:
// http://root#123.456.789.012:14000/webhdfs/v1/?user.name=root&op=OPEN
// ********************************************************************************
val url = makeHttpfsUrl(host, user, srcFile, operation, port)
// ********************************************************************************
// Using HTTP, call the HttpFS server
//
// Exceptions:
// java.net.SocketTimeoutException
// java.net.UnknownHostException
// java.lang.IllegalArgumentException
// Remote Exceptions:
// java.io.FileNotFoundException
// com.sun.jersey.api.NotFoundException
// ********************************************************************************
try {
var response = Http(url)
.timeout(connTimeoutMs = 1000, readTimeoutMs = 5000)
.asBytes
// ********************************************************************************
// Check for an error. We are expecting an HTTP 200 response
// ********************************************************************************
if (response.code < 200 || response.code > 299)
{
val data = ujson.read(response.body)
printf("Error: Cannot download file: %s\n", dstFile)
println(removeQuotes(data("RemoteException")("message").str))
println(removeQuotes(data("RemoteException")("exception").str))
System.exit(1)
}
val is = new FileOutputStream(dstFile)
val bs = new BufferedOutputStream(is)
bs.write(response.body, 0, response.body.length)
bs.close()
is.close()
} catch {
case e: SocketTimeoutException => {
printf("Error: Cannot connect to host %s on port %d\n", host, port)
println(e)
System.exit(1);
}
case e: Exception => {
printf("Error (other): Cannot download file %s\n", srcFile)
println(e)
System.exit(1);
}
}
printf("Success: File downloaded. %s -> %s\n", srcFile, dstFile)
System.exit(0)
}
// ********************************************************************************
// The Json strings are surrounded by quotes.
// This function will remove them (only at the start and the end).
// ********************************************************************************
def removeQuotes(str: String): String = {
// This expression will delete quotes at the beginning and end of a string
return str.replaceAll("^\"|\"$", "");
}
// ********************************************************************************
// Create the URL string that we will use to connect to Hadoop HttpFS
//
// The string will look like this:
// http://root#123.456.789.012:14000/webhdfs/v1/?user.name=root&op=LISTSTATUS
// ********************************************************************************
def makeHttpfsUrl(
host: String,
user: String,
hdfsPath: String,
operation: String,
port: Integer) : String = {
var url = "http://" + user + "#" + host + ":" + port.toString + "/webhdfs/v1"
if (hdfsPath(0) == '/')
url += hdfsPath
else
url += "/" + hdfsPath
url += "?user.name=" + user + "&op=" + operation
return url
}
}
I want use http4s as proxy(like nginx), how to forward all data from my http4s server to another http server?
What I really want do is append a verify function on every request before do forward function. Hopefully like this:
HttpService[IO] {
case request =>
val httpClient: Client[IO] = Http1Client[IO]().unsafeRunSync
if(verifySuccess(request)) { // forward all http data to host2 and
// get a http response.
val result = httpClient.forward(request, "http://host2")
result
} else {
Forbidden //403
}
}
How to do this with http4s and it's client?
Thanks
Updated
with the help of #TheInnerLight, I give it a try with the snippet code:
val httpClient = Http1Client[IO]()
val service: HttpService[IO] = HttpService[IO] {
case req =>
if(true) {
for {
client <- httpClient
newAuthority = req.uri.authority.map(_.copy(host = RegName("scala-lang.org"), port = Some(80)))
proxiedReq = req.withUri(req.uri.copy(authority = newAuthority))
response <- client.fetch(proxiedReq)(IO.pure(_))
} yield response
} else {
Forbidden("Some forbidden message...")
}
}
With a request: http://localhost:28080(http4s server listen at 28080):
but occurred a error:
[ERROR] org.http4s.client.PoolManager:102 - Error establishing client connection for key RequestKey(Scheme(http),localhost)
java.net.ConnectException: Connection refused
at sun.nio.ch.UnixAsynchronousSocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.UnixAsynchronousSocketChannelImpl.finishConnect(UnixAsynchronousSocketChannelImpl.java:252)
at sun.nio.ch.UnixAsynchronousSocketChannelImpl.finish(UnixAsynchronousSocketChannelImpl.java:198)
at sun.nio.ch.UnixAsynchronousSocketChannelImpl.onEvent(UnixAsynchronousSocketChannelImpl.java:213)
at sun.nio.ch.KQueuePort$EventHandlerTask.run(KQueuePort.java:301)
at java.lang.Thread.run(Thread.java:748)
[ERROR] org.http4s.server.service-errors:88 - Error servicing request: GET / from 0:0:0:0:0:0:0:1
java.net.ConnectException: Connection refused
at sun.nio.ch.UnixAsynchronousSocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.UnixAsynchronousSocketChannelImpl.finishConnect(UnixAsynchronousSocketChannelImpl.java:252)
at sun.nio.ch.UnixAsynchronousSocketChannelImpl.finish(UnixAsynchronousSocketChannelImpl.java:198)
at sun.nio.ch.UnixAsynchronousSocketChannelImpl.onEvent(UnixAsynchronousSocketChannelImpl.java:213)
at sun.nio.ch.KQueuePort$EventHandlerTask.run(KQueuePort.java:301)
at java.lang.Thread.run(Thread.java:748)
Latest Version
val httpClient: IO[Client[IO]] = Http1Client[IO]()
override val service: HttpService[IO] = HttpService[IO] {
case req =>
val hostName = "scala-lang.org"
val myPort = 80
if(true) {
val newHeaders = {
val filterHeader = req.headers.filterNot{h =>
h.name == CaseInsensitiveString("Connection") ||
h.name == CaseInsensitiveString("Keep-Alive") ||
h.name == CaseInsensitiveString("Proxy-Authenticate") ||
h.name == CaseInsensitiveString("Proxy-Authorization") ||
h.name == CaseInsensitiveString("TE") ||
h.name == CaseInsensitiveString("Trailer") ||
h.name == CaseInsensitiveString("Transfer-Encoding") ||
h.name == CaseInsensitiveString("Upgrade")
}
filterHeader.put(Header("host", hostName))
}
for {
client <- httpClient
newAuthority = req.uri.authority
.map(_.copy(host = RegName(hostName), port = Some(myPort)))
.getOrElse( Authority(host = RegName(hostName), port = Some(myPort)))
proxiedReq = req.withUri(req.uri.copy(authority = Some(newAuthority)))
.withHeaders(newHeaders)
response <- client.fetch(proxiedReq)(x => IO.pure(x))
} yield {
val rst = response
rst
}
} else {
Forbidden("Some forbidden message...")
}
}
It works fine enough for my REST API web server.
There are some error when proxy scala-lang.org for test:
[ERROR] org.http4s.blaze.pipeline.Stage:226 - Error writing body
org.http4s.InvalidBodyException: Received premature EOF.
How about something like this:
HttpService[IO] {
case req =>
if(verifyRequest(req)) {
for {
client <- Http1Client[IO]()
newHost = "host2"
newAuthority = Authority(host = RegName("host2"), port = Some(80))
proxiedReq =
req.withUri(req.uri.copy(authority = Some(newAuthority)))
.withHeaders(req.headers.put(Header("host", newHost)))
response <- client.fetch(proxiedReq)(IO.pure(_))
} yield response
} else {
Forbidden("Some forbidden message...")
}
}
Note that you should definitely avoid littering your code with calls tounsafeRunSync. You should generally be using it at most once in your program (in Main). In other circumstances, you should focus on lifting the effects into the monad you're working in.
How can I close a WebSocket connection? The example on the documentation works if you want to close it immediately.
But how about the following case: I want to close the connection when some condition occurs in the future. For instance, when I receive a certain message from the client.
def indexWS = WebSocket.using[String] {
request => {
var channel: Option[Concurrent.Channel[String]] = None
var outEnumerator: Enumerator[String] = Concurrent.unicast(c => channel = Some(c))
val myIteratee: Iteratee[String, Unit] = Iteratee.foreach[String] {gotString => {
// received a string from the client
if (gotString == "close_me") {
// outEnumerator = Enumerator.eof // doesn't work
// outEnumerator >>> Enumerator.eof // doesn't work
}
}}
(myIteratee, outEnumerator)
}
}
Thank you for your help!
I got it: I had to go through the channel that I opened at
var outEnumerator: Enumerator[String] = Concurrent.unicast(c => channel = Some(c))
and the commented out block would become
if (gotString == "close_me") {
channel.foreach(_.eofAndEnd())
}
which will push an EOF through the enumerator and close the connection.