I want use http4s as proxy(like nginx), how to forward all data from my http4s server to another http server?
What I really want do is append a verify function on every request before do forward function. Hopefully like this:
HttpService[IO] {
case request =>
val httpClient: Client[IO] = Http1Client[IO]().unsafeRunSync
if(verifySuccess(request)) { // forward all http data to host2 and
// get a http response.
val result = httpClient.forward(request, "http://host2")
result
} else {
Forbidden //403
}
}
How to do this with http4s and it's client?
Thanks
Updated
with the help of #TheInnerLight, I give it a try with the snippet code:
val httpClient = Http1Client[IO]()
val service: HttpService[IO] = HttpService[IO] {
case req =>
if(true) {
for {
client <- httpClient
newAuthority = req.uri.authority.map(_.copy(host = RegName("scala-lang.org"), port = Some(80)))
proxiedReq = req.withUri(req.uri.copy(authority = newAuthority))
response <- client.fetch(proxiedReq)(IO.pure(_))
} yield response
} else {
Forbidden("Some forbidden message...")
}
}
With a request: http://localhost:28080(http4s server listen at 28080):
but occurred a error:
[ERROR] org.http4s.client.PoolManager:102 - Error establishing client connection for key RequestKey(Scheme(http),localhost)
java.net.ConnectException: Connection refused
at sun.nio.ch.UnixAsynchronousSocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.UnixAsynchronousSocketChannelImpl.finishConnect(UnixAsynchronousSocketChannelImpl.java:252)
at sun.nio.ch.UnixAsynchronousSocketChannelImpl.finish(UnixAsynchronousSocketChannelImpl.java:198)
at sun.nio.ch.UnixAsynchronousSocketChannelImpl.onEvent(UnixAsynchronousSocketChannelImpl.java:213)
at sun.nio.ch.KQueuePort$EventHandlerTask.run(KQueuePort.java:301)
at java.lang.Thread.run(Thread.java:748)
[ERROR] org.http4s.server.service-errors:88 - Error servicing request: GET / from 0:0:0:0:0:0:0:1
java.net.ConnectException: Connection refused
at sun.nio.ch.UnixAsynchronousSocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.UnixAsynchronousSocketChannelImpl.finishConnect(UnixAsynchronousSocketChannelImpl.java:252)
at sun.nio.ch.UnixAsynchronousSocketChannelImpl.finish(UnixAsynchronousSocketChannelImpl.java:198)
at sun.nio.ch.UnixAsynchronousSocketChannelImpl.onEvent(UnixAsynchronousSocketChannelImpl.java:213)
at sun.nio.ch.KQueuePort$EventHandlerTask.run(KQueuePort.java:301)
at java.lang.Thread.run(Thread.java:748)
Latest Version
val httpClient: IO[Client[IO]] = Http1Client[IO]()
override val service: HttpService[IO] = HttpService[IO] {
case req =>
val hostName = "scala-lang.org"
val myPort = 80
if(true) {
val newHeaders = {
val filterHeader = req.headers.filterNot{h =>
h.name == CaseInsensitiveString("Connection") ||
h.name == CaseInsensitiveString("Keep-Alive") ||
h.name == CaseInsensitiveString("Proxy-Authenticate") ||
h.name == CaseInsensitiveString("Proxy-Authorization") ||
h.name == CaseInsensitiveString("TE") ||
h.name == CaseInsensitiveString("Trailer") ||
h.name == CaseInsensitiveString("Transfer-Encoding") ||
h.name == CaseInsensitiveString("Upgrade")
}
filterHeader.put(Header("host", hostName))
}
for {
client <- httpClient
newAuthority = req.uri.authority
.map(_.copy(host = RegName(hostName), port = Some(myPort)))
.getOrElse( Authority(host = RegName(hostName), port = Some(myPort)))
proxiedReq = req.withUri(req.uri.copy(authority = Some(newAuthority)))
.withHeaders(newHeaders)
response <- client.fetch(proxiedReq)(x => IO.pure(x))
} yield {
val rst = response
rst
}
} else {
Forbidden("Some forbidden message...")
}
}
It works fine enough for my REST API web server.
There are some error when proxy scala-lang.org for test:
[ERROR] org.http4s.blaze.pipeline.Stage:226 - Error writing body
org.http4s.InvalidBodyException: Received premature EOF.
How about something like this:
HttpService[IO] {
case req =>
if(verifyRequest(req)) {
for {
client <- Http1Client[IO]()
newHost = "host2"
newAuthority = Authority(host = RegName("host2"), port = Some(80))
proxiedReq =
req.withUri(req.uri.copy(authority = Some(newAuthority)))
.withHeaders(req.headers.put(Header("host", newHost)))
response <- client.fetch(proxiedReq)(IO.pure(_))
} yield response
} else {
Forbidden("Some forbidden message...")
}
}
Note that you should definitely avoid littering your code with calls tounsafeRunSync. You should generally be using it at most once in your program (in Main). In other circumstances, you should focus on lifting the effects into the monad you're working in.
Related
I am trying to create a Scala JDBC program where a connection to Hive is being made. To do this, I wrote the below code.
var HIVECON: Connection = null
def hiveConnection(): Connection = {
val conf = new Configuration()
conf.set("hadoop.security.authentication", "Kerberos")
// DEV System Properties
System.setProperty("java.security.krb5.kdc", "ip-address.ec2.internal");
System.setProperty("java.security.krb5.realm", "DEV.COM");
// DEV System Properties
// DEV loginUserFromKeytab
UserGroupInformation.loginUserFromKeytab("username#DEV.COM", "/home/username/username.keytab");
// DEV loginUserFromKeytab
try {
Class.forName("org.apache.hive.jdbc.HiveDriver")
if(HIVECON == null || HIVECON.isClosed)
HIVECON = DriverManager.getConnection("jdbc:hive2://ip-address.ec2.internal:10500/dbname;principal=hive/ip-address.ec2.internal#DEV.COM", "username","password")
else HIVECON
} catch {
case s:SQLException => s.printStackTrace()
case e:Exception => e.printStackTrace()
}
}
But the code gives a compilation error at these lines:
With the way I wrote, the catch statements are returning UNIT where my method is trying to return CONNECTION. Is there any way to handle the exception better ?
I would handle the Exception in a functional way.
If you do not care about specific Exception use Option:
var HIVECON: Option[Connection] = None
def hiveConnection(): Option[Connection] = {
...
try {
Class.forName("org.apache.hive.jdbc.HiveDriver")
if(HIVECON == None || HIVECON.get.isClosed)
HIVECON = Some(DriverManager.getConnection("jdbc:hive2://ip-address.ec2.internal:10500/dbname;principal=hive/ip-address.ec2.internal#DEV.COM", "username","password"))
HIVECON // return Some(Connection)
} catch {
case s:Exception =>
s.printStackTrace()
None
}
If you care for the Exception use Try:
var HIVECON: Connection = null
def hiveConnection(): Try[Connection] = {
...
Try {
Class.forName("org.apache.hive.jdbc.HiveDriver")
if(HIVECON == null || HIVECON.isClosed)
HIVECON = DriverManager.getConnection("jdbc:hive2://ip-address.ec2.internal:10500/dbname;principal=hive/ip-address.ec2.internal#DEV.COM", "username","password")
HIVECON // return Success(Connection)
}
In case of a Failure it returns Failure(Exception).
See here the Docs: https://docs.scala-lang.org/overviews/scala-book/functional-error-handling.html
Using these parameters:
canada {
hosts = ["dd.weather.gc.ca"]
username = "anonymous"
password = "anonymous"
port = 5671
exchange = "xpublic"
queue = "q_anonymous_gsk"
routingKey = "v02.post.observations.swob-ml.#"
requestedHeartbeat = 300
ssl = true
}
I can connect to a weather service in Canada using NewMotion/Akka, but when I try op-rabbit, I get:
ACCESS_REFUSED - access to exchange 'xpublic' in vhost '/' refused for user 'anonymous'
[INFO] [foo-akka.actor.default-dispatcher-7] [akka://foo/user/$a/connection] akka://foo/user/$a/connection connected to amqp://anonymous#{dd.weather.gc.ca:5671}:5671//
[INFO] [foo-op-rabbit.default-channel-dispatcher-6] [akka://foo/user/$a/connection/$a] akka://foo/user/$a/connection/$a connected
[INFO] [foo-akka.actor.default-dispatcher-4] [akka://foo/user/$a/connection/confirmed-publisher-channel] akka://foo/user/$a/connection/confirmed-publisher-channel connected
[INFO] [foo-akka.actor.default-dispatcher-4] [akka://foo/user/$a/connection/$b] akka://foo/user/$a/connection/$b connected
[ERROR] [foo-akka.actor.default-dispatcher-3] [akka://foo/user/$a/subscription-q_anonymous_gsk-1] Connection related error while trying to re-bind a consumer to q_anonymous_gsk. Waiting in anticipating of a new channel.
...
Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=403, reply-text=ACCESS_REFUSED - access to exchange 'xpublic' in vhost '/' refused for user 'anonymous', class-id=40, method-id=10)
The following works in NewMotion/Akka:
val inQueue = "q_anonymous_gsk"
val inExchange = "xpublic"
val canadaQueue = canadaChannel.queueDeclare(inQueue, false, true, false, null).getQueue
canadaChannel.queueBind(canadaQueue, inExchange, inQueue)
val consumer = new DefaultConsumer(canadaChannel) {
override def handleDelivery(consumerTag: String, envelope: Envelope, properties: BasicProperties, body: Array[Byte]) {
val s = fromBytes(body)
if (republishElsewhere) {
// ...
}
}
}
canadaChannel.basicConsume(canadaQueue, true, consumer)
but using op-rabbit like this:
val inQueue = "q_anonymous_gsk"
val inExchange = "xpublic"
val inRoutingKey = "v02.post.observations.swob-ml.#""
val rabbitCanada: ActorRef = actorSystem.actorOf(Props(classOf[RabbitControl], connParamsCanada))
def runSubscription(): SubscriptionRef = Subscription.run(rabbitCanada) {
channel(qos = 3) {
consume(topic(queue(inQueue), List(inRoutingKey))) {
(body(as[String]) & routingKey) { (msg, key) =>
ack
}
}
}
}
I get the ACCESS_REFUSED error near the top of this post. Why? How do I fix this if I want to use op-rabbit?
Have you tried to use the correct vhost with permission to anonymous user
Edit the Web socket header response sent from server to client.
I am creating a websocket server application using playframework. Right now the websocket response from the server is
taken care by Play. Following is the response header,
Request Header:
(UpgradeToWebSocket,),
(Host,localhost:8083),
(Connection,Upgrade),
(Upgrade,websocket),
(Sec-WebSocket-Version,13),
(Accept-Encoding,gzip, deflate, br),
(Accept-Language,en-US,en;q=0.9),
(Sec-WebSocket-Key,ZvfzpVo3EX4DFA4BRcgRIA==)
def chatSystem(): WebSocket = WebSocket.acceptOrResult[String, String] { request =>
Future.successful{
AuthenticationService.doBasicAuthentication(request.headers) match {
case Results.Ok => Right(ActorFlow.actorRef { out => ChatServiceActor.props(out) })
case _ => Left(Unauthorized)
}
}
}
I want to validate Sec-WebSocket-Protocol if it is present in the request header or add the same with value in the server response if it is not present.
I used the following code:
// Defined at http://tools.ietf.org/html/rfc6455#section-4.2.2
val MagicGuid = "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"
def websocketAcceptForKey(key: String): String = {
val sha1 = MessageDigest.getInstance("sha1")
val salted = key + MagicGuid
val hash = sha1.digest(salted.asciiBytes)
val acceptKey: String = Base64.rfc2045().encodeToString(hash, false)
acceptKey
}
Use it like the following:
val wsKey: Optional[HttpHeader] = request.getHeader("Sec-WebSocket-Key")
val wsAccept = if (wsKey.isPresent) Some(RawHeader("Sec-WebSocket-Accept", websocketAcceptForKey(wsKey.get.value()))) else None
My question is how to use a Buffered stream when using Scalaj-Http.
I have written the following code which is a complete working example that will download a file from Hadoop HDFS using HttpFS. My goal is to handle very large files and this will require using a buffered approach with multiple I/O writes to a local file.
I have not been able to find documentation on how to use a stream with the ScalaJ-Http interface. I am interested in an example for both download and upload that can handle large multi GB files. My code below uses in memory buffering which is appropriate for only prototyping.
import scalaj.http._
import ujson.Js
import java.text.SimpleDateFormat
import java.net.SocketTimeoutException
import java.io.InputStream
import java.io.BufferedOutputStream
import java.io.FileOutputStream
import java.io.FileNotFoundException
object CopyFileFromHdfs {
def main(args: Array[String]) {
val host = "hadoop.example.com"
val user = "root"
var dstFile = ""
var srcFile = ""
val operation = "OPEN"
val port = 14000
System.setProperty("sun.net.http.allowRestrictedHeaders", "true")
if (args.length != 2)
{
println("Error: Missing or too many arguments")
println("Usage: CopyFileFromHdfs <srcfile> <dstfile>")
System.exit(1)
}
srcFile = args(0)
dstFile = args(1)
// ********************************************************************************
// Create the URL string that we will use to connect to Hadoop HttpFS
//
// The string will look like this:
// http://root#123.456.789.012:14000/webhdfs/v1/?user.name=root&op=OPEN
// ********************************************************************************
val url = makeHttpfsUrl(host, user, srcFile, operation, port)
// ********************************************************************************
// Using HTTP, call the HttpFS server
//
// Exceptions:
// java.net.SocketTimeoutException
// java.net.UnknownHostException
// java.lang.IllegalArgumentException
// Remote Exceptions:
// java.io.FileNotFoundException
// com.sun.jersey.api.NotFoundException
// ********************************************************************************
try {
var response = Http(url)
.timeout(connTimeoutMs = 1000, readTimeoutMs = 5000)
.asBytes
// ********************************************************************************
// Check for an error. We are expecting an HTTP 200 response
// ********************************************************************************
if (response.code < 200 || response.code > 299)
{
val data = ujson.read(response.body)
printf("Error: Cannot download file: %s\n", dstFile)
println(removeQuotes(data("RemoteException")("message").str))
println(removeQuotes(data("RemoteException")("exception").str))
System.exit(1)
}
val is = new FileOutputStream(dstFile)
val bs = new BufferedOutputStream(is)
bs.write(response.body, 0, response.body.length)
bs.close()
is.close()
} catch {
case e: SocketTimeoutException => {
printf("Error: Cannot connect to host %s on port %d\n", host, port)
println(e)
System.exit(1);
}
case e: Exception => {
printf("Error (other): Cannot download file %s\n", srcFile)
println(e)
System.exit(1);
}
}
printf("Success: File downloaded. %s -> %s\n", srcFile, dstFile)
System.exit(0)
}
// ********************************************************************************
// The Json strings are surrounded by quotes.
// This function will remove them (only at the start and the end).
// ********************************************************************************
def removeQuotes(str: String): String = {
// This expression will delete quotes at the beginning and end of a string
return str.replaceAll("^\"|\"$", "");
}
// ********************************************************************************
// Create the URL string that we will use to connect to Hadoop HttpFS
//
// The string will look like this:
// http://root#123.456.789.012:14000/webhdfs/v1/?user.name=root&op=LISTSTATUS
// ********************************************************************************
def makeHttpfsUrl(
host: String,
user: String,
hdfsPath: String,
operation: String,
port: Integer) : String = {
var url = "http://" + user + "#" + host + ":" + port.toString + "/webhdfs/v1"
if (hdfsPath(0) == '/')
url += hdfsPath
else
url += "/" + hdfsPath
url += "?user.name=" + user + "&op=" + operation
return url
}
}
How can I close a WebSocket connection? The example on the documentation works if you want to close it immediately.
But how about the following case: I want to close the connection when some condition occurs in the future. For instance, when I receive a certain message from the client.
def indexWS = WebSocket.using[String] {
request => {
var channel: Option[Concurrent.Channel[String]] = None
var outEnumerator: Enumerator[String] = Concurrent.unicast(c => channel = Some(c))
val myIteratee: Iteratee[String, Unit] = Iteratee.foreach[String] {gotString => {
// received a string from the client
if (gotString == "close_me") {
// outEnumerator = Enumerator.eof // doesn't work
// outEnumerator >>> Enumerator.eof // doesn't work
}
}}
(myIteratee, outEnumerator)
}
}
Thank you for your help!
I got it: I had to go through the channel that I opened at
var outEnumerator: Enumerator[String] = Concurrent.unicast(c => channel = Some(c))
and the commented out block would become
if (gotString == "close_me") {
channel.foreach(_.eofAndEnd())
}
which will push an EOF through the enumerator and close the connection.