I am using httpClient lib to do some REST API calls using scala. I am able to post the data.
I am using following code to read the content. However, When I run on Spark Databricks Cluster it give me error.
val entity = response.getEntity
var content = ""
if (entity != null) {
val inputStream = entity.getContent
content = io.Source.fromInputStream(inputStream).getLines.mkString
inputStream.close()
}
Error
error: object Source is not a member of package io
content = io.Source.fromInputStream(inputStream).getLines.mkString
is there a way I can fix this error, or a different way to read HTTP response content.
Please try to import scala.io.Source
OR use the completed package name like this:
content = scala.io.Source.fromInputStream(inputStream).getLines.mkString
Related
I am working on usecase where i have to read (id , token and type values delimited with pipe )from kafka topic using spark streaming job.
Using the above values, i have to make rest api call to pull data belongs to id.
I am using below code which i am able to get the data in json format.
val httpget = new HttpGet("///////");
val httpParams = httpClient.getParams
HttpConnectionParams.setConnectionTimeout(httpParams,timeoutConnection)
val httpResponse = httpClient.execute(httpget)
println(httpResponse.getStatusLine().getStatusCode)
val entity = httpResponse.getEntity()
var content = ""
if (entity != null) {
val inputStream = entity.getContent()
content = scala.io.Source.fromInputStream(inputStream).getLines.mkString
inputStream.close
}
My problem here is using this logic in dStream.foreachRDD() block and parsing the json content effectively.
Any help will be appreciated.
I have a REST client that returns me a list of systems.
I need this list to be as a parameter for a jenkins job.
I think I need Actice Choices Parameter plugin with Groovy and HTTPBuilder in order to do that.
What do you guys think?
I did not find a way to install HTTPBuilder into Jenkins.
Is there any other way that you guys think it is possible?
I have run into the same problem trying to parse parameters via groovy script. Arun's answer did not work for me. However, I have managed to get it to work using the following:
import java.io.BufferedReader
import java.io.InputStreamReader
import java.io.OutputStreamWriter
import java.net.URL
import java.net.URLConnection
import groovy.json.JsonSlurper
def choices = []
def url = new URL("some.data.url")
def conn = url.openConnection()
conn.setDoOutput(true)
def reader = new BufferedReader(new InputStreamReader(conn.getInputStream()))
def results = new JsonSlurper().parseText(reader.getText());
reader.close()
results.each { data -> choices.push(data.field) }
return choices.sort()
First paste the JSON body snapshot output -or whatever your REST client is going to return. That'll help.
For ex: if it'll return a JSON object then you can use Active Choice Parameter's Groovy script step - OR Scriptler script (within the Active Choice Parameter plugin). PS: Scriptler script runs in the same JVM of Jenkins process so it has access to Jenkins/etc object for free. You don't need HTTPBuilder or anything. See the code sample below.
Assuming if your REST client is returning a JSON object and from that object if you want to list hostname of the system or some field name then replace the following variable with that and you'll get it listed while doing "Build with parameters" from Jenkins job's dashboard.
import groovy.json.JsonSlurper
//this will be your URL which will return something, tweak it if you want to pass parameters or username/password acc.
def SOME_URL = "https://koba.baby.com/some_url"
// now connect to the URL and create a connection variable 'conn'
def conn = SOME_URL.toURL().openConnection()
// create a list variable 'servernames'
def servernames = []
// if connection response was successful i.e. http protocol return code was 200, then do the following
if( conn.responseCode == 200 ) {
// get the results / output of the URL connection in a variable 'results'
def results = new JsonSlurper().parseText(conn.content.text)
// to see results variable output uncomment the next line
//println results
// now read each element in the 'results' variable and pick servername/somefield variable into the list variable 'servernames'
results.each { id, data -> servernames.push(data.someField_or_HostName) }
}
return servernames.sort().unique()
// return servernames.sort()
I am using following code to get URI but it gives exception:
s3.putObject(bucket, fileToPut, file.ref.file)
val s3Object = s3.getObject(bucket, fileToPut).orNull
if(s3Object != null){
val assetUrl = s3Object.getObjectContent.getHttpRequest.getURI
}
I am unable to get uploaded object URI
Please help
If you just want the url of the object you could use
s3Client.getResourceUrl("your-bucket", "path/file.txt");
What is the preferred method to serve a static file from a Play Framework 2 Scala controller?
The file is bundled with my application, so it's not possible to hardcode a filesystem absolute /path/to/the/file, because its location depends on where the Play app happens to be installeld.
The file is placed in the public/ dir, but not in app/assets/, because I don't want Play to compile it.
(The reason I don't simply add a route to that file, is that one needs to login before accessing that file, otherwise it's of no use.)
Here is what I've done so far, but this breaks on my production server.
object Application ...
def viewAdminPage = Action ... {
... authorization ...
val adminPageFile = Play.getFile("/public/admin/index.html")
Ok.sendFile(adminPageFile, inline = true)
}
And in my routes file, I have this line:
GET /-/admin/ controllers.Application.viewAdminPage
The problem is that on my production server, this error happens:
FileNotFoundException: app1/public/admin/index.html
Is there some other method, rather than Play.getFile and OK.sendFile, to specify which file to serve? That never breaks in production?
(My app is installed in /some-dir/app1/ and I start it from /some-dir/ (without app1/) — perhaps everything would work if I instead started the app from /some-dir/app1/. But I'd like to know how one "should" do, to serve a static file from inside a controller? So that everything always works also on the production servers, regardless of from where I happen to start the application)
Check Streaming HTTP responses doc
def index = Action {
Ok.sendFile(
content = new java.io.File("/tmp/fileToServe.pdf"),
fileName = _ => "termsOfService.pdf"
)
}
You can add some random string to the fileName (individual for each logged user) to avoid sharing download link between authenticated and non-authinticated users and also make advanced download stats.
I did this: (but see the Update below!)
val fileUrl: java.net.URL = this.getClass().getResource("/public/admin/file.html")
val file = new java.io.File(adminPageUrl.toURI())
Ok.sendFile(file, inline = true)
(this is the controller, which is (and must be) located in the same package as the file that's being served.)
Here is a related question: open resource with relative path in java
Update
Accessing the file via an URI causes an error: IllegalArgumentException: URI is not hierarchical, if the file is then located inside a JAR, which is the case if you run Play like so: play stage and then target/start.
So instead I read the file as a stream, converted it to a String, and sent that string as HTML:
val adminPageFileString: String = {
// In prod builds, the file is embedded in a JAR, and accessing it via
// an URI causes an IllegalArgumentException: "URI is not hierarchical".
// So use a stream instead.
val adminPageStream: java.io.InputStream =
this.getClass().getResourceAsStream("/public/admin/index.html")
io.Source.fromInputStream(adminPageStream).mkString("")
}
...
return Ok(adminPageFileString) as HTML
Play has a built-in method for this:
Ok.sendResource("public/admin/file.html", classLoader)
You can obtain a classloader from an injected Environment with environment.classLoader or from this.getClass.getClassLoader.
The manual approach for this is the following:
val url = Play.resource(file)
url.map { url =>
val stream = url.openStream()
val length = stream.available
val resourceData = Enumerator.fromStream(stream)
val headers = Map(
CONTENT_LENGTH -> length.toString,
CONTENT_TYPE -> MimeTypes.forFileName(file).getOrElse(BINARY),
CONTENT_DISPOSITION -> s"""attachment; filename="$name"""")
SimpleResult(
header = ResponseHeader(OK, headers),
body = resourceData)
The equivalent using the assets controller is this:
val name = "someName.ext"
val response = Assets.at("/public", name)(request)
response
.withHeaders(CONTENT_DISPOSITION -> s"""attachment; filename="$name"""")
Another variant, without using a String, but by streaming the file content:
def myStaticRessource() = Action { implicit request =>
val contentStream = this.getClass.getResourceAsStream("/public/content.html")
Ok.chunked(Enumerator.fromStream(contentStream)).as(HTML)
}
I'm evaluating the possibility of using Play2-mini with Scala to develop a service that will sit between a mobile client and existing web service. I'm looking for the simplest possible example of a piece of code where Play2-mini implements a server and a client. Ideally the client will use Akka2 actors.
With this question, I'm trying to find out how it is done, but also to see how Play2-Mini and Akka2 should co-operate. Since Play2-Mini appears to be the replacement for the Akka HTTP modules.
Play2-mini contains the following code example, in which I created two TODO's. If someone can help me with some sample code to get started, I will be really grateful.
package com.example
import com.typesafe.play.mini._
import play.api.mvc._
import play.api.mvc.Results._
object App extends Application {
def route = {
case GET(Path("/testservice")) & QueryString(qs) => Action{ request=>
println(request.body)
//TODO Take parameter and content from the request them pass it to the back-end server
//TODO Receive a response from the back-end server and pass it back as a response
Ok(<h1>Server response: String {result}</h1>).as("text/html")
}
}
}
Here's the implementation of your example.
Add the following imports:
import play.api.libs.ws.WS
import play.api.mvc.BodyParsers.parse
import scala.xml.XML
Add the following route:
case GET(Path("/testservice")) & QueryString(qs) => Action{ request =>
Async {
val backendUrl = QueryString(qs,"target") map (_.get(0)) getOrElse("http://localhost:8080/api/token")
val tokenData = QueryString(qs,"data") map (_.get(0)) getOrElse("<auth>john</auth>")
WS.url(backendUrl).post(XML loadString tokenData).map { response =>
Ok(<html><h1>Posted to {backendUrl}</h1>
<body>
<div><p><b>Request body:</b></p>{tokenData}</div>
<div><p><b>Response body:</b></p>{response.body}</div>
</body></html>).as("text/html") }
}
}
All it does, is forwarding a GET request to a back-end serivce as a POST request. The back-end service is specified in the request parameter as target and the body for the POST request is specified in the request parameter as data (must be valid XML). As a bonus the request is handled asynchronously (hence Async). Once the response from the back-end service is received the front-end service responds with some basic HTML showing the back-end service response.
If you wanted to use request body, I would suggest adding the following POST route rather than GET (again, in this implementation body must be a valid XML):
case POST(Path("/testservice")) & QueryString(qs) => Action(parse.tolerantXml){ request =>
Async {
val backendUrl = QueryString(qs,"target") map (_.get(0)) getOrElse("http://localhost:8080/api/token")
WS.url(backendUrl).post(request.body).map { response =>
Ok(<html><h1>Posted to {backendUrl}</h1>
<body>
<div><p><b>Request body:</b></p>{request.body}</div>
<div><p><b>Response body:</b></p>{response.body}</div>
</body></html>).as("text/html") }
}
}
So as you can see, for your HTTP Gateway you can use Async and play.api.libs.ws.WS with Akka under the hood working to provide asynchronous handling (no explicit Actors required). Good luck with your Play2/Akka2 project.
Great answer by romusz
Another way to make a (blocking) HTTP GET request:
import play.api.libs.ws.WS.WSRequestHolder
import play.api.libs.ws.WS.url
import play.api.libs.concurrent.Promise
import play.api.libs.ws.Response
val wsRequestHolder: WSRequestHolder = url("http://yourservice.com")
val promiseResponse: Promise[Response] = wsRequestHolder.get()
val response = promiseResponse.await.get
println("HTTP status code: " + response.status)
println("HTTP body: " + response.body)