I am working on usecase where i have to read (id , token and type values delimited with pipe )from kafka topic using spark streaming job.
Using the above values, i have to make rest api call to pull data belongs to id.
I am using below code which i am able to get the data in json format.
val httpget = new HttpGet("///////");
val httpParams = httpClient.getParams
HttpConnectionParams.setConnectionTimeout(httpParams,timeoutConnection)
val httpResponse = httpClient.execute(httpget)
println(httpResponse.getStatusLine().getStatusCode)
val entity = httpResponse.getEntity()
var content = ""
if (entity != null) {
val inputStream = entity.getContent()
content = scala.io.Source.fromInputStream(inputStream).getLines.mkString
inputStream.close
}
My problem here is using this logic in dStream.foreachRDD() block and parsing the json content effectively.
Any help will be appreciated.
Related
I am using httpClient lib to do some REST API calls using scala. I am able to post the data.
I am using following code to read the content. However, When I run on Spark Databricks Cluster it give me error.
val entity = response.getEntity
var content = ""
if (entity != null) {
val inputStream = entity.getContent
content = io.Source.fromInputStream(inputStream).getLines.mkString
inputStream.close()
}
Error
error: object Source is not a member of package io
content = io.Source.fromInputStream(inputStream).getLines.mkString
is there a way I can fix this error, or a different way to read HTTP response content.
Please try to import scala.io.Source
OR use the completed package name like this:
content = scala.io.Source.fromInputStream(inputStream).getLines.mkString
I want to send bulk xml to my soap request body. I want to know is there any way to generate them dynamically through Gatling/Scala
This is how I am doing
private val request=exec( http("Post request to create")
.post("/endPoint")
.headers(Utils.soapHeaders)
.body(ElFileBody("requestbody/ids.xml"))
.check(status.is(Utils.successStatus)) )
private val feedId = Iterator.continually( Map(
"id" -> Random.alphanumeric.take(13).mkString,
"transactionId" -> Random.alphanumeric.take(14).mkString
) )
val scnPostVehicleAsn=Utils.createScenario("soapService", feedId,
request)
and requestbody/ids.xml has the template with dynamic values ${transactionId} & ${id}.
So Is there any way to dynamically generate xml based on the template, I am not looking for repeat functionality.
Just generate xml once before execution and pass it and later I will make rest calls to validate them
I figured it out myself. I have created a list of random number and passed it to xml directly
val randomNumbers =
List.fill(number)(prefix.concat(Random.alphanumeric.take(13).mkString))
{ randomNumbers.map( i => i }
Then,
val file = new File(System.getProperty("user.dir")
+"/performance-tests/src/test/resources/requestBody/ids.xml") val bw = new BufferedWriter(new FileWriter(file))
bw.write(VehicleAsns.toString()) bw.close()
I am using Akka and I want to modify the entity of a HttpResponse. In particular, I want to modify the body or header of the particular HttpResponse. Since HttpResponse is a final class, I cannot modify it, but copying and setting a slightly modified body would be sufficient.
Here's my code:
val handler = Source.single(context.request)
.via(flow)
.runWith(Sink.head)
.flatMap { r =>
logger.info(s"Status code: ${r.status}.")
val copyR = r.copy(status = r.status, headers = r.headers, entity = ???, protocol = r.protocol)
context.complete(copyR)
}
Is there an elegant way to parse the entity and modify certain elements of the DOM?
I'm using the twitter4j library to access the public twitter stream. I'm trying to make a project involving geotagged tweets, and I need to collect a large number of them for testing.
Right now I am getting the unfiltered stream from twitter and only saving tweets with geotags. This is slow though because the VAST majority of tweets don't have geo tags. I want the twitter stream to send me only tweets with geotags.
I have tried using the method mentioned in [this question][1], where you filter with a bounding box of size 360* by 180* but that's not working for me. I'm not getting any errors when using that filter, but I'm still getting 99% of tweets with no geotags. Here is how I'm doing it:
DefaultHttpClient httpclient = new DefaultHttpClient();
httpclient.getParams().setParameter(HttpMethodParams.RETRY_HANDLER,
new DefaultHttpMethodRetryHandler());
HttpGet httpget = new HttpGet("https://developers.facebook.com/docs/reference/api/examples/");
HttpResponse response = httpclient.execute(httpget);
HttpEntity entity = response.getEntity();
if (entity != null) {
entity.consumeContent();
}
List<Cookie> cookies = httpclient.getCookieStore().getCookies();
HttpPost httpost = new HttpPost(
"https://www.facebook.com/login.php?login_attempt=1");
List<NameValuePair> nvps = new ArrayList<NameValuePair>();
nvps.add(new BasicNameValuePair("email", "xxxxxxxxxxxxxx"));
nvps.add(new BasicNameValuePair("pass", "ssssssss"));
httpost.setEntity(new UrlEncodedFormEntity(nvps, HTTP.UTF_8));
response = httpclient.execute(httpost);
entity = response.getEntity();
if (entity != null) {
entity.consumeContent();
}
CookieStore cookiestrore = httpclient.getCookieStore();
//cookies = httpclient.getCookieStore().getCookies();
//httpclient.getConnectionManager().shutdown();
return cookiestrore;
Any this is not getting any error but i am not getting any results.
When you track keyword it is separate job from tracking locations. These are logical ORs
I want to add the CacheControl intormation to a GET service that use the json binding.
I found that to add the cacheControl to a response the REST service sound like this:
#GET
#Path("cacheheadertest")
#Produces({"*/*"})
def testcache():javax.ws.rs.core.Response {
val rb:Response.ResponseBuilder = javax.ws.rs.core.Response.ok("chached test message")
val cc = new CacheControl()
cc.setMaxAge(60)
cc.setNoCache(false)
rb.cacheControl(cc).build()
}
but I have a REST service that produce json messages and the jersey library transform automaticcally the java object from java to xml/json.
#GET
#Path("jsontestcache")
#Produces(Array(MediaType.APPLICATION_JSON, MediaType.APPLICATION_XML))
def myjsontestservice(#HeaderParam("X-TOKENID") tokenId: String,
#QueryParam("clientId") clientId: String):com.test.MyResultClass = {
val response= new com.test.MyResultClass
[...]
response
}
How can I add the cache control to the response of myjsontestservice service? Do I need to use a filter and append the cachecontrol once the response has been created by jersey?
thanks million
Flavio
You would still need to return a Response object.
def somejson() : Response = {
val builder = Response.ok(new com.test.MyResultClass);
val cc = new CacheControl()
cc.setMaxAge(60)
cc.setNoCache(false)
builder.cacheControl(cc).build()
}
Jersey's interceptors will automatically convert your class into a JSON object.