All other questions seem to address getting of Spark applicationId. I want to cancel the spark job programmatically which requires jobId.
spark.sparkContext.cancelJob(jobId)
Similar to the following way.
sc.applicationId
You can use below code logic for this use case.
Step-01: getting the job details .
import requests
import json
class BearerAuth(requests.auth.AuthBase):
def __init__(self, token):
self.token = token
def __call__(self, r):
r.headers["authorization"] = "Bearer " + self.token
return r
response = requests.get('https://databricksinstance/api/2.0/jobs/list', auth=BearerAuth('token')).json()
print(response)
Step-02: cancelling the job rest api call
same code , just change the URL as like this
https://<databricks-instance>/api/2.1/jobs/runs/cancel
ref: link
The spark status tracker is meant for monitoring job and stage progress.
In your case you could fetch all active job ids:
sc.statusTracker.getActiveJobIds
The official scala doc
Related
The test below checks the performance of a graphql endpoint. A CSV file of id's is fed into the test. When run, about 1% of the requests fail because the endpoint returns an error for some of the id's fed in. But, the message returned from graphql is not very descriptive, so I have no idea which id's actually failed. I'd like to be able to add a step to the test which logs the request body and response for all the failed requests.
I could enable the debug log but this will log everything. I'm only interested in logging the requests which fail. Is it possible to add something like a on failure step which would let me log out the request body and response so that I know which id's failed?
class Test extends CommonSimulation {
val graphqlQuery: String =
"""
|{"query":"{person(personId:\"${id}\")}"}
|""".stripMargin
val gqsPerson: ScenarioBuilder = scenario("Service Test")
.feed(csv(Data.getPath + "id.csv").random)
.exec(http("My test")
.post("https://localhost:4000/graphql")
.body(StringBody(graphqlQuery)).asJson
.check(jsonPath("$.errors").notExists)
.headers(headers)
)
setUp(
authToken.inject(atOnceUsers(1))
.andThen(
gqsPerson.inject(constantConcurrentUsers(1) during 1)
))
}
Please have a look at the documentation: https://gatling.io/docs/gatling/guides/debugging/#logback
I am trying to build a simple Http Get request that requires me to submit an api key as authentication (api key as unsername and blank password). I have seen some solutions using the groovyx.net.http.HTTPBuilder library. However, the piece of code will need to be deployed in an evironment that does not allow for libraries. So I tried the following where is the url of the website i am trying to reach:
// GET
def get = new URL("<url>").openConnection();
def getRC = get.getResponseCode();
println(getRC);
if(getRC.equals(200)) {
println(get.getInputStream().getText());
}
As expected this returns error 400 since I do not include any authentication with the api key, so I tried the following where is the api key:
def get = new URL("<url>");
def authString = "<api_key>:".getBytes().encodeBase64().toString();
def conn = get.openConnection();
conn.setRequestProperty("Authorization", "Basic ${authString}");
def getRC = conn.getResponseCode();
println(getRC);
println(conn.getInputStream().getText());
But I still get the 400 error. I tried picking up the request through Fiddler but it doesn't seem to be tracking it (executing Groovy code through GroovyConsole).
The second approach works. My mistake was to not substitute spaces in the URL with % signs.
I have groovy spript in test step, and MockOperation in MockService.
How to get last request in script?
Now I can check only time of last mock request:
def project = testRunner.testCase.testSuite.project
def mock = testRunner.testCase.testSuite.project.mockServices["mock"]
def service = mock.getMockOperationByName("service")
def req = service.getLastMockResult()
Welcome to Stackoverflow.
For a specific response, you really need to clarify a few things.
In your description, you describe you want "last request". In your code, you do "getLastMockResult". Do you want the last request or the last response?
Is it SOAP or REST?
You also make use of "mockIss". Where does that come from? Is just a typo, and really you mean the "mock" from the previous line?
On a more generic note, given this is a SOAP request, and the Groovy Script teststep is situated in the same testcase as the Soap Request teststep, you should be able to do something like this:
def request = context.expand( '${Name of Soap Request teststep#Request#declare namespace soap=\'http://www.w3.org/2003/05/soap-envelope\'; //soap:Envelope[1]}' )
def response = context.expand( '${Name of Soap Request teststep#Response#declare namespace soap=\'http://www.w3.org/2003/05/soap-envelope\'; //soap:Envelope[1]}' )
log.info request
log.info response
I have found an [example][1] where akka-http is used with Source.single to make a request. Now I'd like to use Source.tick to implement polling requests which are execute every X seconds like this:
import akka.http.scaladsl.model._
import scala.concurrent.duration._
val request: HttpRequest = RequestBuilding.Get(Uri("http://api.someSite.com"))
val source: Source[HttpRequest, Cancellable] = Source.tick(1.seconds, 1.seconds, request)
val sourceWithDest = source.via(Http().superPool())
However, I get a compile error in the last line which I cant resolve(Type mismatch). Any ideas on what I am doing wrong or suggestions for alternatives?
[1]: https://gist.github.com/steinybot/a1f79fe9a67693722164
As per the docs:
The Flow returned by Http().superPool(...) is very similar to the one
from the Host-Level Client-Side API, so the Using a Host Connection
Pool section also applies here.
And then
The “pool client flow” returned by
Http().cachedHostConnectionPool(...) has the following type:
Flow[(HttpRequest, T), (Try[HttpResponse], T), HostConnectionPool]
This is to give client-side code the possibility to implement some logic to match the original requests to the corresponding response. Assuming you don't need this kind of behaviour in your case, you can always proceed by appending NotUsed to your request before feeding it to the pool flow. E.g.
val sourceWithDest: Source[Try[HttpResponse], Cancellable] =
source.map(req ⇒ (req, NotUsed)).via(Http().superPool[NotUsed]()).map(_._1)
In my play framework 2 application I'd like to have a log message with the request, response, and some details about the response - such as the number of search results returned from an external web call.
What I have now is a filter like this:
object AccessLog extends Filter {
import play.api.mvc._
import play.api.libs.concurrent.Execution.Implicits._
def apply(next: RequestHeader => Future[SimpleResult])(request: RequestHeader): Future[SimpleResult] = {
val result = next(request)
result map { r =>
play.Logger.info(s"Request: ${request.uri} - Response: ${r.header.status}")
}
result
}
}
At the point of logging, I've alread converted my classes into json, so it seems wasteful to parse the json back into objects so I can log information about it.
Is it possible to compute the number of search results earlier in the request pipeline, maybe into a dictionary, and pull them out when I log the message here?
I was looking at flash, but don't want the values to be sent out in a cookie at any cost. Maybe I can clear the flash instead. Buf if there's a more suitable way I'd like to see that.
This is part of a read-only API that does not involve user accounts or sessions.
You could try using the play.api.cache.Cache object if you can come up with a reproducible unique request identifier. Once you have logged your request, you can remove it from the Cache.