Extracting the n´th element of a json response in Scala/Gatling - scala

I have a response body in json where I want to extract the value: ID_ADFA0741-6A46-4A90-8AAE-BC7DB18B27E1
In the below json structure.
.check(jsonPath("$..key").saveAs("key")))
gives me the value M01A
How could I use a check to obtain the value "key" for the element number 2 (the second occuerence of "key" below?
{"libItems":[{"hasNotes":false,"drugs":[{"ingredientNamesAndStrengths":[{"ingredientNames":["Diklofenak"],"strengths":["50 mg"]}],"order":0,"form":"Enterotab"}],"treatmentType":"Brand","isSupplement":false,"nameOfBrand":"Voltaren","nameFormStrength":"Diklofenak Enterotab 50 mg","medicationUsedFor":{"value":"TEST"},"clinicalWarnings":[],"dataWarnings":[{"practitionerName":"ELENA FOS ENGELUND","importDate":"2021-08-16T18:13:06.597+02:00","isNewTreatment":true,"type":"Import","isMarkedAsHandled":false}],"warningReferences":[{"key":"M01A","drugName":"Diklofenak Enterotab 50 mg","refType":"Atc"},{"key":"ID_ADFA0741-6A46-4A90-8AAE-BC7DB18B27E1","drugName":"Diklofenak Enterotab 50 mg","refType":"Atc"},{"key":"ID_BD1BDC3F-1FCF-4944-99BC-6B3E8EAAD6FD","drugName":"Diklofenak Enterotab 50 mg","refType":"Atc"}],"usage":"Fixed","atc":"M01AB05","atcNames":"Diklofenak","shortDose":{"key":"160","value":"1x3"},"drugId":"ID_BD1BDC3F-1FCF-4944-99BC-6B3E8EAAD6FD","hasIngredientNames":true,"prescriptionActions":["Stop","ConfirmUse","Renew","Prescribe"],"id":"440a01f5-1df8-4f26-a6c1-08d95cab00c7","treatmentId":"00be4297-cc9f-432d-30d6-08d95cab00c7","treatmentStart":"2021-06-02T00:00:00","dssnText":"1 tablett 3 ganger daglig","actionStatus":"ActionRequired","treatmentStatus":"Active","externalUpdate":"New","approvalStatus":"Empty","resepts":[{"localReseptState":"None","rfReseptState":"AvailableForDispatch"}],"diffPreviousValues":[],"hasRfError":false,"isLocked":false,"guardianAccessReservation":false,"paperReseptDispensation":false,"reseptPidState":"NoChange"}],"fibItems":[],"nibItems":[],"vaccines":[],"vibInformation":{"vibStatus":"ActionRequired","messagesToSign":{"new":0,"renewed":0,"changed":0,"stopped":0,"removed":0,"recalled":0,"registration":0,"stoppedLocalRegistration":0},"actionRequired":{"treatments":1,"nutritions":0,"consumables":0,"vaccines":0,"allergies":0,"warnings":0},"previousLibItems":["ID_BD1BDC3F-1FCF-4944-99BC-6B3E8EAAD6FD"]}}

Okay, there is a two ways.
The first - you can 'directly' define which element you need to extract:
.check(jsonPath("$.[1].key").saveAs("key")))
The second way - you can save all elements and then use the desired via Gatling EL:
.check(jsonPath("$..key").findAll.saveAs("keys")))
"${keys(1)}"

Since, in the comments, the OP requested an example of using a Json parsing/serialization library for constructing case classes from the provided Json, for the accessing and processing of the desired fields; here is a simplified working example with circe:
import io.circe._, io.circe.generic.auto._, io.circe.syntax._, io.circe.parser._
val myShortJsonString =
"""{"libItems":
[
{
"hasNotes":false,
"warningReferences":[
{"key":"M01A", "drugName":"Diklofenak Enterotab 50 mg", "refType":"Atc" },
{ "key":"ID_ADFA0741-6A46-4A90-8AAE-BC7DB18B27E1", "drugName":"Diklofenak Enterotab 50 mg", "refType":"Atc" },
{ "key": "ID_BD1BDC3F-1FCF-4944-99BC-6B3E8EAAD6FD", "drugName":"Diklofenak Enterotab 50 mg", "refType":"Atc" }
]
}
]
}"""
val myFullJsonString =
"""{"libItems":
[
{
"hasNotes":false,
"drugs":[
{"ingredientNamesAndStrengths":[{"ingredientNames":["Diklofenak"],"strengths":["50 mg"]}],
"order":0,
"form":"Enterotab"}
],
"treatmentType":"Brand",
"isSupplement":false,
"nameOfBrand":"Voltaren",
"nameFormStrength":"Diklofenak Enterotab 50 mg",
"medicationUsedFor":{"value":"TEST"},
"clinicalWarnings":[],
"dataWarnings":[{"practitionerName":"ELENA FOS ENGELUND","importDate":"2021-08-16T18:13:06.597+02:00","isNewTreatment":true,"type":"Import","isMarkedAsHandled":false}],
"warningReferences":[
{"key":"M01A","drugName":"Diklofenak Enterotab 50 mg","refType":"Atc"},
{"key":"ID_ADFA0741-6A46-4A90-8AAE-BC7DB18B27E1","drugName":"Diklofenak Enterotab 50 mg","refType":"Atc"},
{"key":"ID_BD1BDC3F-1FCF-4944-99BC-6B3E8EAAD6FD","drugName":"Diklofenak Enterotab 50 mg","refType":"Atc"}],
"usage":"Fixed",
"atc":"M01AB05",
"atcNames":"Diklofenak",
"shortDose":{"key":"160","value":"1x3"},
"drugId":"ID_BD1BDC3F-1FCF-4944-99BC-6B3E8EAAD6FD",
"hasIngredientNames":true,
"prescriptionActions":["Stop","ConfirmUse","Renew","Prescribe"],
"id":"440a01f5-1df8-4f26-a6c1-08d95cab00c7",
"treatmentId":"00be4297-cc9f-432d-30d6-08d95cab00c7",
"treatmentStart":"2021-06-02T00:00:00",
"dssnText":"1 tablett 3 ganger daglig",
"actionStatus":"ActionRequired",
"treatmentStatus":"Active",
"externalUpdate":"New",
"approvalStatus":"Empty",
"resepts":[{"localReseptState":"None","rfReseptState":"AvailableForDispatch"}],
"diffPreviousValues":[],
"hasRfError":false,
"isLocked":false,
"guardianAccessReservation":false,
"paperReseptDispensation":false,
"reseptPidState":"NoChange"
}
],
"fibItems":[],
"nibItems":[],
"vaccines":[],
"vibInformation":{
"vibStatus":"ActionRequired",
"messagesToSign":{
"new":0,
"renewed":0,
"changed":0,
"stopped":0,
"removed":0,
"recalled":0,
"registration":0,
"stoppedLocalRegistration":0
},
"actionRequired":{
"treatments":1,
"nutritions":0,
"consumables":0,
"vaccines":0,
"allergies":0,
"warnings":0
},
"previousLibItems":["ID_BD1BDC3F-1FCF-4944-99BC-6B3E8EAAD6FD"]
}
}"""
case class Library(libItems: List[LibItem],
// fibItems: List[FibItem], nibItems: List[NibItem], vaccines: List[Vaccine], vibInformation: VibInformation
)
case class LibItem(hasNotes: Boolean,
//drugs: List[Drug],
warningReferences: List[WarningReference])
case class WarningReference(key: String, drugName: String, refType: String)
val maybeLibraryJsonObject = parse(myShortJsonString)
val maybeLibrary = maybeLibraryJsonObject match {
case Right(libraryJsonObject) => libraryJsonObject.as[Library]
case Left(failure) => println(failure)
}
maybeLibrary match {
case Right(libraryResult) =>
println(libraryResult)
val libItems = libraryResult.asInstanceOf[Library].libItems
val libItem = libItems(0)
val warningReferenceKey = libItem.warningReferences(1).key
val warningReferenceKeySections = warningReferenceKey.split("-")
val code = warningReferenceKeySections(3)+"-"+warningReferenceKeySections(4)
println("code: " + code)
case Left(failure) => println(failure)
}
To make it work for the full json string, you need to create case classes for all of its fields, as in the commented section.

Related

Make JSON Parsing & Error Handling More Functional in Scala

I have the following piece of code that I use to read the incoming JSON which looks like this:
{
"messageTypeId": 2,
"messageId": "19223201",
"BootNotification" :
{
"reason": "PowerUp",
"chargingStation": {
"serialNumber" : "12345",
"model" : "",
"vendorName" : "",
"firmwareVersion" : "",
"modem": {
"iccid": "",
"imsi": ""
}
}
}
}
I have the following reads using play-json that would process this JSON:
implicit val ocppCallRequestReads: Reads[OCPPCallRequest] = Reads { jsValue =>
val messageTypeId = (jsValue \ 0).toOption
val messageId = (jsValue \ 1).toOption
val actionName = (jsValue \ 2).toOption
val payload = (jsValue \ 3).toOption
(messageTypeId.zip(messageId.zip(actionName.zip(payload)))) match {
case Some(_) => JsSuccess(
OCPPCallRequest( // Here I know all 4 exists, so safe to call head
messageTypeId.head.as[Int],
messageId.head.as[String],
actionName.head.as[String],
payload.head
)
)
case None => JsError( // Here, I know that I have to send a CallError back!
"ERROR OCCURRED" // TODO: Work on this!
)
}
}
It is not playing nicely when it comes to delivering the exact error message for the case None. It is all in or nothing, but what I want to avoid is that in my case None block, I would like to avoid looking into each of the Option and populate the corresponding error message. Any ideas on how I could make it more functional?

How do you post form data using pytest?

I'm trying to write a unit test that posts form data. The actual line in question is:
def test_create_request():
with app.test_client() as test_client:
app_url = '/requests/'
with app.app_context():
new_request = get_new_request()
form_data = json.dumps(new_request, default=str)
print('FORM DATA: ', form_data)
resp = test_client.post(app_url, data=form_data, headers={'Content-Type': 'application/json'})
assert resp.status_code == 200
s = json.loads(resp.data)
assert s['success'] == True
Where new_request is a dict representation of an object. The print statement yields (I've formatted it a bit):
FORM DATA: {
"request_id": "6",
"state_id": 1,
"orig_project_id": "1",
"orig_project_code": "QQQ",
"orig_allocated_funding": "123.45",
"orig_ytd_spend": "123.45",
"orig_ytd_commit": "123.45",
"orig_ytd_ocnr": "123.45",
"new_project_id": 2,
"new_funding_amount": 123.45,
"new_ytd_spend": 123.45,
"new_ytd_commit": 123.45,
"new_ytd_ocnr": 123.45,
"plan": "this is the plan",
"reason": "this is the reason",
"sm_director": "sm.dir#example.com",
"submitted_by": "xfgbn#vexample.com",
"created_on": "2021-09-14 16:32:55",
"meets_approval_guidelines": null
}
In the flask form, most fields are required. When I try to post the data, the form.validate_on_submit() function in the view's route returns False, and the 2nd assert fails. WTForms claims that none of the required orig_allocated_funding, orig_ytd_spend, orig_ytd_commit, orig_ytd_ocnr, new_project_id, new_funding_amount, new_ytd_spend, new_ytd_commit, new_ytd_ocnr, reason, plan, sm_director, or submitted_by fields are supplied.
I've read several tutorials, and I can't see what I'm doing wrong. Can anyone help?
What I was able to make work was scraping form_data altogether:
def test_create_request():
with app.test_client() as test_client:
app_url = '/requests/'
with app.app_context():
new_request = get_new_request()
resp = test_client.post(app_url, data=new_request)
assert resp.status_code == 200
s = json.loads(resp.data)
print(s['html'])
assert s['success'] == True

How to upload files and get formfields in akka-http

I am trying to upload a file via akka-http, and have gotten it to work with the following snippet
def tempDestination(fileInfo: FileInfo): File =
File.createTempFile(fileInfo.fileName, ".tmp")
val route =
storeUploadedFile("csv", tempDestination) {
case (metadata, file) =>
//Do my operation on the file.
complete("File Uploaded. Status OK")
}
But I'd also want to send a param1/param2 in the posted form.
I tried the following, and it works, but I am having to send the parameters via the URL (http://host:port/csv-upload?userid=arvind)
(post & path("csv-upload")) {
storeUploadedFile("csv", tempDestination) {
case (metadata, file) =>
parameters('userid) { userid =>
//logic for processing the file
complete(OK)
}
}
}
The restriction on the file size is around 200-300 MB. I added the following property to my conf
akka{
http{
parsing{
max-content-length=200m
}
}
}
Is there a way, I can get the parameters via the formFields directive ?
I tried the following
fileUpload("csv") {
case (metadata, byteSource) =>
formFields('userid) { userid =>
onComplete(byteSource.runWith(FileIO.toPath(Paths.get(metadata.fileName)))) {
case Success(value) =>
logger.info(s"${metadata}")
complete(StatusCodes.OK)
case Failure(exception) =>
complete("failure")
But, with the above code, I hit the following exception
java.lang.IllegalStateException: Substream Source cannot be materialized more than once
at akka.stream.impl.fusing.SubSource$$anon$13.setCB(StreamOfStreams.scala:792)
at akka.stream.impl.fusing.SubSource$$anon$13.preStart(StreamOfStreams.scala:802)
at akka.stream.impl.fusing.GraphInterpreter.init(GraphInterpreter.scala:306)
at akka.stream.impl.fusing.GraphInterpreterShell.init(ActorGraphInterpreter.scala:593)
Thanks,
Arvind
I got this working with sth like:
path("upload") {
formFields(Symbol("payload")) { payload =>
println(s"Server received request with additional payload: $payload")
def tempDestination(fileInfo: FileInfo): File = File.createTempFile(fileInfo.fileName, ".tmp.server")
storeUploadedFile("binary", tempDestination) {
case (metadataFromClient: FileInfo, uploadedFile: File) =>
println(s"Server stored uploaded tmp file with name: ${uploadedFile.getName} (Metadata from client: $metadataFromClient)")
complete(Future(FileHandle(uploadedFile.getName, uploadedFile.getAbsolutePath, uploadedFile.length())))
}
}
}
Full example:
https://github.com/pbernet/akka_streams_tutorial/blob/master/src/main/scala/akkahttp/HttpFileEcho.scala

Querying in mongodb 3.4

I have large XML files which I had to convert in json and store in mongodb. The python code for conversion and insertion is:
import pymysql
import re
import json
import xmltodict
from pymongo import MongoClient
# Open Database Connection.
db = pymysql.connect("fffff","ddd","fgf","hnj")
# prepare a cursor object
cursor = db.cursor()
# execute SQL query
cursor.execute("SELECT jlp.appid, convert(MAX(lex.response) using utf8) FROM jos_lender_portfolio jlp INNER JOIN jos_lexnex_data lex ON jlp.appid = lex.appid\
group by appid limit 10;")
# Fetch all rows
data = cursor.fetchall()
a = (r'(?=<response>)(.*)(?<=</response>)')
def cleanxml(xml):
if re.findall(a, xml, re.S):
file = re.findall(a, xml, re.S)[0]
else:
file = "<response>NA</response>"
return file
data = list(data)
client = MongoClient()
db = client['lexnex']
collection = db['test']
for row in data:
thexml = cleanxml(row[1])
jsonString = json.dumps(xmltodict.parse(thexml), indent = 4)
d = json.loads(jsonString)
newdict = {"caseid" : row[0]}
newdict.update(d)
jsondata = json.dumps(newdict, indent = 3)
f = json.loads(jsondata)
db.test.insert_one(f)
Now, the problem: I'm very new to mongodb and having problem in querying my database.I have the following json:
"_id":ObjectId("5aeff8537871560bf05d8c25"),
"caseid":44136,
"response":{
"Header":{
"TransactionId":"18092257R1069402",
"Status":"0"
},
"Records":{
"Record":[
{
"Filings":{
"Filing":{
"Type":"INITIAL FILING",
"Date":{
"Day":"23",
"Month":"9",
"Year":"2008"
}
}
},
"FilingJurisdiction":"NY",
"MatchedParty":{
"PartyType":"D",
"Address":{
"City":"BROOKLYN",
"State":"NY",
},
"OriginName":"GOLDLINE"
},
"Secureds":{
"Secured":{
"Addresses":{
"Address":{
"City":"SCHAUMBURG",
"State":"IL"
}
}
}
}
},
{
,
"Filings":{
"Filing":{
"Type":"INITIAL FILING",
"Date":{
"Day":"23",
"Month":"9",
"Year":"2008"
}
}
},
"FilingJurisdiction":"NY",
"MatchedParty":{
"PartyType":"D",
"Address":{
"City":"BROOKLYN",
"State":"NY",
},
"OriginName":"GOLD"
},
"Secureds":{
"Secured":{
"Addresses":{
"Address":{
"City":"SCHAUMBURG",
"State":"IL"
}
}
}
}
}
]
}
}
This is a small portion of a very big document and there are more than a million such documents. Now, the expected result which I want is for every caseid, some part of the Filings and the Secureds. Here's the sample expected output:
"_id":ObjectId("5aeff8537871560bf05d8c25"),
"caseid":44136,
"Filings":{
[
"Filing":{
"Type":"INITIAL FILING",
"Date":{
"Day":"23",
"Month":"9",
"Year":"2008"
}
},
"Secureds":{
"Secured":{
"Addresses":{
"Address":{
"City":"SCHAUMBURG",
"State":"IL"
}
}
}
},
{
"Filing":{
"Type":"INITIAL FILING",
"Date":{
"Day":"23",
"Month":"9",
"Year":"2008"
}
}
},
"Secureds":{
"Secured":{
"Addresses":{
"Address":{
"City":"SCHAUMBURG",
"State":"IL"
}
}
}
}
]
}
There are several caseids and each one has 0 or more filings. I have no clue how to do it. I know the basics like simple queries. But this, I think, requires $unwind and $group together. What I have written so far is nothing but just this:
db.test.aggregate([{$unwind:{path: '$response'}},{"$group":{_id:{caseid:"$caseid"}}}])
Please help.

How to check for 200 OK response status using Scala and Play Framework

I have a following Actor class that is responsible for sending a JSON message to a URL using POST.
import play.api.libs.ws._
class Worker extends Actor {
val logger: Logger = Logger("superman")
val supermanURL = "http://localhost:9000/superman/send"
def receive = {
case message: JsValue => {
val transactionID = (message \ "transactionID").get
println("Received JSON Object =>" + message)
val responseFromSuperman = WS.url(supermanURL).withHeaders("Content-Type" -> "application/json").post(message)
responseFromSuperman.map(
result => {
//TODO: Make sure to only log if response status is 200 OK
logger.info("""Message="ACK received from Superman" for transactionID=""" + transactionID)}
).recover { case error: Throwable =>
logger.error("""Message="NACK received from Superman" for transactionID=""" + transactionID) + " errorMessage:" + error.getLocalizedMessage()
}
}
}
}
So, if you look into my TODO above, I would like to add a check for a response type 200 OK. The current implementation is not doing that and it logs the message even if I manually send in a BadRequest. I tried checking for result.allHeaders which returns:
Map(Date -> Buffer(Wed, 27 Jan 2016 21:45:31 GMT), Content-Type -> Buffer(text/plain; charset=utf-8), Content-Length -> Buffer(7))
but no information about response status 200 OK
Simply:
import play.api.http.Status
if(result.status == Status.OK) {
// ...
}
Maybe I am missing here something but you have "status" on the response.
So you can do:
WS.url(url).withHeaders("Content-Type" -> "application/json").post(message).map{
case response if ( response.status == OK) => //DO SOMETHING?
}