Passing multiple json as a payload for a request in Gatling - scala

sample json payload:
'{
"Stub1": "XXXXX",
"Stub2": "XXXXX-3047-4ed3-b73b-83fbcc0c2aa9",
"Code": "CodeX",
"people": [
{
"ID": "XXXXX-6425-EA11-A94A-A08CFDCA6C02"
"customer": {
"Id": 173,
"Account": 275,
"AFile": "tel"
},
"products": [
{
"product": 1,
"type": "A",
"stub1": "XXXXX-42E1-4A13-8190-20C2DE39C0A5",
"Stub2": "XXXXX-FC4F-41AB-92E7-A408E7F4C632",
"stub3": "XXXXX-A2B4-4ADF-96C5-8F3CDCF5821D",
"Stub4": "XXXXX-1948-4B3C-987F-B5EC4D6C2824"
},
{
"product": 2,
"type": "B",
"stub1": "XXXXX-42E1-4A13-8190-20C2DE39C0A5",
"Stub2": "XXXXX-FC4F-41AB-92E7-A408E7F4C632",
"stub3": "XXXXX-A2B4-4ADF-96C5-8F3CDCF5821D",
"Stub4": "XXXXX-1948-4B3C-987F-B5EC4D6C2824"
}
]
}
]
}'
I am working on a POST call. Is there any way to feed multiple json files as a payload in Gatling. I am using body(RawFileBody("file.json")) as json here.
This works fine for a single json file. I want to check response for multiple json files. Is there any way we can parametrize this and get response against multiple json files.

As far as I can see, there's a couple of ways you could do this.
Use a JSON feeder (https://gatling.io/docs/current/session/feeder#json-feeders). This would need your multiple JSON files to be in a single file, with the root element being a JSON array. Essentially you'd put the JSON objects you have inside an array inside a single JSON file
Create a Scala Iterator and have the names of the JSON files you're going to use in it. e.g:
val fileNames = Iterator("file1.json", "file2.json)
// and later, in your scenario
body(RawFileBody(fileNames.next())
Note that this method cannot be used across users, as the iterator will initialize separately for each user. You'd have to use repeat or something similar to send multiple files as a single user.
You could do something similar by maintaining the file names as a list inside Gatling's session variable, but this session would still not be shared between different users you inject into your scenario.

Related

How to map a json string into object type in sink transformation

Using Azure Data Factory and a data transformation flow. I have a csv that contains a column with a json object string, below an example including the header:
"Id","Name","Timestamp","Value","Metadata"
"99c9347ab7c34733a4fe0623e1496ffd","data1","2021-03-18 05:53:00.0000000","0","{""unit"":""%""}"
"99c9347ab7c34733a4fe0623e1496ffd","data1","2021-03-19 05:53:00.0000000","4","{""jobName"":""RecipeB""}"
"99c9347ab7c34733a4fe0623e1496ffd","data1","2021-03-16 02:12:30.0000000","state","{""jobEndState"":""negative""}"
"99c9347ab7c34733a4fe0623e1496ffd","data1","2021-03-19 06:33:00.0000000","23","{""unit"":""kg""}"
Want to store the data in a json like this:
{
"id": "99c9347ab7c34733a4fe0623e1496ffd",
"name": "data1",
"values": [
{
"timestamp": "2021-03-18 05:53:00.0000000",
"value": "0",
"metadata": {
"unit": "%"
}
},
{
"timestamp": "2021-03-19 05:53:00.0000000",
"value": "4",
"metadata": {
"jobName": "RecipeB"
}
}
....
]
}
The challenge is that metadata has dynamic content, meaning, that it will be always a json object but the content can vary. Therefore I cannot define a schema. Currently the column "metadata" on the sink schema is defined as object, but whenever I run the transformation I run into an exception:
Conversion from ArrayType(StructType(StructField(timestamp,StringType,false),
StructField(value,StringType,false), StructField(metadata,StringType,false)),true) to ArrayType(StructType(StructField(timestamp,StringType,true),
StructField(value,StringType,true), StructField(metadata,StructType(StructField(,StringType,true)),true)),false) not defined
We can get the output you expected, we need the expression to get the object Metadata.value.
Please ref my steps, here's my source:
Derived column expressions, create a JSON schema to convert the data:
#(id=Id,
name=Name,
values=#(timestamp=Timestamp,
value=Value,
metadata=#(unit=substring(split(Metadata,':')[2], 3, length(split(Metadata,':')[2])-6))))
Sink mapping and output data preview:
The key is that your matadata value is an object and may have different schema and content, may be 'value' or other key. We only can manually build the schema, it doesn't support expression. That's the limit.
We can't achieve that within Data Factory.
HTH.

Extract values from JSON file in Talend

I have json file like this:
{"2020-04-28":
{
"37,N7L2H4,Carry,CHOPA,PLX":
{
"inter_results": {
"inter_mark": "GITA"
,"down": null
,"up": null
,"wiki": {"included": "false", "options": ["RRR", "SSS","HHH"]
}}
,"38, N5L2J4, HURT, SERRA, PZT": {
"inter_results": {
"inter_mark": "MARI"
,"down": "250"
,"up": "1250"
,"wiki": {"included": "true", "options": ["XXX", "YYY"]
}}
,"39, N4L2H4, HIBA, FILA, PFG": {
"inter_results": {
"inter_mark": "HILO"
,"down": "100"
,"up": "250"
,"wiki": {"included": "true", "options": ["RTG", "VTH","HJI","JKL"]
}}
}
}
And i want extract the Values N7L2H4,N5L2J4,N4L2H4 from this json file using tFileInputJson with jsonPath.
Using the native components of Talend, this is hard to achieve. You can do it with some java code but it's not elegant.
Here's a solution using json components suite from Talend Exchange which you can download here
The component tJSONDocTraverseFields allows you to list all fields, paths and values of your json.
It gives this output:
$.2020-04-28.37,N7L2H4,Carry,CHOPA,PLX.inter_results.inter_mark|4|inter_mark|"GITA"|false|21
$.2020-04-28.37,N7L2H4,Carry,CHOPA,PLX.inter_results.down|4|down|null|false|21
$.2020-04-28.37,N7L2H4,Carry,CHOPA,PLX.inter_results.up|4|up|null|false|21
$.2020-04-28.37,N7L2H4,Carry,CHOPA,PLX.inter_results.wiki.included|5|included|"false"|false|21
$.2020-04-28.37,N7L2H4,Carry,CHOPA,PLX.inter_results.wiki.options[0]|6|options|"RRR"|true|21
$.2020-04-28.37,N7L2H4,Carry,CHOPA,PLX.inter_results.wiki.options[1]|6|options|"SSS"|true|21
$.2020-04-28.37,N7L2H4,Carry,CHOPA,PLX.inter_results.wiki.options[2]|6|options|"HHH"|true|21
$.2020-04-28.38, N5L2J4, HURT, SERRA, PZT.inter_results.inter_mark|4|inter_mark|"MARI"|false|21
$.2020-04-28.38, N5L2J4, HURT, SERRA, PZT.inter_results.down|4|down|"250"|false|21
$.2020-04-28.38, N5L2J4, HURT, SERRA, PZT.inter_results.up|4|up|"1250"|false|21
You can then parse the json path to get the values you want:
I split the path by "." to get the field "37,N7L2H4,Carry,CHOPA,PLX", then split the result again on "," and get the first value.
tJSONDocOpen allows you to initialize your json file, it acts as a connection. You then select it in tJSONDocTraverseFields.

Use output from Web Activity call as variable

I'm using ADFv2 to transfer some data. As a part of this operation I need some configuration values to pass into the pipeline.
The config values must be pulled at runtime from a REST service - not as parameters.
I can successfully query the REST service with Web Activity and I can see the output in the debug view.
Now the problem :)
How do I use this output in other activities further in the pipeline?
My Web Activity configuration is like this:
{
"name": "Web1",
"type": "WebActivity",
"policy": {
"timeout": "7.00:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false
},
"typeProperties": {
"url": "https://myazurefunction.azurewebsites.net/api/MyFunction",
"method": "GET",
"headers": {
"Content-Type": "application/json"
}
}
I have tried to access the output after is has executed, but it seems empty:
#activity('Web1').Output
#activity('Web1').output
#string(activity('Web1').Output)
they are all empty. Any suggestions?
Thanks!
I set up an ADF2 and try to get a response.
This works for me:
#string(activity('Post').output)
Have you checked the output in the debugging?
Here is my output:
{
"test": {
"value": 123,
"text": abc
},
"concat": 123abc
}
I use the stored procedure to insert the values into the destination table on a Logical Server.
In ADFv2, you access the output of previous activities using #activity('ActivityName').output.
For the web activity defined, the response from your function should be in JSON format, so you would reference specific JSON values using their attribute names in the response. For example, your defined web activity, named Web1, calls a function that returns a response of:
{
"foo": "bar",
"some": "value"
}
To use the value of foo in a subsequent ADF activity, you would reference #activity('Web1').output.foo. ADFv2 provides multiple type conversion functions, should you need the returned value converted to another type.
If your function is returning an empty JSON response back, you may want to inspect the response from your function using Postman or another tool to ensure you are returning a properly formatted response, and that your function isn't failing for another reason.
Inside your Azure function code, you should be returning a JSON object, along with a success code, similar to return req.CreateResponse(HttpStatusCode.OK, json);.
Also note that if you reference a property of the response and it does not exist, ADF will fail at that point, so you can use an If Condition activity to check for the required values to better handle failures in ADFv2.

Talend read JSON data from tRESTRequest

I am trying to learn Talend.
Scenario:
I have to create a REST endpoint (i am using tRESTRequest) which takes a POST request at http://localhost:8086/emp/create and accepts below json and prints each json field and sends a sample json response containing only name field.
How can I do so ?
How to read the json data into a java component like tJava?
Structure:
{
"emp" :
[
{
"id":"123",
"name": "testemp1"
},
{
"id":"456",
"name": "testemp2"
}
]
}
Expected Response:
{
"emp" :
[
{
"name": "testemp1"
},
{
"name": "testemp2"
}
]
}
I am using tRESTRequest -> tExtractJSONFields -> tRESTResponse.
For looping on the right elements and parsing the contents, please see my answer JSON Deserialization on Talend
I did not understand the second question. When deserializing JSON, the data will already be available in the usual row format for processing further. Beginner tutorials will show you the standard structure. The component tJava is - of course - an exception to that rule. Handling data is different in this component and not neccessarily row based.
Talend has an excellent knowledge base for components and examples, see https://help.talend.com/

Pagination response payload from a RESTful API

I want to support pagination in my RESTful API.
My API method should return a JSON list of product via /products/index. However, there are potentially thousands of products, and I want to page through them, so my request should look something like this:
/products/index?page_number=5&page_size=20
But what does my JSON response need to look like? Would API consumers typically expect pagination meta data in the response? Or is only an array of products necessary? Why?
It looks like Twitter's API includes meta data: https://dev.twitter.com/docs/api/1/get/lists/members (see Example Request).
With meta data:
{
"page_number": 5,
"page_size": 20,
"total_record_count": 521,
"records": [
{
"id": 1,
"name": "Widget #1"
},
{
"id": 2,
"name": "Widget #2"
},
{
"id": 3,
"name": "Widget #3"
}
]
}
Just an array of products (no meta data):
[
{
"id": 1,
"name": "Widget #1"
},
{
"id": 2,
"name": "Widget #2"
},
{
"id": 3,
"name": "Widget #3"
}
]
ReSTful APIs are consumed primarily by other systems, which is why I put paging data in the response headers. However, some API consumers may not have direct access to the response headers, or may be building a UX over your API, so providing a way to retrieve (on demand) the metadata in the JSON response is a plus.
I believe your implementation should include machine-readable metadata as a default, and human-readable metadata when requested. The human-readable metadata could be returned with every request if you like or, preferably, on-demand via a query parameter, such as include=metadata or include_metadata=true.
In your particular scenario, I would include the URI for each product with the record. This makes it easy for the API consumer to create links to the individual products. I would also set some reasonable expectations as per the limits of my paging requests. Implementing and documenting default settings for page size is an acceptable practice. For example, GitHub's API sets the default page size to 30 records with a maximum of 100, plus sets a rate limit on the number of times you can query the API. If your API has a default page size, then the query string can just specify the page index.
In the human-readable scenario, when navigating to /products?page=5&per_page=20&include=metadata, the response could be:
{
"_metadata":
{
"page": 5,
"per_page": 20,
"page_count": 20,
"total_count": 521,
"Links": [
{"self": "/products?page=5&per_page=20"},
{"first": "/products?page=0&per_page=20"},
{"previous": "/products?page=4&per_page=20"},
{"next": "/products?page=6&per_page=20"},
{"last": "/products?page=26&per_page=20"},
]
},
"records": [
{
"id": 1,
"name": "Widget #1",
"uri": "/products/1"
},
{
"id": 2,
"name": "Widget #2",
"uri": "/products/2"
},
{
"id": 3,
"name": "Widget #3",
"uri": "/products/3"
}
]
}
For machine-readable metadata, I would add Link headers to the response:
Link: </products?page=5&perPage=20>;rel=self,</products?page=0&perPage=20>;rel=first,</products?page=4&perPage=20>;rel=previous,</products?page=6&perPage=20>;rel=next,</products?page=26&perPage=20>;rel=last
(the Link header value should be urlencoded)
...and possibly a custom total-count response header, if you so choose:
total-count: 521
The other paging data revealed in the human-centric metadata might be superfluous for machine-centric metadata, as the link headers let me know which page I am on and the number per page, and I can quickly retrieve the number of records in the array. Therefore, I would probably only create a header for the total count. You can always change your mind later and add more metadata.
As an aside, you may notice I removed /index from your URI. A generally accepted convention is to have your ReST endpoint expose collections. Having /index at the end muddies that up slightly.
These are just a few things I like to have when consuming/creating an API.
I would recommend adding headers for the same. Moving metadata to headers helps in getting rid of envelops like result , data or records and response body only contains the data we need. You can use Link header if you generate pagination links too.
HTTP/1.1 200
Pagination-Count: 100
Pagination-Page: 5
Pagination-Limit: 20
Content-Type: application/json
[
{
"id": 10,
"name": "shirt",
"color": "red",
"price": "$23"
},
{
"id": 11,
"name": "shirt",
"color": "blue",
"price": "$25"
}
]
For details refer to:
https://github.com/adnan-kamili/rest-api-response-format
For swagger file:
https://github.com/adnan-kamili/swagger-response-template
As someone who has written several libraries for consuming REST services, let me give you the client perspective on why I think wrapping the result in metadata is the way to go:
Without the total count, how can the client know that it has not yet received everything there is and should continue paging through the result set? In a UI that didn't perform look ahead to the next page, in the worst case this might be represented as a Next/More link that didn't actually fetch any more data.
Including metadata in the response allows the client to track less state. Now I don't have to match up my REST request with the response, as the response contains the metadata necessary to reconstruct the request state (in this case the cursor into the dataset).
If the state is part of the response, I can perform multiple requests into the same dataset simultaneously, and I can handle the requests in any order they happen to arrive in which is not necessarily the order I made the requests in.
And a suggestion: Like the Twitter API, you should replace the page_number with a straight index/cursor. The reason is, the API allows the client to set the page size per-request. Is the returned page_number the number of pages the client has requested so far, or the number of the page given the last used page_size (almost certainly the later, but why not avoid such ambiguity altogether)?
just add in your backend API new property's into response body.
from example .net core:
[Authorize]
[HttpGet]
public async Task<IActionResult> GetUsers([FromQuery]UserParams userParams)
{
var users = await _repo.GetUsers(userParams);
var usersToReturn = _mapper.Map<IEnumerable<UserForListDto>>(users);
// create new object and add into it total count param etc
var UsersListResult = new
{
usersToReturn,
currentPage = users.CurrentPage,
pageSize = users.PageSize,
totalCount = users.TotalCount,
totalPages = users.TotalPages
};
return Ok(UsersListResult);
}
In body response it look like this
{
"usersToReturn": [
{
"userId": 1,
"username": "nancycaldwell#conjurica.com",
"firstName": "Joann",
"lastName": "Wilson",
"city": "Armstrong",
"phoneNumber": "+1 (893) 515-2172"
},
{
"userId": 2,
"username": "zelmasheppard#conjurica.com",
"firstName": "Booth",
"lastName": "Drake",
"city": "Franks",
"phoneNumber": "+1 (800) 493-2168"
}
],
// metadata to pars in client side
"currentPage": 1,
"pageSize": 2,
"totalCount": 87,
"totalPages": 44
}
This is an interessting question and may be perceived with different arguments. As per the general standard meta related data should be communicated in the response headers e.g. MIME type and HTTP codes. However, the tendency I seem to have observed is that information related to counts and pagination typically are communicated at the top of the response body. Just to provide an example of this The New York Times REST API communicate the count at the top of the response body (https://developer.nytimes.com/apis).
The question for me is wheter or not to comply with the general norms or adopt and do a response message construction that "fits the purpose" so to speak. You can argue for both and providers do this differently, so I believe it comes down to what makes sense in your particular context.
As a general recommendation ALL meta data should be communicated in the headers. For the same reason I have upvoted the suggested answer from #adnan kamili.
However, it is not "wrong" to included some sort of meta related information such as counts or pagination in the body.
generally, I make by simple way, whatever, I create a restAPI endpoint for example "localhost/api/method/:lastIdObtained/:countDateToReturn"
with theses parameters, you can do it a simple request.
in the service, eg. .net
jsonData function(lastIdObtained,countDatetoReturn){
'... write your code as you wish..'
and into select query make a filter
select top countDatetoreturn tt.id,tt.desc
from tbANyThing tt
where id > lastIdObtained
order by id
}
In Ionic, when I scroll from bottom to top, I pass the zero value, when I get the answer, I set the value of the last id obtained, and when I slide from top to bottom, I pass the last registration id I got