I'm creating an API on API Gateway and the idea is to receive a webhook, map the data in the required format for Kinesis and use putRecord to send that event to Kinesis data streams.
One of the required pieces of info is present in the headers in a field named x-site-context.
For example:
x-site-context: {"field1": "Data1", "field2": "Data2"}
Now, I would like to extract field2 from x-site-context in a mapping template.
So far, I have tried to extract this field in the following way:
#foreach($header in $input.params().header.keySet())
#if($header=="x-site-context")
"x-site-context": "$util.escapeJavaScript($input.params().header.get($header)).get("field2")"
#end
#end
But it doesn't seem to be working, any help would be much appreciated, thanks!
Related
I have a project scenario to get all values from an endpoint URL. I am using ADF Pipeline but I'm having some issues with pagination.
To get the following values, I need to make requests with the PaginationCursor value in the current body response in the following request header.
I have read that ADF supports the following case, which would be mine.
Next request’s header = property value in current response body ADF - Pagination support
I don't know how to use the following attributes in order to use the paginationCursor value from the current response body in the header of the next request.
Attributes for pagination in ADF
I tried to reproduce above but not successful. Instead, if you want to do it without pagination, you can try this approach.
First create a web activity with any page URL of your API to get the total number of pages count.
In ForEach create an array for page numbers using the count from web activity as
#range(1,activity('Web1').output.total_pages)
Inside ForEach use the copy activity and give the source REST dataset parameter for the page number like ?page=#{item()}.
In the sink dataset also, create a dataset for each page with the dataset parameter value like APIdataset#{item()}.csv. This generates the sink dataset names like APIdataset1.csv, APIdataset2.csv,...
Now, you can copy from your REST API without pagination.
My repro for your reference:
Copy activity:
I could solve this problem with the following attributes.
Solution
In the Headers I had to put the name of the header of the next call. In my case the name is PaginationCursor and I got the value of this header from the actual body response called paginationCursor.
I am trying to use debezium server to stream "some" changes in a postgresql table. Namely this table being tracked has a json type column named "payload". I would like the message streamed to pubsub by debezium to contain only the contents of the payload column. Is that possible?
I ve explored the custom transformations provided by debezium but from what I could get it would only allow me to enrich the published message with extra fields, but not to publish only certain fields, which is what I want to do.
Edit:
The closest I got to what I wanted was to use the outbox transform but that published the following message:
{
"schema":{
...
},
"payload:{
"key":"value"
}
Whereas what I would like the message to be is:
{"key":"value"}
I ve tried adding an ExtractNewRecordState transform but still got the same results. My application.properties file looks like:
debezium.transforms=outbox,unwrap
debezium.transforms.outbox.type=io.debezium.transforms.outbox.EventRouter
debezium.transforms.outbox.table.field.event.key=grouping_key
debezium.transforms.outbox.table.field.event.payload.id=id
debezium.transforms.outbox.route.by.field=target
debezium.transforms.outbox.table.expand.json.payload=true
debezium.transforms.unwrap.type=io.debezium.transforms.ExtractNewRecordState
debezium.transforms.unwrap.add.fields=payload
Many thanks,
Daniel
I have just started in a new company and they would like me to use Talend jobs to update the stocks of web sites. I learned towards the web services of prestashop except that I do not know the exchanges with the web services well and not at all talend.
I need to modify the body of the tREST component, for each iteration of a contentfile my ID and quantities.
Here is the body structure and my job. (Which works for a given ID and quantity)
I don't know very well the tREST component. But if you use a tRESTClient , with a tXMLMap before it, you should be able to create your XML schema in tXMLMap, then send the document produced to the tRESTClient.
tXMLMap allows you to use metadata from repository, so you can create a metadata associated with your XML example.
Type of the output in your tXMLMap should be Document (this is the java-type associated with XML), and the name of the output flow should be "payload" (i think it is mandatory , but not sure about it)
We are trying to put data in Apahe Ignite Cache using this REST API provided by Ignite. https://apacheignite.readme.io/docs/rest-api.
I want to know if I can pass JSON data to it from spring boot application. Tried the basic GET and PUT it's working fine. But how to pass lots of Data from the JSON.?
Like Example JSON
{
Name : CYZ,
Id:12345
Dept: xyz
}
P.S The JSON is for understanding purposes only. I will tweak the answer as per my requirement.
Thanks.
You can use a ConnectorMessageInterceptor to convert the JSON representation into a Java object.
You can specify it in Ignite configuration as ConnectorConfiguration#messageInterceptor property. ConnectorConfiguration can be specified as IgniteConfiguration#connectorConfiguration property.
Storm - Conditionally consuming stream from kafka spout?
How do i get the schema of the data inside the Split Bolt when I try to output it using the declareOutputFields().
Fields schema = new Fields(?)
How do i get the schema of the all the fields in the data inside this bolt without basically reparsing all the data and recreating it?
You need to know the schema beforehand, ie, before you process the first tuples. The method declareOutputFields() is called during deployment before the first call to execute().
Storm cannot handle a variable schema. If you have JSON data with unknown structure, you could declare new Fields("json") and put the whole JSON object into a single field.