I've got a data flow that has a REST API source.
The REST API is paginated, an example JSON that links to the next page is shown below:
"links": [
{
"href": "/api/v1/widgets/?page_number=1",
"method": "GET",
"rel": "first"
},
{
"href": "/api/v1/widgets/?page_number=2",
"method": "GET",
"rel": "next"
},
{
"href": "/api/v1/widgets/?page_number=30",
"method": "GET",
"rel": "last"
}
]
To retrieve all the data, I need to use a Pagination Rule. This can be done using the JSON path to the correct value in the document.
I've got the following rule set, but this does not trigger data factory to call the next page:
$.links[?(#.rel=="next")].href
RFC5988 is set to false, which I believe should trigger the rule to be run instead.
What am I doing wrong that's causing the page to not be called?
Related
world! We are trying to use the NetSuite SuiteQL REST API to pull down data. It works perfectly fine for most of the records we are sourcing, but we hit an interesting snag when we hit large tables.
Since the known limit of a single page of data is 1,000 rows, we are simply calling it with a limit of 1,000 and setting the offsets in 1,000 row increments. Something interesting happens when we get to an offset of 100,000 rows for a record that is larger than 100,000 rows.
If we call https://myinstance.suitetalk.api.netsuite.com/services/rest/query/v1/suiteql?limit=1000&offset=98000, we get all the links that we expect.
{
"links": [
{
"rel": "previous",
"href": "https://myinstance.suitetalk.api.netsuite.com/services/rest/query/v1/suiteql?limit=1000&offset=97000"
},
{
"rel": "first",
"href": "https://myinstance.suitetalk.api.netsuite.com/services/rest/query/v1/suiteql?limit=1000&offset=0"
},
{
"rel": "next",
"href": "https://myinstance.suitetalk.api.netsuite.com/services/rest/query/v1/suiteql?limit=1000&offset=99000"
},
{
"rel": "last",
"href": "https://myinstance.suitetalk.api.netsuite.com/services/rest/query/v1/suiteql?limit=1000&offset=753000"
},
{
"rel": "self",
"href": "https://myinstance.suitetalk.api.netsuite.com/services/rest/query/v1/suiteql?limit=1000&offset=98000"
}
],
"count": 1000,
"hasMore": true,
"items": [
{
[heres my data]...
Setting it to 99,000, a lot of that information disappears. It's almost as if it thinks this is the last page.
{
"links": [
{
"rel": "previous",
"href": "https://myinstance.suitetalk.api.netsuite.com/services/rest/query/v1/suiteql?limit=1000&offset=98000"
},
{
"rel": "first",
"href": "https://myinstance.suitetalk.api.netsuite.com/services/rest/query/v1/suiteql?limit=1000&offset=0"
},
{
"rel": "self",
"href": "https://myinstance.suitetalk.api.netsuite.com/services/rest/query/v1/suiteql?limit=1000&offset=99000"
}
],
"count": 1000,
"hasMore": false,
"items": [
{
[heres my data]...
Setting it to 100,000 flat out gives me an error.
{
"type": "https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.5",
"title": "Not Found",
"status": 404,
"o:errorDetails": [
{
"detail": "The specified query parameter 'offset' is out of bounds. Provide value between 0 and 753000.",
"o:errorQueryParam": "offset",
"o:errorCode": "INVALID_PARAMETER"
}
]
Has anyone seen this kind of behavior before? I scoured the documentation and couldn't find any mention of limitation in page offsets, so I'm thinking this might be a bug of some sort (the fact that it even tells you the maximum bounds and it's clearly higher than the offset specified makes me think it's a bug), but hoping someone may have seen this before, and even better, has ideas on how to get around this!
the official documentation for NetSuite REST Web Services mentions this limitation [1]:
Using SuiteQL queries, you can return a maximum of 100,000 results. For more information, see query.runSuiteQLPaged(options).
The HATEOAS definition say that each response must contains the associated links, but following the REST pattern an URI must be the same for all operations, what changes is the HTTP method.
Per example:
Request: GET http://example.com/book
Response:
[
{
"id": 1,
"title:" "foo",
"links": {
"self": http://example.com/book/1
}
}
]
The response inform that the link that can be used to read the book data is "http://example.com/book/1", but with only this link i unknown if the user that did this request is authorized to do a DELETE or PUT for this resource.
The HATEOAS or other specification define some pattern for inform what methods are available for each resource?
Per example:
Request: GET http://example.com/book
Response:
[
{
"id": 1,
"title:" "foo",
"links": {
"self": {
"url": "http://example.com/book/1",
"methods": [
"GET",
"PUT",
"DELETE"
]
}
}
]
You should look for Siren (http://hyperschema.org/mediatypes/siren)
It will tell your clients what methods do they can use with links.
I'm currently attempting to integrate my app with Future Payments and in the documentation, it mentions:
Unlike the standard REST API docs that demonstrate a one time payment,
a future payment doesn't require you to separately get payment
approval after getting initial user consent. The payment is
pre-approved by the user.
So looking at the example, I should get a response which contains:
"state": "authorized"
"links": [
{
"href": "https://api.paypal.com/v1/payments/authorization/4TD55050SV609544L",
"method": "GET",
"rel": "self"
},
{
"href": "https://api.paypal.com/v1/payments/authorization/4TD55050SV609544L/capture",
"method": "POST",
"rel": "capture"
},
{
"href": "https://api.paypal.com/v1/payments/authorization/4TD55050SV609544L/void",
"method": "POST",
"rel": "void"
},
{
"href": "https://api.paypal.com/v1/payments/authorization/4TD55050SV609544L/reauthorize",
"method": "POST",
"rel": "reauthorize"
},
{
"href": "https://api.paypal.com/v1/payments/payment/PAY-2C433581AX997613HKJFBVLI",
"method": "GET",
"rel": "parent_payment"
}
],
And from what I understand, the transaction, along with the Client Metadata ID and Access Token in the request header, should be automatically processed, without further approval, because the user has already given consent.
So if the transaction intent is 'sale', the success response 'state' would be 'completed' and if the intent is 'authorize', the state would be 'authorized'.
This makes sense, but when testing my app, I'm getting a response with an approval url that I need to redirect the user to and the state is 'created' not 'completed/authorized' ? -
"state": "created"
"create_time": "2016-03-20T00:42:25Z",
"links": [
{
"href": "https://api.sandbox.paypal.com/v1/payments/payment/PAY-3NA62949E72063722K3W7D4I",
"rel": "self",
"method": "GET"
},
{
"href": "https://www.sandbox.paypal.com/cgi-bin/webscr?cmd=_express-checkout&token=EC-41A06151Y6402822R",
"rel": "approval_url",
"method": "REDIRECT"
},
{
"href": "https://api.sandbox.paypal.com/v1/payments/payment/PAY-3NA62949E72063722K3W7D4I/execute",
"rel": "execute",
"method": "POST"
}
]
I managed to resolve the issue by removing express_checkout from the scope. If this is enabled, it appears to override future payments so it no longer works and uses the express checkout pay flow instead.
The JavaScript PayPal button, http://paypal.github.io/JavaScriptButtons/ allows custom data to be sent in the data-custom field.
When using IPN, these data are clearly visible and usable.
However, I don't find any mention of custom data in the webhook documentation; I would expect the "Sale Completed" event to receive something about custom data.
So my question is twofold:
Has anyone managed to get the data and knows what field contains them?
Is there a way to simulate this, given the webhook simulator does not allow any field to be entered?
Webhooks do not support any custom data for simulator. Simulator provides a sample of payload for an event. It does not allow any other data field except URL/EventType. If you want to use the custom data you may use them and don't want to use a live account for testing, you can try it with a sandbox account and go through flow for the Webhook event type for which you want to send custom data.
Also sample for PAYMENT.SALE.COMPLETED for your reference:
{
"id": "WH-2WR32451HC0233532-67976317FL4543714",
"create_time": "2014-10-23T17:23:52Z",
"resource_type": "sale",
"event_type": "PAYMENT.SALE.COMPLETED",
"summary": "A successful sale payment was made for $ 0.48 USD",
"resource": {
"id": "80021663DE681814L",
"create_time": "2014-10-23T17:22:56Z",
"update_time": "2014-10-23T17:23:04Z",
"amount": {
"total": "0.48",
"currency": "USD"
},
"payment_mode": "ECHECK",
"state": "completed",
"protection_eligibility": "ELIGIBLE",
"protection_eligibility_type": "ITEM_NOT_RECEIVED_ELIGIBLE,UNAUTHORIZED_PAYMENT_ELIGIBLE",
"clearing_time": "2014-10-30T07:00:00Z",
"parent_payment": "PAY-1PA12106FU478450MKRETS4A",
"links": [
{
"href": "https://api.paypal.com/v1/payments/sale/80021663DE681814L",
"rel": "self",
"method": "GET"
},
{
"href": "https://api.paypal.com/v1/payments/sale/80021663DE681814L/refund",
"rel": "refund",
"method": "POST"
},
{
"href": "https://api.paypal.com/v1/payments/payment/PAY-1PA12106FU478450MKRETS4A",
"rel": "parent_payment",
"method": "GET"
}
]
},
"links": [
{
"href": "https://api.paypal.com/v1/notifications/webhooks-events/WH-2WR32451HC0233532-67976317FL4543714",
"rel": "self",
"method": "GET"
},
{
"href": "https://api.paypal.com/v1/notifications/webhooks-events/WH-2WR32451HC0233532-67976317FL4543714/resend",
"rel": "resend",
"method": "POST"
}
]
}
Has anyone managed to get the data and knows what field contains them?
Not Supported.
You may refer for the fields at https://github.com/paypal/JavaScriptButtons#editable-inputs
Is there a way to simulate this, given the webhook simulator does not allow any field to be entered? Not Supported
I'm designing a REST api and to be as Restful as it gets I want to implement HATEOAS into the json responses.
If the linked entities are not present in database say for example every user has a passport entity but if someone doesn't have then i want to send the set of rel and href links like this:
{
"links": [
{
"rel": "drivingLicence",
"href": "http://localhost:9090/api/v1/drivingLicence/6"
},
{
"rel": "passport",
"href": null
},
{
"rel": "self",
"href": "http://localhost:9090/api/v1/user/4028818347818fd80147819111d60001"
}
],
"userId": "4028818347818fd80147819111d60001",
"userName": "username",
"fullName": "Some User",
"activeFlag": "ACTIVE",
"createTimestamp": 1406628074000,
"updateTimestamp": 1505697126548
}
Is there any way to achieve this using Spring HATEOAS or manually i have to check for null and then set href as null.
There's no way to do this with the built in Link class as it doesn't let href be null or empty (checks in constructor). Perhaps you can derive from Link and pass in non-empty href, but override the getHref to return null.
FWIW: i've never seen the href as null strategy. I've seen have the link, and if it returns 404 then that means it's not present & just not have the link. As a hypermedia client, i'd be really surprised to see href of null...but this is a new field with new practices happening all the time.