GET Service for listing Buckets
With the tutorial in the link above, I am able to list Buckets in my account. However, the response only return 1000 Buckets which I believe is the limit for a single response. The problem is how can I list the remaining Buckets in my account? For listing Bucket Objects, we can add the marker param to indicate from which object to continue listing but it seems there is no such param for listing Buckets.
Edit:
I am using HMAC keys instead of OAuth2 to make request so I cannot use the JSON API, "HMAC keys can only be used to make requests to the XML API, not the JSON API", according to HMAC keys Restrictions
Current XML response which doesn't include anything like marker, cursor or pageToken:
<?xml version='1.0' encoding='UTF-8'?>
<ListAllMyBucketsResult
xmlns='http://doc.s3.amazonaws.com/2006-03-01'>
<Buckets>
<Bucket>
<Name>bucket-1</Name>
<CreationDate>2019-09-23T07:03:11.892Z</CreationDate>
</Bucket>
<Bucket>
<Name>bucket-2</Name>
<CreationDate>2019-09-23T07:03:12.578Z</CreationDate>
</Bucket>
...
<Bucket>
<Name>bucket-999</Name>
<CreationDate>2019-09-23T07:03:11.892Z</CreationDate>
</Bucket>
<Bucket>
<Name>bucket-1000</Name>
<CreationDate>2019-09-23T07:03:12.578Z</CreationDate>
</Bucket>
</Buckets>
</ListAllMyBucketsResult>
You could try using the JSON API's list method. With this method whenever your results are over the maximum a pageToken item will be returned, you can use it as a parameter for your next request in order to paginate through the results.
Before implementing it you can test this API from the documentation page.
I have a list of entity IDs that I need to fetch from Dynamics into my application. Using the WebAPI the query must be URL encoded and passed as a GET request which limits the number of entities to about 130 (more than that hits the URL length limit)
Previously we could use the SOAP endpoint to make a single POST request but this is on the deprecation path.
So what is the recommended way to make such a query now?
Some options:
Manually paginate and run through the WebAPI making parallel requests. This is less than ideal, especially when making abstractions around Dynamics since requests may fail.
UPDATE
Thanks to #Guido Preite I was able to use a WebAPI batch request. Basically you issue the GET request with the 130+ entities but encode it in the batch request body to work around the character limit. I'll post the request below for Postman ...
Post to {{myDynamicsURL}}/api/data/v9.0/$batch
Headers
Authorization:Bearer {{token}}
Content-Type:multipart/mixed;boundary=batch_AAA123
Accept:application/json
Cache-Control:no-cache
OData-MaxVersion:4.0
OData-Version:4.0
Prefer:odata.maxpagesize=500, odata.include-annotations="*", return=representation
Body
--batch_AAA123
Content-Type: application/http
Content-Transfer-Encoding:binary
GET {{myDynamicsURL}}/connections?$filter=(connectionid eq 9b704176-2f60-e911-a830-000d3a385a17 or connectionid eq 1ccc526c-2e6c-e911-a831-000d3a385a17 ... <REST OF QUERIES HERE>) HTTP/1.1
Accept: application/json
--batch_AAA123--
you can query using a FetchXML passed inside a POST request
https://dreamingincrm.com/2017/01/15/executing-large-fetchxml-with-webapi/
We are trying to post a really long query string, that exceeds 2048 characters to sharepoint rest api.I am making a post to the below api
http://sitename/_api/web/sitegroups?$select=LoginName,Id,Title&$Filter=LoginName%20eq%20'Test_Grp'
We modified the config file of our sharepoint site to the below,
<system.web>
<httpRuntime maxQueryStringLength="65536"/>
<system.webServer>
<security>
<requestFiltering>
<requestLimits maxQueryString="65536" />
But still sharepoint throws a 400 error stating that the maxquery length is exceeded. Do I have to change the configurations in some other place?
Is it possible to send the query string as part of the http post body so that the query string length could be let alone ?
SharePoint REST supports maximum query length of 256 characters
excluding query options.
Looking at your REST query, there is not issue. Please send the error message you are getting to resolve your query.
I am using the Business Rules service on Bluemix, and it does not seem to use the Decision Warehouse. Is there another way to trace the rule firing?
As you noticed, the Decision Warehouse is not a feature within the Business Rules service. Also, rule traces are disabled.
You can add trace filter tags to the payload request. Use the REST form (&trace=true) to view the trace filter tags:
https://brsv2-instanceID.domain/DecisionService/run.jsp?path=/ruleApp/1.0/ruleset/1.0&trace=true
(You can get the values for instanceID, domain, ruleApp, ruleset and the corresponding versions from the “URL” in the Ruleset Details page.)
If you work with XML, choose the XML option, in which case the generated payload will contain filter tags such as:
<!--Optional:-->
<trac:all>false</trac:all>
<!--Optional:-->
<trac:none>true</trac:none>
<!--Optional:-->
<trac:executionDuration>true</trac:executionDuration>
<!--Optional:-->
<trac:executionDate>true</trac:executionDate>
If you work with JSON, choose the JSON option instead. The generated payload will contain filter entries, for example:
"_TraceFilter_":{
"infoRulesetProperties":true,
"infoOutputString":false,
"infoInputParameters":false,
"infoOutputParameters":true,
...
Add these tags/entries with the appropriate filter enabled in the execution request. The payload response will contain the requested traces.
Another way is to enable the Decision trace information when getting the WSDL or WADL. See the documentation at “Decision trace information” in the IBM ODM documentation at "IBM Operational Decision Manager 8.7.0>Operational Decision Manager 8.7> Decision Server Rules>Rule Execution Server console online help>Viewing and managing transparent decision services>Viewing or downloading an HTDS description file", Step 6 and collect the execution information on the client side.
According to REST style, it's generally assumed that HTTP POST, GET, PUT, and DELETE methods should be used for CREATE, READ, UPDATE and DELETE (CRUD) operations.
But if we stick to the HTTP method definitions, it might not be so clear.
In this article it's explained that:
In a nutshell: use PUT if and only if you know both the URL where the resource will live, and the entirety of the contents of the resource. Otherwise, use POST.
Mainly because
PUT is a much more restrictive verb. It takes a complete resource and stores it at the given URL. If there was a resource there previously, it is replaced; if not, a new one is created. These properties support idempotence, which a naive create or update operation might not. I suspect this may be why PUT is defined the way it is; it's an idempotent operation which allows the client to send information to the server.
In my case I usually issue updates passing all the resource data, so I could use PUT for updates, but every time I issue an update I save a LastUser and LastUpdate column, with the user id that made the modification and the time of the operation.
I'd like to know your opinion, because strictly speaking those two columns are not part of the resource, but they do prevent the operation from being idempotent.
Ignoring the comment about the REST style mapping CRUD to the HTTP methods, this is an excellent question.
The answer to your question is, yes you are free to use PUT in this scenario even though there are some elements of the resource that are updated by the server in a non-idempotent manner. Unfortunately, the reasoning behind the answer is quite vague. The important thing, is to understand what was the intent of the client request. The client intended to completely replace the contents of resource with the values passed. The client is not responsible for the server doing other operations and therefore the semantics of the HTTP method are not violated.
This is the reasoning that is used to allow a server to update a page counter when you do a GET operation. The client didn't ask for an update therefore the GET is safe even though the server chose to make an update.
The whole, complete resource versus partial resource debate has finally been spelled out in an update to the HTTP spec
An origin server SHOULD reject any PUT
request that contains a
Content-Range header field, since it
might be misinterpreted as partial
content (or might be partial content
that is being mistakenly PUT as a
full representation). Partial content
updates are possible by targeting a
separately identified resource with
state that overlaps a portion of
the larger resource, or by using a
different method that has been
specifically defined for partial
updates (for example, the PATCH
method defined in [RFC5789]).
So, what we are supposed to do is now clear. What is not so clear is why there exists this constraint on only being allowed to send full responses. That question has been asked and IMHO remains unanswered in this thread on rest-discuss.
As LastUser and LastUpdate are not modifiable by the client, I'd remove them from the representation of your resource altogether. Let me explain my reasoning with an example.
Let's say that our typical example API will return the following representation to the client when asked to provide a single resource:
GET /example/123
<?xml version="1.0" encoding="UTF-8" ?>
<example>
<id>123</id>
<lorem>ipsum</lorem>
<dolor>sit amet</dolor>
<lastUser uri="/user/321">321</lastUser>
<lastUpdate>2011-04-16 20:00:00 GMT</lastUpdate>
</example>
If a client wants to modify the resource, it would presumably take the whole representation and send it back to the API.
PUT /example/123
<?xml version="1.0" encoding="UTF-8" ?>
<example>
<id>123</id>
<lorem>foobar</lorem>
<dolor>foobaz</dolor>
<lastUser>322</lastUser>
<lastUpdate>2011-04-16 20:46:15 GMT+2</lastUpdate>
</example>
Since the API generates values for lastUser and lastUpdate automatically and cannot accept data provided by the client, the most appropriate response would be 400 Bad Request or 403 Forbidden (since the client cannot modify these values).
If we want to be compliant with REST and send a full representation of the resource when doing a PUT request, we need to remove lastUser and lastUpdate from the representation of the resource. This will allow clients to send the full entity via PUT:
PUT /example/123
<?xml version="1.0" encoding="UTF-8" ?>
<example>
<id>123</id>
<lorem>foobar</lorem>
<dolor>foobaz</dolor>
</example>
The server would accept a full representation now that it doesn't contain lastUpdate and lastUser.
The question that remains is how to provide clients with access to lastUpdate and lastUser. If they don't need it (and these fields are required just internally by the API), we are fine and our solution is perfectly RESTful. If however clients need access to this data, the cleanest approach would be to use HTTP headers:
GET /example/123
...
Last-Modified: Sat, 16 Apr 2011 18:46:15 GMT
X-Last-User: /user/322
...
<?xml version="1.0" encoding="UTF-8" ?>
<example>
<id>123</id>
<lorem>foobar</lorem>
<dolor>foobaz</dolor>
</example>
Using a custom HTTP header is not ideal because user agents need to be taught on how to read it. If we want to provide clients with access to the same data in a more easier way, the only thing that we can do is to put the data into the representation, and we are facing the same problem as in your original question. I would at least try to mitigate it somehow. If the content type used by the API is XML, we can put the data into node attributes instead of exposing them directly as node values, i.e.:
GET /example/123
...
Last-Modified: Sat, 16 Apr 2011 18:46:15 GMT
...
<?xml version="1.0" encoding="UTF-8" ?>
<example last-update="2011-04-16 18:46:15 GMT" last-user="/user/322">
<id>123</id>
<lorem>foobar</lorem>
<dolor>foobaz</dolor>
</example>
This way we'll at least avoid the problem where a client would attempt to submit all XML nodes in a follow-up PUT request. This won't work with JSON, and the solution is still a bit on the edge of idempotency (since the API would still have to ignore the XML attributes when processing the request).
Even better, as Jonah pointed out in the comments, if clients need access to lastUser and lastUpdate, these can be exposed as a new resource, linked from the original one e.g. as follows:
GET /example/123
<?xml version="1.0" encoding="UTF-8" ?>
<example>
<id>123</id>
<lorem>foobar</lorem>
<dolor>foobaz</dolor>
<lastUpdateUri>/example/123/last-update</lastUpdateUri>
</example>
... and then:
GET /example/123/last-update
<?xml version="1.0" encoding="UTF-8" ?>
<lastUpdate>
<resourceUri>/example/123</resourceUri>
<updatedBy uri="/user/321">321</updatedBy>
<updatedAt>2011-04-16 20:00:00 GMT</updatedAt>
</lastUpdate>
(The above can be also nicely expanded to provide a full audit log with individual changes, providing a resource changelog is available.)
Please note:
I agree with Darrel Miller's take on the question, but I wanted to provide a different approach on top of it. Note that this approach is not backed-up by any standards/RFCs/etc, it's just a different take on the problem.
The disadvantage of using PUT to create resources is that the client has to provide the unique ID that represents the object it is creating. While it usually possible for the client to generate this unique ID, most application designers prefer that their servers (usually through their databases) create this ID. In most cases we want our server to control the generation of resource IDs. So what do we do? We can switch to using POST instead of PUT.
So:
Put = UPDATE
Post = INSERT
Hopefully, this helps for your specific case.
The HTTP methods POST and PUT aren't the HTTP equivalent of the CRUD's create and update. They both serve a different purpose. It's quite possible, valid and even preferred in some occasions, to use PUT to create resources, or use POST to update resources.
Use PUT when you can update a resource completely through a specific resource. For instance, if you know that an article resides at http://example.org/article/1234, you can PUT a new resource representation of this article directly through a PUT on this URL.
If you do not know the actual resource location, for instance, when you add a new article, but do not have any idea where to store it, you can POST it to an URL, and let the server decide the actual URL.