EWS FindItem CalendarView weird handling of MaxEntriesReturned - soap

This is my SOAP FindItem call to retrieve calendar items:
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:typ="http://schemas.microsoft.com/exchange/services/2006/types" xmlns:mes="http://schemas.microsoft.com/exchange/services/2006/messages">
<soapenv:Header>
<typ:RequestServerVersion Version="Exchange2007_SP1"/>
<typ:MailboxCulture>en-US</typ:MailboxCulture>
<typ:TimeZoneContext>
<typ:TimeZoneDefinition Id="W. Europe Standard Time"/>
</typ:TimeZoneContext>
</soapenv:Header>
<soapenv:Body>
<mes:FindItem Traversal="Shallow">
<mes:ItemShape>
<typ:BaseShape>AllProperties</typ:BaseShape>
<typ:AdditionalProperties>
<typ:FieldURI FieldURI="item:LastModifiedTime" />
</typ:AdditionalProperties>
</mes:ItemShape>
<mes:CalendarView MaxEntriesReturned="1000" StartDate="2015-02-18T00:00:00Z" EndDate="2015-12-05T23:59:59Z"/>
<mes:ParentFolderIds>
<typ:DistinguishedFolderId Id="calendar">
<typ:Mailbox>
<typ:EmailAddress>sddress#server.nl</typ:EmailAddress>
</typ:Mailbox>
</typ:DistinguishedFolderId>
</mes:ParentFolderIds>
</mes:FindItem>
</soapenv:Body>
</soapenv:Envelope>
The MaxEntriesReturned in the optional element
<mes:CalendarView MaxEntriesReturned="1000" StartDate="2015-02-18T00:00:00Z" EndDate="2015-12-05T23:59:59Z"/>
is supposed to mean the maximum number of results to return in the FindItem response
However:
1) The results easily returns more items:
<m:FindItemResponse xmlns:m="http://schemas.microsoft.com/exchange/services/2006/messages" xmlns:t="http://schemas.microsoft.com/exchange/services/2006/types">
<m:ResponseMessages>
<m:FindItemResponseMessage ResponseClass="Success">
<m:ResponseCode>NoError</m:ResponseCode>
<m:RootFolder TotalItemsInView="2516" IncludesLastItemInRange="false">
<t:Items>
<t:CalendarItem>
(and note the IncludesLastItemInRange="false", so there are more items. Then why return 2516?)
2) If I leave MaxEntriesReturned="1000" out of the query, I get a ErrorServerBusy response code with text The server cannot service this request right now. Try again later.
This may have to do with Exchange Server throttling policies, but that link says:
The EWSFindCountLimit parameter specifies the maximum result size of FindItem or FindFolder calls that can exist in memory on the Client Access server at the same time for this user in this current process
so I expect to get the ErrorExceededFindCountLimit response code with text You have exceeded the maximum number of objects that can be returned for the find operation. Use paging to reduce the result size and try your request again.
The issues I have with these observations:
1) I specify MaxEntriesReturned to prevent being overwhelmed with data, yet I get much more than I ask for. (Fine with ice cream, but not here).
2) If I do indeed bump into a throttling policy I'd like a decent ErrorExceededFindCountLimit response that tells me what is going on, so that my code can suggest corrective measures; ErrorServerBusy can mean many things.
What can be done about both?
This is all testing against Release to Manufacturing (RTM) version of Exchange Server 2013:
<ServerVersionInfo MajorVersion="15" MinorVersion="0" MajorBuildNumber="516" MinorBuildNumber="29" Version="Exchange2013" .../>

The TotalItemsInView attribute in the FindItem response corresponds to the total number of items in the entire CalendarView. The resulting dataset will still be bound by the set MaxEntriesReturned if you take a look at the actual amount of items returned.
So for instance, if I set MaxEntriesReturned to 1 for a date range I know has 4 events total, I'll get back: <m:RootFolder TotalItemsInView="4" IncludesLastItemInRange="false”>. But in the response, only one CalendarItem is returned.
[Note added by OP] That solves Item 1) of the question (item 2 is in this answer)

The CalenderView isn't page-able eg have a look at https://msdn.microsoft.com/en-us/library/microsoft.exchange.webservices.data.calendarview_properties(v=exchg.80).aspx and you will see that there is no property that allows you to specify an offset (or index) value.
What you need to do is reduce the time window so you get less then 1000 items returned. Then page the timewindow manually yourself to get the full Appointment list you want.
Cheers
Glen

Not content with waiting for an answer, I kept digging and came across Throttling Policies and the EWSFindCountLimit.
This explains issue 2) (item 1 is in this answer)
The article says If the call has a RequestServerVersion that is earlier than Exchange2010, you will receive a failure response with an error code of ErrorServerBusy.
That was exactly what I was doing: <typ:RequestServerVersion Version="Exchange2007_SP1"/>. Changing it to <typ:RequestServerVersion Version="Exchange2010"/> returns:
<m:FindItemResponseMessage ResponseClass="Error">
<m:MessageText>You have exceeded the maximum number of objects that can be returned for the find operation. Use paging to reduce the result size and try your request again.</m:MessageText>
<m:ResponseCode>ErrorExceededFindCountLimit</m:ResponseCode>
<m:DescriptiveLinkKey>0</m:DescriptiveLinkKey>
<m:MessageXml>
<t:Value Name="PolicyLimit">1000</t:Value>
</m:MessageXml>
</m:FindItemResponseMessage>
... nicely telling me that the limit I broke was 1000.
The article concludes: the moral of the story is: Always use a paging mechanism when calling FindItem or FindFolder

Related

how to unlock objects in IBM content navigator from a SOA webservice call

In my SOA, there are two apps trading document information back and forth. One of them is IBM's filenet/content navigator. Now the other app cannot call upon documents in filenet when these document are checked out in filenet. This usually is solvable by manually logging into Filenet and right clicking the document and selecting to undo the checkout.
Since the holdup really screws with my SOA integration I want to be able to perform this "undo checkout" action in filenet through a webservice call in my SOA. This would save a lot of time spent on manual actions unlocking the documents. I am using Oracle's SOA suite 11g (and 12c), and my process is heavily carried by BPELs. I already have a nice webservice interacting with Filenet. However, I will need to create a new operation "UnlockDocument" to interact and perform this action in filenet.
What I need: I need to have the code that would cover the "UnlockDocument" operation in a filenet environment, or some similar trick that would get the job done. Any information (also non-code!) on how I could proceed is very welcome, and I´ll keep updating my post if I find more info myself!
Thank you for your help!
Jesper
It turns out there is no possible "UnlockDocument" or "CancelCheckout" operation in filenet's webservice. However, I have found a neat workaround that let's you do just that.
When a document is checked out in filenet through the client or through a webservicecall of the operation:
"CheckoutAction".
A copy of the document is made internally in filenet with the same VersionSeriesId as the original document, but with the property Isreserved = 'true'. If you perform a "DeleteAction" on this copy, you essentially recreate the manual "Cancel Checkout" step available in the filenet client. "DeleteAction" requires an ObjectIdand doesn't work on the VersionSeriesId. In order to get this ObjectID through a webservice call, you need to make a SOAPCall that obtains this ObjectID. To cancel the initial checkout, a second SOAPCall should be made that deletes the document with the in the previous step obtained ObjectID, aka: "the copy". Herefollows two examples of usable SOAPCalls:
ExecuteSearchRequest SoapCall:
<soap:Envelope xmlns:soap="http://www.w3.org/2003/05/soap-envelope" xmlns:sch="http://www.filenet.com/ns/fnce/2006/11/ws/schema">
<soap:Header>
<sch:Localization>
<sch:Locale>en_EN</sch:Locale>
<sch:Timezone/>
</sch:Localization>
</soap:Header>
<soap:Body>
<sch:ExecuteSearchRequest xsi:type="RepositorySearch" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<sch:SearchScope xsi:type="ObjectStoreScope" objectStore="ObjectStoreXXX"/>
<sch:SearchSQL>SELECT [Id] FROM Document WHERE VersionSeries = {"enter the VersionSeriesID of the initial document without quotes"} AND IsReserved = true</sch:SearchSQL>
</sch:ExecuteSearchRequest>
</soap:Body>
</soap:Envelope>
DeleteActionRequest SoapCall:
<soap:Envelope xmlns:soap="http://www.w3.org/2003/05/soap-envelope" xmlns:sch="http://www.filenet.com/ns/fnce/2006/11/ws/schema">
<soap:Header>
<sch:Localization>
<sch:Locale>en-EN</sch:Locale>
<sch:Timezone/>
</sch:Localization>
</soap:Header>
<soap:Body>
<sch:ExecuteChangesRequest xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<sch:ChangeRequest>
<sch:TargetSpecification classId="Document" objectId="{"enter the objectId of the previously obtained document without quotes"}" objectStore="ObjectStoreO7"/>
<sch:Action xsi:type="sch:DeleteAction"/>
</sch:ChangeRequest>
</sch:ExecuteChangesRequest>
</soap:Body>
</soap:Envelope>
Now in order to get this to work from a SOA, you'll need to invoke filenet's webservice twice from your BPEL. First with the first operation: ExecuteSearchRequest, which yields you the ObjectId required to cancel the checkout, afterwards with the second operation ExecuteChangesRequest, which deletes the correct document, undoing the initial checkout. These operations are listed in the above SOAP examples. Additionally you need to add WS-security in your outgoing header with working credentials to access the Filenet service. Otherwise you won't be able to connect with filenet.
This has cost me a lot of time so I hope this helps someone besides me. Enjoy your mastery of filenet checkout deletion!

QBWC: "Index Out Of Range" in do_authenticate()

I'm using QuickBooks WebConnector 2.2.0.71 and my WCF web service (on .NET 4.6.1). After pressing "Update selected" in WebConnector serverVersion and clientVersion requests successfully processed, but authenticate failed:
20170705.06:31:00 UTC : QBWebConnector.SOAPWebService.do_authenticate() : *** Calling authenticate() with following parameters:<userName="username"><password=<MaskedForSecurity>
20170705.06:31:00 UTC : QBWebConnector.SOAPWebService.do_authenticate() : QBWC1012: Authentication failed due to following error message.
Index Out Of Range.
More info:
StackTrace = в QBWebConnector.WebService.do_authenticate(String& ticket, String& companyFileName)
Source = QBWebConnector
Response of my WCF service (from WCF Test Client):
<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
<s:Header />
<s:Body>
<authenticateResponse xmlns="http://developer.intuit.com/">
<authenticateResult xmlns:a="http://schemas.microsoft.com/2003/10/Serialization/Arrays" xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
<a:string>a3f10876-e027-419d-8dd8-3752a852ddae</a:string>
<a:string>nvu</a:string>
<a:string>3</a:string>
<a:string>60</a:string>
<a:string>60</a:string>
</authenticateResult>
</authenticateResponse>
</s:Body>
</s:Envelope>
Docs says "Your callback must return A string array with 4 possible elements. The first element contains either NONE or NVU (invalid user) or BUSY., or empty
string, or a string that is the QB company file name." but in samples first element is guid token, so I send array of 5 elements.
Same error occured when I send not an array, but int value, so I guess - maybe something wrong with my xml?
Page 21 of the QBWC Programmers Guide.
Your return to the authenticate call will be a string array with a maximum of four strings.
The first member of the array is a session token, which could be a GUID or anything else that you want to use to identify the session. This token will be returned by QBWC in subsequent callbacks in the session.
The second member of the string array can contain a variety of things.
a. If the username and password in the authenticate call is invalid, you would supply the value “nvu”.
b. If on the other hand the user data is valid but you have no work to do for that user, you would supply the value “none”.
c. If you do have work to do for the that user, you can supply the full pathname of the company to be used in the current update.
d. If you want to use whatever QuickBooks company is currently open at the client end, simply supply an empty string.
The optional third member of the string array contains the number of seconds to wait before the next update. You would use this to in effect tell that QBWC client not to bother you for a specified time.
The optional fourth member of the string array contains the number of seconds to be used as the MinimumRunEveryNSeconds time for your web service, which tells QBWC how frequently your web service needs to be contacted.
I am not sure why the Authenticate at the end of the doc is different than this.
Solved by adding [XmlSerializerFormat] to IService, so responce become
<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
<s:Header />
<s:Body xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<authenticateResponse xmlns="http://developer.intuit.com/">
<authenticateResult>
<string>d0297d33-859d-4259-a598-5fbf328bac3b</string>
<string>nvu</string>
<string>3</string>
<string>60</string>
</authenticateResult>
</authenticateResponse>
</s:Body>
</s:Envelope>

Outlook rest api "synchronize messages" ignores $expand in subsequent calls

I am trying to synchronize an Outlook folder (say the Inbox) using the beta version of the Outlook Rest Api see doc here
I need only to retrieve the property IsRead and the PR_INTERNET_MESSAGE_ID
So following documentation, for the first synchronization my requests look like:
The following Http headers are always added:
request.Headers.Add("Prefer", "odata.track-changes");
request.Headers.Add("Prefer", "odata.maxpagesize=5"); //Use a small page size easier for debugging
The first initial full synchronization request
https://outlook.office365.com/api/beta/Me/MailFolders('inbox')/messages?$select=IsRead&$expand=SingleValueExtendedProperties($filter=(PropertyId eq 'String 0x1035'))
Good results the value array contain what I need.
The second request after the first request uses the deltatoken
https://outlook.office365.com/api/beta/Me/MailFolders('inbox')/messages?$select=IsRead,Subject&$expand=SingleValueExtendedProperties($filter=(PropertyId eq 'String 0x1035'))&$deltatoken=a758b90491954a61ad463ef3a0e690a2
Bad results, no SingleValueExtendedProperties entries
Next requests for paginations with skiptoken...
https://outlook.office365.com/api/beta/Me/MailFolders('inbox')/messages?$select=IsRead,Subject&$expand=SingleValueExtendedProperties($filter=(PropertyId eq 'String 0x1035'))&$skiptoken=e99ad10324464488b6b219ca5ed6be1c
Bad results again, same as 2.
It looks like a bug to me. Can you provide a workaround? From a list of ItemId is possible to retrieve easily the list of corresponding PR_InternetMessage_Id efficiently (not item per item)?
Note also that in the documentation it is written that:
The response will include a Preference-Applied: odata.track-changes
header. If you attempt to sync a resource that is not supported, this
header will not be returned in the response. Check for this header
before processing the response to avoid errors.
It seems that for 2. and 3. calls this response header "Preference-Applied" is not set.
The sync functionality today doesn't support extended properties. However, we are working to enable this and it should start working in a few weeks.
EDIT:
For a workaround for the very special case of the PR_INTERNETMESSAGE_ID look at the comment below.

Rest v2 report execution success, check status of the report 404 error

I run http://localhost:8081/jasperserver/rest_v2/reportExecutions with the relevant reportExecutionRequest and it returns 200 ok message. The following is the result of the post request.
<reportExecution>
<exports>
<export>
<id>5eb00aeb-0f42-4195-8aec-b306535c325f</id>
<outputResource>
<contentType>application/pdf</contentType>
<fileName>ReportTest1.pdf</fileName>
<outputFinal>false</outputFinal>
</outputResource>
<status>ready</status>
</export>
</exports>
<reportURI>/reports/test/ReportTest1</reportURI>
<requestId>8da22862-35b7-448a-8169-e282d12a02f5</requestId>
<status>ready</status>
<totalPages>0</totalPages>
</reportExecution>
But i can't access the report using http://localhost:8081/jasperserver/rest_v2/reportExecutions/5eb00aeb-0f42-4195-8aec-b306535c325f/status. Is totalpages 0 is the problem?.
You put wrong GUID into status check URL. It must come from <requestId> element (or requestId property in JSON response), not <id> in <export> block.
So, proper status check address for example is http://localhost:8081/jasperserver/rest_v2/reportExecutions/8da22862-35b7-448a-8169-e282d12a02f5/status.
And both calls must be performed with same session cookie, basic authentication will lead to 404 on status check or generated output retrieval.
Yes. If totalpages are 0 then there is no report that could be shown to you.
You might want to include a band in your report that is printed every time anyway so a report is generated in any case. That way you would get an empty report instead of nothing.

REST design - An operation with multiple possible non-success payloads

I'm trying to design a REST method for an 'Add person' operation that has a bunch of business rules. There are multiple possible non-success payloads (for the business purposes), requiring defined structure (to allow the consumer to parse the detail).
For 'Add a person', one of the following non-successes could happen:
We believe the system already has person.
Payload: The ID of that person
There are some possible matches.
Payload: A list of possible duplicates, and an override code to submit the record 'for sure'
General validation errors
Payload: Array of 'Error' object. (Standard across the API)
Question - Response object
If they're all to return under a single HTTP error status code, would it be right to have a varied object like:
OverrideCode (for (1))
PersonPossibleMatches [] (also for (1))
PersonDuplicateId (for (2))
ErrorList [] (for (3))
And have the consumer + documentation explain the interpretation?
Question - Response code
Is 400 (Bad Request) the correct (or correct enough) HTTP status code for this? We use it largely for the field validation (also scenario (3) - just wondering if business rule / 'intermediate state' things like this are any different.
Are there a more appropriate codes to spread the 3x scenarios over? And is it ok for the payloads to be different?
Thanks.
There are two aspects you need to consider
HTTP response code.
Error response payload.
Point number 1 is relatively simple. You have 400 error code for bad requests. And 409 for conflicting resources. So far simple.
Now let us consider your scenarios:
We believe the system already has person.
Payload: The ID of that person
Design suggestion: you can send a response like below
Response code: 409
{
"error_code": "resource_exists",
"error_description": "Resource person with ID XXX already exists"
"debug_info": "",
"link" : [
{
"href": "http://host-name/persons/123456",
"rel": "person"
}
]
}
2. There are some possible matches.
Payload: A list of possible duplicates, and an override code to submit the record 'for sure'
Design suggestion:
In this case - you may want to use PUT to override the resource. No need to use special code.
Response Code: 400
{
"error_code": "potential_duplicates",
"error_description": "Potentially the resource is duplicate of one of the following. Please use PUT with the resource ID to update"
"debug_info": "",
"link" : [
{
"href": "http://host-name/persons/234",
"rel": "person"
},
{
"href": "http://host-name/persons/456",
"rel": "person"
},
{
"href": "http://host-name/persons/789",
"rel": "person"
}
]
}
General validation errors
Payload: Array of 'Error' object. (Standard across the API)
Design suggestion: Here you can simply use 400 response code and a meaningful response like the examples above.
This depends in part on how the operation is performed. Since you said the operation has a bunch of business rules, and the system returns a payload with an ID when the person already exists, let's assume the operation is non-idempotent due to unrelated side-effects, performed with a POST to a factory endpoint.
1. We believe the system already has person.
This is a no-brainer. As suggested by others, you should use a 409 Conflict status code, with a body describing the nature of the conflict. In this case, it seems like there's nothing else the user needs to do, and he can move forward to the next step in the workflow. If there's something he can do, it should follow a procedure similar to the next case.
2. There are some possible matches.
Assuming that the clients don't have any key to unambiguously identify a person, which seems to be your case since you're considering possible matches, here you should also use a 409 Conflict status code, with a body describing the nature of the conflict, but with instructions on how to solve it.
Some other answer suggests you to allow an overwrite parameter that could be used any time, other suggests using a PUT, but I disagree with that since there's nothing preventing a client from using the overwrite all the time, or skipping the POST and use the PUT to replace an existent close-match. Also, you may have concurrent clients trying to add or change a person that match each other, or a common existent group, which will lead to an ABA conflict.
The conflict resolution body should return a valid tag for each possible match, and the client should be instructed to resubmit the same request with the If-Match header and the collection of tags. It may be a single tag, as long as it's generated from key data from each member in the collection. This will enforce that the user first must try the request without any override. If there's a conflict the user is forced to specify the exact entities that will be overwritten, and you're protected from inconsistent updates in case someone changes the current state between the first and the second request.
If the tags don't match in the second request, meaning the state was changed by something else between them, you should fail with a 412 Precondition Failed error.
3. General validation errors
This is also a no-brainer. A 400 Bad Request detailing the error, which seems to be standard across your API.
You could use 409 for the duplicate entry - and arguably for the possible duplicate entries with the extra info in the payload.
http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.10
A 400 for validation errors would be the expected response.
It is a judgement call at the end of the day and it depends what would be easier for your rest clients and what they are doing.
Here's the design process I use when creating RESTful API responses for error cases:
Design the response payload for the error condition. Regardless of the error code that's used, it is always good practice to return some content within error responses so that clients can learn more about the error and how to avoid it in the future.
If there is an HTTP status code that accurately describes this error already, and it's not already in use for another error case, use that.
If the closest matching error code is already being used for another error case, it's still OK to use that code, but the response payload becomes the place where the different error cases under that code get distinguished from each other. Your documentation should clearly state that inspecting this code wouldn't be enough, and that clients should then also look into the response to see exactly what happened.
If none of the above are applicable, use the closest error code that's appropriate. Just as in #3, the documentation of your response payloads makes this approach possible. If it's an error that the client influenced, make it a 400-range error, probably 400 - Bad Request. If it's the server's fault, then it should be a 500-range error, probably 500 - Internal Server Error.
Please, please, never throw 200 - OK for errors. The world left that nonsense behind in SOAP land, and nobody wants to go back.
Now let's apply that thinking to your error cases:
We believe the system already has that person. As correctly stated in another answer, 409 - Conflict accurately describes that error, so you should just use that. Putting some descriptive error information in the response payload would help new users of your API, even with such a definitive and understandable code.
There are some possible matches. There really isn't an HTTP code that describes this, and it's something that the client could influence, so the closest would be the catch-all code of 400 - Bad Request. Including the list of possible duplicates is an interesting idea, but make sure you don't end up returning enormous responses with huge numbers of possible matches. Also, make sure to also return URIs to those matching resources so that your clients can easily consume them. As for the "override code" suggestion, I wouldn't return that in the payload. Rather, I'd just document a parameter to your "Add a Person" operation that would allow for overrides to occur at any time, not just after a failed first attempt. For example: POST /people?overwrite=true.
General validation errors This is definitely a job for 400 - Bad Request, along with a descriptive error payload. It sounds like you're already allowing an array of errors to be returned from any API call, so that should be good enough to capture all the validation failures for the client-supplied data.
How about explaining it with a payload back; That is how we deal with REST responses for clients.
Response HTTP 409 with following payload response indicating to client what should they do next
`
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<request-result>
<http-code>200</http-code>
<description>REST Request is successfully processed</description>
<internal-error-info>Person already Exists</internal-error-info>
<message>Person with <id> already exists in sytem. Try picking different ID/Name combination</message>
<requested-operation>Add a Person</requested-operation>
<resource-name>Person</resource-name>
<status>SUCCESSFUL</status>
</request-result>
`