How to allow for POST to an MVC5 Controller for large sets of data. - rest

I have seen several posts addressing this issue or similar to this issue for requests or GETs. I am not having this problem getting the data from the server, its solely on the POST.
The Errors I get are
The JSON request was too large to be deserialized.
or either
Error during serialization or deserialization using the JSON JavaScriptSerializer. The length of the string exceeds the value set on the maxJsonLength property. Parameter name: input
I haven't been able to consistently determine which actions result in which error, but it is predominately the latter one.
In an effort to get the value of the MaxJsonSize value, on the Index method of the controller, I get this data and dump it into a viewbag to write to console on the client side. Every time it comes back at 10k (102400).
If I reduce the data package size, and still serialize as previously, I get no errors.
In fiddler I can inspect the package and all the JSON is deserializable in fiddler, so I don't see an issue in my JSON. Additionally if I console.log(data) chrome sees no problems with it either.
The VM in the controller is the same for both POST and GET. With the exception there is more data with the POST than the GET. To test this I got a huge data set from the server;
GeoJSON data for all 50 states. Following was the result.
GET Content-Length: 3229309 return 200
POST Content-Length: 2975244 return 500
The POST failed in this scenario and returned the second error listed previously.
I only changed the data minimally (one string) and don't know why when sent back its smaller, but the JSON for both the GET and the POST is virtually identical.
I've tried changing the web.config file:
<system.web.extensions>
<scripting>
<webServices>
<jsonSerialization maxJsonLength="2147483644"/>
</webServices>
</scripting>
</system.web.extensions>
I just added this to the end of my config file just prior to
I've also added a parameter in Settings.config
<add key="aspnet:MaxJsonDeserializerMembers" value="2147483644" />
I have also verified that this param loads as part of the application settings in IIS.
Is there something else I can try to change to allow for these large data sets to be sent in a POST.
As a last resort, I was going to pull all of the GeoJSON data out of the POST. However when a user navigates back and they haven't changed what they were mapping, we'd have to find all the GeoJSON data again, causing undue work on the server etc. I thought if I only had to fetch it once that would be best from an efficiency perspective.

I struggled with this too, nothing I changed in web.config helped, despite several SO answers looking relevant. They helped with returning large JSON data, but the large JSON post kept failing. In the end I found this:
increase maxJsonLength for JSON POST and used the solution there, and it worked for me.
Quoting from there :
the MVC json serializer does not look at the webconfig to get the max length (thats for asp.net web services). you need to use your own serializer. you override ExecuteResult and supply you own json serializer. to override the input, create a new JsonValueProviderFactory, then override ValueProvider in the controller to return your new json factory when its a json request.

Related

axios data differs from request.response

I'm facing a quite improbable situation. My http request is returning the right data if I read it in the network tab but when trying to retrieve it in the code the data I receive isn't the one I'm expected.
Would you have any lead on understanding what's going on ?
When printing the whole response object, I can see that the response.data doesn't have the same content than the response.request.response
One return an object whose amount is 1450 the other 3750.
Would you have any idea why ?
Screenshot of the console content display different value for data and request.response

Apache ManifoldCF: Get a history report for a repository connection over REST API

I'm trying to get a history report for a repository connection over ManifoldCF REST API. According to the documentation:
https://manifoldcf.apache.org/release/release-2.11/en_US/programmatic-operation.html#History+query+parameters
It should be possible with the following URL (connection name: myConnection):
http://localhost:8345/mcf-api-service/json/repositoryconnectionhistory/myConnection
I have also tried to use some of the history query parameters:
http://localhost:8345/mcf-api-service/json/repositoryconnectionhistory/myConnection?report=simple
But I am not sure if I am using them correctly or how they should be attached to the URL, because it is not mentioned in the documentation.
The problem is also that I don't receive any error, but an empty object, so it is difficult to debug. The API returns an empty object even for a non-existing connection.
However it works for resources, which doesn't have any attributes, e.g.:
http://localhost:8345/mcf-api-service/json/repositoryconnectionjobs/myConnection
or
http://localhost:8345/mcf-api-service/json/repositoryconnections/myConnection
Thanks in advace for any help.
I also wrote a message to ManifoldCF team and they gave me an answer. So I summed up it for you below.
Query parameters go after the fixed "path" part of the URL and are of the form ?parameter=value&parameter2=value2...
So in the same way as in any other URL.
The problem was that I didn't supply the activity(s) that I wanted to match. Possible activities are e.g. fetch, process. My example:
http://localhost:8345/mcf-api-service/json/repositoryconnectionhistory/myConnection?activity=process&activity=fetch
Finally, the reason why I didn't get an error when I used a connection name that is bogus is because the underlying implementation is merely doing a dumb query and not checking for the legality/existence of the connection name.

How to handle error responses in a REST endpoint that accepts different Accept header values.

I'm trying to add a new content type to a REST endpoint. Currently it only returns json but I now need to be able to return also a CSV file.
As far as I know, the best way to do this is by using the Accept header with value text/csv and then add a converter that is able to react to this and convert the returned body to the proper CSV representation.
I've been able to do this but then I have a problem handling exceptions. Up until know, all the errors returned are in json. The frontend expects any 500 status code to contain a specific body with the error. But now, by adding the option to return either application/json or text/csv to my endpoint, in case of an error, the converter to be used to transform the body is going to be either the jackson converter or my custom one depending on the Accept header passed. Moreover, my frontend is going to need to read the content-type returned and parse the value based on the type of representation returned.
Is this the normal approach to handle this situation?
A faster workaround would be to forget about the Accept header and include a url parameter indicating the format expected. Doing it this way, I'd be able to change the content-type of the response and the parsing of the data directly in the controller as the GET request won't include any Accept header and it will be able to accept anything. There are some parts of the code already doing this where the only expected response format is CSV so I'm going to have a difficult time defending the use of the Accept header unless there is a better way of handling this.
my frontend is going to need to read the content-type returned and parse the value based on the type of representation returned.
Is this the normal approach to handle this situation?
Yes.
For example, RFC 7807 describes a common format for describing problems. So the server would send an application/problem+json or an application/problem+xml representation of the issue in the response, along with the usual meta data in the headers.
Consumers that understand application/problem+json can parse the data with in, and forward a useful description of the problem to the user/logs whatever. Consumers that don't understand that representation are limited to acting on the information in the headers.
A faster workaround would be to forget about the Accept header and include a url parameter indicating the format expected.
That's also fine -- more precisely, you can have a different resource responsible for the each of the different media-types that you support.
It may be useful to review section 3.4 of RFC 7231, which describes the semantics of content negotiation.

GoodData Export Reports API Call results in incomplete file

I've developed a method that does the following steps, in this order:
1) Get a report's metadata via /gdc/md//obj/
2) From that, get the report definition and use that as payload for a call to /gdc/xtab2/executor3
3) Use the result from that call as payload for a call to /gdc/exporter/executor
4) Perform a GET on the returned URI to download the generated CSV
So this all works fine, but the problem is that I often get back a blank CSV or an incomplete CSV. My workaround has been to put a sleep() in between getting the URI back and actually calling a GET on the URI. However, as our data grows, I have to keep increasing the delay on this, and even then it is no guarantee that I got complete data. Is there a way to make sure that the report has finished exporting data to the file before calling the URI?
The problem is that export runs as asynchronous task - result on the URL returned in payload of POST to /gdc/exporter/executor (in form of /gdc/exporter/result/{project-id}/{result-id}) is available after exporter task finishes its job.
If the task has not been done yet, GET to /gdc/exporter/result/{project-id}/{result-id} should return status code 202 which means "we are still exporting, please wait".
So you should periodically poll on the result URL until it returns status 200 which will contain a payload (or 40x/50x if something wrong happened).

RESTful, efficient way to query List.contains(element)?

Given:
/images: list of all images
/images/{imageId}: specific image
/feed/{feedId}: potentially huge list of some images (not all of them)
How would you query if a particular feed contains a particular image without downloading the full list? Put another way, how would you check whether a resource state contains a component without downloading the entire state? The first thought that comes to mind is:
Alias /images/{imageId} to /feed/{feedId}/images/{imageId}
Clients would then issue HTTP GET against /feed/{feedId}/images/{id} to check for its existence. The downside I see with this approach is that it forces me to hard-code logic into the client for breaking down an image URI to its proprietary id, something that REST frowns upon. Ideally I should be using the opaque image URI. Another option is:
Issue HTTP GET against /feed/{feedId}?contains={imageURI} to check for existence
but that feels a lot closer to RPC than I'd like. Any ideas?
What's wrong with this?
HEAD /images/id
It's unclear what "feed" means, but assuming it contains resources, it'd be the same:
HEAD /feed/id
It's tricky to say without seeing some examples to provide context.
But you could just have clients call HEAD /feed/images/{imageURI} (assuming that you might need to encode the imageURI). The server would respond with the usual HEAD response, or with a 404 error if the resource doesn't exist. You'd need to code some logic on the server to understand the imageURI.
Then the client either uses the image meta info in the head, or gracefully handles the 404 error and does something else (depending on the application I guess)
There's nothing "un-RESTful" about:
/feed/{feedId}?contains={imageURI}[,{imageURI}]
It returns the subset as specified. The resource, /feed/{feedid}, is a list resource containing a list of images. How is the resource returned with the contains query any different?
The URI is unique, and returns the appropriate state from the application. Can't say anything about the caching semantics of the request, but they're identical to whatever the caching semantics are of the original /feed/{feedid}, it simply a subset.
Finally, there's nothing that says that there even exists a /feed/{feedid}/image/{imageURL}. If you want to work with the sub-resources at that level, then fine, but you're not required to. The list coming back will likely just be a list of direct image URLS, so where's the link describing the /feed/{feedid}/image/{imageURL} relationship? You were going to embed that in the payload, correct?
How about setting up a ImageQuery resource:
# Create a new query from form data where you could constrain results for a given feed.
# May or may not redirect to /image_queries/query_id.
POST /image_queries/
# Optional - view query results containing URIs to query resources.
GET /image_queries/query_id
This video demonstrates the idea using Rails.