We are working on a new RESTful api using ASP.NET Web API. Many of our customers receive a nightly datafeed from us. For each feed we run a schedule SQL Agent job that fires off a stored procedure which executes an SSIS package and delivers files via Email/FTP. Several customers would benefit from being able to run this job on demand and then receive either their binary file (xml, xls, csv, txt, etc.) or a direct transfer of the data in JSON or XML.
The main issue is that the feeds generally take a while to run. Most run within a few minutes but there are a couple that can take 20 minutes (part of the project is optimizing these jobs). I need some help finding a best practice for setting up this api.
Here are our actions and proposed REST calls
Create Feed Request
POST ./api/feedRequest
Status 201 Created
Returns feedID in the body (JSON or XML)
We thought POST would be the correct request type because we're creating a new request.
Poll Feed Status
GET ./api/feedRequest/{feedID}
Status 102 Processing (feed is processing)
Status 200 OK (feed is completed)
Cancel Feed Request
DELETE .api/feedRequest/{feedID}
Status 204 No Content
Cancels feed request.
Get Feed
GET .api/feed/{feedID}
Status 200 OK
This will return the feed data. We'll probably pass parameters into the header to specify how they want their data. Setting feedType to "direct" would require JSON or XML setting in Content-Type. Setting feedType to "xml", "xls", "csv", etc., will transfer a binary data file back to the user. For some feeds this is a custom template that is specified in the feed definition already stored in our tables.
Questions
Does it appear that we're on the right track? Any immediate suggestions or concerns?
We are trying to decide whether to have a /feed resource and a /feedRequest resource or whether to keep it all under /feed. The above scenario is the two resource approach. The single resource would POST /feed to start request, PUT /feed to check the status, GET /feed when it's done. The PUT doesn't feel right and right now we're leaning towards the stated solution above. Does this seem right?
We're concerned about very large dataset returns. Should we be breaking these into pieces or will REST service handle these large returns. Some feeds can be in excess of 100MB.
We also have images that may be generated to accompany the feed, they're zipped up in a separate file when the feed stored procedure and package are called. We can keep this all in the same request and call GET /feed/{feedID}/images on the return.
Does anyone know of a Best Practice or a good GitHub example we could look at that does something similar to this with MS technologies? (We considered moving to ASP.NET Core as well.
Related
I'm implementing a voting system (like/dislike) according to RESTful API and wondering what should I do if the intention is unknown while a request.
My real case:
If a user voting the first time - create a vote, if the user already has a vote - delete his vote (a vote can be only like or dislike, not 0). So what HTTP method and HTTP status code are suitable for this?
PUT sounds like a good option for this. PUT is used to create or replace the resource at the target uri.
PUT /article/[id]/myvote
When the vote is created, you can return 201. When it is overwritten, you can return 204 or 200.
My real case: If a user voting the first time - create a vote, if the user already has a vote - delete his vote (a vote can be only like or dislike, not 0).
How would you do it with a web site?
You would have some HTML page with a form on it; the form would include some input controls to collect information from the user. When the user submits the form, the browser would -- using the standardized HTML form processing rules, use the data and metadata of the form to create an application/x-www-form-urlencoded document, which it would send to the server as described by the form.
Supposing for this particular case that the action of voting is not "essentially read only", we would choose a form method that is not safe, which is to say POST.
The status codes to be returned would be exactly the same status codes (with the same meanings) that every other POST on the web uses. In particular, if the origin server successfully handles the request, then one of the 2xx status codes would be returned, with an appropriate entity and meta data.
If you aren't sure which 2xx code to use, 200 is the safe bet.
Remember, the world wide web is the reference implementation in the REST architectural style. You won't be far off if you storyboard a website, then use that to guide your api design.
There are other alternatives, that require a bit more thinking. One would be to treat the voter's ballot as a document that is stored on the origin server. The voter might GET a copy of his ballot, make local changes to it, and then request that the origin server apply those changes to its local copy of the ballot.
In that design, we would normally use PUT to copy our local edits to the web server; and the server would -- as before -- return some 2xx status code if request was successful.
The client, of course, will need to understand the representation of the ballot well enough to make the appropriate edits to it. You'll also need some sort of link convention that allows the client to find the right instance of ballot to edit.
The actual representation on the server doesn't need to be a document, of course. The document delivered to the origin server might instead be copied into rows in a relational database (an arbitrary example).
In the specific case of associating a representation (document) with a resource that previously didn't have one (in other words, if GET /ballot would have returned a 404), then the HTTP standard requires that you respond with a 201 status code, rather than a 200 status code, and include the appropriate metadata to identify the newly created resource.
GET /ballots/68D7165A-D172-4E32-84F3-E494A01B5C8E
404 Not Found
Ballot: /ballots/68D7165A-D172-4E32-84F3-E494A01B5C8E
has not yet been submitted
PUT /ballots/68D7165A-D172-4E32-84F3-E494A01B5C8E
Content-Type: text/plain
LIKE
201 Created
Location: /ballots/68D7165A-D172-4E32-84F3-E494A01B5C8E
Thank you for voting!
GET /ballots/68D7165A-D172-4E32-84F3-E494A01B5C8E
200 OK
Content-Type: text/plain
LIKE
I'm writing a stateless API. You send it a document, it processes the document, and then returns the processed document. I'm struggling to understand how the RESTFUL rules apply. I'm not retrieving data, creating data or updating data on the server. There is no data on the server. What do I need to use in this case as the http method and why?
Good news - you are right that it is confusing.
Nothing on the server changes in response to the request. That suggests that the request is safe. So GET is the natural choice here... BUT -- GET doesn't support message payloads
A payload within a GET request message has no defined semantics; sending a payload body on a GET request might cause some existing implementations to reject the request.
HEAD, the other ubiquitous safe method, has the same problem (and is unsuitable when you want to return a document in any case).
The straight forward thing to do at this point is just use POST. It's important to realize that POST doesn't promise that a request is unsafe, only that it doesn't promise that it is safe -- generic components won't know that the request is safe, and won't be able to take advantage of that.
Another possibility is to look through the method registry, to see if somebody has already specified a method that has the semantics that you want. Candidates include SEARCH and REPORT, from the WebDAV specifications. My read of those specifications is that they don't actually have the right semantics for your case.
A Lot of ways to do what you want. But here is a small guideline.
I would create an endpoint that receives the document:
/receive_document
with a 'POST' method. Since you are 'sending' your document to the server
I would create an endpoint that serves up the processed document:
/processed_document
with a 'GET' method. Since you want to retrieve / see your document from the server?
The problem that you are trying to solve is mainly related to the document size, and processing time before returning the response.
Theorically, in order to use a restful approach, you have an endpoint, like yourhost.com/api/document-manager (it can be a php script, or whatever you are using as backend).
OK, so instead of naming the endpoint differently for each operation type, you just change the HTTP method, I'll try to make an example:
POST: used to upload the document, returns 200 OK when the upload is completed.
GET: returns the processed document, you can also return a different HTTP code, in case the document is not ready or even different if the document hasn't been uploaded. 204 no content or 412 precondition failed can be good candidates in case of unavailable document. I'm not sure about the 412, seems like it's returned when you pass a header in the request, that tells the server which resource to return. In your case, I think that the user processes one document at time. But to make a more solid api, maybe you can return an ID token to the user, into the POST response, then forward that token to the GET request, so the server will be able to know exactly which file the user is requesting.
PUT: this method should be used when updating a resource that has been already created with POST
DELETE: removes a resource, then return 204 or 404
OPTIONS: the response contains the allowed methods on this endpoint. You can use it to know for example which privileges has the currently logged user on a resource.
HEAD: is the same as a GET call, but it shouldn't return the response body. This is another good candidate for you to know when the document has been processed. You can upload with POST, then when the upload is done, start some kind of polling to the same endpoint with the HEAD method, finally when it will return "found", the polling will stop, and make the final GET call, which will start the download of the processed document.
HTTP methods are a neat way of managing HTTP communications, if used properly are the way to go, and easily understandable by other developers. And not to forget, you don't have to invent lots of different names for your endpoints, one is enough.
Hope that this helped you a little... But there are loads of guides on the net.
Bye!
I don't make this very general question post lightly, but I've maxed out what I can figure out on my own.
I'm starting up the QA Test Automation at my new company (they don't automate anything currently) and they've elected to use SoapUI for this procedure.
The application they're developing is a REST application (I don't really have any idea what that means) so I'm trying to build a REST request and Test Suite to reach our internal test server (which gets me XML that I am not allowed to post here - but it does reach the server successfully!) and then try to do a Login/Logout test.
I'm asking for help with methodology, because I have no idea where to start. I;ve Googled and trolled their support forums and looked in every corner of YouTube. Everyone is doing something different enough that I can't relate or use it.
Does anybody out there use SoapUI and test functional login in a REST application? I can write HTML/CSS and I'm pretty Java savvy, so I can do technical stuff if I know what to look for and what to learn.
Feeling overwhelmed. This was not in my job description when I started.
You should start with REST, and after that with SoapUI.
It is hard to catch the essence of REST.
It is like the hybrid of SOAP and a simple HTML driven web application. By SOAP you describe your web service with a WSDL. By a web application you send back hypermedia, so you don't have to write a WSDL or any descriptor to your application. This is convention over configuration...
REST uses the same approach, so it sends back hypermedia as well, but it sends not HTML, because it is not machine processable. The hypermedia sent by a REST API is usually an XML or a JSON derivative, for example ATOM+XML, JSON-LD, etc... If your web service does not send back hyperlinks, then it is not a real REST service just a SOAP web service with some REST constraints. There is a big difference. By SOAP you have to know everything about the operation name and the parameters if you want to send a request. If something changes, then your SOAP client breaks immediately. By REST your automated client follows links, checks their link-relation, or the bound linked data and recognizes which link is what it was looking for. So the modification of the link's url is irrelevant in the client, because it follows the vocabulary of the application, for example: hydra is a project which tries to describe these application level semantics in a general way, and tries to bind it to open linked data.
So at first you have to check that you have a real REST API, which follows the HATEOAS principle, or just a REST like SOAP web service. This is very important if you want to write end to end tests against it. By testing REST, you have to follow the links in your tests returned by the web API. By testing REST like SOAP, you have to build the links yourself in your tests... How to build such a link? I am sure you got a description of your REST API, but a link looks usually something like this in a JSON format:
{
rel: "link-relations",
method: "METHOD",
href: "domain/api-root/version/resource-path?map-reduce",
data: {...},
title: "...",
...
}
Ofc. there is some difference by every hypermedia, so you have to check your XML hypermedia type, how it represents links... The link-relations and maybe other attributes bind your data to the semantics of your REST API. The METHOD is always a verb, usually: GET, POST, PUT, PATCH, DELETE, maybe OPTIONS, and so on... There are only a few REST verbs, each of them has a specific meaning. In the url: The domain is the domain name of your application, e.g. https://example.com. The api-root is the root of your REST API, usually /api. The version is the version number of the currently used API, usually /v1. Only non backward compatible vocabulary changes should affect this version number. The resource-path is the path of your resource, usually /users or /users/inf3rno, etc... By REST you have resources. Each of them has a unique resource-path, and as you can see, every word in that path is a noun. So resources are something you can modify or display with a verb. For example a GET /users/inf3rno should return a representation of my profile page, and a PATCH /users/inf3rno {nick: "Leslie"} will turn my nick name: inf3rno into Leslie. By REST every resource should have only a single resource-path, so this is always a unique identifier, therefore the previous example with PATCH was not so perfect if you want to have multiple users with the same nick... The map-reduce in the queryString of the url, and it contains the sorting, pagination and filtering settings of the resource you want to modify or display. For example you can retrieve some data of every user with a first name: "Leslie" with GET /users?filters="firstName: 'Leslie'"&page=3&count=25. There is a difference between the following url-s: /users?id="inf3rno" and /users/inf3rno. The first one points to a collection resource and filters the result by its representation, the second one points to a single item resource. So a GET should return a collection representation with a single item by the first one, and an item representation by the seconds one. By the resource modifying methods there is no difference between the 2 urls... So it is recommended to add only a unique identifier to the resource-path if you want to select an item resource from a collection. By reducing the collection representation in any other ways, you have to add the filters to the queryString. The data part contains the params from the input fields. The title is the title of the link, and so on... You can use url-templates of you want to put input params to the url as well...
By REST the client maintains the session, and it sends the credentials (username, password) with every request. This is because the REST service is like John Snow, it does not know anything about the session or the identity of the user. It has to authenticate every request. To do that it uses a credentials -> permissions cache. This is a good approach, because the service scales very well if it does not have to maintain the session, which is part of the application state (the state of the client)... The REST service maintains only the resource state, which is not dependent on the clients...
The response to your REST requests is usually a hypermedia which contains the links you can follow and the data you requested. By REST like SOAP web services you get only the data in a JSON or XML format. Every response should contain a proper status header. The most frequent status codes are:
200 - ok (by successful PUT, PATCH and GET)
201 - created (by successful POST)
202 - accepted (by async request with eventual consistency)
204 - no content (by successful DELETE)
206 - partial content (by pagination with range headers)
301 - moved permanently (by migration)
304 - not modified (by cache)
400 - bad request (by invalid input)
401 - unauthorized (if no password given, or wrong username or password)
403 - access denied (if your account does not have permission to perform the task)
404 - not found (by unknown resource)
409 - conflict (by concurrency issues or duplicated request or db constraint problems)
410 - gone (if the resource was present before, but it is already deleted)
415 - unsupported media type (if the client wants the response in an unknown media type)
500 - internal server error (if the request was okay, but something went wrong by processing it)
By any error you have to send a detailed error message with a custom error code, which is understandable for the users, not just the developers...
That's how a REST API looks like.
To test it with e2e tests you have to set fixtures send REST requests and check their response. So it is like any other test... The SoapUI is not necessarily the best tool to do that, I read many complaints about it... I personally never used it, but it is not so hard to write your custom testing system. You need a testing framework, which can compare expected and actual values. You need something to send HTTP requests, or simply mock out the HTTP framework of the REST API. You need something for the fixture. By integration tests you can mock out the business logic and the HTTP framework as well, so by those you just inject the mock dependencies and check the calls. By e2e tests you need a test data set, and compare it with the result XML in your case... If you want to e2e test your client, you can use selenium if it is HTML based, maybe with nightwatch.js. By testing a real REST API, you'll need an automated browser, like selenium for your REST API implementation, which can select and follow the proper links. If you are developing the REST API you would write a browser like that anyways if you want an example client for your 3rd party client developers.
Thinking in a RESTful way, is it correct to use POST to create in a single call a resource and its sub-resource?
In my application I have the resource /notices/{notice} and sub-resource /notices/{notice}/photos/{photo}. A {photo} can't exists without a {notice}, but a {notice} doesn't have necessarily photos. Normally, I have to do first a POST to create a notice, then another POST to add a photo.
Now I want to allow the creation of a notice with a photo directly attached, enabling the creation of /notices/{notice} and /notices/{notice}/photos/{photo} with a single POST request to /notices/{notice}/photos/{photo}, with a multipart content describing both the resources (JSON for notice, binary for the photo). I think I will return the Location header only for the sub-resource.
Essentially, I want this to prevent Android clients to send two POST request to the server to upload a notice with a photo.
Is this correct? Or does it infringe REST principles? Should I consider to keep them separate and make two different requests? Or is it wrong to consider photos a separate entity from the notice? Should I keep only /notices/{notice} as resource, using PUT to add photos?
Which is the best solution?
Yes, there is nothing wrong with creating sub-resources at the same time you create the parent resource. It would even be OK to use PUT instead of POST to do this, as everything under the parent URL is part of/belongs to the resource you are uploading.
EDIT:
Now I want to allow the creation of a notice with a photo directly attached, enabling the creation of /notices/{notice} and /notices/{notice}/photos/{photo} with a single POST request to /notices/{notice}/photos/{photo}
This I disagree with. I suggest POSTing to the collection resource's URL, /notices. You provide the notice and its photos as a single representation (request body). The back end will then create resources for both the notice and any constituent photographs.
Although its essential in many cases, multiple edits/creates are not formally addressed by the RESTful architecture style.
Problem starts when you need to report failures on part of the collection, and the problem is worsen when they failures have different cause.
An it will effect choosing the right Hypermedia controls which are essential for the client to find a way forward in the given transaction/conversation.
so my suggestion is to have a nested cycle or POST requests rather than a sing POST to create nested Resources , so that it'd be easier and clearer to address each Resource state change.
We are trying to implement a REST API for an application we have now. We want to expose read/write capabilities for various resources using the REST API. How do we implement the "form" part of this? I get how to expose "read" of our data by creating RESTful URLs that essentially function as method calls and return the data:
GET /restapi/myobject?param=object-id-maybe
...and an XML document representing some data structure is returned. Fine.
But, normally, in a web application, an "edit" would involve two requests: one to load the current version of the resources and populate the form with that data, and one to post the modified data back.
But I don't get how you would do the same thing with HTTP methods that REST is sort of mapped to. It's a PUT, right? Can someone explain this?
(Additional consideration: The UI would be primarily done with AJAX)
--
Update: That definitely helps. But, I am still a bit confused about the server side? Obviously, I am not simply dealing with files here. On the server, the code that answers the requests should be filtering the request method to determine what to do with it? Is that the "switch" between reads and writes?
There are many different alternatives you can use. A good solution is provided at the microformats wiki and has also been referenced by the RESTful JSON crew. As close as you can get to a standard, really.
Operate on a Record
GET /people/1
return the first record
DELETE /people/1
destroy the first record
POST /people/1?_method=DELETE
alias for DELETE, to compensate for browser limitations
GET /people/1/edit
return a form to edit the first record
PUT /people/1
submit fields for updating the first record
POST /people/1?_method=PUT
alias for PUT, to compensate for browser limitations
I think you need to separate data services from web UI. When providing data services, a RESTful system is entirely appropriate, including the use of verbs that browsers can't support (like PUT and DELETE).
When describing a UI, I think most people confuse "RESTful" with "nice, predictable URLs". I wouldn't be all that worried about a purely RESTful URL syntax when you're describing web UI.
If you're submitting the data via plain HTML, you're restricted to doing a POST based form. The URI that the POST request is sent to should not be the URI for the resource being modified. You should either POST to a collection resource that ADDs a newly created resource each time (with the URI for the new resource in the Location header and a 202 status code) or POST to an updater resource that updates a resource with a supplied URI in the request's content (or custom header).
If you're using an XmlHttpRequest object, you can set the method to PUT and submit the data to the resource's URI. This can also work with empty forms if the server supplies a valid URI for the yet-nonexistent resource. The first PUT would create the resource (returning 202). Subsequent PUTs will either do nothing if it's the same data or modify the existing resource (in either case a 200 is returned unless an error occurs).
The load should just be a normal GET request, and the saving of new data should be a POST to the URL which currently has the data...
For example, load the current data from http://www.example.com/record/matt-s-example and then, change the data, and POST back to the same URL with the new data.
A PUT request could be used when creating a new record (i.e. PUT the data at a URL which doesn't currently exist), but in practice just POSTing is probably a better approach to get started with.