API Exploration
Although there are several questions that touch on the topic, none that I've been able to find address the core concept I'm trying to understand
If the root structure of an API is know, let's imaginatively say http://stackoverflow.com/api/service/ and we can successfully retrieve results from a know endpoint, let's say http://stackoverflow.com/api/service/answers/?id=39234, are there any methods, programmatic or otherwise, to identify other available endpoints?
Example
As an example, that's likely very source specific, googling the root url has revealed methods used in tags in pages from the source. I'm interested in similar broadly applicable techniques that may work.
I think it is not easy to answer your question in general, because there are a lot of different kinds of API's around in the world. For example:
Simple Object Access Protocol for which the API is often described via Web Services Description Language WSDL so you need to find the wsdl
For RESTful API's there are some more API description languages Overview of RESTful API Description Languages very famous is the OpenAPI specification
Also sometime you have XML documents that describe API's like in AVM's FritzBox router - which can be downloaded from the device.
Over the decades a lot more API descriptions have been developed like for CORBA
So it always depends on what kind of API's you are interested in and knowing the description languages for them and then to find the descriptions via search engines like google or Shodan etc.
Related
I'm new to web services world. I have created a rest api in ERP software that creates sales order. How will anyone outside the world know what parameters to send to this and in what format ? From all the videos I have watched and all materials read they talk about no api documentation is needed as it is REST service, People just know what request payload to send. I m not sure if I understand how that is possible. Its like I give somebody the url and tell them go figure. I tried sending orders with different parameter list and it is creating errors. But, if I send it in the way it accepts then it is working fine. Not sure if I understand the concepts well. Should I be creating documentation of this api telling what the request payload should look like ?
I completely disagree with the answer given by Joessel, which just propagates a typical RPC take on it, which is NOT REST at all!
In regards to how a service utilizing the REST architecture style should inform clients about what properties a resource supports and stuff like that, just look at traditional HTML pages. How are Web server able to tell your Browser what input it expects?
HTML is a media type that specifies the syntax to use as well as the semantics of each of the elements and attributes wich are admissible in a HTML document. I.e. HTML Forms enables a server to inform a client on the respective properties a resource support. In addition to that, a server also teaches a client on the respective target URI to send the request to, the HTTP operation to use upon sending the request as well as the media type to use for marshalling the request to a respective representation format. This is why you DON'T NEED any other documentation to interact with Web pages. Most arbitrary Web clients support HTML documents by default and therefore you don't need to reimplement the wheel to process such documents.
For non-HTML resources it is also just a matter of whether your client supports the respective media type or not. I.e. PNG files also follow a certain standard which allows arbitrary clients to show images instead of the actual bytes on your screen.
Most of those so called "REST APIs", which are truely RPC ones, just use custom JSON based message structures. JSON itself just defines the basic syntax but no semantics for any elements, attributes or other properties. It doesn't even add support for links. JSON Hyper-Schema is an extension which at least triest o add support for it, though it already requires to use an other media type than application/json. Though, if such formats are not well-defined and standardized, widespread adoption will not be possible on the long run. Hence creating a common media type is of importance to increase interoperability for such media types. I.e. for JSON based formats, HAL+JSON, HAL Forms, ION, and others provide definitions for basic JSON based message structures, with support for links and other features like form-support and other things.
So, if you take a closer look at the Web, you will find many concepts that you can reuse for a truely RESTful design. After all, REST just takes the ideas used on the Web for decades and attempts to offer the benefits resulting from these concepts to applications rather than humans alone. As such, it is always a good idea to first design the interaction flow as if one would interact with a traditional Web page and then take the concepts used in that design and apply it onto your application domain model.
As you don't need external documentation to interact with Web pages, so you don't need external documentation to interact with well defined message formats that follow a common media type as well. Through content-type negotiation both server and client will communicate with representation formats both support. So, the more (different) media types you support, the more likely you will be in the end to interact with different parties in that environment and if all of the supported media types are standardized you might not need any external documentation at all.
If you follow the academic definition of REST defined by Dr Fielding in his dissertation, it is true that such a service doesn't need any documentation. Hovewer, the chance are high that your service is not RESTful by his definition and that you need a documention to help your consumers. You can find an informative discussion about why a REST api doesn't need a documentation here.
That being said, you will probably need a documentation in most case. Depending on your needs and consumers of your api, a simple text file could be enough; maybe a markdown that you can easily share with other developers. Just remember to add all information needed to understand each route (verb, path, query, body, response...), and you can even throw some example code to make it easy to start.
And if you need something more robust, I can only advise you to follow the open api specification. One of the easiest way to get started is to use the swagger editor . You will even be able to publish your documentation in a nice way using one of the many tools out there (ex: redoc).
Writing a documention is never fun but nobody will use a product that they cannot understand, however great the product is.
Good luck,
Edit:
I rewrote the introduction as it was misleading, thanks to #Roman Vottner for pointing it out. THe previous intro was:
I don't know who told you that a documentation is not needed for a REST Api but I find this highly misleading for newcomers... A documentation is (super) important, especially if you are not the sole consumer of your API, and in that case I would say it's mandatory.
I have been trying to understand what a RESTful API/Web Application is supposed to look like and I cant help but wonder if the Web is already built under a RESTful architecture. Every website is basically a collection of web pages (A specific representation of the state of the Web App) and you change states by clicking links. Therefore you have Representational State Transfer!
Am I wrong in thinking this? Also, every SO answer to the question 'What is REST?' which I have managed to find fails to provide an example and show the difference between a non-RESTful API and a RESTful API.
Can someone provide that please? It would help in clearing things up. I do not want abstract terms in the answer because it would be more helpful to fit the abstract answer into an example rather than try to figure out an example from an abstract answer. Beginner speaking :)
Deepak,
in a nutshell, REST in an architectural style which helps your application make the most of the Web. The term was coined in the thesis by Roy Fielding called Architectural Styles and the Design of Network-based Software Architectures.
While a lot of REST services use HTTP and JSON, they are not limited to the aforementioned protocol and format. To tell whether a service is RESTful or not, one can use the so-called Richardson Maturity Model. The services that are not at the level three of the model are sometimes called RESTlike.
First of all, REST services are stateless, that is if you receive a huge list from the server, which is paginated, such as a list of books from the Amazon, the page you are on isn't stored at the server side, it is the responsibility of the client to inform the server about the next page's number. Also URIs are used to identify resources, e.g. each book's description can be retrieved using its unique URI, and HTTP methods such as GET and POST are used to work with resources, for instance, the former can be used to retrieve the book info and the latter to add a new book.
The crux of the style is a concept called Hypermedia As The Engine Of An Application state(HATEOAS). A simplistic explanation of what it is could be that the representations of resources (book info can be in JSON, XML, HTML etc.) should use hyperlinks to make those representations self-describing, e.g. if the book info can be edited, the link to do so should be added.
There are various proposals how to add hypermedia to representations. One possible format is Hypertext Application Language (HAL). Others include but not limited to JSON API, JSON-LD, UBER.
Personally I ventured to elaborate on the topic in my blog:
What is REST?
REST: Uniform interface
Introduction to Hypertext Application Language (HAL)
Regards, Dmitry
For a SaaS startup I'm involved in, I am building both a RESTful web API and a couple of client apps on different platforms that consume it. I think I've got the API figured out, but now I'm turning to the clients. As I've been reading about REST, I see that a key part of REST is discovery, but there seems to be a lot of debate between two different interpretations of what discovery really means:
Developer discovery: The developer hard-codes copious amounts of API details into the client, such as resource URI's, query parameters, supported HTTP methods, and other details that they've discovered through browsing the docs and experimenting with the API's responses. This type of discovery IMHO necessitates cool linkage and the API versioning question, and leads to hard coupling of the client code to the API. Not much better than if using a well-documented collection of RPC's it seems.
Runtime discovery - The client app itself is able to figure out everything it needs with little or no out-of-band information (presumably, only a knowledge of the media types the API deals with.) Links can be hot. But to make the API very efficient, a lot of link templating for query parameters seems to be needed, which makes out-of-band info creep back in. There are possibly other difficulties I haven't thought of yet since I haven't gotten to that point in development. But I do like the idea of loose coupling.
Runtime discovery seems to be the holy grail of REST, but I'm seeing precious little discussion about how to implement such a client. Almost all REST sources I've found seem to assume Developer discovery. Anyone know of some Runtime discovery resources? Best practices? Examples or libraries with real code? I'm working in PHP (Zend Framework) for one client. Objective-C (iOS) for the other.
Is Runtime discovery a realistic goal, given the present set of tools and knowledge in the developer community? I can write my client to treat all of the URI's in an opaque manner, but how to do this most efficiently is a question, especially over low-bandwidth connections. Anyway, URI's are only part of the equation. What about link templating in the Runtime context? How about communicating what methods are supported, aside from making a lot of OPTIONS requests?
This is definitely a tough nut to crack. At Google, we've implemented our Discovery Service that all our new APIs are built against. The TL;DR version is we generate a JSON Schema-like spec that our clients can parse - many of them dynamically.
That results means easier SDK upgrades for the developer and easy/better maintenance for us.
By no means the perfect solution, but many of our devs seem to like.
See link for more details (and make sure to watch the vid.)
Fascinating. What you are describing is basically the HATEOAS principle. What is HATEOAS you ask? Read this: http://en.wikipedia.org/wiki/HATEOAS
In layman's terms, HATEOAS means link following. This approach decouples your client from specific URL's and gives you the flexibility to change your API without breaking anyone.
You did your home work and you got to the heart of it: runtime discovery is holy grail. Don't chase it.
UDDI tells a poignant story of runtime discovery: http://en.wikipedia.org/wiki/Universal_Description_Discovery_and_Integration
One of the requirements that should be satisfied before you can call an API 'RESTful' is that it should be possible to write a generic client application on top of that API. With the generic client, a user should be able to access all the API's functionality. A generic client is a client application that does not assume that any resource has a specific structure beyond the structure that is defined by the media type. For example, a web browser is a generic client that knows how to interpret HTML, including HTML forms etc.
Now, suppose we have a HTTP/JSON API for a web shop and we want to build a HTML/CSS/JavaScript client that gives our customers an excellent user experience. Would it be a realistic option to let that client be a generic client application? No. We want to provide a specific look-and-feel for every specific data element and every specific application state. We don't want to include all knowledge about these presentation-specifics in the API, on the contrary, the client should define the look and feel and the API should only carry the data. This implies that the client has hard-coded coupling of specific resource elements to specific layouts and user interactions.
Is this the end of HATEOAS and thus the end of REST? Yes and no.
Yes, because if we hard-code knowledge about the API into the client, we loose the benefit of HATEOAS: server-side changes may break the client.
No, for two reasons:
Being "RESTful" is a property of the API, not of the client. As long as it is possible, in theory, to build a generic client that offers all capabilities of the API, the API can be called RESTful. The fact that clients don't obey the rules, is not the API's fault. The fact that a generic client would have a lousy user experience is not an issue. Why is it important to know that it is possible to have a generic client, if we don't actually have that generic client? This brings me to the second reason:
A RESTful API offers clients the option to choose how generic they want to be, i.e. how resilient to server-side changes they want to be. Clients which need to provide a great user experience may still be resilient to URI changes, to changes in default values and more. Clients doing batch jobs without user interaction may be resilient to other kinds of changes.
If you are interested in practical examples, checkout my JAREST paper. The last section is about HATEOAS. You will see that with JAREST, even highly interactive and visually attractive clients can be quite resilient to server-side changes, though not 100%.
I think the important point about HATEOAS is not that it is some holy grail client-side, but that it isolates the client from URI changes - it is assumed you are using known (or developer discovered custom) Link Relations that will allow the system to know which link for an object is the editable form. The important point is to use a media type that is hypermedia aware (e.g. HTML, XHTML, etc).
You write:
To make the API very efficient, a lot of link templating for query parameters seems to be needed, which makes out-of-band info creep back in.
If that link template is supplied in the previous request, then there is no out-of-band information. For example a HTML search form uses link templating (/search?q=%#) to generate a URL (/search?q=hateoas), but nothing is known by the client (the web browser) other than how to use HTML forms and GET.
This is a multi-part question. I just watched a very interesting presentation on YQL by the lead developer (a graduate of my MS program). While it was very compelling, and I am looking forward to trying it out, I am wondering if anyone knows of alternative frameworks for querying multiple web service APIs to make them appear seamless, the apparent purpose of YQL?
Yahoo's strategy has been to create XML schema definitions that bind a given web service's parameters into their YQL Open Table query parameters, which I think is very clever. Is there any tool that attempts (perhaps I am naive here) to automate the discovery of parameters in say a REST API? I am aware that with SOAP APIs, because there is a published WSDL, it makes automation easier, but is there yet no way to do this with REST? Is anyone trying?
Yes people are trying to produce description languages for REST. The most popular effort is WADL. There are lots of questions about WADL here on SO. Is it a good idea? In my opinion no.
REST does not need a discovery model beyond what it already has with hypermedia, because is trying to solve a problem at a different architectural layer than web services. Web services deliver data to an application's business logic/domain model. REST is about delivering content and behaviour to a presentation layer.
How about an analogy? Think of the different between an object and struct in C++. A struct is just simple data that some client process is going to manipulate. That's what a web service does, it returns a chunk of data, a struct. Sure maybe it did a bunch of server side processing to produce the result, but the end result is a lump of data. A REST interface delivers an object. i.e. It contains both data and the methods that can be used to manipulate that object. By definition, if you understand the uniform interface and you understand the returned media type, you already know what you can do with the response. Discovery mechanisms are redundant.
If you find this hard to believe, the think about the web. How does a web browser discover web pages? The web has no formalized discovery mechanism, and yet there is a world of information out there that we can discover with a web browser.
There is this little website http://zachgrav.es/yql/tablesaw/ which indeed auto-discovers parameters in a REST api and turns it into a YQL compatible table.
There are two ways to find information. Either you use a 100% unambiguous language or you use a natural language. Anything in between like YQL is doomed to fail because it delivers neither and works well only with the examples its authors tout.
I blogged about this at http://zscraper.wordpress.com/2012/05/30/enough-with-crawling-2. My personal stance is that you'll always get the most accurate results if you do your homework first, i.e. study the target domain and figure out how to query it unambiguously.
To answer your question and give you an alternative -- try Bobik. This is a cloud-backed scraping service that you control via REST API. Compose your "queries" in traditional syntax (Bobik supports Javascript, JQuery, XPATH and CSS) and call Bobik to run them from any client-side environment (webpages, mobile apps, or your server).
Hope this helps.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 5 years ago.
Improve this question
I'm writing a specification for a RESTful API for a new internal web service. It's not hugely long and fairly simple, but even so, it's my first time using strict REST (as opposed to cheating for practical reasons - avoiding PUT and DELETE because they're a pain in PHP, and so on). I was wondering if there were any standard methods or best practices for documenting a REST interface? I want the rest of the team to understand it at a glance, and for anyone that wants to write a client to be able to do so without understanding the underlying code.
Sure, REST APIs should ideally use HATEOAS and be hypertext driven (with heavy use of media types), but also having simple human-friendly documentation for developers to work off of is helpful.
Some specific tools that are helpful for generating documentation like this:
Swagger
An open spec for describing REST APIs [ github ]
Tools for auto-generating
Documentation
Code for your API
Donated to the OpenAPI initiative and renamed OpenAPI in 2015
Mashery
An open source project [ github ]
Tools for generating
Documentation
An exploration interface for your API
Apiary and API Blueprint
Write the API description in a DSL within markdown
Tools for auto-generating
Documentation
Mock server
Seems to be focused on ruby+mac devs
RAML
A spec for describing REST APIs [ github ]
WADL
A spec for writing discoverable API docs with XML
Some discussion comparing WSDL and WADL
APIgee
A commercial product with some documentation features
3scale
A commercial product with some documentation features
miredot
Commercial REST API documentation generator
Java specific
I've been using http://apiary.io, which is pretty nice. You can also export the API documentation to github.
In Roy's post here he states
A REST API should spend almost all of
its descriptive effort in defining the
media type(s) used for representing
resources and driving application
state, or in defining extended
relation names and/or
hypertext-enabled mark-up for existing
standard media types. Any effort spent
describing what methods to use on what
URIs of interest should be entirely
defined within the scope of the
processing rules for a media type
(and, in most cases, already defined
by existing media types).
A good ReST documentation would mean documenting your media type and only your media type.
In a typical scenario, you'd produce a document like so:
The Acme Corp XML formats
Link Discovery
Links to various resources are described in a document that can be found by issuing a GET or HEAD request to the server on a bookmark URI (typically the root of the server, http://www.acme.org), and looking for an HTTP Link header:
Link: <xxx>;rel="http://rel.acme.org/services";type=application/vnd.acme.services+xml
where the rel part is the link relationship, and the xxx is the URI for which the relationship has been established.
Link Relationships
This document defines the following relationship names:
http://rel.acme.org/services
The link relationship describes the list of links that can be navigated.
http://rel.acme.org/customers
The link for which this relationship is used is the list of customers.
Media Types
The application/vnd.acme.services+xml is a document with an xml serialization that describes a list of links an application may want to process.
<links>
<link rel="http://rel.acme.org/customers" href="http://www.acme.org/services/customers" type="application/vnd.acme.customers+xml" />
</link>
The applcation/vnd.acme.customers+xml is a document with an xml serialization that describes customers.
Example documents:
<customers>
<customer firstname="Darth" lastname="Vador" href="http://www.acme.org/services/customers/28" />
</customer>
etc...
The point is to give a way to the developer to follow the links you define. First find the link to the index so they can get the list of things they can navigate to.
Once they discover that document, they discover that they can see a list of customers at a certain Uri, and can do a GET against it.
If they find a customer of interest, they can follow the link defined in /customers/customer/#href and issue a GET to retrieve a representation of that customer.
From there, your media type could embed actions that are available to the user, using more links. You also have the additional option of issuing an OPTIONS request on the resource to know if you can allow deleting the resource, or a PUT if you can save the document back after modification.
So a good documentation doesn't ever:
give static links
give interaction such as "you can issue POST on Customer with this media type and that will mean the move operation". The client should issue a POST against Customer only because your XML document has specified it that way.
The point of all this is to achieve minimum coupling between clients and servers. The client can be very smart in displaying and discovering resources (showing forms and god knows what else), but is totally dumb as to what the actual workflow is: the server decides.
At my company, we've been very happy using WADL, Web Application Description Language. Wikipedia describes it as: "an XML-based file format that provides a machine-readable description of HTTP-based web applications". I find raw WADL easy to write, read, and understand, and it maps directly to RESTful concepts. The official project provides a simple spec, XSD and RELAX NG schemata, and Java tools.
A number of tools and resources exist for working with WADL, including:
wadl_stylesheets, XSLT stylesheets to create HTML documentation from WADL files
Restlet, a Java framework for building RESTful servers and clients, includes a WADL extension
A tip: try including human-readable documentation, such as descriptions, concepts, getting started, usage tips, etc, in the WADL document's doc element by including HTML elements, using the XHTML namespace. It can make a big difference!
You might find rest-tool useful.
It follows a language agnostic approach to write specification, mock implementation and automated unit-testing for RESTful APIs. It also provides a cook-book however it is in a very early stage, but its content is continuously growing.
The services you just described can be immediately used, so it is also good for experimenting.
Initially, we went for static documentation of resources but just had to field too many questions. Eventually, we moved to using Live documentation pages using IO/Docs (actually a fork).
Been working great.
To create understanding/documentation, heavyweight solutions aren't always needed. Examples of (great) heavyweight tools are: IO/Docs / Apigee (although great tools).
For tiny projects that already have a docchain setup (doxygen/phpdoc/phpdoctor/custom/etc) I use the following shellscript to just include the page in the full generated documentation:
https://gist.github.com/4496972
A demo: http://pastie.org/5657190
It just use custom comment-tags in your sourcecode.
It can also be a nice starting point for documenting any sourcecode (language).