I would like to make a small program that scrapes information from a 3rd-party GWT-enabled website. Is it possible to somehow call this RPC directly? What would I need to reverse-engineer to do this (i.e. hopefully not the entire low-level protocol).
I am hoping I could somehow just call this from within my own server-side GWT or Servlet app.
It is possible... but you would have to go through their code to understand how the serialization/deserialization works.
Classes of interest are
RPC.java
ClientSerializationStreamWriter -> ServerSerializationStreamReader are the classes involved in making a GWT request.
ServerSerializationStreamWriter -> ClientSerializationStreamReader are the classes involved in creating and interpreting a response.
I am trying out a similar thing as an academic project, and I will add more information if I can decipher these classes.
I'm pretty sure GWT makes it difficult to do cross-site requests, for the sake of security.
Any solution you come up with will probably be hacky (and not flexible to future changes), and since you're presumably doing it without the consent of the site in question, probably a bad idea to begin with.
Is there some reason you can't ask the site to publish their data using a REST API?
Related
If I develop Booking REST service in Golang (i.e., in package booking). Is it a "GO way" to create BookingClient interface (backed up by struct) with business operations allowed, so that clients of my restful service would use BookingClient (imported from package booking) instead of sending http requests directly?
In general, no – if you provide a client in a particular language it'd only be a convenience, so (some) users can use your API easier. This of course assumes your client is well designed. I wouldn't provide merely an interface in Go just to indicate a set of possible API calls. This would be beneficial to a very narrow range of audience, probably for people developing a client for your API themselves, in programming language which just happened to be the same as implementation of your server. And even then they might not really like the idea of using the interface (e.g. they might only need a specific set of methods).
If you want to provide a client for your API, go ahead, do it, but separate it from the actual server (different package, maybe even different repo). In general one develops APIs over HTTP to allow for wide range of clients to access it, which could be written in any language. Instead of providing some interfaces I would invest my time in writing a good documentation.
In my opinion the answer to your question, assuming there is no more context provided, should be no different if you asked yourself if you should provide a client in, say, Python. The whole situation might change though if, for example, your API is used internally by your company and you develop mainly in Go.
It's usually preferable to do this, and most companies do, but provide documentation for working directly with the API. The main use case for that is people working with different languages than the ones you intended.
You can have a look at a new RESTful framework I wrote, that includes infrastructure to automatically compile clients with Go templates, although I haven't gotten to writing a Go client compiler. If you want to write one it would be greatly appreciated :) https://github.com/EverythingMe/vertex
Testing is important in Go, so writing testable code is something you should do. If you use direct http requests you will have a harder time writing unit tests, compared to using a mocked struct.
Is there any reason to use a Client rather than calling the functions that call the REST endpoints? It's usually harder to mock a bigger thing, such as a Client struct, rather than a group of small functions.
You should put the client at booking.Client to avoid repeating yourself (booking.BookingClient) and maybe rename Client to something more descriptive.
I am looking at backbone for a project. I have many legacy services which are non-restful that I have no choice but to use them as is. I see that I have to override Backbone.Model.sync, parse and many other methods and handle the ajax service calls. I am not sure how the routing is going to work but I can see that there will be a lot of extra code to make that work. My question is: Is Backbone really recommended if I have to work with non-restful services? I don't find any examples or discussion anywhere online which talks about it.
Backbone's automatic understanding of REST conventions boils down to probably about 50 lines of code. If your back-end APIs are all odd and unique, yes, you'll need to write code to talk to them, but you'll need that no matter which framework you use because no framework is going to understand the unique weirdnesses of your back end services. If you feel good about the basic MVC with event bindings design of backbone, stick with it. That's the core of it. And it's a tiny core, that's why it's called backbone.
As per routing, that's really handled in the browser as a single page app and the browser URL routing and associated backbone router/view code is entirely separate from the API patterns and URLs that provide the back end services. The two can be utterly unrelated and that's fine. You'll still be able to define your own browser routing however you see fit.
they're exposing a REST and SOAP API to access data but none of them seems really fitted to my purpose of inserting lots of data through external RPC calls, which are somehow time-critical (There are a dozen users working the same time on the data).
I only spend a few hours to get an idea how things work in SugarCRM but I figured out its best to place an independent RPC server beside of SugarCRM and then do the processing(huge) part and afterwards storing it into the database, using preferable Sugars own API.
Is there a way to include their core API in order to benefit from their model and persistence API ?
I'd like to know which files I need to include and how to access the running instance, if there is one.
Its actually quite simple what I am trying to do :
Receiving RPC calls
Do some matching
Update database
You see, there is no GUI part involved which justifies the long way of using modules, etc..
Well, I hope someone knows it. Its seems a pretty big beast to me and I'd love to keep it quick'n dirty.
thanks !
Here You find a starting point for your own API Extending the REST API and here is a overview of what API versions exist.
We are using a time critical custom API to speed up some calls.
For a SaaS startup I'm involved in, I am building both a RESTful web API and a couple of client apps on different platforms that consume it. I think I've got the API figured out, but now I'm turning to the clients. As I've been reading about REST, I see that a key part of REST is discovery, but there seems to be a lot of debate between two different interpretations of what discovery really means:
Developer discovery: The developer hard-codes copious amounts of API details into the client, such as resource URI's, query parameters, supported HTTP methods, and other details that they've discovered through browsing the docs and experimenting with the API's responses. This type of discovery IMHO necessitates cool linkage and the API versioning question, and leads to hard coupling of the client code to the API. Not much better than if using a well-documented collection of RPC's it seems.
Runtime discovery - The client app itself is able to figure out everything it needs with little or no out-of-band information (presumably, only a knowledge of the media types the API deals with.) Links can be hot. But to make the API very efficient, a lot of link templating for query parameters seems to be needed, which makes out-of-band info creep back in. There are possibly other difficulties I haven't thought of yet since I haven't gotten to that point in development. But I do like the idea of loose coupling.
Runtime discovery seems to be the holy grail of REST, but I'm seeing precious little discussion about how to implement such a client. Almost all REST sources I've found seem to assume Developer discovery. Anyone know of some Runtime discovery resources? Best practices? Examples or libraries with real code? I'm working in PHP (Zend Framework) for one client. Objective-C (iOS) for the other.
Is Runtime discovery a realistic goal, given the present set of tools and knowledge in the developer community? I can write my client to treat all of the URI's in an opaque manner, but how to do this most efficiently is a question, especially over low-bandwidth connections. Anyway, URI's are only part of the equation. What about link templating in the Runtime context? How about communicating what methods are supported, aside from making a lot of OPTIONS requests?
This is definitely a tough nut to crack. At Google, we've implemented our Discovery Service that all our new APIs are built against. The TL;DR version is we generate a JSON Schema-like spec that our clients can parse - many of them dynamically.
That results means easier SDK upgrades for the developer and easy/better maintenance for us.
By no means the perfect solution, but many of our devs seem to like.
See link for more details (and make sure to watch the vid.)
Fascinating. What you are describing is basically the HATEOAS principle. What is HATEOAS you ask? Read this: http://en.wikipedia.org/wiki/HATEOAS
In layman's terms, HATEOAS means link following. This approach decouples your client from specific URL's and gives you the flexibility to change your API without breaking anyone.
You did your home work and you got to the heart of it: runtime discovery is holy grail. Don't chase it.
UDDI tells a poignant story of runtime discovery: http://en.wikipedia.org/wiki/Universal_Description_Discovery_and_Integration
One of the requirements that should be satisfied before you can call an API 'RESTful' is that it should be possible to write a generic client application on top of that API. With the generic client, a user should be able to access all the API's functionality. A generic client is a client application that does not assume that any resource has a specific structure beyond the structure that is defined by the media type. For example, a web browser is a generic client that knows how to interpret HTML, including HTML forms etc.
Now, suppose we have a HTTP/JSON API for a web shop and we want to build a HTML/CSS/JavaScript client that gives our customers an excellent user experience. Would it be a realistic option to let that client be a generic client application? No. We want to provide a specific look-and-feel for every specific data element and every specific application state. We don't want to include all knowledge about these presentation-specifics in the API, on the contrary, the client should define the look and feel and the API should only carry the data. This implies that the client has hard-coded coupling of specific resource elements to specific layouts and user interactions.
Is this the end of HATEOAS and thus the end of REST? Yes and no.
Yes, because if we hard-code knowledge about the API into the client, we loose the benefit of HATEOAS: server-side changes may break the client.
No, for two reasons:
Being "RESTful" is a property of the API, not of the client. As long as it is possible, in theory, to build a generic client that offers all capabilities of the API, the API can be called RESTful. The fact that clients don't obey the rules, is not the API's fault. The fact that a generic client would have a lousy user experience is not an issue. Why is it important to know that it is possible to have a generic client, if we don't actually have that generic client? This brings me to the second reason:
A RESTful API offers clients the option to choose how generic they want to be, i.e. how resilient to server-side changes they want to be. Clients which need to provide a great user experience may still be resilient to URI changes, to changes in default values and more. Clients doing batch jobs without user interaction may be resilient to other kinds of changes.
If you are interested in practical examples, checkout my JAREST paper. The last section is about HATEOAS. You will see that with JAREST, even highly interactive and visually attractive clients can be quite resilient to server-side changes, though not 100%.
I think the important point about HATEOAS is not that it is some holy grail client-side, but that it isolates the client from URI changes - it is assumed you are using known (or developer discovered custom) Link Relations that will allow the system to know which link for an object is the editable form. The important point is to use a media type that is hypermedia aware (e.g. HTML, XHTML, etc).
You write:
To make the API very efficient, a lot of link templating for query parameters seems to be needed, which makes out-of-band info creep back in.
If that link template is supplied in the previous request, then there is no out-of-band information. For example a HTML search form uses link templating (/search?q=%#) to generate a URL (/search?q=hateoas), but nothing is known by the client (the web browser) other than how to use HTML forms and GET.
I am working on a experimental website (which is accessible through web browser) that will act as a front-end to a restful interface (a sub-system). The website will serve as an interface between a user and the restful interface, as it will make http requests to the restful interface for almost all database operations. Authentication will probably be done using openid and authorization for the database operations will be done via oAuth.
Just out of curiousity, is this a feasible solution or I should develop two systems that accesses the database in parallel (i.e. the website has its own data access logic, and the restful interface has another data access logic)? And what are the pros/cons if I insist on doing it this way (it is just an experiment project for me to learn things like how OpenID and oAuth work in real life anyway) besides there will be more database queries and http requests generated for each transaction?
Your concept sounds quite feasible. I'd say that you'll get some fairly good wins out of this approach. For starters you'll get a large degree of code reuse since you'll be able to put other front ends on top of the RESTful service. Additionally, you'll be able to unit test this architecture with relative ease. Finally, you'll be able to give 3rd party developers access to the same API that you use (subject possibly to some restrictions) which will be a huge win when it comes to attracting customers and developers to your platform.
On the down side, depending on how you structure your back end you could run into the standard problem of granularity. Too much granularity and you'll end up making lots of connections for very little amounts of data. Too little and you'll get more data than you need in some cases. As for security, you should be able to lock down the back end so that requests can only be made under certain conditions: requests contain an authorization token, api key, etc.
Sounds good, but I'd recommend that you do this only if you plan to open up the restful API for other UI's to use, or simply to learn something cool. Support HTML XML and JSON for the interface.
Otherwise, use a great MVC framework instead (asp.net MVC, rails, cakephp). You'll end up with the same basic result but you'll be "strongerly" typed to the database.
with a modern javascript library your approach is quite straightforward.
ExtJS now has always had Ajax support, but it is now able to do this via a REST interface.
So, your ExtJS user interface components populate receive a URL. They populate themselves via a GET to the URL, and store update via POST to the URL.
This has worked really well on a project I'm currently working on. By applying RESTful principles there's an almost clinical separation between the front & backends - meaning it would be trivial undertaking to replace other. Plus, the API barely needs documenting, since it's an implementation of an existing mature standard.
Good luck,
Ian
woow! A question from 2009! And it's funny to read the answers. Many people seem to disagree with the web services approach and JS front end - which has nowadays become kind of standard, known as Single Page Applications..
I think the general approach you outline is quite feasible -- the main pro is flexibility, the main con is that it won't protect clueless users against their own ((expletive deleted)) abuses. As most users are likely to be clueless, this isn't feasible for mass consumption... but, it's fine for really leet users!-)
So to clarify, you want to have your web UI call into your web service, which in turn calls into the database?
This is exactly the path I took for a recent project and I think it was a mistake because you end up creating a lot of extra work. Here's why:
When you are coding your web service, you will create a library to wrap database calls, which is typical. No problem there.
But then when you code your web UI, you will end up creating another library to wrap calls into the REST interface... because otherwise it will get cumbersome making all the raw HTTP calls.
So you essentially created 2 data access libraries, one to wrap DB and the other to wrap the Web service calls. This basically doubles the amount of work you do, because for every operation on a resource, you will end up implementing in both libraries. This gets tiring real fast.
The simpler alternative is to create a single library that wraps access to the database, as before, then use that library from BOTH the web UI and web service.
This is assuming that your web UI and web service reside on the same network and both have direct access to the backend database server (which was the case for me). In this setup having both go directly to the database is also a lot more efficient then having the UI go through the web service.