I see that Go itself has a package net/http, which is adequate at providing everything you need to get your own REST APIs up and running. However, there are a variety of frameworks; the most popular maybe say gorilla.
Considering that one of the main things I need to do going forward is to build REST APIs that will access some back-end storage (databases, caches, etc.) to perform CRUD operation, is it good to go with Go's standard library itself, or should I consider using some frameworks?
Normally, people write a new library or framework which solves the problem present in the existing library. But a lot of the frameworks also tend to make things worse when actual demands are simple.
So I have few questions:
Is the basic library in go lang good enough to support basic to moderate functionality for REST?
If I do end up using the inbuilt library and tomorrow have to change it to use some framework (like a gorilla), how difficult/costly would that be?
Are frameworks really addressing the problems or just making simple problems complex?
I would be extremely grateful for someone to share his thoughts here (who has been through making this choice himself) while I research more of my own.
The net/http package is probably sufficient for most scenarios, but if you want to ease your development, you should use a third-party package, such as Gorilla.
For example, net/http's ServeMux does a great job at routing incoming requests for fixed URL paths but for pretty paths which use variables, you will need to implement a custom multiplexer while using Gorilla, you are getting this for free.
Another example is if you want to specify RESTful resources with
proper HTTP methods, it is hard to work with the standard
http.ServeMux, while with Gorilla's mux package,
requests can be matched based on URL host, path, path prefix,
schemes, header and query values, and HTTP methods.
One of the great benefits of Gorilla is that it is fully compatible with the net/http package and can be substituted in the future.
See 1.
I totally encourage you to use Gorilla's toolkit to develop REST services.
The built-in net/http package is sufficient to build a complete REST API. However, some of the libraries can make building an API slightly easier, particularly if the REST API is complex. Changing from the built-in facilities to any decent framework is relatively straightforward - they generally accept handlers of the http.Handler type.
In the end, though, this is an extremely situational choice. The best thing you can do is examine each available solution, contrast and compare, and build a proof of concept with the top options if you possibly can. First-hand experience will guide you best.
Related
I am looking into something lightweight, that, at a minimum should support the following features:
Support for easy definition of actions through metadata
Wrapper that extracts parameters from request into clojure map, or as function parameters
Support for multiple forms of authentication (basic, form, cookie)
basic authorization based of api method metadata
session object wrapped in clojure map
live coding from REPL (no need to restart server)
automatic serialization of return value to json and xml
have nice (pluggable) url parameter handling (eg /action/par1/par2 instead of /action?par1=val1&par2=val2)
I know it is relatively easy to roll my own micro-framework for each one of these options, but why reinvent the wheel if something like that already exists? Especially if it is:
Active project with rising number of contributors/users
Have at least basic documentation and tutorial online.
First of all, I think that you are unlikely to find a single shrinkwrapped solution to do all this in Clojure (except in the form of a Java library to be used through interop). What is becoming Clojure's standard Web stack comprises a number of libraries which people mix and match in all sorts of ways (since they happily tend to be perfectly compatible).1
Here's a list of some building blocks which you might find useful:
Ring -- Clojure's basic HTTP request handling library; all the other webby libraries (for writing routes &c.) that I know of are compatible with Ring. Ring is being actively developed, has a robust community, is very well-written and has a nice SPEC document detailing its design philosophy. This blog post provides a nice example of how it might be used (reacting to GitHub commits).
Sandbar -- currently an authentication library, more types of functionality planned; under development.
Compojure -- a mature and robust library which provides a nice DSL for writing routes to be used on top of Ring. This will give you the nice URL parameter handling.
Compojure-rest -- "a library for building RESTful applications on top of Compojure". Compojure-rest is, as far as I can tell, in its early stages of development; perhaps you might see this as an opportunity to influence its design. :-)
For dealing with XML, there's clojure.contrib.lazy-xml (and the helper library clojure.contrib.zip-filter.xml) and Enlive (the built-in clojure.xml namespace is currently not very usable); these would be used in tandem (though for your purposes the former might suffice).
For JSON, there a library in contrib and clojure-json (and I think there was at least one other lib I seem to be forgetting now...); pick the one you like best.
All of will be perfectly happy with a REPL-driven development style (see the accepted answer to this SO question for a Ring trick which is very much to the purpose here). I suppose the above collection of links does leave a few blind spots (in particular, the authentication story is still being ironed out, as far as I can tell), but hopefully it's a good start.
1The only single-package solution for building webapps in Clojure that I know of is Conjure, inspired by Rails; unfortunately I have to admit that I don't know much about it, so if you feel interested, follow the link and look around the sources, wiki &c.
While building my first Clojure rest service I found myself asking often the same question. The Clojure Toolbox helped me a lot: http://www.clojure-toolbox.com/
If you are looking for some sample, real-world, illustrative code to get you started, then you could study this clojure-news-feed on github project which demonstrates how to implement a non-trivial RESTful web service with compojure/ring that wraps both SQL (postgresql or mysql) and NoSQL (cassandra), search (solr), caching (redis), event logging (kafka), connection pooling (c3po), and real-time metrics via JMX.
This blog about Building a Scalable News Feed Web Service in Clojure provides a good introduction. I ran some load tests against this service on a humble AWS deployment and got about eighty transactions per second with less than a half second average latency per transaction.
Take a look at liberator library http://clojure-liberator.github.io/liberator/ It's noy a standalone solution, buy very good for rest service definition.
Just to provide an updated answer to this old question, currently (in 2018) I think Luminus provides an excellent starting point. It's using many of the libraries (ring, compojure, etc.) mentioned in previous answers, is modular and as close to "single package" as you can get with Clojure. Specifically for REST, take a look at compojure-api. Luminus recommends buddy for authentication, I've had good success using it both for traditional session-based auth as well as Oauth and stateless JWTs.
I'm just getting into the client-server data sync stage of my iPhone app project, and have managed to get my CoreData data model loading on both the iPhone client and my TurboGears server (which is good). I'm now beginning to tackle the problem of sync'ing data between the server and multiple clients, and while I could roll my own, this seems like one of those problems that is quite general and therefore there should be frameworks or libraries out there that provide a good deal of the functionality.
Does anyone know of one that might be applicable to this environment (e.g. Objective-C on iPhone, pyobjc / Python on the server)? If not, does anyone know of a design pattern or generally-agreed upon approach for this stuff that would be a good road to take for a self-implementation? I couldn't find a generally accepted term for this problem beyond "data synchronization" or "remote object persistence", neither of which hit much useful on Google.
I did come across the Funambol framework which looks like it provides this exact type of functionality, however, it is C++ / Java based and therefore seems like it might not be a good fit for the specific languages in my project.
Any help much appreciated.
Since you are using TurboGears already, take a look at the RestController documentation. Using RESTful services has become a widely adopted architecture with many implementations for both clients and servers. Matt Gemmell's MGTwitterEngine is a good example of the client implementation of a specific API, Twitter.
Has anybody done or seen a deployment of Apache Thrift in an iPhone app?
I am wondering if is a reasonable solution for a high-volume, low(er)-latency network service for iPhones compared to HTTP.
One noteworthy thing I found is a bug report about running Thrift on the iPhone, which seems to have been fixed. But that doesn't necessarily indicate that it's a done deal.
Thrift and HTTP aren't mutually exclusive. In fact thrift now ships with an HTTP transport implementation to use. It's also a really nice way to auto-generate server/client code that avoids a lot of marshalling/unmarshalling boilerplate while still being really fast. Its internal representation is basically binary JSON, so it's very similar to a RESTful web service (except being easier to code and much, much faster).
So... anyone able to answer the original question? If not, I'll dive in myself with thrift's included Cocoa support and see how it works on the iphone.
Just my two cents..
The accepted answer to this question, is an opinion to not use a technology, not an answer of whether it is possible.
Thrift, is an interface definition language, IDL, like Protobuf and Capt'n'Proto. They permit the definition of a client/server/server protocol which is platform agnostic. JSON and Plist don't provide the same level of type conformance.
Having previously lead an iOS team with 10Ms MAU using Google Protobuf v2.5 on iOS, Android, Windows, and server teams, I can attest that IDLs are great on mobile. Apple uses them for syncing iWork content.
My current team uses Thrift for iOS and Android clients, with a mostly Scala backend. I much prefer it to Protobuf.
We send Thrift payloads over HTTPS and WebSockets. Once you have defined (in Thrift) your our wire communication protocol (i.e. frame structure), it's very easy to evolve your APIs.
However, on iOS in particular there are some implementation issues. The current version of the library is quite poorly packaged, and if you hope to make an Objective-C framework (e.g. for iOS 8+), then you will not be able to out of the box with v0.9.2. This is because the library headers include local imports, (#import "TProtocol.h" instead of #import <Thrift/TProtocol.h>) with no umbrella headers. Worst of all, the Objective-C compiler generates very messy Objective-C classes, also including local imports from the Thrift library.
Some of these issues are pretty damning. It indicates to me that while use of an IDL is very much a good engineering decision, not many iOS teams are using Thrift, unless they're huge with the resources to write their own library.
I've always disliked frameworks that use a common interface definition that builds out both server and client code. It keeps both sides too much in lockstep where in reality server API changes must be very flexible in the versions of clients that are communicating with it.
There are helpful libraries that make JSON or PLIST communication over HTTP pretty easy, and decades of debugging and understanding the HTTP protocol and how to use it well. I would ignore that at your peril.
I have used thrift's objective c bindings for a large iPhone app with a few million users. As one of the posters mentioned we can use Http which gets the best of both worlds. However there is no asynchronous HTTP client for thrift. We had to build an event based wrapper to allow non-blocking I/O calls. The underlying layer still issues one call at a time which hit us in a big way because we have one server call that takes a long time but it does not block UI flow and another really fast one that does block UI flow. If the underlying layer is busy with the slow command our fast command just has to wait. I am trying to build asyc http in c++ which can then be used on the iPhone but that is someways off from being ready.
Thrift as an external API doesn't make sense. Use it internally rock and roll.
I know that Flickr provides both XML-RPC and REST ways of working with it.
There are standard XML-RPC libraries for every language (For example, Python has a built-in one xmlrpclib).
Standard XML-RPC libraries takes care of the serializing/deserializing as well as sending/receiving the responses.
It seems to me that websites that use the REST style for the same API would end up writing their own libraries in each language. Example: the Yahoo! Search SDK.
To me, it seems that the XML-RPC way is better, but all the evidence is to the contrary. Why?
So:
Why are most web services in REST style, and not in XML-RPC?
Are there downsides to XML-RPC that is not apparent?
Rest is not just easier, its a lot easier.
Xml-Rpc/soap has a lot of moving parts and a hefty amount of overhead, cognitive
and otherwise which (very often) is not needed, its complex and unless you
specifically need some of the features it provides it's just not worth it
Not every service request needs to be packaged up as a formal function call with
parameters
REST is also a formal system that's well defined and a great model for representing
the resources available on the web (hence the term REST)
Having said that, it's easy to make a lot of newbie mistakes using REST so google around for how to use it first, you'll be happy you did.
This is a great question. Unless you are taking advantage of hypermedia for discovery and standard media formats then you are not likely to be getting the benefits of REST. You might as well stick with XML-RPC.
Simple Answer: REST tends to be easier to implement
there are many discussions on that on the web, so I won't go deep on the answer. In short: It's easy. Easy to write, easy to understand, easy to debug. You can write it on your browser and it will probably bring back something useful. Very good.
This easiness come at the price of less "possibilities" but the theory goes that in the long run, easiness might be more worthy.
REST is the native architectural style of the Web. (In fact, it was reverse-engineered from the way the Web already works.) XML-RPC and SOAP attempt to take a very different (procedural, imperative) programming model and adapt it to the web. The result is that REST ends up being cleaner and more flexible.
Is there a general procedure for programming extensibility capability into your code?
I am wondering what the general procedure is for adding extension-type capability to a system you are writing so that functionality can be extended through some kind of plugin API rather than having to modify the core code of a system.
Do such things tend to be dependent on the language the system was written in, or is there a general method for allowing for this?
I've used event-based APIs for plugins in the past. You can insert hooks for plugins by dispatching events and providing access to the application state.
For example, if you were writing a blogging application, you might want to raise an event just before a new post is saved to the database, and provide the post HTML to the plugin to alter as needed.
This is generally something that you'll have to expose yourself, so yes, it will be dependent on the language your system is written in (though often it's possible to write wrappers for other languages as well).
If, for example, you had a program written in C, for Windows, plugins would be written for your program as DLLs. At runtime, you would manually load these DLLs, and expose some interface to them. For example, the DLLs might expose a gimme_the_interface() function which could accept a structure filled with function pointers. These function pointers would allow the DLL to make calls, register callbacks, etc.
If you were in C++, you would use the DLL system, except you would probably pass an object pointer instead of a struct, and the object would implement an interface which provided functionality (accomplishing the same thing as the struct, but less ugly). For Java, you would load class files on-demand instead of DLLs, but the basic idea would be the same.
In all cases, you'll need to define a standard interface between your code and the plugins, so that you can initialize the plugins, and so the plugins can interact with you.
P.S. If you'd like to see a good example of a C++ plugin system, check out the foobar2000 SDK. I haven't used it in quite a while, but it used to be really well done. I assume it still is.
I'm tempted to point you to the Design Patterns book for this generic question :p
Seriously, I think the answer is no. You can't write extensible code by default, it will be both hard to write/extend and awfully inefficient (Mozilla started with the idea of being very extensible, used XPCOM everywhere, and now they realized it was a mistake and started to remove it where it doesn't make sense).
what makes sense to do is to identify the pieces of your system that can be meaningfully extended and support a proper API for these cases (e.g. language support plug-ins in an editor). You'd use the relevant patterns, but the specific implementation depends on your platform/language choice.
IMO, it also helps to use a dynamic language - makes it possible to tweak the core code at run time (when absolutely necessary). I appreciated that Mozilla's extensibility works that way when writing Firefox extensions.
I think there are two aspects to your question:
The design of the system to be extendable (the design patterns, inversion of control and other architectural aspects) (http://www.martinfowler.com/articles/injection.html). And, at least to me, yes these patterns/techniques are platform/language independent and can be seen as a "general procedure".
Now, their implementation is language and platform dependend (for example in C/C++ you have the dynamic library stuff, etc.)
Several 'frameworks' have been developed to give you a programming environment that provides you pluggability/extensibility but as some other people mention, don't get too crazy making everything pluggable.
In the Java world a good specification to look is OSGi (http://en.wikipedia.org/wiki/OSGi) with several implementations the best one IMHO being Equinox (http://www.eclipse.org/equinox/)
Find out what minimum requrements you want to put on a plugin writer. Then make one or more Interfaces that the writer must implement for your code to know when and where to execute the code.
Make an API the writer can use to access some of the functionality in your code.
You could also make a base class the writer must inherit. This will make wiring up the API easier. Then use some kind of reflection to scan a directory, and load the classes you find that matches your requirements.
Some people also make a scripting language for their system, or implements an interpreter for a subset of an existing language. This is also a possible route to go.
Bottom line is: When you get the code to load, only your imagination should be able to stop you.
Good luck.
If you are using a compiled language such as C or C++, it may be a good idea to look at plugin support via scripting languages. Both Python and Lua are excellent languages that are used to script a large number of applications (Civ4 and blender use Python, Supreme Commander uses Lua, etc).
If you are using C++, check out the boost python library. Otherwise, python ships with headers that can be used in C, and does a fairly good job documenting the C/python API. The documentation seemed less complete for Lua, but I may not have been looking hard enough. Either way, you can offer a fairly solid scripting platform without a terrible amount of work. It still isn't trivial, but it provides you with a very good base to work from.