Is there a general procedure for programming extensibility capability into your code?
I am wondering what the general procedure is for adding extension-type capability to a system you are writing so that functionality can be extended through some kind of plugin API rather than having to modify the core code of a system.
Do such things tend to be dependent on the language the system was written in, or is there a general method for allowing for this?
I've used event-based APIs for plugins in the past. You can insert hooks for plugins by dispatching events and providing access to the application state.
For example, if you were writing a blogging application, you might want to raise an event just before a new post is saved to the database, and provide the post HTML to the plugin to alter as needed.
This is generally something that you'll have to expose yourself, so yes, it will be dependent on the language your system is written in (though often it's possible to write wrappers for other languages as well).
If, for example, you had a program written in C, for Windows, plugins would be written for your program as DLLs. At runtime, you would manually load these DLLs, and expose some interface to them. For example, the DLLs might expose a gimme_the_interface() function which could accept a structure filled with function pointers. These function pointers would allow the DLL to make calls, register callbacks, etc.
If you were in C++, you would use the DLL system, except you would probably pass an object pointer instead of a struct, and the object would implement an interface which provided functionality (accomplishing the same thing as the struct, but less ugly). For Java, you would load class files on-demand instead of DLLs, but the basic idea would be the same.
In all cases, you'll need to define a standard interface between your code and the plugins, so that you can initialize the plugins, and so the plugins can interact with you.
P.S. If you'd like to see a good example of a C++ plugin system, check out the foobar2000 SDK. I haven't used it in quite a while, but it used to be really well done. I assume it still is.
I'm tempted to point you to the Design Patterns book for this generic question :p
Seriously, I think the answer is no. You can't write extensible code by default, it will be both hard to write/extend and awfully inefficient (Mozilla started with the idea of being very extensible, used XPCOM everywhere, and now they realized it was a mistake and started to remove it where it doesn't make sense).
what makes sense to do is to identify the pieces of your system that can be meaningfully extended and support a proper API for these cases (e.g. language support plug-ins in an editor). You'd use the relevant patterns, but the specific implementation depends on your platform/language choice.
IMO, it also helps to use a dynamic language - makes it possible to tweak the core code at run time (when absolutely necessary). I appreciated that Mozilla's extensibility works that way when writing Firefox extensions.
I think there are two aspects to your question:
The design of the system to be extendable (the design patterns, inversion of control and other architectural aspects) (http://www.martinfowler.com/articles/injection.html). And, at least to me, yes these patterns/techniques are platform/language independent and can be seen as a "general procedure".
Now, their implementation is language and platform dependend (for example in C/C++ you have the dynamic library stuff, etc.)
Several 'frameworks' have been developed to give you a programming environment that provides you pluggability/extensibility but as some other people mention, don't get too crazy making everything pluggable.
In the Java world a good specification to look is OSGi (http://en.wikipedia.org/wiki/OSGi) with several implementations the best one IMHO being Equinox (http://www.eclipse.org/equinox/)
Find out what minimum requrements you want to put on a plugin writer. Then make one or more Interfaces that the writer must implement for your code to know when and where to execute the code.
Make an API the writer can use to access some of the functionality in your code.
You could also make a base class the writer must inherit. This will make wiring up the API easier. Then use some kind of reflection to scan a directory, and load the classes you find that matches your requirements.
Some people also make a scripting language for their system, or implements an interpreter for a subset of an existing language. This is also a possible route to go.
Bottom line is: When you get the code to load, only your imagination should be able to stop you.
Good luck.
If you are using a compiled language such as C or C++, it may be a good idea to look at plugin support via scripting languages. Both Python and Lua are excellent languages that are used to script a large number of applications (Civ4 and blender use Python, Supreme Commander uses Lua, etc).
If you are using C++, check out the boost python library. Otherwise, python ships with headers that can be used in C, and does a fairly good job documenting the C/python API. The documentation seemed less complete for Lua, but I may not have been looking hard enough. Either way, you can offer a fairly solid scripting platform without a terrible amount of work. It still isn't trivial, but it provides you with a very good base to work from.
Related
I see that Go itself has a package net/http, which is adequate at providing everything you need to get your own REST APIs up and running. However, there are a variety of frameworks; the most popular maybe say gorilla.
Considering that one of the main things I need to do going forward is to build REST APIs that will access some back-end storage (databases, caches, etc.) to perform CRUD operation, is it good to go with Go's standard library itself, or should I consider using some frameworks?
Normally, people write a new library or framework which solves the problem present in the existing library. But a lot of the frameworks also tend to make things worse when actual demands are simple.
So I have few questions:
Is the basic library in go lang good enough to support basic to moderate functionality for REST?
If I do end up using the inbuilt library and tomorrow have to change it to use some framework (like a gorilla), how difficult/costly would that be?
Are frameworks really addressing the problems or just making simple problems complex?
I would be extremely grateful for someone to share his thoughts here (who has been through making this choice himself) while I research more of my own.
The net/http package is probably sufficient for most scenarios, but if you want to ease your development, you should use a third-party package, such as Gorilla.
For example, net/http's ServeMux does a great job at routing incoming requests for fixed URL paths but for pretty paths which use variables, you will need to implement a custom multiplexer while using Gorilla, you are getting this for free.
Another example is if you want to specify RESTful resources with
proper HTTP methods, it is hard to work with the standard
http.ServeMux, while with Gorilla's mux package,
requests can be matched based on URL host, path, path prefix,
schemes, header and query values, and HTTP methods.
One of the great benefits of Gorilla is that it is fully compatible with the net/http package and can be substituted in the future.
See 1.
I totally encourage you to use Gorilla's toolkit to develop REST services.
The built-in net/http package is sufficient to build a complete REST API. However, some of the libraries can make building an API slightly easier, particularly if the REST API is complex. Changing from the built-in facilities to any decent framework is relatively straightforward - they generally accept handlers of the http.Handler type.
In the end, though, this is an extremely situational choice. The best thing you can do is examine each available solution, contrast and compare, and build a proof of concept with the top options if you possibly can. First-hand experience will guide you best.
I have integrated Lua with my ObjC code (iphone game). The setup was pretty easy, but now, I have a little problem with the bridging. I have googled for results, etc... and it seems there isn't anything that could work without modifications. I mean, I have checked luaobjc bridge (it seems pretty old and dicontinued), I heard about LuaCocoa but it seems not to work on iphone, and wax is too thick.
My needs are pretty spare, I just need to be able to call objc methods from lua and don't mind having to do extra work to make it work (I don't need a totally authomatic bridging system).
So, I have decided to build a little bridge myself based on this page http://anti-alias.me/?p=36. It has key information about how to accomplish what I need, but the tutorial is not completed and I have some doubts about how to deal with method overloading when called from lua, etc...
Do anybody know if there exist any working bridge between objc and lua on the iphone or if it could be so hard to complete the bridge the above site offers?
Any information will be welcomed.
Don't reinvent the wheel!
First, you are correct that luaobjc and some other variants are outdated. A good overview can be found on the LuaCocoa page. LuaCocoa is fine but apparently doesn't support iPhone development, so the only other choice is Wax. Both LuaCocoa and Wax are runtime bridges, which means that you can (in theory) access every Objective-C class and method in Lua at the expense of runtime performance.
For games and from my experience the runtime performance overhead is so significant that it doesn't warrant the use of any runtime binding library. From a perspective of why one would use a scripting language, both libraries defy the purpose of favoring a scripting language over a lower-level language: they don't provide a DSL solution - which means you're still going to write what is essentially Objective-C code but with a slightly different syntax, no runtime debugging support, and no code editing support in Xcode. In other words: runtime Lua binding is a questionable solution at best, and has lots of cons going against it. Runtime Lua bindings are particularly unsuited for fast-paced action games aiming at a constantly high framerate.
What you want is a static binding. Static bindings at a minimum require you to declare what kind of methods will be available in Lua code. Some binding libraries scan your header files, others require you to provide a special declaration file similar to a header file. Most binding libraries can use both approaches. The benefit is optimal runtime performance, and being able to actually design what classes, methods and variables Lua scripts have access to.
There are but 3 candidates to bind Lua code to an iPhone app. To be fair, there are a lot more but most have one or more crucial flaws or are simply not stable or for special purposes only, or simply don't work for iPhone apps. The candidates are:
tolua and tolua++
luabind
SWIG
Big disadvantage shared by all Lua static binding libraries: none of them can bind directly to Objective-C code. All require to have an additional C or C++ layer available that ultimately interfaces with your Objective-C code. This has to do with how Objective-C works as a language and how small a role it has played (so far) when it comes to embedding Lua in Objective-C apps.
I recently evaluated all three binding libraries and came to enjoy SWIG. It is very well documented but has a bit of a learning curve. But I believe that learning curve is warranted because SWIG can be used to combine nearly any programming and scripting language, it can be advantageous to know how to use SWIG for future projects. Plus, once you understand their definition file implementation it turns out to be very easy (especially when compared to luabind) and considerably more flexible than tolua.
OK, bit late to the party but in case others come late also to this post here's another approach to add to the choices available: hand-code your LUA APIs.
I did a lecture on this topic where I live coded some basic LUA bindings in an hour. Its not hard. From the lecture I made a set of video tutorials that shows how to get started.
The approach of using a bindings generation tool like SWIG is a good one if you already have exactly the APIs that you need to call written in Objective-C and it makes sense to bring all those same API's over into LUA.
The pros of the hand-coding approach:
your project just compiles with one standard Xcode target
your project is all C & Obj-C
the LUA is just data shipped along with your images
no fiddling with "do I check in generated code" to Git
you create LUA functions for just the things you want
you can easily have hosted scripts that live inside an object
the API is under your control and is well known
dont expose engine APIs to level building team/tools
The last point is just that if you have detail functions that only make sense at the engine level and you don't want to see those when coding the game play you'll need to tell SWIG not to bind those.
Steffens answer is perfect and this approach is just another option, that may suit some folks better depending on the project.
In my existing software I have an implementation of genetic programing using home grown decission making tree that is able to apply basic logic operators (AND OR NOT) in some boolean data that are provided to it in a form of an array. The platform I am using is .NET / C# with SQLServer back end. Looking for ways to improve the performance of my genetic program I concluded that I need almost all the additional functionality that comes with a functional language and I believe Scheme or to a lesser extend LISP are the best solutions for it unless I want to implement features like COND, IF, comparisson operators etc myself extending the existing implementation.
My question to the forum is if there is any efficient way to call Scheme (or LISP) from a .NET application passing data back and front in some array form. If this is not possible, do you thing that it will better just to bite the bullet and implement it from scratch or I should look for alternative ways, like for example communicating using a text file?
There is an R6RS compliant Scheme implementation for the DLR called IronScheme. Since IronScheme uses the DLR, it can be embedded into any .NET application using the standardized DLR embedding APIs in exactly the same way that you would embed, say, IronRuby or IronPython:
dynamic Scheme = new SchemeEnvironment();
var list = Scheme.list;
var map = Scheme.map;
// and so on
The full snippet can be found in a blog post by IronScheme's author, leppie. It also shows how to pass a C# lambda to a Scheme higher-order function and other cool stuff.
Unless you go with IronScheme (above), I'd probably use something like ZeroMQ (which has both Common Lisp and .Net drivers) to pass messages between the two systems.
I built a lightweight, embeddable Scheme-like language interpreter exactly for the purpose of complex and re-usable configuration. It has a small footprint (~1500 line of code) and does not introduce any other dependency to your application.
I open-sourced it from work. It's called schemy. Here is also an example application demonstrating how to use it in really complex way.
I also provided some detailed motivation behind building it for work in this stackoverflow answer.
Hope it helps:)
Why not look at F#?
(www.fsharp.net)
It's basically an adaptation of OCaml in .Net.
Or you can always use IronScheme, but I don't think it's as mature.
I've seen it in the context of classes. I suspect it means that the class could use being broken down into logical subunits, but I can't find a good definition. Could you give some examples?
Thanks for the help.
Edit: I love the smart replies, but I'm obviously referring to "monolithic" within a software context. I know about monoliths, megaliths, dolmens, and all the stone-related contexts. Gee, I have enough of them in my country...
Interesting question. I don't think there are any formal definitions of what a monolithic class is, but you've got the idea. A class that contains multiple components that are logically unconnected, or pointlessly coupled, is a monolithic class.
If you've read The Pragmatic Programmer, which I strongly recommend, you can define a monolithic class as an anti-pattern that goes against almost everything from that book.
As for examples, you'll find more in the realm of chip and OS design, where there are formal definitions of monolithic chips/kernels, which are similar to a monolithic class. Here are some examples, although each of them can be argued against being on this list:
JOGL - Java bindings for OpenGL. This could be arguable, and with good reason.
Most academic projects - For obvious reasons.
If you started programming alone, rather than joining a team, then chances are you can open one of your first projects, and there will be a class that is monolithic.
If you look up the etymology of the word you'll see it comes from the Greek monos (single) and lithos (stone). In the context of software as you mention it, it describes a single-tiered application in which the code for the user interface and the data access are combined into a single program from a single platform.
"Monolithic" is a term that has been used to flame succesful software. This link exposes the assumptions inherent in the term, and their limited usefulness.
The basic assumption is that a system works better if it is built from software components that each have an individual, well-defined task. Intuitively, this seems right. If each component works, the entire system must work, right?
In reality, it's not that easy. A larger, compositional (non-monolithic) system can miss a critical function, even when there is no single component to blame. This happens when the architectural design fails to allocate a function to any specific component. This can happen especially if it's a function which doesn't map cleanly to a single component.
Now Linux (to continue with the linked example) in reality is not monolithic. It has a modular userspace on top of a monolithic kernel, a userspace that comes with many separate utilities. Except when it doesn't.
My definition of a Monolithic design in software development, is a design which requires additional functionality to be added to a single indivisible block of code.
PRO:
Everything is in one place, and therefore easy to find
Can be simpler, given there less relations to consider (can also be more complex see cons)
CONS:
Over time as functionality is added the complexity of the system may exponentially increase, to the point new features are extremely hard or impossible to implement
Can make it difficult for multiple developers to work with e.g Entity Framework EDMX files have the entire database in a single file which can be extremely difficult for multiple developers to work on.
Reduced re-usability, by definition it does not have smaller components which can be then reused and re-purposed to solve other problems, unless a complete copy of the code is made and then modified.
A monolithic architecture is a model of software structure which is created as one piece where all Rails tools (ActionMailer, ActiveJob, ActionCable, etc.) can be gathered together with the code that these tools applies. The tools are not connected with each other but they are also not autonomous.
If one feature needs changes, it will influence the work of the whole process and other features because they are parts of one process.
Let’s recall what Ruby on Rails is, what it can offer, its pros and cons. Its most important benefit is that it is easy to work with.
If you write rails new you immediately get a new application at once, then you can create any REST API you want and use Rails helpers and generators, which makes development even easier.
If you need to send emails in your Rails app, then use Rails ActionMailer. When you need to do some hard processing, ActiveJob will help you. With Rails 5 you will also be able to use websockets out of the box. Thus, it will be easy to create chats or make your application more interactive.
In case you use correct DSL syntax, you can use all that and even more immediately. Moreover, you don’t have to know everything about the internal implementation of these tools, consider it’s DSL, and receive the expected result.
It means something is the opposite of modular. A modular application can have parts, referred to as modules, replaced without requiring replacement of the entire application. Whereas a monolithic application, after having a part fixed or upgraded, must be replaced in it's entirety.
From Wikipedia: "Modularity is desirable, in general, as it supports reuse of parts of the application logic and also facilitates maintenance by allowing repair or replacement of parts of the application without requiring wholesale replacement."
So in the context of a monolithic class, all its features are self-contained and if you want to add or alter a feature to the class you would need to alter/add code in the class and recompile it. Conversely a modular class exposes access to functionality which is implemented externally. For example a "Calculator" class may use a separate "Add" class for actually adding numbers; call a "Multiply" function from a separate library; or even call an "Amortize" function from a web service. As long as the each of these functional parts can be altered externally from the class, it is modular.
A repeating theme in my development work has been the use of or creation of an in-house plug-in architecture. I've seen it approached many ways - configuration files (XML, .conf, and so on), inheritance frameworks, database information, libraries, and others. In my experience:
A database isn't a great place to store your configuration information, especially co-mingled with data
Attempting this with an inheritance hierarchy requires knowledge about the plug-ins to be coded in, meaning the plug-in architecture isn't all that dynamic
Configuration files work well for providing simple information, but can't handle more complex behaviors
Libraries seem to work well, but the one-way dependencies have to be carefully created.
As I seek to learn from the various architectures I've worked with, I'm also looking to the community for suggestions. How have you implemented a SOLID plug-in architecture? What was your worst failure (or the worst failure you've seen)? What would you do if you were going to implement a new plug-in architecture? What SDK or open source project that you've worked with has the best example of a good architecture?
A few examples I've been finding on my own:
Perl's Module::Plugable and IOC for dependency injection in Perl
The various Spring frameworks (Java, .NET, Python) for dependency injection.
An SO question with a list for Java (including Service Provider Interfaces)
An SO question for C++ pointing to a Dr. Dobbs article
An SO question regarding a specific plugin idea for ASP.NET MVC
These examples seem to play to various language strengths. Is a good plugin architecture necessarily tied to the language? Is it best to use tools to create a plugin architecture, or to do it on one's own following models?
This is not an answer as much as a bunch of potentially useful remarks/examples.
One effective way to make your application extensible is to expose its internals as a scripting language and write all the top level stuff in that language. This makes it quite modifiable and practically future proof (if your primitives are well chosen and implemented). A success story of this kind of thing is Emacs. I prefer this to the eclipse style plugin system because if I want to extend functionality, I don't have to learn the API and write/compile a separate plugin. I can write a 3 line snippet in the current buffer itself, evaluate it and use it. Very smooth learning curve and very pleasing results.
One application which I've extended a little is Trac. It has a component architecture which in this situation means that tasks are delegated to modules that advertise extension points. You can then implement other components which would fit into these points and change the flow. It's a little like Kalkie's suggestion above.
Another one that's good is py.test. It follows the "best API is no API" philosophy and relies purely on hooks being called at every level. You can override these hooks in files/functions named according to a convention and alter the behaviour. You can see the list of plugins on the site to see how quickly/easily they can be implemented.
A few general points.
Try to keep your non-extensible/non-user-modifiable core as small as possible. Delegate everything you can to a higher layer so that the extensibility increases. Less stuff to correct in the core then in case of bad choices.
Related to the above point is that you shouldn't make too many decisions about the direction of your project at the outset. Implement the smallest needed subset and then start writing plugins.
If you are embedding a scripting language, make sure it's a full one in which you can write general programs and not a toy language just for your application.
Reduce boilerplate as much as you can. Don't bother with subclassing, complex APIs, plugin registration and stuff like that. Try to keep it simple so that it's easy and not just possible to extend. This will let your plugin API be used more and will encourage end users to write plugins. Not just plugin developers. py.test does this well. Eclipse as far as I know, does not.
In my experience I've found there are really two types of plug-in Architectures.
One follows the Eclipse model which is meant to allow for freedom and is open-ended.
The other usually requires plugins to follow a narrow API because the plugin will fill a specific function.
To state this in a different way, one allows plugins to access your application while the other allows your application to access plugins.
The distinction is subtle, and sometimes there is no distiction... you want both for your application.
I do not have a ton of experience with Eclipse/Opening up your App to plugins model (the article in Kalkie's post is great). I've read a bit on the way eclipse does things, but nothing more than that.
Yegge's properties blog talks a bit about how the use of the properties pattern allows for plugins and extensibility.
Most of the work I've done has used a plugin architecture to allow my app to access plugins, things like time/display/map data, etc.
Years ago I would create factories, plugin managers and config files to manage all of it and let me determine which plugin to use at runtime.
Now I usually just have a DI framework do most of that work.
I still have to write adapters to use third party libraries, but they usually aren't that bad.
One of the best plug-in architectures that I have seen is implemented in Eclipse. Instead of having an application with a plug-in model, everything is a plug-in. The base application itself is the plug-in framework.
http://www.eclipse.org/articles/Article-Plug-in-architecture/plugin_architecture.html
I'll describe a fairly simple technique that I have use in the past. This approach uses C# reflection to help in the plugin loading process. This technique can be modified so it is applicable to C++ but you lose the convenience of being able to use reflection.
An IPlugin interface is used to identify classes that implement plugins. Methods are added to the interface to allow the application to communicate with the plugin. For example the Init method that the application will use to instruct the plugin to initialize.
To find plugins the application scans a plugin folder for .Net assemblies. Each assembly is loaded. Reflection is used to scan for classes that implement IPlugin. An instance of each plugin class is created.
(Alternatively, an Xml file might list the assemblies and classes to load. This might help performance but I never found an issue with performance).
The Init method is called for each plugin object. It is passed a reference to an object that implements the application interface: IApplication (or something else named specific to your app, eg ITextEditorApplication).
IApplication contains methods that allows the plugin to communicate with the application. For instance if you are writing a text editor this interface would have an OpenDocuments property that allows plugins to enumerate the collection of currently open documents.
This plugin system can be extended to scripting languages, eg Lua, by creating a derived plugin class, eg LuaPlugin that forwards IPlugin functions and the application interface to a Lua script.
This technique allows you to iteratively implement your IPlugin, IApplication and other application-specific interfaces during development. When the application is complete and nicely refactored you can document your exposed interfaces and you should have a nice system for which users can write their own plugins.
I once worked on a project that had to be so flexible in the way each customer could setup the system, which the only good design we found was to ship the customer a C# compiler!
If the spec is filled with words like:
Flexible
Plug-In
Customisable
Ask lots of questions about how you will support the system (and how support will be charged for, as each customer will think their case is the normal case and should not need any plug-ins.), as in my experience
The support of customers (or
fount-line support people) writing
Plug-Ins is a lot harder than the
Architecture
Usualy I use MEF. The Managed Extensibility Framework (or MEF for short) simplifies the creation of extensible applications. MEF offers discovery and composition capabilities that you can leverage to load application extensions.
If you are interested read more...
In my experience, the two best ways to create a flexible plugin architecture are scripting languages and libraries. These two concepts are in my mind orthogonal; the two can be mixed in any proportion, rather like functional and object-oriented programming, but find their greatest strengths when balanced. A library is typically responsible for fulfilling a specific interface with dynamic functionality, whereas scripts tend to emphasise functionality with a dynamic interface.
I have found that an architecture based on scripts managing libraries seems to work the best. The scripting language allows high-level manipulation of lower-level libraries, and the libraries are thus freed from any specific interface, leaving all of the application-level interaction in the more flexible hands of the scripting system.
For this to work, the scripting system must have a fairly robust API, with hooks to the application data, logic, and GUI, as well as the base functionality of importing and executing code from libraries. Further, scripts are usually required to be safe in the sense that the application can gracefully recover from a poorly-written script. Using a scripting system as a layer of indirection means that the application can more easily detach itself in case of Something Bad™.
The means of packaging plugins depends largely on personal preference, but you can never go wrong with a compressed archive with a simple interface, say PluginName.ext in the root directory.
I think you need to first answer the question: "What components are expected to be plugins?"
You want to keep this number to an absolute minimum or the number of combinations which you must test explodes. Try to separate your core product (which should not have too much flexibility) from plugin functionality.
I've found that the IOC (Inversion of Control) principal (read springframework) works well for providing a flexible base, which you can add specialization to to make plugin development simpler.
You can scan the container for the "interface as a plugin type advertisement" mechanism.
You can use the container to inject common dependencies which plugins may require (i.e. ResourceLoaderAware or MessageSourceAware).
The Plug-in Pattern is a software pattern for extending the behaviour of a class with a clean interface. Often behaviour of classes is extended by class inheritance, where the derived class overwrites some of the virtual methods of the class. A problem with this solution is that it conflicts with implementation hiding. It also leads to situations where derived class become a gathering places of unrelated behaviour extensions. Also, scripting is used to implement this pattern as mentioned above "Make internals as a scripting language and write all the top level stuff in that language. This makes it quite modifiable and practically future proof". Libraries use script managing libraries. The scripting language allows high-level manipulation of lower level libraries. (Also as mentioned above)