Considering that I am developing an end-user software program (as an uberjar) I am wondering what my options are to make it possible for the user to download a plugin and load that during runtime.
The plugin(s) should come compiled and without source code, so sth. like load is not an option.
What existing libraries (or ways of Java...?) exist to build this on?
EDIT: If you are not sure I would also be satisfied with a way that costs a reboot/-start of the main-program. However, what is important is that the source-code won't be included in any JAR file (neither main application nor plugin-jars, see :omit-source of Leiningen documentation).
To add a jar during runtime, use pomegranate which lets you add a .jar file to the classpath. Plugins of your software should be regular Clojure libs that follow certain conventions that you need to establish:
Make them provide (e. g. in an edn) a symbol to an object implementing a constructor/destructor mechanism such as the Lifecycle protocol in Stuart Sierras component library. In runtime, require and resolve that symbol, start the resulting object and hand it over to rest your programs plugin coordination facilities.
Provide a public API in your program that allows the plugins to interact with it in ways that you coordinate asynchronously e. g. with clojure.core.async (don't let one plugin block the entire program).
Make sure that the plugins have a coordinated way to expose their functionality to each other only if they desire so to enable a high degree of modularity among your plugins. Make sure that your plugin loader is capable of detecting dependencies among plugins and is capable of loading and unloading them in the right order.
I've not tried it myself, but you should in theory be able to get OSGi to work with Clojure.
There's a Clojure / OSGi integration library here:
https://github.com/aav/clojure.osgi
If I were to attempt to role my own solution, I would try using tools.namespace to load and unload plugins. I'm not entirely sure it will work, but it's certainly in the right direction. I think one key piece is that the plugin jars will have to be "installed" in a location that's already on the classpath.
Again, this is only the start of one possible solution. I haven't tried doing this.
Related
we have some dependent/common jars of the plugin which we are currently placing in .lib of ContentNavigator application and referring the same in build.xml of ContentNavigator. while we placed the jars in local and tried to give the local path in build.xml it didn't worked out.Can anyone suggest what is the recommended way of doing it?
I've applied the following approaches:
Embed/extract all the content(.class files) of the dependency-jars within the navigator plugin. While the upside is that you have a nice bundle containing all your dependency-classes, it might cause some problems with signed jar's. Or on files that are duplicate amongst your dependencies.
Make use of the "Shared Library" concept. You basically "attach" jar's to the classloader of your server/JVM, Application(ear) or module(war). In your case you would probably attach the shared-library to the navigator.ear, allowing all plugins to access the dependencies you register as a sharedlibrary. One huge downside is (in my experience) that you need to reboot websphere every time you update one of the library-jars.
A third option is to move the functionality out of your ICN Plugin, and let another EAR/EJB do the work (and reference to it from the plugin using a jndi lookup).
I personally prefer option 3 because it's the cleanest way to bundle your functionality&dependencies in an EAR. But option 2 will be a good call when libraries are truely common and won't be updated often.
p.s. In your question you mention the build aspect of your software as well (e.g. "how to correctly reference the dependencies"). You might want to look into the concepts of "dependency management" (using Ant, IVY is a good choice, but i would suggest moving to Maven, or Gradle)
As of Play 2.0, it appears that there is no longer a way of creating a module for an existing Play application, other than by creating a new Play application. Having searched around a bit, I came across these instructions, which indicate that I must (or at least should) delete any routes created in the new module (application), and that the module's application.conf file is really just a stub that is required in order for the module to be recognized as a Play application.
I am new to Play, but apparently there used to be a console command ('new-module') for generating a module, which presumably created only those files which were needed for the module to be discovered by the application. It seems to me like it would still be useful to be able to quickly create a new module in this way, especially if registering a new module from the console also added the module to your build, and to the repository of your choice, thus removing the requirement for (as much) manual wiring.
I would also like to be able to maintain Play modules upon which my application depends as part of the same codebase/build, such that, when I make changes to a module, they are picked up at application compile time (for example, when play is ~ running and and a changed file is saved). Does this already happen with modules registered as dependencies, or must I rebuild modules independently of my application?
Because I am a newcomer, I'm not positive that there isn't a way to do accomplish these tasks in an automated manner. There is a chapter on packages and modules listed in the Play for Scala book (Part III, chapter 9), but the book is not yet complete and that chapter is, unfortunately, yet unwritten.
If an experienced Play! developer would be so kind as to either confirm that the instructions to which I linked above are still the recommended procedure for creating a module and registering/maintaining it as a dependency, or else list a better procedure, I would greatly appreciate it.
Most of the information is valid.
To Play 2.x there is no difference between a regular library and a Play module (a library which itself depends on the play library jar).
The part about the routes file is still valid, but they introduced 'Sub-Router composition' to give you some extra freedom (search for 'Allow more modularization for your projects' on the highlight page).
Libraries (and thus Play modules) are referenced in the Build.scala file with version, for example:
"play.modules.mailer" %% "play-mailer" % "1.1.0"
If you are developing a module yourself you could use the 'publish-local' command to make sure other projects on your computer can find the dependency. Because modules are essentially versioned libraries you need to compile them separately from your application. However no-one is preventing you from running scripts to automate things.
Can go run dynamically in order to be used for a plugin based application ?
In eclipse, we can create some plugins that Eclipse can run dynamically.
Would the same thing be possible in Go ?
I'll argue that those are two separate problems :
having dynamic load
having plugins
The first one is simply no : A Go program is statically linked, which means you can't add code to a running program. And which also means you must compile the program to let it integrate plugins.
Fortunately, you can define a program accepting plugins in Go as in most languages, and Go, with interfaces and fast compilation doesn't make that task hard.
Here are two possible approaches :
Solution 1 : Plugin integrated in the main program
Similarly to Eclipse plugins, we can integrate the "plugins" in the main program memory, by simply recompiling the program. In this sense we can for example say that database drivers are plugins.
This may not feel as simple as in Java, as you must have a recompilation and you must in some point of your code import the "plugin" (see how it's done for database drivers) but, given the standardization of Go regarding directories and imports, it seems easy to handle that with a simple makefile importing the plugin and recompiling the application.
Given the ease and speed of compilation in Go, and the standardization of package structure, this seems to me to be a very viable solution.
Solution 2 : Separate process
It's especially easy in Go to communicate and to handle asynchronous calls. Which means you could define a solution based on many process communicating by named pipes (or any networking solution). Note that there is a rpc package in Go. This would probably be efficient enough for most programs and the main program would be able to start and stop the plugin processes. This could very well feel similar to what you have in Eclipse with the added benefits of memory space protection.
A last note from somebody who wrote several Eclipse plugins : you don't want that mess; keep it simple.
Go 1.8 supports plugins (to be released soon Feb 2017.)
https://tip.golang.org/pkg/plugin/
As dystroy already said, it's not possible to load packages at runtime.
In the future (or today with limitations) it may be possible to have this feature with projects like go-eval, which is "the beginning of an interpreter for Go".
A few packages I found to do this:
https://golang.org/pkg/net/rpc/
https://github.com/hashicorp/go-plugin
https://github.com/natefinch/pie
I think (see below) that I would like to structure a Clojure project as multiple modules, with ordered dependencies - just like Maven lets me do with multi-modules projects.
But I can't see how to do this with Leiningen - all I can see is the checkouts fix described in the FAQ which doesn't seem to be as powerful.
Will lein do this? Should I be using Gradle instead? Or is this kind of thing not needed?
Some more context: I am wondering how to architect a modular application that supports plugins (which I imagine means jars dumped on the classpath). And am wondering to what extent I can structure that as a core + plugins (I am thinking I should be able to do something with Clojure's dynamic code loading and not have to go with Java/OSGi). So I guess the driving motivation for a single project comes from wanting some way of packaging everything (the core + default plugins) as a single blob that is easy for the end user, but which can also be divided up (and which is built and tested in fragments, testing the logical independence of each module). More general advice about this is welcome
Update
A possible solution that wasn't mentioned below is using a Maven plugin - I assume that supports everything Maven does, but compiles Clojure, so will work with nested modules, etc.
First, it does not seem like Leiningen supports a module hierarchy like Maven does. The checkouts are the next closest thing it has. It should be sufficient though to develop a modular application in Clojure though.
For the project structure, I would have an API project, a "core" project, the plugins themselves, and a separate packaging project. The core and the plugins should only depend on the API. Which build tool you use to create the packaging project is up to you. Gradle would probably be more effective at handling the packaging, however having the "checkout" functionality Leiningen offers could make development of the system as a whole easier.
I would take a look at the code for Leiningen and Noir to figure out how to effectively handle this.
For dynamically loading the plugins, I would start with looking how Noir handles it in two of their files:
server.clj has namespace loading for all files under a particular namespace. Under the hood it uses tools.namespace, but you can easily see how it's used to require every namespace under a particular base. This is how Leiningen handles custom tasks as well - the base definition for the task should be in the leiningen.$task namespace.
core.clj has what I would use for plugin registration. In summary, use a map under an atom and add plugins to that map. I would advice wrapping the registration with a macro to keep your code cleaner.
What I listed above should be sufficient if you don't need to handle adding plugins at run time. If you don't have every plugin on the classpath during start-up, I would recommend utilizing pomegranite to add entries to the classpath. You can see an example in classpath.clj.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 years ago.
Improve this question
Wether you call it Addons, Plugins or extra peices of code that is connected with the original software later, it really doesn't matter. I would love to understand how they work, there has to be a simple explanation of how to design a Plugin System. Unfortunately, I never understood it, and there remains a lot of open question in my mind. For example, how does the program find a plugin? how does it interface with it? when is it preferable for a software to have Plugin System?
Thanks for all helpful answers. It seems I asked too open question, fortunately I got keywords to look for. I liked David answer though I am not a Java guy, but his talk made sense to me :)
Plug-ins work by conforming to well-known interfaces that the main application expects to work with.
There are several ways in which a plug-in architecture actually works, but in general, these are the steps:
Plug-ins are designed to match an interface that the application expects. For example, a simple application might require that plug-ins implement a IPlugin interface.
Plug-ins are loaded by the application, usually when the app is starting up
Plug-ins are often provided access to much of the data that the application manages. For example, Firefox plug-ins can access the current web page, and Eclipse plug-ins can access the open files.
Here are two ways (out of several) in which an application can find plug-ins:
The plug-ins are known to exist in a particular folder, and the application knows to load plug-ins from that folder
Each plug-in runs as a service, and the services are designed to work together (this is how an OSGi-based application works)
When plug-ins are found, they are loaded by the application (sometimes the job of a Class Loader).
A software architect might design a plug-in architecture when they expect that either the software provider or the user community will implement new features that were not originally part of the system. Two great examples are Eclipse and Firefox; other applications include Adobe Photoshop (for artistic techniques and graphical tools) and Winamp (for visualizations).
Create an interface that all plugins of a particular type will implement
Write the code that will 'consume' the plugin against the interface only.
Have a dynamic way to load a DLL containing the plugin type that implements your interface (for instance, have a configurable folder location to test whether any DLLs in that folder contain any types that implement your interface, and dynamically load any that do. In .NET this might use Assembly.LoadFile())
If you want to have a look at some source code, Paint.NET is free and open source, and has a plugin architecture.
A program typically has to be designed to look for a plug-in, and the plug-in has to have a standard access point to accept control from the main program. Every application or website does it a little differently.
The simplest type of plug-in is accessed something like this:
if (a plug-in exists/is configured)
call predefined plug-in code
In this case, the main program is coded to only handle a specific set of plug-ins (many php-based wordpress templates are like this). A slightly more advanced plug-in
perform application specific logic
if any plug-in exists that exposes the run_after_app_specific_logic function
call plug-in code
This second case can handle ridiculously complex plug-ins ... the plug-in would just need to implement more functions called by the master program.
Eclipse in an example of a application-framework which is entirely plugin-based, meaning that all functionality is implemented as plugins. There is a thin layer at the bottom for startup/shutdown and plugin-management, but everything else is implemented as plugins on top of that. This results in a framework which can be used for just about everything. More info about Eclipse plugin architecture can be found here: http://www.eclipse.org/articles/Article-Plug-in-architecture/plugin_architecture.html.
It's very language dependent.
In an interpreted language it simply involves calling a file that follows a pattern.
In C it's pretty hard to do without help. In C+windows a "DLL" can be a plug-in and are often used that way.
In an OO language with reflection, you might create an object that implements an interface and load it reflectively. After it's loaded, you can ignore the fact that it was a plug-in because it's treated as any other object in your code.
.net has a plugin architecture (is it COM?) Well anyway COM can be used as (is?) a plugin system.
Your question is probably too open-ended because of all the possibilities. There is no single answer.
I've never written a plugin system. But this is how I imagine it in my head:
Your program has a subdirectory for plugins (e.g. "C:\Program Files\My Program Name\plugins").
You create plugins as DLL files and place them in the plugins folder.
These DLLs would export functions with predefined names.
When you run your program, it looks through all the DLLs in your plugins folder. In each one it would look for an exported function with a certain name (e.g. "Load") and call that function. The plugin could then do any setup that it needed to do.
The program would then call an exported function on the plugin with a name like "GetPluginName". The plugin would return it's name and the program could then use that name when it displays a list of plugins to the user.
When it comes time to invoke the plugin, the program would call another exported function (maybe "Activate") and probably pass the plugin a pointer to the data that the plugin is going to work on. The program would then do its work on the data.
The plugin might also export another function that the program would call to show a setup dialog where you could change the plugin options.
A plugin system can be implemented in many ways, but the common way for a lot of C/C++ applications is a DLL-based plugin SDK.
The DLL will expose various automated function calls which may allow the plugin to "set itself up" in the running application such as adding menu items, new functionality or extra options for systems (like 3D rendering implementations).
More ofthen there's no need for any special discovery - the plugin mechanizm is generally dumb: Here's a code signature I understand, and here's a call(s) I can make. I have no clue how the thing I'm calling will do the job, but I expect result to be in certain format. And that is pretty much a contract. Now - the plugin will implement the contract and make itself available. In Java, for example "make available" simply means that implementing classes are loaded into memory. JDBC driver for a particular database would be a good example.