Can go run dynamically in order to be used for a plugin based application ?
In eclipse, we can create some plugins that Eclipse can run dynamically.
Would the same thing be possible in Go ?
I'll argue that those are two separate problems :
having dynamic load
having plugins
The first one is simply no : A Go program is statically linked, which means you can't add code to a running program. And which also means you must compile the program to let it integrate plugins.
Fortunately, you can define a program accepting plugins in Go as in most languages, and Go, with interfaces and fast compilation doesn't make that task hard.
Here are two possible approaches :
Solution 1 : Plugin integrated in the main program
Similarly to Eclipse plugins, we can integrate the "plugins" in the main program memory, by simply recompiling the program. In this sense we can for example say that database drivers are plugins.
This may not feel as simple as in Java, as you must have a recompilation and you must in some point of your code import the "plugin" (see how it's done for database drivers) but, given the standardization of Go regarding directories and imports, it seems easy to handle that with a simple makefile importing the plugin and recompiling the application.
Given the ease and speed of compilation in Go, and the standardization of package structure, this seems to me to be a very viable solution.
Solution 2 : Separate process
It's especially easy in Go to communicate and to handle asynchronous calls. Which means you could define a solution based on many process communicating by named pipes (or any networking solution). Note that there is a rpc package in Go. This would probably be efficient enough for most programs and the main program would be able to start and stop the plugin processes. This could very well feel similar to what you have in Eclipse with the added benefits of memory space protection.
A last note from somebody who wrote several Eclipse plugins : you don't want that mess; keep it simple.
Go 1.8 supports plugins (to be released soon Feb 2017.)
https://tip.golang.org/pkg/plugin/
As dystroy already said, it's not possible to load packages at runtime.
In the future (or today with limitations) it may be possible to have this feature with projects like go-eval, which is "the beginning of an interpreter for Go".
A few packages I found to do this:
https://golang.org/pkg/net/rpc/
https://github.com/hashicorp/go-plugin
https://github.com/natefinch/pie
Related
I would like to write JUnit test for my Eclipse RCP while I continue developing the code. When starting the application the different plugins initialize variables of various plugins/classes (mostly within their start methods) which are needed for the correct functionality.
If this initialization doesn't happen, it is impossible to test code because it depends on those values.
How do I solve this issue without creating a lot of dummy values?
What is the general approach to testing Eclipse RCPs?
You're facing a common problem: Too many dependencies. You need to cut them.
With Eclipse 3, this is going to be somewhat hard. Try to split the code into things that depend on the Eclipse platform running and everything else. Eclipse often uses interfaces, so you can test many things using mocks.
With e4, things got more simple since many services will be injected, making mocking and testing even easier.
But the goal must always be to have as much code as possible that doesn't depend on SWT or the platform. Create your own interfaces if you have to. The runtime imlementations just wrap Eclipse code. For tests, you can use mocks to simulate the Eclipse runtime.
You can run tests using JUnit plugintest, that will start up the plugin framework and will allow for testing of plugins. But this usually only solves some of the issues. The best suggestion is as Aaron suggests to separate functionality as much as possible to the point where all your actual code are just plain old java objects that you can test normally. All dependencies to Eclipse are in different classes and are kept as thin as possible so that they dont require testing.
This can be difficult to achieve, so mocking may be required. Another trick I've resorted to at times is to use Java reflection to change values of private fields, see this question
Considering that I am developing an end-user software program (as an uberjar) I am wondering what my options are to make it possible for the user to download a plugin and load that during runtime.
The plugin(s) should come compiled and without source code, so sth. like load is not an option.
What existing libraries (or ways of Java...?) exist to build this on?
EDIT: If you are not sure I would also be satisfied with a way that costs a reboot/-start of the main-program. However, what is important is that the source-code won't be included in any JAR file (neither main application nor plugin-jars, see :omit-source of Leiningen documentation).
To add a jar during runtime, use pomegranate which lets you add a .jar file to the classpath. Plugins of your software should be regular Clojure libs that follow certain conventions that you need to establish:
Make them provide (e. g. in an edn) a symbol to an object implementing a constructor/destructor mechanism such as the Lifecycle protocol in Stuart Sierras component library. In runtime, require and resolve that symbol, start the resulting object and hand it over to rest your programs plugin coordination facilities.
Provide a public API in your program that allows the plugins to interact with it in ways that you coordinate asynchronously e. g. with clojure.core.async (don't let one plugin block the entire program).
Make sure that the plugins have a coordinated way to expose their functionality to each other only if they desire so to enable a high degree of modularity among your plugins. Make sure that your plugin loader is capable of detecting dependencies among plugins and is capable of loading and unloading them in the right order.
I've not tried it myself, but you should in theory be able to get OSGi to work with Clojure.
There's a Clojure / OSGi integration library here:
https://github.com/aav/clojure.osgi
If I were to attempt to role my own solution, I would try using tools.namespace to load and unload plugins. I'm not entirely sure it will work, but it's certainly in the right direction. I think one key piece is that the plugin jars will have to be "installed" in a location that's already on the classpath.
Again, this is only the start of one possible solution. I haven't tried doing this.
I think (see below) that I would like to structure a Clojure project as multiple modules, with ordered dependencies - just like Maven lets me do with multi-modules projects.
But I can't see how to do this with Leiningen - all I can see is the checkouts fix described in the FAQ which doesn't seem to be as powerful.
Will lein do this? Should I be using Gradle instead? Or is this kind of thing not needed?
Some more context: I am wondering how to architect a modular application that supports plugins (which I imagine means jars dumped on the classpath). And am wondering to what extent I can structure that as a core + plugins (I am thinking I should be able to do something with Clojure's dynamic code loading and not have to go with Java/OSGi). So I guess the driving motivation for a single project comes from wanting some way of packaging everything (the core + default plugins) as a single blob that is easy for the end user, but which can also be divided up (and which is built and tested in fragments, testing the logical independence of each module). More general advice about this is welcome
Update
A possible solution that wasn't mentioned below is using a Maven plugin - I assume that supports everything Maven does, but compiles Clojure, so will work with nested modules, etc.
First, it does not seem like Leiningen supports a module hierarchy like Maven does. The checkouts are the next closest thing it has. It should be sufficient though to develop a modular application in Clojure though.
For the project structure, I would have an API project, a "core" project, the plugins themselves, and a separate packaging project. The core and the plugins should only depend on the API. Which build tool you use to create the packaging project is up to you. Gradle would probably be more effective at handling the packaging, however having the "checkout" functionality Leiningen offers could make development of the system as a whole easier.
I would take a look at the code for Leiningen and Noir to figure out how to effectively handle this.
For dynamically loading the plugins, I would start with looking how Noir handles it in two of their files:
server.clj has namespace loading for all files under a particular namespace. Under the hood it uses tools.namespace, but you can easily see how it's used to require every namespace under a particular base. This is how Leiningen handles custom tasks as well - the base definition for the task should be in the leiningen.$task namespace.
core.clj has what I would use for plugin registration. In summary, use a map under an atom and add plugins to that map. I would advice wrapping the registration with a macro to keep your code cleaner.
What I listed above should be sufficient if you don't need to handle adding plugins at run time. If you don't have every plugin on the classpath during start-up, I would recommend utilizing pomegranite to add entries to the classpath. You can see an example in classpath.clj.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 years ago.
Improve this question
Wether you call it Addons, Plugins or extra peices of code that is connected with the original software later, it really doesn't matter. I would love to understand how they work, there has to be a simple explanation of how to design a Plugin System. Unfortunately, I never understood it, and there remains a lot of open question in my mind. For example, how does the program find a plugin? how does it interface with it? when is it preferable for a software to have Plugin System?
Thanks for all helpful answers. It seems I asked too open question, fortunately I got keywords to look for. I liked David answer though I am not a Java guy, but his talk made sense to me :)
Plug-ins work by conforming to well-known interfaces that the main application expects to work with.
There are several ways in which a plug-in architecture actually works, but in general, these are the steps:
Plug-ins are designed to match an interface that the application expects. For example, a simple application might require that plug-ins implement a IPlugin interface.
Plug-ins are loaded by the application, usually when the app is starting up
Plug-ins are often provided access to much of the data that the application manages. For example, Firefox plug-ins can access the current web page, and Eclipse plug-ins can access the open files.
Here are two ways (out of several) in which an application can find plug-ins:
The plug-ins are known to exist in a particular folder, and the application knows to load plug-ins from that folder
Each plug-in runs as a service, and the services are designed to work together (this is how an OSGi-based application works)
When plug-ins are found, they are loaded by the application (sometimes the job of a Class Loader).
A software architect might design a plug-in architecture when they expect that either the software provider or the user community will implement new features that were not originally part of the system. Two great examples are Eclipse and Firefox; other applications include Adobe Photoshop (for artistic techniques and graphical tools) and Winamp (for visualizations).
Create an interface that all plugins of a particular type will implement
Write the code that will 'consume' the plugin against the interface only.
Have a dynamic way to load a DLL containing the plugin type that implements your interface (for instance, have a configurable folder location to test whether any DLLs in that folder contain any types that implement your interface, and dynamically load any that do. In .NET this might use Assembly.LoadFile())
If you want to have a look at some source code, Paint.NET is free and open source, and has a plugin architecture.
A program typically has to be designed to look for a plug-in, and the plug-in has to have a standard access point to accept control from the main program. Every application or website does it a little differently.
The simplest type of plug-in is accessed something like this:
if (a plug-in exists/is configured)
call predefined plug-in code
In this case, the main program is coded to only handle a specific set of plug-ins (many php-based wordpress templates are like this). A slightly more advanced plug-in
perform application specific logic
if any plug-in exists that exposes the run_after_app_specific_logic function
call plug-in code
This second case can handle ridiculously complex plug-ins ... the plug-in would just need to implement more functions called by the master program.
Eclipse in an example of a application-framework which is entirely plugin-based, meaning that all functionality is implemented as plugins. There is a thin layer at the bottom for startup/shutdown and plugin-management, but everything else is implemented as plugins on top of that. This results in a framework which can be used for just about everything. More info about Eclipse plugin architecture can be found here: http://www.eclipse.org/articles/Article-Plug-in-architecture/plugin_architecture.html.
It's very language dependent.
In an interpreted language it simply involves calling a file that follows a pattern.
In C it's pretty hard to do without help. In C+windows a "DLL" can be a plug-in and are often used that way.
In an OO language with reflection, you might create an object that implements an interface and load it reflectively. After it's loaded, you can ignore the fact that it was a plug-in because it's treated as any other object in your code.
.net has a plugin architecture (is it COM?) Well anyway COM can be used as (is?) a plugin system.
Your question is probably too open-ended because of all the possibilities. There is no single answer.
I've never written a plugin system. But this is how I imagine it in my head:
Your program has a subdirectory for plugins (e.g. "C:\Program Files\My Program Name\plugins").
You create plugins as DLL files and place them in the plugins folder.
These DLLs would export functions with predefined names.
When you run your program, it looks through all the DLLs in your plugins folder. In each one it would look for an exported function with a certain name (e.g. "Load") and call that function. The plugin could then do any setup that it needed to do.
The program would then call an exported function on the plugin with a name like "GetPluginName". The plugin would return it's name and the program could then use that name when it displays a list of plugins to the user.
When it comes time to invoke the plugin, the program would call another exported function (maybe "Activate") and probably pass the plugin a pointer to the data that the plugin is going to work on. The program would then do its work on the data.
The plugin might also export another function that the program would call to show a setup dialog where you could change the plugin options.
A plugin system can be implemented in many ways, but the common way for a lot of C/C++ applications is a DLL-based plugin SDK.
The DLL will expose various automated function calls which may allow the plugin to "set itself up" in the running application such as adding menu items, new functionality or extra options for systems (like 3D rendering implementations).
More ofthen there's no need for any special discovery - the plugin mechanizm is generally dumb: Here's a code signature I understand, and here's a call(s) I can make. I have no clue how the thing I'm calling will do the job, but I expect result to be in certain format. And that is pretty much a contract. Now - the plugin will implement the contract and make itself available. In Java, for example "make available" simply means that implementing classes are loaded into memory. JDBC driver for a particular database would be a good example.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
I've seen questions about IDE's here -- Which is the best IDE for Scala development? and What is the current state of tooling for Scala?, but I've had mixed experiences with IDEs. Right now, I'm using the Eclipse IDE with the automatic workspace refresh option, and KDE 4's Kate as my text editor. Here are some of the problems I'd like to solve:
use my own editor IDEs are really geared at everyone using their components. I like Kate better, but the refresh system is very annoying (it doesn't use inotify, rather, maybe a 10s polling interval). The reason I don't use the built-in text editor is because broken auto-complete functionalities cause the IDE to hang for maybe 10s.
rebuild only modified files The Eclipse build system is broken. It doesn't know when to rebuild classes. I find myself almost half of the time going to project->clean. Worse, it seems even after it has finished building my project, a few minutes later it will pop up with some bizarre error (edit - these errors appear to be things that were previously solved with a project > clean, but then come back up...). Finally, setting "Preferences / Continue launch if project contains errors" to "prompt" seems to have no effect for Scala projects (i.e. it always launches even if there are errors).
build customization I can use the "nightly" release, but I'll want to modify and use my own Scala builds, not the compiler that's built into the IDE's plugin. It would also be nice to pass [e.g.] -Xprint:jvm to the compiler (to print out lowered code).
fast compiling Though Eclipse doesn't always build right, it does seem snappy -- even more so than fsc.
I looked at Ant and Maven, though haven't employed either yet (I'll also need to spend time solving #3 and #4). I wanted to see if anyone has other suggestions before I spend time getting a suboptimal build system working. Thanks in advance!
UPDATE - I'm now using Maven, passing a project as a compiler plugin to it. It seems fast enough; I'm not sure what kind of jar caching Maven does. A current repository for Scala 2.8.0 is available [link]. The archetypes are very cool, and cross-platform support seems very good. However, about compile issues, I'm not sure if fsc is actually fixed, or my project is stable enough (e.g. class names aren't changing) -- running it manually doesn't bother me as much. If you'd like to see an example, feel free to browse the pom.xml files I'm using [github].
UPDATE 2 - from benchmarks I've seen, Daniel Spiewak is right that buildr's faster than Maven (and, if one is doing incremental changes, Maven's 10 second latency gets annoying), so if one can craft a compatible build file, then it's probably worth it...
Points 2 and 4 are extremely difficult to manage with the current scalac. The problem is that Scala's compiler is a little dumb about building files. Basically, it will build whatever you feed it, regardless of whether or not that file really needs to be built. Scala 2.8.0 will have some tremendous improvements in this respect, but until then... Eclipse SDT actually has some very elaborate (and very hackish) code for doing change detection and dependency tracking. On the whole, it does a decent job, but as you have seen, there are wrinkles. Eclipse SDT 2.8.0 will rely on the aforementioned improvements to scalac itself.
So, building only modified files is pretty much out of the question. Aside from SDT, the only tool I know of which even tries this is SBT (Simple Build Tool). It uses a compiler plugin to track files as they are compiled and query the dependency graph computed by the compiler itself. In practice, this yields about a 50% improvement over the recompile-the-world approach. Once again, this is a hack to get around deficiencies in pre-2.8.0 scalac.
The good news is that reasonably fast compilation is still achievable even without worrying about change detection. FSC uses the same technology (ooh, that sounded so "Charlie Eppes") that Eclipse SDT uses to implement fast incremental compilation. In short, it's pretty snappy.
Personally, I use Apache Buildr. Its configuration is significantly cleaner than either Maven's or SBT's and its startup time is orders of magnitude less (when running under MRI). It integrates with FSC and attempts to do some basic change detection on its own (fairly primitive). It also has auto-magical support for the major Scala test frameworks (ScalaTest, ScalaCheck and Specs) as well as support for joint compilation with Java sources and IDE meta generation for IntelliJ and Eclipse. Oh, and it supports all of Maven's features (dependency resolution, etc) and then some. I'm even working on an extension which would allow interactive shell support integrated with JavaRebel and supporting several shell providers (Scala, JIRB, Clojure REPL, etc). It's not ready for the SVN yet, but I'll commit once it's ready (possibly in time for 1.3.5).
As you can see, I'm very firmly of the opinion that Buildr is the best Scala build tool out there. Its documentation is a little spotty where Scala is concerned, but that's because everything is so straightforward that it's hard to document without feeling verbose. You can always check out one of my GitHub repositories for examples. Good luck!
Have you looked at Intellij IDEA and its Scala integration ? Intellij has a loyal (fanatical?) following amongst Java developers, so you may find this is appropriate for your needs.
Am also quite frustrated with the scala plugin on Eclipse and I can add a few more problems to the list:
auto-complete only works some of the time
the debugger doesn't work properly (especially when trying to debug scala xml)
the debugger forgets breakpoints
'go to definition' doesn't work more often than not.
I'm glad to hear that Buildr sounds like a better alternative (on the build front anyhow), I'll give that a try - thanks!
If you use Emacs, I think Ensime is a pretty good IDE. I think at the time writing, Ensime is the only IDE that will give you fast and accurate autocompletion on both Scala and Java objects, including implicit conversions.
There's code browsing support using Speedbar, code templates using the excellent Yasnippet, and code completion menu using Autocomplete. These are all very modern, actively maintained Emacs packages. There's also out of the box incremental building support for Maven and SBT.
There's a lot more in there such as interactive debugging, refactoring, and the Scala interpreter in an inferior process. All the things you want in a modern IDE for Scala is already there in Ensime. Highly recommended for Emacsens.
For the reasons of completeness, I have to say that there is also Pants -- the build tool that in use in Twitter (one of the early scala adopters)
The main difference it that it is intended not only for scala (and written in python, by the way) and is modeled after google build system.
It's not so bloated as sbt, so for the freshmans it's much simplier, but I've never heard about Pants usage outside of twitter and foursquare.
If you scared of SBT, maybe another no-so-popular build tool, ABT, could be an alternative for you?
I went down the same road, and here is where I am at:
- After some initial investigation, I dropped Kate. I love to use it for most things, but when it came to things like defining tab completions, I found it sorely lacking. I would recommend that you look into gedit instead, which is much more robust for Scala development
- With gedit as my editor, I use SBT and have found it to be a great build tool. I can put it into a 'test' mode where when any code changes it recompiles the relevant files and runs my test suite. This has been an extremely effective way to work.
I have not taken a look at Buildr yet. I would like to say that I will, but honestly with SBT at my disposal I don't really have a compelling need to look at another build tool.
If you want to use Eclipse, but build the project using sbt, and still be able to debug, take a look at this post here:
zikaprog.wordpress.com/2010/04/19/scala-eclipse-sbt-and-debugging/
It also can be applied to builders other than sbt.
The latest version of the Maven Scala plugin supports Zinc/Nailgun for faster start times and faster incremental builds. See Zinc and Incremental Compilation.