What is the best way to convert scala (or bytecode) to native binary in order to increase performance
At this moment I see two solutions to convert jvm bytecode to sort of self-contained native binary:
Avian - lightweight embeddable JVM with AOT features
Excelsior JET - Commercial Java native compiler
Both should be compatible with Scala.
There are no direct native compilers for scala as I know. There some projects like Scala LLVM, but they are more about research and proof of concepts than ready to use tools
Although not a fully capable tool yet, the scala-native project is starting to become usable, though it's still at an early stage, it's under active development and is becoming more capable by the month. It's based on LLVM and clang, and will compile your scala sources to binaries, if the libraries you depend on are among those implemented at this early stage. (it's not yet working in Windows or cygwin, although it does work in the WSL environment).
Update: Windows support has been improving recently (fall 2021).
Whether performance is increased or not is a separate question, although most programs are likely to start up much more quickly.
Here's a link to the User's Guide
To create your own project: Minimal sbt project
The biggest limitations are that only a subset of java and scala standard libraries have been implemented so far, so you'll need to limit yourself to what's currently available, and not every project will only be feasible if you restrict yourself to 100% scala. Also, the documentation is a work in progress.
As a test, I created a command line tool for processing text files, and I was able to get it to work finally, although I did spend a bit of time figuring out how to accomplish various things, and mostly how to live with the available libraries. If necessary, you can also link to C/C++ libraries although I didn't need to for my small project.
Footnote as of June 2019: I'm having good luck with graalvm native-image. Here's the link:
https://www.graalvm.org/docs/reference-manual/aot-compilation/
I couldn't get clear idea in differences between framework and libraries, can you suggest your idea or refer me any website or article thanks.
Apple's Doc is always a good place to start:
https://developer.apple.com/library/ios/documentation/MacOSX/Conceptual/BPFrameworks/Frameworks.html
In sum, Frameworks are an Apple specific mechanism for packaging dynamic code in a well defined directory structure. Frameworks are a particular type of Bundle which allow for versioning, storing and loading resources (icons, string files). Dynamic libraries provide only the shared module (with the *.dylib extension).
Note, OS X supports both methods, but Frameworks provide a great deal of flexibility both in development, deployment and management and are usually the preferred choice.
Using Frameworks requires compiler/linker specific flags to properly use. You should consult the Clang/XCode documentation.
I have been using CDH and HDP for a while (both in the pseudo-distributed mode) on a VM as well as installing natively on Ubuntu. Although my question is probably relevant to all Projects within the Apache Hadoop Ecosystem, let me ask this specifically in the context of Avro.
What is the best way to go about figuring out what the different packages and the classes within the packages do. I usually end up referring to the Javadoc for the project (Avro in this case) but the overviews for packages and classes end up being awfully inadequate.
For e.g. Take two of the Avro packages: org.apache.avro.specific and org.apache.avro.generic These are used for creating Specific and Generic Readers and Writers (respectively) but I'm not a 100% sure what these are for. I have used the Specific Package for in cases when I have used Avro Code Generation and the Generic ones when I don't want to use code generation. However, I am not sure if that is the only reason for using one vs. the other.
Another example: The Encoder\Decoder Classes are used for low-level SerDe, the DatumReader\DatumWrite for a "medium-level" Serde while most application layer interactions with Avro will probably use Generic\Specific Readers\Writers. Without having struggled through the pain of using these classes, how is a user to know what to use for what?
Is there a better way to get a good overview of each package (clearly the javadoc is not well documented) and the classes within the package?
PS: I have similar questions for essentially all other Hadoop Projects (Hive, HBASE etc.) - the Javadocs seem to be grossly inadequate overall. I just wonder what other developers end up doing to figure these out.
Any inputs would be great.
I download the source code and skim through it to get the idea what it does. If there is javadoc, I read that too. I tend to concentrate on the interfaces that I need and move on from there, that way I put everything into context and it makes it easier to figure out the usage. I use the call hierarchy and the type hierarchy views a lot.
These are very general guidelines, and ultimately it is the time you spend with the project that will make you understand it.
Hadoop ecosystem is quickly growing and changes are introduced on monthly bases. that's why javadoc is not so good. Another reason is that hadoop software tends to lean towards the infrastructure and not towards the end user. People developing tools will spend time learning the APIs and internals while everybody else is kinda supposed to be blissfully ignorant of all those, and just use some high level domain specific language for the tool.
A repeating theme in my development work has been the use of or creation of an in-house plug-in architecture. I've seen it approached many ways - configuration files (XML, .conf, and so on), inheritance frameworks, database information, libraries, and others. In my experience:
A database isn't a great place to store your configuration information, especially co-mingled with data
Attempting this with an inheritance hierarchy requires knowledge about the plug-ins to be coded in, meaning the plug-in architecture isn't all that dynamic
Configuration files work well for providing simple information, but can't handle more complex behaviors
Libraries seem to work well, but the one-way dependencies have to be carefully created.
As I seek to learn from the various architectures I've worked with, I'm also looking to the community for suggestions. How have you implemented a SOLID plug-in architecture? What was your worst failure (or the worst failure you've seen)? What would you do if you were going to implement a new plug-in architecture? What SDK or open source project that you've worked with has the best example of a good architecture?
A few examples I've been finding on my own:
Perl's Module::Plugable and IOC for dependency injection in Perl
The various Spring frameworks (Java, .NET, Python) for dependency injection.
An SO question with a list for Java (including Service Provider Interfaces)
An SO question for C++ pointing to a Dr. Dobbs article
An SO question regarding a specific plugin idea for ASP.NET MVC
These examples seem to play to various language strengths. Is a good plugin architecture necessarily tied to the language? Is it best to use tools to create a plugin architecture, or to do it on one's own following models?
This is not an answer as much as a bunch of potentially useful remarks/examples.
One effective way to make your application extensible is to expose its internals as a scripting language and write all the top level stuff in that language. This makes it quite modifiable and practically future proof (if your primitives are well chosen and implemented). A success story of this kind of thing is Emacs. I prefer this to the eclipse style plugin system because if I want to extend functionality, I don't have to learn the API and write/compile a separate plugin. I can write a 3 line snippet in the current buffer itself, evaluate it and use it. Very smooth learning curve and very pleasing results.
One application which I've extended a little is Trac. It has a component architecture which in this situation means that tasks are delegated to modules that advertise extension points. You can then implement other components which would fit into these points and change the flow. It's a little like Kalkie's suggestion above.
Another one that's good is py.test. It follows the "best API is no API" philosophy and relies purely on hooks being called at every level. You can override these hooks in files/functions named according to a convention and alter the behaviour. You can see the list of plugins on the site to see how quickly/easily they can be implemented.
A few general points.
Try to keep your non-extensible/non-user-modifiable core as small as possible. Delegate everything you can to a higher layer so that the extensibility increases. Less stuff to correct in the core then in case of bad choices.
Related to the above point is that you shouldn't make too many decisions about the direction of your project at the outset. Implement the smallest needed subset and then start writing plugins.
If you are embedding a scripting language, make sure it's a full one in which you can write general programs and not a toy language just for your application.
Reduce boilerplate as much as you can. Don't bother with subclassing, complex APIs, plugin registration and stuff like that. Try to keep it simple so that it's easy and not just possible to extend. This will let your plugin API be used more and will encourage end users to write plugins. Not just plugin developers. py.test does this well. Eclipse as far as I know, does not.
In my experience I've found there are really two types of plug-in Architectures.
One follows the Eclipse model which is meant to allow for freedom and is open-ended.
The other usually requires plugins to follow a narrow API because the plugin will fill a specific function.
To state this in a different way, one allows plugins to access your application while the other allows your application to access plugins.
The distinction is subtle, and sometimes there is no distiction... you want both for your application.
I do not have a ton of experience with Eclipse/Opening up your App to plugins model (the article in Kalkie's post is great). I've read a bit on the way eclipse does things, but nothing more than that.
Yegge's properties blog talks a bit about how the use of the properties pattern allows for plugins and extensibility.
Most of the work I've done has used a plugin architecture to allow my app to access plugins, things like time/display/map data, etc.
Years ago I would create factories, plugin managers and config files to manage all of it and let me determine which plugin to use at runtime.
Now I usually just have a DI framework do most of that work.
I still have to write adapters to use third party libraries, but they usually aren't that bad.
One of the best plug-in architectures that I have seen is implemented in Eclipse. Instead of having an application with a plug-in model, everything is a plug-in. The base application itself is the plug-in framework.
http://www.eclipse.org/articles/Article-Plug-in-architecture/plugin_architecture.html
I'll describe a fairly simple technique that I have use in the past. This approach uses C# reflection to help in the plugin loading process. This technique can be modified so it is applicable to C++ but you lose the convenience of being able to use reflection.
An IPlugin interface is used to identify classes that implement plugins. Methods are added to the interface to allow the application to communicate with the plugin. For example the Init method that the application will use to instruct the plugin to initialize.
To find plugins the application scans a plugin folder for .Net assemblies. Each assembly is loaded. Reflection is used to scan for classes that implement IPlugin. An instance of each plugin class is created.
(Alternatively, an Xml file might list the assemblies and classes to load. This might help performance but I never found an issue with performance).
The Init method is called for each plugin object. It is passed a reference to an object that implements the application interface: IApplication (or something else named specific to your app, eg ITextEditorApplication).
IApplication contains methods that allows the plugin to communicate with the application. For instance if you are writing a text editor this interface would have an OpenDocuments property that allows plugins to enumerate the collection of currently open documents.
This plugin system can be extended to scripting languages, eg Lua, by creating a derived plugin class, eg LuaPlugin that forwards IPlugin functions and the application interface to a Lua script.
This technique allows you to iteratively implement your IPlugin, IApplication and other application-specific interfaces during development. When the application is complete and nicely refactored you can document your exposed interfaces and you should have a nice system for which users can write their own plugins.
I once worked on a project that had to be so flexible in the way each customer could setup the system, which the only good design we found was to ship the customer a C# compiler!
If the spec is filled with words like:
Flexible
Plug-In
Customisable
Ask lots of questions about how you will support the system (and how support will be charged for, as each customer will think their case is the normal case and should not need any plug-ins.), as in my experience
The support of customers (or
fount-line support people) writing
Plug-Ins is a lot harder than the
Architecture
Usualy I use MEF. The Managed Extensibility Framework (or MEF for short) simplifies the creation of extensible applications. MEF offers discovery and composition capabilities that you can leverage to load application extensions.
If you are interested read more...
In my experience, the two best ways to create a flexible plugin architecture are scripting languages and libraries. These two concepts are in my mind orthogonal; the two can be mixed in any proportion, rather like functional and object-oriented programming, but find their greatest strengths when balanced. A library is typically responsible for fulfilling a specific interface with dynamic functionality, whereas scripts tend to emphasise functionality with a dynamic interface.
I have found that an architecture based on scripts managing libraries seems to work the best. The scripting language allows high-level manipulation of lower-level libraries, and the libraries are thus freed from any specific interface, leaving all of the application-level interaction in the more flexible hands of the scripting system.
For this to work, the scripting system must have a fairly robust API, with hooks to the application data, logic, and GUI, as well as the base functionality of importing and executing code from libraries. Further, scripts are usually required to be safe in the sense that the application can gracefully recover from a poorly-written script. Using a scripting system as a layer of indirection means that the application can more easily detach itself in case of Something Bad™.
The means of packaging plugins depends largely on personal preference, but you can never go wrong with a compressed archive with a simple interface, say PluginName.ext in the root directory.
I think you need to first answer the question: "What components are expected to be plugins?"
You want to keep this number to an absolute minimum or the number of combinations which you must test explodes. Try to separate your core product (which should not have too much flexibility) from plugin functionality.
I've found that the IOC (Inversion of Control) principal (read springframework) works well for providing a flexible base, which you can add specialization to to make plugin development simpler.
You can scan the container for the "interface as a plugin type advertisement" mechanism.
You can use the container to inject common dependencies which plugins may require (i.e. ResourceLoaderAware or MessageSourceAware).
The Plug-in Pattern is a software pattern for extending the behaviour of a class with a clean interface. Often behaviour of classes is extended by class inheritance, where the derived class overwrites some of the virtual methods of the class. A problem with this solution is that it conflicts with implementation hiding. It also leads to situations where derived class become a gathering places of unrelated behaviour extensions. Also, scripting is used to implement this pattern as mentioned above "Make internals as a scripting language and write all the top level stuff in that language. This makes it quite modifiable and practically future proof". Libraries use script managing libraries. The scripting language allows high-level manipulation of lower level libraries. (Also as mentioned above)
I have a set of functionality (classes) that I would like to share with an application I'm building for the iPhone and for the Blackberry (Java). Does anyone have any best practices on doing this?
This is not going to be possible as far as I understand your question - the binary format for the iPhone and Java are not compatible - and even for a native library on a blackberry device.
This is not like building for OS X where you can use Java unfornately the iPhone doesn't support Java.
The best idea is probably to build you library in Objective-C and then port it to Java which is an easier transition than going the other way. If you programme for Objective-C and make sure you code has no memory leaks - then the changes are not so complex.
If you keep the structure of your classes the same then you should find maintenance much simpler - fix a bug in the Java and you should find it easy to check for the same bug in the ObjC methods etc.
Hope this helps - sorry that it is not all good news.
As Grouchal mentioned - you are not going to be able to share any physical components of your application between the two platforms. However you should be able to share the logical design of your application if you carefully separate it into highly decoupled layers. This is still a big win because the logical application design probably accounts for a large part of your development effort.
You could aim to wrap the sections of the platform specific APIs (iPhone SDK etc.) that you use with your own interfaces. In doing so you are effectively hiding the platform specific libraries and making your design and code easier to manage when dealing with differences in the platforms.
With this in place you can write your core application code so that it appears very similar on either platform - even though they are written in different languages. I find Java and Objective-C to be very similar conceptually (at least at the level at which I use it) and would expect to be able to achieve parity with at least the following:
An almost identical set of Java and Objective-C classes with the same names and responsibilities
Java/Objective-C classes with similarly named methods
Java/Objective-C methods with the same responsibilities and logical implementations
This alone will make the application easier to understand across platforms. Of course the code will always look very different at the edges - i.e when you start dealing with the view, threading, networking etc. However, these concerns will be handled by your API wrappers which once developed should have fairly static interfaces.
You might also stand to benefit if you later developer further applications that need to be delivered to both platforms as you might find that you can reuse or extend your API wrappers.
If you are writing a client-server type application you should also try and keep as much logic on your server as possible. Keep the amount of extra business logic on the device to a minimum. The more you can just treat the device as a view layer the less porting you'll have to do over all.
Aside from that, following the same naming conventions and package structure across all the projects helps greatly, especially for your framework code.
The UI API's and usability paradigms for BlackBerry and iPhone are so different that it won't be possible in most cases to directly port this kind of logic between apps. The biggest mistake one could make (in my opinion) is to try and transplant a user experience designed for one mobile platform on to another. The way people interact with BlackBerrys vs iPhones is very different so be prepared to revamp your user experience for each mobile platform you want to deploy on.
Hope this is helpful.
It is possible to write C++ code that works in both a BB10 Native app and an iOS app.
XCode would need to see the C++ files as ObjectiveCPP code.
I am currently working on such a task in my spare time. I have not yet completed it enough to either show or know if it is truly possible, but I haven't run in to any road-blocks yet.
You will need to be disciplined to write good cross-platform code designed w/ abstractions for platform-specific features.
My general pattern is that I have "class Foo" to do cross platform stuff, and a "class FooPlatform" to do platform specific stuff.
Class "Foo" can call class "FooPlatform" which abstracts out anything platform specific.
The raw cross-platform code is itself not compile-able on its own.
Separate BB10 and XCode projects are created in their respective IDEs.
Each project implements a thin (few [dozen] line) "class FooPlatform" and references the raw cross-platform code.
When I get something working that I can show I will post again here...