I am beginner in apache spark-scala, and I have been created a project with SBT (self-contained application).
I searched on Internet and I asked many people, why we create the self-contained application on scala with SBT ? Why we need for self-contained application ? What is the interest of the self-contained application ?
I arrived to these results:
- we create the self-contained application in scala with sbt to create an arborescence of project and we can run a full program.
- In the running of program we genere a file jar.
I run my program like this:
spark-submit --class "name_file" --master local[*] target/scala-2.11/"name_project"_2.11-1.0.jar
With the self-contained application we can give my file jar to another person and he can running my program without install spark or scala in another machine (like the files .dll in C++).
And finally, may be my question is not normaly, but I told, that I can ask my question in StackoverFlow and I am sure that I'll find the best answer.
Thank you!
Name itself states that it have no external dependencies while running (not developing) it in production environments.
A self-contained application consists of a single, installable bundle
that contains your application and a copy of the JRE or any Runtime
environment needed to run the application. When the application is
installed, it behaves the in the same way as any native application.
Read more from here
And in you case a simple self contained application is here
Related
I'm building an application using Scala + Play!. This is not a normal web application. Instead, it's just a software that will do something on localhost. In other words it will be distributed to multiple clients' machines. As a result, I want it to be really portable and light-weight. I don't want the clients to have to manually install Scala, Play, and JVM before they can run the program.
So my question is, is there a way to bundle a Play! application into a single executable? It doesn't really have to be just a single file, i.e. it just have to be really easy for a client with little technical knowledge to run this application easily.
Hope I'm being clear...
Thank you!
Or maybe you are just looking for a way to easily dist your application? :) have a look here: http://www.playframework.com/documentation/2.1.0/ProductionDist
This will produce a self-contained .zip file in your project's brand new dist sub directory. The only thing missing in that archive would be the JVM
You could use things like sbt-native-packager and sbt-assembly.
I am starting to use Cascading library, but I search information and all is about cascading on linux... I have executed fine the Impatient examples in a ubuntu server.
But I want to develop and test my application using eclipse in windows...
Is that posssible?? How I can do it?
Thanks
Glad to hear the "Impatient" examples helped out -
There are two concerns: (1) Windows and (2) Eclipse.
Hadoop runs in Java, and is primarily meant for running apps on clusters. You must be careful on Windows, because the Java support is problematic. I've seen many students attempt to use Cygwin, thinking that would provide a Java layer -- it does not. Running Hadoop atop Cygwin typically is more trouble than it's worth. Obviously the HDInsight work by Microsoft is a great way to run Hadoop on Windows, on Azure. To run Hadoop on your desktop Windows, it's best to use a virtual machine. Then be certain to run in "Standalone Mode", instead of pseudo-distributed mode or attempting to create a cluster on your desktop. Otherwise, it'd be better to run Cascading apps in HDInsight for Hadoop on Azure.
Eclipse is a much simpler answer. Gradle build scripts in for the "Impatient" series show how to use "gradle eclipse" to generate a project to import into your IDE. Even so, you may have to clean up some paths -- Eclipse doesn't handle Gradle imports as cleanly as it should, from what I've seen.
Hope that helps -
To develop and test your Cascading application using eclipse in windows, you need to apply a patch (https://github.com/congainc/patch-hadoop_7682-1.0.x-win). Download the patch jar, then add to your application's CLASSPATH. In your code, you need to set the properties "fs.file.impl"
Properties properties = new Properties();
AppProps.setApplicationJarClass(properties, Main.class);
if (System.getProperty("os.name").toLowerCase().indexOf("win") >= 0) {
properties.put("fs.file.impl",
"com.conga.services.hadoop.patch.HADOOP_7682.WinLocalFileSystem");
}
HadoopFlowConnector flowConnector = new HadoopFlowConnector(properties);
When using Perl script as mapper & reducer in Hadoop streaming, how we can manage perl module dependencies.
I want to use "Net::RabbitMQ" in my perl mapper & reducer script.
Is there any standard way in perl/hadoop streaming to handle dependencies similar to the DistributedCache (for Hadoop java MR)
There are a couple of ways to handle dependencies including specifying a custom library path or creating a packed binary of your Perl application with PAR::Packer. There are some examples of how to accomplish these tasks in the Examples section of the Hadoop::Streaming POD, and the author includes a good description of the process, as well as some considerations for the different ways to handle dependencies. Note that the suggestions provided in the Hadoop::Streaming documentation about handling Perl dependencies are not specific to that module.
Here is an excerpt from the documentation for Hadoop::Streaming (there are detailed examples therein, as previously mentioned):
All perl modules must be installed on each hadoop cluster machine. This proves to be a challenge for large installations. I have a local::lib controlled perl directory that I push out to a fixed location on all of my hadoop boxes (/apps/perl5) that is kept up-to-date and included in my system image. Previously I was producing stand-alone perl files with PAR::Packer (pp), which worked quite well except for the size of the jar with the -file option. The standalone files can be put into hdfs and then included with the jar via the -cacheFile option.
I've been contemplating moving my project over to cloud 9 IDE but have been having trouble running coffee script in the project. I copied over all my js and coffee files but can't seem to get a run configuration working using the coffee files. I tried compiling the coffee files in the console command line as well as creating a run configuration that calls the app.coffee directly but no luck.
What is the coffee script support on cloud 9 and how does it work. Does it compile the coffee script to js automatically? How do I need to configure my run settings in cloud 9?
I got it working... your results may vary, but this is what I did...
Assuming you have your app which is called app.coffee, I created a file called runner.js with the following code:
require("coffee-script");
require("./app");
From the Cloud9 IDE, I just tell it to run the runner.js file and it seems to work.
I also wasn't able to use my global npm-installed packages... not sure why, but I am guessing it is just a pathing issue. Anyways, I just installed my packages into my project directory:
npm install coffee-script
I was also using express and restler in my project so I did the same thing in the project. It worked beautiffuly :)
There is an entry on the Cloud9 support page on running node.js applications written in CoffeeScript: Create a CoffeeScript node.js project. Note that you can only debug the javascript files.
We have set of unix shell(ksh) scripts used for deployment of our product.
Actually there is a mixture of ksh+sed+awk+ant code.
Our product works on AIX only so we did not try to do our scripts portable.
But now we need to run part of our scripts not only on AIX but on Windows also.
We need to have ability to create/update DB both from AIX and from Windows.
Now for this part of functionality we use ksh+ant.
We have a requirement to install as few as possible tools on this Windows box.
In the best case it should be JRE+our products only.
What do you propose to use instead of ksh?
As I know we can put Groovy jar into our project and write this part of functionality on Groovy.
In this case this code will be portable.
May be there are better solutions than Groovy?
Any JVM language such as Jython or Scala should work as well as Groovy so it’s really a choice of what the developers are comfortable with. I’ve had good success with Groovy and have been able to bundle Groovy as a jar file and execute any script I wanted in the following way
Java -jar groovy.jar myscript.groovy
I’ve been able to do this on z/OS, Windows, and Linux.