Can some one please let me know how to connect to Spark application from a web application. The Spark code whatever we are having is on Scala and we want to access the code from web application. If we can provide a small code snippet that will be helpful.
As mentioned in the comment section, try the following approaches,
spark-jobserver
spark-jobserver provides a RESTful interface for submitting and managing Apache Spark jobs, jars, and job contexts.
Approach mentioned by Jamal in spark-user-list
1) In the tomcat's classpath, add the spark distribution jar for spark code to be available at runtime
2) In the Web application project, add the spark distribution jar in the classpath ( Could be Java / Web project).
3) Setup the FAIR scheduling mode, which would help send parallel requests from web application to the spark cluster.
4) In the application startup, initialize the connection to the spark cluster. This is composed of creating the JavaSparkContext and making it available throughout the web application, in case this needs to be the only Driver Program required by the web application.
5) Using the JavaSpark Context, Create RDD's and make them available globally to the web application code.
6) invoke transformation / actions as required.
Related
I integrate spring batch into a restful controller of a spring boot, which means now we operate spring batch program by send a restful call. In this case, we cannot make a jar and register the jar on spring data flow server. So my question is that how to register a task if we don't have jar
You've asked a few similar questions today.
My recommendation is that you could consider referring to the ref. guide of Spring Cloud Task and Spring Cloud Data Flow. Specifically, pay attention to the Spring Batch section.
Once you have the understanding as to what to do, you can build a batch-job as a Spring Cloud Task application, and run it standalone successfully.
If it runs locally as expected, you can switch to SCDF and register the JAR using the REST-API, Shell, or in the GUI. You'd need a physical uber-jar of the application for it. With that registered, you can then build a Task definition with it, and launch it from SCDF.
If you want to do all of the above programmatically, please have a look at the acceptance-test suite for examples.
I create an instance of the Apache Service but when when I try to bring up it's console the console page hangs indefinitely with a spinning cursor.
For this stage of the Bluemix Apache Spark beta, you can only create apps from the Spark Starter boilerplate at the top of the catalog. A Spark Starter app will automatically create an instance of the Apache Spark service and will give you access to that service instance strictly for the purpose of using Jupyter/IPython notebooks. Later on in the beta, you should expect more functionality to be made available; and direct usage of a spark service instance is quite possible ;-)
Once you create an app from the Spark Starter boilerplate, you can click on the route/URL from the dashboard for the app and that will take you into a home page for that app. The Launch button you see there will take you to the Jupyter/IPython notebook server that utilizes the Apache Spark bluemix service.
I am beginner for Google App Engine.
I have simple java application with a class in which there is main method which executes some threads.
Actually, it's Java application which is used to execute some back-end activities.
On my Linux server, i have created a cron job which executes this class at some specific time interval.
Now, I want to move this application to Google App Engine.
I did search about that and what i find is i have to convert this java application to web application to deploy on GAE.
Please guide me how i can deploy JAR to GAE.
Thanks
Yes, your java application needs to be a web application. That is, you'd need to have a WEB-INFO directory within your jar, with the correct xml files (i.e. web.xml and appengine-web.xml).
Please, start by following one of the tutorials here:
https://cloud.google.com/appengine/docs/java/gettingstarted/introduction
You also mentioned that you use threads in your application. I believe that on App Engine you cannot just create threads, and instead you need to rely on the provided threads api:
https://cloud.google.com/appengine/docs/java/javadoc/com/google/appengine/api/ThreadManager
Thus, some of your code will need to be changed to use such an api.
Finally ! i figured out the solution without using Google App Engine and just using Google Compute Engine.
I created a JAR and put it into my home folder using SFTP.
After that i created a cron file at /etc/cron.d/myfile as below
*/10 * * * * myusername java -jar /home/myusername/myjar.jar
Then i reload the crontab
sudo service cron reload
That's it ! Thanks for your input and interest !
If you mean just a simple JAR with a main method, that won't work. Your application has to be a web service that responds to requests.
Check this link to create an app using JAR file:
APP-ENGINE
You cannot deploy a single jar file to App Engine. You can deploy an application.
This tutorial explains how to create an App Engine project in Java:
https://cloud.google.com/appengine/docs/java/gettingstarted/introduction
Yes, you can only deploy web applications to Google App Engine, so you may have to convert your class into a Servlet class if you want to deploy it to App engine.
You can use the Servlet's url if you are looking to execute it as a cron job, so you can configure this url in a file cron.xml by specifying the frequency and time when cron job should be executed before deploying.
You can use the link below which will guide you through the steps :
http://www.vogella.com/tutorials/GoogleAppEngineJava/article.html
Google's documentation on Cron Job :
https://cloud.google.com/appengine/docs/java/config/cron
You can not however use threads when working with App Engine.
I've been developing a Spark Streaming application with Eclipse, and I'm using sbt to run it locally.
Now I want to deploy the application on AWS using a jar, but when I try to use the command package of sbt it creates a jar without all dependencies so when I upload it on AWS it won't work because of Scala being missing.
Is there a way to create a uber-jar with SBT? Am I doing something wrong with the deployment of Spark on AWS?
For creating uber-jar with sbt, use sbt plugin sbt-assembly. For more details about creating uber-jar using sbt-assembly refer the blog post
After creating you can run the assembly jar using java -jar command.
But from Spark-1.0.0 onwards the spark-submit script in Spark’s bin directory is used to launch applications on a cluster for more details refer here
You should really be following Running Spark on EC2 that reads:
The spark-ec2 script, located in Spark’s ec2 directory, allows you to
launch, manage and shut down Spark clusters on Amazon EC2. It
automatically sets up Spark, Shark and HDFS on the cluster for you.
This guide describes how to use spark-ec2 to launch clusters, how to
run jobs on them, and how to shut them down. It assumes you’ve already
signed up for an EC2 account on the Amazon Web Services site.
I've only partially followed the document so I can't comment on how well it's written.
Moreover, according to Shipping Code to the Cluster chapter in the other document:
The recommended way to ship your code to the cluster is to pass it
through SparkContext’s constructor, which takes a list of JAR files
(Java/Scala) or .egg and .zip libraries (Python) to disseminate to
worker nodes. You can also dynamically add new files to be sent to
executors with SparkContext.addJar and addFile.
Is there anyone that has done a java interpreter using groovy-all jar file? Maybe sample or example can share it to me or teach me? I meant a interpreter that can parse string(java code) into the textarea and output it as a result like(hello world)
As you need some sample code to implement a web-console using groovy-all.jar, it would strongly recommend taking a look at Groovy Web Console.
Although it's not exactly a Java EE / Tomcat app and it is fairly similar as its a standard Java Servlets API 2.5 based web app. It runs on Google App Engine, which you can try out here. All you need from it is the script execution logic which for most of the part is not app engine specific. Keep in mind, it has dependencies on GAE Apis (through Gaelyk) so you should prune that part out of it to run in it outside Google App Engine.