I want to activate the Kafka Spark pipeline for the Thingsboard platform (community edition).
As per the mentioned Stack question "Couldn't able to find plugins in ThingsBoard 2.0.3 Home screen"
It's been said that we can do it via Rule chains itself since the plugin section has been removed, but I am not able to understand how to configure it using rule chains. I am not able to get the complete documentation to configure Kafka via rule chains. So need help on that.
I figured it out. By using this link it can be done easily "https://thingsboard.io/docs/samples/analytics/kafka-streams/"
The thing is that using the Thingsboard CE we can get data into Kafka-topic. However, to fetch data from Kafka you will need to have TB Professional Edition integration.
The alternate option to Thingsboard PE is to write your own REST API script to push the insights back to ThingsBoard.
i have a Spring Cloud Task fat jar that i have successfully deployed to SCDF running on PCF. i have created a definition for it and can therefore run it from the dashboard. fwiw it reads and writes from a database using Spring JDBC.
i'm trying to now set it up to run in a scheduled way and am having issues. i created a stream with a triggertask source and a task-launcher-local sink, and have configured the triggertask URI to point to the fat jar (via http, using a staticfile PCF pushed app).
the dashboard shows the two PCF apps (one for triggertask, one for task-local-launcher) both starting successfully, and it all runs, but the task fails every time with the error:
Caused by: java.io.IOException: Cannot run program "java" (in directory "/home/vcap/tmp/spring-cloud-dataflow-5903184636016162160/Task--582903409-1502669137014/Task--582903409"): error=2, No such file or directory
from what i can tell and surmise, the PCF app running the stream tries to fork and exec a java call, but since java is not in the path for PCF app containers i get the error
am i right? either way, how can i get the Spring Cloud Task (jar) to successfully run?
Spring Cloud Data Flow: Server
1.2.3 (using built spring-cloud-dataflow-server-cloudfoundry-1.2.3.BUILD-SNAPSHOT.jar)
Spring Cloud Data Flow: Shell
1.2.3 (using downloaded spring-cloud-dataflow-shell-1.2.3.RELEASE.jar)
Deployment Environment
PCF v1.11.6 (on Azure)
pcf dev v0.26.0 (on mac)
App Starters
http://bit-dot-ly/1-0-4-GA-stream-applications-rabbit-maven
Logs
link to log
The stream definition is missing from the post. It is possible that you're using the tasklauncher-local sink, which is compatible only when using SCDF's local-server and it will fail with the attached error when running in CF. Please make sure you're using tasklauncher-cloudfoundry sink. This application was added in the latest release of app-starters.
As pointed in the previous SO thread, it is highly recommended that you use the latest release of app-starters (1.0.4 is at least 10 months old). The latest releases can be found at the project site.
Can some one please let me know how to connect to Spark application from a web application. The Spark code whatever we are having is on Scala and we want to access the code from web application. If we can provide a small code snippet that will be helpful.
As mentioned in the comment section, try the following approaches,
spark-jobserver
spark-jobserver provides a RESTful interface for submitting and managing Apache Spark jobs, jars, and job contexts.
Approach mentioned by Jamal in spark-user-list
1) In the tomcat's classpath, add the spark distribution jar for spark code to be available at runtime
2) In the Web application project, add the spark distribution jar in the classpath ( Could be Java / Web project).
3) Setup the FAIR scheduling mode, which would help send parallel requests from web application to the spark cluster.
4) In the application startup, initialize the connection to the spark cluster. This is composed of creating the JavaSparkContext and making it available throughout the web application, in case this needs to be the only Driver Program required by the web application.
5) Using the JavaSpark Context, Create RDD's and make them available globally to the web application code.
6) invoke transformation / actions as required.
If I'm using Play! to build a scala web app, how can I connect to a remote instance, run a map reduce job there? I have the scripts for the map reduce job, just not sure how I can connect to the instance within play and run the map reduce job on the instance using parameter data I get from the user through the web app?
I love Heroku but I would prefer to develop in Scala rather than Ruby on Rails.
Does anyone know of any services like Heroku that work with Scala?
UPDATE: Heroku now officially supports Scala - see answers below for links
As of October 3rd 2011, Heroku officially supports Scala, Akka and sbt.
http://blog.heroku.com/archives/2011/10/3/scala/
Update
Heroku has just announced support for Java.
Update 2
Heroku has just announced support for Scala
Also
Check out Amazon Elastic Beanstalk.
To deploy Java applications using
Elastic Beanstalk, you simply:
Create your application as you
normally would using any editor or IDE
(e.g. Eclipse).
Package your
deployable code into a standard Java
Web Application Archive (WAR file).
Upload your WAR file to Elastic
Beanstalk using the AWS Management
Console, the AWS Toolkit for Eclipse,
the web service APIs, or the Command
Line Tools.
Deploy your application.
Behind the scenes, Elastic Beanstalk
handles the provisioning of a load
balancer and the deployment of your
WAR file to one or more EC2 instances
running the Apache Tomcat application
server.
Within a few minutes you will
be able to access your application at
a customized URL (e.g.
http://myapp.elasticbeanstalk.com/).
Once an application is running,
Elastic Beanstalk provides several
management features such as:
Easily deploy new application versions
to running environments (or rollback
to a previous version).
Access
built-in CloudWatch monitoring metrics
such as average CPU utilization,
request count, and average latency.
Receive e-mail notifications through
Amazon Simple Notification Service
when application health changes or
application servers are added or
removed.
Access Tomcat server log
files without needing to login to the
application servers.
Quickly restart
the application servers on all EC2
instances with a single command.
Another strong contender is Cloud Foundry. One of the nice features of Cloud Foundry is the ability to have a local version of "the cloud" running on your laptop so you can deploy and test offline.
I started working on the exact same thing as what you said a few weeks ago. I use Lift, which is a great framework and has a lot of potential, on top of Linux chroot environment.
I'm done with a demo version, but Linux chroot is not that stable (nor secure), so I'm now switching to FreeBSD jail on Amazon EC2, and hopefully it'll be done soon.
http://lifthub.net/
There are also other Java hosting environment including VMForce mentioned above.
If you are looking for a custom setup which also has the ease of deployment that heroku offers: http://dotcloud.com. They are invite only right now but I was given access in under three days. I am working on a Lift/MongoDB project there and it works well.
Off the top of my head, only VMForce comes to mind, but its not available yet. This will be a Java-oriented service, so that probably means you'll have to spend a wee bit of time figuring out how to package the app.
For more discussion, there was a debate about this in 2008.
I'm not entirely sure if it's really suitable or not, but people have deployed Scala applications to Google App Engine, for example http://mawson.wordpress.com/2009/04/10/first-steps-with-scala-on-google-app-engine/
Actually you can run scala on heroku right now. You don't believe it?
https://github.com/lstoll/heroku-playframework-scala
I'm not sure the tricks lstoll has used are legit but using the
new cedar platform where you can run custom processes and some
ingenious Gemfile hacking he has managed to bootstrap the Java
play platform into a process. Seems to work as he has a live
site running a test page.
Stax cloud service offers preconfigured lift project skeleton. Also, there is a tutorial on how to deploy lift project to appengine.