Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 6 years ago.
Improve this question
I am working with Spark.
I want the Spark application to be a long running application which doesn't exit after it finishes computation and to listen for HTTP requests and return the computed data.
How can I do it out-of-the box? Now, I can only write a while loop to ensure the program keeps going.
Spark doesn't have such features out of the box. Spark streaming does via awaitTermination() method which you call on the StreamingContext. Then you just need to implement a HTTP endpoint in your Spark application.
Using Spark Streaming functionality would be the easiest, you can still leave your Spark jobs to use regular RDDs and not DStreams, you can use the StreamingContext just for the awaitTermination.
If you don't want to use Spark Streaming you still probably can have a look at how they implement it using locks here ContextWaiter#waitForStopOrError()
Related
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 19 days ago.
Improve this question
I have to implement a solution that consists of processing a large amount of data by applying business requirement rules. The input and the output will be a file.
I haven't been using Kafka before, I am wondering if I can use Kafka streams to process these rules or use spring batch combined with Kafka streams.
Is there any other frameworks/technologies that can be used in Java?
Thank you
Kafka Streams is a stream processing solution; what you're talking about is more of a batch workload. The difficulties you will encounter using KStreams are:
Kafka Streams doesn't have a good way of working with files as input and output.
In Stream Processing, there's no real concept of "beginning" and "end," whereas I gather from the nature of your question that you do indeed have a beginning and end in your use-case.
As such I would recommend another batch solution.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 1 year ago.
Improve this question
I have a bunch of scripts collecting data from internet and local services, writing them to disk, scripts transforming the data and writing it into a database, scripts reading data from the database and generating new data, etc, written in bash, Python, SQL, ... (Linux).
Apart from a few time-triggered scripts, the glue between the scripts is currently me, running the scripts now and then in a particular order to update everything.
What is the simplest way to replace me by a tool that observes dependencies and triggers the next step as soon as the preconditions are met?
I've found many ETL and data warehousing tools, but these seem too heavy weight for my simple setting. I'd prefer a CLI solution with text-based configuration (maybe able to visualise the graph of dependencies). Any suggestions?
Try airflow: airflow.apache.org
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 3 years ago.
Improve this question
How easy is to switch from Rabbit to Kafka in existing solution, to replace one implementation (Rabbit) with other (Kafka)? We are about to use Rabbit in our implementation but we want to see if it is possible in the future to replace it with Kafka.
It is possible, and I've seen people do it - but it is a big project.
Not only the APIs are different, but the semantics are different. So you need to rethink your data model, scaling model, error handling, etc. And then there's testing.
If you don't have tons of code to update, and the code is localized and you have both RabbitMQ and Kafka experts on the team you may be able to get it done in a month or two.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 4 years ago.
Improve this question
I'm writing a uber-like mobile app for an academic project. This app is only a front-end for a system running on a server. This system, the backend is responsible for the user database, a search engine, messaging system and a location service (tracking system as in uber). I'm totally new to server development. My problems are:
How scala can be used to implement a backend?
How to handle concurrency, user authorizations, messaging and GPS data?
Additionally, In what cases should I use Kafka and Spark?
I would suggest you to go with play-framework. It has some learning curve, but once you start learning it seems quite interesting and is quite powerful too.
You must be familiar with scala, concurrency terms like future, actor and some database knowledge would be enough to start.
You can start by doing the examples provided in the documentation.
https://www.playframework.com/
All the best with your learning:)
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 8 years ago.
Improve this question
Currently, I am using finagle Scala web framework as microservice for our project, They are very easy to use and also convenient to be deployable. At the same time, my colleague are trying to use Play framework for micro service, but I think it is too huge. It is not micro anymore.
May I know what is your opinion about this and is there any other good microservice framework in scala should be taken into consideration ?
Many thanks in advance
We switched from Finagle to Spray. It's minimal framework to write RESTful micro services, much smaller than Play. It's literally takes less than 10 lines of spray-based code to run new service.
If you don't plan to build web UI with play I don't see why to use this heavy framework for building just REST services.