Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking us to recommend or find a tool, library or favorite off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.
Closed 9 years ago.
Improve this question
Are there some good demo project using Apache Kafka (version 0.8 is preferred) on Github (or somewhere else)? We've been testing it with some toy projects. But I'd like to check out some real-world bigger projects.
Toy projects are as good as demo projects. It will be fun writing your own simple producers/consumers. You can create your own demo project (and while doing so you will learn a lot). Think of a problem where you require large amount of say streaming data (Think logs of a running application). Now make Kafka to read those Logs. Kafka is just a kind of message queue. Until and unless you write your consumers I don't think there will be any fun. So for a real world consumer pickup say Twitter Storm. Send all the log lines of your application to the brokers where the Storm Consumer (Aka Kafka Spout) picks up those lines and send them to Bolts (Spout/Bolt are Storm terminology similar to Map Reduce but for real time).
This way you will have a full fledged demo application.
Now the main question. How to generate logs to feed to Kafka (for a demo project, if you dont have any application). There are plenty of huge data sets available (open source). From Youtube to Amazon to Twitter, all provide them. Just download it and think of some application. For example consider Youtube video logs (http://netsg.cs.sfu.ca/youtubedata/). Simulate as if they are coming online. Input them to Kafka. Let Storm Consumer (or Kafka Spout) pick that log line for you from the Kafka broker. Give each line of log to Bolt where in say the bolt just reads the line (do some analytics) and calculates the hottest/trending genres for videos watched in last X minutes.
Writing all of this should not take much time. Enjoy!
i have been using kafka from quite sometime. i am using franz-kafka nodejs client to implement pubsubhubbub spec.
i too didn't find any projects using kafka but you can ask me any questions that you have, i will try to answer them.
Thanks
You can try this https://github.com/wurstmeister/storm-kafka-0.8-plus
uses the 0.8 build
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 2 years ago.
Improve this question
I've been meaning to create MicroServices using Scala. Later we may need some web pages as well.
What is Reactive programming in terms of Micro Services or REST HTTP?
What is the relationship (if any) between Play, Lagom and Akka? Are they all used in making Micro Services?
Is Play Reactive?
Can MicroServices serve dynamic Web Pages? Or do I need a separate application for that?
I have done some work with ReactJS. Does that fit anywhere in the picture?
Please help
The question is a bit too broad to answer it here. But Jonas Bonér, the creator of Akka, explores the relationship between Microservices and Reactive Systems in his free ebook "Reactive Microservices Architecture", why don't you read that for a start.
Akka is a library/tookit, it's more low-level and doesn't guide you towards using certain patterns as frameworks do. Play and Lagom are both frameworks that prescribe, to a certain extend, a structure, and provide more architectural guidance for your application. They are both built on top of Akka. There's quite a bit of overlap between Lagom and Play, e.g. you can use either to write RESTful HTTP services. Historically Play is a "full stack web framework", including server side rendering of HTML pages. Lagom is younger, was built explicitly for microservices, and has a lot of built-in support for typical architectural patterns and challenges in that area (e.g. service lookup, async messaging between services etc.)
Yes, absolutely. For more info on that see for example "What Makes Play Frameworks Fast"?
There are different schools of thought, some say services should provide their own UI, others will have a separate service for that. If you look at Lagom sample applications like Chirper or Auction, you'll see that server-side rendering of pages in there is done by Play applications, which use Lagom services as their backends.
Yes, you can write you client application (which will run in the browser) with ReactJS and the server side with Lagom and/or Play, no problem.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
For a while I was looking for a fast and simple solution for a microservice framework. I am quite new to all Lightbend products & scala but since its looks very interesting I decided to give it a try.
Few Questions:
1) I don’t understand why there is a need for the new framework Lagom ?
If play can already give me the same solution (to serve as microservice) then why there is a need for another framework ?
2) With play I manage to create an “Hello World” project very fast and also the deployment was very easy and straight forward(via dist).
I like the fact that I can merge all in one ZIP and run it via script . In Lagom from what I understand I need to use ConductR.
For my current needs its look like a big overhead. Is there a simple why to deploy it something like in play ?
Thank you all
Lagom is built on top of Play. Whereas Play is intended to be a general-purpose (asynchronous) web framework, Lagom's more specific goal is to add some tools/opinions focussed on deploying your app as microservices.
A couple of examples that Lagom provides that helps you realise a microservices style architecture (that Play does not):-
Persistence
One thing it adds, for example, is an API for CQRS-based persistence on top of the persistence support Play currently provides - this (if you do not know) is a pattern which helps you realize a microservices architecture by decoupling queries and commands.
Container Orchestration
Let's say you have a Play application that has 25 different microservices - which is probably a conservative number for say even a relatively small corporate application - how do you manage the deployment/orchestration of all those JVM's? Well containers are all the rage. How do you manage all those containers? ConductR is a tool which takes some of the pain away from that task, and Lagom provides you integration tools for ConductR to make it easier for you to use it with your Lagom project - something you do not get with Play on it's own.
I could still achieve this with Play
Ok, there are loads of SBT modules that you could use in your Play project to help you realise the same thing but then you need to choose what tools you need, figure out which of the many modules available are right for your project, configure and wire them as necessary - this is one of Lagom's goals - to take these decisions and configuration tasks away from you so you can focus on writing your application logic.
If my application was small, maybe just 5 services, then you could argue quite convincingly that you really don't need Lagom (or any other microservice framework for that matter). However, if your application is likely to grow, then Play on it's own will cost you more time in the long run.
There are obviously many more considerations when designing microservices but you get the jist of Play vs Lagom.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
I am working on a project where we are looking to build a REST API from scratch. So my initial thoughts were to utilise one of the relatively new aysnc non-blocking I/O frameworks.
I've been looking at the Typesafe framework (Scala, Akka, Play).
Is this a good solution for a RESTful API?
Are there any good (up to date) tutorials/examples that people could provide links to?
Also, I came across spray.io. From what I can gather this seems to be a good alternative to using Play if all I want to build is a REASTful API?
Is that true?
Any good tutorials for this?
I would really like to use Akka and its Actors model from the outset.
Separately I read that Typesafe acquired spray.io to integrate it into the Typesafe framework.
Has this been done yet? If I download the latest version of Typesafe do I get spray bundled in with it?
Finally, hosting. What would people suggest for hosting such an API with this type of framework? AWS, Google, Cloud Foundry, Heroku?
The service will ultimately need to be extremely scalable.
Does anyone know of any good links that cover a lot of the above? i.e. deploying a spray based API on Cloud Foundry for example?
Thanks for the help.
Your question is far too broad, prone to opinion-based answers. Anyway, my two cents:
I've been looking at the Typesafe framework (Scala, Akka, Play). Is
this a good solution for a RESTful API?
IMHO, the best current solution if you want to both grow your system and sleep in peace.
Are there any good (up to date) tutorials/examples that people could
provide links to?
Plenty of them, all around. Check the following, for example:
http://akka.io/docs/ (the official Akka documentation, which I find great)
https://typesafe.com/activator/templates (templates for almost anything you would want to learn/do, including Akka, Play, Spray, and many more)
https://www.playframework.com/documentation/2.3.x/Home (official Play documentation, in case you want to use it)
Also, I came across spray.io. From what I can gather this seems to be
a good alternative to using Play if all I want to build is a REASTful
API? Is that true? Any good tutorials for this?
If you are planning to do only a RESTful API, yes, seems like a good way to go. For tutorials, see the official site and the activator templates listed above.
Separately I read that Typesafe acquired spray.io to integrate it into
the Typesafe framework. Has this been done yet? If I download the
latest version of Typesafe do I get spray bundled in with it?
As far as I understand, Akka is implementing its very own http module, based on the work of spray.io team. See here for more details. Play will be eventually built upon the Akka streams and http modules. It's still marked as experimental and the process is still going. You can use spray in your project just adding the dependencies. There's no such thing as "latest version of Typesafe", unless you are speaking about the Typesafe Activator, which is the bundle of tools (including Play and Akka and some nice web interface) provided by Typesafe. Normally, what you need will be downloaded automatically. See the examples for more information.
Finally, hosting. What would people suggest for hosting such an API
with this type of framework? AWS, Google, Cloud Foundry, Heroku?
No idea. Until now I have been using AWS successfully, but since I have not tried other options, I can not give opinion on that.
Does anyone know of any good links that cover a lot of the above? i.e.
deploying a spray based API on Cloud Foundry for example?
For Cloud Foundry specifically, I don't know one.
Hope it helped!
P.S.: your question is too broad. Try in the future to present real concrete problems and your attempts to (unsuccessfully) solve it.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking us to recommend or find a tool, library or favorite off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.
Closed 9 years ago.
Improve this question
I am a C# developer which I specialize in Win forms and web application. I work in financial field and most of my experience is connecting to Oracle/sql/sybase and get data and display on the screen.
Recently I taught myself how to develop iPhone / iPad applications. It went very well. Now I want to learn how to connect using web services to my own databases and get data or upload data.
So I need to learn Web services, SOAP, WSDL and whatever else that I need. I don't have any experience in it but if someone can direct me to the right books I will buy them and read them. i want to start writing in my office and connect to my databases and be able to do a proof of concept. Any ideas?
Particularly because you're starting out, I would suggest looking at RESTful services. The API is essentially a URL using HTTP GET, PUT, POST, or DELETE. The output can be XML, JSON, whatever you want. Very simple to construct and test. And because the API is so simple, you don't necessarily need to add another library to your project and increase the code size.
The second chapter of the book iPhone Games Projects talks about how to use a RESTful rankings system to record game scores to a server from an iPhone game.
I don't think you need books for that subject. Try using some API's to make your life easier, and check some code samples. For example:
For JSON, SBJSON is a standard: https://github.com/stig/json-framework/
For XML, have a look at this Apple example: https://developer.apple.com/library/ios/#samplecode/SeismicXML/Introduction/Intro.html#//apple_ref/doc/uid/DTS40007323
An easy way to retrieve info from a online database is to use PHP. You simply call a PHP file located on your server from your iPhone application. The PHP file takes in info you sent it via GET/POST methods (if required), retrieves info from your database, and echo's it in XML/JSON.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 5 months ago.
The community reviewed whether to reopen this question 5 months ago and left it closed:
Original close reason(s) were not resolved
Improve this question
Which of these technology suite for Multi user online game project?
Project requirements:
Able to handle 2k-5k user at any given time.
client library for iphone and android (native, no javascript).
client library for Microsoft Windows (most important), also for mac os x and linux.
Good documentation specially for mod development
Project is not open-source. So, can only use libraries with suitable license.
I can program in erlang and java as well, programming language isn't an issue.
I was looking at following server technologies like Openfire, Tigase, ejabberd and RabbitMQ.
All are good for my project but I want to know more about what suite my needs, AMQP or XMPP.
What AMQP offer specially for real time online games. Is it better option then xmpp?
A key difference between XMPP and AMQP is binary content. AMQP handles binary data just fine, and XMPP seems more designed for XML. Personally for online games I use Google Protocol Buffers for message formatting and parsing, and with their very small binary footprint, I'd be more inclined to use AMQP to deliver those messages.
But do consider what AMQP server you want to use. I've been bitten by using RabbitMQ for my AMQP server in the past. RabbitMQ does not have any flow control facilities, at all. So if your clients are sending messages faster than your server can consume them, buffers on the server can fill up and blow the server up. More recent versions of RabbitMQ implement flow control in an exceedingly coarse way: they halt all consumers in the system until memory clears up.
I've never tried zeromq; perhaps it'd be better for the things I've been using RabbitMQ for...
5K users doesn't tell me much about their behaviour, but if they all submited one request within the same 10 second window then let's say you'd be looking in the 500-1000 requests per second.
I've had Active/MQ running on my relatively low-powered lap-top easily handling 300 requests per second and so I'd happily recommend it here. You can also set up clusters of brokers and achieve horizontal scalability. You can use an http protocol (STOMP) or its native binary protocol. Lots of client API libraries also for C/C++, Java, JavaScript and others. There is some initial AMQP support.
You didn't mention any persistence requirement, but again I would have thought that most RDBMSs would suffice. That said, some of the document oriented and big-table type databases look interesting from a horizontal scaling perspective.
I've also found Apache Camel highly performant and I strongly recommend it. Camel is used to implement your logic layer.