We are at the beginning of building an IoT cloud platform project. There are certain well known portions to achieve complete IoT platform solution. One of them is real-time rule processing/engine system which is needed to understand that streaming events are matched with any rules defined dynamically by end users with readable format (SQL or Drools if/when/then etc.)
I am so confused because there are lots of products, projects (Storm, Spark, Flink, Drools, Espertech etc.) in internet so, considering we have 3-person development team (a junior, a mid-senior, a senior), what would it be the best choice ?
Choosing one of the streaming projects such as Apache Flink and learn well ?
Choosing one of the complete solution (AWS, Azure etc.)
The BRMS(Business Rule Management System) like Drools is mainly built for quickly adapting changes in business logic and are more matured and stable compared to stream processing engines like Apache Storm, Spark Streaming, and Flink. Stream processing engines are built for high throughput and low latency. The BRMS may not be suitable to serve hundreds of millions of events in IOT scenarios and may be difficult to deal with event-time-based window calculations.
All these solutions can be used in Iaas providers. In AWS you may also want to take a look at AWS EMR and Kinesis/Kinesis Analytics.
Some use cases I've seen.
Stream data directly to FlinkCEP.
Use rule engines to do fast response with low latency, at the same time stream data to Spark for analysis and machine learning.
You can also run Drools in Spark and Flink to hot-deploy user-defined rules.
Disclaimer, I work for them. But, you should check out Losant. It's developer friendly and it's super easy to get started. We also have a workflow engine, where you can build custom logic/rules for your application.
check out the Waylay rules engine built specifically for real-time IoT data streams.
In the beginning phase Go for the cloud based IoT platform like predix,AWA,SAP or Watson for rapid product development and initial learning.
Related
On Db2 v11.1, how do we get or setup the notification for DBA team if there is any hang or slowness situation in off shift working hours?
The answer depends on the external monitoring and alerting solution you deployed, and how you configure that tooling in your environment.
This application layer tooling is not built into Db2-LUW, although APIs exist in Db2-LUW for such tooling to get the data it needs in order to operate.
IBM and several third parties offer solutions for real time monitoring and alerting in this space. Many cover app-servers, web-servers, database layers, networks and operating-system layers and have different alerting configurability. Many have plugin type architecture with plugins for Db2-LUW monitoring. Do not use stackoverflow for product recommendations however.
For "slowness", this is only meaningful to measure usually at the application layer, in terms of response times and other metrics etc.
For database-hangs, IBM offers a db2-hang_detect script that tooling can orchestrate , requires careful interpretation and even more careful testing.
I need to develop a Scala based application that will write\read to\from managed AWS Redis at verry high rate. On official Redis page they mention several clients, without comparission. For my project every microsecond matters. I saw similar questions here, on SO, but they all are outdated.
Please advice what client has better performance.
As another pointed out, you can use Jedis: https://github.com/xetorthio/jedis/blob/master/src/main/java/redis/clients/jedis/JedisPool.java
The latency may depend more on requesting within the same AZ/VPC (avoiding external networks) and using Redis pipelines, which batch together transactions and reduce number of requests. See pipeline usage examples here:
https://github.com/xetorthio/jedis/wiki/AdvancedUsage
Here is another example combining AWS client libraries with Jedis:
https://github.com/fishercoder1534/AmazonElastiCacheExample/blob/master/src/main/java/AmazonElastiCacheExample.java
I'm interested in deploying an enterprise service bus on a fault tolerant system with a variety of use cases that include tracking network traffic and analyzing social media data. I'd prefer to use a streaming application, but open to the idea of micro-batching. The solution will need to be able to take imports and exports from a variety of sources (bus).
I have been researching various types of stream-processing software platforms here:
https://thenewstack.io/apache-streaming-projects-exploratory-guide/
But I've been struggling with the fact that many (all) of these projects are open source and I don't like the large implementation risk.
I have found Apache Kafka attractive because of the Confluent Platform built on-top, but I can't seem to find anything similar to Confluent out there and want to know if there are any direct competitors built on top of another Apache project. Or an entirely private solution.
Any help would be appreciated! Thanks in advance!
I know theres already a good question on this, but it doesn't really answer what I'm looking for.
From what I understand:
1.both are used as a central focal point between applications
2.both can use routing/mediation/transformation etc. between services/apps
But the only difference i can really see is that hub and spoke typically have many different formats entering the hub(SOAP/REST/XML/JSON...) while ESB typically has a standard format(Usually just SOAP.)
Also I keep reading that hub and spoke introduces a single point of failure compared to an ESB. So is the physical deployment the difference here? Where a hub has every possible endpoint and as ESB has endpoints deployed across multiple hubs? So an ESB is just multiple hubs(for want of better words)?
Can anyone help clear this up for me?
There is no exact answer here, since you can talk about ESB as a specific design pattern, or as the discourse about the evolution of software integration tools and SOA.
ESB as a design pattern means that you manage communication between different services using a bus where clients can easily plug in and out. This is usually done by forcing them to use standard data formats and protocols, whereas with Hub and Spoke you might use custom connectors and data transformations for each client. This limits the number of problems you may have with running multiple integrations, but you may still have a single point of failure in ESB.
ESB as a discourse (or marketing term) is a more complex issue, where people argue over what is "True ESB". Some people say you need to have a modular architecture where you can select which components you deploy, or you need to be able to distribute the components across different machines to allow scaling and fault tolerance. In the extreme definition you would need to deploy even your data transformers as distributed services.
From Here
The ESB is the next generation of enterprise integration technology, taking over where EAI(hub-spoke) leaves off.
Smarter Endpoints : The ESB enables architectures in which more intelligence is placed at the point
where the application interfaces with the outside world. The ESB allows each endpoint to present
itself as a service using standards such as WSDL and obviates the need for a unique interface written
for each application. Integration intelligence can be deployed natively on the end-points (clients and
servers) themselves. Canonical formats are bypassed in favor of directly formatting the payload to
the targeted format. This approach effectively removes much of the complexity inherent in EAI
products.
Distributed Architecture : Where EAI is a purely hub and spoke approach, ESB is a lightweight
distributed architecture. A centralized hub made sense when each interaction among programs had
to be converted to a canonical format. An ESB, distributes much more of the
processing logic to the end points.
No integration stacks : As customers used EAI products to solve more problems, each vendor added
stacks of proprietary features wedded to the EAI product. Over time these integration stacks got
monolithic and require deep expertise to use. ESBs, in contrast, are a relatively thin layer of software
to which other processing layers can be applied using open standards. For example, if an ESB user
wants to deploy a particular business process management tool, it can be easily integrated with the
ESB using industry standard interfaces such as BPEL for coordinating business processes.
The immediate short-term advantage of the ESB approach is that it achieves the same overall effect
as the EAI(hub-spoke) approach, but at a much lower total-cost-of-ownership. These savings are realized not
only through reduced hardware and software expenses, but also via labor savings that are realized by
using a framework that is distributed and flexible.
I don't know if you mean this when you say is physical deployment the difference here? but actually the main difference between Hubs and ESP is that, its communication system is in different Layer.
When we talking about an ESP we are reffering to a software architecture model where as a hub is reffering to a strict hardware connecting topology.
Profiously this hardware topology, (a collection of hubs) implements an ESP, but there is a distinct line in communication layers between the two.
I found this guide in the google documentation for the cloud platform.
https://cloud.google.com/developers/articles/high-availability-lamp-stack-on-google-compute-engine
although the files it asks you to download, are not found when the link is clicked?
anyone know where they actually are?
Unfortunately we deprecated this sample application as it was focused on migrating applications around maintenance windows.
Now that we have live migration and transparent maintenance windows, developers no longer need to move components of their application around zones due to impeding maintenance. Additionally, Cloud SQL now supports MySQL wire protocol which significantly reduces the complexities of managing applications for high availability.
In the future we may develop a new application, but it will be greatly simplified since we can offload load balancing to Compute Engine load balancer and persistency to Cloud Datastore and Cloud SQL.