What is the best (and simplest) way to regularly send data from a database to an IoT device at a certain interval?
In this case I have the data in Google Cloud Datastore, and want to send it to Particle Photons (possibly via Particle Cloud, but not necessarily). But I might also be using other IoT devices and/or other database alternative like Cloud Firestore for instance, in future, so it's great if the solution is easily adoptable to this situation.
Seems like you want some kind of a cron job that takes data from datastore (or any database for that matter) and sends it to your IoT device. Assuming your IoT can be reached via a REST end point, you can use Cloud Scheduler (https://cloud.google.com/scheduler/) and do the logic. The target to the cloud scheduler can be an app engine instance or a cloud function.
Related
This is an architectural question.
I am currently in the process of designing a web application and I am used to a basic: frontend, api, database, microservices setup.
For the sake of saving money and making my architecture a little bit more modern than what I am used to I decided to look into serverless.
The two main parts I am interested in are google cloud functions and firebase. My understanding is that google cloud functions can be fired when a database entry in firebase has been manipulated.
The way I used to communicate between services was through message queues such as RabbitMQ but it seems to me that by using firebase and cloud functions you can build communication through the database without the need for message queues. What I mean by communication in this case, would be that one service would be able to react to the execution of another service by seeing that an entry in the database was changed.
My question therefore is, what are the upsides and downsides of letting all your "communication" between microservices run through firebase instead of message queues, and is this an architecture that is generally used?
AFAIK, cloud function triggers is a beta feature in Firebase, and according to the doc, there are some limitations for firestore trigger events:
It can take up to 10 seconds for a function to respond to changes in Cloud Firestore.
Ordering is not guaranteed. Rapid changes can trigger function invocations in an unexpected order.
Events are delivered at least once, but a single event may result in multiple function invocations. Avoid depending on exactly-once mechanics, and write idempotent functions.
Cloud Firestore triggers for Cloud Functions is available only for Cloud Firestore in Native mode. It is not available for Cloud Firestore in Datastore mode.
The most concerning limitation here is the first one. 10 seconds for an update is a long time if you need that update to be visible to the user.
Another disadvantage I see is that it may run out of control (in terms of system design) as the complexity increases. You may be tempted to add events for everything, and it may be hard to partition them by category, for example (in message queues, you can use topics for that).
Also, according to the doc, cloud functions are rate-limited to 16 invocations per 100 seconds, which may quickly be reached if you got some traffic on your app.
I would use trigger-events for isolated scenarios and use a message queue for the backbone communication between microservices.
I'm trying to do real-time analytics with Azure, and when I have gone through services, I have seen three services provided by Azure are HDInsight(Kafka), Azure stream Analytics, and Azure Events hub what are the services do I have to use.
I'm trying stream data on real-time either from SQL server or from twitter or some other and to store in on Azure Data warehouse or Data Lake.
To answer the high level question, Event Hubs (including Event Hubs Kafka) and HDInsight can be used for data ingestion. The first one provides a serverless service while the second one provides a managed Kafka cluster.
Azure Stream Analytics focuses on data processing, transformation and analytics. You can use a SQL query to do this and take data from Event Hubs (or IoT Hubs) and move it to various sinks such as SQL, SQL Data Warehouse, Data Lake, etc.
To answer your particular question, you can look at this tutorial showing how to use Event Hubs and Stream Analytics to process Twitter data.
Also, you mentioned you want to take data from SQL Server? Is it streaming data? Azure Stream Analytics support data from SQL for reference data (slow moving data used to enrich a stream of data). Are you looking for something to do ETL and move data from SQL server to other places at regular pace? For this Azure Data Factory or SQL Server Integration Services could be a good choice.
Let me know if this answers your question, I'll be happy to give you more info.
Jean-Sébastien
(Azure Stream Analytics)
Hello I actually have a REST api running on Cloud Functions for Firebase using http request, but now I need to sync the data on real time requesting to the functions. I read something about web sockets.
Is there a way to run a web socket on Cloud Functions for Firebase?
This is not going to be a good fit for Cloud Functions. Websockets rely on long-lived connections to the same server over time; Cloud Functions are ephemeral compute instances that are spun down when there's no traffic. There's no way to force or guarantee that a Cloud Function will keep running or hold a connection open indefinitely.
I would encourage you to investigate using the Firebase Realtime Database as a conduit here instead of trying to add realtime to Cloud Functions.
Theoretically you could use two different layers: one to manage the websocket connections and another layer to handle the data processing.
The websocket layer will not be Cloud Functions, but a Docker container running Push Pin in Cloud Run and that’ll route HTTP calls to your Cloud Functions to do the actual data processing.
This is possible using an Apigee Java callout, where the Java (if needed) calls a Cloud Function. See https://cloud.google.com/apigee/docs/api-platform/develop/how-create-java-callout
I have streaming data coming in to a platform that looks like this:
RestAPI -> AWS Kinesis Stream A -> Custom Analytics Engine
Once each datum is processed by the Analytics engine, the result needs to be pushed to web front ends so users can look at it in real time. The web front end is a webapp that connects to a cluster of web servers (spray or play or whatever) via a websocket... and there can be many users interested in the same data.
Question: Since the front end user can connect to any web server in the cluster, how do I get the data back to the cluster in a way where it can be pushed to all the users interested in that particular datum? Do I get the datum to a single server and then somehow distribute it to all of the machines in the cluster? Is there a way for that single datum to go to all servers in the cluster and then each server decides whether there is a connected user that is interested in it and, if not, drops it?
I read through azure documentation and found that the message size limit of Queues is 64 Kb and Service Bus is 256 KB. We are trying to develop an application which will read sensor data from the some devices, call a REST Service and upload it to cloud . This data will be stored in the queues and then dumped in to a Cloud database.
There could be chances that the sensor data collected is more than 256 KB... In such cases what is the recommended approach... Do we need to split the data
in the REST service and then put chunks of data in the queue or is there any other recommended pattern
Any help is appreciated
You have several conflicting technology statements. I will begin by clarifying a few.
Service Bus/IoT Hub are not post calls. A post call would use a
restful service, which exists separately. IoT Hub uses a low
latency message passing system that is abstracted from you. These
are intended to be high volume small packets and fits most IoT
scenarios.
In the situation in which a message is larger than 256 KB (which is very interesting for an IoT scenario, I would be interested to
see why those messages are so large), you should ideally upload to
blob storage. You can still post packets
If you have access to blob storage api's with your devices, you should go that route
If you do not have access to this, you should post big packets to a rest endpoint and cross your fingers it makes it or chop it up.
You can run post analytics on blob storage, I would recommend using the wasb prefix as those containers are Hadoop compliant and you can stand up analytics clusters on top of those storage mechanisms.
You have no real need for a queue that I can immediately see.
You should take a look at the patterns leveraging:
Stream Analytics: https://azure.microsoft.com/en-us/services/stream-analytics/
Azure Data Factory: https://azure.microsoft.com/en-us/services/data-factory/
Your typical ingestion will be: Get your data up into the cloud into super cheap storage as easily as possible and then deal with analytics later using clusters you can stand up and tear down on demand. That cheap storage is typically blob and that analytics cluster is usually some form of Hadoop. Using data factory allows you to pipe your data around as you figure out what you are going to use specific components of it for.
Example of having used HBase as ingestion with cheap blob storage as the underlayment and Azure Machine Learning as part of my analytics solution: http://indiedevspot.com/2015/07/09/powering-azureml-with-hadoop-hbase/