I'm trying to figure out how I can raise a notification when a new value is inserted on my influxDB and send a notification to an HTTP endpoint with the data of the new inserted measurement sample. I'm not sure if it's the goal of Kapacitor (I'm new on the TICK stack) or it's better to use another tool (any suggestion will be welcome).
Thanks in advance.
Best regards,
Albert.
In Kapacitor there is two types of task namely batch and stream. The former is meant for processing historical data and stream is for real time purpose.
Looking at your requirement I guess it is obvious that stream is the way to go as it will enable you to watch data from an influxdb's measurement in real time. For invoking an endpoint in TICK script you can use the HttpPostNode node.
Example (Pseudo code ONLY):
var data = stream
|from()
.database('myInfluxDB')
.retentionPolicy('autogen')
.measurement('measurement_ABCD')
|window()
.period(10s)
.every(10s)
data
|httpPost('http://your.service.url/api/endpoint_xyz')
In this instance the TICK script will watch for new inserted data on measurement, measurement_ABCD for a window period of 10 seconds before doing a HTTP POST to the defined endpoint and this entire process will repeat again every 10 seconds.
That is, you have a moving window of 10 seconds.
Reference:
https://docs.influxdata.com/kapacitor/v1.3/nodes/http_post_node/
Related
I have a materialized view that is updated from many streams. Every one enrich it partially. Order doesn't matter. Updates comes in not specified time. Is following algorithm is a good approach:
Update comes and I check what is stored in materialized view via get(), that this is an initial one so enrich and save.
Second comes and get() shows that partial update exist - add next information
... and I continue with same style
If there is a query/join, object that is stored has a method that shows that the update is not complete isValid() that could be used in KafkaStreams#filter().
Could you share please is this a good plan? Is there any pattern in Kafka streams world that handle this case?
Please advice.
Your plan looks good , you have the general idea, but you'll have to use the lower Kafka Stream API : Processor API.
There is a .transform operator that allow you to access a KeyValueStatestore, inside this operation implementation you are free to decide if you current aggregated value is valid or not.
Therefore send it downstream or returning null waiting for more information.
I have the following challenge I would like to solve preferably in Google Data Fusion:
I have one web service that returns about 30-50 elements describing an invoice in a JSON payload like this:
{
"invoice-services": [
{
"serviceId": "[some-20-digit-string]",
// some other stuff omitted
},
[...]
]
}
For each occurrence of serviceId I then need to call another webservice https://example.com/api/v2/services/{serviceId}/items repeatedly where each serviceId comes from the first call. I am only interested in the data from the second call which is to be persisted into BigQuery. This second service call doesn't support wildcards or any other mechanism to aggregate the items - i.e. if I have 30 serviceId from the first call, I need to call the second webservice 30 times.
I have made the first call work, I have made the second call work with a hard coded serviceId and also the persistence into BigQuery. These calls simply use the Data Fusion HTTP adapter.
However, how can I use the output of the first service in such a way that I issue one webservice call for the second service for each row returned from the first call - effectively looping over all serviceId?
I completely appreciate this is very easy in Python Code, but for maintainability and fit with our environment I would prefer to solve this in Data Fusion or need be any of the other -as-a-Service offerings from Google.
Any help is really appreciated!
J
PS: This is NOT a big data problem -I am looking at about 50 serviceId and maybe 300 items.
I started a OPC UA project using the Milo project to create a OPC UA Client. I am still very new to OPC UA. Right now I am stuck looking for the best practice to read values from several Nodes after a data change of one specific node.
The information model looks like this:
RfidSensorType
On my server i will have several objects of this RfidSensorType. The client creates a subscription on the CurrentAtTag Node to listen for data changes.
My Question:
When the value of CurrentAtTag is changed a callback function will be called in my client which contains the UaMonitoredItem and the DataValue of the CurrentAtTag.
In my application i need to process (at the same time) also the values of Station, IOLPort and CurrentValue which are changed at that moment too.
How can i access those values within the callback from CurrentAtTag?
My only solution is: Using a synchronous read request within that callback
-> Is that an legit approach?
My Research:
1) TriggeringService
I've seen that a TriggerigService exists, which monitors items will send reports only if one specific node changes it values.
Problem: This will call several callsbacks and noz just one..i need all the informations at the same time to further process them..
2) Event Monitoring
In event monitoring one can select "Event fields" which will be returned for each Event notificaiton. I am not sure if i could select the CurrentAtTag, Station, IOLPort and CurrentValue...
Just like you can subscribe to the server's ServerStatus (nodeid "i=2256"), you should be able to subscribe to the nodeid corresponding to 'RfidSensor_Station1'. The server will send PublishResponse with data of type 'RfidSensorType' encoded as an ExtensionObject. The trick is decoding the ExtensionObject.
As Kevin corrected, because 'RfidSensor_Station1' is not node class 'Variable' then it doesn't have a value attribute and you can not monitor the node for data changes. If you are using a PLC, I might combine all properties of the sensor into a string, or byte array. Then I monitor the new variable, and parse the string in the client.
Or you could make ReadRequest as you describe. That will work just fine.
Explanation:
I am able to query the Google Core reporting APIv3 using the client library to get data on pageviews for specific URLs of a website I am working on. I want to get data(pageviews) for each day within a specified range. So far I am simply looping through the range, sending individual request to the API. in each request I am setting the same value for the start date and the end date.
Problem:
Obviously this gets the job done, BUT it is certainly not the best way to go about it. Because, assumming I want to get data for the past 3 months for each of about 2000 URIs. Then I will need 360000 number of requests and that value is well over the limit quota defined by Google.
Potential solution: So one way I thought of solving this issue is probably to send a request setting start-date and end-date to be a week apart but the API will return a sum of the values rather than the individual values.
main question: So is there a way to insist that these values should not be added up and returned as a sum but rather returned (as associative array or something like that) separately for each.
I hope the question is clear and that there is a solution! Thank you!
Very straightforward:
Metric: ga:pageview, Dimension: ga:date, Set a filter for your pagepath, and set a start-date and end-date.
Example:
https://www.googleapis.com/analytics/v3/data/ga?ids=ga%3Axxyyzz&dimensions=ga%3Adate&metrics=ga%3Apageviews&filters=ga%3Apagepath%3D%3D%2Ffaq.html&start-date=2013-06-27&end-date=2013-07-11&max-results=50
This will return the pageviews for that the faq.html& page for each day in the time-frame.
You should check out the QueryExplorer. Great tool to find out how to structure queries.
Basically, I want to implement SYNC functionality; where, if internet connection is not available, data gets stored on local sqlite database. Whenever, internet connection is available, SYNC gets into the action.
Now, Say for example; 5 records are stored locally, and then internet connection is available. I want the server to be updated. So, What I do currently is:
Post first record to the server.
Wait for the success of first request.
Post local NSNotification to routine, that the first record has been updated on server & now second request can go.
The routine fires the second post request on server and so on...
Question: Is this approach right and efficient enough to implement SYNC functionality; OR anything I should change into it ??
NOTE: Records to be SYNC will have no limit in numbers.
Well it depends on the requirements on the data that you save. If it is just for backup then you should be fine.
If the 5 records are somehow dependent on each other and you need to access this data from another device/application you should take care on the server side that either all 5 records are written or none. Otherwise you will have an inconsistent state if only 3 get written.
If other users are also reading / writing those data concurrently on the server then you need to implement some kind of lock on all records before writing and also decide how to handle conflicts when someone attempts to overwrite somebody else changes.