Sensu: best practices to only handle a check after n occurences - sensu

I've recently upgraded from Sensu 0.24 to 1.2 and noticed that checks where triggered immediately to the referenced handlers.
On the old checks (with v0.24), checks had an "occurences" attribute to filter out noise. I only want checks to be handled by handlers on every n occurences, i.e. a http check must fail 5 times before the pagerduty handler will be triggered. This behaviour seems to have changed with the sensu upgrade.
As I understand, a handler is supposed to include a filter to sort out events based on attributes. So suppose this is my check:
{
"checks": {
"examplecom_http": {
"command": "check-http.rb --url https://example.com -s -k -q 'Keyword'",
"handlers": ["default","pagerduty"],
"subscribers": ["network"],
"interval": 60,
"occurrences": 5
}
}
}
In previous versions (or at least that was my understanding), this check would only be handled after 5 minutes (5 occurences for 60 second intervals) of failure. This doesn't work anymore, so now the handler should include a filter to handle occurences:
{
"handlers": {
"pagerduty": {
"type": "pipe",
"command": "/etc/sensu/plugins/pagerduty.rb",
"severities": ["critical"],
"filter": "occurences"
}
}
And the "occurences" filter would look like this:
{
"filters": {
"occurences": {
"attributes": {
"occurrences": "eval: value >= 5"
}
}
}
}
However, whatever comes after the eval part, be it value >= 5 or value < 5, the effect is the same and the pagerduty handler gets executed. I've tried using the negate directive with true and false but it seems my understanding of how filtering and occurences work for checks is just not correct. Maybe checks don't count their occurences at all?
Can somebody help and explain this?

As it stands right now, it looks like occurrences in your handlers JSON block is spelled incorrectly as occurences.
You will want "negate": false the way that your eval is currently written. When negate is false, this means that the handler will only trigger when the filter evaluation is true - negate will simply reverse the result so handlers trigger when the filter evaluation is false.
Finally, if you are looking to trigger the handler "every n occurrences" instead of every occurrence after the 5th, consider this evaluation:
{
"filters": {
"every_five": {
"attributes": {
"occurrences": "eval: value % 5 == 0"
}
}
}
}
This way, when the number of occurrences is divisible by 5, the handler will trigger, but you will need to add the every_five filter to your handler definition. You can change filter to filters and pass an array instead, like so:
{
"handlers": {
"pagerduty": {
"type": "pipe",
"command": "/etc/sensu/plugins/pagerduty.rb",
"severities": ["critical"],
"filters": [
"occurrences",
"every_five"
}
}
}
This will allow you to use the built-in filter "occurrences" as well as your custom filter.

Related

How to edit a value (list of entries) from an api response to use in a request body in Gatling/Scala

I have an issue that I'm hoping someone can help me with. I'm pretty new to coding and Gatling, so I'm not sure how to proceed.
I'm using Gatling (with Scala) to create a performance test scenario that contains two API-calls.
GetInformation
SendInformation
I'm storing some of the values from the GetInformation response so I can use it in the body for the SendInformation request. The problem is that some information from the GetInformation response needs to be edited/removed before it is included in the body for SendInformation.
Extract of the GetInformation response:
{
"parameter": [
{
"name": "ResponseFromGetInfo",
"type": "document",
"total": 3,
"entry": [
{
"fullUrl": "urn:uuid:4ea859d0-daa4-4d2a-8fbc-1571cd7dfdb0",
"resource": {
"resourceType": "Composition"
}
},
{
"fullUrl": "urn:uuid:1b10ed79-333b-4838-93a5-a40d22508f0a",
"resource": {
"resourceType": "Practitioner"
}
},
{
"fullUrl": "urn:uuid:650b8e7a-2cfc-4b0b-a23b-a85d1bf782de",
"resource": {
"resourceType": "Dispense"
}
}
]
}
]
}
What I want is to store the list in "entry" and remove the entries with resourceType = "Dispense" so I can use it in the body for SendInformation.
It would have been ok if the entry list always had the same number of entries and order, but that is not the case. The number of entries can be several hundred and the order of entries varies. The number of entries are equal to the "total" value that is included in the GetInformation response.
I've thought about a few ways to solve it, but now I'm stuck. Some alternatives:
Extract the entire "entry" list using .check(jsonPath("$.parameter[0].entry").saveAs("entryList")) and then iterate through the list to remove the entries with resourceTypes = "Dispense".
But I don't know how to iterate over a value of type io.gatling.core.session.SessionAttribute, or if this is possible. It would have been nice if I could iterate over the entry list and check if parameter[0].entry[0].resourceType = "Dispense", and remove the entry if the statement is true.
I'm also considering If I can use StringBuilder in some way. Maybe if I check one entry at the time using .check(parameter[0].entry[X].resourceType != dispense, and if true then append it to a stringBuilder.
Does someone know how I can do this? Either by one of the alternatives that I listed, or in a different way? All help is appreciated :)
So maybe in the end it will look something like this:
val scn = scenario("getAndSendInformation")
.exec(http("getInformation")
.post("/Information/$getInformation")
.body(ElFileBody("bodies/getInformtion.json"))
// I can save total, så I know the total number of entries in the entry list
.check(jsonPath("$.parameter[0].total").saveAs("total"))
//Store entire entry list
.check(jsonPath("$.parameter[0].entry").saveAs("entryList"))
//Or store all entries separatly and check afterwards who have resourceType = "dispense"? Not sure how to do this..
.check(jsonPath("$.parameter[0].entry[0]").saveAs("entry_0"))
.check(jsonPath("$.parameter[0].entry[1]").saveAs("entry_1"))
//...
.check(jsonPath("$.parameter[0].entry[X]").saveAs("entry_X"))
)
//Alternativ 1
.repeat("${total}", "counter") {
exec(session => {
//Do some magic here
//Check if session("parameter[0]_entry[counter].resourceType") = "Dispense" {
// if yes, remove entry from entry list}
session})}
//Alternativ 2
val entryString = new StringBuilder("")
.repeat("${total}", "counter") {
exec(session => {
//Do some magic here
//Check if session("parameter[0]_entry[counter].resourceType") != "Dispense" {
// if yes, add to StringBuilder}
// entryString.append(session("parameter[0]_entry[counter]").as[String] + ", ")
session})}
.exec(http("sendInformation")
.post("/Information/$sendInformation")
.body(ElFileBody("bodies/sendInformationRequest.json")))
I'm pretty new to coding
I'm using Gatling (with Scala)
Gatling with Java would probably be an easier solution for you.
check(jsonPath("$.parameter[0].entry").saveAs("entryList"))
This is going to capture a String, not a list. In order to be able to iterate, you have to use ofXXX/ofType[], see https://gatling.io/docs/gatling/reference/current/core/check/#jsonpath
Then, in order to generate the next request's body, you could consider a templating engine such as PebbleBody (https://gatling.io/docs/gatling/reference/current/http/request/#pebblestringbody) or indeed use StringBody with a function with a StringBuilder.

Working on historical data (incrementing properties)

I'm trying to write a CEP rule that would take all the existing ACTIVE alarms and increase a specific fragment bike_alarm.priority by 1 every minute. This is the whole structure of alarm:
{
"count": 1,
"creationTime": "2018-07-09T15:30:20.790+02:00",
"time": "2014-03-03T12:03:27.845Z",
"history": {
"auditRecords": [],
"self": "https://cumulocity.com/audit/auditRecords"
},
"id": "1024",
"self": "https://cumulocity.com/alarm/alarms/1024",
"severity": "WARNING",
"source": {
"id": "22022",
"name": "01BIKE_STATION_NORTH",
"self": "https://cumulocity.com/inventory/managedObjects/22022"
},
"status": "ACTIVE",
"text": "Bike disconnected",
"type": "bike_error_01",
"bike_alarm" {
"priority": 10
}
}
This is what I managed to create (based mainly on this question):
create schema Alarm as Alarm;
create schema CollectedAlarms(
alarms List
);
create schema SingleAlarm(
alarm Alarm
);
#Name("Collecting alarms")
insert into CollectedAlarms
select findAllAlarmByTimeBetween(
current_timestamp().minus(100 day).toDate(),
current_timestamp().toDate()
) as alarms
from pattern[every timer:interval(30 sec)];
#Name("Splitting alarms")
insert into SingleAlarm
select
singleAlarm as alarm
from
CollectedAlarms as alarms unidirectional,
CollectedAlarms[alarms#type(Alarm)] as singleAlarm;
#Name("Rising priority")
insert into UpdateAlarm
select
sa.alarm.id as id,
{
"bike_alarm.priority", GetNumber(sa.alarm, "bike_alarm.priority". 0) + 1
} as fragments
from pattern [every sa = SingleAlarm -> (timer:interval(1 minutes))];
the problem is that not all alarms are founded and even those that are the incrementation don't work, priority is set to null.
Additionally, could you point me in direction of some better documentation? Is this something you use?
In general the esper documentation that you linked is the best place to look for the generic syntax.
In combination you probably sometimes also need the Cumulocity documentation for the specific stuff: http://cumulocity.com/guides/event-language/introduction
Coming to your problems:
You are miss-using a realtime processing engine to do cron-like batch operations. While it technically can be done this might not be the best approach and I will show you a different approach that you can take.
But first solving your approach:
The database queries like findAllAlarmByTimeBetween() will only return up to 2000 results and there is no way to get the next page like on the REST API of Cumulocity. Also if you say you want to handle only active alarms you should use a function that also filters for the status.
Getting null out of a function like getNumber() means the JsonPath wasn't found or the dataType is incorrect (using getNumber for a String). You can set a default value for that case as the third parameter. From you Json that you provided it looks correct though. The syntax errors in your code are copy paste errors I assume as otherwise you wouldn't have been able to deploy it.
In my opinion you should approach that differently:
On each incoming alarm raise the priority after one minute if it hasn't been cleared. Additionally trigger the 1 minute timer again. Like a loop until the alarm is cleared.
The pattern for this would like like that:
from pattern [
every e = AlarmCreated(alarm.status = CumulocityAlarmStatuses.ACTIVE)
-> (timer:interval(1 minutes)
and not AlarmUpdated(alarm.status != CumulocityAlarmStatuses.ACTIVE, alarm.id.value = e.alarm.id.value))
];
You need one with AlarmCreated which will only cover the initial increase and a second statement that triggers on your own schema which is then run in a loop.
In general try to avoid as many database calls as you can. So keep the loop counter in your schema so you only always need to execute the update call of the alarm.

Orion subscription when condition on expression is true

My purpose is to make a subscription so that accumulator server from the Orion's test package receives notifications when attribute's value exceeds a treshold. If I am correct this is not implemented on NGSI v1. So on NGSI v2 when I use the above subscription payload with the suitable Service and Subservice Headers
{
"description":"mydescription",
"subject":{
"entities":[
{
"id":"room1",
"type":"room",
"isPattern":"false"
}
],
"condition":{
"attrs":[
"temperature"
],
"expression":{
"q":"temperature>5"
}
}
},
"notification":{
"http":{
"url":"http://myURL:1028/accumulate"
},
"attrs":[
]
},
"expires":"2040-01
-01T14:00:00.00Z"
}
I don't receive any notifications on accumulator server. Without the expression I receive notifications on first place when i make the subscription and also every time that the attribute's value is changed.
Orion version: 1.7.0
I have also tried the solution with noCache for subscriptions on Context Broker.
EDIT: When I query the context for room1
curl -X GET localhost:1026/v2/entities/room1 -H "Fiware-Service: myService" -H "Fiware-ServicePath: /mySubService"
this is the reply from Orion.
{
"id":"room1",
"type":"room",
"TimeInstant":{
"type":"ISO8601",
"value":"2017-05-15T13:33:35.632Z",
"metadata":{
}
},
"temperature":{
"type":"float",
"value":"6",
"metadata":{
"TimeInstant":{
"type":"ISO8601",
"value":"2017-05-15T13:33:35.632Z"
}
}
}
}
Note that your temperature is a string ("6") and not a number. In order the filter to work it should be a number, i.e. you have to get:
{
..
"temperature":{
"type": "float",
"value": 6,
..
}
Note that the NGSIv1 API doesn't allow to create/update attributes with JSON native types other than strings (a more detailed explanation can be found in this presentation, "Native JSON datatypes" slides) If you attemp to create/update temperature as a number using NGSIv1 operations, it will be casted to string.
Thus you have to use the NGSIv2 API (which doesn't have such limitation) in order to create/update attributes with numeric values. For instance, the following request will update temperature value to the (numeric) value 6:
PUT /v2/entities/room1/attrs/temperature
{
"type": "float",
"value": 6
}

How I can set a 'limit' of alerts on sensu?

I wish to set a limit of alerts from sensu. For instance, after 5 alerts from host X about disk capacity, I would like to stop receive alerts about the same issue.
Is it possible?
Thanks
You can do that with sensu filters.
Something like:
{
"filters": {
"firstfive": {
"attributes": {
"occurrences": "eval: value <= 5"
}
}
}
}
You need to add such filter to the handler you have created for the email.

RESTful master/detail

Having 3 dropdown pickers in a web application. The web application uses a Restful service to populate pickers data.
The two first pickers get their values from something like /years and /colors. The third one should get its values depending on the settings of the two.
So it could be something like /models?year=1&color=red.
The question is, how to make this HATEOAS-compliant (so that the dev does not have to know the way he should create an url to get the models).
The root / gets me a number of links, such as:
{
"_links": {
"colors": "/colors",
"years": "/years",
"models": "???" }
}
What should be instead of ???? If there was some kind of template /models?color={color}&year={year}, the dev would have to create the url. Is this OK?
Or there could be a link to list of years on each color got from /colors and then a link to list of models on each year got from /years?color=red, but i'd have to first choose color, then populate years and then populate models. Any idea if i want to have the model dependent on both color and year, not just the year populated from color?
Is it even possible in this situation to make it hateoas-compliant?
I have not heard of HATEOAS before, but based on what I just read about it, it seems that it supposed to return links to where the consumer of the service can go forward in the "state machine".
In your case that would translate to the links being "function calls". The first two (/colors and /years) are functions that take no parameters (and return "something" at this point), while the third is a function call that takes two parameters: one that is a representation of a color, the other a year. For the first two having a simple URL will suffice for the link, but for the third, you need to include the parameter name/type information as well. Something like:
{
"_links": {
"colors": "/colors",
"years": "/years",
"models": {
"url": "/models",
"param1": {"color"}
"param2": {"year"}
}
}
}
Note: you can use the same layout as "models" for "colors" and "years" as well.
At this point the client knows what the URL to access the functions are and what the parameter (if any) names are to be passed to the function.
One more thing is missing: types. Although you could just use "string", it will not be obvious that the "color" parameter is actually a value from what "/colors" returns. You can be introducing a "type" Color that describes a color (and any functions that operate on a color: give a displayable name, HTML color code, etc.)
The "beefed up" signature becomes:
{
"_links": {
"colors": {
"url": "/colors",
"return": "/type/List?type=/type/Color"
},
"years": {
"url": "/years",
"return": "/type/List?type=/type/Integer"
},
"models": {
"url": "/models",
"param1": {
"name": "color",
"type": "/type/Color"
},
"param2": {
"name": "year",
"type": "/type/Integer"
}
"return": "/type/List?type=/type/Model"
}
}
}
Note: the path "/type" is used just to separate the types from functions, but is not necessary.
This will interchangeably and discoverably describe the functions, what parameters they take, and what values they are returning, so you can use the right value at the right place.
Of course implementing this on the service end will not be easy (especially with parameterized types, like "/type/List" -- think Generics in Java or templates in C++), but this is the most "safe" and "portable" way you can describe your interface to your clients.