I hope you can help me.
What I have is some metric data I am collecting. I want to send out an alert, when I reach a specific error-rate on these metrics.
To make clear, my data looks something like this:
Timestamp
value (the runtime of a query)
state (error, success)
api-endpoint called
I have a grafana-Board doing some calculations, drops out something linke this:
error-rate
api-endpoint
number of calls made to the api endpoint
Fine for now - as I can read out on my grafana, I am able to send some error-messages/warnings, if the error-rate is too high. Works like a charm. But now comes the point:
If the first two (e.g.) calls to a specific api fail, I will instantly receive an alarm send by my grafana. I do not wan't that!
Is it possible - and if: how? - to alert me ONLY if this specific request was executed at least 5 times? It is no problem if this is a generic alert like "hey, something is wrong!" - but I need to figure out if the request triggering the alarm with 50-100% error-Rate was at least executed a specific amount of time before alarming.
It has to be done based on tags/fields, I do not want to add a single query for all of my 35+ APIs (number growing).
Any Idea anybody?
Using Grafana 8.0
Using InfluxDb 1.8 (with Flux enabled)
Just to make clear, if everyone ever want's to do the same: FluxQL is king - you can use filter functionality in there and only base data on datasets where value count is greater than X.
Yes: FluxQL is damn hot. I love it since a few month.
Related
As described in the Sensu documentation, I've written a custom check script that returns 0 for OK, 1 for Warning, 2 for Critical, and prints out the description of the status. It shows up as expected on Sensu's built-in web interface, but I'm not sure how to make it show up in Grafana. I have some canned metrics that work through InfluxDB, but this is just a status check, not a metric.
I gather that I need some sort of handler on the Sensu side and/or some sort of datasource on the Grafana side that talks to the Sensu API, but the one for Sensu Core (1.x) doesn't seem to work with the newer Sensu Go (5.x). So, do I:
Rewrite the check to do graphite_plaintext output and use the
influxdb handler?
Write a custom Grafana datasource and/or Sensu handler?
Revert to Sensu Core?
Sensu Go seems to have been re-oriented around metrics, so it's not clear from the docs how to deal with simple checks anymore.
It's probably terribly inefficient, but for now I've simply chosen option 1, to rewrite the check to use influxdb handler.
All I had to do for this was to print my output with the form:
metric_path value timestamp\n
Where metric_path is something like computer_name.topic.status, value is just an integer status, and then timestamp is the current unix time as an integer. That last bit took an embarrassingly long time to figure out...nothing was showing up in InfluxDB database (and thus, Grafana) because the sensu-influxdb-handler errors out if the timestamp is not an integer.
Then, on the Grafana side, I installed the Status Panel plugin developed by Vonage here:
https://grafana.com/plugins/vonage-status-panel
Once the data was finally showing up in InfluxDB, I could select it from Grafana. I set the thresholds for warning and critical to 1 and 2 respectively, and it now works like I wanted. Still, if anyone has a more efficient way to handle this, I'd like to know about it, because I'm going to want to track the status of a large number of things, and I want to do it the right way.
I have a JMeter project with multiple GET and POST requests and assertions for these. I use Aggregate results and View results tree listeners, but none of these can store results on hourly basis. I tried JMeterPlugins-Standard and JMeterPlugins-Extras packages and jp#gc - Graphs Generator listener, but all of them use aggregated data instead of hourly data. So I would like to get number of successful and failed requests/assertions per hour, maybe a bar chart would be most suitable for this purpose.
I'm going to suggest a non-conventional design-level solution: name your samplers dynamically with hour (or date and hour), so that each hour the name will change, and thus they will appear in different category, i.e.:
The code for such name is:
${__time(dd:hh,)} the rest of sampler name
Such sampler will appear in the following way in Aggregate Report (here I simulated it with minutes/seconds, but same will happen with days/hours, just on larger scale):
Pros and cons of such approach:
Simple, you can aggregate anything by hour, minute, or any other time slice while test is running, and not by analysis after execution.
Not listener-dependant, can be used with pretty much any listener or visualizer
If you want to also have overall stats, it will require to sum up every sub-category. So it alters data, but in the way that it can still can be added back to original relatively easy.
Calculating __time before every sampler will not be unnoticed completely from performance perspective, but I don't think it will add visible overhead to a script.
You could get the same data by properly aggregating JTL or CSV (whichever you use) after execution, so it doesn't provide you with anything that is not possible to achieve using standard methods
Script needs altering to make this happen. if you have 100s of samplers, it's going to take a while. And if you want to change back...
You might want to use Filter Results Tool which has --start-offset and --end-offset parameters, you can "cut" your results file into "interesting" pieces and plot them according to your requirements.
You can install Filter Results Tool using JMeter Plugins Manager
Also be aware that according to JMeter Best Practices you should
Use as few Listeners as possible; if using the -l flag as above they can all be deleted or disabled.
Don't use "View Results Tree" or "View Results in Table" listeners during the load test, use them only during scripting phase to debug your scripts.
You can get whatever information you need from the .jtl results file, you can specify test results location via -l command-line argument
To get summarized results per hour add to your test plan Generate Summary Results:
Generates a summary of the test run so far to the log file and/or standard output
Update interval in jmeter.properties to your needs ,1 hour, 3600 seconds:
summariser.interval=3600
You will get summary per hour of your requests.
You can try with Jmeter backend Listener. It has integration with graphite and Influxdb. After storing the results in these time series database you can display the result in Grafana dashboard. Grafana has its own filtering of showing the results in hourly, monthly, daily basis and so on.
I am new to MongoDB and have a table like below
boxname
time_create
box_data
Basically what we are logging here is which box is sending what data and at what time.
Now my requirement is that is to create an alert in system if a box sends more request than a threshold values indicating probably something is wrong or suspicious from that box. I can get the count of records for a box for a time period say 10 mins - but this query is different - each 10 mins check counts for all boxes and check if any exceed more than threshold.
Do I need to do regular polling after 10 mins? Do a program needs to run infinitely to count # and alert - what would be best method to implement the same?
What would be the best mechanism of implementation for such a requirement?
To solve this problem - query scheduler is needed.
That means - every x minutes we need to execute query and check if any box have been matched with threshold criteria.
Scheduler implementation is most based on your current solution architecture (as I am in c# - HangFire will be my selection to implement scheduler).
I am trying to get the current weather conditions for a certain location, but for some reason I always get the conditions for a semi-random day/time (more or less past week). I am using this query: https://query.yahooapis.com/v1/public/yql?q=select%20item.condition%20from%20weather.forecast%20where%20woeid%20%3D%20639660%20AND%20u%3D%22c%22&format=json&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys&callback=
Even trying out the example query on their website doesn't work for me. When I click on "Current conditions for San Diego, CA" I get the same, random results. Is there any way to get the current conditions?
I am seeing the same thing. It has to be on Yahoo's side. Unfortunately it looks like their support page is down too. Returned a 404
I've also notice the random ( 1 day up to 2 wks) of old data being transmitted
http://xml.weather.yahoo.com/forecastrss/55364_f.xml It seems to change about every 30 seconds
We are using the plugin https://goodies.pixabay.com/jquery/tag-editor/demo.html for our AutoComplete feature. We load the source with 3500 items. The performance gets too bad when the user starts typing and the autocomplete loads the filtered result after 6 to 8 seconds.
What are alternate approach that we can take for upto 4000 items for Autocomplete.
Appreciate your response!
are you using the minLength attribute from autocomplete?
on their homepage, they have something like this:
$('#my_textarea').tagEditor({ autocomplete: { 'source': '/url/', minLength: 3 } });
this effectively means, that the user has to enter at least 3 charaters before autocomplete will be used. doing so will usually reduce the amount of results from the autocomplete to a more sane count (like 20-30 maybe).
However, this might not necessarily be your problem. first you should figure out, if it's your server that's got a problem with responding fast (you can use your browser developer toolbar to see how long the requests takes to complete).
If the request takes 6-8 seconds, then you will have to optimize your server's code. On the other hand, if the response is quick, but tageditor needs a long time to build the suggestion list, then the problem is, that it might not be optimized for so many suggestions. in that case, the ultimate solution would be to rewrite the autocompletion module yourself or patch the existing one to better scale to your needs.
Do you go back to the server every time the user types in something to get the matching results?
I am using SPRING ehcache which gets all the items from database and stores in the server cache when the server is started. Whenever the user types the cached data is used which gets the results with few milliseconds. Some one else recommended me to use this.Below is the example for it
http://www.mkyong.com/ehcache/ehcache-hello-world-example/
I am using the jQuery autocomplete features with 2500 items without any issue.
here is the link where it being used http://www.all4sportsonline.com