I'm trying to query a Prometheus database to determine how many customers have recorded data for one metric with a specific label filter, but not another metric with a different label filter. I.e. all the customer_id's that show up in
sum(usage{usage_type="type_b"}) by (customer_id)
but not in
count(service_plan{plan_type=~".*plan_b.*"}) by (customer_id)
I could run each and just mash them together outside Prometheus, but I want to do this either in a single query in Prometheus, or with some fancy transformation tricks in Grafana.
You need unless operator - see these docs. The following query should return customer ids, which exist in the first query and are missing in the second query:
sum(usage{usage_type="type_b"}) by (customer_id)
unless
count(service_plan{plan_type=~".*plan_b.*"}) by (customer_id)
Related
I am using Grafana 9.3.1 for monitoring of our system. Among other things, I am trying to monitor the remaining FUP of a phone number for each unit we operate.
Basically, we intend to use two data sources.
Database mapping of the unit ID to its phone number (e.g. "unit_id=123, phone_number="00 123456789")
Prometheus time series remaining_fup{phone_number="00 123456789"}. However, remaining_fup is a 3rd party data and does not include unit_id.
In my unit-detail dashboard I have unit_id variable which indicates which unit FUP should be displayed (among other things depending on unit_id)
My original approach was this:
Create a mixed datasource dashboard
Add database datasource as data A. SELECT phone_number FROM units WHERE unit_id='$unit_id'
Add prometheus datasource remaining_fup and filter it based on A.phone_number: remaining_fup{phone_number="${A.phone_number}"}
Unfortunatelly such use of A isn't supported. I used to hope for applying some transformation like Merge or Join by field and then Filter but with no success. After a lot of googling and trying I feel hopeless.
Could you help please? Is such filter even possible? Thanks!
TL;DR: In grafana dashboard I want to query one datasource in order to obtain a value which I subsequently want to use in another datasource query.
1.) Create variable - name phone_number, type: Query and query your database datasource SELECT phone_number FROM units WHERE unit_id='$unit_id'. You can hide this variable if you don't want it to be visible for the dashboard users.
2.) Variable phone_number may have multiple values, so use advance variable formatting to create valid regex query syntax for your prometheus datasource, e.g.
remaining_fup{phone_number=~"${phone_number:pipe}"}
Of course this queries are just examples and they may need some (syntax) tweaking for the use case. Main idea: don't use 2 queries, but one variable and one query (where you use that variable).
I have managed to create a query to get datasets by tags. For reference I have the following query:
http://localhost/api/3/action/package_search?fq=tags:(my-first-tag%20AND%20my-second-tag)&rows=2
Now, I need to filter these results by the group also. I added a dataset to a group and can see the group as:
http://localhost/api/3/action/group_list
But how can I add the group to the previous query? I can't seem to find anything that works.
You can use the filter query (fq) to search using the group name. For e.g to search the datasets that are in the test group
https://demo.ckan.org/api/3/action/package_search?fq=groups:test-group
To include the two fq, we need to make the query like this:
http://localhost/api/3/action/package_search?fq=tags:(my-tag-1%20AND%20my-tag-2)+groups:my-group-1&rows=2
Now that groupBy is deprecated, how can I mimic a SQL command like SELECT COUNT(*) FROM table GROUP BY xxx using the Waterline ORM ?
This page recommends using .sum() and .avg() but these methods are for number-type columns. Here I want to be able to count the rows of grouped columns, whatever type it is.
I think for specific groupBy query, you've got two choices.
The first one is a 2 step action.
1) Select all the unique element "group by field" you've got in the database.
2) Then count the record for each unique group by field element.
The second one is to use .sendNativeQuery() wich allow you to send a native SQL query to the datastore (you can use this only if you use a real SQL server and not the embedded Sails.JS database)
sendNativeQuery() Documentation
I have a prometheus metric like this:
lines_added{project="xx",user="xx"}
the project label and user label are integer ids, not good for legend on graph.
and I have another variable members as query from postgresql datasource:
select u.id as __value,u.name as __text
from project_authorizations pa left join users u
on pa.user_id = u.id
where pa.project_id=[[project]]
now for the lines_added metric how can I make it display user's real name according "id"->"name" mapping in variable members
Is that possible with grafana?
Would it perhaps work if you stashed the SQL mappings into a variable and then perform some combination of label_replace regex magic on Prometheus query? At least that's how you get mappings into Prometheus world. But from there, I am yet to figure out how to pull it off in bulk. How many IDs you need to map?
I'm confused about this annotation for an entity field that is of type of another entity:
#BatchFetch(value = BatchFetchType.JOIN)
In the docs of EclipseLink for BatchFetch they explain it as following:
For example, consider an object with an EMPLOYEE and PHONE table in
which PHONE has a foreign key to EMPLOYEE. By default, reading a list
of employees' addresses by default requires n queries, for each
employee's address. With batch fetching, you use one query for all the
addresses.
but I'm confused about the meaning of specifying BatchFetchType.JOIN. I mean, doesn't BatchFetch do a join in the moment it retrieves the list of records associated with employee? The records of address/phone type are retrieved using the foreign key, so it is a join itself, right?
The BatchFetch type is an optional parameter, and for join it is said:
JOIN – The original query's selection criteria is joined with the
batch query
what does this means? Isn't the batch query a join itself?
Joining the relationship and returning the referenced data with the main data is a fetch join. So a query that brings in 1 Employee that has 5 phones, results in 5 rows being returned, with the data in Employee being duplicated for reach row. When that is less ideal, say a query over 1000 employees, you resort to a separate batch query for these phone numbers. Such a query would run once to return 1000 employee rows, and then run a second query to return all employee phones needed to build the read in employees.
The three batch query types listed here then determine how this second batch query gets built. These will perform differently based on the data and database tuning.
JOIN - Works much the same away a fetch join would, except it only returns the Phone data.
EXISTS - This causes the DB to execute the initial query on Employees, but uses the data in an Exists subquery to then fetch the Phones.
IN - EclipseLink agregates all the Employee IDs or values used to reference Phones, and uses them to filter Phones directly.
Best way to find out is always to try it out with SQL logging turned on to see what it generates for your mapping and query. Since these are performance options, you should test them out and record the metrics to determine which works best for your application as its dataset grows.