I have log lines that contain a few timestamp fields. Here is an example of log line I am filtering in order to process it:
{
"time": "2022-06-22T10:33:08.710037238Z",
"#version": "1",
"message": "Duration at response processing ends",
"logger_name": "com.mks.cloud.filters.MainFilter",
"thread_name": "reactor-http-epoll-1",
"level": "INFO",
"level_value": 20000,
"rqst_id_ts": "b65c37d9284584e71b1dcd84b6a74075",
"rqst_end_ts": "1655893988698",
"rqst_start_ts": "1655893988698",
"rsp_start_ts": "1655893988709",
"rsp_end_ts": "1655893988709"
}
What I would like to do is calculate a value that would represent the duration between 2 timestamps in the log line so that I can then put the obtained range into quantile_over_time() or any other aggregate function.
For instance, the following works:
quantile_over_time(0.99, {app="myapp"} |~ "rsp_end_ts"
| json
| __error__ = ""
| unwrap rsp_end_ts | __error__="" [5m]) by (tsNs)
However this is not what I want to do since calculating the quantile of epoch timestamps makes no sense. What I want to calculate is the p99 of (rsp_end_ts - rqst_start_ts).
I tried the following which of course doesn't work, but gives an idea as to what I am attempting to do:
quantile_over_time(0.99, {app="myapp"} |~ "rsp_end_ts"
| json
| __error__ = ""
| unwrap (rsp_end_ts - rqst_start_ts) | __error__="" [5m]) by (tsNs)
If somehow there was a way to create a new label like rqst_duration=(rsp_end_ts - rqst_start_ts)
Then the following would be what I am looking for:
quantile_over_time(0.99, {app="myapp"} |~ "rsp_end_ts"
| json
| __error__ = ""
| unwrap rqst_duration | __error__="" [5m]) by (tsNs)
I couldn't find any documentation about this, which is very surprising, I would think (but it seems I might be wrong) this to be a common use case. Any help would be greatly appreciated :).
You can use template functions for that. Here's a sample query on the Grafana playground that you can inspire from.
So your query will look something like:
{app="myapp"} |~ "rsp_end_ts"
| json
} label_format result=`{{sub .rsp_end_ts rqst_start_ts}}` | line_format "{{ .result }}"
Then you can use the result.
Related
I have got a string that represents a query. It begins with a function, and the argument is a dictionary.
"runQuery `syms`columns`fastQuery`exchange!((`AAPL`MSFT`GOOG`AMD);(`sym`price`date);1b;`nasdaq)"
How can I extract the dictionary from the string, and save it in kdb as a dictionary type?
parse the string, to get the parse tree, and then take the param (the dict):
q)eval last parse "runQuery `syms`columns`fastQuery`exchange!((`AAPL`MSFT`GOOG`AMD);(`sym`price`date);1b;`nasdaq)"
syms | `AAPL`MSFT`GOOG`AMD
columns | `sym`price`date
fastQuery| 1b
exchange | `nasdaq
For this example value can have the desired effect:
q)myDict:value {(first where x=" ")_x}"runQuery `syms`columns`fastQuery`exchange!((`AAPL`MSFT`GOOG`AMD);(`sym`price`date);1b;`nasdaq)"
q)myDict
syms | `AAPL`MSFT`GOOG`AMD
columns | `sym`price`date
fastQuery| 1b
exchange | `nasdaq
If you have free rein to (re)define the function then you could just do:
q)runQuery:{x}
q)value"runQuery `syms`columns`fastQuery`exchange!((`AAPL`MSFT`GOOG`AMD);(`sym`price`date);1b;`nasdaq)"
syms | `AAPL`MSFT`GOOG`AMD
columns | `sym`price`date
fastQuery| 1b
exchange | `nasdaq
This could be quite useful if you're replaying a tickerplant-style log using -11!
I mean that in a column there are numbers, [NULL], and empty cell - how to exclude it from the output?
The column looks like that:
|123 |
|23,45 |
| |
|5,67 |
|1,06 |
|[NULL]|
Using expression
SELECT column from scheme.table1...,
how I can get the output like that:
123,
23,45
5,67
1,06
[NULL]
Take care with NULL / empty / 'NULL' values - it's significantly different.
About your task - try to use combine condition, but before - check this dbfiddle
I have a postgresql query that I want to save as .json, just from a especific part of the query result:
SELECT info FROM d.tests where tag like 'HMIZP'
The result of this query is:
{"blabla":{a lot of blabla}, "Body":[{....
I just want everything after "Body" (including " Body")
How can I do it?
You can combine the extraction with building a json
SELECT json_build_object('Body',json_extract_path('{"blabla": { "a": "a lot of blabla"},"Body": [{"a": [1,2]}, {"b":2}]}','Body'))
| json_build_object |
| :--------------------------------- |
| {"Body" : [{"a": [1,2]}, {"b":2}]} |
db<>fiddle here
I am following documentation and thanks to | line_format and regexReplaceAll I was able to fetch some substring from a line.
Let's say now I have those columns:
line
123
7
123
54
14
Having that I want to perform some transform operation, ex sum, or transform operation with grouping by and taking total.
It is not working as I am suspecting those values are not being numbers, but only strings.
Is it possible to convert it to numbers?
I was trying using unwrap but it didn't worked:
sum_over_time(
{service="some"}
|="text expression"
| json
| line_format `{{ regexReplaceAll "text expression to remove from (\\d+)" .label_id "${1}" | trim }}`
| unwrap label_id [1m]
)
it ends up with
pipeline error: 'SampleExtractionErr' for series:
when I am filtering out errors, there is no results.
I'm currently attempting to modify an existing API that interacts with a postgres database. Long story short, it's essentially stores descriptors/metadata to determine where an actual 'asset' (typically this is a file of some sort) is storing on the server's hard disk.
Currently, its possible to 'tag' these 'assets' with any number of undefined key-value pairs (i.e. uploadedBy, addedOn, assetType, etc.) These tags are stored in a separate table with a structure similar to the following:
+---------------+----------------+-------------+
|assetid (text) | tagid(integer) | value(text) |
|---------------+----------------+-------------|
|someStringValue| 1234 | someValue |
|---------------+----------------+-------------|
|aDiffStringKey | 1235 | a username |
|---------------+----------------+-------------|
|aDiffStrKey | 1236 | Nov 5, 1605 |
+---------------+----------------+-------------+
assetid and tagid are foreign keys from other tables. Think of the assetid representing a file and the tagid/value pair is a map of descriptors.
Right now, the API (which is in Java) creates all these key-value pairs as a Map object. This includes things like timestamps/dates. What we'd like to do is to somehow be able to store different types of data for the value in the key-value pair. Or at least, storing it differently within the database, so that if we needed to, we could run queries checking date-ranges and the like on these tags. However, if they're stored as text items in the db, then we'd have to a.) Know that this is actually a date/time/timestamp item, and b.) convert into something that we could actually run such a query on.
There is only 1 idea I could think of thus far, without complete changing changing the layout of the db too much.
It is to expand the assettag table (shown above) to have additional columns for various types (numeric, text, timestamp), allow them to be null, and then on insert, checking the corresponding 'key' to figure out what type of data it really is. However, I can see a lot of problems with that sort of implementation.
Can any PostgreSQL-Ninjas out there offer a suggestion on how to approach this problem? I'm only recently getting thrown back into the deep-end of database interactions, so I admit I'm a bit rusty.
You've basically got two choices:
Option 1: A sparse table
Have one column for each data type, but only use the column that matches that data type you want to store. Of course this leads to most columns being null - a waste of space, but the purists like it because of the strong typing. It's a bit clunky having to check each column for null to figure out which datatype applies. Also, too bad if you actually want to store a null - then you must chose a specific value that "means null" - more clunkiness.
Option 2: Two columns - one for content, one for type
Everything can be expressed as text, so have a text column for the value, and another column (int or text) for the type, so your app code can restore the correct value in the correct type object. Good things are you don't have lots of nulls, but importantly you can easily extend the types to something beyond SQL data types to application classes by storing their value as json and their type as the class name.
I have used option 2 several times in my career and it was always very successful.
Another option, depending on what your doing, could be to just have one value column but store some json around the value...
This could look something like:
{
"type": "datetime",
"value": "2019-05-31 13:51:36"
}
That could even go a step further, using a Json or XML column.
I'm not in any way PostgreSQL ninja, but I think that instead of two columns (one for name and one for type) you could look at hstore data type:
data type for storing sets of key/value pairs within a single
PostgreSQL value. This can be useful in various scenarios, such as
rows with many attributes that are rarely examined, or semi-structured
data. Keys and values are simply text strings.
Of course, you have to check how date/timestamps converting into and from this type and see if it good for you.
You can use 2 different technics:
if you have floating type for every tagid
Define table and ID for every tagid-assetid combination and actual data tables:
maintable:
+---------------+----------------+-----------------+---------------+
|assetid (text) | tagid(integer) | tablename(text) | table_id(int) |
|---------------+----------------+-----------------+---------------|
|someStringValue| 1234 | tablebool | 123 |
|---------------+----------------+-----------------+---------------|
|aDiffStringKey | 1235 | tablefloat | 123 |
|---------------+----------------+-----------------+---------------|
|aDiffStrKey | 1236 | tablestring | 123 |
+---------------+----------------+-----------------+---------------+
tablebool
+-------------+-------------+
| id(integer) | value(bool) |
|-------------+-------------|
| 123 | False |
+-------------+-------------+
tablefloat
+-------------+--------------+
| id(integer) | value(float) |
|-------------+--------------|
| 123 | 12.345 |
+-------------+--------------+
tablestring
+-------------+---------------+
| id(integer) | value(string) |
|-------------+---------------|
| 123 | 'text' |
+-------------+---------------+
In case if every tagid has fixed type
create tagid description table
tag descriptors
+---------------+----------------+-----------------+
|assetid (text) | tagid(integer) | tablename(text) |
|---------------+----------------+-----------------|
|someStringValue| 1234 | tablebool |
|---------------+----------------+-----------------|
|aDiffStringKey | 1235 | tablefloat |
|---------------+----------------+-----------------|
|aDiffStrKey | 1236 | tablestring |
+---------------+----------------+-----------------+
and correspodnding data tables
tablebool
+-------------+----------------+-------------+
| id(integer) | tagid(integer) | value(bool) |
|-------------+----------------+-------------|
| 123 | 1234 | False |
+-------------+----------------+-------------+
tablefloat
+-------------+----------------+--------------+
| id(integer) | tagid(integer) | value(float) |
|-------------+----------------+--------------|
| 123 | 1235 | 12.345 |
+-------------+----------------+--------------+
tablestring
+-------------+----------------+---------------+
| id(integer) | tagid(integer) | value(string) |
|-------------+----------------+---------------|
| 123 | 1236 | 'text' |
+-------------+----------------+---------------+
All this is just for general idea. You should adapt it for your needs.