We have a situation where I want to add a MIB variable label to another query. This another query gives me the value result that I want but I need to add the label from first variable in order to then sort them by what I want (for example just as we do it with instance label).
E.g.
variable1{alert,env, index, instance, ..., labelneeded}
variable2{alert,env, index, instance}
For example there I wanted to get the index of both and somehow add the label-needed by
I tried the following queries but they didnt work after they are giving me both variables with its labels but not concatenated together so my question is if there is a possibility to concatenate them together?
Query example:
max by {index, instance} (variable2 * 5) or max(variable2) by (labelneeded, index, instance)
Thank you in advance :).
In order to explain how to merge labels labels from two metrics, I'll take a common case:
a value metric value{instance="foo",a_label="bar"} 42
an info metric info{instance="foo",version="1.2.3",another="bar"} 1
Info metrics (such as version, compiler, ...) have value 1 such that you can apply operators between the metrics:
value * on(instance) group_left(version) info
Result
{instance="foo",a_label="bar",version="1.2.3"} 42
The parameter(s) of on() keyword specify the criteria(s) for matching the info and the parameter(s) of group_left() operator specify the labels to pull.
It can happens that the metric you want to pull from doesn't have value 1. In that case, you can use the bool modifier with a comparison always true to obtain 1:
other_metric{instance="foo",baz="void"} 0.5555
value * on(instance) group_left(baz) other_metric != bool NaN
Result
{instance="foo",a_label="bar",baz="void"} 42
Related
I have a very simple query:
SELECT * FROM someTable
WHERE instrument = '{instrument}' AND ts >= '{fromTime}' AND ts < '{toTime}'
ORDER BY ts
That query is applied to 3 tables across 2 databases.
I receive a list of rows that have timestamps (ts). I take the last timestamp and it serves as the basis for the 'fromTime' of the next iteration. toTime is usually equal to 'now'.
This allows me to only get new rows at every iteration.
I have about 30 instrument types and I need an update every 1s.
So that's 30 instruments * 3 queries = 90 queries per second.
How can I rewrite the query so that I could use a function like this:
getData table [(instrument, fromTime) list] toTime
and get back some dictionary, in the form:
Dictionary<instrument, MyDataType list>
To use a list of instruments, I could do something like:
WHERE instrument in '{instruments list}'
but this wouldn't help with the various fromTime as there is one value per instrument.
I could take the min of all fromTime values, get the data for all instruments and then filter the results out, but that's wasteful since I could potentially query a lot of data to throw is right after.
What is the right strategy for this?
So there is a single toTime to test against per query, but a different fromTime per instrument.
One solution to group them in a single query would be to pass a list of (instrument, fromTime) couples as a relation.
The query would look like this:
SELECT [columns] FROM someTable
JOIN (VALUES
('{name of instrument1}', '{fromTime for instrument1}'),
('{name of instrument2}', '{fromTime for instrument2}'),
('{name of instrument3}', '{fromTime for instrument3}'),
...
) AS params(instrument, fromTime)
ON someTable.instrument = params.instrument AND someTable.ts >= params.fromTime
WHERE ts < 'toTime';
Depending on your datatypes and what method is used by the client-side driver
to pass parameters, you may have to be explicit about the datatype of
your parameters by casting the first value of the list, as in, for
example:
JOIN (VALUES
('name of instrument1', '{fromTime for instrument1}'::timestamptz),
If you had much more than 30 values, a variant of this query with arrays as parameters (instead of the VALUES clause) could be preferred. The difference if that it would take 3 parameters: 2 arrays + 1 upper bound, instead of N*2+1 parameters. But it depends on the ability of the client-side driver to support Postgres arrays as a datatype, and the ability to pass them as a single value.
I have a Set Variable activity which uses the logic:
#range(int(pipeline().parameters.start),int(pipeline().parameters.end))
It is wierd that I cant find any logic in documents where I can mention a step so that I can generate few numbers as shown below
1,3,5,7,9,...
Is there work around to it, other than introducin a new parameter that is equal to step and generate next number using logic last = last+step.
It is possible to do this using the Filter activity and the range function. Use the range function to generate all numbers and then the Filter condition with mod to get odd numbers, ie
Property
Value
Items
#range(1,10)
Condition
#equals(mod(item(),2),1)
A screenprint of the results:
The other way to do it would be just use a Lookup activity and query a numbers table.
I agree with you that it's a shame range does not have a step argument, and that generally the ADF expression language isn't a bit more fully featured.
I would like to count all tables in the same instance.
I have not used kdb for a while and I forgot how to make this work.
This is what I got:
tablelist:tables[]
{select count i from x} each tablelist
but I got a type error
Your statement doesn't contain a trailing semi colon ; at the end of the first line which will cause an error in an IDE like qpad (assuming you are running it as written).
If not running from an IDE I would check my hdb for any possible missing data and run some sanity checks (i.e can I select from each of my tables normally, do types match across partitions, i is a virtual column representing row count so issues with non-conforming types in your other columns is probably not a cause but investigating may yield the right answer)
One way to achieve what you're trying is (using dummy data):
q){flip select counts:count i,tab:1#x from x}each tablelist:tables[]
counts tab
-------------
5469 depth
3150 quotes
3005 trades
Here I select the count for each table, but also add on the name of the table, flip each result into a dictionary, which results in a list of dictionaries of conforming types and key names which is in fact a table, hence my result. In this way you have a nice way to track what you're actually counting.
Each select query you run is returning a table in the form:
x
-
3
It would be better to use exec as opposed to select to simply return the value of the count e.g:
q){exec count i from x} each tables[]
3 2
Your current method would be attempting to return a list of tables: e.g:
q){select count i from x} each tables[]
+(,`x)!,,3
+(,`x)!,,2
However, the type error makes me think there may be an issue with your tables as this should not error for in-memory tables.
Here's one way
count each `. tables[]
I am using 3.6 2018.05.17 and your expression worked for me. I then change the select to an exec to return just a list of counts.
q){exec count i from x} each tables[]
Below code helps us get the count of each table along with tablename.
q)flip (`table;`msgcount)! flip {x, count value x}#'tables[]
To get only the count and not the tablename along with it.
q){count value x}#'tables[]
I have something like the below in spark of which I'm grouping and then trying to find the one with the highest value from my struct.
test.map(x => tester(x._1, x._2, x._3, x._4, x._5))
.toDS
.select($"ac", $"sk", struct($"num1", struct($"time", $"num1")).as("grp"))
.groupBy($"ac", $"sk")
.agg(max($"grp")).show(false)
I am not sure how the max function figures out how to decide the max. The reason I used a nested struct is because it seemed to make the max function using num1 instead of the next numbers when everything was in the same struct.
The StructTypes are compared lexicographically - field by field, from left to right and all fields have to recursively orderable. So in your case:
It will compare the first element of the struct.
If the elements are not equal it will return the struct with higher value.
Otherwise it will proceed to the point 2.
Since the second field is complex as well, it will repeat procedure from point 1 this time comparing time fields first.
Note that nested num1 can be evaluated on if top level num1 fields are equal, therefore it doesn't affect the ordering in practice.
If I want to compute a value that relies on the previous one (Recursive functions) how can I do it in SPSS ? Example:
Q0 = 0
Qn = Q(n-1) + Constant
If by "... the previous one ..." you mean the value of the same field (or a different field) for the previous record, you can use the #OFFSET(FIELD, EXPR) function.
The function allows you to access values from records other than the current one based on a relative reference.
After many research I couldn't find any way to do recursive function with SPSS Modeler. The only work around is to use R Transform node within SPSS. HTH.
Depending on what you need to do, you can either chain many derive nodes or refer to the previous value in a column after sorting them.
I started with creating a domain context for the stream data flow (iterations) with a simple csv source file with records keeping one field N (range from 1 to 100), just to limit the example. Then I connected this data source with a derive node that defines the variable field Q:
if not(#NULL(#OFFSET(N,1))) then #OFFSET(Q,1) + 2 else 0 endif
Here I used the value 2 for the Constant in the example above. I see this being a recursive function and it relies on the OFFSET just as Kenneth suggested above.