How to run a pivot operation on a table selection within a ticker function in KDB+/Q - kdb

I am trying to run a pivot operation on a selection made within a ticker function as follows:
// Trades table
trades:(
[]time:`timespan$();
sym:`symbol$();
exch:`symbol$();
side:`symbol$();
price:`float$();
size:`float$());
// Generic pivot function
piv:{[t;k;p;v]f:{[v;P]`${raze "_" sv x} each string raze P,'/:v};v:(),v; k:(),k; p:(),p;G:group flip k!(t:.Q.v t)k;F:group flip p!t p;key[G]!flip(C:f[v]P:flip value flip key F)!raze{[i;j;k;x;y]a:count[x]#x 0N;a[y]:x y;b:count[x]#0b;b[y]:1b;c:a i;c[k]:first'[a[j]#'where'[b j]];c}[I[;0];I J;J:where 1<>count'[I:value G]]/:\:[t v;value F]};
// Ticker function that "ticks" every 5 seconds
if[not system"t";system"t 5000";
.z.ts:{
// Selection is made from the trades table with the index reset
trds:0!select first_price:first price, last_price:last price, mean_size:avg size, volume:sum size, min_price:min price, max_price:max price by time:1 xbar time.second, exch, sym from trades;
// Check if selection returns at least one row and run pivot
if[(count trds)>0; (
ptrds:piv[`trds;`time;`exch`sym;`first_price`last_price`mean_size`min_price`max_price`volume];
show ptrds;
)];
}];
However this does not function as the trds reference made in the pivot function is not picked up, presumably because the trds selection is not global, however whilst using trds (without) as a parameter the piv function somehow is not applied and returns the result as if no operation had been performed. In addition, having to create an additional variable with ptrds seems inneficcient and also throws a reference error. Could someone please advise me how I can go about implementing this logic in a canonical manner that is efficient. Thanks

I think the source of your error here is the () brackets in your final if statement. The problem is that, when multiple arguments are enclosed within brackets, the object becomes a list. In kdb, elements of a list are evaluated from right to left, e.g.
q)(a:10;a+15)
'a
[0] (a:10;a+15)
^
q)(a+15;a:10)
25 10
To solve your problem, simply remove the brackets such that it reads
if[(count trades)>0;ptrds:piv[...];show ptrds;]

Related

PL/PGSQL function set upper and lower bound to an integer value

I am working on a pgsql function that calculates a score based on some logic. But one of the requirements is that a parameter after calculation should be in the range [100000, 9900000].
I can't figure out how to do this with existing functions, obviously possible with if conditions, any help?
v_running_sum += (30 - v_calcuated_value)* 100000;
I want v_running_sum to be in the range mentioned above. Is there any way to bound the value of the variable if lower than the lower bound (100,000) to 100,000 and vice versa for the upper bound?
This is how you can easily do this check, using a range:
SELECT 1 <# int4range(100000, 9900000,'[]');
There are many options how to implement this in your logic.
----edit----
When the outcome of a calculation should always be something between 100000 and 9900000, you can use this:
SELECT LEAST(GREATEST(var, 100000), 9900000);
Whatever you stick into "var", the result will always be between these boundaries
If you want a verbose solution use a CASE statement
case when val <= 100000 then 100000
when val >= 9900000 then 9900000
else val end as val

Is there a function in R to create a new column in a tibble that depends on values from a previous row?

First time poster and quite new to R.
I'm trying to add a new variable to a tibble ("joined") that adds value nrow-1 from column 22 ("NurseID"), if the value of the variable in column 3("AccountID") on nrow matches the one on nrow-1.
I can do it with a sorted loop, but this is a large dataset and it takes a long time to run and I wonder if there is a faster/easier way to do this
arrange (joined, AccountID, date_day, shift)
tie <- "."
for (i in 2:nrow(joined))
{
ifelse (joined[i,3]==joined[i-1,3], temp<-joined[i-1,22], temp<-".")
tie <- c(tie,temp)
}
temptie <- as.numeric(tie)
joined <- as_tibble(cbind(joined,temptie))
Any help / input is much appreciated. Please kindly let me know if you need more information on the tibble

KDB: select and round off each row

I created my own function of round off:
.q.rnd:{$[x < 0; -1; 1] * floor abs[x] + 0.5}
I have a table Test with a string column of COL
select "F"$(COL) from Test
24549.18741328
48939.50717263
-274853.33568872
-24549.18741328
298753.62574861
84822.70074144
-7468840.64371524
117944.21228603
-117944.21228603
7468840.64371524
-7468840.64371524
I want to derive a table that would round-off the records in Test
One would think that the statement below would work. But it does not.
select .q.rnd "F"$(COL) from Test
I get the error "type". So how do I round off the records?
The result if the if-else conditional must be an atomic boolean. When you run .q.rnd on a column, you are operating on a list and x<0 is going to return a list of booleans, not an atom. The vector conditional is ?
Nonetheless, it looks like you want a resulting integer/long anyway, so just use parse here
q)t:([]string (10?-1 1)*10?10000f)
q)select "F"$x from t
x
-------------------
4123.1701336801052
-9877.8444156050682
-3867.3530425876379
7267.8099689073861
4046.5459413826466
-8355.0649625249207
6427.3701561614871
-5830.2619284950197
1424.9352994374931
-9149.8820902779698
q)select "j"$"F"$x from t
x
-----
4123
-9878
-3867
7268
4047
-8355
6427
-5830
1425
-9150
To add to what Sean's said, if you wanted to use your function as well you could use each which will apply .q.rnd to each item in the list.
q)select .q.rnd each "F"$x from t
x
-----
-3928
5171
5160
-4067
-1781
3018
-7850
5347
-7112
-4116
but using select "F"$x from t is better as it is vectorised.
q)\t:1000 select "j"$"F"$x from t
22
q)\t:1000 select .q.rnd each "F"$x from t
33
Also it should be noted that the .q namespace isn't necessary and is "reserved for kx use". A lot of the default q functions are in the .q namespace and there's always a chance future kdb updates could add a .q.rnd that has different behaviour and will break any code where you have used your function in.

Strange object value instead of float by using mapReduce in mongodb with Doctrine

I use mongo query for calculating sum price for every item.
My query looks like so
$queryBuilder = new Query\Builder($this, $documentName);
$queryBuilder->field('created')->gte($startDate);
$queryBuilder->field('is_test_value')->notEqual(true);
..........
$queryBuilder->map('function() {emit(this.item, this.price)}');
$queryBuilder->reduce('function(item, valuesPrices) {
return {sum: Array.sum(valuesPrices)}
}');
And this works, no problem. But I found that in some cases (approximately 20 cases from 200 results) I have strange result in field sum - instead of sum value I see construction like
[objectObject]444444444444444
4 - is price for item.
I tried to replace reduce block to block like this:
var sum = 0;
for (var i = 0; i < valuesPrices.length; i++) {
sum += parseFloat(valuesPrices[i]);
}
return {sum: sum}
In that case I see NAN value.
I suspected that some data in field price was inserted incorrectly (not as float, but as string, object etc). I tried execute my query from mongo cli and I see that all price values are integer.
It's not "strange" at all. You "broke the rules" and now you are paying for it.
"MongoDB can invoke the reduce function more than once for the same key. In this case, the previous output from the reduce function for that key will become one of the input values to the next reduce function invocation for that key."
The primary rule of mapReduce (as cited ) is that you must return exactly the same structure from the "reducer" as you do from the "mapper". This is because the "reducer" can actually run several times for the same "key". This is how mapReduce processes large lists.
You fix this by just returning a singular value, just like you did in the emit:
return Array.sum(values);
And then there will not be a problem. Adding an object key to that makes the data inconsistent, and thus you get an error when the "reduced" result gets fed back into the "reducer" again.

Function equivalent to SUM() for multiplication in SQL Reporting

I'm looking for a function or solution to the following:
For the chart in SQL Reporting i need to multiply values from a Column A. For summation i would use =SUM(COLUMN_A) for the chart. But what can i use for multiplication - i was not able to find a solution so far?
Currently i am calculating the value of the stacked column as following:
=ROUND(SUM(Fields!Value_Is.Value)/SUM(Fields!StartValue.Value),3)
Instead of SUM i need something to multiply the values.
Something like that:
=ROUND(MULTIPLY(Fields!Value_Is.Value)/MULTIPLY(Fields!StartValue.Value),3)
EDIT #1
Okay tried to get this thing running.
The expression for the chart looks like this:
=Exp(Sum(Log(IIf(Fields!Menge_Ist.Value = 0, 10^-306, Fields!Menge_Ist.Value)))) / Exp(Sum(Log(IIf(Fields!Startmenge.Value = 0, 10^-306, Fields!Startmenge.Value))))
If i calculate my 'needs' manually i have to get the following result:
In my SQL Report i get the following result:
To make it easier, these are the raw values:
and you have the possibility to group the chart by CW, CQ or CY
(The values from the first pictures are aggregated Sum values from the raw values by FertStufe)
EDIT #2
Tried your expression, which results in this:
Just to make it clear:
The values in the column
=Value_IS / Start_Value
in the first picture are multiplied against each other
0,9947 x 1,0000 x 0,59401 = 0,58573
Diffusion Calenderweek 44 Sums
Startvalue: 1900,00 Value Is: 1890,00 == yield:0,99474
Waffer unbestrahlt Calenderweek 44 Sums
Startvalue: 620,00 Value Is: 620,00 == yield 1,0000
Pellet Calenderweek 44 Sums
Startvalue: 271,00 Value Is: 160,00 == yield 0,59041
yield Diffusion x yield Wafer x yield Pellet = needed Value in chart = 0,58730
EDIT #3
The raw values look like this:
The chart ist grouped - like in the image - on these fields
CY (Calendar year), CM (Calendar month), CW (Calendar week)
You can download the data as xls here:
https://www.dropbox.com/s/g0yrzo3330adgem/2013-01-17_data.xls
The expression i use (copy / past from the edit window)
=Exp(Sum(Log(Fields!Menge_Ist.Value / Fields!Startmenge.Value)))
I've exported the whole report result to excel, you can get it here:
https://www.dropbox.com/s/uogdh9ac2onuqh6/2013-01-17_report.xls
it's actually a workaround. But I am pretty sure is the only solution for this infamous problem :D
This is how I did:
Exp(∑(Log(X))), so what you should do is:
Exp(Sum(Log(Fields!YourField.Value)))
Who said math was worth nothing? =D
EDIT:
Corrected the formula.
By the way, it's tested.
Addressing Ian's concern:
Exp(Sum(Log(IIf(Fields!YourField.Value = 0, 10^-306, Fields!YourField.Value))))
The idea is change 0 with a very small number. Just an idea.
EDIT:
Based on your updated question this is what you should do:
Exp(Sum(Log(Fields!Value_IS.Value / Fields!Start_Value.Value)))
I just tested the above code and got the result you hoped for.