I saw a technique to access directory elements by its symbol rather then its name (see q.k):
`.q `svar`sdev`scov`med / instead of .q `svar`sdev`scov`med
Why and when this approach is useful?
Also for some reason the behaviour is reversed compared to the # apply:
q)l:til 5
q)`l[2 3]: 20 30 / 'assign, `l not changed
/ (upd: `l is not just a symbol here, it is exactly refers to a list: `l[2 3] gets l elements)
q)l[2 3]: 21 31; l / l changed
0 1 21 31 4
But when we use # apply syntax, the result is reversed:
q)#[l;2 3;:;22 32]; l / l not changed
0 1 21 31 4
q)#[`l;2 3;:;23 33]; l / `l changed
0 1 23 33 4
upd:
Applying indexes to a symbol does no work in shakti, looks like this idea hadn't withstand the test of time.
By directory, I think you mean dictionary as this is the data structure which is returned when you call .q.
You can access dictionary elements a number of ways:
q)d:`a`b`c!1 2 3
q)d
a| 1
b| 2
c| 3
q)d`a
1
q)`d `a
1
q)d[`a]
1
q)#[d;`a]
1
q)#[`d;`a]
1
q) / etc ...
which are all syntactic sugar for each other, it just depends which you prefer (or what the situation dictates is better).
In the code below,
q)`l[2 3]:20 30
'assign
[0] `l[2 3]:20 30
`l is simply the symbol `l, not a reference to the list l, which is why you get an assign error.
The # operator is slightly different,
q)#[l;0;:;20]
20 1 2 3 4
q)l
0 1 2 3 4
q) / -vs-
q)#[`l;0;:;20]
`l
q)l
20 1 2 3 4
adding the backtick to l is telling q that you want to update l not just apply the operation to the list and return the result.
Related
I want to maintain a list L. So I want to write a function func[] that always take the last table of list L, work on it, return a value that appends to the end of L.
For example:
Let L be [a,b,c]
func() will take c, compute d = func[c]
Now L becomes [a,b,c,d]
func() will take d, compute e = func[d]
Now L becomes [a,b,c,d,e]
Et cetera.
How can I implement this in kdb/q?
Do or while should allow you to achieve this e.g.
q)l:0 2;f:{x,2*last x}
q)
q)5 f/ l // do op 5 times
0 2 4 8 16 32 64
q)
q)(64>last#) f/ l // perform op while last item in list is less than 64
0 2 4 8 16 32 64
I think the over function is what you need.
https://code.kx.com/q/ref/over/
q)l:1 2 3
q)f:{if[100<count x;:x];x,1+last x}
q)over[f;l]
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29..
I added a condition to break the loop in f. When using over if the same result is returned twice it will stop.
You could also use .z.s. This will repeat the current function with new input.
https://code.kx.com/q/ref/dotz/#zs-self
Adding on to #CWD's answer, you could do this with explicit recursion using .z.s like so:
func: {[x;f;threshold] $[threshold <= count x;x;.z.s [x,enlist f last x;f;threshold]]}
func[1 2 3;{x*10};5} /returns 1 2 3 30 300
This will recursively apply your function to the last element until the total count of the list hits your specified threshold. Of course, set the threshold too high, and you will end up with a stack error. As such, it is preferable to use the standard iterators in production-grade code.
In q/kdb, we can apply a function to a number of arguments, as below:
f each (1;2;3)
We can also apply a defined argument to a list of functions:
flist: (f1:{x+y+z},f2:{x+y-z},f3:{x-y+z});
flist .\: 1 2 3
What is the most efficient way to combine both of these- to apply every function in a list to each value in a list as parameters. For example, to apply 3 unary functions, f1, f2 and f3, to a list containing values 1, 2 and 3 (producing 9 calls).
Any help with this is much appreciated!
You can use the eachboth (') operator:
q)f1:1+;f2:2+;f3:3+
q)(f1;f2;f3) #' 10 20 30
11 22 33
or in the case of multi-argument functions,
q)g1:+;g2:-;g3:*
q)(g1;g2;g3) .' (2 1;3 2;2 2)
3 1 4
and if you want to apply each function to each value, you need to form a cross product first:
q)(#/)each(f1;f2;f3) cross 10 20 30
11 21 31 12 22 32 13 23 33
You can use the unary apply-at # (since you are dealing with unary functions), in combination with each-left & each-right. For example:
q)({x+1};{neg x};{x*x}) #\:/: (1 2 3)
2 -1 1
3 -2 4
4 -3 9
I'm playing around with an example on https://code.kx.com/q/ref/amend/#cross-sections
$ q
KDB+ 3.6 2019.04.02 Copyright (C) 1993-2019 Kx Systems
q)d:((1 2 3;4 5 6 7);(8 9;10;11 12);(13 14;15 16 17 18;19 20))
q)i:(2 0; 0 1 0)
q)y:(100 200 300; 400 500 600)
q)r:.[d; i; ,; y]
It all works fine except if I'll try to reference a single element d[1;1]:
q)i:(1 0; 0 1 0)
q)r:.[d; i; ,; y]
'type
[0] r:.[d; i; ,; y]
But if I use join , for lists and for just single element it works as supposed:
q)10,200
10 200
q)10,((),200)
10 200
So why the amend operation breaks on this simple join?
Upd:
One more example:
q)#[(1; 2; 3);1;,;10]
'type
[0] #[(1; 2; 3);1;,;10]
^
but it's ok if one use lists:
q)#[(1; (),2; 3);1;,;10]
1
2 10
3
The answer lies in examining the data you are extracting to join, here the operator -3! is your friend to help reveal the actual structure through multiple layers of enlistment
q)d:((1 2 3;4 5 6 7);(8 9;10;11 12);(13 14;15 16 17 18;19 20))
q)i1:(2 0; 0 1 0)
q)i2:(1 0; 0 1 0)
q)y:(100 200 300; 400 500 600)
q)-3!r1:.[d; i1]
"((13 14;15 16 17 18;13 14);(1 2 3;4 5 6 7;1 2 3))"
q)q){type each x} each .[d; i1]
7 7 7
7 7 7
q)-3!r2:.[d; i2]
"((8 9;10;8 9);(1 2 3;4 5 6 7;1 2 3))"
q){type each x} each .[d; i2]
7 -7 7
7 7 7
No here we can see that in the first case, each element of r1 is a list of lists, but for r2 the first element is 2 lists of longs with an atomic long 10.
From the cross section documentation
The shape of y is 2 3, the same shape as the cross-section selected by d . i
i.e., Shape should be the counts and types matching, the type of each item of y is a 7h, that should match the type of each selection from d.
Essentially when you are using the amendment operators , and # it will be expecting conformity as it is using an amend in place. a:1;a,:1 2 3 will also fail.
We can confirm this with your other examples
q)type #[(1; (),2; 3);1]
7h
Changing that example to adjust the first element
q)#[(1; (),2; 3);0;,;10]
'type
The reason that just using the , operator as 10,((),200) didn't cause any errors, is because you are using it outside the amend overloads, within the amend overload , is expecting to be working with matching shape. When used directly it can promote and adjust shape.
The only explanation I can offer is that kdb+ tries to use amend-in-place (also known as assignment through the operator) when it can.
For example, this works:
q)l:(13 14;15 16 17 18;19 20)
q)l[1],:200
q)l
13 14
15 16 17 18 200
19 20
But this doesn't:
q)l:(8 9;10;11 12)
q)l[1],:200
'type
[0] l[1],:200
^
The latter fails because kdb can't substitute the vector 10 200 for the atom 10 -- changing types is not something that amend-in-place is supposed to do.
If you used your own function {x,y} instead of the plus operator the second
expression from your example would work as expected too because kdb will replace existing values with return values of the function (which, in contrast to the built-in ,, is a black box as far as kdb is concerned):
q)i:(1 0; 0 1 0)
q)r:.[d; i; ,; y]
q).[d; ii; {x,y}; y]
(1 2 3 400 600;4 5 6 7 500)
(8 9 100 300;10 200;11 12)
(13 14;15 16 17 18;19 20)
I am trying create 2d buckets in q
Given a 2d grid
5 o---o---o
| | |
3 o---o---o
| | |
0 o---o---o
0 3 5
each node on the grid defines the boundary of 2d buckets for positive integers. For example the center node would contain tuples (x;y) where 3<=x<5 and 3<=y<5. The nine buckets are indexed from 0,...,8.
The way I tried to implemented this in q is
bucketidx:{((0 3 5i) cross (0 3 5i)) bin "i"$(first x;last x)}
To traverse through the buckets:
bucketidx each ((0 3 5i) cross (0 3 5i))
/0j, 1j, 2j, 3j, 4j, 5j, 6j, 7j, 8j
However I get a strange behavior on bucketidx 6 0. I expect this to be in the upper left node
(5<=y) and (x=0)
but it returns index 8 which would be the upper right node. I hope it is clear what I am trying to do.
Thanks for the help
Thats because of the bin behavior.
Binary search returns the index of the last item in x which is <=y
https://code.kx.com/q/ref/search/#bin-binr
Your list is :
q) a:(0 3 5i) cross (0 3 5i)
q) a / (0 0; 0 3;0 5;3 0; 3 3; 3 5;5 0;5 3; 5 5)
You are searching (6 0) in this list using bin function and last item in that list which is <=(6;0) is (5;5) and index of that item is 8.
q) a bin 6 0 / 8
thats the reason you are getting 8.
I think 'tuple with bin' approach is not the right way to go for this problem.
You could use something similar to below idea. First argument to function is X coordinate and second is Y coordinate.
q) node:{b:0 3 5;(b bin x)+3*b bin y}
q) node[0;6] / 6
Another approach is to use a dictionary with sorted attribute, which makes it a step function.
q)d:`s#0 3 5!0 1 2
q)3 sv' d#(0 3 5i) cross (0 3 5i)
0 1 2 3 4 5 6 7 8
q)3 sv' d#enlist 6 0
,6
I have two lists:
l1:`a`b`c;
l2: til 20;
I am trying to create a dictionary 'd' that contains the elements of 'l1' as key and the elements of 'l2' evenly distributed over it. So like this:
d:(`a`b`c)!(0j, 3j, 6j, 9j, 12j, 15j, 18j;1j, 4j, 7j, 10j, 13j, 16j, 19j;2j, 5j, 8j, 11j, 14j, 17j)
The order of the elements is not relevant, I just need them balanced. I was able to achieve that in an iterative way (happy to add the code, if that's considered helpful), but there must be a more elegant way (potentially with adverbs?).
It can be done using the group :
q)group (count[l2]#l1)
(`a`b`c)!(0j, 3j, 6j, 9j, 12j, 15j, 18j;1j, 4j, 7j, 10j, 13j, 16j, 19j;2j, 5j, 8j, 11j, 14j, 17j)
If your l2 is something else instead of til 20 , then you have to lookup the items back after grouping :
q)l2: 20#.Q.a
q)l2
"abcdefghijklmnopqrst"
q)l2 group (count[l2]#l1) // lookup the items back from l2 after grouping
(`a`b`c)!("adgjmps";"behknqt";"cfilor")
You can use the reshape functionality of the take operator #. It takes two arguments: a LHS of at least 2 dimensions and the list to reshape.
For example (3;4)#til 12 will reshape the list 0 1 ... 12 into a 3 by 4 matrix
In our case, the number of the number of elements in l1 will will not necessary divide exactly into the number of elements in l2 (we don't want a rectangular matrix). Instead we can supply a null as the second dimension which will take care of distributing the remainders.
q) l1!(count[l1];0N)#l2
a| 0 1 2 3 4 5
b| 6 7 8 9 10 11 12
c| 13 14 15 16 17 18 19
This method performs very well for larger input lists.
As a side note, when using .Q.fc to split a vector argument over n slaves for multi-threading, kdb uses the # operator to reshape the vector into n vectors, one for each slave.
q)d:`a`b`c!{a where x = (a:til 20) mod y}'[til 3;3]
q)d
a| 0 3 6 9 12 15 18
b| 1 4 7 10 13 16 19
c| 2 5 8 11 14 17