Table transformation, table as list of dicts - kdb

Please help me with t1 transformation through the following path:
t1:enlist `a`b!1 2;
t2:exec val from ([]val:t1);
-3!t1 // +`a`b!(,1;,2)
-3!t2 // ,`a`b!1 2
t1~t2[;] // 1b
I expect that the 2nd line (exec) returns the same object as t1, but it is not. For some reason only [;] gets t1 from t2.
So what (and why) happens on lines 2 and 5?
UPD
Why the enlist is so fruitful? It makes enlist for each element, and also flips the entire object
-3!enlist `a`b!1 2 // +`a`b!(,1;,2)
-3!enlist each `a`b!1 2 // `a`b!(,1;,2)
-3!flip enlist each `a`b!1 2 // +`a`b!(,1;,2)

I think what's happening here is that the special promotion of lists of (conforming) dictionaries to a table is essentially "undone" when it becomes the column of a table. It goes back to being lists of dictionaries as if they were non-conforming, though misleadingly the terminal still displays it as if it was a table. To see this, consider this example:
/promoted to table
q)-3!(`a`b!4 5;`a`b!1 2)
"+`a`b!(4 1;5 2)"
/still a table (embedded)
q)-3!enlist(`a`b!4 5;`a`b!1 2)
",+`a`b!(4 1;5 2)"
/still a table (embedded)
q)-3!(1#`val)!enlist(`a`b!4 5;`a`b!1 2)
"(,`val)!,+`a`b!(4 1;5 2)"
/no longer a table, now a list of dictionaries
q)-3!flip(1#`val)!enlist(`a`b!4 5;`a`b!1 2)
"+(,`val)!,(`a`b!4 5;`a`b!1 2)"
/extracting the column gives an "un-promoted" list of dictionaries
q)-3!exec val from flip(1#`val)!enlist(`a`b!4 5;`a`b!1 2)
"(`a`b!4 5;`a`b!1 2)"
/even though there are two of them and they both have type 99h, it is not a table
q)count exec val from flip(1#`val)!enlist(`a`b!4 5;`a`b!1 2)
2
q)type each exec val from flip(1#`val)!enlist(`a`b!4 5;`a`b!1 2)
99 99h
q)type exec val from flip(1#`val)!enlist(`a`b!4 5;`a`b!1 2)
0h
/so it is not comparable to something that has the promotion, even though the terminal makes them appear the same
q)((`a`b!4 5;`a`b!1 2))~exec val from flip(1#`val)!enlist(`a`b!4 5;`a`b!1 2)
0b
q)(`a`b!4 5;`a`b!1 2)
a b
---
4 5
1 2
q)exec val from flip(1#`val)!enlist(`a`b!4 5;`a`b!1 2)
a b
---
4 5
1 2
Your case is the same thing but for a single dictionary rather than two, but I think two dictionaries makes it easier to see.

Related

Select from a table with Limit expression works, without - fails

For a table t with a custom field c which is dictionary I could use select with limit expression, but simple select failes:
q)r1: `n`m`k!111b;
q)r2: `n`m`k!000b;
q)t: ([]a:1 2; b:10 20; c:(r1; r2));
q)t
a b c
----------------
1 10 `n`m`k!111b
2 20 `n`m`k!000b
q)select[2] c[`n] from t
x
-
1
0
q)select c[`n] from t
'type
[0] select c[`n] from t
^
Is it a bug, or am I missing something?
Upd:
Why does select [2] c[`n] from t work here?
Since c is a list, it does not support key indexing which is why it has returned a type
You need to index into each element instead of trying to index the column.
q)select c[;`n] from t
x
-
1
0
A list of confirming dictionaries outside of this context is equivalent to a table, so you can index like you were
q)c:(r1;r2)
q)type c
98h
q)c[`n]
10b
I would say that the way complex columns are represented in memory makes this not possible. I suspect that any modification that creates a copy of a subset of the elements will allow column indexing as the copy will be formatted as a table.
One example here is serialising and deserialising the column (not recommended to do this). In the case of select[n] it is selecting a subset of 2 elements
q)type exec c from t
0h
q)type exec -9!-8!c from t
98h
q)exec (-9!-8!c)[`n] from t
10b

Table cols definition

There are a bunch of tricks used in kdb+ to work with keyed/splayed/partitioned and simple tables. I see lots of .Q functions which work as a facade for these varieties. One of them is cols. Could you help me please with one of the cases - what the 11h= case stands for?
cols
k){$[
.Q.qp x:.Q.v x; / If partitioned
.Q.pf,!+x; / add "partitioned field" to table cols
98h=#x; / If simple table
!+x; / just table cols (convert to dict of lists, get keys)
11h=#!x; / ?
!x;
!+0!x / (keys-dict)!(data-dict): remove keys, get table cols
]}
It's for dictionaries:
q)cols`a`b`c!1 2 3
`a`b`c
Where the type (#) of the key (!) is a list of symbols (11h)

kdb/q functional update for "prev 5 mavg col1"

How do i write functional update for below? instead of spelling out each column (of 0, 1, 2, 3 ... ) because there are a lot of them.
table: update pma0: prev 5 mavg fa_0, pma1: prev 5 mavg fa_1, pma2: prev 5 mavg fa_2, pma3: prev 5 mavg fa_3, pma4: prev 5 mavg fa_4 by sym from table
It's particularly difficult for me because "mavg" is not really a "function" I call like like mavg[a;b] and there is another function "prev" ahead of it.
I can't even figure out an easier version of the problem. if I just want to
table: update pma0: prev fa_0, pma1: prev fa_1, pma2: prev fa_2, pma3: prev fa_3, pma4: prev fa_4 by sym from table
below apparently didn't work. It gives me the same values for newCols and origCols.
ttt2: ![table; (); enlist[`ric]!enlist[`ric]; newCols!({prev; x} each origCols)]
You may use functional form similar to following
table: ![table;();(1#`sym)!1#`sym;(`$"pma",/:string til 5)!{(prev;(mavg;5;x))} each `$"fa_",/:string til 5];
Where (`$"pma",/:string til 5) creates list column names. And {(prev;(mavg;5;x))} each `$"fa_",/:string til 5 creates list of corresponding column values in functional form

TorQ: .loader.loadallfiles and referential integrity leads to `cast error

I have a table volatilitysurface and a detail table volatilitysurface_smile as part of the detail table I define a foreign key to the master table i.e.
volatilitysurface::([date:`datetime$(); ccypair:`symbol$()] atm_convention:`symbol$(); ...);
volatilitysurface_smile::([...] volatilitysurface:`volatilitysurface$(); ...);
When I try using AquaQ's TorQ .loader.loadallfiles to load the detail table volatilitysurface_smile I need as part of the "dataprocessfunc" function to dynamically build the foreign key field i.e.
rawdatadir:hsym `$("" sv (getenv[`KDBRAWDATA]; "volatilitysurface_smile"));
.loader.loadallfiles[`headers`types`separator`tablename`dbdir`partitioncol`partitiontype`dataprocessfunc!(`x`ccypair...;"ZS...";enlist ",";`volatilitysurface_smile;target;`date;`month;{[p;t] select date,ccypair,volatilitysurface,... from update date:x,volatilitysurface:`volatilitysurface$(x,'ccypair) from t}); rawdatadir];
Note the part:
update date:x,volatilitysurface:`volatilitysurface$(x,'ccypair) from t
The cast error is pointing to the construction of the volatilitysurface key. However, this works outside .loader.loadallfiles and the tables are globally :: and fully defined before calling the .loader.loadallfiles function.
Any ideas how to deal with this use-case? If the detail table foreign key is not initialized then the insertion will fail.
The error may be due to the scoping in the update. As you are running the cast/update within the .loader namespace the tablename would need to be full scoped (`..volatilitysurface).
eg. update date:x,volatilitysurface:`..volatilitysurface$(x,'ccypair) from t
Regards,
Scott
Are you sure that all possible x & ccypair combinations are in the volatilitysurface table? The 'cast error would seem to suggest this is not the case e.g.
q)t:([a:1 2 3;b:`a`b`c] c:"ghi")
q)update t:`t$(a,'b) from ([] a:2 3 1;b:`b`c`a)
a b t
-----
2 b 1
3 c 2
1 a 0
q)update t:`t$(a,'b) from ([] a:2 3 1 5;b:`b`c`a`d)
'cast
[0] update t:`t$(a,'b) from ([] a:2 3 1 5;b:`b`c`a`d)
^
Note in the second case I have the a-b pair of (5;`d), which isn't present in the table t, and so I get the 'cast error
You can determine if there are missing keys, and which they are, like so:
q)all (exec (a,'b) from ([] a:2 3 1;b:`b`c`a)) in key t //check for presence, all present
1b
q)all (exec (a,'b) from ([] a:2 3 1 5;b:`b`c`a`d)) in key t //check for presence, not all present
0b
q)k where not (k:exec (a,'b) from ([] a:2 3 1 5;b:`b`c`a`d)) in key t //check which keys AREN'T present
5 `d
If this is the case, I guess you kind of have two options:
Make sure the volatilitysurface table is loaded correctly - assuming you have full data coverage in your files, presumably every possible key should be present in this table
If there is the possibility of possibly keys not being present in the volatilitysurface table, you could perhaps add dummy records to it before making the foreign key (which could be replaced if an actual record comes in later
The second option could perhaps work something like this:
q.test){if[count k:k where not (k:exec (a,'b) from x) in key `..t;#[`..t;;:;value[`..t](0N;`)]'[k]];update t:`t$(a,'b) from x}([] a:2 3 1;b:`b`c`a)
a b t
-----
2 b 1
3 c 2
1 a 0
q.test){if[count k:k where not (k:exec (a,'b) from x) in key `..t;#[`..t;;:;value[`..t](0N;`)]'[k]];update t:`t$(a,'b) from x}([] a:2 3 1 5 6;b:`b`c`a`d`e)
a b t
-----
2 b 1
3 c 2
1 a 0
5 d 3
6 e 4
q.test)value `..t //check table t, new dummy records added by previous call
a b| c
---| -
1 a| g
2 b| h
3 c| i
5 d|
6 e|
I've done these tests inside a namespace as this is how the dataprocess function will run in TorQ (i.e. at certain places you need to use `..t to access t in the root namespace.) The analogous version of this function for your setup (with some nicer formatting than the one-liners above) would be something like:
{
if[count k:k where not (k:exec (x,'ccypair from volatilitysurface_smile) in key `..volatilitysurface; //check for missing keys
#[`..volatilitysurface;;:;value[`..volatilitysurface](0Nz;`)]'[k]]; //index into null key of table to get dummy record and upsert to global volatilitysurface table
update volatilitysurface:`volatilitysurface$(x,'ccypair) from x //create foreign key
}

KDB select regex from

Is there a way to use a regex for defining all the columns in a select statement. something like
select myColumnPrefix* from myTable
that would show all columns that start with myColumnPrefix?
Not using qSQL, however you can use a regex to get the column names and then use a functional select. For example,
c: cols[myTable] where cols[myTable] like "myColumnPrefix*";
?[myTable;();0b;c!c]
Or as a one-liner,
?[myTable;();0b;{x!x#:where x like "myColumnPrefix*"} cols myTable]
If you really want to use qSQL (not advised - difficult to read/maintain) then you could expand on what #ostewart suggested:
Define table:
q)t:([]foo1:1 2;foo2:3 4;foo3:5 6;bar1:1 2;bar2:3 4;bar3:5 6)
q)t
foo1 foo2 foo3 bar1 bar2 bar3
-----------------------------
1 3 5 1 3 5
2 4 6 2 4 6
Extract columns of interest and prepare as string:
q)c:", " sv string cols[t] where cols[t] like "foo*";
q)c
"foo1, foo2, foo3"
Join columns to select query and value expression:
q)value "select ",c," from t"
foo1 foo2 foo3
--------------
1 3 5
2 4 6