KDB/Q: How to write a loop and append output table? - kdb

Question

Disclaimer: I am very new to the Q language so please excuse my silly question.
I have a function that currently is taking on 2 parameters (date;sym).It runs fine for 1 sym and 1 day. however, I need to perform this on multiple syms and dates which will take forever.
How do I create a loop that run the function on every sym, and on every date?
In python, it is straighforward as :
for date in datelist:
for sym in symlist:
func(date,sym)
How can I do something similar to this in Q? and how can I dynamically change the output table names and append them to 1 single table?
Currently, I am using the following:
output: raze .[function] peach paralist
where paralist is a list of parameter pairs: ((2020.06.01;ABC);(2020.06.01;XYZ)) but imho this is nowhere near efficient.
What would be the best way to achieve this in Q?

Answer 1

I'll generalize everything, if you have a given function foo which will operate on an atom dt with a vector s
q)foo:{[dt;s] dt +\: s}
q)dt:10?10
q)s:100?10
q)dt
8 1 9 5 4 6 6 1 8 5
q)s
4 9 2 7 0 1 9 2 1 8 8 1 7 2 4 5 4 2 7 8 5 6 4 1 3 3 7 8 2 1 4 2 8 0 5 8 5 2 8..
q)foo[;s] each dt
12 17 10 15 8 9 17 10 9 16 16 9 15 10 12 13 12 10 15 16 13 14 12 9 11 11 ..
5 10 3 8 1 2 10 3 2 9 9 2 8 3 5 6 5 3 8 9 6 7 5 2 4 4 ..
13 18 11 16 9 10 18 11 10 17 17 10 16 11 13 14 13 11 16 17 14 15 13 10 12 12 ..
9 14 7 12 5 6 14 7 6 13 13 6 12 7 9 10 9 7 12 13 10 11 9 6 8 8 ..
The solution is to project the symList over the function in question, then use each (or peach) for the date variable.
If your function requires an atomic date and sym, then you can just create a new function to implement this
q)bar:{[x;y] foo[x;] each y};

Answer 2

datelist:`date$10?10
symlist:10?`IBM`MSFT`GOOG
function:{0N!(x;y)}
{.[function;x]} peach datelist cross symlist
cross will return all combinations of sym and date
Is this what you need?

Answer 3

Try to use two "double" '
raze function'[datelist]'[symlist]
peach or each won't work here. They are not operators, but anonymous functions with two parameters: each is k){x'y}. That is why function each list1 each list2 statement is invalid, but function'[list1]'[list2] works.

Answer 4

From reading your response to another answer you are looking to save the results with unique names yes? Take a look at this solution using set to save and get to retrieve.
q)t:flip enlist each `colA`colB!(100;`name)
q)t
colA colB
---------
100 name
q)f:{[date;sym]tblName:`$string[date],string sym;tblName set update date:date,sym:sym from t}
q)newTbls:f'[.z.d+til 3;`AAA`BBB`CCC]
q)newTbls
`2020.09.02AAA`2020.09.03BBB`2020.09.04CCC
q)get each newTbls
+`colA`colB`date`sym!(,100;,`name;,2020.09.02;,`AAA)
+`colA`colB`date`sym!(,100;,`name;,2020.09.03;,`BBB)
+`colA`colB`date`sym!(,100;,`name;,2020.09.04;,`CCC)
q)get first newTbls
colA colB date sym
------------------------
100 name 2020.09.02 AAA
Does this meet your needs?

Answer 5

This could be a stab in the dark, but why not create a hdb rather than all these variables output20191005ABC, output20191006ABC .. etc and given you want to append them to 1 table.
Below I have outlined how to create a date partitioned hdb called outputHDB which has one table outputTbl. I created the hdb by running a function by date and sym and then upserting those rows to disk.
C:\Users\Matthew Moore>mkdir outputHDB
C:\Users\Matthew Moore>cd outputHDB
// can change the outputHDB as desired
// start q
h:hopen `::6789; // as a demo I connected to another hdb process and extracted some data per sym / date over IPC
hdbLoc:hsym `$"C:/Users/Matthew Moore/outputHDB";
{[d;sl]
{[d;s]
//output:yourFunc[date;sym];
// my func as a demo, I'm grabbing rows where price = max price by date and by sym from another hdb using the handle h
output:{[d;s]
h({[d;s] select from trades where date = d, sym = s, price = max price};d;s)
}[d;s];
// HDB Part
path:` sv (hdbLoc;`$string d;`outputTbl;`);
// change `outputTbl to desired table name
// dynamically creates the save location and then upserts one syms data directly to disk
// e.g. `:C:/Users/Matthew Moore/outputHDB/2014.04.21/outputTbl/
// extra / at the end saves the table as splayed i.e. each column is it's own file within the outputTbl directory
path upsert .Q.en[`:.;output];
// .Q.en enumerates syms in a table which is required when saving a table splayed
}[d;] each sl;
// applies the parted attribute to the sym column on disk, this speeds up querying for on disk data
#[` sv (hdbLoc;`$string d;`outputTbl;`);`sym;`p#];
}[;`AAPL`CSCO`DELL`GOOG`IBM`MSFT`NOK`ORCL`YHOO] each dateList:2014.04.21 2014.04.22 2014.04.23 2014.04.24 2014.04.25;
Now that the hdb has been created, you can load it from disk and query with qSQL
q)\l .
q)select from outputTbl where date = 2014.04.24, sym = `GOOG
date sym time src price size
------------------------------------------------------------
2014.04.24 GOOG 2014.04.24D13:53:59.182000000 O 46.43 2453