I would like to pass a column name into a Q function to query a loaded table.
Example:
getDistinct:{[x] select count x from raw}
getDistinct "HEADER"
This doesn't work as the Q documentation says I cannot pass column as arguments. Is there a way to bypass this?
When q interprets x it will treat it as a string, it has no reference to the column, so your output would just be count "HEADER".
If you want to pass in the column as a string you need to build the whole select statement then use value
{value "select count ",x," from tab"} "HEADER"
However, the recommended method would be to use a functional select. Below I use parse to build the functional select equivalent using the parse tree.
/Create sample table
tab:([]inst:10?`MSFT`GOOG`AAPL;time:10?.z.p;price:10?10f)
/Generate my parse tree to get my functional form
.Q.s parse "select count i by inst from tab"
/Build this into my function
{?[`tab;();(enlist x)!enlist x;(enlist `countDistinct)!enlist (#:;`i)]} `inst
Note that you have to pass the column in as a symbol. Additionally the #:i is just the k equivalent to count i.
Update for multiple columns
tab:([]inst:10?`MSFT`GOOG`AAPL;time:10?.z.p;price:10?10f;cntr:10`HK`SG`UK`US)
{?[`tab;();(x)!x;(enlist `countDistinct)!enlist (#:;`i)]} `inst`cntr
To get the functional form of a select statement, I recommend using buildSelect. Also, reduce the scope of parenthesis, i.e. use enlist[`countDistinct] instead of (enlist `countDistinct).
Related
I want to be able to run something like the following:
f:{[dt] syms:`sym1;eval parse"select from tbl where date = dt, sym=syms"}
f[.z.D]
Given the following :
tbl:([] date:2022.01.01 2022.01.01; Id:1000000 2000000; sym:`sym1`sym2;price:10 20;qty:3 4)
f:{[dt] syms:`sym1; ?[tbl;((=;`date;`dt);(=;`sym;`syms));0b;()]}
f1:{[dt] syms:`sym1; (?) . (tbl;((=;`date;`dt);(=;`sym;`syms));0b;())}
f2:{[dt] syms:`sym1; value (?;tbl;((=;`date;`dt);(=;`sym;`syms));0b;())}
f[.z.D] // works
f1[.z.D] // Gives Error - dt not recognized/out of scope
f2[.z.D] // Gives Error - dt not recognized/out of scope
Value within a function seems to not be able to detect the local variables and surprisingly (?) . also fails. (maybe because this in itself is a function and dt is not defined here?)
Is there any work around for this?
For context, I have a function that takes a select string/functional select, parses it, does some checks and manipulations on the functional form and returns a modified functional form.
I want users to be able to call this function from their own functions and that parameters they have defined in their function can be in the outputted functional form and that functional form can be valued some how.
I don't want users to be forced to pass more variables into my function etc.
What you need to do here is remove the backtick for dt and syms
I would also recommend using a backtick when calling your table name.
Further, you should make sure syms is enlisted if it is only one symbol.
So your function should be:
f:{[dt] syms:(),`sym1; ?[`tbl;((=;`date;dt);(=;`sym;syms));0b;()]}
If you parse your select statement you can see the correct form for functional selects:
q)parse "select from tbl where date=2022.01.01,sym=`sym1"
?
`tbl
,((=;`date;2022.01.01);(=;`sym;,`sym1)) // comma in front of `sym1 means enlist
0b
()
The backtick is not needed as this is a variable, defined in your function, it would be the same as doing:
?[`tbl;((=;`date;2022.01.01);(=;`sym;enlist `sym1));0b;()]
This should allow you to use your function correctly:
q)f[2022.01.01]
date Id sym price qty
---------------------------------
2022.01.01 1000000 sym1 10 3
For more information, see the kx documentation
The following lines of Q code all throw an error, because when the statement "local" is parsed, the local variable is not in the correct scope.
{local:1; value "local"}[]
{[local]; value "local"}[1]
{local:1; eval parse "local"}[]
{[local]; eval parse "local"}[1]
Is there a way to reach the local variable from inside the parsed string?
Note: This is a simplification of the actual problem I'm grappling with, which is to write a function that executes a query, accepting a list of columns which it should return. I imagine the finished product looking something like this:
getData:{[requiredColumns, condition]
value "select ",(", " sv string[requiredColumns])," from myTable where someCol=condition"
}
The condition parameter in this query is the one that isn’t recognised and I do realise I could append it’s value rather than reference it inside a string, but the real query uses lots of local variables including tables etc, so it’s not as easy as just pulling all the variables out of the string before calling value on it.
I'm new to KDB and Q, so if anyone has a better way to achieve the same effect I'm happy to be schooled on the proper way to achieve this outcome in Q. Would still be interested to know in the variable access thing is possible though.
In the first example, you are right that local is not within the correct scope, as value is looking for the global variable local.
One way to get around this is to use a namespace, which will define the variable globally, but can only be accessed by calling that namespace. In the modified example below I have defined local in the .ns namespace
{.ns.local:1; value ".ns.local"}[]
For the problem you are facing with selecting, if requiredColumns is a symbol list of columns you can just use the take operator # to select them.
getData:{[requiredColumns] requiredColumns#myTable}
For more advanced queries using variables you may have to use functional select form, explained here. This will allow you to include variables in the where and by clause of the select statement
The same example in functional form would be (no by clause, only select and where):
getData:{[requiredColumns;condition] requiredColumns:(), requiredColumns;
?[myTable;enlist (=;`someCol;condition);0b;requiredColumns!requiredColumns]}
The first line ensures that requiredColumns is a list even if the user enters a single column name
value will look for a variable in the global scope that's why you are getting an error. You can directly use local variables like you are doing that in your function.
Your function is mostly correct, just need a slight correction to append condition(I have mentioned that below). However, a better approach would be to use functional select in this case.
Using functional select:
q) t:([]id:`a`b; val:3 4)
q) gd: {?[`t;enlist (=;`val;y);0b;((),x)!(),x]}
q) gd[`id;3] / for single column
Output:
id
-
1
q) gd[`id`val;3] / for multiple columns
In case your condition column is of type symbol, then enlist your condition value like:
q) gd: {?[`t;enlist (=;`id;y);0b;((),x)!(),x]}
q) gd[`id;enlist `a]
You can use parse to get a functional form of qsql queries:
q) parse " select id,val from t where id=`a"
?
`t
,,(=;`id;,`a)
0b
`id`val!`id`val
Using String concat(your function):
q)getData:{[requiredColumns;condition] value "select ",(", " sv string[requiredColumns])," from t where id=", .Q.s1 condition}
q) getData[enlist `id;`a] / for single column
q) getData[`id`val;`a] / for multi columns
I have a query that returns data for a bunch of syms for a date from a table.
dataFromTab:delete date from select from table where date=2015.01.01,any (sym like) each ("sym1";"sym2";"sym3";"sym4")
dataFromTab:`sym xasc dataFromTab;
#[`dataFromTab;`sym;`p#];
type var is 98h and sorting by sym is working fine but when adding p# using functional amend is giving me a 'type error.
Any inputs on where i am going wrong would be appreciated.
Thanks!
Are you calling all of this from within a function or from the global namespace?
If dataFromTab is a local variable then applying the p# to the global referencedataFromTab will not work.
q){data:select from tab;#[`data;`sym;`p#]}[]
'type
[1] {data:select from tab;#[`data;`sym;`p#]}
^
q))\
You will need to reassign dataFromTab locally for this to work;
q){data:select from tab;data:#[data;`sym;`p#]}[]
You could also use 'set' to create your table globally, which would allow your original syntax to work. However you probably do not want to create unnecessary globals;
q){`data set select from tab;#[`data;`sym;`p#]}[]
`data
Is sym a string column? I get that error if I convert to string:
q)trades:update string sym from trades
q)trades:`sym xasc trades
q)#[`trades;`sym;`p#]
'type
[0] #[`trades;`sym;`p#]
^
Can you try converting it to a symbol and see if that works?
Let's say I have a column of a table whose data type is a character array. I want to pass in a functional select where clause, where the column is in a list of given strings. However, I cannot simply use (in; `col; myList) for reasons. Instead, I need to do the equivalent of:
max col like/: myList
which effectively gives the same result. However, I have tried to put this in functional form
(max; (like/:; `col; myList))
And I am getting a type error. Any ideas on how I could make this work?
A nice trick when dealing with this problem is using parse on a string of the select statement you want to functionalize. For example:
q)parse"select from t where max col like/: myList"
?
`t
,,(max;((/:;like);`col;`myList))
0b
()
Or specifically in your case you want the 3rd element of the result list (the where clause):
q)(parse"select from t where max col like/: myList")2
max ((/:;like);`col;`myList)
I even think using this pattern in your actual code can be a good idea, as functionalized statements like max ((/:;like);`col;`myList) can get pretty unreadable pretty quickly!
Hope that helps!
(any; ((/:;like); `col; enlist,myList))
it should be: (max;((/:;like);`col;`mylist))
I am trying to programmatically construct arguments to functional select call having the form:
?[ `t; () ; groupBy; ()]
The problematic part is groupBy, which should be a dictionary.
Suppose the objective is to arrive at the parse tree in the form:
parse "select by sym,month:`date$dt.month from t"
I start constructing the by part with:
groupBy: enlist[`sym]!enlist(`sym)
Then I try to append the month part of the group by statement (note that periodicity is parameterised):
per: `month / monthly periodicity
groupBy,: {enlist[x]!enlist[ $[x = `day;
`dt;
$[x=`month;
((parse "select by month:`date$dt.month from x")#3)#`month
;` sv (`dt,x)]
]]
}[per]
However, that throws type error. What goes wrong?
I think it doesn't like the compound assignment
groupBy,:{...}
Try
groupBy:groupBy,{...}
The difference is that in the first case it's trying to directly alter the data in memory without creating a copy, whereas in the second case it is creating a copy of the data in memory and then re-assigning it to that variable. Perhaps the compound assignment only works when the types are uniform