I am trying to programmatically construct arguments to functional select call having the form:
?[ `t; () ; groupBy; ()]
The problematic part is groupBy, which should be a dictionary.
Suppose the objective is to arrive at the parse tree in the form:
parse "select by sym,month:`date$dt.month from t"
I start constructing the by part with:
groupBy: enlist[`sym]!enlist(`sym)
Then I try to append the month part of the group by statement (note that periodicity is parameterised):
per: `month / monthly periodicity
groupBy,: {enlist[x]!enlist[ $[x = `day;
`dt;
$[x=`month;
((parse "select by month:`date$dt.month from x")#3)#`month
;` sv (`dt,x)]
]]
}[per]
However, that throws type error. What goes wrong?
I think it doesn't like the compound assignment
groupBy,:{...}
Try
groupBy:groupBy,{...}
The difference is that in the first case it's trying to directly alter the data in memory without creating a copy, whereas in the second case it is creating a copy of the data in memory and then re-assigning it to that variable. Perhaps the compound assignment only works when the types are uniform
Related
I want to be able to run something like the following:
f:{[dt] syms:`sym1;eval parse"select from tbl where date = dt, sym=syms"}
f[.z.D]
Given the following :
tbl:([] date:2022.01.01 2022.01.01; Id:1000000 2000000; sym:`sym1`sym2;price:10 20;qty:3 4)
f:{[dt] syms:`sym1; ?[tbl;((=;`date;`dt);(=;`sym;`syms));0b;()]}
f1:{[dt] syms:`sym1; (?) . (tbl;((=;`date;`dt);(=;`sym;`syms));0b;())}
f2:{[dt] syms:`sym1; value (?;tbl;((=;`date;`dt);(=;`sym;`syms));0b;())}
f[.z.D] // works
f1[.z.D] // Gives Error - dt not recognized/out of scope
f2[.z.D] // Gives Error - dt not recognized/out of scope
Value within a function seems to not be able to detect the local variables and surprisingly (?) . also fails. (maybe because this in itself is a function and dt is not defined here?)
Is there any work around for this?
For context, I have a function that takes a select string/functional select, parses it, does some checks and manipulations on the functional form and returns a modified functional form.
I want users to be able to call this function from their own functions and that parameters they have defined in their function can be in the outputted functional form and that functional form can be valued some how.
I don't want users to be forced to pass more variables into my function etc.
What you need to do here is remove the backtick for dt and syms
I would also recommend using a backtick when calling your table name.
Further, you should make sure syms is enlisted if it is only one symbol.
So your function should be:
f:{[dt] syms:(),`sym1; ?[`tbl;((=;`date;dt);(=;`sym;syms));0b;()]}
If you parse your select statement you can see the correct form for functional selects:
q)parse "select from tbl where date=2022.01.01,sym=`sym1"
?
`tbl
,((=;`date;2022.01.01);(=;`sym;,`sym1)) // comma in front of `sym1 means enlist
0b
()
The backtick is not needed as this is a variable, defined in your function, it would be the same as doing:
?[`tbl;((=;`date;2022.01.01);(=;`sym;enlist `sym1));0b;()]
This should allow you to use your function correctly:
q)f[2022.01.01]
date Id sym price qty
---------------------------------
2022.01.01 1000000 sym1 10 3
For more information, see the kx documentation
I would like to pass a column name into a Q function to query a loaded table.
Example:
getDistinct:{[x] select count x from raw}
getDistinct "HEADER"
This doesn't work as the Q documentation says I cannot pass column as arguments. Is there a way to bypass this?
When q interprets x it will treat it as a string, it has no reference to the column, so your output would just be count "HEADER".
If you want to pass in the column as a string you need to build the whole select statement then use value
{value "select count ",x," from tab"} "HEADER"
However, the recommended method would be to use a functional select. Below I use parse to build the functional select equivalent using the parse tree.
/Create sample table
tab:([]inst:10?`MSFT`GOOG`AAPL;time:10?.z.p;price:10?10f)
/Generate my parse tree to get my functional form
.Q.s parse "select count i by inst from tab"
/Build this into my function
{?[`tab;();(enlist x)!enlist x;(enlist `countDistinct)!enlist (#:;`i)]} `inst
Note that you have to pass the column in as a symbol. Additionally the #:i is just the k equivalent to count i.
Update for multiple columns
tab:([]inst:10?`MSFT`GOOG`AAPL;time:10?.z.p;price:10?10f;cntr:10`HK`SG`UK`US)
{?[`tab;();(x)!x;(enlist `countDistinct)!enlist (#:;`i)]} `inst`cntr
To get the functional form of a select statement, I recommend using buildSelect. Also, reduce the scope of parenthesis, i.e. use enlist[`countDistinct] instead of (enlist `countDistinct).
The following lines of Q code all throw an error, because when the statement "local" is parsed, the local variable is not in the correct scope.
{local:1; value "local"}[]
{[local]; value "local"}[1]
{local:1; eval parse "local"}[]
{[local]; eval parse "local"}[1]
Is there a way to reach the local variable from inside the parsed string?
Note: This is a simplification of the actual problem I'm grappling with, which is to write a function that executes a query, accepting a list of columns which it should return. I imagine the finished product looking something like this:
getData:{[requiredColumns, condition]
value "select ",(", " sv string[requiredColumns])," from myTable where someCol=condition"
}
The condition parameter in this query is the one that isn’t recognised and I do realise I could append it’s value rather than reference it inside a string, but the real query uses lots of local variables including tables etc, so it’s not as easy as just pulling all the variables out of the string before calling value on it.
I'm new to KDB and Q, so if anyone has a better way to achieve the same effect I'm happy to be schooled on the proper way to achieve this outcome in Q. Would still be interested to know in the variable access thing is possible though.
In the first example, you are right that local is not within the correct scope, as value is looking for the global variable local.
One way to get around this is to use a namespace, which will define the variable globally, but can only be accessed by calling that namespace. In the modified example below I have defined local in the .ns namespace
{.ns.local:1; value ".ns.local"}[]
For the problem you are facing with selecting, if requiredColumns is a symbol list of columns you can just use the take operator # to select them.
getData:{[requiredColumns] requiredColumns#myTable}
For more advanced queries using variables you may have to use functional select form, explained here. This will allow you to include variables in the where and by clause of the select statement
The same example in functional form would be (no by clause, only select and where):
getData:{[requiredColumns;condition] requiredColumns:(), requiredColumns;
?[myTable;enlist (=;`someCol;condition);0b;requiredColumns!requiredColumns]}
The first line ensures that requiredColumns is a list even if the user enters a single column name
value will look for a variable in the global scope that's why you are getting an error. You can directly use local variables like you are doing that in your function.
Your function is mostly correct, just need a slight correction to append condition(I have mentioned that below). However, a better approach would be to use functional select in this case.
Using functional select:
q) t:([]id:`a`b; val:3 4)
q) gd: {?[`t;enlist (=;`val;y);0b;((),x)!(),x]}
q) gd[`id;3] / for single column
Output:
id
-
1
q) gd[`id`val;3] / for multiple columns
In case your condition column is of type symbol, then enlist your condition value like:
q) gd: {?[`t;enlist (=;`id;y);0b;((),x)!(),x]}
q) gd[`id;enlist `a]
You can use parse to get a functional form of qsql queries:
q) parse " select id,val from t where id=`a"
?
`t
,,(=;`id;,`a)
0b
`id`val!`id`val
Using String concat(your function):
q)getData:{[requiredColumns;condition] value "select ",(", " sv string[requiredColumns])," from t where id=", .Q.s1 condition}
q) getData[enlist `id;`a] / for single column
q) getData[`id`val;`a] / for multi columns
I'm a total erlang noob and I just want to see what's in a particular table I have. I want to just "select *" from a particular table to start with. The examples I'm seeing, such as the official documentation, all have column restrictions which I don't really want. I don't really know how to form the MatchHead or Guard to match anything (aka "*").
A very simple primer on how to just get everything out of a table would be very appreciated!
For example, you can use qlc:
F = fun() ->
Q = qlc:q([R || R <- mnesia:table(foo)]),
qlc:e(Q)
end,
mnesia:transaction(F).
The simplest way to do it is probably mnesia:dirty_match_object:
mnesia:dirty_match_object(foo, #foo{_ = '_'}).
That is, match everything in the table foo that is a foo record, regardless of the values of the fields (every field is '_', i.e. wildcard). Note that since it uses record construction syntax, it will only work in a module where you have included the record definition, or in the shell after evaluating rr(my_module) to make the record definition available.
(I expected mnesia:dirty_match_object(foo, '_') to work, but that fails with a bad_type error.)
To do it with select, call it like this:
mnesia:dirty_select(foo, [{'_', [], ['$_']}]).
Here, MatchHead is _, i.e. match anything. The guards are [], an empty list, i.e. no extra limitations. The result spec is ['$_'], i.e. return the entire record. For more information about match specs, see the match specifications chapter of the ERTS user guide.
If an expression is too deep and gets printed with ... in the shell, you can ask the shell to print the entire thing by evaluating rp(EXPRESSION). EXPRESSION can either be the function call once again, or v(-1) for the value returned by the previous expression, or v(42) for the value returned by the expression preceded by the shell prompt 42>.
I am currently successfully using a CASE expression to update an empty column based on attributes from other columns. For example
UPDATE table SET cat = CASE
WHEN term = '{"Boulder"}' then 'Boulder'
However, I need to do the same but on an text array and particularly when an element is in a specific position within that array.
For example if the data looks like
{"Boulder, Tree, Bush"}
WHEN position('Tree' in term) > 0 then 'Boulder'
But I receive an error message
function pg_catalog.position(character varying[], unknown) does not exist
I have used position in a function before so I am not sure why PostgreSQL does not like it in this situation.
Is there a way to using a CASE expression whilst determining the position of a text element within an array.
Apparently your term column is defined as an array, e.g. varchar[]. The position function only works with scalar values, not with arrays.
If you want to test if an element is contained in an array you need to use a different operator: #>
update foobar
set cat = 'Boulder'
where term #> '{"Boulder"}'
The expression '{"Boulder"}' creates an array with a single element. It's equivalent to array['Boulder'] (which I find more readable). So the above where condition updates all rows where the array term contains all elements of the array on the right hand side of the operator. In this case it's only a single element you are testing for.
More details about the array functions and operators can be found in the manual: http://www.postgresql.org/docs/current/static/functions-array.html
Edit after the requirements have changed
To find and update only those where boulder is in the first, second or third place, you can use this:
update foobar
set cat = 'Boulder'
where 'Boulder' in (term[1], term[2], term[3]);