kdb: can I use the set keyword to achieve #[`.;`upd;:;.some.function]? - kdb

Say there's an existing function definition .my.func:{[tbl;data] ...} and I want to set the upd function to this lambda.
Are the two lines below equivalent?
#[`.;`upd;:;.my.func]; / #1
`upd set .my.func; / #2
Asking because I see a lot of #1 in the codebase I work with, but #2 seems more succinct, so was wondering if they're somehow different.
I checked https://code.kx.com/q/ref/amend/
"Amend at" #[d; i; v; vy]
This seems to simply define the function upd in the global namespace.
d = `.
i = `upd
v = :
vy = .my.func
After running #1/#2 myself, get`. also seems to suggest #1/2 are equivalent.

The lines are not equivalent in the scenario where \d is used to change namespace. In the examples above set creates the variable in the current namespace and this happens to be the global namespace. # creates the variable in the global namespace.
Example for set using .test namespace creates .test.upd instead.
d).my.func:{x}
q)\d .test
q.test)`upd set .my.func;
q.test)upd
{x}
q.test)\d .
q)upd / upd does not exist in global namespace
'upd
[0] upd
^
q).test.upd / upd exists in .test namespace
{x}
Example for # using upd2 to highlight differences to upd example above. upd2 is created in the global namespace.
q)\d .test
q.test)#[`.;`upd2;:;.my.func];
q.test)upd2 / upd2 does not exist in .test namespace
'upd2
[0] upd2
^
q.test)\d .
q)upd2 / upd2 exists in global namespace
{x}

Following on from Thomas' answer, it's also helpful to know that you can use # to create your function in the current namespace also. The d system command, without an argument, will return the current namespace, and you can use this as the first parameter to #.
q).my.func:{x}
q)\d .test
q.test)\d
`.test
q.test)system"d"
`.test
q.test)#[system"d";`upd;:;.my.func]
`.test
q.test)upd
{x}
Note that this method will only work once the namespace has been initialized first, i.e. something else must exist in the namespace, otherwise you'll see a 'type error.
q)key`
`q`Q`h`o`test
q)\d .new
q.new)#[system"d";`upd;:;.my.func]
'type
q.new)type each(.new;.test)
0 99h

Asking because I see a lot of #1 in the codebase I work with, but #2 seems more succinct, so was wondering if they're somehow different.
You can still use set to assign a value to a variable in the global namespace regardless of a namespace you're currently in, you just need to be a bit more explicit:
q)`upd set .my.func / sets upd in the current namespace
q)`..upd set .my.func / sets upd in the global namespace
q)\d .test
q.test)`upd set {x+1}
`upd
q.test)`..upd set {x+42}
`..upd
q.test)upd
{x+1}
q.test)\d .
q)upd
{x+42}

Related

kdb - passing in a symbol as parameter for variable name

I've been trying to pass in a symbol from a process outside KDB and am wishing to use this symbol to create a variable where we then load a specific file in case it doesn't exist.
//create some sample data and write to disk
.data.area52.varName1: ([] name:`billy`allen`terry; val:90 100 200);
.data.area52.varName2: 10 #200;
.data.area53.varName1: 9 #100;
.data.area53.varName2: ([] name:`joe`jim`bob; val:10 20 30);
.data.area53.varName3: `random;
`:area52 set .data.area52;
`:area53 set .data.area52;
I am trying to pass in the parameter for reference area52for example which will test a conditional if it exists (do nothing, else) then create the variable loading viaget `:area52
First, I wrote a conditional below which seems to check (however trhe `.areaXX isn't parametrized yet.
// fresh instance of KDB
.data.area52: $[`area52 in key `.data; .data.area52; get `:area52
And then stumbled onto the link here kdb+: use string as variable name
Which gets me most of the way there.
Is there a way to parameterize the beginning of the lambda below passing in some combination of `.area52 ? Much of this variable can be assembled / edited outside KDB and only passing in the
for an example, we can have several hundred `.areaXX that we could pass into KDB as data is changed and refreshed.
{.data.area52:()!(); #[`.data; `area52;:; (get `:area52)]} [];
To safeguard against the situation where the namespace .data doesn't exist you could use the following lambda which takes parameter area52 as an argument.
q){(` sv ``data,x) set ()!(); #[`.data; x; :; get hsym x]}`area52
`.data
Tying it back to the original problem, this lambda will create the variable if it does not exist by reading from the file of the same name:
/ .data does not exist yet
q).data
'.data
[0] .data
^
/ create variable if necessary
q){if[not x in key `.data; (` sv ``data,x)set get hsym x]}`area52
/ check .data
q).data
| ::
area52| ``varName1`varName2!(::;+`name`val!(`billy`allen`terry;90 100 200);20..

How to change a namespace in a single query?

If I try to switch namespace in-place (say via qcon) to define function exactly in namespace (I want to get[`f][3;0] will give exacly the namespace in which it was defined), I got empy result - it seems like no namespace switching occured:
\d .
system"d .ns1"; f1:{x+y}; get[.ns1.f1][3;0]~`
but:
\d .
system"d .ns1"; // notice the only difference is the newline
f1:{x+y}; get[.ns1.f1][3;0]~`ns1
Is it a bug of 3.6 32bit version? or a feature?
If it is a feature, then how to switch to new namespace in a single qcon query?
A code line is evaluated in the context before the code line is executed. Therefore, f1's definition in the first sample was evaluated in the context before namespace switching occurred.

Access locally scoped variables from within a string using parse or value (KDB / Q)

The following lines of Q code all throw an error, because when the statement "local" is parsed, the local variable is not in the correct scope.
{local:1; value "local"}[]
{[local]; value "local"}[1]
{local:1; eval parse "local"}[]
{[local]; eval parse "local"}[1]
Is there a way to reach the local variable from inside the parsed string?
Note: This is a simplification of the actual problem I'm grappling with, which is to write a function that executes a query, accepting a list of columns which it should return. I imagine the finished product looking something like this:
getData:{[requiredColumns, condition]
value "select ",(", " sv string[requiredColumns])," from myTable where someCol=condition"
}
The condition parameter in this query is the one that isn’t recognised and I do realise I could append it’s value rather than reference it inside a string, but the real query uses lots of local variables including tables etc, so it’s not as easy as just pulling all the variables out of the string before calling value on it.
I'm new to KDB and Q, so if anyone has a better way to achieve the same effect I'm happy to be schooled on the proper way to achieve this outcome in Q. Would still be interested to know in the variable access thing is possible though.
In the first example, you are right that local is not within the correct scope, as value is looking for the global variable local.
One way to get around this is to use a namespace, which will define the variable globally, but can only be accessed by calling that namespace. In the modified example below I have defined local in the .ns namespace
{.ns.local:1; value ".ns.local"}[]
For the problem you are facing with selecting, if requiredColumns is a symbol list of columns you can just use the take operator # to select them.
getData:{[requiredColumns] requiredColumns#myTable}
For more advanced queries using variables you may have to use functional select form, explained here. This will allow you to include variables in the where and by clause of the select statement
The same example in functional form would be (no by clause, only select and where):
getData:{[requiredColumns;condition] requiredColumns:(), requiredColumns;
?[myTable;enlist (=;`someCol;condition);0b;requiredColumns!requiredColumns]}
The first line ensures that requiredColumns is a list even if the user enters a single column name
value will look for a variable in the global scope that's why you are getting an error. You can directly use local variables like you are doing that in your function.
Your function is mostly correct, just need a slight correction to append condition(I have mentioned that below). However, a better approach would be to use functional select in this case.
Using functional select:
q) t:([]id:`a`b; val:3 4)
q) gd: {?[`t;enlist (=;`val;y);0b;((),x)!(),x]}
q) gd[`id;3] / for single column
Output:
id
-
1
q) gd[`id`val;3] / for multiple columns
In case your condition column is of type symbol, then enlist your condition value like:
q) gd: {?[`t;enlist (=;`id;y);0b;((),x)!(),x]}
q) gd[`id;enlist `a]
You can use parse to get a functional form of qsql queries:
q) parse " select id,val from t where id=`a"
?
`t
,,(=;`id;,`a)
0b
`id`val!`id`val
Using String concat(your function):
q)getData:{[requiredColumns;condition] value "select ",(", " sv string[requiredColumns])," from t where id=", .Q.s1 condition}
q) getData[enlist `id;`a] / for single column
q) getData[`id`val;`a] / for multi columns

Retain the value of loop variable even on making a call to another function in the loop

I want to preserve the value of variable q in the below mentioned code when it makes a call to the funtion gm.
demo.m is:
for q=1:Nqueries
disp(['Matching query ' db.queries{q}]);
qPath=[db.folder '/' db.fqueryFolder '/' db.queries{q} '_' featMethod '_' num2str(PeakThreshold) '.mat'];
fq=load(qPath);
query_path=[db.folder '/' db.queryFolder '/' db.queries{q} '.jpg'];
matches=cell(1,Nrefs);
fr=cell(1,Nrefs);
ref_paths=cell(1,Nrefs);
for r=1:Nrefs
rPath=[db.folder '/' db.frefFolder '/' db.references{r} '_' featMethod '_' num2str(PeakThreshold) '.mat'];
ref_paths{r}=[db.folder '/' db.refFolder '/' db.references{r} '.jpg'];
fr{r}=load(rPath);
%Matching things
[idx, dists] = vl_ubcmatch(fq.d,fr{r}.d,thRatio);
matches{r}.idx=idx;
matches{r}.dists=dists;
end
%We run the Generative Model
sim(q,:)=gm(query_path,ref_paths,fq,fr,matches,K);
end
and this code generates following error:
Matching query 1
??? Undefined function or variable 'q'.
Error in ==> gm at 86
Iq=imread(sprintf('db/queries/%d.jpg',q));
Error in ==> demo at 65
sim(q,:)=gm(query_path,ref_paths,fq,fr,matches,K);
The gm function uses q as follows:
Iq=imread(sprintf('db/queries/%d.jpg',q));
Adding more variables to the function call is the cleanest way of resolving this issue, of course. But if modifying the called function is too painful, e.g. because you'd have to change many functions until you reach the one where you want to use your variable, you might want to consider making this variable a global variable:
global YOURVARIABLE %choose a good name here to avoid
%overwriting existing global variables
YOURVARIABLE can now be accessed from any other function's workspace although you have to declare this in each function separately, see:
Declaring a global variable in MATLAB
Also, you should be very careful when using them:
http://www.mathworks.com/help/matlab/matlab_prog/share-data-between-workspaces.html
As described in the documentation global variables are risky because they have their own workspace that can be edited from anywhere, so if the same variable is used by several functions you might get unexpected results. Therefore, they should only be used when really necessary.
I modified the code in the for loop to
sim(q,:)=gm(query_path,ref_paths,fq,fr,matches,K,q);
and the definition of the called function gm as
gm(query_path,ref_paths,fq,fr,matches,K,q);

Why is this xmlns attribute messing up my xpath query?

I'm parsing a simple jhove output using LibXML. However, I don't get the values I expect. Here's the code:
use feature "say";
use XML::LibXML;
my $PRSR = XML::LibXML->new();
my $xs=<DATA>;
say $xs;
my $t1 = $PRSR->load_xml(string => $xs);
say "1:" . $t1->findvalue('//date');
$xs=<DATA>;
say $xs;
$t1 = $PRSR->load_xml(string => $xs);
say "2:" . $t1->findvalue('//date');
__DATA__
<jhove xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://hul.harvard.edu/ois/xml/ns/jhove" xsi:schemaLocation="http://hul.harvard.edu/ois/xml/ns/jhove http://hul.harvard.edu/ois/xml/xsd/jhove/1.3/jhove.xsd" name="Jhove" release="1.0 (beta 3)" date="2005-02-04"><date>2006-10-06T09:11:34+02:00</date></jhove>
<jhove><date>2006-10-06T09:11:34+02:00</date></jhove>
As you can see, the line "1:" is returning an empty string, while "2:" is returning the expected date. What is in the jhove-root-element that keeps the xpath query from working properly? I even tried in XML-Spy and there it works, even with the full header.
Edit: When I remove the xmlns-attribute from the root element, the xpath query works. But how is that possible?
The XML::LibXML::Node documentation specifically mentions this issue and how to deal with it...
NOTE ON NAMESPACES AND XPATH:
A common mistake about XPath is to assume that node tests consisting of an element name with no prefix match elements in the default namespace. This assumption is wrong - by XPath specification, such node tests can only match elements that are in no (i.e. null) namespace.
So, for example, one cannot match the root element of an XHTML document with $node->find('/html') since '/html' would only match if the root element <html> had no namespace, but all XHTML elements belong to the namespace http://www.w3.org/1999/xhtml. (Note that xmlns="..." namespace declarations can also be specified in a DTD, which makes the situation even worse, since the XML document looks as if there was no default namespace).
There are several possible ways to deal with namespaces in XPath:
The recommended way is to use the XML::LibXML::XPathContext module to define an explicit context for XPath evaluation, in which a document independent prefix-to-namespace mapping can be defined. For example:
my $xpc = XML::LibXML::XPathContext->new;
$xpc->registerNs('x', 'http://www.w3.org/1999/xhtml');
$xpc->find('/x:html',$node);
Another possibility is to use prefixes declared in the queried document (if known). If the document declares a prefix for the namespace in question (and the context node is in the scope of the declaration), XML::LibXML allows you to use the prefix in the XPath expression, e.g.:
$node->find('/x:html');
I found another solution. Simply using this
say "1:" . $t1->findvalue('//*[local-name()="date"]');
will also find the value and save the hassle of declaring namespaces in an XPathContext. But apart from that, tobyinks answer is the correct one.