Hi is there a possibility to make a distinct select request to mnesia ?
I could copy the content of one table to ets and since ets is a hash table it could work. But i thought there is maybe a more elegant solution to this problem.
Thank you.
I'm not sure if this is what you had in mind, but you could use QLC's {unique, true} option (See QLC documentation for more info).
I created a mnesia table, called test, with bag semantics. Each row consists of the table name, a Key and a Value, so my rows looked like:
1. test, 1, 1
2. test, 2, 1
3. test, 2, 2
4. test, 3, 1
5. test, 3, 2
6. test, 3, 3
... etc.
Then this simple module illustrates my approach. Notice that you have to include the qlc library and that, in my example, I am selecting distinct Keys.
-module(test).
-export([select_distinct/0]).
-include_lib("stdlib/include/qlc.hrl").
select_distinct()->
QH = qlc:q( [K || {_TName, K, _V} <- mnesia:table(test)], {unique, true}),
F = fun() -> qlc:eval(QH) end,
{atomic, Result} = mnesia:transaction(F),
Result.
Compiling and running
> c("/home/jim/test", [{outdir, "/home/jim/"}]).
> test:select_distinct().
> [4,1,2,3,5]
If you wanted sorted output then use the following version of the QH = ... line above
QH = qlc:sort(qlc:q( [K || {_TName, K, _V} <- mnesia:table(test)], {unique, true})),
If you wanted to select distinct values, the following would work:
QH = qlc:sort(qlc:q( [V || {_TName, _K, V} <- mnesia:table(test)], {unique, true})),
Again, the code is just to illustrate an approach
For keys you can get a list of unique keys using:
mnesia:all_keys(Table).
From my tests, for bags it yields a list of unique keys.
Related
I am trying to split a string variable into multiple dummy coded variables. I used these sources to get an idea of how one would achieve this task in SPSS:
https://www.ibm.com/support/pages/making-multiple-string-variables-single-multiply-coded-field
https://www.spss-tutorials.com/spss-split-string-variable-into-separate-variables/
But when I try to adapt the first one to my needs or when I try to convert the second one to a macro, I fail.
In my dataset I have (multiple) variables that contain a comma seperated string that represents different combinations of selected items (as well as missing values). For each item of a specific variable I want to create a dummy variable. If the item was selected, it should be represented with a 1 in the new dummy variable. If it was not selected, that case should be represented with a 0.
Different input variables can contain different numbers of items.
For example:
ID
VAR1
VAR2
DMMY1_1
DMMY1_2
DMMY1_3
1
1, 2
8
1
1
0
2
1
1, 3
1
0
0
3
3, 1
2, 3, 1
1
0
1
4
2, 8
0
0
0
Here is what I came up with so far ...
* DEFINE DATA.
DATA LIST /ID 1 (F) VAR1 2-5 (A) VAR2 6-12 (A).
BEGIN DATA
11, 28
21 1, 3
33, 12, 3, 1
4 2, 8
END DATA.
* MACRO SYNTAX.
* DEFINE VARIABLES (in the long run these should/will be inside the macro function, but for now I will leave them outside).
NUMERIC v1 TO v3 (F1).
VECTOR v = v1 TO v3.
STRING #char (A1).
DEFINE split_var(vr = TOKENS(1)).
!DO !#pos=1 !TO char.length(!vr).
COMPUTE #char = char.substr(!vr, !#pos, 1).
!IF (!#char !NE "," !AND !#char !NE " ") !THEN
COMPUTE v(NUMBER(!#char, F1)) = 1.
!IFEND.
!DOEND.
!ENDDEFINE.
split_var vr=VAR1.
EXECUTE.
As I got more errors than I can count, it's hard to narrow down my problem. But I think the problem has something to do with the way I use the char.length() function (and I am a bit confused when to use the bang operator).
If anyone has some insights, I would really appreciate some help :)
There is a fundamental issue to understand about SPSS macro - the macro does not read or interact in any way with the data. All the macro does is manipulate text to write syntax. The syntax created will later work on the actual data when you run it.
So, for example, Your first error is using char.length(!vr) within the syntax. You are trying to get the macro to read the data, calculate the length and use, but that simply can't be done - the macro can only work with what you gave it.
Another example in your code: you calculate #char and then try to use it in the macro as !#char. So that obviously won't work. ! precedes only macro functions or arguments. #char, in your code, is neither, and it can't become one - can't read the data into the macro...
To give you a litte push forward: I understand you want the macro loop to run a different number of times for each variable, but you can't use char.length(!vr). I suggest instead have the macro loop as many times as necessary to be sure you can deal with the longest variable you'll need to work with.
And another general strategy hint - first, create syntax to deal with one specific variable and one specific delimiter. Once this works, start working on a macro, keeping in mind that the only purpose of the macro is to recreate the same working syntax, only changing the parameters of variable name and delimiter.
With my new understanding of the SPSS macro logic (thanks to #eli-k) the problem was quite easy to solve. Here is the working solution.
* DEFINE DATA.
DATA LIST /ID 1 (F) VAR1 2-5 (A) VAR2 6-12 (A).
BEGIN DATA
11, 28
21 1, 3
33, 12, 3, 1
4 2, 8
END DATA.
* DEFINE MACRO.
DEFINE #split_var(src_var = !TOKENS(1)
/dmmy_var_label = !DEFAULT(dmmy) !TOKENS(1)
/dmmy_var_lvls = !TOKENS(1))
NUMERIC !CONCAT(!dmmy_var_label,1) TO !CONCAT(!dmmy_var_label, !dmmy_var_lvls) (F1).
VECTOR #dmmy_vec = !CONCAT(!dmmy_var_label,1) TO !CONCAT(!dmmy_var_label, !dmmy_var_lvls).
STRING #char (A1).
LOOP #pos=1 TO char.length(!src_var).
COMPUTE #char = char.substr(!src_var, #pos, 1).
DO IF (#char NE "," AND #char NE " ").
COMPUTE #index = NUMBER(#char, F1).
COMPUTE #dmmy_vec(#index) = 1.
END IF.
END LOOP.
RECODE !CONCAT(!dmmy_var_label,1) TO !CONCAT(!dmmy_var_label, !dmmy_var_lvls) (SYSMIS=0) (ELSE=COPY).
EXECUTE.
!ENDDEFINE.
* CALL MACRO.
#split_var src_var=VAR2 dmmy_var_lvls=8.
I am having a couple of issues to put this in a functional format.
select from tableName where i=fby[(last;i);([]column_one;column_two)]
This is what I got:
?[tableName;fby;enlist(=;`i;(enlist;last;`i);(+:;(!;enlist`column_one`column_two;(enlist;`column_one;`column_two))));0b;()]
but I get a type error.
Any suggestions?
Consider using the following function, adjust from the buildQuery function given in the whitepaper on Parse Trees. This is a pretty useful tool for quickly developing in q, this version is an improvement on that given in the linked whitepaper, having been extended to handle updates by reference (i.e., update x:3 from `tab)
\c 30 200
tidy:{ssr/[;("\"~~";"~~\"");("";"")] $[","=first x;1_x;x]};
strBrk:{y,(";" sv x),z};
//replace k representation with equivalent q keyword
kreplace:{[x] $[`=qval:.q?x;x;"~~",string[qval],"~~"]};
funcK:{$[0=t:type x;.z.s each x;t<100h;x;kreplace x]};
//replace eg ,`FD`ABC`DEF with "enlist`FD`ABC`DEF"
ereplace:{"~~enlist",(.Q.s1 first x),"~~"};
ereptest:{((0=type x) & (1=count x) & (11=type first x)) | ((11=type x)&(1=count x))};
funcEn:{$[ereptest x;ereplace x;0=type x;.z.s each x;x]};
basic:{tidy .Q.s1 funcK funcEn x};
addbraks:{"(",x,")"};
//where clause needs to be a list of where clauses, so if only one whereclause need to enlist.
stringify:{$[(0=type x) & 1=count x;"enlist ";""],basic x};
//if a dictionary apply to both, keys and values
ab:{$[(0=count x) | -1=type x;.Q.s1 x;99=type x;(addbraks stringify key x),"!",stringify value x;stringify x]};
inner:{[x]
idxs:2 3 4 5 6 inter ainds:til count x;
x:#[x;idxs;'[ab;eval]];
if[6 in idxs;x[6]:ssr/[;("hopen";"hclose");("iasc";"idesc")] x[6]];
//for select statements within select statements
//This line has been adjusted
x[1]:$[-11=type x 1;x 1;$[11h=type x 1;[idxs,:1;"`",string first x 1];[idxs,:1;.z.s x 1]]];
x:#[x;ainds except idxs;string];
x[0],strBrk[1_x;"[";"]"]
};
buildSelect:{[x]
inner parse x
};
We can use this to create the functional query that will work
q)n:1000
q)tab:([]sym:n?`3;col1:n?100.0;col2:n?10.0)
q)buildSelect "select from tab where i=fby[(last;i);([]col1;col2)]"
"?[tab;enlist (=;`i;(fby;(enlist;last;`i);(flip;(lsq;enlist`col1`col2;(enlist;`col1;`col2)))));0b;()]"
So we have the following as the functional form
?[tab;enlist (=;`i;(fby;(enlist;last;`i);(flip;(lsq;enlist`col1`col2;(enlist;`col1;`col2)))));0b;()]
// Applying this
q)?[tab;enlist (=;`i;(fby;(enlist;last;`i);(flip;(lsq;enlist`col1`col2;(enlist;`col1;`col2)))));0b;()]
sym col1 col2
----------------------
bah 18.70281 3.927524
jjb 35.95293 5.170911
ihm 48.09078 5.159796
...
Glad you were able to fix your problem with converting your query to functional form.
Generally it is the case that when you use parse with a fby in your statement, q will convert this function into its k definition. Usually you should just be able to replace this k code with the q function itself (i.e. change (k){stuff} to fby) and this should run properly when turning the query into functional form.
Additionally, if you check out https://code.kx.com/v2/wp/parse-trees/ it goes into more detail about parse trees and functional form. Additionally, it contains a script called buildQuery which will return the functional form of the query of interest as a string which can be quite handy and save time when a functional form is complex.
I actually got it myself ->
?[tableName;((=;`i;(fby;(enlist;last;`i);(+:;(!;enlist`column_one`column_two;(enlist;`column_one;`column_two)))));(in;`venue;enlist`venueone`venuetwo));0b;()]
The issues was a () missing from the statement. Works fine now.
**if someone wants to add a more detailed explanation on how manual parse trees are built and how the generic (k){} function can be replaced with the actual function in q feel free to add your answer and I'll accept and upvote it
I am a new SystemVerilog user and I have faced a strange (from my point of view) behavior of combination of unique method called for fixed array with with operator.
module test();
int arr[12] = '{1,2,1,2,3,4,5,8,9,10,10,8};
int q[$]
initial begin
q = arr.unique() with (item > 5 ? item : 0);
$display("the result is %p",q);
end
I've expected to get queue {8,9,10} but instead I have got {1,8,9,10}.
Why there is a one at the index 0 ?
You are trying to combine the operation of the find method with unique. Unfortunately, it does not work the way you expect. unique returns the element, not the expression in the with clause, which is 0 for elements 1,2,3,4 and 5. The simulator could have chosen any of those elements to represent the unique value for 0(and different simulators do pick different values)
You need to write them separately:
module test();
int arr[$] = '{1,2,1,2,3,4,5,8,9,10,10,8};
int q[$]
initial begin
arr = arr.find() with (item > 5);
q = arr.unique();
$display("the result is %p",q);
end
Update explaining the original results
The with clause generates a list of values to check for uniqueness
'{0,0,0,0,0,0,0,8,9,10,10,8};
^. ^ ^ ^
Assuming the simulator chooses the first occurrence of a replicated value to remain, then it returns {arr[0], arr[7], arr[8], arr[9]} from the original array, which is {1,8,9,10}
I have a table
t: flip `S`V ! ((`$"|A|B|"; `$"|B|C|D|"; `$"|B|"); 1 2 3)
and some dicts
t1: 4 10 15 20 ! 1 2 3 5;
t2: 4 10 15 20 ! 0.5 2 4 5;
Now I need to add a column with values on the the substrings in S and the function below (which is a bit pseudocode because I am stuck here).
f:{[s;v];
if[`A in "|" vs string s; t:t1;];
else if[`B in "|" vs string s; t:t2;];
k: asc key t;
:t k k binr v;
}
problems are that s and v are passed in as full column vectors when I do something like
update l:f[S,V] from t;
How can I make this an operation that works by row?
How can I make this a vectorized function?
Thanks
You will want to use the each-both adverb to apply a function over two columns by row.
In your case:
update l:f'[S;V] from t;
To help with your pseudocode function, you might want to use $, the if-else operator, e.g.
f:{[s;v]
t:$["A"in ls:"|"vs string s;t1;"B"in ls;t2;()!()];
k:asc key t;
:t k k binr v;
};
You've not mentioned a final else clause in your pseudocode but $ expects one hence the empty dictionary at the end.
Also note that in your table the columns S and V have been cast to a symbol. vs expects a string to split so I've had to use the stringoperation - this could be removed if you are able to redefine your original table.
Hope this helps!
The first one below works, but, just wanted to see if there's a better way...
If I'm trying to find all records that start with 'C' with either a flag of 2 or a status of 9, do I need to incorporate the 'C' criteria twice?
i.e.,
"SELECT * FROM mytable WHERE name like 'C%' AND flag = 2 OR name like 'C%' AND status = 9"
Or, is there a way quicker way to write it so that I only need to set 'C%' once?
Logically, AND and OR are distributive to each other, i.e.
a & (b | c) = (a & b) | (a & c)
that means your condition can be rewritten as
name LIKE 'C%' AND (flag = 2 OR status = 9)
OR has lower priority than AND so you need parentheses.
WHERE name LIKE "C%" AND (flag = 2 OR flag = 9)
you can also check for membership of a set of values
... AND flag IN (1, 9)