I have a table of returns that I want to calculate correlation for every n rows, but I am not sure how to do that. To be more illustrative, my table t is
sym1
sym2
sym3
sym4
3
4
5
1
0
-1
6
4
-3
10
8
9
-4
19
-1
6
How can I calculate the correlation between sym1 and sym2/sym3/sym4 for row 1-3, then row 2-4 etc?
Currently I can only specify column then calculate, but I am trying to find a way that I can fix sym1 and iterate through sym2-4. Here's my current code
(cor).'flip flip each {(til x)xprev\:y}[3;]each (sym1;sym2) from t
Here's a solution which may work for you.
q)update s1s2:(cor).'flip{-3#'reverse{-1_x}\[x;y]}[-1+count i]each(sym1;sym2)from t
sym1 sym2 sym3 sym4 s1s2
------------------------------
3 4 5 1
0 -1 6 4 1
-3 10 8 9 -0.5447048
-4 19 -1 6 -0.9751578
This is only one correlation found between columns, namely sym1 & sym2. The first two results can be discarded since they do not have the 3 values required for the correlation.
Related
There are already moving average in kdb/q.
https://code.kx.com/q/ref/avg/#mavg
But how do I compute moving median?
Here is a naive approach. It starts with an empty list and null median and iterates over the list feeding in a new value each time.
Sublist is used fix the window, and this window is passed along with the median as the state of into the next iteration.
At the end scan \ will output the state at every iteration from which we take the median (first element) from each one
mmed:{{(med l;l:neg[x] sublist last[y],z)}[x]\[(0n;());y][;0]}
q)mmed[5;til 10]
0 0.5 1 1.5 2 3 4 5 6 7
q)i:4 9 2 7 0 1 9 2 1 8
q)mmed[3;i]
4 6.5 4 7 2 1 1 2 2 2
There's also a generic "sliding window" function here which you can pass your desired aggregator into: https://code.kx.com/q/kb/programming-idioms/#how-do-i-apply-a-function-to-a-sequence-sliding-window
q)swin:{[f;w;s] f each { 1_x,y }\[w#0;s]}
q)swin[avg; 3; til 10]
0 0.33333333 1 2 3 4 5 6 7 8
q)update newcol:swin[med;10;mycol] from tab
Say I have a list of tables. (sym1, sym2, sym3 etc)
How would I add a new column to each table called Sym containing the table name?
Thank you
You could use something like:
q){#[value x;`Sym;:;x]}each tables[]
+`a`b`c`Sym!(0 1 2 3 4;0 1 2 3 4;0 1 2 3 4;`sym1`sym1`sym1`sym1`sym1)
+`a`b`c`Sym!(0 1 2 3 4;0 1 2 3 4;0 1 2 3 4;`sym2`sym2`sym2`sym2`sym2)
+`a`b`c`Sym!(0 1 2 3 4;0 1 2 3 4;0 1 2 3 4;`sym3`sym3`sym3`sym3`sym3)
If you remove value from the first argument of #, this will update the tables in place.
Otherwise, since this returns a list, you can use indexing to return the table you want from the list:
q)({#[value x;`Sym;:;x]}each tables[])0
a b c Sym
----------
0 0 0 sym1
1 1 1 sym1
2 2 2 sym1
3 3 3 sym1
4 4 4 sym1
Hope this helps,
James
Another way to achieve this :
q){update Sym:x from x}each `sym1`sym2`sym3
q)raze (sym1;sym2;sym3)
p s Sym
----------------
2.08725 75 sym1
2.065687 6 sym1
2.058972 63 sym2
2.095509 62 sym2
2.036151 90 sym3
2.090895 63 sym3
If you are getting these tables (sym1,sym2,sym3) as the output of another function call like :
f each `s1`s2`s3
then I'll suggest updating the function to add the column Sym just before return these individual tables.
f:{
/some logic
update Sym:x from t
}
This will save an operation of adding a new column separately
I have the following vector a:
a=[8,8,9,9,1,1,2,2,3,3,4,4,5,5,6,6,7,7,8,8]
From a I want to delete all "adjacent" repetitions to obtain:
b=[8,9,1,2,3,4,5,6,7,8]
However, when I do:
unique(a,'stable')
ans =
8 9 1 2 3 4 5 6 7
You see, unique only really gets the unique elements of a, whereas what I want is to delete the "duplicates"... How do I do this?
It looks like a run-length-encoding problem (check here). You can modify Mohsen's solution to get the desired output. (i.e. I claim no credit for this code, yet the question is not a duplicate in my opinion).
Here is the code:
a =[8,8,9,9,1,1,2,2,3,3,4,4,5,5,6,6,7,7,8,8]
F=find(diff([a(1)-1, a]));
Since diff(a) returns an array of length (length(a) -1), we want to add a value at the beginning (i.e the a(1)) to get a vector the same size as a. Here we subtract 1 so that, as mentioned by #surgical_tubing, the command find effectively finds it because it looks for non zero elements, so we want to make sure the value is non zero.
Hence diff([a(1)-1, a]) looks like this:
Columns 1 through 8
1 0 1 0 -8 0 1 0
Columns 9 through 16
1 0 1 0 1 0 1 0
Columns 17 through 20
1 0 1 0
Now having found the repeated elements, we index back into a with the positions found by find:
newa=a(F)
and output:
newa =
Columns 1 through 8
8 9 1 2 3 4 5 6
Columns 9 through 10
7 8
Say I have a list (1 3 4) and after 1 I want to insert another element 2 resulting in (1 2 3 4).
How can this be done efficiently in a generic way?
An alternative approach which allows for multiple inserts at once.
If the indices are to index the original list:
q){raze cut[(0,z);x],'(y,enlist ())}[til 10;999 998 994;2 4 8]
0 1 999 2 3 998 4 5 6 7 994 8 9
If the indices are to index consecutive iterations of the list:
q){raze cut[(0,z);x],'(y,enlist ())}/[til 10;999 998 994;2 4 8]
0 1 999 2 998 3 4 5 994 6 7 8 9
I think you need to be more specific about what you want, but for now here's an example of how you could achieve it
q)list:1 3 4
q)list
1 3 4
q)list: asc list,:2
q)list
`s#1 2 3 4
Or another way is let's say you know the index at which you want to add the element to the list, in this case at index 1, then you could create a function as such:
q)add:{[lst;ele;ind] (ind#lst),ele,(ind _ lst)}
q)list:1 3 4
q)add[list;2;1]
1 2 3 4
I've got a matrix (n x m). And I'd like to know, for each row, the indexes of the coloums that contain the first two maximum values:
2 3 4 2
2 4 7 1
1 1 2 4
5 5 9 6
1 4 2 1
9 8 1 2
The answer should be:
2 3
2 3
3 4
3 4
2 3
1 2
How can I obtain it with matlab commands? I'd like not to use for loops. I tried with:
[x,y]=max(matrix')
y=y';
y gives me the colum indexes for the maximum elements. Now I'd set to zero these elements and repeat the instructions but I have no idea how to do. I treid:
matrix(:,y)=0;
but it doesn't work.
if A is your matrix, then sort and pick the top two indices,
[a ix]=sort(A,2)
ans= ix(:,end-1:end)