I have one table with in y the regions, while in several years (2010, 2011, 2012, 2013 etc). Each year has one value for each region. I would like to nomalize the data of each column. Is it possible to normalize using the mean of the year? I tried this, but I am not sure.
t=readtable('Cartel3.xlsx')
t_norm a= normalize((t{2:end, 2:end}),'norm')
I have two tables with different data. I would like to compare them.I was asked to use the annual average to normalize. But it is not clear for me, the formula.
You can use normalize directly on your table providing you specify the 'DataVariables' parameter, like this:
>> t = table(["A";"B";"C"], [1;2;3], [0;10;20], 'VariableNames', {'Name', '2011', '2012'})
t =
3x3 table
Name 2011 2012
____ ____ ____
"A" 1 0
"B" 2 10
"C" 3 20
>> normalize(t, 'DataVariables', t.Properties.VariableNames(2:end))
ans =
3x3 table
Name 2011 2012
____ ____ ____
"A" -1 -1
"B" 0 0
"C" 1 1
There's a specific example like this on the doc page.
Related
I have a table
t:([]a:`a`b`c;b:1 2 3;c:`x`y`z)
I would like to iterate and process each row.
The thing is that the processing logic for each row may result in arbitrary lines of data, after the full iteration the result maybe as such e.g.
results:([]a:`a1`b1`b2`b3`c1`c2;x:1 2 2 2 3 3)
I have the following idea so far but doesn't seem to work:
uj { // some processing function } each t
But how does one return arbitrary number of data append the results into a new table?
Assuming you are using something from the table entries to indicate your arbitrary value, you can use a dictionary to indicate a number (or a function) which can be used to apply these values.
In this example, I use the c column of the original table to indicate the number of rows to return (and the number from 1 to count to).
As each entry of the table is a dictionary, I can index using the column names to get the values and build a new table.
I also use raze to join each of the results together, as they will each have the same schema.
raze {[x]
d:`x`y`z!1 3 2;
([]a:((),`$string[x[`a]],/:string 1+til d[x[`c]]);x:((),d[x[`c]])#x[`b])
} each t
Not sure if this is what you want, but you can try something like this:
ungroup select a:`${y,/:x}[string b]'[string a],b from t
Or you can use accumulators if you need the result of the previous row calculations like this:
{y[`b]+:last[x]`b;x,y}/[t;t]
If your processing function is outputting tables that conform, just raze should suffice:
raze {y#enlist x}'[t;1 3 2]
a b c
-----
a 1 x
b 2 y
b 2 y
b 2 y
c 3 z
c 3 z
Otherwise use (uj/)
(uj/) {y#enlist x}'[t;1 3 2]
a b c
-----
a 1 x
b 2 y
b 2 y
b 2 y
c 3 z
c 3 z
Your best answer will depend very much on how you want to use the results computed from each row of t. It might suit you to normalise t; it might not. The key point here:
A table cell can be any q data structure.
The minimum you can do in this regard is to store the result of your processing function in a new column.
Below, an arbitrary binary function f returns its result as a dictionary.
q)f:{n:1+rand 3;(`$string[x],/:"123" til n)!n#y}
q)f [`a;2]
a1| 2
a2| 2
q)update d:a f'b from t
a b c d
---------------------
a 1 x `a1`a2`a3!1 1 1
b 2 y (,`b1)!,2
c 3 z `c1`c2!3 3
But its result could be any q data structure.
You were considering a unary processing function:
q)pf:{#[x;`d;:;] f . x`a`b}
q)pf each t
a b c d
---------------------
a 1 x `a1`a2`a3!1 1 1
b 2 y `b1`b2!2 2
c 3 z `c1`c2`c3!3 3 3
You might find other suggestions at KX Community.
If I understand correctly your question you need something like this :
(uj/){}each t
Check this bit :
(uj/)enlist[t],{x:update x:i from?[rand[20]#enlist x;();0b;{x!x}rand[4]#cols[x]];{(x;![x;();0b;(enlist`a)!enlist($;enlist`;((';{raze string(x;y)});`a;`i))])[y~`a]}/[x;cols x]}each t
This part :
x:update x:i from
// functional form of a function that takes random rows/columns
?[rand[20]#enlist x;();0b;{x!x}rand[4]#cols[x]];
// some for of if-else and an update to generate column a (not bullet proof)
{(x;![x;();0b;(enlist`a)!enlist($;enlist`;((';{raze string(x;y)});`a;`i))])[y~`a]}/[x;cols x]
Basically the above gives something like :
q){x:update x:i from?[rand[20]#enlist x;();0b;{x!x}rand[4]#cols[x]];{(x;![x;();0b;(enlist`a)!enlist($;enlist`;((';{raze string(x;y)});`a;`i))])[y~`a]}/[x;cols x]}each t
+`a`b`c`x!(`a0`a1`a2`a3`a4`a5`a6`a7;1 1 1 1 1 1 1 1;`x`x`x`x`x`x`x`x;0 1 2 3 ..
+`a`x!(`a0`a1`a2`a3`a4`a5;0 1 2 3 4 5)
+`a`b`c`x!(`a0`a1`a2;1 1 1;`x`x`x;0 1 2)
+`a`b`c`x!(`a0`a1`a2`a3`a4`a5`a6`a7`a8`a9`a10`a11;1 1 1 1 1 1 1 1 1 1 1 1;`x`..
or taking the first one :
q)first{x:update x:i from?[rand[20]#enlist x;();0b;{x!x}rand[4]#cols[x]];{(x;![x;();0b;(enlist`a)!enlist($;enlist`;((';{raze string(x;y)});`a;`i))])[y~`a]}/[x;cols x]}each t
a b x
--------
a0 1 0
a1 1 1
a2 1 2
a3 1 3
a4 1 4
a5 1 5
a6 1 6
a7 1 7
a8 1 8
a9 1 9
a10 1 10
You can do
(uj/)enist[t],{ // some function }each t
to get what you want. Drop the enlist[t] if you don't want the table you start with in your result
Hope this helps.
I have two tables
table 1 (orders) columns: (date,symbol,qty)
table 2 (marketData) columns: (date,symbol,close price)
I want to add the close for T+0 to T+5 to table 1.
{[nday]
value "temp0::update date",string[nday],":mdDates[DateInd+",string[nday],"] from orders";
value "temp::temp0 lj 2! select date",string[nday],":date,sym,close",string[nday],":close from marketData";
table1::temp
} each (1+til 5)
I'm sure there is a better way to do this, but I get a 'loop error when I try to run this function. Any suggestions?
See here for common errors. Your loop error is because you're setting views with value, not globals. Inside a function value evaluates as if it's outside the function so you don't need the ::.
That said there's lots of room for improvement, here's a few pointers.
You don't need the value at all in your case. E.g. this line:
First line can be reduced to (I'm assuming mdDates is some kind of function you're just dropping in to work out the date from an integer, and DateInd some kind of global):
{[nday]
temp0:update date:mdDates[nday;DateInd] from orders;
....
} each (1+til 5)
In this bit it just looks like you're trying to append something to the column name:
select date",string[nday],":date
Remember that tables are flipped dictionaries... you can mess with their column names via the keys, as illustrated (very noddily) below:
q)t:flip `a`b!(1 2; 3 4)
q)t
a b
---
1 3
2 4
q)flip ((`$"a","1"),`b)!(t`a;t`b)
a1 b
----
1 3
2 4
You can also use functional select, which is much neater IMO:
q)?[t;();0b;((`$"a","1"),`b)!(`a`b)]
a1 b
----
1 3
2 4
Seems like you wanted to have p0 to p5 columns with prices corresponding to date+0 to date+5 dates.
Using adverb over to iterate over 0 to 5 days :
q)orders:([] date:(2018.01.01+til 5); sym:5?`A`G; qty:5?10)
q)data:([] date:20#(2018.01.01+til 10); sym:raze 10#'`A`G; price:20?10+10.)
q)delete d from {c:`$"p",string[y]; (update d:date+y from x) lj 2!(`d`sym,c )xcol 0!data}/[ orders;0 1 2 3 4]
date sym qty p0 p1 p2 p3 p4
---------------------------------------------------------------
2018.01.01 A 0 10.08094 6.027448 6.045174 18.11676 1.919615
2018.01.02 G 3 13.1917 8.515314 19.018 19.18736 6.64622
2018.01.03 A 2 6.045174 18.11676 1.919615 14.27323 2.255483
2018.01.04 A 7 18.11676 1.919615 14.27323 2.255483 2.352626
2018.01.05 G 0 19.18736 6.64622 11.16619 2.437314 4.698096
I have a MatLab "struct", with different "level" and "sub-structures". When printed to a cell, the data contained in the "struct", look like that:
report.COUNTRY.SOURCE.SCENARIO.CATEGORY.ENTITY = YEAR YEAR ...;
As a minimal example:
report.HUN.CRF2014.BASEYEAR.CAT0.CO2 = 1991 1992 1993 1994
report.HUN.CRF2014.BASEYEAR.CAT0.CH4 = 1995 1996 1997
report.HUN.CRF2014.BASEYEAR.CAT0.H2S = 1990 1991 1992
report.HUN.CRF2014.BASEYEAR.CAT1.N2 = 1991 1992 1993
report.HUN.CRF2014.BASEYEAR.CAT1.FGASES = 1990 1991 1992
In order to produce tables listing the different variables combinations, I would like to loop over the fieldnames contained within the "struct".
I am currently trying to write a function able to do that:
fields=fieldnames(struct);
for categoryidx=1:length(fields)
categoryname=fields{categoryidx};
if isstruct(struct.(categoryname))
category=fieldnames(struct.(categoryname));
for entityidx = 1:length(category);
entityname = category{entityidx};
if isstruct(struct.(categoryname).(entityname))
gases=fieldnames(struct.(categoryname).(entityname));
end
end
end
end
Unfortunately, this is just producing anything! Does anyone has any idea how to loop over fieldnames in such a matlab structure? Thank you!
You might want to check out:
struct2tabler. This is a MATLAB function that recursively goes through a structure to convert into a table.
For example:
a.a = 5
a.b.c = 10
a.b.d = 15
Would return a table:
a_a a_b_c a_b_d
---------------------------
5 10 15
Disclaimer: I have written struct2tabler, so might be a little biased, however it was created out of a requirement, I think, very similar to yours.
I have 7 different datasets that all contain the exact same columns (same names and same type of data) and I want to add them all together into one big dataset - they are all files from the same database but were originally extracted in 7 different smaller files due to size.
So I have:
df1
var1 var2 var3
NY 300 abc
LA 500 def
And
df2
var1 var2 var3
MI 200 ght
OR 900 pth
And so on with df3 to df7. I'd like to attach them one after the other to get:
Big.Dataset
var1 var2 var3
NY 300 abc
LA 500 def
MI 200 ght
OR 900 pth
I tried the following:
Big.Dataset <- rbind(df1, df2, df3, df4, df5, df6, df7)
But when I then do
View(Big.Dataset)
All I can see is 1 row and 1 column. So clearly, the function didn't work.
What is it exactly that I'm doing wrong? Is rbind the right function? Is it something wrong with my files that I need to fix before doing rbind perhaps?
Solved. The issue was that I had to define the variables first - as.numeric or as.character. That way R could recognise what it had to do.
If I have:
t=(1:1:5)'
time=1:3:100
How do I arrange data t in each column starting from 1 until the end, with an interval of 3. Which means that the data t (1 to 5) at column 1,4,7 and so on.
I've tried:
t=[1:1:5];
nt=length(temp);
time=[1:1:100];
nti=length(time);
x=zeros(nt,nti);
temp=temp';
initiator=2;
monomer=3;
post=1:3:100;
for l=1:post
step=1;
maxstep=100;
while (step<maxstep)
step=step+3;
temp=(1:1:5)';
end
t(:,l)=t;
x=[t];
end
This only shows result X with temp at column 1. I do not know how to to arrange this data at columns that I want.
Hope someone will help me. Thank you in advance.
How many dimensions does your data have? If you already have "temp" (temperature?) and "time" as your first two dimensions and you want "t" to be the third dimension, then create a three-dimension matrix.
To extract from indexes [1 4 7 10 13 16 ... ], use (1:3:end)
To extract from indexed [2 5 8 11 14 17 ... ], use (2:3:end)
In MATLAB's colon notation, the first value is the start. Second value is increment. Third value is the end value and is inclusive.