Aggregate Postgresql rows into groups based on interval - postgresql

I have a result set from a CTE that returns the interval between each row insertion like follows:
row
interval
1
years 0 mons 0 days 1 hours 0 mins 16.0 secs
2
0 years 0 mons 0 days 0 hours 45 mins 42.0 se
3
0 years 0 mons 0 days 0 hours 0 mins 20.0 sec
4
0 years 0 mons 0 days 5 hours 4 mins 19.0 sec
5
0 years 0 mons 0 days 0 hours 2 mins 32.0 sec
6
0 years 0 mons 0 days 0 hours 0 mins 25.0 sec
7
0 years 0 mons 0 days 0 hours 1 mins 9.0 secs
8
0 years 0 mons 0 days 0 hours 0 mins 25.0 sec
9
0 years 0 mons 0 days 0 hours 0 mins 16.0 sec
10
0 years 0 mons 0 days 2 hours 50 mins 24.0 s
11
0 years 0 mons 0 days 1 hours 6 mins 49.0 se
12
0 years 0 mons 0 days 0 hours 4 mins 6.0 sec
13
0 years 0 mons 0 days 0 hours 1 mins 6.0 sec
14
0 years 0 mons 0 days 0 hours 5 mins 48.0 se
15
0 years 0 mons 0 days 0 hours 3 mins 42.0 se
16
0 years 0 mons 0 days 0 hours 0 mins 22.0 se
17
0 years 0 mons 0 days 0 hours 0 mins 30.0 se
18
0 years 0 mons 0 days 0 hours 0 mins 19.0 se
19
0 years 0 mons 0 days 0 hours 0 mins 16.0 se
20
0 years 0 mons 0 days 0 hours 1 mins 55.0 se
What I need to extract is an aggregation of all rows after a cut off value to be grouped together until another cut off initiates a new grouping. For example, say the interval cut off is 1 hour, I want (starting with row 1) rows 1-3 to be grouped together because row 1 is above the cut off and would be the beginning a new grouping. 2&3 would be included the group as well because they are below the cut off. At row 4 a new group would be created because 4 is above the cut off. Then all subsequent rows would be included up to 9. 10 would be its own group because 11 is also above the cutoff and so on.
Thanks very much for any assistance.
insert into tmp (x, delta)
values (1, '0 years 0 mons 0 days 1 hours 0 mins 16.0 secs'),
(2, '0 years 0 mons 0 days 0 hours 45 mins 42.0 secs'),
(3, '0 years 0 mons 0 days 0 hours 0 mins 20.0 secs'),
(4, '0 years 0 mons 0 days 0 hours 4 mins 19.0 secs'),
(5, '0 years 0 mons 0 days 0 hours 2 mins 32.0 secs'),
(6, '0 years 0 mons 0 days 0 hours 0 mins 25.0 secs'),
(7, '0 years 0 mons 0 days 0 hours 1 mins 9.0 secs'),
(8, '0 years 0 mons 0 days 0 hours 0 mins 25.0 secs'),
(9, '0 years 0 mons 0 days 0 hours 0 mins 16.0 secs'),
(10, '0 years 0 mons 0 days 2 hours 50 mins 24.0 secs'),
(11, '0 years 0 mons 0 days 1 hours 6 mins 49.0 secs'),
(12, '0 years 0 mons 0 days 0 hours 4 mins 6.0 secs'),
(13, '0 years 0 mons 0 days 0 hours 1 mins 6.0 secs'),
(14, '0 years 0 mons 0 days 0 hours 5 mins 48.0 secs'),
(15, '0 years 0 mons 0 days 0 hours 3 mins 42.0 secs'),
(16, '0 years 0 mons 0 days 0 hours 0 mins 22.0 secs'),
(17, '0 years 0 mons 0 days 0 hours 0 mins 30.0 secs'),
(18, '0 years 0 mons 0 days 0 hours 0 mins 19.0 secs'),
(19, '0 years 0 mons 0 days 0 hours 0 mins 16.0 secs'),
(20, '0 years 0 mons 0 days 0 hours 1 mins 55.0 secs');

It's unclear to me from your question if 10 consecutive rows with a delta of 10 minutes each should be lumped into one group or two. If you want them all to be one group, you can do something like this:
WITH ranges AS (
SELECT x as start, lead(x) over (order by x) as end
FROM tmp
WHERE delta > interval '1 hour'
)
SELECT ranges.start, string_agg(x::text, ', ')
FROM tmp, ranges
WHERE tmp.x >= ranges.start
AND (ranges.end is null OR tmp.x < ranges.end)
GROUP BY ranges.start, ranges.end
ORDER BY ranges.start;
start | string_agg
-------+----------------------------------------
1 | 1, 2, 3, 4, 5, 6, 7, 8, 9
10 | 10
11 | 11, 12, 13, 14, 15, 16, 17, 18, 19, 20
First we identify the rows with a delta greater than the cutoff value, which represent the cutoff points, and we use a window function to determine the range. Then it's just a matter of aggregating over those ranges.
However, if you want the groups to be no more than one hour long, then that's trickier. I'm not sure if it can be done without a loop.

Related

Convert seconds into 0 years 0 mons 0 days 0 hours 0 mins 0 secs

I'm trying to convert my seconds into the numbers of hours, days,... contained into my time.
I have tried the following request (fiddle):
SELECT TO_CHAR((1670857661 || ' second')::interval, 'YYYY" years" MM" mons "DD" days "HH24" hours "MI" mins "SS" secs"')
Current Output:
0000 years 00 mons 00 days 464127 hours 07 mins 41 secs
Expected output:
54 years 8 mons 12 days 10 hours 51 mins 12 secs
Use justify_interval()
select justify_interval(make_interval(secs => 1670857661));
Online example

Implementation of FIFO pnl in kdb/q

Consider the table below:
Id
Verb
Qty
Price
1
Buy
6
10.0
2
Sell
5
11.0
3
Buy
4
10.0
4
Sell
3
12.0
5
Sell
8
9.0
6
Buy
7
8.0
I would like to compute the PnL in a FIFO way. For example for Id=1, PnL is -6*(10.0) +5*(11.0) + 1*(12.0) = +$7.00. For Id=5, this case is a bit different: our position is +2, and we will firstly fill this position(which will not take account into the PnL of Id=5), then we sell the remaining 6 assets. At Id=6, the -6 position is fulfilled and we get the PnL of Id=5 which is +6*(9.0)-6*(8.0)=+$6.00. Hence this table with PnL is what I want to have :
Id
Verb
Qty
Price
PnL
1
Buy
6
10.0
7.0
2
Sell
5
11.0
0.0
3
Buy
4
10.0
2.0
4
Sell
3
12.0
0.0
5
Sell
8
9.0
6.0
6
Buy
7
8.0
0.0(with 1 asset remaining)
I have read this post and KDB: pnl in FIFO manner and https://code.kx.com/q4m3/1_Q_Shock_and_Awe/#114-example-fifo-allocation. But in their approach, they don't care about the order between buy orders and sell orders, which is not my case.
My idea is to firstly produce the FIFO allocation matrix where the dimension is the trades number:
Id
1
2
3
4
5
6
1
6
0
0
0
0
0
2
1
0
0
0
0
0
3
1
0
4
0
0
0
4
0
0
2
0
0
0
5
0
0
0
0
-6
0
6
0
0
0
0
0
1
Then I compute the diff(price). The inner product of each column and diff(price) is PnL of each trade.
I am having trouble to implement this allocation matrix. Or any advice on solving this problem more directly?
Here's one approach. It's more convoluted than I'd like but it covers a lot of the intermediary steps and generates a type of allocation matrix as you suggested. There are likely edge-cases and tweaks needed but this should give you some ideas at least.
t:([]id:1+til 6;side:`b`s`b`s`s`b;qty:6 5 4 3 8 7;px:10 11 10 12 9 8f);
t:update pos:sums delta from update delta:qty*(1;-1)side=`s from t;
f:{signum[x]*x,{#[(-). z;x;:;abs[y]-sum z 1]}[y;x y]{(x;deltas y&sums x)}[abs where[signum[x]<>signum x y]#x;abs x y]};
t:update fifo:deltas[id!delta;f\[id!delta;id]] from t;
q)update pnl:sum each(id!px)*/:fifo from t
id side qty px delta pos fifo pnl
-----------------------------------------------------
1 b 6 10 6 6 1 2 3 4 5 6!-6 5 0 1 0 0 7
2 s 5 11 -5 1 1 2 3 4 5 6!0 0 0 0 0 0 0
3 b 4 10 4 5 1 2 3 4 5 6!0 0 -4 2 2 0 2
4 s 3 12 -3 2 1 2 3 4 5 6!0 0 0 0 0 0 0
5 s 8 9 -8 -6 1 2 3 4 5 6!0 0 0 0 6 -6 6
6 b 7 8 7 1 1 2 3 4 5 6!0 0 0 0 0 0 0

All pair maximum flow in Matlab

Is there a way to find the maximum flow between each pair of vertices in matlab?
c = sparse([1 1 2 2 3 4 4 5 5 6 7 8 9 9],[2 3 3 4 5 6 7 6 7 8 9 10 8 10],[15 10 3 8 9 7 5 6 2 12 10 6 10 8],10,10)
a = [2 3 4 5 6 7 8 9 10]
b = arrayfun(#(x)max_flow(c,1,x),a)
OR
b = arrayfun(#(x)graphmaxflow(c,1,x),a)
b =
15 13 8 9 13 7 16 7 13
So, I can take a sparse matrix and get the maximum flow from one vertex to all others. Is there a way to continue this to obtain the max flow for all of the pairs?
I'd eventually like to be able to find the all-pair max flow for a directed, weighted graph. . .
Got it to work:
c = sparse([1 1 2 2 3 4 4 5 5 6 7 8 9 9],[2 3 3 4 5 6 7 6 7 8 9 10 8 10],[15 10 3 8 9 7 5 6 2 12 10 6 10 8],10,10)
for a=1:10
for b=1:10
if a==b
continue
end
t(b,a)=graphmaxflow(c,a,b);
p=t(:);
end
end
I couldn't figure out a way to use arrayfun to do this.
Each maximum flow value:
t =
0 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0
13 3 0 0 0 0 0 0 0 0
8 8 0 0 0 0 0 0 0 0
9 3 9 0 0 0 0 0 0 0
13 10 6 7 6 0 0 0 0 0
7 7 2 5 2 0 0 0 0 0
16 11 8 12 8 12 10 0 10 0
7 7 2 5 2 0 10 0 0 0
13 11 8 11 8 6 10 6 14 0
p =
0
15
13
8
9
13
7
...

How to create a matrix with different elements using Matlab [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking for code must demonstrate a minimal understanding of the problem being solved. Include attempted solutions, why they didn't work, and the expected results. See also: Stack Overflow question checklist
Closed 9 years ago.
Improve this question
how to create 9×9 matrix with the first 3 rows all zeros, 4 to 6 rows are all filled with 5, and the remaining rows first elements are 1's and the remaining elements are 5's, using MATLAB?
Here's an answer that'll teach you how to use MATLAB if you're interested enough:
A = bsxfun(#times, ones(9), kron([0 5 5], [1 1 1])') - ...
[kron([0 0 4], [1 1 1])' zeros(9,8)]
result:
A =
0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0
5 5 5 5 5 5 5 5 5
5 5 5 5 5 5 5 5 5
5 5 5 5 5 5 5 5 5
1 5 5 5 5 5 5 5 5
1 5 5 5 5 5 5 5 5
1 5 5 5 5 5 5 5 5
subZero = zeros(3, 9);
subFive = 5*ones(3, 9);
subsubOnes = ones(3, 1);
subsubFive = 5*ones(3, 8);
subOneFive = [subsubOnes subsubFive];
yourMatrix = [subZero; subFive; subOneFive];
Have you tried creating matrix with values at the time of initialization like this:
myMatrix = [...
0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0
5 5 5 5 5 5 5 5 5
5 5 5 5 5 5 5 5 5
5 5 5 5 5 5 5 5 5
1 5 5 5 5 5 5 5 5
1 5 5 5 5 5 5 5 5
1 5 5 5 5 5 5 5 5];
I know there are simpler ways to initialize.

matlab efficient copying of matrix

I have matrix (a) with (1:10),<10 x 1> double. I would like to copy the values and rearrange them column wise into another matrix var. (b). See example below. Also, what method would be most efficient at this task?
matrix a matrix b
1 1
2 2 2
3 3 3 3
4 4 4 4 4
5 5 5 5 5 5
6 6 6 6 6 6 6
7 7 7 7 7 7 7 7
8 8 8 8 8 8 8 8 8
9 9 9 9 9 9 9 9 9 9
10 10 10 10 10 10 10 10 10 10 10
update:
Hi once again Amro. How about if I wanted to define which values to copy. See below example:
matrix a matrix b
column: 1 2 3 4 5 6 7
1 1
2 2 2
3 3 3
4 4
5 5
6 6
7 7
8 8
9 9
10 10 10
Try:
>> a = (1:10)'
a =
1
2
3
4
5
6
7
8
9
10
>> b = tril(repmat(a,1,10))
b =
1 0 0 0 0 0 0 0 0 0
2 2 0 0 0 0 0 0 0 0
3 3 3 0 0 0 0 0 0 0
4 4 4 4 0 0 0 0 0 0
5 5 5 5 5 0 0 0 0 0
6 6 6 6 6 6 0 0 0 0
7 7 7 7 7 7 7 0 0 0
8 8 8 8 8 8 8 8 0 0
9 9 9 9 9 9 9 9 9 0
10 10 10 10 10 10 10 10 10 10
I think in the second matrix you specified you made an error. I'm assuming you wanted to do something like this:
b =
1 0 0 0 0 0
2 2 0 0 0 0
0 3 3 0 0 0
0 0 4 4 0 0
0 0 0 5 5 0
0 0 0 0 6 6
this is simple to do:
%define vector of arbitrary length
a=1:6;
%generate b with shifted diagonal matrices
b=diag(a)+diag(a(2:end),-1);
the second argument of diag just shifts the resulting diagonal.