LWP::UserAgent memory usage while saving content - perl
EDIT: The issue below was produced on stock Perl 5.16.3 on an AWS ec2 micro instance. It was ultimately solved by installing and switching to Perl 5.24.3 with perlbrew.
LWP::UserAgent seems to be causing some kind of memory leak when saving content.
use v5.10;
use strict;
use warnings;
use IO::Socket::SSL;
use LWP::UserAgent;
use Memory::Usage;
use Time::HiRes 'usleep';
my $ua = LWP::UserAgent->new;
my $mu = Memory::Usage->new;
$ua->ssl_opts(SSL_ocsp_mode => SSL_OCSP_NO_STAPLE); ## disable OCSP stapling
for (1..50) {
$mu->record("before request: $_");
$ua->get('http://www.wikipedia.org/wiki/Special:Random', ':content_file' => 'temp.html');
usleep 50000;
}
$mu->dump();
Output:
time vsz ( diff) rss ( diff) shared ( diff) code ( diff) data ( diff)
0 213964 ( 213964) 25112 ( 25112) 7532 ( 7532) 8 ( 8) 17976 ( 17976) before request: 1
0 225744 ( 11780) 31428 ( 6316) 8120 ( 588) 8 ( 0) 23580 ( 5604) before request: 2
0 226824 ( 1080) 32500 ( 1072) 8120 ( 0) 8 ( 0) 24660 ( 1080) before request: 3
0 227920 ( 1096) 33596 ( 1096) 8120 ( 0) 8 ( 0) 25756 ( 1096) before request: 4
0 229016 ( 1096) 34692 ( 1096) 8120 ( 0) 8 ( 0) 26852 ( 1096) before request: 5
1 230112 ( 1096) 35784 ( 1092) 8120 ( 0) 8 ( 0) 27948 ( 1096) before request: 6
1 231204 ( 1092) 36880 ( 1096) 8120 ( 0) 8 ( 0) 29040 ( 1092) before request: 7
1 232296 ( 1092) 37968 ( 1088) 8120 ( 0) 8 ( 0) 30132 ( 1092) before request: 8
1 233388 ( 1092) 39060 ( 1092) 8120 ( 0) 8 ( 0) 31224 ( 1092) before request: 9
1 234484 ( 1096) 40152 ( 1092) 8120 ( 0) 8 ( 0) 32320 ( 1096) before request: 10
1 235572 ( 1088) 41248 ( 1096) 8120 ( 0) 8 ( 0) 33408 ( 1088) before request: 11
2 236668 ( 1096) 42344 ( 1096) 8120 ( 0) 8 ( 0) 34504 ( 1096) before request: 12
2 237764 ( 1096) 43440 ( 1096) 8120 ( 0) 8 ( 0) 35600 ( 1096) before request: 13
2 238864 ( 1100) 44536 ( 1096) 8120 ( 0) 8 ( 0) 36700 ( 1100) before request: 14
2 239956 ( 1092) 45632 ( 1096) 8120 ( 0) 8 ( 0) 37792 ( 1092) before request: 15
2 241044 ( 1088) 46720 ( 1088) 8120 ( 0) 8 ( 0) 38880 ( 1088) before request: 16
3 242140 ( 1096) 47816 ( 1096) 8120 ( 0) 8 ( 0) 39976 ( 1096) before request: 17
3 243240 ( 1100) 48916 ( 1100) 8120 ( 0) 8 ( 0) 41076 ( 1100) before request: 18
3 244336 ( 1096) 50008 ( 1092) 8120 ( 0) 8 ( 0) 42172 ( 1096) before request: 19
3 245428 ( 1092) 51096 ( 1088) 8120 ( 0) 8 ( 0) 43264 ( 1092) before request: 20
3 246516 ( 1088) 52196 ( 1100) 8120 ( 0) 8 ( 0) 44352 ( 1088) before request: 21
4 247612 ( 1096) 53288 ( 1092) 8120 ( 0) 8 ( 0) 45448 ( 1096) before request: 22
4 248704 ( 1092) 54380 ( 1092) 8120 ( 0) 8 ( 0) 46540 ( 1092) before request: 23
4 249796 ( 1092) 55476 ( 1096) 8120 ( 0) 8 ( 0) 47632 ( 1092) before request: 24
4 250892 ( 1096) 56568 ( 1092) 8120 ( 0) 8 ( 0) 48728 ( 1096) before request: 25
4 251988 ( 1096) 57660 ( 1092) 8120 ( 0) 8 ( 0) 49824 ( 1096) before request: 26
4 253076 ( 1088) 58752 ( 1092) 8120 ( 0) 8 ( 0) 50912 ( 1088) before request: 27
5 254168 ( 1092) 59852 ( 1100) 8120 ( 0) 8 ( 0) 52004 ( 1092) before request: 28
5 255272 ( 1104) 60940 ( 1088) 8120 ( 0) 8 ( 0) 53108 ( 1104) before request: 29
5 256360 ( 1088) 62036 ( 1096) 8120 ( 0) 8 ( 0) 54196 ( 1088) before request: 30
5 257456 ( 1096) 63132 ( 1096) 8120 ( 0) 8 ( 0) 55292 ( 1096) before request: 31
5 258556 ( 1100) 64236 ( 1104) 8120 ( 0) 8 ( 0) 56392 ( 1100) before request: 32
6 259652 ( 1096) 65324 ( 1088) 8120 ( 0) 8 ( 0) 57488 ( 1096) before request: 33
6 260740 ( 1088) 66416 ( 1092) 8120 ( 0) 8 ( 0) 58576 ( 1088) before request: 34
6 261832 ( 1092) 67504 ( 1088) 8120 ( 0) 8 ( 0) 59668 ( 1092) before request: 35
6 262928 ( 1096) 68600 ( 1096) 8120 ( 0) 8 ( 0) 60764 ( 1096) before request: 36
6 264016 ( 1088) 69692 ( 1092) 8120 ( 0) 8 ( 0) 61852 ( 1088) before request: 37
7 265120 ( 1104) 70796 ( 1104) 8120 ( 0) 8 ( 0) 62956 ( 1104) before request: 38
7 266212 ( 1092) 71888 ( 1092) 8120 ( 0) 8 ( 0) 64048 ( 1092) before request: 39
7 267304 ( 1092) 72980 ( 1092) 8120 ( 0) 8 ( 0) 65140 ( 1092) before request: 40
7 268396 ( 1092) 74072 ( 1092) 8120 ( 0) 8 ( 0) 66232 ( 1092) before request: 41
7 269484 ( 1088) 75168 ( 1096) 8120 ( 0) 8 ( 0) 67320 ( 1088) before request: 42
7 270584 ( 1100) 76260 ( 1092) 8120 ( 0) 8 ( 0) 68420 ( 1100) before request: 43
8 271676 ( 1092) 77356 ( 1096) 8120 ( 0) 8 ( 0) 69512 ( 1092) before request: 44
8 272776 ( 1100) 78448 ( 1092) 8120 ( 0) 8 ( 0) 70612 ( 1100) before request: 45
8 273868 ( 1092) 79540 ( 1092) 8120 ( 0) 8 ( 0) 71704 ( 1092) before request: 46
8 274964 ( 1096) 80632 ( 1092) 8120 ( 0) 8 ( 0) 72800 ( 1096) before request: 47
8 276052 ( 1088) 81728 ( 1096) 8120 ( 0) 8 ( 0) 73888 ( 1088) before request: 48
9 277144 ( 1092) 82816 ( 1088) 8120 ( 0) 8 ( 0) 74980 ( 1092) before request: 49
9 278232 ( 1088) 83912 ( 1096) 8120 ( 0) 8 ( 0) 76068 ( 1088) before request: 50
Perl version is 5.16.3 ; Modules versions below using cpan -D
CPAN: Module::CoreList loaded ok (v5.20170923)
(no description)
S/SU/SULLR/IO-Socket-SSL-2.056.tar.gz
/home/ec2-user/perl5/lib/perl5/IO/Socket/SSL.pm
Installed: 2.056
CPAN: 2.056 up to date
Steffen Ullrich (SULLR)
Steffen_Ullrich#genua.de
...
CPAN: Module::CoreList loaded ok (v5.20170923)
(no description)
M/MI/MIKEM/Net-SSLeay-1.85.tar.gz
/home/ec2-user/perl5/lib/perl5/x86_64-linux-thread-multi/Net/SSLeay.pm
Installed: 1.85
CPAN: 1.85 up to date
Mike McCauley (MIKEM)
mikem#airspayce.com
...
CPAN: Module::CoreList loaded ok (v5.20170923)
(no description)
O/OA/OALDERS/libwww-perl-6.33.tar.gz
/home/ec2-user/perl5/lib/perl5/LWP/UserAgent.pm
Installed: 6.33
CPAN: 6.33 up to date
Olaf Alders (OALDERS)
olaf#wundersolutions.com
...
CPAN: Module::CoreList loaded ok (v5.20170923)
(no description)
O/OA/OALDERS/LWP-Protocol-https-6.07.tar.gz
/home/ec2-user/perl5/lib/perl5/LWP/Protocol/https.pm
Installed: 6.07
CPAN: 6.07 up to date
Olaf Alders (OALDERS)
olaf#wundersolutions.com
There is a memory leak in the OCSP handling which is relevant if the target site uses OCSP stapling (only in this case is revocation checking done by default). A first leak in this area was fixed in Net::SSLeay1 1.83. But I've found a second leak which is not fixed yet although a patch is available (since a few minutes). See issue 125273 for details of the problem and the patch.
It is also possible to work around this problem by disabling OCSP stapling:
use IO::Socket::SSL;
use LWP::UserAgent;
my $ua = LWP::UserAgent->new;
$ua->ssl_opts(SSL_ocsp_mode => SSL_OCSP_NO_STAPLE); ## disable OCSP stapling
$ua->get('http://www.wikipedia.org', ':content_file' => 'save.txt') for 0..500;
1 Net::SSLeay is the underlying library for IO::Socket::SSL which itself provides SSL functionality to LWP.
Related
How to write DAX function to calculate database growth? DATESBETWEEN not gives the answer what I want
Database part
I have created a table:
Here is the postgresql query that I have use to create this table,
INSERT INTO growth_test.trax_qa_db_growth
SELECT
now()::date as executed_date,
schemaname as table_schema,
relname as table_name,
pg_size_pretty(pg_relation_size(relid)) as data_size,
pg_relation_size(relid) as full_data_size
FROM pg_catalog.pg_statio_user_tables
WHERE schemaname='production'
order by pg_relation_size(relid) desc;
Power BI part
I have to get the data in this table to powerbi. I want to visualize the size of db growth based on "full_data_size" column based on specific two dates using sliser. How can I visualize dbgrowth based on specific takes?
I have tried the "DATESBETWEEN" but that not give the output that I want.
Start Date & End Date are the measures that I have create for the sliser,
DAX function
Data Size in the period =
VAR total =
SUM ( trax_qa_db_growth[full_data_size] ) + 0
RETURN
IF (
total < 1024,
CALCULATE(FORMAT ( total, "#0.0# B" ),DATESBETWEEN('trax_qa_db_growth'[executed_date],[Start Date],[End Date])),
IF (
total < POWER ( 2, 20 ),
CALCULATE(FORMAT ( total / POWER ( 2, 10 ), "#0.0# KB" ),DATESBETWEEN('trax_qa_db_growth'[executed_date],[Start Date],[End Date])),
IF (
total < POWER ( 2, 30 ),
CALCULATE(FORMAT ( total / POWER ( 2, 20 ), "#0.0# MB" ),DATESBETWEEN('trax_qa_db_growth'[executed_date],[Start Date],[End Date])),
IF (
total < POWER ( 2, 40 ),
CALCULATE(FORMAT ( total / POWER ( 2, 30 ), "#0.0# GB" ),DATESBETWEEN('trax_qa_db_growth'[executed_date],[Start Date],[End Date])),
CALCULATE(FORMAT ( total / POWER ( 2, 40 ), "#0.0# TB" ),DATESBETWEEN('trax_qa_db_growth'[executed_date],[Start Date],[End Date]))
)
)
))
This DAX function didn't provide me the result the way I want. Instead it shows the size from full_db_size.
So, I have tried some different method. I hope that will be the correct way.
New DAX function
Data Size in the period =
VAR total =
SUM ( trax_qa_db_growth[full_data_size] ) + 0
RETURN
IF (
total < 1024,
CALCULATE(FORMAT ( total, "#0.0# B" )),
IF (
total < POWER ( 2, 20 ),
CALCULATE(FORMAT ( total / POWER ( 2, 10 ), "#0.0# KB" ),FILTER(trax_qa_db_growth, trax_qa_db_growth[executed_date] = [End Date]))-CALCULATE(FORMAT ( total / POWER ( 2, 10 ), "#0.0# KB" ),FILTER(trax_qa_db_growth, trax_qa_db_growth[executed_date] = [Start Date])),
IF (
total < POWER ( 2, 30 ),
CALCULATE(FORMAT ( total / POWER ( 2, 20 ), "#0.0# MB" ),FILTER(trax_qa_db_growth, trax_qa_db_growth[executed_date] = [End Date]))-CALCULATE(FORMAT ( total / POWER ( 2, 10 ), "#0.0# KB" ),FILTER(trax_qa_db_growth, trax_qa_db_growth[executed_date] = [Start Date])),
IF (
total < POWER ( 2, 40 ),
CALCULATE(FORMAT ( total / POWER ( 2, 30 ), "#0.0# GB" ),FILTER(trax_qa_db_growth, trax_qa_db_growth[executed_date] = [End Date]))-CALCULATE(FORMAT ( total / POWER ( 2, 10 ), "#0.0# KB" ),FILTER(trax_qa_db_growth, trax_qa_db_growth[executed_date] = [Start Date])),
CALCULATE(FORMAT ( total / POWER ( 2, 40 ), "#0.0# TB" ),FILTER(trax_qa_db_growth, trax_qa_db_growth[executed_date] = [End Date]))-CALCULATE(FORMAT ( total / POWER ( 2, 10 ), "#0.0# KB" ),FILTER(trax_qa_db_growth, trax_qa_db_growth[executed_date] = [Start Date]))
)
)
))
What is the issue with the "New DAX function"? How to calculate the DB growth?
Is there any way to calculate db growth in postgresql? (In the oracle DB has that type of options)
torchtext TabularDataset returns ValueError: too many values to unpack (expected
I just exported a dataset with
df[['src', 'trg']].to_csv('dataset.csv', index=False, quoting=csv.QUOTE_ALL)
I checked that there are no commas at all in the cells of the table. However, when I try to read the file with
from torchtext.data import TabularDataset
dataset = TabularDataset(os.path.abspath('dataset.csv'), format='csv', fields=['src', 'trg'])
I get:
ValueError: too many values to unpack (expected 2)
The data looks like this:
$ head dataset.csv
"src","trg"
"S( CC) /C(=N\ [H] ) N","[H] /N=C(/ N) S CC"
"[CH2:0] 1 [CH2:0] [N:0] ( [CH2:0] [CH:0] 2 [CH2:0] [N:0] ( [C:0] ( [O:0] [CH3:0] ) = [O:0] ) [CH2:0] [CH2:0] [N:0] 2 [C:0] ( [CH2:0] [c:0] 2 [cH:0] [c:0] ( [Cl:0] ) [c:0] ( [Cl:0] ) [cH:0] [cH:0] 2) = [O:0] ) [CH2:0] [CH2:0] 1","[CH3:0] [O:0] [C:0] ( = [O:0] ) [N:0] 1 [CH2:0] [CH2:0] [N:0] ( [C:0] ( = [O:0] ) [CH2:0] [c:0] 2 [cH:0] [cH:0] [c:0] ( [Cl:0] ) [c:0] ( [Cl:0] ) [cH:0] 2) [CH:0] ( [CH2:0] [N:0] 2 [CH2:0] [CH2:0] [CH2:0] [CH2:0] 2) [CH2:0] 1"
Notably, the cells contain \'s.
from torchtext.data import TabularDataset, Field
dataset = TabularDataset('dataset.csv', format='csv', fields=[('src', Field()), ('trg', Field())])
worked. For fields a list of tuples had to be provided.
Optimize SQL statement for this Query?
I have two tables T1 and T2. T1 have ID,F1,F2,F3,F4,F5,F6,F7,F8 T2 have ID,T1_ID,F1,F2,F3,F4,F5,F6,F7,F8,SUM Examples Data for T1 and T2 T1 ID,F1,F2,F3,F4,F5,F6,F7,F8 1, 1, 2, 3, 0, 0, 5, 0, 0 2, 0, 0, 0, 1, 0, 4, 5, 0 3, 4, 1, 3, 2, 0, 0, 0, 5 4, 1 ,3, 4, 0, 0 ,0, 0, 0 5, 7, 2, 1, 3, 0, 0, 0, 0 . . . T2 ID,T1_ID,F1,F2,F3,F4,F5,F6,F7,F8,SUM 1, 1, 2, 3, 5, 0, 0, 3, 0, 0,100 2, 5, 9, 8, 8, 1, 0, 0, 0, 0,200 3, 2, 0, 0, 0, 5, 0, 6, 6, 0,300 4, 1 ,3, 4, 2, 0 ,0, 3, 0, 0,255 5, 4, 8, 8, 8, 0, 0, 0, 0, 0,155 . . Select * from T2 where T1.F1....T1.F8 have (1 and 2 and 3) query must return records 1,2,4 1, 1, 2, 3, 5, 0, 0, 3, 0, 0,100 2, 5, 9, 8, 8, 1, 0, 0, 0, 0,200 4, 1 ,3, 4, 2, 0 ,0, 3, 0, 0,255 I create this query Select T2.ID,T2.F1,T2.F2,T2.F3,T2.F4.T2.F5,T2.F6,T2.F7,T2.F8,T2.SUM,T1.ID from T2 join T1 on T1.ID = T2.T1_ID where (CASE WHEN ( T1_ID.F1 = 1 ) THEN T2.F1 between 0 and 1000 end) and(CASE WHEN ( T1_ID.F1 = 2 ) THEN T2.F1 between 0 and 1000 end) and(CASE WHEN ( T1_ID.F1 = 3 ) THEN T2.F1 between 0 and 1000 end) or (CASE WHEN ( T1_ID.F1 = 1 ) THEN T2.F1 between 0 and 1000 end) and(CASE WHEN ( T1_ID.F2 = 2 ) THEN T2.F2 between 0 and 1000 end) and(CASE WHEN ( T1_ID.F1 = 3 ) THEN T2.F1 between 0 and 1000 end) or (CASE WHEN ( T1_ID.F1 = 1 ) THEN T2.F1 between 0 and 1000 end) and(CASE WHEN ( T1_ID.F3 = 2 ) THEN T2.F3 between 0 and 1000 end) and(CASE WHEN ( T1_ID.F1 = 3 ) THEN T2.F1 between 0 and 1000 end) or (CASE WHEN ( T1_ID.F1 = 1 ) THEN T2.F1 between 0 and 1000 end) and(CASE WHEN ( T1_ID.F4 = 2 ) THEN T2.F4 between 0 and 1000 end) and(CASE WHEN ( T1_ID.F1 = 3 ) THEN T2.F1 between 0 and 1000 end) . . . or (CASE WHEN ( T1_ID.F2 = 1 ) THEN T2.F2 between 0 and 1000 end) and(CASE WHEN ( T1_ID.F1 = 2 ) THEN T2.F1 between 0 and 1000 end) and(CASE WHEN ( T1_ID.F1 = 3 ) THEN T2.F1 between 0 and 1000 end) or (CASE WHEN ( T1_ID.F2 = 1 ) THEN T2.F2 between 0 and 1000 end) and(CASE WHEN ( T1_ID.F2 = 2 ) THEN T2.F2 between 0 and 1000 end) and(CASE WHEN ( T1_ID.F1 = 3 ) THEN T2.F1 between 0 and 1000 end) or (CASE WHEN ( T1_ID.F2 = 1 ) THEN T2.F2 between 0 and 1000 end) and(CASE WHEN ( T1_ID.F3 = 2 ) THEN T2.F3 between 0 and 1000 end) and(CASE WHEN ( T1_ID.F1 = 3 ) THEN T2.F1 between 0 and 1000 end) . . . this is too big statement. How can I optimize statement ?
SELECT * FROM T2 WHERE EXISTS ( SELECT N.ID FROM( SELECT T1.ID , T1.F1 AS F FROM T1 UNION ALL SELECT T1.ID , T1.F2 AS F FROM T1 UNION ALL SELECT T1.ID , T1.F3 AS F FROM T1 UNION ALL SELECT T1.ID , T1.F4 AS F FROM T1 UNION ALL SELECT T1.ID , T1.F5 AS F FROM T1 UNION ALL SELECT T1.ID , T1.F6 AS F FROM T1 UNION ALL SELECT T1.ID , T1.F7 AS F FROM T1 UNION ALL SELECT T1.ID , T1.F8 AS F FROM T1 UNION ALL ) N WHERE N.F IN ( 1, 2, 3 ) AND N.ID = T2.T1_ID GROUP BY N.ID HAVING COUNT(DISTINCT N.F) = 3 );
Finding chronological age on Filemaker to be used in equation
I'm trying to calculate a chronological age based on a person's birth date (DOB) and the date of evaluation (DOE). I need the answer to be expressed as a numerical number that can be put in an equation. For example if a person's DOB is 12/26/07 and the DOE is 10/13/15 the chronological age will be 7.75, NOT 7;9 (7 years, 9 months). The decimal place needs to be a function out of 12, not simply representing the age in months. I have accurately calculated this but every time the chronological age is configured to be an even age (9 years 0 months) the calculation shows up as 90, I really need this "90" to be a "9.0" is there anything I can do to configure this into my calculation? The current calculation that is working for every other age is: Year ( GetAsNumber ( DOE )) - Year ( DOB ) - If ( GetAsNumber ( DOE ) < Date ( Month ( DOB ) ; Day ( DOB ) ; Year ( GetAsNumber ( DOE ) ) ); 1 ; 0 ) & ( Mod ( Month ( GetAsNumber ( DOE ) ) - Month ( DOB ) + 12 - If ( Day ( GetAsNumber ( DOE ) ) < Day ( DOB ) ; 1 ; 0 ) ; 12 ) / 12) Thank you!
Your Calculation is ok, but you are concatenating the 2 parts rather than adding them. That's why you get a "90" instead of a "9" Also, if you want it to be "9.0", the calculation result must be numeric; and in layout mode set the format to decimal with a fixed number of decimals. Round ( Year ( GetAsNumber ( DOE )) - Year ( DOB ) - If ( GetAsNumber ( DOE ) < Date ( Month ( DOB ) ; Day ( DOB ) ; Year ( GetAsNumber ( DOE ))); 1 ; 0) + ( Mod ( Month ( GetAsNumber ( DOE ) ) - Month ( DOB ) + 12 - If ( Day ( GetAsNumber ( DOE ) ) < Day ( DOB ); 1; 0) ; 12 ) / 12) ; 2 )
I would suggest a simpler approach: Let ( elapsedMonths = 12 * ( Year ( DOE ) - Year ( DOB ) ) + Month ( DOE ) - Month ( DOB ) - ( Day ( DOE ) < Day ( DOB ) ) ; Div ( elapsedMonths ; 12 ) + Mod ( elapsedMonths ; 12 ) / 12 ) Note that this counts only fully elapsed months.
sql window function to detect change in column values
I'd like to detect changes in column values in this (example) db WITH events(id, row,event) AS ( VALUES (1,1, 0 ) ,(1,2, 0 ) ,(1,3, 1 ) ,(1,4, 0 ) ,(1,5, 1 ) ,(2,1, 0 ) ,(2,2, 1 ) ,(3,1, 0 ) ,(3,2, 0 ) ) select * from events What I am looking for is code for a new column 'code' which switches to 1 AFTER de event column shows a 1. Within the same id the code stays 1. For this example this new column wil look like this WITH events2(id, row,event, code) AS ( VALUES (1,1, 0, 0 ) ,(1,2, 0, 0 ) ,(1,3, 1, 0 ) ,(1,4, 0, 1 ) -- notice the switch here ,(1,5, 1, 1 ) -- ,(2,1, 0, 0 ) ,(2,2, 1, 0 ) ,(3,1, 0, 0 ) ,(3,2, 0, 0 ) ) select * from events2 I have a hunch that the answer will be related to the answer on this question : PostgreSQL window function: partition by comparison Somehow I cannot figure this out myself.. Peter
COALESCE over a scalar subquery: WITH events(id, zrow, zevent) AS ( VALUES (1,1, 0 ) ,(1,2, 0 ) ,(1,3, 1 ) ,(1,4, 0 ) ,(1,5, 1 ) ,(2,1, 0 ) ,(2,2, 1 ) ,(3,1, 0 ) ,(3,2, 0 ) ) SELECT id, zrow, zevent , COALESCE((SELECT 1 FROM events ex WHERE ex.id = ev.id AND ex.zrow < ev.zrow AND ex.zevent> 0),0) AS oevent FROM events ev ; Or, avoid the COALESCE() by typecasting the boolean EXISTS() to INTEGER: WITH events(id, zrow,event) AS ( VALUES (1,1, 0 ) ,(1,2, 0 ) ,(1,3, 1 ) ,(1,4, 0 ) ,(1,5, 1 ) ,(2,1, 0 ) ,(2,2, 1 ) ,(3,1, 0 ) ,(3,2, 0 ) ) SELECT id, zrow, event , EXISTS(SELECT 1 FROM events ex WHERE ex.id = ev.id AND ex.zrow < ev.zrow AND ex.event> 0)::integer AS oevent FROM events ev ; Find the MAX() value over the previous records within the same group (frame): WITH events(id, zrow,event) AS ( VALUES (1,1, 0 ) ,(1,2, 0 ) ,(1,3, 1 ) ,(1,4, 0 ) ,(1,5, 1 ) ,(2,1, 0 ) ,(2,2, 1 ) ,(3,1, 0 ) ,(3,2, 0 ) ) , drag AS ( SELECT id, zrow, event, MAX(event) OVER (PARTITION BY id ORDER BY zrow ROWS BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING ) AS lagged FROM events ev ) SELECT id, zrow, event , COALESCE(lagged,0) AS oevent FROM drag dr ; The same without the extra CTE: WITH events(id, zrow,event) AS ( VALUES (1,1, 0 ) ,(1,2, 0 ) ,(1,3, 1 ) ,(1,4, 0 ) ,(1,5, 1 ) ,(2,1, 0 ) ,(2,2, 1 ) ,(3,1, 0 ) ,(3,2, 0 ) ) SELECT id, zrow, event, COALESCE(MAX(event) OVER (PARTITION BY id ORDER BY zrow ROWS BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING ),0) AS lagged FROM events ev ; Another way to perform the self-join would be to use a recursive query.