Splitting a dataset into training and test set in postgres - postgresql

Question

I have a dataset and I want to use postgres sql to split it into 70:30 ratio into training and test set. How can i do that. I used the following code but it doesn't seem to work
create table training_test as
(
WITH TEMP as
(
SELECT ROW_NUMBER() AS ROW_ID , Random() as RANDOM_VALUE,D.*
FROM analytics.model_data_discharge_v1 as D
ORDER BY RANDOM_VALUE
)
SELECT 'Training',T.* FROM TEMP T
WHERE ROW_ID <= 493896*0.70
UNION
SELECT 'Test',T.* FROM TEMP T
WHERE ROW_ID > 493896*0.70
) distributed by(hospitalaccountrecord);

Answer 1

select t.*,
case
when random() < 0.7 then 'training'
else 'test'
end as split
from analytics.model_data_discharge_v1 t

Answer 2

Don't use random splitting is NOT repeatable! random() will return different results each time.
Instead, you can, for example, use hashing and modulo to split a dataset as Google Cloud suggests.
Hash using a fingerprint function one the columns/features that is not correlated to target (to avoid leaving valuable information out of the training set). Otherwise, concatenate all the fields as JSON string and hash on that
Take the absolute value of the hash
Calculate the abs(hash(column)) modulo 10
If the result < 8 then it will be part of the 80% training set
If the result == 8 then it will be part of the 20% test set
Example using BiQuery (I've taken from a GCP ML course) :
Training set
Test set
In that way, you get the exact 80% data each time.

Answer 3

If you want a stratified split, you can use the following code.
The first bit guarantees that each group has the minimum size to be split.
with ssize as (
select
group
from to_split_table
group by group
having count(*) >= {{ MINIMUM GROUP SIZE }}) -- {{ MINIMUM GROUP SIZE }} = 1 / {{ TEST_THRESHOLD }}
select
id_aux,
ts.group,
case
when
cast(row_number() over (partition by ts.group order by rand()) as double) / cast(count() over (partition by ts.group) as double)
< {{ TEST_THRESHOLD }} then 'test'
else 'train'
end as splitting
from to_split_table ts
join ssize
on ts.group = ssize.group