Insert failed on table with 300 integers and 300 char(20) - postgresql

Question

I've tried to test TOAST functionality and created the code:
int length = 20;
using (NpgsqlConnection conn = new NpgsqlConnection(""))
{
conn.Open();
StringBuilder ct = new StringBuilder();
ct.Append("CREATE TABLE t300 (");
for (int i = 0; i < 300; i++)
{
ct.Append("i").Append(i).Append(" int not null, n").Append(i).Append(" varchar(").Append(length).Append(") not null, ");
}
ct.Remove(ct.Length - 2, 2).Append(");");
using (NpgsqlCommand cmd = new NpgsqlCommand(ct.ToString(), conn))
{
cmd.ExecuteNonQuery();
}
StringBuilder isql = new StringBuilder();
isql.Append("INSERT INTO t300 (");
StringBuilder vsql = new StringBuilder();
vsql.Append("VALUES (");
for (int i = 0; i < 300; i++)
{
isql.Append("i").Append(i).Append(", n").Append(i).Append(", ");
vsql.Append(":i").Append(i).Append(", :n").Append(i).Append(", ");
}
isql.Remove(isql.Length - 2, 2).Append(") ").Append(vsql).Remove(isql.Length - 2, 2).Append(");");
using (NpgsqlCommand cmd = new NpgsqlCommand(isql.ToString(), conn))
{
for (int i = 0; i < 300; i++)
{
cmd.Parameters.AddWithValue("i" + i.ToString(), NpgsqlDbType.Integer, i);
cmd.Parameters.AddWithValue("n" + i.ToString(), NpgsqlDbType.Varchar, length, i.ToString() + new string('n', length - i.ToString().Length));
}
for (int i = 0; i < 10000; i++)
{
cmd.ExecuteNonQuery();
}
}
}
This code fails on INSERT with exception '54000, row size (8424) exceeds limit (8160)'.
When I set 'length' variable to 26, the code works fine. Please tell me the workaround to eliminate this situation.
Postgres 12, Npgsql 4.1.5

Answer 1

Perhaps you have a misconception of how TOAST storage works. PostgreSQL does not compress the whole row and store it in the TOAST table, but each column of a varying length data type independently.
So after toasting, the row still consists of 600 columns, 300 of which (the integers) won't be toasted (4 bytes), and the other 300 toasted columns (the varchars) will now contain a TOAST header and a TOAST pointer.
Together this happens to be more than fits into a single block, and rows cannot span more than a single block. That causes the error.
The solution is not to use tables with so many columns. You should split the data in several tables (normalization usually takes care of that). If there are truly very many attributes to a single entity, chances are that not all of these attributes will get used in join or WHERE conditions. You could consider storing such attributes in a single jsonb column, where TOASTing will be much more efficient.