I tried issuing a read all request (select * from tblName) using JDBC cursors (setFetchSize). The temp_file_limit property in postgresql.conf is 500 KB. When I execute the prepared statement, I get a psql exception -
org.postgresql.util.PSQLException: ERROR: temporary file size exceeds temp_file_limit (500kB)
The documentation in PostgreSQL.conf says "# limits per-session temp file space".
https://jdbc.postgresql.org/documentation/head/query.html#query-with-cursor as per this link, the connection must have disabled auto-commit and I had not disabled it. I did that as well, yet I ended up with the same issue. My understanding is that such large read operations are written to a temporary file before loading to the resultset. If this is the case then with a low temp_file_limit, I will never be able to read very large data, even with a cursor. This does not make sense to why it is made configurable in the first place.
Setting the temp_file_limit to -1 (unbounded file size) solved this issue for me. Am I correct in taking this approach?
I have seen queries that would use a lot of space (sometimes over 1TiB), kept going until all free space was used and other queries started crashing. Setting it to lets say reasonable (in our case) 100GiB would have protected other sessions.
Setting it to 500kB seems indeed pointless.
FYI temp files are used not only for cursors.
Related
I'm trying to use the Postgres JDBC driver to query data from a table where each rows can be up to about 50MB. Unfortunately, without any memory restrictions, the Postgres driver can use too much memory and cause OOMs (even with a very healthy Xmx) because it buffers so much data locally.
I've tried to restrict the driver to using less memory, for example 1GB, and telling it to buffer less too. Since no one row is bigger than 50MB this should work fine, but unfortunately I'm now getting Exceptions thrown from the Postgres driver itself. The exceptions are because it is trying to allocate more memory than I have configured it with.
If I use this configuration:
"jdbc:postgresql://localhost/dbname?maxResultBuffer=1G&adaptiveFetch=true&adaptiveFetchMaximum=2&defaultRowFetchSize=1"
I'll get an Exception thrown here, in PGStream
if (resultBufferByteCount > maxResultBuffer) {
throw new PSQLException(GT.tr(
"Result set exceeded maxResultBuffer limit. Received: {0}; Current limit: {1}",
String.valueOf(resultBufferByteCount), String.valueOf(maxResultBuffer)),PSQLState.COMMUNICATION_ERROR);
}
If I set a breakpoint there I can see:
value = 41155480
resultBufferByteCount = 1021091718
maxResultBuffer = 1000000000
Which shows it's picking up the config fine. I've also inspected it to make sure it's getting the fetch size config and it is.
Is there some other config I'm missing? Clearly the Postgres driver is reading more rows than I've allowed it to.
thanks
(postgreqsl 42.5.1, java 17.0.5, hikaricp 5.0.1 with max connections of 1)
The adaptive buffer, like setFetchSize, only works if autocommit is off. If Autocommit is on, then they are silently ignored. I don't know if there is a way to turn autocommit off though the jdbc connect string, I haven't found one.
I have a database that can not be started. It started when I change max_wal_size < wal_segement_size. The error is FATAL ERROR "max wal sender must be at least twice wal_segment_size"
I have changed the value back to its original size, but the error still comes up. it seems that the postgres is still having hang over or something is stuck within the "internal cache".
Why postgres does not pick up the new config file ?
How can I clear the "cache" ?
How can I overwrite the postgres.conf during start ? this is mainly to bypass whatever the postgres.conf that postgres read.
Thanks
That error message does not exist in PostgreSQL.
Perhaps you mean
"min_wal_size" must be at least twice "wal_segment_size"
If my guess is correct, and you didn't change the WAL segment size from the default 16MB, you can fix the problem by changing min_wal_size to a value of at least 32MB.
max_wal_size must be no less than min_wal_size. Typically, it should be much bigger for decent performance.
I recently upgraded a Postgres 9.6 instance to 11.1 on Google Cloud SQL. Since then I've begun to notice a large number of the following error across multiple queries:
org.postgresql.util.PSQLException: ERROR: could not resize shared
memory segment "/PostgreSQL.78044234" to 2097152 bytes: No space left
on device
From what I've read, this is probably due to changes that came in PG10, and the typical solution involves increasing the instance's shared memory. To my knowledge this isn't possible on Google Cloud SQL though. I've also tried adjusting work_mem with no positive effect.
This may not matter, but for completeness, the instance is configured with 30 gigs of RAM, 120 gigs of SSD hd space and 8 CPU's. I'd assume that Google would provide an appropriate shared memory setting for those specs, but perhaps not? Any ideas?
UPDATE
Setting the database flag random_page_cost to 1 appears to have reduced the impact the issue. This isn't a full solution though so would still love to get a proper fix if one is out there.
Credit goes to this blog post for the idea.
UPDATE 2
The original issue report was closed and a new internal issue that isnt viewable by the public was created. According to a GCP Account Manager's email reply however, a fix was rolled out by Google on 8/11/2019.
This worked for me, I think google needs to change a flag on how they're starting the postgres container on their end that we can't influence inside postgres.
https://www.postgresql.org/message-id/CAEepm%3D2wXSfmS601nUVCftJKRPF%3DPRX%2BDYZxMeT8M2WwLSanVQ%40mail.gmail.com
Bingo. Somehow your container tech is limiting shared memory. That
error is working as designed. You could figure out how to fix the
mount options, or you could disable parallelism with
max_parallel_workers_per_gather = 0.
show max_parallel_workers_per_gather;
-- 2
-- Run your query
-- Query fails
alter user ${MY_PROD_USER} set max_parallel_workers_per_gather=0;
-- Run query again -- query should work
alter user ${MY_PROD_USER} set max_parallel_workers_per_gather=2;
-- -- Run query again -- fails
You may consider increasing Tier of the instance, that will have influence on machine memory, vCPU cores, and resources available to your Cloud SQL instance. Check available machine types
In Google Cloud SQL PostgreSQL is also possible to change database flags, that have influence on memory consumption:
max_connections: some memory resources can be allocated per-client, so the maximum number of clients suggests the maximum possible memory use
shared_buffers: determines how much memory is dedicated to PostgreSQL to use for caching data
autovacuum - should be on.
I recommend lowering the limits, to lower memory consumption.
I'm running a really big query, that insert a lot of rows in table, almost 8 million of rows divide in some smaller querys, but in some moment appear that error : "I get an error "could not write block .... of temporary file no space left on device ..." using postgresql". I don't know if i need to delete temporary files after each query and how I can to do that, or if it is related with another issue.
Thank you
OK. As there are still some facts missing, an attempt to answer to maybe clarify the issue:
It appears that you are running out of disk space. Most likely because you don't have enough space on your disk. Check on a Linux/Unix df -h for example.
To show you, how this could happen:
Having a table with maybe 3 integers the data alone will occupy about 12Byte. You need to add some overhead to it for row management etc. On another answer Erwin mentioned about 23Byte and linked to the manual for more information about. Also there might needs some padding betweens rows etc. So doing a little math:
Even with a 3 integer we will end up at about 40 Byte per row. Having in mind you wanted to insert 8,000,000 this will sum up to 320,000,000Byte or ~ 300MB (for our 3 integer example only and very roughly).
Now giving, you have a couple of indexes on this table, the indexes will also grow during the inserts. Also another aspect might could be bloat on the table and indexes which might can be cleared with a vacuum.
So what's the solution:
Provide more disk space to your database
Split your inserts a little more and ensure, vacuum is running between them
Inserting data or index(create) always needs temp_tablespaces, which determines the placement of temporary tables and indexes, as well as temporary files that are used for purposes such as sorting large data sets.according to your error, it meant that your temp_tablespace location is not enough for disk space .
To resolve this problem you may need these two ways:
1.Re-claim the space of your temp_tablespace located to, default /PG_DATA/base/pgsql_tmp
2. If your temp_tablespace space still not enough for temp storing you can create the other temp tablespace for that database:
create tablespace tmp_YOURS location '[your enough space location';
alter database yourDB set temp_tablespaces = tmp_YOURS ;
GRANT ALL ON TABLESPACE tmp_YOURS to USER_OF_DB;
then disconnect the session and reconnect it.
The error is quite self-explanatory. You are running a big query yet you do not have enough disk space to do so. If postgresql is installed in /opt...check if you have enough space to run the query. If not LIMIT the output to confirm you are getting the expected output and then proceed to run the query and write the output to a file.
I'm getting an Error
ERROR: invalid page header in block 411 of relation "t_value_time"
in my PostgreSQL database. This keeps happening on different machines. Is there a way to prevent it from happening, or at least telling PSQL to ignore the data on the invalid block and move on?
I'd rather lose the data from the block and have him skip over it, reading the rest of the data. Is there a way to tell PSQL to skip this block?
WARNING: You will lose some data!
We managed to get over it (crashed DEV VM) by issuing:
database=# SET zero_damaged_pages = on;
SET
database=# VACUUM FULL damaged_table;
WARNING: invalid page header in block xxx of relation base/yyy/zzz; zeroing out page
[..]
REINDEX TABLE damaged_table;
Fix via pwkg.ork.
Same block every time?
From what I've read, the most common cause of invalid blocks is hardware. Red Hat has a utility, pg_filedump, that formats "PostgreSQL heap, index, and control files into a human-readable form". I don't think they support any PostgreSQL version greater than 8.4.0, but I could be wrong.
You want to prove your hardware is good by using tough, thorough disk, RAM, and NIC diagnostics.
There's no simple way to do it, but it's reasonably easy to do just by editing the data file directly (relfilenode of the pg_class entry gives the filename).
Just copy a block from elsewhere in the file over the bad block. Ideally, synthesise an empty block or update the one you're overwriting to have no valid tuples in it.
Once you've got something that doesn't produce that error, dump the table and reload it for safety.
these are almost always hardware problems btw. Verify and test RAM, disk, CPU. Make sure your environment is good (bad power input can cause problems as can overheating). That's the best way to prevent it. Best way to address it is point in time recovery from a base backup.
If you have a slave, set hot_standby_feedback to 'on' on it if not already.
Do pg_dump and write it to /dev/null so that you don't consume any space.
nohup pg_dump db_name -v -Fc -f /dev/null &
If the dump succeeds, then your slave is fine. Do a failover. There will be no data loss.
Another way to validate your slave is to do, explain select count(*) from table_name;
If it succeeds and if it is using a sequence scan, then your slave is good.
You may not have to consider this option if it is using index scan.
Note: This works only if your master is affected with storage level corruption.
I happened to face same issue just today and i was able to fix it.