I'm ingesting billions of rows into a postgresql table using a copy statement in a SQL script. Once the table is built I need to add a couple indexes. This is the only thing I'm doing with the database right now, so I would like to optimize it for copying/indexing. I heard it is best to adjust the maintenance_work_mem parameter value. But when I look at the value in RDS I see:
maintenance_work_mem = GREATEST({DBInstanceClassMemory*1024/63963136},65536)
The database I am using is a db.r6g.12xlarge so it has 384GB of memory. What do you think I should set the value to? Is adjusting the parameter in configuration > parameter group the right place?
Related
I'm managing a PostgreSQL database server for some users who need to create temporary tables. One user accidentally sent a query with ridiculously many outer joins, and that completely filled the disk up.
PostgreSQL has a temp_file_limit parameter but it seems to me that it is not relevant:
It should be noted that disk space used for explicit temporary tables, as opposed to temporary files used behind-the-scenes in query execution, does not count against this limit.
Is there a way then to put a limit on the size on disk of "explicit" temporary tables? Or limit the row count? What's the best approach to prevent this?
The only way to limit a table's size in PostgreSQL is to put it in a tablespace on a file system of an appropriate size.
Since temporary tables are created in the default tablespace of the database you are connected to, you have to place your database in that size restricted tablespace. To keep your regular tables from being limited in the same way, you'd have to explicitly create them in a different, less limited tablespace. Make sure that your user has no permissions on that less limited tablespace.
This is a rather unappealing solution, so maybe you should rethink your requirement. After all, the user could just as well fill up the disk by inserting the data into a permanent table.
Recently, we are trying to migrate our database from SQL Server to PostgreSQL. But, we didn't know that by default, tables in Potsgres are ot clustered. Now, when our data has increased so much, we want to CLUSTER our table like so
CLUSTER table USING idx_table;
But seems like my data is a lot (maybe), so that it produces
SQL Error [53400]: ERROR: temporary file size exceeds temp_file_limit
(8663254kB)
Since, its not resulted by a query, which I cannot tune it to perform better, Is there any solution for this?
If for example I am needed to increase my temp_file_limit, is it possible to increase it only for temporary? Since I'm only running this CLUSTER once.
There is some important differences between SQL Server and PostgreSQL.
Sybase SQL Server has been designed from INGRES in the beginning of the eighties when INGRES was using massively the concept of CLUSTERED indexes which means that table is organized as an index. The SQL Engine was designed especially to optimize the use of CLUSTERED index. That is the ways that SQL Server actually works...
When Postgres was designed, the use of CLUSTERED indexes disappeared.
When Postgres switched to the SQL language, an then be renamed to PostgreSQL nothing have changed to use CLUSTERED indexes.
So the use of CLUSTER tables in PostgreSQL is rarely optimal in execution plans. You have to prove individually for each table and for some queries involving those tables, if there is a benefit or not...
Another thing is that CLUSTERing a table in PostgreSQL is not the equivalent of MS SQL Server's CLUSTERED indexes...
More information about this will be find in my paper :
PostgreSQL vs. SQL Server (MSSQL) – part 3 – Very Extremely Detailed Comparison
An especially in § : "6 – The lack of Clustered Index (AKA IOT)"
I'm working on a script to push data into MySQL workbench. While exploring the workbench, I saw there is this option which says "Limit to 50000 rows" and it can be changed from Don't Limit to 50000. Is this just to limit the data/row being displayed for each table, or it will only store 50000 rows for each table ? Is there a limit to the data or row being stored for each table? Or as long as my hard disk size is sufficient it will keep saving the data. Also is there a way to check how much size each table is ? Thanks.
This limit option simply applies a LIMIT clause to queries which support it (unless the user has already added such a clause).
This is a simple measure to help new users to avoid the common pitfall of running a SELECT on a large data set unconditionally. More advanced users usually disable this option and manually apply a LIMIT clause where needed.
I have an optimization issue. At the moment I am using DBI in Perl to connect to IQ(Sybase) then load the values into a hash, I then connect to PostgreSQL and use that hash to do a line by line insert with insert() value(). This is very slow. Does anyone know of a faster way to run this? My main problem is the two different DB servers and I need to avoid a hash when inserting so some type of bulk insert would be ideal just not sure how?
Populating a PostgreSQL database is documented here. The COPY command is your fiend.
Use COPY
Remove Indexes
Remove Foreign Key Constraints
Increase maintenance_work_mem
Increase checkpoint_segments
Disable WAL Archival and Streaming Replication
Run ANALYZE Afterwards
I have a rather large Insert Query, and upon running this my slow disks fill up towards 100% upon where I revive:
Transaction aborted because DBD::Pg::db do failed: ERROR: could not write to hash-join temporary file: No space left on device
Sounds believable, I have a Fast drive with lots of space on it that i could use instead of the slow disk but I dont want to make the fast disk the default table-space or move the table im inserting into to the fast disk, I just want that data-blob that is generated as part of the insert query to be on the fast disk table-space. Is this possible is PostgreSQL and if so how?
version 9.1
You want the temp_tablespaces configuration directive. See the docs.
Temporary files for purposes such as sorting large data sets are also
created in these tablespaces
You must CREATE TABLESPACE the tablespace(s) before using them in an interactive SET temp_tablespaces command.
SET LOCAL temp_tablespaces may be used to set it only for the current transaction.