I have a large PostgreSQL table - 2.8 millions rows; 2345 MB in size; 49 columns, mainly short VARCHAR fields, but with a large json field.
It's running on an Ubuntu 12.04 VM with 4GB RAM.
When I try doing a SELECT * against this table, my psql connection is terminated. Looking in the error logs, I just get:
2014-03-19 18:50:53 UTC LOG: could not send data to client: Connection reset by peer
2014-03-19 18:50:53 UTC STATEMENT: select * from all;
2014-03-19 18:50:53 UTC FATAL: connection to client lost
2014-03-19 18:50:53 UTC STATEMENT: select * from all;
Why is this happening? Is there a maximum amount of data that can be transferred or something - and is that configurable in postgres?
Having one large, wide table is dictated by the system we're using (I know it's not an ideal DB structure). Can postgres handle tables of this size, or will we keep having problems?
Thanks for any help,
Ben
Those messages in the server log just mean that the client went away unexpectedly. In this case, it probably died with an out of memory error.
By default psql loads the entire results into memory before displaying anything. That way it can best decide how to format the data. You can change that behavior by setting FETCH_COUNT
I have seen a similar issue, however, the issue I faced was not on the client side, but most probably on the postgres driver side. The query was required to fetch a lot of rows, and as a result, there could have been temporary memory spike requirement on postgres driver. As a result, the cursor that I was using to fetch the records closed, and I got the exactly same logs.
I would really love if someone validated if this is possible, one thing I am sure is that there was not any issue on the client side.
Related
Background of the issue (could be irrelevent, but only relation to these issues makes sense for me)
In our production environment, disk space had run out. (We do have monitoring and notifications for this, but no-one read them - the classical)
Anyway, after fixing the issue Postgresql PostgreSQL 9.4.17 on x86_64-unknown-linux-gnu, compiled by gcc (Debian 4.9.2-10) 4.9.2, 64-bit has shown couple of weird behaviors.
1. Unique indexes
I have couple of (multi column) unique indexes specified for the database, but they do not appear to be functioning. However I can find duplicate rows from the database.
2. Sorting based on date
We got one table which is basically just logging some json data. We got three columns: id, json and insertedAt DEFAULT NOW(). If I do simple query where I try to sort based on the insertedAt column, the sorting doesn't work around the area of disk overflow. All of the data is valid and readable, but order is invalid
3. Db dumps / backups are having some corruption.
Again, when I was browsing this logging data and tried to recover a backup to my local machine for better observation it gave an error around some random row. When I examined the sql-file with text editor, I encountered that data was otherwise valid expect that it was missing some semicolons on some rows. I think I'll give a try shortly for the never backup if it's still having the same error or if it was random issue with the one backup I tried playing with.
I've tried the basic ones: restarting the machine and PG process.
I have a PostgreSQL database containing a table with several 'timestamp with timezone' fields.
I have a tool (DBSync) that I want to use to transfer the contents of this table to another server/database.
When I transfer the data to a MSSQL server all datetime values are replaced with '1753-01-01'. When I transfer the data to a PostgreSQL database all datetime values are replaced with '0001-01-01'.
The smallest possible date for those systems.
Now i recreate the source-table (including contents) in a different database on the same PostgreSQL server. The only difference: the sourcetable is in a different database. Same server, same routing. Only ports are different.
User is different but in each database I have the same rights.
How can it be that the database is responsible for an apparant different interpretation of the data? Do PostgreSQL databases have database-specific settings that can cause such behaviour? What database-settings can/should I check?
To be clear, I am not looking for another way to transfer data. I have several available. The thing that I am trying to understand is: how can it be that, if an application reads datetime info from table A in database Y on server X, it gives me the the wrong date while when reading the same table from database Z on server X will give me the data as it should be.
It turns out that the cause is probably the difference in server-version. One is a Postgres 9 (works ok), the other is a Postgres 10 (does not work okay).
They are different instances on the same machine. Somehow I missed that (blush).
With transferring I meant that I am reading records from a sourcedatabase (Postgresql) and inserting them in a targetdatabase (mssql 2017).
This is done through the application, I am not sure what drivers it is using.
I wil work with the people who made the application.
For those wondering: it is this application: https://dbconvert.com/mssql/postgresql/
When a solution is found I will update this answer with the found solution.
Yes, I know there is a stack overflow question with the exact same name:
Queries in pg_stat_activity are truncated?
I set this value to 7168, rebooted the server, and verified it with show track_activity_query_size. It actually shortened the amount of text shown, its now truncating at 256 characters.
What am I missing?
Edit: My database is an AWS RDS instance (db.t2.small) runningPostgreSQL 9.3.6
I was using PGAdmin. The query actually was returning a larger value, but PGAdmin was truncating it and I was copy/pasting it from the column out. The solution (for anybody else having this problem) is to use Query -> execute to file. This will write the full results, which you can then look at in the CSV file that it writes.
What the title says -
Msg 9002, Level 17, State 4, Line 1
The transaction log for database 'tempdb' is full. To find out why space in the log cannot be reused, see the log_reuse_wait_desc column in sys.databases
The query in question first pulls out some rows from a databased on the linked server (matching strings in the server I'm querying from), stores them in a table, then uses this table to pull out more matching records from the linked server. This is what I get on the second part.
The basic question is, is there something else hiding in this error message? Which tempdb is it, my tempdb, or the linked server's tempdb? I don't think mine can be a problem, as increasing the size doesn't help, the recover mode is simple, autogrowth is on, etc.
You first need to check your SQL Server's tempDB.... is the drive that TempDB and its log got lots of free disc space? It might be on two different drives. I would expect such an error to write a message in the SQL Server error log - have you checked there as well at the time of the problem? You then need to do the same on the remote server, if you have access.
Whether it's tempDB or a user/application database, just because the recovery model is simple doesn't mean that the transaction log won't grow - and take all the disk space! It does make it less likely, but long transactions can cause it to "blow".
All,
SQL 2005 sp3, database is about 70gb in size. Once in a while when I reindex all of my indexes in all of my tables, the front end seems to freeze up or run very slowly. These are queries coming from the front end, not stored procedures in sql server. The front end is using JTDS JDBC connection to access the SQL Server. If we stop and restart the web services sending the queries the problem seems to go away. It is my understandning that we have a connection pool in which we re-use connections and dont establish a new connection each time.
This problem does not happen every time we reindex. I have tried both ways with dbcc dbreindex and alter index online = on and sort in tempdb = on.
Any insight into why this problem occurs once in a while and how to prevent this problem would be very helpful.
Thanks in advance,
Gary Abbott
When this happens next time, look into sys.dm_exec_requests to see what is blocking the requests from the clients. The blocking_session_id will indicate who is blocking, and the wait_type and wait_resource will indicate what is blocking on. You can also use the Activity Monitor to the same effect.
On a pre-grown database an online index rebuild will not block normal activity 9select/insert/update/delete). The load on the server may increase as a result of the online index rebuild and this could result in overall slower responses, but should not cause blocking.
If the database is not pre-grown though then the extra allocations of the index rebuild will trigger database growth events, which can be very slow if left default at 10% increments and without instant file initialisation enabled. During a database growth event all activity is frozen in that database, and this may be your problem even if the indexes are rebuilt online. Again, Activity Monitor and sys.dm_exec_requests would both clearly show this as happening.