I have very starnge behavior on Azure PostgreSQL server.
I try to read DB size using the next query:
SELECT CAST(pg_database_size(current_database()) * 1.0/(1024 * 1024) AS DECIMAL(12, 5))
But this query runs very long and I have timeout exception in my C# application:
Exception while reading from stream; Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
I have no idea why pg_database_size takes a lot of time...
To resolve my problem I see two options:
Increase query performance using Azure PostgreSQL Server settings
Replace the current query to another query with some result
I'm open for any suggestions.
Please give me any ideas about how I can read DB size fast and correctly.
Here is very much the same sql, runs fine directly on postgresql
SELECT pg_database.datname, pg_size_pretty(pg_database_size(pg_database.datname)) AS size FROM pg_database; --- as option #2 ?
Related
I am trying to copy data from a MySQL database to Azure SQL Server but I am getting a timeout error:
Operation on target Copy MyTable failed:
ErrorCode=UserErrorFailedToConnectOdbcSource,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=ERROR
[08001] [Microsoft][MySQL] (1004) The connection has timed out while
connecting to server: xxxxx.xxxxxx.us-west-2.rds.amazonaws.com at
port:
1234.,Source=Microsoft.DataTransfer.Runtime.GenericOdbcConnectors,''Type=System.Data.Odbc.OdbcException,Message=ERROR
[08001] [Microsoft][MySQL] (1004) The connection has timed out while
connecting to server: xxxxx.xxxxxx.us-west-2.rds.amazonaws.com at
port: 1234.,Source=,'
I can preview data while looking at the source of my Copy Data task. There is no timeout. I see all of the rows and columns. I even changed the query to limit the results to 2 rows.
SELECT mytable.id, mytable.name FROM myschema.mytable LIMIT 2;
However, when I publish the pipeline and trigger it to run I get the timeout error. How can I resolve the timeout using Azure Data Factory (ADF) when connecting to MySQL?
The error message was not the most helpful. I discovered what the problem was. The problem was that the IP Addresses used by ADF had to be added to the "Outbound IP" list from AWS MySQL. Everything started working once I updated the outbound IP address list.
After a period of inactivity, my go web service is getting a net.OpError with message read tcp x.x.x.x:52086->x.x.x.x:24414: read: connection reset by peer when executing the first postgres sql query. After the error, the subsequent requests will work fine.
The postgres database is hosted with compose.com which has haproxy in front of the postgres db. My go web app is using standard sql and sqlx.
I've tried running a ticker invoking db.Ping() every 15 minutes, but this hasn't fixed the issue.
Why is the go standard sql lib not handling these connection drops?
Since no one wrote that explicity. The solution to this problem is setting db.SetConnMaxLifetime(time.Minute). I tried it and it works. Connection reset occurs often on AWS where there is inactivity limit set to 350 seconds, after that TCP RST is returned.
I am facing the below issue
ERROR: canceling statement due to user request
inconsistently after I enabled query timeout for xa datasource in my xxx-ds.xml. I have added the following in my ds xml file.
<query-timeout>180</query-timeout>
The query timeout is set for 180 seconds, which means any sql query
that takes more than 180 seconds will get cancelled from application
server side.
But the issue I am facing is inconsistent and the queries gets timed out now and then without taking exactly 180 seconds.
We are using connection pooling also.
While searching stackoverflow found this question, which discusses about the possible causes for this issue while using connection pooling.
The solution suggested there was to set statement_timeout setting in postgresql.conf file. But it is bit difficult for me to enable statement_timeout setting in my database environment as the DB server is shared by multiple applications.
I would like to have a solution to terminate timed out queries from client side effectively and consistently while using connection pooling. I am using
JBoss 4.2.2-GA
postgresql 9.2 (64 bit)
java 1.7
postgresql-9.2-1002.jdbc4.jar
It looks like the issue is with postgresql driver 9.2. When I upgraded to 9.3, issue is fixed.
I have two EC2 instances, one of them needs to insert large amounts of data into a Postgresql db that lives on the other one. Incidentally it's Open Street Map data and I am using the osm2pgsql utility to insert the data, not sure if that's relevant, I don't think so.
For smaller inserts everything is fine but for very large inserts something times out after around 15 minutes and the operation fails with:
COPY_END for planet_osm_point failed: SSL SYSCALL error: Connection timed out
Is this timeout enforced by Postgresql, Ubuntu or AWS? Not too sure where to start troubleshooting.
Thanks
Could be caused by renegotiation. Check the log, and maybe tweak
ssl_renegotiation_limit = 512MB (the default)
setting it to zero will disable negotiation
I fixed and use Connection Pool to get the DB Connection from JSP to Oracle Database now.
But then I found out from the OC4J console at once that there are many UNCLOSED/NOT-CLOSED connection.
I am using OC4J 9.0.4.1 and Oracle 10g. Is there any effective way to find out the location or area that the connection is not closed.
In addition, I would like to know the followings:
1) The UNCLOSED connection means the Connection or ResultSet or Statment ?
2) IF Connection for the ResultSet and Statement is closed, the corresponding ResultSet and Statement are closed automatically too ?
I'd expect that this is easiest to detect on the server side - one assumes (!) that there is a 'show me all the open connections' report or command on the server that should show up a list of those connections together with the last executed SQL on that connection (am coming from the MSSQL side of things here - this doesn't sound like too much of a stretch, tho').
Unclosed means that the connection has not been closed.