I have a mobile application (C#) which call an API using phoenix and ecto.
This API makes several calls to the postgres database. Each call lasts approximately 60ms and we make approximately 25 calls to the database.
But at some point, I get a timeout from the database.
Here is the postgres error:
2020-06-04 09:40:03.503 CEST [24455] postgres#view_models ERROR: canceling statement due to user request
2020-06-04 09:40:03.503 CEST [24455] postgres#view_models STATEMENT: SELECT DISTINCT ON (i0."intervention_id") i0."intervention_id" FROM "interventions" AS i0 LEFT OUTER JOIN "appointments" AS a1 ON a1."intervention_id" = i0."intervention_id" WHERE ((i0."account_id" = $1) AND ((i0."updated_at" > $2) OR (a1."updated_at" > $3))) LIMIT 1
2020-06-04 09:40:03.504 CEST [24455] postgres#view_models LOG: could not send data to client: Broken pipe
2020-06-04 09:40:03.504 CEST [24455] postgres#view_models FATAL: connection to client lost`
Here is the ecto error:
DBConnection.ConnectionError: tcp recv: closed (the connection was closed by the pool, possibly due to a timeout or because the pool has been terminated)
The postgres statement_tiemout parameter is 0
Here is the ecto config:
config :query_backend, QueryBackend.V1.Repo,
username: System.get_env("POSTGRES_USERNAME"),
password: System.get_env("POSTGRES_PASSWORD"),
database: System.get_env("VIEW_POSTGRES_DB_NAME"),
hostname: System.get_env("POSTGRES_HOST"),
pool_size: 10,
queue_target: 3_000,
queue_interval: 15_000,
connect_timeout: 20_000,
timeout: 30_000
On the mobile, the HTTP client timeout is 15s.
This problem is reproduced only by our customers and our technical team cannot reproduce it.
Can you tell me if my configuration is valid? What is the origin of timeout? the mobile HTTP client, Ecto, Postgres ?
Thank you in advance for your help
Best regards.
Julien.
Related
I have a Go program which uses Mongo DB. I run it from my home pc and I keep getting this error very often:
panic: server selection error: server selection timeout, current
topology: { Type: ReplicaSetNoPrimary, Servers: [{ Addr:
cluster0-shard-00-00.tvm1o.mongodb.net:27017, Type: Unknown, Last
error: connection() error occurred during connection handshake: dial
tcp
3.216.112.85:27017: i/o timeout }, { Addr: cluster0-shard-00-01.tvm1o.mongodb.net:27017, Type: Unknown, Last
error: connection() error occurred during connection handshake: dial
tcp 34.197.85.254:27017: i/o timeout }, { Addr:
cluster0-shard-00-02.tvm1o.mongodb.net:27017, Type: Unknown, Last
error: connection() error occurred during connection handshake: dial
tcp 18.206.5.2:27017: i/o timeout }, ] }
And this is the exact code where it breaks:
if err := clientMongo.Ping(context.TODO(), readpref.Primary()); err != nil {
panic(err)
}
I understand this is a connection timeout, but I don't understand how can this happen at all during a simple client connection. I made a speedtest and my current upload speed is 22 Mbps, I am not uploading big json arrays or anything. It happens always when I try to connect to the client. So I would like to know if this can be caused because my internet connection or something on Mongo's end?
You might need to add your IP to the whitelist of MongoDB.
A few things --
we would need to see the complete code for creating a connection. I'm going to assume you're using exactly what is in the documentation here?
You should try to connect with mongosh and Compass also. If you have problems with another tool, then the odds are it is your Atlas cluster OR your security settings on the cluster, rather than your application code.
that being said about 95% of the time the issue is whitelist or database users. do you have a user created in database access area of the UI that has admin/read/write any database? is your IP in the whitelist?
if 3 is good, and 2 doesn't work, there is free Atlas support available in the green chat button of the UI in MongoDB.
I have a view in Postgres with the following definition:
CREATE VIEW participant_data_view AS
SELECT participant_profile.*,
"user".public_id, "user".created, "user".status, "user".region,"user".name, "user".email, "user".locale,
(SELECT COUNT(id) FROM message_log WHERE message_log.target_id = "user".id AND message_log.type = 'diary') AS diary_reminder_count,
(SELECT SUM(pills) FROM "order" WHERE "order".user_id = "user".id AND "order".status = 'complete') AS pills
FROM participant_profile
JOIN "user" ON "user".id = participant_profile.id
;
The view creation works just fine. However, when I query the view SELECT * FROM participant_data_view, postgres crashes with
10:24:46.345 WARN HikariPool-1 - Connection org.postgresql.jdbc.PgConnection#172d19fe marked as broken because of SQLSTATE(08006), ErrorCode(0) c.z.h.p.ProxyConnection
org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend.
this question suggests to me that it might be an internal assertion that causes it to crash.
If I remove the diary_reminder_count field from the view definition, the select works just fine.
What am I doing wrong? How can I fix my view, or change it so I can query the same data in a different way?
Note that creating the view works just fine, it only crashes when querying it.
I tried running explain (analyze) select * from participant_data_view; from the IntelliJ query console, which only returns
[2020-12-08 11:13:56] [08006] An I/O error occurred while sending to the backend.
[2020-12-08 11:13:56] java.io.EOFException
I ran the same using psql, there it returns
my-database=# explain (analyze) select * from participant_data_view;
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The connection to the server was lost. Attempting reset: Failed.
Looking at the log files, it contains:
2020-12-08 10:24:01.383 CET [111] LOG: server process (PID 89670) was terminated by signal 9: Killed: 9
2020-12-08 10:24:01.383 CET [111] DETAIL: Failed process was running: select "public"."participant_data_view"."id", "public"."participant_data_view"."study_number", <snip many other fields>,
"public"."participant_data_view"."diary_reminder_count", "public"."participant
2020-12-08 10:24:01.383 CET [111] LOG: terminating any other active server processes
In all likelihood, the Linux kernel out-of-memory killer killed your query because the system ran out of memory.
Either restrict the number of database sessions (for example with a connection pool) or reduce work_mem.
It is usually a good idea to set vm.overcommit_memory = 2 in the kernel and tune vm.overcommit_ratio appropriately.
I am following IdentityServer4 quickstart and trying to migrate in memory data to my local SQL Server (not SQL express or LocalDB that came with VS). My connection string is:
#"Server=localhost,1434;Database=MyIDS;user id=tester_1;Password=tester_1;trusted_connection=yes;".
When I start my IdentityServer, it creates the enpty db, MyIDS, and then throw an exception with 2 inner exceptions:
Inner Exception 1:
SqlException: A connection was successfully established with the server, but then an error occurred during the pre-login handshake. (provider: TCP Provider, error: 0 - An established connection was aborted by the software in your host machine.)
Inner Exception 2:
Win32Exception: An established connection was aborted by the software in your host machine.
Can anyone tell me what's going on here? Why a working connection always gets dropped?
localhost,1434 looks wrong, you don't need to provide the port 1434 and the commma should not be used either, it should be a colon in that case.
I typically use for local development:
server=.;Database=ASPIdentity;Trusted_Connection=True;
dot means localhost, if you use sqlexpress the connetion string would become
server=.\\sqlexpress;Database=ASPIdentity;Trusted_Connection=True;
I want to load a large dataset (750 GB) into Skyrise. For this I use
copy LINEITEM from 's3://myBucket/'
credentials 'aws_access_key_id=key;aws_secret_access_key=secret'
null as '\000'
DELIMITER ','
region 'us-east-1'
ESCAPE;
After about 10 minutes I get
Unable to execute HTTP request: Connect to <some IP> failed: Connection refused (Connection refused)
I am able to load other datasets. What is the issue here?
It's likely that your connection was dropped due a timeout. Please review the following document for steps to correct this issue:
"Troubleshooting connection issues in Amazon Redshift"
EDIT
I put the same pg_hba rule on the postgresql installed on my WINDOWS laptop, and it works... so I changed titled : how can I make my linux server work to authenticate users with AD, like the windows Pg server does ?
/ EDIT
I need to perform authentication of postgresql db users with our Active Directory servers. I've tested lots of configurations but so far, I couldn't find why postgresql users can't be authenticated with this authentication methode.
LDAP : Active Directory / Postgresql : Postgresql 9.4
Here is the pg_hba rule I use :
host myDB myUser localhost ldap ldapserver="192.168.10.1" ldapbasedn="DC=companygroup,DC=priv" ldapbinddn="cn=LDAP - Lecture,ou=Users,ou=Specials Objects,dc=companygroup,dc=priv" ldapbindpasswd="ldapPassWord" ldapsearchattribute="sAMAccountName"
When logging with 'myUser' with the correct password for this user, I have the following logs in postgresql log file :
2015-11-18 10:01:50 CET [25991-1] [unknown]#[unknown] LOG: 00000: connection received: host=127.0.0.1 port=39074
2015-11-18 10:01:50 CET [25991-2] [unknown]#[unknown] LOCATION: BackendInitialize, postmaster.c:4003
2015-11-18 10:01:50 CET [25991-3] myUser#myDB LOG: 00000: could not search LDAP for filter "(sAMAccountName=myUser)" on server "192.168.10.1": Operations error
2015-11-18 10:01:50 CET [25991-4] myUser#myDB LOCATION: CheckLDAPAuth, auth.c:2030
2015-11-18 10:01:50 CET [25991-5] myUser#myDB FATAL: 28000: LDAP authentication failed for user "myUser"
2015-11-18 10:01:50 CET [25991-6] myUser#myDB DETAIL: Connection matched pg_hba.conf line 104: "host myDB myUser localhost ldap ldapserver="192.168.10.1" ldapbasedn="DC=companygroup,DC=priv" ldapbinddn="cn=LDAP - Lecture,ou=Users,ou=Specials Objects,dc=companygroup,dc=priv" ldapbindpasswd="ldapPassWord" ldapsearchattribute="sAMAccountName"
I saw that if I change somehow ldapbinddn or ldapbindpasswd, I have another error like 'couldn't perform initial LDAP bind for ldapbinddn "...". so these parameters should be ok.
"Operations error" was no very detailed, so I tcpdump the authentication process and here is what I found. It seems that Postgres perform two queries :
First to search the user via the search attribute. This action seems OK because in the response of Active Directory, I saw information tied to my user.
Then another query is performed. On this one, the real message from the LDAP Active directory server is :
LdapErr: DSID-0C0906E8, comment: In order to perform this operation a successful bind must be completed on the connection., data 0 , v1db1
On this second query, I see that PG seem to change slightly the base search with
"DC=ForestDnsZones,DC=companygroup,dc=priv" instead of "DC=companygroup,DC=priv"
(I saw it in the tcp trace :
LDAPMessage searchRequest(3) "DC=ForestDnsZones,DC=companygroup,dc=priv" wholeSubtree ...
)
When I tried a research thanks to windows soft "ldapbrowser", I was abled to find my account with a simple filter (sAMAccountName=myUser), with the searchDN DC=companygroup,DC=priv
Is my understanding correct ? is it possible that the search is not successful just because of the basedn that is maybe changed ? or do I miss something else ?
Based on your log, your configuration for ldapsearchattribute="sAMAccountName" doesn't work.
You can use a LDAP tool such as LDAPAdmin or OpenLDAP to test your filter. Make sure that the above return result when your filter the attribute below
(sAMAccountName=myUser)