I have Postgresql 12.8.1 version and python 3.7.6 version installed on my system. I want to create a trigger function using plpython so I created plpython3 extension using CREATE EXTENSION plpython3u . While trying to compile the trigger function, I face the following error -
ERROR: server closed the connection unexpectedly This probably means
the server terminated abnormally before or while processing the
request
Trigger function code:
CREATE OR REPLACE FUNCTION getTemperature() RETURNS trigger as $pgsqlTrigger$
city = TD["new"]["City"]
Station = TD["new"]["Station"]
dtime = TD["new"]["dtime"]
import requests, json, time, pandas as pd, numpy as np
from datetime import datetime
from calendar import monthrange
from copy import deepcopy
import sys
template_request = "https://api.weather.com/v1/location/{station}/observations/historical.json?apiKey=apikey&units=m&startDate={start_date}&endDate={end_date}"
df_header = ["City", "Year", "Month", "Day", "Hour", "Temperature(C)", "Condition"]
def get_weather_data(city, year, month, day, station):
start_date = "%d%02d%02d" % (year, month, day)
end_date = "%d%02d%02d" % (year, month, day)
request = template_request.format(station=station, start_date=start_date, end_date=end_date)
request_data = json.loads(requests.get(request).content)
weather_data = []
last_dt = None
for observation in request_data["observations"]:
dt = datetime.fromtimestamp(observation["valid_time_gmt"]+3600)
if last_dt and dt.hour > (last_dt.hour + 1):
last_row = deepcopy(weather_data[-1])
last_row[4] = last_row[4]+1
weather_data.append(last_row)
weather_data.append([city, year, month, dt.day, dt.hour, observation["temp"], observation["wx_phrase"]])
last_dt = dt
return weather_data
dtime = datetime.strptime(dtime, '%Y-%m-%d %H:%M:%S%z')
data = get_weather_data(city, dtime.year, dtime.month, dtime.day, station)
weather_df = pd.DataFrame(data, columns=df_header).drop_duplicates(subset=["City", "Year", "Month", "Day", "Hour"])
avg = (weather_df["Temperature(C)"].values).mean()
weather_df = pd.DataFrame()
TD["new"]["temp"] = avg;
return NEW;
$pgsqlTrigger$ LANGUAGE plpython3u;
CREATE TRIGGER pgsqlTrigger
BEFORE INSERT ON tweets
FOR EACH ROW EXECUTE FUNCTION getTemperature();
I have not yet found a solution for this. Any ideas on what should I do?
UPDATE:
I am using windows 10 as my os.
Regarding how I installed my plpython3u extension. I was facing the error " no such module named plpython3u found" and the fix I found on net was to copy and paste the python37.dll from the local python folder to my C:/windows/System32 folder. After doing that I was able to create plpython3u extension by using the command - CREATE EXTENSION plpython3u.
I am using the default user "postgres" which has the superuser permission.
below is the output of the server log:
log file output:
2021-09-22 11:46:28.518 IST [10620] LOG: server process (PID 2316) was terminated by exception 0xC0000409
2021-09-22 11:46:28.518 IST [10620] DETAIL: Failed process was running: CREATE FUNCTION public.proc1()
2021-09-22 11:46:28.518 IST [10620] HINT: See C include file "ntstatus.h" for a description of the hexadecimal value.
2021-09-22 11:46:28.522 IST [10620] LOG: terminating any other active server processes
2021-09-22 11:46:28.547 IST [13168] WARNING: terminating connection because of crash of another server process
2021-09-22 11:46:28.547 IST [13168] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2021-09-22 11:46:28.547 IST [13168] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2021-09-22 11:46:28.566 IST [7776] WARNING: terminating connection because of crash of another server process
2021-09-22 11:46:28.566 IST [7776] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2021-09-22 11:46:28.566 IST [7776] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2021-09-22 11:46:28.586 IST [10620] LOG: all server processes terminated; reinitializing
2021-09-22 11:46:28.668 IST [9204] LOG: database system was interrupted; last known up at 2021-09-21 19:23:24 IST
2021-09-22 11:46:29.103 IST [14208] FATAL: the database system is in recovery mode
2021-09-22 11:46:29.549 IST [9204] LOG: database system was not properly shut down; automatic recovery in progress
2021-09-22 11:46:29.555 IST [9204] LOG: redo starts at 0/1D19AB8
2021-09-22 11:46:29.562 IST [9204] LOG: invalid record length at 0/1D21978: wanted 24, got 0
2021-09-22 11:46:29.563 IST [9204] LOG: redo done at 0/1D21940
2021-09-22 11:46:29.618 IST [10620] LOG: database system is ready to accept connections
Related
I'm running an application with about 20 processes connected to a postgres DB (10.0) on windows server 2016.
Since about a month I have unexpected crashes of postgres.exe.
To isolate the problem I extended the logging by setting log_min_duration_statement = 0
This creates more detailed logfile. What I can see is:
LOG: server process (PID xxxxx) was terminated by exception
0xFFFFFFFF DETAIL: Failed process was running: COMMIT HINT: See C
include file "ntstatus.h" for a description of the hexadecimal value.
Then it tears down all 20 processes like this:
LOG: terminating any other active server processes
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and repeat your command.
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.Then DB recovers:
HINT: In a moment you should be able to reconnect to the database and repeat your command.
LOG: all server processes terminated; reinitializing
LOG: database system was interrupted; last known up at 2021-06-11 18:17:18 CEST
DB enters recovery mode
FATAL: the database system is in recovery mode
FATAL: the database system is in recovery mode
FATAL: the database system is in recovery mode
FATAL: the database system is in recovery mode
LOG: database system was not properly shut down; automatic recovery in progress
...
LOG: redo starts at 1B2/33319E58
FATAL: the database system is in recovery mode
LOG: invalid record length at 1B2/33D29930: wanted 24, got 0
LOG: redo done at 1B2/33D29908
LOG: last completed transaction was at log time 2021-06-11 18:21:39.830526+02
FATAL: the database system is in recovery mode
...
FATAL: the database system is in recovery mode
LOG: database system is ready to accept connections
Now it's running again like normal
The crashed PID xxxxx I can identify to a postgres.exe running for one of the 20 application processes. It's not always the same one. This happens about every 5-10 days.
Can anybody give me some advice how to track down the reason of this crash?
Extensions used:
oracle_fdw 2.0.0, PostgreSQL 10.0, Oracle client 11.2.0.3.0, Oracle server 11.2.0.2.0
Crashdump:
Followed the link :
https://wiki.postgresql.org/wiki/Getting_a_stack_trace_of_a_running_PostgreSQL_backend_on_Windows
Although the postgres user has "full control" of the crashdump folder in the security tab it does not write something. Folder stays empty.
Follow-Up on the comment #Laurenz Albe:
The COMMIT is not the reason of the crash. It is the last successfull executed command of the session. Explained on the following example:
Process gets a job and starts to do it's job
2021-06-15 16:27:51.100 CEST [25604] LOG: duration: 0.061 ms statement: DISCARD ALL
2021-06-15 16:27:51.100 CEST [25604] LOG: duration: 0.012 ms statement: BEGIN
2021-06-15 16:27:51.100 CEST [25604] LOG: duration: 0.015 ms statement: SET TRANSACTION ISOLATION LEVEL READ COMMITTED
now a lot of action going on within session 25604
and among others the oracle foreign datawrapper
2021-06-15 16:28:13.792 CEST [25604] LOG: duration: 0.016 ms execute <unnamed>: FETCH ALL FROM "<unnamed portal 689>"
finishes action successfully (data of the transaction in the database)
2021-06-15 16:28:13.823 CEST [25604] LOG: duration: 0.059 ms statement: COMMIT
a lot of action is going in different sessions
among others the oracle foreign datawrapper
more the 7 minutes afterwards the next job is requested and now postgres.exe crash
2021-06-15 16:36:01.524 CEST [17904] LOG: server process (PID 25604) was terminated by exception 0xFFFFFFFF
The process does not do DISCARD ALL, BEGIN and SET TRANSACTION ISOLATION LEVEL READ COMMITTED
It crashes immediately
My Conclusion:
"the possibly corrupted shared memory" was initiated by one of the processes before. Meaning between the last successful COMMIT and the new request.
That's a 7 minutes time span where the problem occurs.
Some feedback on this conclusion?
We have an application pipeline and Postgres-12(TimescaleDB, managed through Patroni) on a separate server (VM with Ubuntu 18.04 LTS).
We are facing an issue with the DB, it suddenly stuck in the recovery mode, and also we can’t access it from the psql client and select queries also hung.
After an hour or late all got back to normal (As my current pipeline terminated) and able to run queries against the DB server.
Master DB error details:
2020-11-03 18:35:08.612 IST [9773] [unknown]#[unknown] LOG: connection received: host=x.x.x.x port=58780
2020-11-03 18:35:08.612 IST [9773] FATAL: the database system is in recovery mode
2020-11-03 18:35:08.596 IST [18276] LOG: could not send data to client: Broken pipe
Replica server error details:
2020-11-03 18:34:55 IST [18316]: [85649-1] user=postgres,db=postgres,app=[unknown],client=x.x.x.x LOG: duration: 10.228 ms statement: SELECT * FROM pg_stat_bgwriter;
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and repeat your command.
2020-11-03 18:35:08 IST [18322]: [2-1] user=,db=,app=,client= FATAL: could not receive data from WAL stream: SSL SYSCALL error: EOF detected
2020-11-03 18:35:08 IST [20500]: [1-1] user=,db=,app=,client= FATAL: could not connect to the primary server: FATAL: the database system is in recovery mode
FATAL: the database system is in recovery mode
Pipeline error details:
Job aborted due to stage failure: Task 4 in stage 0.0 failed 3 times, most recent failure: Lost task 4.2 in stage 0.0 (TID 29, ip-x-x-x-x.ap-southeast-1.compute.internal, executor 19): org.postgresql.util.PSQLException: FATAL: the database system is in recovery mode at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:514) at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:141) at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:192) at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49) at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195) at org.postgresql.Driver.makeConnection(Driver.java:454) at org.postgresql.Driver.connect(Driver.java:256) at org.apache.spark.sql.execution.datasources.jdbc.DriverWrapper.connect(DriverWrapper.scala:45)
Please any advise on this issue?
What version of TimescaleDB are you running? In particular, there were some issues with 1.7.x if you try to query a read replica; we recommend upgrading to 1.7.4.
(Otherwise, there's not much information about to suggest what might have happened.)
https://github.com/timescale/timescaledb/releases/tag/1.7.4
Database is postgresql-9.5.1 in docker. My host machine has 3.75 GB memory, linux. In some methods I am inserting 490000 rows one after another using psycopg2 with below code.
student_list = [(name, surname, explanation)]
args_str = ','.join(cur.mogrify("(%s,%s,%s)", x) for x in student_list)
cur.execute('INSERT INTO students (name, surname, explanation) VALUES ' + args_str)
This makes my database docker memory seems full and gives these errors:
LOG: server process (PID 11219) was terminated by signal 9:
Killed DETAIL: Failed process was running LOG: terminating
any other active server processes docker#test_db WARNING:
terminating connection because of crash of another server process
docker#test_db DETAIL: The postmaster has commanded this server
process to roll back the current transaction and exit, because another
server process exited abnormally and possibly corrupted shared
memory. docker#test_db HINT: In a moment you should be able to
reconnect to the database and repeat your command. docker#test_db
WARNING: terminating connection because of crash of another server
process docker#test_db DETAIL: The postmaster has commanded this
server process to roll back the current transaction and exit, because
another server process exited abnormally and possibly corrupted shared
memory. ... docker#test_db FATAL: the database system is in
recovery mode LOG: all server processes terminated;
reinitializing LOG: database system was interrupted; last known
up at 2017-06-06 09:39:40 UTC LOG: database system was not
properly shut down; automatic recovery in progress docker#test_db
FATAL: the database system is in recovery mode docker#test_db
FATAL: the database system is in recovery mode docker#test_db
FATAL: the database system is in recovery mode LOG: autovacuum
launcher started
Script gives that log:
Inner exception
SSL SYSCALL error: EOF detected
I tried put some sleep time between consecutive queries but got same result. Is there any limitation for that?
Also I tried to connect and disconnect for each query but got same result. These are my connect and disconnect methods.
def connect():
conn = psycopg2.connect(database=database_name,
user=database_user,
host=database_host,
port=database_port)
conn
.set_isolation_level(psycopg2.extensions.ISOLATION_LEVEL_AUTOCOMMIT)
cur = conn.cursor()
return conn, cur
def disconnect(conn, cur):
cur.close()
conn.close()
Here is what I did. Actually my memory was full enough. That's why linux OS used to kill the process in Postgresql. There were 1M values in every insert process. The trick was I divided data lists to chunks and tried it 100k by 100k. That works very well. Thanks for your helps.
i am run pg_dump on my vps server, it throw me error:
pg_dump: [archiver (db)] query failed: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
pg_dump: [archiver (db)] query was: SELECT
( SELECT alias FROM pg_catalog.ts_token_type('22171'::pg_catalog.oid) AS t
WHERE t.tokid = m.maptokentype ) AS tokenname,
m.mapdict::pg_catalog.regdictionary AS dictname
FROM pg_catalog.pg_ts_config_map AS m
WHERE m.mapcfg = '22172'
ORDER BY m.mapcfg, m.maptokentype, m.mapseqno
Then I notice the sql on the above error:
SELECT
( SELECT alias FROM pg_catalog.ts_token_type('22171'::pg_catalog.oid) AS t
WHERE t.tokid = m.maptokentype ) AS tokenname,
m.mapdict::pg_catalog.regdictionary AS dictname
FROM pg_catalog.pg_ts_config_map AS m
WHERE m.mapcfg = '22172'
ORDER BY m.mapcfg, m.maptokentype, m.mapseqno
So I try to run SELECT alias FROM pg_catalog.ts_token_type('22171'::pg_catalog.oid) on psql
So it throw me error:
pzz_development=# SELECT alias FROM pg_catalog.ts_token_type('22171'::pg_catalog.oid);
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The connection to the server was lost. Attempting reset: Failed.
!> \q
How can I figure out the problem, and dump my data properly?
EDIT:
Then i check postgresql log at /var/log/postgresql/postgresql-9.3-main.log
2015-08-10 16:22:49 CST LOG: server process (PID 4029) was terminated by signal 11: Segmentation fault
2015-08-10 16:22:49 CST DETAIL: Failed process was running: SELECT
( SELECT alias FROM pg_catalog.ts_token_type('22171'::pg_catalog.oid) AS t
WHERE t.tokid = m.maptokentype ) AS tokenname,
m.mapdict::pg_catalog.regdictionary AS dictname
FROM pg_catalog.pg_ts_config_map AS m
WHERE m.mapcfg = '22172'
ORDER BY m.mapcfg, m.maptokentype, m.mapseqno
2015-08-10 16:22:49 CST LOG: terminating any other active server processes
2015-08-10 16:22:49 CST WARNING: terminating connection because of crash of another server process
2015-08-10 16:22:49 CST DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2015-08-10 16:22:49 CST HINT: In a moment you should be able to reconnect to the database and repeat your command.
2015-08-10 16:22:49 CST LOG: all server processes terminated; reinitializing
2015-08-10 16:22:49 CST LOG: database system was interrupted; last known up at 2015-08-10 16:22:45 CST
2015-08-10 16:22:50 CST LOG: database system was not properly shut down; automatic recovery in progress
2015-08-10 16:22:50 CST LOG: unexpected pageaddr 0/2AE6000 in log segment 000000010000000000000004, offset 11427840
2015-08-10 16:22:50 CST LOG: redo is not required
2015-08-10 16:22:50 CST LOG: MultiXact member wraparound protections are now enabled
2015-08-10 16:22:50 CST LOG: autovacuum launcher started
2015-08-10 16:22:50 CST LOG: database system is ready to accept connections
The database in use is Postgres database V8.Every one hour there is a server connection error.The server gets disconnected and needs to be re connected again.
Please find below the log of the error and let know on a solution to resolve this error
2012-01-05 13:28:52 CEST LOG: server process (PID 6128) was terminated by exception 0xC0000017
2012-01-05 13:28:52 CEST HINT: See C include file "ntstatus.h" for a description of the hexadecimal value.
2012-01-05 13:28:52 CEST LOG: terminating any other active server processes
2012-01-05 13:28:52 CEST WARNING: terminating connection because of crash of another server process
2012-01-05 13:28:52 CEST DETAIL:The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2012-01-05 13:28:52 CEST HINT:In a moment you should be able to reconnect to the database and repeat your command.
2012-01-05 13:28:52 CEST WARNING:terminating connection because of crash of another server process
2012-01-05 13:28:52 CEST DETAIL:The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory
Thanks in Advance
Apparently that status is STATUS_NO_MEMORY, so look at your server memory setup (shared_buffers, work_mem et al) and monitor the memory usage on the machine around the time it crashes (if it is regular). Does it always coincide with some sort of scheduled task?