I want to import osm planet-131113.osm.pbf whit osm2pgsql.
But whent it proccessing relation the window throw and say "osm2pgsql has stopped working" and then proccess killed.What is the problem?
My hardware is:
Cpu:intel xeon E5-2620.
Ram:24GB
Mainbord:supermico x9dr3-f-o
HDD:seagate 2tb barracuda
My os:
windows server 2008 R2
and Postgresql9.3 with postgis2.1 bundle
Postgresql config:
effective-chache-size=28GB
shared_buffers 2GB
maintenance_work_mem = 2GB
work-mem=512MB
checkpoint_segments = 100
chekpoint_completion-target=0.9
autovacuum = off
fsync=off
synchronous-commit=off
full-page-writes=off
and osm2pgsql command:
F:\OSM\x64>osm2pgsql -d OSMPlanet -s -S default.style -C 16000 -U postgres -r pbf -k -v --number-processes 12 planet-131113.osm.pbf
osm2pgsql SVN version af61cae663 (64bit id space)
release notes: 'Windows version built by Dominik Perpeet (http://www.customdebug
.com/osm2pgsql/index.html)'
WARNING: osm2pgsql was compiled without fork, only using one process!
Using projection SRS 900913 (Spherical Mercator)
Setting up table: planet_osm_point
NOTICE: table "planet_osm_point" does not exist, skipping
NOTICE: table "planet_osm_point_tmp" does not exist, skipping
Setting up table: planet_osm_line
NOTICE: table "planet_osm_line" does not exist, skipping
NOTICE: table "planet_osm_line_tmp" does not exist, skipping
Setting up table: planet_osm_polygon
NOTICE: table "planet_osm_polygon" does not exist, skipping
NOTICE: table "planet_osm_polygon_tmp" does not exist, skipping
Setting up table: planet_osm_roads
NOTICE: table "planet_osm_roads" does not exist, skipping
NOTICE: table "planet_osm_roads_tmp" does not exist, skipping
Allocating memory for sparse node cache
Node-cache: cache=16000MB, maxblocks=2048001*zd, allocation method=8192
Mid: pgsql, scale=100 cache=16000
Setting up table: planet_osm_nodes
NOTICE: table "planet_osm_nodes" does not exist, skipping
Setting up table: planet_osm_ways
NOTICE: table "planet_osm_ways" does not exist, skipping
Setting up table: planet_osm_rels
NOTICE: table "planet_osm_rels" does not exist, skipping
Reading in file: planet-131113.osm.pbf
Processing: Node(2084251k 142.6k/s) Way(204149k 0.84k/s) Relation(95360 5.64/s)
postgresql log:
2013-12-09 00:46:19 PST ERROR: unexpected EOF on client connection with an open transaction
2013-12-09 00:46:19 PST CONTEXT: COPY planet_osm_nodes, line 1
2013-12-09 00:46:19 PST STATEMENT: COPY planet_osm_nodes FROM STDIN;
2013-12-09 00:46:19 PST ERROR: unexpected EOF on client connection with an open transaction
2013-12-09 00:46:19 PST CONTEXT: COPY planet_osm_rels, line 1
2013-12-09 00:46:19 PST STATEMENT: COPY planet_osm_rels FROM STDIN;
2013-12-09 00:46:19 PST LOG: could not send data to client: No connection could be made because the target machine actively refused it.
2013-12-09 00:46:19 PST STATEMENT: COPY planet_osm_ways FROM STDIN;
2013-12-09 00:46:19 PST LOG: could not send data to client: No connection could be made because the target machine actively refused it.
2013-12-09 00:46:19 PST STATEMENT: COPY planet_osm_nodes FROM STDIN;
2013-12-09 00:46:19 PST LOG: could not send data to client: No connection could be made because the target machine actively refused it.
2013-12-09 00:46:19 PST STATEMENT: COPY planet_osm_rels FROM STDIN;
2013-12-09 00:46:19 PST FATAL: connection to client lost
2013-12-09 00:46:19 PST FATAL: connection to client lost
2013-12-09 00:46:19 PST FATAL: connection to client lost
2013-12-09 00:46:19 PST FATAL: connection to client lost
2013-12-09 00:48:53 PST LOG: received fast shutdown request
2013-12-09 00:48:53 PST LOG: aborting any active transactions
2013-12-09 00:48:53 PST LOG: shutting down
2013-12-09 00:48:53 PST LOG: database system is shut down
Related
I have Postgresql 12.8.1 version and python 3.7.6 version installed on my system. I want to create a trigger function using plpython so I created plpython3 extension using CREATE EXTENSION plpython3u . While trying to compile the trigger function, I face the following error -
ERROR: server closed the connection unexpectedly This probably means
the server terminated abnormally before or while processing the
request
Trigger function code:
CREATE OR REPLACE FUNCTION getTemperature() RETURNS trigger as $pgsqlTrigger$
city = TD["new"]["City"]
Station = TD["new"]["Station"]
dtime = TD["new"]["dtime"]
import requests, json, time, pandas as pd, numpy as np
from datetime import datetime
from calendar import monthrange
from copy import deepcopy
import sys
template_request = "https://api.weather.com/v1/location/{station}/observations/historical.json?apiKey=apikey&units=m&startDate={start_date}&endDate={end_date}"
df_header = ["City", "Year", "Month", "Day", "Hour", "Temperature(C)", "Condition"]
def get_weather_data(city, year, month, day, station):
start_date = "%d%02d%02d" % (year, month, day)
end_date = "%d%02d%02d" % (year, month, day)
request = template_request.format(station=station, start_date=start_date, end_date=end_date)
request_data = json.loads(requests.get(request).content)
weather_data = []
last_dt = None
for observation in request_data["observations"]:
dt = datetime.fromtimestamp(observation["valid_time_gmt"]+3600)
if last_dt and dt.hour > (last_dt.hour + 1):
last_row = deepcopy(weather_data[-1])
last_row[4] = last_row[4]+1
weather_data.append(last_row)
weather_data.append([city, year, month, dt.day, dt.hour, observation["temp"], observation["wx_phrase"]])
last_dt = dt
return weather_data
dtime = datetime.strptime(dtime, '%Y-%m-%d %H:%M:%S%z')
data = get_weather_data(city, dtime.year, dtime.month, dtime.day, station)
weather_df = pd.DataFrame(data, columns=df_header).drop_duplicates(subset=["City", "Year", "Month", "Day", "Hour"])
avg = (weather_df["Temperature(C)"].values).mean()
weather_df = pd.DataFrame()
TD["new"]["temp"] = avg;
return NEW;
$pgsqlTrigger$ LANGUAGE plpython3u;
CREATE TRIGGER pgsqlTrigger
BEFORE INSERT ON tweets
FOR EACH ROW EXECUTE FUNCTION getTemperature();
I have not yet found a solution for this. Any ideas on what should I do?
UPDATE:
I am using windows 10 as my os.
Regarding how I installed my plpython3u extension. I was facing the error " no such module named plpython3u found" and the fix I found on net was to copy and paste the python37.dll from the local python folder to my C:/windows/System32 folder. After doing that I was able to create plpython3u extension by using the command - CREATE EXTENSION plpython3u.
I am using the default user "postgres" which has the superuser permission.
below is the output of the server log:
log file output:
2021-09-22 11:46:28.518 IST [10620] LOG: server process (PID 2316) was terminated by exception 0xC0000409
2021-09-22 11:46:28.518 IST [10620] DETAIL: Failed process was running: CREATE FUNCTION public.proc1()
2021-09-22 11:46:28.518 IST [10620] HINT: See C include file "ntstatus.h" for a description of the hexadecimal value.
2021-09-22 11:46:28.522 IST [10620] LOG: terminating any other active server processes
2021-09-22 11:46:28.547 IST [13168] WARNING: terminating connection because of crash of another server process
2021-09-22 11:46:28.547 IST [13168] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2021-09-22 11:46:28.547 IST [13168] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2021-09-22 11:46:28.566 IST [7776] WARNING: terminating connection because of crash of another server process
2021-09-22 11:46:28.566 IST [7776] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2021-09-22 11:46:28.566 IST [7776] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2021-09-22 11:46:28.586 IST [10620] LOG: all server processes terminated; reinitializing
2021-09-22 11:46:28.668 IST [9204] LOG: database system was interrupted; last known up at 2021-09-21 19:23:24 IST
2021-09-22 11:46:29.103 IST [14208] FATAL: the database system is in recovery mode
2021-09-22 11:46:29.549 IST [9204] LOG: database system was not properly shut down; automatic recovery in progress
2021-09-22 11:46:29.555 IST [9204] LOG: redo starts at 0/1D19AB8
2021-09-22 11:46:29.562 IST [9204] LOG: invalid record length at 0/1D21978: wanted 24, got 0
2021-09-22 11:46:29.563 IST [9204] LOG: redo done at 0/1D21940
2021-09-22 11:46:29.618 IST [10620] LOG: database system is ready to accept connections
I have taken dump for the db and it make it run short of memory for Postgresql.
I have then restarted postgresql but it failed to restart.And kept on giving me error
[FAIL] Starting PostgreSQL 9.4 database server: main[....] The PostgreSQL server failed to start. Please check the log output: ... failed!
failed!
and in log file there were following lines
2017-05-05 05:49:25 UTC LOG: could not close temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2017-05-05 05:49:30 UTC LOG: using stale statistics instead of current ones because stats collector is not responding
2017-05-05 05:49:35 UTC LOG: using stale statistics instead of current ones because stats collector is not responding
2017-05-05 05:49:35 UTC LOG: could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device
2017-05-05 05:49:35 UTC LOG: could not write temporary statistics file "pg_stat_tmp/db_85990.tmp": No space left on device
2017-05-05 05:49:35 UTC LOG: could not close temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2017-05-05 05:49:40 UTC LOG: could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device
2017-05-05 05:49:40 UTC LOG: could not close temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2017-05-05 05:49:45 UTC LOG: using stale statistics instead of current ones because stats collector is not responding
Please Help me to solve this issue if some one can.
thanks.
I want to move my postgresql databases to an external hard drive (HDD 2TB USB 3.0). I copied the whole directory:
/var/lib/postgresql/9.4/main/
to the external drive, preserving permissions, with a command (ran by the user postgres):
$ rsync -aHAX /var/lib/postgresql/9.4/main/* new_dir_path
First run of this command was interrupted, but in the second attempt I copied everything (basically one database of size 800 GB). In the file
/etc/postgresql/9.4/main/postgresql.conf
I changed the line
data_directory = '/var/lib/postgresql/9.4/main'
to point to the new location. I restarted the postgresql service, and when from the user postgres I run the command psql, I get:
psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
I didn't change any other settings. There is no pidfile 'postmaster.pid' in the new location (or in the old one). When I run a command
$ /usr/lib/postgresql/9.4/bin/postgres --single -D /etc/postgresql/9.4/main -P -d 1
I get
2017-03-16 20:47:39 CET [2314-1] DEBUG: mmap with MAP_HUGETLB failed, huge pages disabled: Cannot allocate memory
2017-03-16 20:47:39 CET [2314-2] NOTICE: database system was shut down at 2017-03-16 20:01:23 CET
2017-03-16 20:47:39 CET [2314-3] DEBUG: checkpoint record is at 647/4041B3A0
2017-03-16 20:47:39 CET [2314-4] DEBUG: redo record is at 647/4041B3A0; shutdown TRUE
2017-03-16 20:47:39 CET [2314-5] DEBUG: next transaction ID: 1/414989450; next OID: 112553
2017-03-16 20:47:39 CET [2314-6] DEBUG: next MultiXactId: 485048384; next MultiXactOffset: 1214064579
2017-03-16 20:47:39 CET [2314-7] DEBUG: oldest unfrozen transaction ID: 259446705, in database 12141
2017-03-16 20:47:39 CET [2314-8] DEBUG: oldest MultiXactId: 476142442, in database 12141
2017-03-16 20:47:39 CET [2314-9] DEBUG: transaction ID wrap limit is 2406930352, limited by database with OID 12141
2017-03-16 20:47:39 CET [2314-10] DEBUG: MultiXactId wrap limit is 2623626089, limited by database with OID 12141
2017-03-16 20:47:39 CET [2314-11] DEBUG: starting up replication slots
2017-03-16 20:47:39 CET [2314-12] DEBUG: oldest MultiXactId member is at offset 1191132700
2017-03-16 20:47:39 CET [2314-13] DEBUG: MultiXact member stop limit is now 1191060352 based on MultiXact 476142442
PostgreSQL stand-alone backend 9.4.9
backend>
but I don't now how to understand this output. When I revert the changes in the postgresql.conf file, everything works fine. Interestingly, few months ago I moved the database in the same way, but to the local directory, and it worked.
I use postgresql-9.4 and debian-jessie.
Thanks for your help!
UPDATE
Content of the log file:
$ cat /var/log/postgresql/postgresql-9.4-main.log
2017-03-14 17:07:16 CET [13822-2] LOG: received fast shutdown request
2017-03-14 17:07:16 CET [13822-3] LOG: aborting any active transactions
2017-03-14 17:07:16 CET [13827-3] LOG: autovacuum launcher shutting down
2017-03-14 17:07:16 CET [13824-1] LOG: shutting down
2017-03-14 17:07:16 CET [13824-2] LOG: database system is shut down
i am run pg_dump on my vps server, it throw me error:
pg_dump: [archiver (db)] query failed: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
pg_dump: [archiver (db)] query was: SELECT
( SELECT alias FROM pg_catalog.ts_token_type('22171'::pg_catalog.oid) AS t
WHERE t.tokid = m.maptokentype ) AS tokenname,
m.mapdict::pg_catalog.regdictionary AS dictname
FROM pg_catalog.pg_ts_config_map AS m
WHERE m.mapcfg = '22172'
ORDER BY m.mapcfg, m.maptokentype, m.mapseqno
Then I notice the sql on the above error:
SELECT
( SELECT alias FROM pg_catalog.ts_token_type('22171'::pg_catalog.oid) AS t
WHERE t.tokid = m.maptokentype ) AS tokenname,
m.mapdict::pg_catalog.regdictionary AS dictname
FROM pg_catalog.pg_ts_config_map AS m
WHERE m.mapcfg = '22172'
ORDER BY m.mapcfg, m.maptokentype, m.mapseqno
So I try to run SELECT alias FROM pg_catalog.ts_token_type('22171'::pg_catalog.oid) on psql
So it throw me error:
pzz_development=# SELECT alias FROM pg_catalog.ts_token_type('22171'::pg_catalog.oid);
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The connection to the server was lost. Attempting reset: Failed.
!> \q
How can I figure out the problem, and dump my data properly?
EDIT:
Then i check postgresql log at /var/log/postgresql/postgresql-9.3-main.log
2015-08-10 16:22:49 CST LOG: server process (PID 4029) was terminated by signal 11: Segmentation fault
2015-08-10 16:22:49 CST DETAIL: Failed process was running: SELECT
( SELECT alias FROM pg_catalog.ts_token_type('22171'::pg_catalog.oid) AS t
WHERE t.tokid = m.maptokentype ) AS tokenname,
m.mapdict::pg_catalog.regdictionary AS dictname
FROM pg_catalog.pg_ts_config_map AS m
WHERE m.mapcfg = '22172'
ORDER BY m.mapcfg, m.maptokentype, m.mapseqno
2015-08-10 16:22:49 CST LOG: terminating any other active server processes
2015-08-10 16:22:49 CST WARNING: terminating connection because of crash of another server process
2015-08-10 16:22:49 CST DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2015-08-10 16:22:49 CST HINT: In a moment you should be able to reconnect to the database and repeat your command.
2015-08-10 16:22:49 CST LOG: all server processes terminated; reinitializing
2015-08-10 16:22:49 CST LOG: database system was interrupted; last known up at 2015-08-10 16:22:45 CST
2015-08-10 16:22:50 CST LOG: database system was not properly shut down; automatic recovery in progress
2015-08-10 16:22:50 CST LOG: unexpected pageaddr 0/2AE6000 in log segment 000000010000000000000004, offset 11427840
2015-08-10 16:22:50 CST LOG: redo is not required
2015-08-10 16:22:50 CST LOG: MultiXact member wraparound protections are now enabled
2015-08-10 16:22:50 CST LOG: autovacuum launcher started
2015-08-10 16:22:50 CST LOG: database system is ready to accept connections
Lets say that due to some corruption, automatic recovery is triggered by postgres. This results in "redo start at 0/9A3F58" as I can in the database logs. As part of the recovery, I suppose it would try to insert the records for a table. Does it cause database insert triggers for that table to be executed as well. We are using postgres 8.4.
Snippet from postgres logs:
2015-06-17 10:43:34 PDT LOG: unexpected EOF on client connection
2015-06-17 10:43:34 PDT LOG: unexpected EOF on client connection
2015-06-17 10:43:34 PDT LOG: unexpected EOF on client connection
2015-06-17 10:43:34 PDT LOG: unexpected EOF on client connection
2015-06-19 08:55:30 CDT LOG: database system was interrupted; last known up at 2015-06-17 20:05:02 CDT
2015-06-19 08:55:30 CDT LOG: database system was not properly shut down; automatic recovery in progress
2015-06-19 08:55:30 CDT LOG: redo starts at 0/9A3F58
2015-06-19 08:55:30 CDT LOG: incomplete startup packet
2015-06-19 08:55:30 CDT FATAL: the database system is starting up
2015-06-19 08:55:30 CDT LOG: record with zero length at 0/E90334
2015-06-19 08:55:30 CDT LOG: redo done at 0/E90308
2015-06-19 08:55:30 CDT LOG: last completed transaction was at log time 2015-06-17 12:43:32.471831-05
2015-06-19 08:55:31 CDT LOG: database system is ready to accept connections
2015-06-19 08:55:31 CDT LOG: autovacuum launcher started
2015-06-19 08:59:29 CDT LOG: unexpected EOF on client connection
2015-06-19 08:59:29 CDT LOG: unexpected EOF on client connection
2015-06-19 08:59:29 CDT LOG: unexpected EOF on client connection
No, replay from the write-ahead logs absolutely do not cause triggers to fire. In fact, they can't because the write-ahead logs store block level changes, not row-level changes, and have no idea which SQL statement changed which row.