I've looked at previous posts on this topic but couldn't understand why the import process died. Therefore I cannot start it again until I understand what the errors in the log mean. (import took 12 days)
here's the famous nominatim setup log
CREATE TABLE
CREATE TABLE
CREATE TYPE
Import
osm2pgsql version 0.93.0-dev (64 bit id space)
Using projection SRS 4326 (Latlong)
NOTICE: table "place" does not exist, skipping
Allocating memory for dense node cache
Allocating dense node cache in one big chunk
Allocating memory for sparse node cache
Sharing dense sparse
Node-cache: cache=2800MB, maxblocks=44800*65536, allocation method=11
Mid: loading persistent node cache from /srv/nominatim/data/flatnode.file
Mid: pgsql, cache=2800
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels
Reading in file: /srv/nominatim/data/europe-latest.osm.pbf
Using PBF parser.
Processing: Node(1916319k 644.1k/s) Way(234287k 0.25k/s) Relation(3519490 21.55/s) parse time: 1109177s
Node stats: total(1916319008), max(4937556462) in 2975s
Way stats: total(234287257), max(503447033) in 942887s
Relation stats: total(3519494), max(7358761) in 163315s
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels
Going over pending ways...
0 ways are pending
Using 1 helper-processes
Finished processing 0 ways in 0 s
Going over pending relations...
0 relations are pending
Using 1 helper-processes
Finished processing 0 relations in 0 s
Stopping table: planet_osm_nodes
Stopped table: planet_osm_nodes in 0s
Stopping table: planet_osm_ways
Stopped table: planet_osm_ways in 0s
Stopping table: planet_osm_rels
Building index on table: planet_osm_rels
ERROR: Error executing external command: /srv/nominatim/Nominatim/build/osm2pgsql/osm2pgsql --flat-nodes /srv/nominatim/data/flatnode.file -lsc -O gazetteer --hstore --number-processes 1 -C 2800 -P 5432 -d nominatim /srv/nominatim/data/europe-latest.osm.pbf
Error executing external command: /srv/nominatim/Nominatim/build/osm2pgsql/osm2pgsql --flat-nodes /srv/nominatim/data/flatnode.file -lsc -O gazetteer --hstore --number-processes 1 -C 2800 -P 5432 -d nominatim /srv/nominatim/data/europe-latest.osm.pbf
Can anyone help?
Thanks in advance
Related
Can someone suggest me what is the problem, why I am not able to enable integrity, although no constraint found with name of CATEGORY.
create function FCOK_ACCOUNT_CATEGORY_C2(xmlrecord XML) returns integer language sql contains sql no external action deterministic return xmlcast(xmlquery('$d/row/c2' passing xmlrecord as "d") as integer);
reorg table FCOK_ACCOUNT_TEST inplace
db2pd -db admindb –reorg
Database Member 0 -- Database ADMINDB -- Active -- Up 10 days 03:14:17 -- Date 2021-07-26-19.28.05.648688
Table Reorg Information:
Address TbspaceID TableID PartID MasterTbs MasterTab TableName Type IndexID TempSpaceID
0x0A001F00174DF588 4 21566 n/a n/a n/a FCOK_ACCOUNT Offline 0 4
0x0A001F0027981508 3 26880 n/a n/a n/a FCOK_ACCOUNT_TEST Online 0 3
Table Reorg Stats:
Address TableName Start End PhaseStart MaxPhase Phase CurCount MaxCount Status Completion
0x0A001F00174DF588 FCOK_ACCOUNT 07/16/2021 18:38:46 07/16/2021 18:43:15 07/16/2021 18:39:40 3 IdxRecreat 0 0 Done 0
0x0A001F0027981508 FCOK_ACCOUNT_TEST 07/26/2021 18:13:16 07/26/2021 18:14:38 n/a n/a n/a 0 0 Done 0
bash-4.2$
set integrity for FCOK_ACCOUNT_TEST off;
ALTER TABLE FCOK_ACCOUNT_TEST ADD CATEGORY INTEGER generated always as (FCOK_ACCOUNT_CATEGORY_C2(XMLRECORD))
set integrity for FCOK_ACCOUNT_TEST immediate checked;
db2 "set integrity for DB2ADMIN.FCOK_ACCOUNT_TEST immediate checked"
DB21034E The command was processed as an SQL statement because it was not a
valid Command Line Processor command. During SQL processing it returned:
SQL3603N Integrity processing through the SET INTEGRITY statement has found
an integrity violation involving a constraint, a unique index, a generated
column, or an index over an XML column. The associated object is identified by
"DB2ADMIN.FCOK_ACCOUNT_TEST.CATEGORY". SQLSTATE=23514
bash-4.2$
bash-4.2$ db2 "select TYPE, ENFORCED from SYSCAT.TABCONST where CONSTNAME='CATEGORY'"
TYPE ENFORCED
0 record(s) selected.
bash-4.2$ db2 "select COLSEQ,COLNAME from SYSCAT.KEYCOLUSE where CONSTNAME='CATEGORY'"
COLSEQ COLNAME
0 record(s) selected.
bash-4.2$ db2 "reorg table db2admin.FCOK_ACCOUNT_TEST inplace"
SQL0668N Operation not allowed for reason code "1" on table
"DB2ADMIN.FCOK_ACCOUNT_TEST". SQLSTATE=57016
I am new to Postgres and one of my reports that is using select and extracting JSON return the following error.
ERROR: unexpected chunk number 0 (expected 1) for toast value 12599063 in pg_toast_16687
SQL state: XX000
I do not know how to proceed in fixing my query. Any idea?
Run this command:
select reltoastrelid::regclass from pg_class where relname = 'table_name';
Where the table_name is where the error occur. Then check the result if it is the same toast# like pg_toast.pg_toast_XXXXX. Mine happen to be 16687
Then run these commands to reindex:
REINDEX table table_name;
REINDEX table pg_toast.pg_toast_16687;
VACUUM analyze table_name;
That is data corruption:
restore from backup
upgrade to the latest PostgreSQL minor release
check the hardware
I have a VPS with 6 core, 60GB RAM and 1TB SSD space.
I installed Nominatim on Ubuntu 18.04 following the official installation here Nominatim Installation
I tried to import the openstreet planet pbf file located here Planet OpenStreetMap
./utils/setup.php --osm-file planet-latest.osm.pbf --all
--osm2pgsql-cache 28000 2>&1 | tee setup.log
After few hours I got the error Segmentation fault ERROR: No Data see below:
> 2019-11-30 16:50:55 == Setup DB Postgres version found: 10 Postgis
> version found: 2.4 set_config
> ------------
>
> (1 row)
>
> 2019-11-30 16:51:04 == Import data osm2pgsql version 1.2.0 (64 bit id
> space)
>
> Allocating memory for dense node cache Processing: Node(618450k
> 2425.3k/s) Way(0k 0.00k/s) Relation(0 0.00/s) Allocating memory for sparse node cache Sharing dense sparse Node-cache: cache=28000MB,
> maxblocks=448000*65536, allocation method=11 Mid: loading persistent
> node cache from /srv/nominatim/flatnode/flatnode.file Mid: pgsql,
> cache=28000 Setting up table: planet_osm_nodes Setting up table:
> planet_osm_ways Setting up table: planet_osm_rels Processing:
> Node(621390k 2427.3k/s) Way(0k 0.00k/s) Relation(0
> 0.00/s)import-full.style'. Using projection SRS 4326 (Latlong) NOTICE: table "place" does not exist, skipping
>
> Reading in file:
> /srv/nominatim/Nominatim-3.4.0/build/planet-latest.osm.pbf Using PBF
> parser. Processing: Node(5612922k 3148.0k/s) Way(437617k 33.05k/s)
> Relation(0 0.00/s)Segmentation fault ERROR: No Data string(7) "No
> Data"
Any idea why I got this error please?
Thanks.
I am using a docker project in a rather strong server(120GB RAM and plenty of disc space).
When trying to run an import on postgres server I get the following error
Using projection SRS 4326 (Latlong)
NOTICE: table "place" does not exist, skipping
Allocating memory for dense node cache
Allocating dense node cache in one big chunk
Allocating memory for sparse node cache
Sharing dense sparse
Node-cache: cache=1207MB, maxblocks=154496*8192, allocation method=11
Mid: pgsql, scale=10000000 cache=1207
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels
Reading in file: /app/src/data.osm.pbf
Using PBF parser.
node cache: stored: 0(-nan%), storage efficiency: -nan% (dense blocks: 0, sparse nodes: 0), hit rate: -nan%
Osm2pgsql failed due to ERROR: PBF error: invalid BlobHeader size (> max_blob_header_size)
ERROR: Error executing external command: /app/src/osm2pgsql/osm2pgsql -lsc -O gazetteer --hstore --number-processes 1 -C 1207 -P 5432 -d nominatim /app/src/data.osm.pbf
How could I increase the max_blob_header_size?
I stumbled upon the same issue while feeding an S3 hosted PBF file into a Nominatim docker container.
Unfortunately I failed to configure the access to the PBF file properly so the docker container saved the XML error response as /app/src/data.osm.pbf. That's why the file header check fails.
I installed Postgres/Postgis for mac osx. I followed all the steps needed to import osm data into the database as required (this includes setting up a user, a database and having spherical mercator projection done) but for some reason, the tables planet_osm_line, planet_osm_roads, planet_osm_polygon, etc are not even showing up in the database I created. Following is the success message for importing osm data in the database:
Using projection SRS 900913 (Spherical Mercator)
Setting up table: planet_osm_point
NOTICE: table "planet_osm_point_tmp" does not exist, skipping
Setting up table: planet_osm_line
NOTICE: table "planet_osm_line_tmp" does not exist, skipping
Setting up table: planet_osm_polygon
NOTICE: table "planet_osm_polygon_tmp" does not exist, skipping
Setting up table: planet_osm_roads
NOTICE: table "planet_osm_roads_tmp" does not exist, skipping
Using built-in tag processing pipeline
Allocating memory for dense node cache
Allocating dense node cache in one big chunk
Allocating memory for sparse node cache
Sharing dense sparse
Node-cache: cache=800MB, maxblocks=102400*8192, allocation method=3
Mid: Ram, scale=100
Reading in file: map.osm
Processing: Node(0k 0.9k/s) Way(0k 0.09k/s) Relation(0 0.00/s) parse time: 0s
Node stats: total(923), max(3044483353) in 0s
Way stats: total(86), max(291932583) in 0s
Relation stats: total(0), max(0) in 0s
Committing transaction for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Committing transaction for planet_osm_roads
...
Creating indexes on planet_osm_roads finished
All indexes on planet_osm_roads created in 0s
Completed planet_osm_roads
Osm2pgsql took 0s overall
I followed the steps listed here: http://skipperkongen.dk/2011/01/20/how-to-import-open-street-map-data-into-postgresql/
and here: https://wiki.archlinux.org/index.php/GpsDrive
but still the relations above do not show up. I just get an error saying such relations do not exist when I query my database. What else am I missing here?