As described here: https://community.openstreetmap.org/t/stuck-at-this-step-sudo-make-import-sql-area-planet-latest/6843 ,
I followed the indications found here: https://www.linuxbabe.com/ubuntu/basemap-tileserver-gl-openmaptiles-ubuntu-22-04
I first tried the whole process for a small data, the Isle of Man, and everything works fine
But with the data of the entire world, I'm stuck at this sub-step, of step 8 :
root#vmi1046457:~/openmaptiles# sudo make import-sql area=planet-latest
Time: 1.587 ms
psql:/sql/parallel/water__waterway.sql:1016: NOTICE: schema "waterway_important" already exists, skipping
CREATE SCHEMA
Time: 9.567 ms
psql:/sql/parallel/water__waterway.sql:1027: NOTICE: relation "changes" already exists, skipping
CREATE TABLE
Time: 39.958 ms
CREATE FUNCTION
Time: 197.301 ms
psql:/sql/parallel/water__waterway.sql:1048: NOTICE: relation "updates" already exists, skipping
CREATE TABLE
Time: 1.386 ms
CREATE FUNCTION
Time: 10.781 ms
CREATE FUNCTION
Time: 54.163 ms
CREATE FUNCTION
Time: 13.839 ms
CREATE TRIGGER
Time: 159.570 ms
CREATE TRIGGER
Time: 1100.683 ms (00:01.101)
CREATE TRIGGER
Time: 42.035 ms
CREATE TRIGGER
Time: 71.131 ms
CREATE FUNCTION
Time: 18.919 ms
This step is very cpu-intentive : the 10 vcpus usage is within the this range :75% - 99%
The VPS I'm using of a Cloud Service Provider has the following features:
A 10 core CPU
60 GB RAM
1.6 TB Intel Optane SSD
Are these technical features enough for importing into postgresql DB the data of the entire world?
What can I do to make this step : sudo make import-sql speed up and finish?
Related
I am reading 1 GB of CSV file ( record count : 10 million, columns : 13 ) and trying to dump it into the SQL server. Below are the infra details :
CSV file location : azure blob storage
Code : Spark + Scala
Cluster : Databricks
Size :
Code Used to read the file and dump it :
val df = spark.read.format(fileparser_config("fileFormat").as[String]).option("header", fileparser_config("IsFirstRowHeader").toString).load(fileparser_config("FileName").as[String]).withColumn("_ID", monotonically_increasing_id)
val bulkCopyConfig = Config(Map(
"url" -> connConfig("dataSource").as[String],
"databaseName" -> connConfig("dbName").as[String],
"user" -> connConfig("userName").as[String],
"password" -> connConfig("password").as[String],
"dbTable" -> tableName,
"bulkCopyBatchSize" -> "500000",
"bulkCopyTableLock" -> "true",
"bulkCopyTimeout" -> "600"))
println(s" ${LocalDateTime.now()} ************ sql bulk insert start ************")
df.bulkCopyToSqlDB(bulkCopyConfig)
println(s" ${LocalDateTime.now()} ************ sql bulk insert end ************")
Problem :
the cluster goes into a limbo and my job never completes. One time when it ran long enough it threw an error :
org.apache.spark.SparkException: Job aborted due to stage failure: Task 13 in stage 38.0 failed 4 times, most recent failure: Lost task 13.3 in stage 38.0 (TID 1532, 10.0.6.6, executor 4): com.microsoft.sqlserver.jdbc.SQLServerException: The connection is closed.\n\tat com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDriverError(SQLServerException.java:227)\n\tat com.microsoft.sqlserver.jdbc.SQLServerConnection.checkClosed(SQLServerConnection.java:796)\n\tat com.microsoft.sqlserver.jdbc.SQLServ
Cluster Event logs :
Other observations :
While the job runs for a very long time, the cluster is not completely unresponsive. I tried this by submitting more jobs in that same window. The job ran but took comparatively more time than usual( around 10x time)
I tried increasing the worker nodes and the node type ( even chose 128 GB nodes ) but still the outcome was same.
While the job was running, I tried checking the SQL table row count with nolock query. I ran this after 3-4 minutes while the job was running, it gave me around 2 million records in the table. But when I ran it again after 10 minutes, the query kept running forever and never returned any records.
I have tried tweaking the bulkCopyBatchSize property but it hasnt helped much.
I have tried to remove the sqlinsertion code and used an aggregation operation on the dataframe that i create from 1 GB file and the entire thing takes only 40-50 seconds, so the problem is only with sql driver/sql server.
I was facing the same issue.
Azure SQL Server - Standard S7: 800 DTUs
HDInsight - 6 node (2 D13V2 Head and 4 D13V2 Worker)
Data Size - 100GB Parquet with 1.7 billion rows.
Initially I was using "bulkCopyTimeout" as 600 seconds and I observed the loading was restarting after the timeout passed. Then I changed the timeout to a very large value and it worked fine.
For performance improvement:
Create columnstore Index in the target table and use
"bulkCopyBatchSize" = 1048576 (Load whole batch into row-groups maximum capacity and compresses them directly into the column-store rather than loading into delta store and compressing later)
"bulkCopyTableLock" = "false" (in order to allow parallelism)
I am following this link and try to simulate the deadlock issue:
http://www.dba-db2.com/2012/06/how-to-monitor-a-deadlock-in-db2.html
I can see my command run successful.
After that I go to simulate a deadlock error through DbVisualiser tool. However I didnt see any file being generated to the path.
Can someone point the mistake to me?
And also, I try to read back those old 0000000.evt file, it show me something as follow:
EVENT LOG HEADER
Event Monitor name: DB2DETAILDEADLOCK
Server Product ID: SQL10059
Version of event monitor data: 12
Byte order: BIG ENDIAN
Number of nodes in db2 instance: 1
Codepage of database: 1208
Territory code of database: 1
Server instance name: db2inst1
--------------------------------------------------------------------------
--------------------------------------------------------------------------
Database Name: MYDB
Database Path: /db2home/db2inst1/NODE0000/SQL00003/MEMBER0000/
First connection timestamp: 01/29/2018 10:00:17.694784
Event Monitor Start time: 01/29/2018 10:00:18.951331
--------------------------------------------------------------------------
--------------------------------------------------------------------------
Database Name: MYDB
Database Path: /db2home/db2inst1/NODE0000/SQL00003/MEMBER0000/
First connection timestamp: 01/29/2018 10:12:54.382936
Event Monitor Start time: 01/29/2018 10:12:54.697223
--------------------------------------------------------------------------
This means no deadlock?
Works correctly for me (linux, Db2 v11.1). Here are some command lines with annotations. You need to have suitable authorisation/privilege for each command. I was using the instance owner account.
Disable default db2detaildeadlock monitor first and then create your own:
$ db2 "set event monitor db2detaildeadlock state=0"
DB20000I The SQL command completed successfully.
$
$ db2 "create event monitor dlmon for deadlocks write to file '/tmp'"
DB20000I The SQL command completed successfully.
$
$ db2 "set event monitor dlmon state=1"
DB20000I The SQL command completed successfully.
$
Generate a deadlock, ensure you see this SQLCODE -911 with reason code 2.
If you dont' see the reason code 2 then you don't have any deadlock but you might have a timeout and timeouts don't get recorded in the deadlock monitor.
Here I show the victim of the deadlock getting notified of rollback and you can see the correct reason code:
$ db2 +c "select * from db2inst1.dlk where a=4 with rr"
SQL0911N The current transaction has been rolled back because of a deadlock
or timeout. Reason code "2". SQLSTATE=40001
Investigate the monitor output with db2evmon and view resulting file
$ db2evmon -db mydb -evm dlmon > /tmp/db2evmon.dlmon.1
Reading /tmp/00000000.evt ...
$ view /tmp/db2evmon.dlmon.1
...<snip>
...
3) Deadlock Event ...
Deadlock ID: 2
Number of applications deadlocked: 2
Deadlock detection time: 01/03/2018 09:06:39.019854
Rolled back Appl participant no: 2
Rolled back Appl Id: *LOCAL.db2inst1.180301090546
Rolled back Appl seq number: 00001
Rolled back Appl handle: 11872
...<snip>
I've looked at previous posts on this topic but couldn't understand why the import process died. Therefore I cannot start it again until I understand what the errors in the log mean. (import took 12 days)
here's the famous nominatim setup log
CREATE TABLE
CREATE TABLE
CREATE TYPE
Import
osm2pgsql version 0.93.0-dev (64 bit id space)
Using projection SRS 4326 (Latlong)
NOTICE: table "place" does not exist, skipping
Allocating memory for dense node cache
Allocating dense node cache in one big chunk
Allocating memory for sparse node cache
Sharing dense sparse
Node-cache: cache=2800MB, maxblocks=44800*65536, allocation method=11
Mid: loading persistent node cache from /srv/nominatim/data/flatnode.file
Mid: pgsql, cache=2800
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels
Reading in file: /srv/nominatim/data/europe-latest.osm.pbf
Using PBF parser.
Processing: Node(1916319k 644.1k/s) Way(234287k 0.25k/s) Relation(3519490 21.55/s) parse time: 1109177s
Node stats: total(1916319008), max(4937556462) in 2975s
Way stats: total(234287257), max(503447033) in 942887s
Relation stats: total(3519494), max(7358761) in 163315s
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels
Going over pending ways...
0 ways are pending
Using 1 helper-processes
Finished processing 0 ways in 0 s
Going over pending relations...
0 relations are pending
Using 1 helper-processes
Finished processing 0 relations in 0 s
Stopping table: planet_osm_nodes
Stopped table: planet_osm_nodes in 0s
Stopping table: planet_osm_ways
Stopped table: planet_osm_ways in 0s
Stopping table: planet_osm_rels
Building index on table: planet_osm_rels
ERROR: Error executing external command: /srv/nominatim/Nominatim/build/osm2pgsql/osm2pgsql --flat-nodes /srv/nominatim/data/flatnode.file -lsc -O gazetteer --hstore --number-processes 1 -C 2800 -P 5432 -d nominatim /srv/nominatim/data/europe-latest.osm.pbf
Error executing external command: /srv/nominatim/Nominatim/build/osm2pgsql/osm2pgsql --flat-nodes /srv/nominatim/data/flatnode.file -lsc -O gazetteer --hstore --number-processes 1 -C 2800 -P 5432 -d nominatim /srv/nominatim/data/europe-latest.osm.pbf
Can anyone help?
Thanks in advance
I installed Postgres/Postgis for mac osx. I followed all the steps needed to import osm data into the database as required (this includes setting up a user, a database and having spherical mercator projection done) but for some reason, the tables planet_osm_line, planet_osm_roads, planet_osm_polygon, etc are not even showing up in the database I created. Following is the success message for importing osm data in the database:
Using projection SRS 900913 (Spherical Mercator)
Setting up table: planet_osm_point
NOTICE: table "planet_osm_point_tmp" does not exist, skipping
Setting up table: planet_osm_line
NOTICE: table "planet_osm_line_tmp" does not exist, skipping
Setting up table: planet_osm_polygon
NOTICE: table "planet_osm_polygon_tmp" does not exist, skipping
Setting up table: planet_osm_roads
NOTICE: table "planet_osm_roads_tmp" does not exist, skipping
Using built-in tag processing pipeline
Allocating memory for dense node cache
Allocating dense node cache in one big chunk
Allocating memory for sparse node cache
Sharing dense sparse
Node-cache: cache=800MB, maxblocks=102400*8192, allocation method=3
Mid: Ram, scale=100
Reading in file: map.osm
Processing: Node(0k 0.9k/s) Way(0k 0.09k/s) Relation(0 0.00/s) parse time: 0s
Node stats: total(923), max(3044483353) in 0s
Way stats: total(86), max(291932583) in 0s
Relation stats: total(0), max(0) in 0s
Committing transaction for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Committing transaction for planet_osm_roads
...
Creating indexes on planet_osm_roads finished
All indexes on planet_osm_roads created in 0s
Completed planet_osm_roads
Osm2pgsql took 0s overall
I followed the steps listed here: http://skipperkongen.dk/2011/01/20/how-to-import-open-street-map-data-into-postgresql/
and here: https://wiki.archlinux.org/index.php/GpsDrive
but still the relations above do not show up. I just get an error saying such relations do not exist when I query my database. What else am I missing here?
I am having a weird, recurring but not constant, error where I get "2013, 'Lost connection to MySQL server during query'". These are the premises:
a Python app runs around 15-20minutes every hour and then stops (hourly scheduled by cron)
the app is on a GCE n1-highcpu-2 instance, the db is on a D1 with a per package pricing plan and the following mysql flags
max_allowed_packet 1073741824
slow_query_log on
log_output TABLE
log_queries_not_using_indexes on
the database is accessed only by this app and this app only so the usage is the same, around 20 consecutive minutes per hour and then nothing at all for the other 40 minutes
the first query it does is
SELECT users.user_id, users.access_token, users.access_token_secret, users.screen_name, metadata.last_id
FROM users
LEFT OUTER JOIN metadata ON users.user_id = metadata.user_id
WHERE users.enabled = 1
the above query joins two tables that are each around 700 lines longs and do not have indexes
after this query (which takes 0.2 seconds when it runs without problems) the app starts without any issues
Looking at the logs I see that each time this error presents itself the interval between the start of the query and the error is 15 minutes.
I've also enabled the slow query log and those query are registered like this:
start_time: 2014-10-27 13:19:04
query_time: 00:00:00
lock_time: 00:00:00
rows_sent: 760
rows_examined: 1514
db: foobar
last_insert_id: 0
insert_id: 0
server_id: 1234567
sql_text: ...
Any ideas?
If your connection is idle for the 15 minute gap the you are probably seeing GCE disconnect your idle TCP connection, as described at https://cloud.google.com/compute/docs/troubleshooting#communicatewithinternet. Try the workaround that page suggests:
sudo /sbin/sysctl -w net.ipv4.tcp_keepalive_time=60 net.ipv4.tcp_keepalive_intvl=60 net.ipv4.tcp_keepalive_probes=5
(You may need to put this configuration into /etc/sysctl.conf to make it permanent)