I've recently started to work on a project with SRTM datas and I've extracted pbf file using phyghtmap.
To start I'm getting hgt files, converting them to tif using the following command : gdal_fillnodata.py data.hgt data.tif
Then I'm warping them with gdalwarp -co BIGTIFF=YES -co TILED=YES -co COMPRESS=LZW -co PREDICTOR=2 -t_srs "+proj=merc +ellps=sphere +R=6378137 +a=6378137 +units=m" -r bilinear -tr 90 90 data.tif warp-90.tif
And finally creating the pbf file with phyghtmap --max-nodes-per-tile=0 -s 10 -0 --pbf warp-90.tif
The results is a list of pbf files. They are perfectly fine when I load them into PostGIS with osm2pgsql. But I want to merge them to fasten the import.
I've tried all the major solutions:
osmium merge *.pbf -o merged.pbf
convert pbf to o5m then osmconvert64 *.o5m -o=merge.o5m then convert back to pbf
merging two by two with osmosis --read-pbf lon4.00_5.00lat44.00_45.00_local-source.pbf --read-pbf lon5.00_6.00lat44.00_45.00_local-source.osm.pbf --merge --write-pbf osmo_merge.osm.pbf
None of them worked and the result is only a very small fraction of the data merged in the result file.
Am I doing something wrong?
Note: If I load all pbf with --append it works, but it take ages for a very small portion of the world.
I found the issue. I wasn't setting the --start-node-id and --start-way-id in my script so all my pbf was using the same id range. Now I'm assigning unique ID and it work like a charm :)
Related
I was wondering if it is possible to run densepose annotations on a mp4 with detectron2?
In the projects folder, you can run densepose with applynet.py but this only works on images. I tried running this commmand
d demo/
python demo.py --config-file ../configs/COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml \
--video-input video.mp4 \
[--other-options]
--opts MODEL.WEIGHTS detectron2://COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x/137849600/model_final_f10217.pkl
with densepose weights and annotations but Detectron2 gives me this error:
Non-existent config key: MODEL.DENSEPOSE_ON
I know DensePose video exists but it is out of date as it uses caffe2 separated from pytorch.
Is this possible or can you not run on video?
Easiest approach is to use something like ffmpeg to split your video into frames, and then run detectron2 on each frame.
I am trying to use pgbadger to make html report for postgres slow query log files. My postgres logfiles are in csvlog format in folder pg_log. I transfer all logfiles
(80 files with 10 MB each) to my local windows machine and trying to generate single html report for all files. I created all one file from all files in below way,
type postgresql-2020-06-18_075333.csv > postgresql.csv
type postgresql-2020-06-18_080011.csv >> postgresql.csv
....
....
type postgresql-2020-06-18_094812.csv >> postgresql.csv
I downloaded "pgbadger-11.2" and tried below command but getting error.
D:\pgbadger-11.2>perl --version
This is perl 5, version 28, subversion 1 (v5.28.1) built for MSWin32-x64-multi-thread
D:\pgbadger-11.2>perl pgbadger "D:\June-Logs\postgresql.csv" -o postgresql.html
[========================>] Parsed 923009530 bytes of 923009530 (100.00%), queries: 1254764, events: 53
can not load incompatible binary data, binary file is from version < 4.0.
LOG: Ok, generating html report...
postgresql.html is created but no data in any tab.But it works when i create separate report for individual csv. like below.
D:\pgbadger-11.2>perl pgbadger "D:\June-Logs\postgresql-2020-06-18_075333.csv" -o postgresql-2020-06-18_075333.html
D:\pgbadger-11.2>perl pgbadger "D:\June-Logs\postgresql-2020-06-18_080011.csv" -o postgresql-2020-06-18_080011.html
...
D:\pgbadger-11.2>perl pgbadger "D:\June-Logs\postgresql-2020-06-18_094812.csv" -o postgresql-2020-06-18_094812.html
Please suggest me something to fix this issue.
I going to say this due to:
type postgresql-2020-06-18_075333.csv > postgresql.csv
type postgresql-2020-06-18_080011.csv >> postgresql.csv
Pretty sure that is introducing Windows line endings and pgBadger is looking for Unix line endings. Can you do the concatenate on the server?
UPDATE. Hmm. Ran across this
https://github.com/darold/pgbadger/releases
"This new release breaks backward compatibility with old binary or JSON
files. This also mean that incremental mode will not be able to read
old binary file [...] Add a warning about version and skip loading incompatible binary file.
Update code formatter to pgFormatter 4.0."
Not sure why it is failing on CSV logs, still what is version of pgBadger generating logs?
I used to check the sizes of layers in an image using the Docker history command, although now that shows "missing" instead of layer IDs due to the 1.10 migration to content hashes.
I now retrieve the hashes of all layers in an image through these commands:
docker pull ubuntu
ID=$(docker inspect -f {{.Id}} ubuntu)
sudo jq .rootfs.diff_ids /var/lib/docker/image/aufs/imagedb/content/$(echo $ID|tr ':' '/')
This returns a list of content hashes of all layers in the ubuntu image:
"diff_ids": [
"sha256:2a4049cf895d2384cb93d19f46f0d62560a48b2b202787edad2dc6e4b95a923a",
"sha256:01fbb4b5fa1b76ccdc289de098ea61925c7f8d3364159761720617b096f27bcc",
"sha256:d3492de15d7c87ea9db9ab123214d334f4bcb1e40846b77beebb4c37dd134a45",
"sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef"
],
In /var/lib/docker/image/aufs/layerdb/sha256/ I see information about each layer such as parent and size but I noticed that the diff_ids in this folder are not the same as the above output:
> ls /var/lib/docker/image/aufs/layerdb/sha256/
2088e4744016dbe95308d1920060f1fbc4a095ba5b9517d758745fc3986f2632
2a4049cf895d2384cb93d19f46f0d62560a48b2b202787edad2dc6e4b95a923a
8c63d05abe660a2f3f04d754de3ee3d927a17b3623a8e2be6d727e697f4b1e10
f747ac597de13b7f1ff918874f80bb83004232d7d6d4d45ad8890b58cdc79adc
I then tried inspecting another folder such as /var/lib/docker/aufs/layers:
> ls /var/lib/docker/aufs/layers#
58e7ed1f6d4ba047c9c714e66f10c014008ef4aa133d334198b8b1b7673f16e7
c4dd5a81188e36457624849aaeea74d98ef571390db75d4a03efb5bccb8c04e3
d31f918b7f59fcf768a9ae609141152cd5ae63943aac042429e3d2e04d472bcc
e576c6d41b96bd6a47233a6c6ec2f586021aa945aae6bd0e73ab9d4ad051a94e
As you can see these are 4 other content hashes again. Can someone tell me what the connection is between all these hashes and how I can find the size of each layer of the Ubuntu image? I'd like to be able to match each diff_id in the first output with a size but I don't know how all these diff_ids in different folders are related.
EDIT: I solved it like this - /var/lib/docker/image/aufs/layerdb/sha256/ also contains a file called "diff" which contains the diff_id corresponding to the output of the first command. I used this output to map the size to the correct diff_id.
I solved it like this: /var/lib/docker/image/aufs/layerdb/sha256/ also contains a file called "diff" which contains the diff_id corresponding to the output of the first command. I used this output to map the size to the correct diff_id.
I need to figure out the process to load multiple OSM files into a Nominatim database. I have everything setup and can load a single file with no issues.
Basically what I'm trying to do is load some of the GeoFabrik OSM files for only a part of the world. So I'm grabbing like the North America and South America OSM files. Or any 2 on their site.
For the first load I use the setup.php:
./utils/setup.php --osm-file file.osm --all --osm2pgsql-cache 4000
I'm not sure if I have another file (file2.osm) how to load this into the database and keep the original data.
Basically, I just want pieces of the world and I only need to load data like every six months or so. I don't need daily updates/ etc...
I need to split the files up because it just takes too long to load and I want to manage it better.
Can I use the update.php..... But not sure what parameters.
I thought about loading all data with update and the no-index clause...Then maybe building the index??
I did try to re-run the setup.php for the second file but it just hung for a long time
For second file
./utils/setup.php --import-data --osm-file file2.osm --osm2pgsql-cache 4000
But this just hangs on Setting up table: planet_osm_ways. (I tested very small OSM files that should finish within minutes but it just hangs).
The files that I'm using are all non-intersecting so not truly updates. SO I have a North America and a South America...How do I load both into Nominatim separately.
Thanks
The answer can be found at help.openstreetmap.org.
First you need to import it via the update script: ./utils/update.php --import-file <yourfile>. Then you need to trigger a re-indexing of the data: ./utils/update.php --index
But according to lonvia (one of the Nominatim developers) this will be very slow and it is better if you merge all your files first and then import it as one large file.
Sample Merging Code, merging Andorra, Malta and Liechtenstein,
curl -L 'http://download.geofabrik.de/europe/andorra-latest.osm.pbf' --create-dirs -o /srv/nominatim/src/andorra.osm.pbf
curl -L 'http://download.geofabrik.de/europe/malta-latest.osm.pbf' --create-dirs -o /srv/nominatim/src/malta.osm.pbf
curl -L 'http://download.geofabrik.de/europe/liechtenstein-latest.osm.pbf' --create-dirs -o /srv/nominatim/src/liechtenstein.osm.pbf
osmconvert /srv/nominatim/src/andorra.osm.pbf -o=/srv/nominatim/src/andorra.o5m
osmconvert /srv/nominatim/src/malta.osm.pbf -o=/srv/nominatim/src/malta.o5m
osmconvert /srv/nominatim/src/liechtenstein.osm.pbf -o=/srv/nominatim/src/liechtenstein.o5m
osmconvert /srv/nominatim/src/andorra.o5m /srv/nominatim/src/malta.o5m /srv/nominatim/src/liechtenstein.o5m -o=/srv/nominatim/src/data.o5m
osmconvert /srv/nominatim/src/data.o5m -o=/srv/nominatim/src/data.osm.pbf;
More about OsmConvert -> https://wiki.openstreetmap.org/wiki/Osmconvert
Once merged, you can,
sudo -u nominatim /srv/Nominatim/build/utils/setup.php \
--osm-file /srv/nominatim/src/data.osm.pbf \
--all \
--threads ${BUILD_THREADS} \ # 16 Threads?
--osm2pgsql-cache ${OSM2PGSQL_CACHE} # 24000 ?
So, site that used to use FTP now has an HTTP front-end and won't allow FTP connections. The site in question (for an example directory) will show a page with links to different dates. Inside each of these different dates, there are many files, and I typically just need to get some file with some clear pattern e.g. *h17v04*.hdf. I thought this could work:
wget -I "${PLATFORM}/${PRODUCT}/${YEAR}.*" -r -l 4 \
--user-agent="Mozilla/5.0 (Windows NT 5.2; rv:2.0.1) Gecko/20100101 Firefox/4.0.1" \
--verbose -c -np -nc -nd \
-A "*h17v04*.hdf" http://e4ftl01.cr.usgs.gov/$PLATFORM/$PRODUCT/
where PLATFORM=MOLT, PRODUCT=MOD09GA.005 and YEAR=2004, for example. This seems to start looking into all the useful dates, finds the index.html, and then just skips to the next directory, without downloading the relevant hdf file:
--2013-06-14 13:09:18-- http://e4ftl01.cr.usgs.gov/MOLT/MOD09GA.005/2004.01.01/
Reusing existing connection to e4ftl01.cr.usgs.gov:80.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `e4ftl01.cr.usgs.gov/MOLT/MOD09GA.005/2004.01.01/index.html'
[ <=> ] 174,182 134K/s in 1.3s
2013-06-14 13:09:20 (134 KB/s) - `e4ftl01.cr.usgs.gov/MOLT/MOD09GA.005/2004.01.01/index.html' saved [174182]
Removing e4ftl01.cr.usgs.gov/MOLT/MOD09GA.005/2004.01.01/index.html since it should be rejected.
--2013-06-14 13:09:20-- http://e4ftl01.cr.usgs.gov/MOLT/MOD09GA.005/2004.01.02/
[...]
If I ignore the -A option, only the index.html file is downloaded to my system, but it appears it's not parsed and the links are not followed. I don't really know what more is required to make this work, as I can't see why it doesn't!!!
SOLUTION
In the end, the problem was due to an old bug in the local version of wget. However, I ended up writing my own script for downloading MODIS data from the server above. The script is pure Python, and is available from here.
Consider to use pyModis instead of wget which is a Free and Open Source Python based library to work with MODIS data. It offers bulk-download for user selected time ranges, mosaicking of MODIS tiles, and the reprojection from Sinusoidal to other projections, convert HDF format to other formats. See
http://www.pymodis.org/