Get separate layers in an mvt - postgresql

Our platform currently uses pbf tiles(on s3) to render vector layers on Mapbox on the frontend. The pbf tiles have separate layers within them and we use Mapbox layers' source-layer property to style them as separate layers. We are now moving to PostGIS and we want to avoid s3 and directly render the vector tiles as mvt on the frontend.
I am trying to get separate layers from the mvt but the only layer I see is a 'default' layer. I identified this using this vizualizer tool. I am currently getting the mvt using this query -
WITH mvtgeom AS (
SELECT ST_AsMVTGeom(ST_Transform(ST_Force2D(geom), 3857),
ST_TileEnvelope(${z}, ${x}, ${y}))
AS geom,
layer,
gid
FROM site-name
WHERE ST_Intersects(geom, ST_Transform(ST_TileEnvelope(${z}, ${x}, ${y}), 4326)))
SELECT ST_AsMVT(mvtgeom.*) AS mvt FROM mvtgeom;
This issue is very close to my problem but the answer suggests using a UNION, which I want to avoid if possible considering we have to render hundreds of layers.
I also tried this. In this case, when I put the values of my layer-ids into an IN, I see that those set of layers are rendered as one layer.
(Is there something I need to explore in order to understand how vector tiles are generated so I know how exactly the separate layers are generated? I am new to PostGIS, and I'm not sure if I'm doing something wrong or missing something. Any suggestions are appreciated)

Below is the full return signature of ST_AsMvt
bytea ST_AsMVT(anyelement row, text name, integer extent, text geom_name, text feature_id_name);
Later in the docs:
name is the name of the layer. Default is the string "default".
Try using the id (or label, or some other thing specific to each layer) to create a distinct layer name for each layer:
WITH mvtgeom AS (
SELECT
ST_AsMVTGeom(ST_Transform(ST_Force2D(geom), 3857) AS geom
, layer
, gid
FROM
site-name
WHERE
ST_Intersects(
geom,
ST_Transform(ST_TileEnvelope(${z}, ${x}, ${y}), 4326)
)
)
SELECT
ST_AsMVT(
mvtgeom.*,
'layer_' || layer -- this is the layer name argument
) AS mvt
FROM
mvtgeom
GROUP BY
layer;
This name property, by the way, is what mapbox uses as the "source layer"

Related

Errors converting Geometry to Geography

I am getting an error trying to convert data from a Geometry field to a geography field in a separate table.
INSERT INTO PIGeoData
([ID], [geo_name], [geo_wkt] ,[port_geography_binary] )
SELECT [id], [name] ,[wkt], GEOGRAPHY::STGeomFromWKB(em_ports.geom.STAsBinary(),4326)
FROM [guest].[em_ports]
where ID < 4548 and ID not in (select ID from PIGeoData)
The error I get is this
Msg 6522, Level 16, State 1, Line 1
A .NET Framework error occurred during execution of user-defined routine or aggregate "geography":
Microsoft.SqlServer.Types.GLArgumentException: 24205: The specified input does not represent a valid geography instance because it exceeds a single hemisphere. Each geography instance must fit inside a single hemisphere. A common reason for this error is that a polygon has the wrong ring orientation. To create a larger than hemisphere geography instance, upgrade the version of SQL Server and change the database compatibility level to at least 110.
Microsoft.SqlServer.Types.GLArgumentException:
at Microsoft.SqlServer.Types.GLNativeMethods.ThrowExceptionForHr(GL_HResult errorCode)
at Microsoft.SqlServer.Types.GLNativeMethods.GeodeticIsValid(GeoData& g, Double eccentricity, Boolean forceKatmai)
at Microsoft.SqlServer.Types.SqlGeography.IsValidExpensive(Boolean forceKatmai)
at Microsoft.SqlServer.Types.SqlGeography..ctor(GeoData g, Int32 srid)
at Microsoft.SqlServer.Types.SqlGeography.GeographyFromBinary(OpenGisType type, SqlBytes wkbGeography, Int32 srid)
I get the same message if I try to convert from WKT using
,GEOGRAPHY::STGeomFromText(wkt,4326)
Both these formats come from the MS documentation here
But if I copy the polygon data from the wkt and paste it into a query like this
declare #sGeo geography
declare #sWKT varchar(max)
select #sWKT = wkt from guest.em_ports where wkt like '%POLYGON ((73.50667 4.181667,73.50667 4.21,73.48 4.21,73.48 4.1783333,73.50667 4.181667,73.50667 4.181667))%'
set #sGeo = geography::STPolyFromText (#sWKT, 4326 )
Update PIGeoData
Set PortBoundaries = #sGeo
Where wkt like '%POLYGON ((73.50667 4.181667,73.50667 4.21,73.48 4.21,73.48 4.1783333,73.50667 4.181667,73.50667 4.181667))%'
that works.
So I moved all the non-geo data to the new table and started going through record by record to see which WKT was failing:
I used this query
Update PIGeoData
Set port_geography_binary = GEOGRAPHY::STGeomFromText(geo_wkt,4326)
where port_geography_binary is null and ID = <xyz>
where xyz was individual record ids
These WKT values succeeded
POLYGON ((-135.31197 59.451653,-135.32457 59.45799,-135.32996 59.454834,-135.36717 59.455154,-135.36452 59.449005,-135.36488 59.43996,-135.36697 59.43817,-135.33139 59.438065,-135.31197 59.451653,-135.31197 59.451653))
POLYGON ((-4.524549 48.365623,-4.518855 48.361416,-4.4854136 48.367413,-4.436236 48.381382,-4.420772 48.39644,-4.431077 48.398525,-4.4376454 48.393867,-4.438626 48.38611,-4.4559207 48.390007,-4.470995 48.387226,-4.4933248 48.384468,-4.499816 48.38401,-4.512855 48.3754,-4.524549 48.365623,-4.524549 48.365623))
These WKT values failed
POLYGON ((-8.788489 37.773106,-8.989748 37.785244,-9.11148 37.93065,-9.01401 38.13953,-8.993956 38.30128,-9.266149 38.264282,-9.382366 38.33244,-9.435615 38.54836,-9.656681 38.602306,-9.683701 38.883057,-9.1720295 39.00796,-8.444215 39.550682,-8.213643 39.355015,-8.537656 38.037514,-8.712016 37.782127,-8.788489 37.773106))
POLYGON ((-119.71587 34.396824,-119.69837 34.410378,-119.67453 34.41837,-119.62994 34.420082,-119.63012 34.380177,-119.62986 34.3551,-119.71534 34.355022,-119.71587 34.396824,-119.71587 34.396824))
There is nothing obvious to me in the data. Can anyone help with why these records and data are failing?
TIA
The relevant part of the error message is "A common reason for this error is that a polygon has the wrong ring orientation."
The polygons that have failed are in clockwise order.
To convert them to counter-clockwise order, you can use something like this:
DECLARE #t VARCHAR(MAX)='POLYGON ((-119.71587 34.396824,-119.69837 34.410378,-119.67453 34.41837,-119.62994 34.420082,-119.63012 34.380177,-119.62986 34.3551,-119.71534 34.355022,-119.71587 34.396824,-119.71587 34.396824))'
DECLARE #x XML=REPLACE(REPLACE(REPLACE(#t,'POLYGON ((','<root><p>'),'))','</p></root>'),',','</p><p>')
DECLARE #r VARCHAR(MAX)='POLYGON (('+STUFF((
SELECT ','+q.Point
FROM (
SELECT n.value('.','varchar(50)') AS Point, ROW_NUMBER() OVER (ORDER BY t.n) AS Position
FROM #x.nodes('/root/p') t(n)
) q ORDER BY q.Position DESC
FOR XML PATH(''), TYPE
).value('.','VARCHAR(MAX)'),1,1,'')+'))'
DECLARE #g GEOGRAPHY=GEOGRAPHY::STGeomFromText(#r,4326)
SELECT #g, #g.ToString()
Later edit:
There is a convention that says that a polygon should always be represented in counter-clockwise order. Imagine that you have a polygon in the shape of the equator; without this convention it would not be clear if the polygon represents the northern hemisphere or the southern hemisphere. See Spatial Data Types Overview in the Microsoft SQL Server documentation for details.
Additionally, there is a limitation in SQL Server when the compatibility level is 100 or below that each geography instance must fit inside a single hemisphere. If you are using SQL Server 2012 or later and you choose to use at least compatibility level 110, you can avoid the error message, but the polygon would represent the entire area that is outside of what you would normally think that the polygon represents.
If you use compatibility level is 100 or below, you could use a TRY/CATCH to detect the error and if it happens you should try reversing the polygon.
If you use compatibility level 110 or later, you can try to use STArea() to check if the polygon has a surface which is much bigger or much smaller than one hemisphere. If the area approaches 510100000000000 square meters (which approximately the area of the entire earth) then you should reverse the polygon.

Download OSM network (with OSMNx) filtering based on the union of tag values

I would like to download a network from OSM with union of 2 filters based on highway and cycleway tags.
network=ox.core.graph_from_place ( place_name, custom_filter='["highway"~"cycleway"]["bicycle"!~"no”]’
This command makes the intersection of the 2 filters. So it gets all edges with highway = cycleway, and with cycleway tag different from value “no”.
However if I would like to make the union with the filter ["cycleway”~"lane”] I don’t know the boolean ‘OR’ operatore for OSM.
I tried the following but it doesn’t work:
network=ox.core.graph_from_place ( place_name, custom_filter='["highway"~"cycleway"]["bicycle"!~"no”] or ["cycleway”~"lane”]’
network=ox.core.graph_from_place ( place_name, custom_filter='["highway"~"cycleway"]["bicycle"!~"no”] | ["cycleway”~"lane”]’
Is there an easy way to write the custom filter making the union of tag values? or should I download more than I need and then remove out edges as suggested in #151 ?
This is how I extract networks based on union of infrastructure filters.
# get graphs of different infrastructure types, then combine
place = 'Berkeley, California, USA'
G1 = ox.graph_from_place(place, custom_filter='["highway"~"cycleway"]')
G2 = ox.graph_from_place(place, custom_filter='["cycleway”~"lane”]')
G = nx.compose(G1, G2)
osmnx uses Overpass API for downloading OSM data. Overpass API has no or operator in the way you are trying to use it. There is a simple union statement which just means "download X, then download Y" (see an example at overpass-turbo for "highway=cycleway or cycleway=lane"). I guess you have to do the same in osmnx.

Problems with spatial join

I am new to sql, and attempting to use it to speed up spatial analysis on a set of ~1.2 million trips from a csv that contains the lat and lon for pickup and dropoff points.
What I am trying to do in plain English is:
select all trips that start in the area of interest (loaded into my database as a shapefile) into one table
select all trips that end in the area of interest into another
-perform a spatial join between these points and a shapefile of census tracks (which contains neighborhood names)
count by neighborhood name to list the most frequent origins/destination of trips to/ from the area of interest.
The code I am working with is below (If its helpful, NTA or neighborhood tabulation area, is the neighborhood name which I want to display in my table at the end of this operation) :
--Select all trips that end in project area
SELECT *
INTO end_PA
FROM trips, projarea
WHERE ST_Intersects(trips.dropoff, projarea.geom);
--for trips that end in project area - index by NTA of pick up point
ALTER TABLE end_PA ADD COLUMN GID SERIAL;
CREATE TABLE points_ct_end AS
SELECT nyct2010.ntacode as ct_nta, end_PA.gid as point_id
from nyct2010, end_PA WHERE ST_Intersects(nyct2010.geom , end_PA.pickup);
--Count most common NTA
--return count for each NAT as a csv
copy(
select count(ct_nta) from points_ct_end
group by ct_nta
order by count desc)
to 'C://TaxiData//Analysis//Trips_Arriving_LM.csv' DELIMITER ',' CSV HEADER;
However, I am having problems from the very start - ST_Intersects does not return any points within the area of interest!
Troubleshooting solutions I have tried thus far:
My first thought is that the points weren't in the correct SRID. When I created the 'dropoff' point I set the SRID to 4326. I tried both using ST_SetSRID to change the projection of both data sets to 4326, and manually re projecting the shapefiles to 4326 in ArcMap - but neither worked.
I plotted a small sample of the points from the 'trips' data set in Arc Map to ensure they were correctly projected and overlapping with the ProjArea shapefile. They are.
I imported the multipoint shapefile this created into my geo database to test if that worked with ST_Intersects. Nope.
I tried using ST_Within. This threw the error message:
ERROR: function st_within(character varying, geometry) does not exist
....
HINT: No function matches the given name and argument types. You
might need to add explicit type casts.
I am using Big SQL and postgres
Thanks!!
My first thought is that the points weren't in the correct SRID. When I created the 'dropoff' point I set the SRID to 4326. I tried both using ST_SetSRID to change the projection of both data sets to 4326, and manually re projecting the shapefiles to 4326 in ArcMap - but neither worked.
ST_SetSRID doesn't change the projection (reproject). It just changes the internal representation. This can totally screw everything up if the previous SRID matched the input data. You likely wanted ST_Transform().
There isn't enough information here to trouble shoot this problem. However, we can answer this...
ERROR: function st_within(character varying, geometry) does not exist
This simply means the first argument is not a geometery. Of course, we can't do anything with that at all because we don't have your query that you tried with ST_Within().
Your syntax for ST_Intersects() looks to be right. But, there simply isn't enough information provided to help. Show some schema and sample data.

How to create a new SRID in Postgresql/Postgis?

I have some extravagant local spatial reference system and I have a lot of data stored in some old legacy system. Now I want to import this data to my Postgresql/Postgis database. On the client side I'm using JavaScript OpenLayers 3 library (if it matters), on the server side I'm used to storing geometry data with srid 3857, so my tables with layer data have such constraints:
CONSTRAINT enforce_dims_geom_layer_1_ CHECK (st_ndims(geom) = 2),
CONSTRAINT enforce_srid_geom_layer_1_ CHECK (st_srid(geom) = 3857)
So, if I have this legacy data, with some coordinates in a local reference system, how can I approach this problem to get a formula like:
+proj=longlat +ellps=bessel +towgs84=595.48,121.69,515.35,4.115,-2.9383,0.853,-3.408 +no_defs
Have a look at the public.spatial_ref_sys table. There the SRIDs are defined and you can insert your new SRID. The column proj4text includes the formulas.
The website https://epsg.io will generate an insert string for your SRID if you can find the page for your desired EPSG code. Once you find the page for the code you want, scroll down to "Export", below that on the left set it to "PostGIS". You can then "Copy TEXT", and paste that into your terminal or whatever you use to interact with your database.

How to get all points along a way from (osm)PostGIS?

I have import OpenstreetMap data into Postgres with gis extension with tool
osm2pgsql (-s option)
of course, I have the following tables
planet_osm_point
planet_osm_ways
....
Within planet_osm_ways I have a column called way, type geometry(LineString, 4326), content like following
"0102000020E6100000070000005E70BCF1A49F2540D3D226987B134840896764EB749F25403B5DCC858013484040D1860D609F2540C426327381134840CE50DCF1269F2540EF552B137E1348405AAB2CC02D9E2540F978324976134840D66F26A60B9D2540CE8877256E1348403CA81F2FFF9C2540BC1D86FB6D134840"
What is that ? How could I get all points along this way ?
Thanks a lot
That's hex-encoded extended well-known binary (EWKB) of a LINESTRING.
There are several methods to get the points along the way. To get individual coordinates as points, use ST_DumpPoints. Or to simply output the geometry in other human-readable formats (WKT, EWKT, GeoJSON, GML, etc.), see the relevant manual section.