OSM API Overpass - openstreetmap

I am trying to pull all glaciers as entered in OSM in a given country but am noticing that I am only pulling a fraction of what is available. For example, when I run this following code:
import overpass from shapely.geometry
import shape, Polygon
api = overpass.API()
api = overpass.API(endpoint="https://overpass.myserver/interpreter")
api = overpass.API(timeout=600)
query = 'area["ISO3166-1"="IS][admin_level=2];(way["natural"="glacier"](area););'
result = api.get(query, verbosity='geom')
import geopandas
results = geopandas.GeoDataFrame.from_features(result['features'])
The result has 132 features and appears as so:
Iceland Glaciers
I know this is missing one large glacier (Vatnajökull) which does appear in OSM under osm id 406429.
Any thoughts as to why this is not appearing as a result from my query?

OSM Wiki tag documentation is a helpful starting point when writing Overpass queries. Here is the documentation for natural=glacier. The tag/value is applied to nodes and closed ways based on the documentation and also appears to apply to relations based on community preference (even though this is discouraged in the documentation).
To query for nodes, ways, and relations, you can use the abbreviation nwr instead of the union (node[natural=glacier];way[natural=glacier];relation[natural=glacier];);. As a side note, you can drop admin_level=2 since ISO3166-1 codes are unique identifiers.
Here is the Python request:
query = 'area["ISO3166-1"="IS"];nwr[natural=glacier](area);out geom;'
response = api.get(query)

Related

How to use database index correctly when searching on geo field?

Suppose i have the following model:
from django.db import models
from django.contrib.gis.db import models as gis_models
class Place(models.Model):
location = gis_models.PointField(geography=True, srid=4326)
Later i am performing the search on those Places; my query is "fetch all places no further N meters from me":
from django.contrib.gis.db.models.functions import Distance
from django.contrib.gis.geos import Point
location = Point(1.0, 2.0)
distance = 20.0
queryset = queryset.annotate(distance=Distance("location", location)).filter(
distance__lte=distance
)
Is there any way using PostGIS to optimize those queries? For example, using indexes or something related.
Your code does not look like SQL, and you did not tag a programming language, framework or ORM, so I'll give you an SQL answer.
To search for all geometrys that are less than 30 from a certain point, you would use
... WHERE ST_DWithin(geom, 'POINT(1 2)', 30)
The index to support that is
CREATE INDEX ON tab USING gist (geom);
The Postgis funciton ST_DWithin should use relevant bounding box indexes if existing.
To create such a spatial index, you can for example do the following:
CREATE INDEX unique_index_name ON table_name USING gist (geometry_column);
As #fresser and #laurenz-albe helpfully suggested, the correct index is GIST index:
from django.contrib.postgres.indexes import GistIndex
class Place:
...
class Meta:
indexes = [
GistIndex(fields=["location"]),
]
Your problem is that your data is in degrees (EPSG:4326) and you ask for everything with in 20 of a point. That 30 is in degrees and probably includes a large chunk (if not all) of your data so PostGIS is too smart to use your index.

Download OSM network (with OSMNx) filtering based on the union of tag values

I would like to download a network from OSM with union of 2 filters based on highway and cycleway tags.
network=ox.core.graph_from_place ( place_name, custom_filter='["highway"~"cycleway"]["bicycle"!~"no”]’
This command makes the intersection of the 2 filters. So it gets all edges with highway = cycleway, and with cycleway tag different from value “no”.
However if I would like to make the union with the filter ["cycleway”~"lane”] I don’t know the boolean ‘OR’ operatore for OSM.
I tried the following but it doesn’t work:
network=ox.core.graph_from_place ( place_name, custom_filter='["highway"~"cycleway"]["bicycle"!~"no”] or ["cycleway”~"lane”]’
network=ox.core.graph_from_place ( place_name, custom_filter='["highway"~"cycleway"]["bicycle"!~"no”] | ["cycleway”~"lane”]’
Is there an easy way to write the custom filter making the union of tag values? or should I download more than I need and then remove out edges as suggested in #151 ?
This is how I extract networks based on union of infrastructure filters.
# get graphs of different infrastructure types, then combine
place = 'Berkeley, California, USA'
G1 = ox.graph_from_place(place, custom_filter='["highway"~"cycleway"]')
G2 = ox.graph_from_place(place, custom_filter='["cycleway”~"lane”]')
G = nx.compose(G1, G2)
osmnx uses Overpass API for downloading OSM data. Overpass API has no or operator in the way you are trying to use it. There is a simple union statement which just means "download X, then download Y" (see an example at overpass-turbo for "highway=cycleway or cycleway=lane"). I guess you have to do the same in osmnx.

Using VSTS.Feed() in Power BI to access odata

I am trying to use the VSTS.Feed() function in Power BI to read WorkItemSnapshot data. There are multiple problems. If I build the entire URL into a single string and call VSTS.Feed () with that, I get the correct information in Power BI desktop, but it will not refresh in Power BI online. I have been told to use the (undocumented) Query parameter, as shown below, but it is clear that this parameter is ignored. I can see that the select parameter is ignored on smaller projects, because all columns are returned. I can see that the filter parameter is ignored because the query fails on larger projects.
Does anyone have a working example of using the Query parameter with VSTS.Feed()?
let
BaseURL = "https://server.analytics.visualstudio.com/DefaultCollection/project/_odata/WorkItemSnapshot",
Select = "DateSK,WorkItemId,State,WorkItemType",
Filter = "WorkItemType eq Bug and State ne Closed and State ne Removed and DateSK ge 20180517 and DateSK le 20180615",
Source = VSTS.Feed(BaseURL, [Query=[select=#"Select",filter=#"Filter"]])
in
Source
Update:
With the query above, the message I get is shown below. As I said earlier, it is clearly not using the Filter parameter, and I'm assuming it is not using the Select parameter, either. I can't query everything because there is too much data, and I can't use a filter because I can't figure out a way to get the Options parameter to work. With VSTS.AccountContents, the options parameter works well, but those API endpoints don't use $ in parameter names.
Error: Query result contains 36,788,023 rows and it exceeds maximum allowed size of 300,000. Please reduce the number of records by applying additional filters
Details:
DataSourceKind=Visual Studio Team Services
ActivityId=881f7988-9863-4e03-8375-0489028f28f3
Url=https://server.analytics.visualstudio.com/DefaultCollection/Project/_odata/WorkItemSnapshot
error=Record
The query that started this whole line of questioning is simply one with a variable for a start date.
let
startDate = DateTimeZone.ToText (Date.AddDays(DateTimeZone.UtcNow(), -45), "yyyyMMdd"),
URL = "https://server.analytics.visualstudio.com/DefaultCollection/project/_odata/WorkItemSnapshot?$select=DateSK,WorkItemId,State,WorkItemType&$filter=WorkItemType eq 'Bug' and State ne 'Closed' and State ne 'Removed' and DateSK gt " & startDate,
Source = VSTS.Feed(URL)
in
Source
While this query mostly works in Power BI desktop (the select clause is ignored), the message I get when the data source is refreshed online is:
You can't schedule refresh for this dataset because one or more sources currently don't support refresh.
Discover Data Sources
Query contains unknown or unsupported data sources.
The documentation for VSTS.Feed() contradicts itself, saying both
The VSTS.Feed function has the same arguments, options and return value format as OData.Feed.
and
'VSTS.Feed' provides a subset of the Arguments and Options available through 'OData.Feed'.
To to summarize, I know that I can't combine data sources in Power BI. Does VSTS.Feed() support the options parameter? If so, how do I pass a Filter and Select clause to it?
To get WorkItemSnapshot by vsts.feed, please refer below query:
let
Source = OData.Feed("https://account.analytics.visualstudio.com/project/_odata/v1.0-preview", null, [Implementation="2.0"]),
WorkItemSnapshot_table = Source{[Name="WorkItemSnapshot",Signature="table"]}[Data]
in
WorkItemSnapshot_table
Note: the URL format should be https://account.analytics.visualstudio.com/project/_odata/v1.0-preview, or https://account.analytics.visualstudio.com/_odata/v1.0-preview.
And you can refer below documents:
Connect to VSTS using the Power BI OData feed
Connect using Power Query and Visual Studio Team Services (VSTS) functions

Openstreetmap: filter out data that have been edited after some timestamp

I want to get OSM data after some timestamp - in other words the last records after a certain timestamp. I have downloaded the osm file of the area. I went through the osmosis documentation but could not find a way to filter it by time. The result should be same as when we use the timestamp-argument. Well how to do that:
I could use the overpass but the area is large and overpass timed out many times
I could use the osmconvert-tool (cf the manual: m.m.i24.cc/osmconvert.c )
Some of the following statements might be useful for the task:
"--timestamp=<date_time> add a timestamp to the data\n"
"--timestamp=NOW-<seconds> add a timestamp in seconds before now\n"
What I have tried is the following;
./osmfilter austria-latest.osm --keep="$key=$school" |
./osmconvert - --all-to-nodes --csv="#id #lat #lon #timestamp $key name" --csv-headline |
but this fails. How to get the data out of the osm-pbf-file. Should I use the statements drop! or should i name a certain time from timestamp to timestamp!?
Since version 0.7.50 Overpass API provides a way to query for data, which changed since a given timestamp or in a given timeframe. It is even possible to restrict the change analysis to certain tags (or filter criteria). Please check the Overpass API Wiki page for more details on "diff" and "adiff" keywords.
Working with Overpass API ina way is much more convenient than trying to process a full planet history, which takes at least 35GB to download and requires more complex post-processing.
You want to process OSM history planet (extracts): https://wiki.openstreetmap.org/wiki/Planet.osm/full

MapQuest Search API not yielding proper result

I have some data in MapQuest Data Manager v2 (DMv2) and I am trying to search this hosted data set using their search API(Web Service) but I am not getting desired results. Following are the snapshot of my data set and queries I am using :
These are the fields :
"storeid","brandname","onlineorderingenabled","street","zipcode","state","geocodereturncode","city","country","mqap_geography","mqap_quality","landmark","county","mqap_id","storename","longitude","latitude"
and this is sample row :
"1","Chili's","true","12815 Preston Rd","75230-1302","TX","","Dallas","US","POINT (-96.80363 32.92329)","P1AAA","","Dallas","ca6b6bae-945f-45fc-a8d1-3d512796150d","Preston/LBJ-Chili's","",""
Sample search queries :
http://www.mapquestapi.com/search/v2/search?key=[My_Key here]&shapePoints=-80,26&outFormat=json&hostedData=hostedData=mqap.121144_BrinkerStores|storename ILIKE ?|Preston/LBJ-Chili's|storeid,storename,city
http://www.mapquestapi.com/search/v2/radius?key=[My_Key]&origin=Dallas&inFormat=json&json={hostedData:[{tableName:mqap.121144_BrinkerStores,extraCriteria:City ILIKE ?,parameters:[Dallas],columnNames:[storeid,storename,city,state]}]}
I am not getting storedid,storenam,city,state. Please help me.
In the search/v2/search request, make sure your shapePoints are latititude,longitude and not the other way around. And the apostrophe in the storename needs to be double apostrophe'd for the postgres backend.
http://www.mapquestapi.com/search/v2/search?key=KEY&shapePoints=32.778149,-96.795403&hostedData=mqap.121144_BrinkerStores|storename=?|Knox-Chili%27%27s
In the search/v2/radius request, include the origin in the json and rename hostedData to hostedDataList.
www.mapquestapi.com/search/v2/radius?key=KEY&inFormat=json&json={"origin":"Dallas,TX","hostedDataList":[{"tableName":"mqap.121144_BrinkerStores","extraCriteria":"City ILIKE ?","parameters":["Dallas"],columnNames:["storeid","storename","city","state"]}]}