I am trying to figure out if it is possible with the OverpassAPI to download all boundary data for a country.
Small example: Luxembourg
I would like to get all relations that are tagged as (key-value) boundary=administrative. I would like to receive all ways to these relations, and all nodes in these ways.
I assume that could be quite a lot of data (?) so I wonder if one query is the way to go, especially if I would try to do it with Germany or France (with about 20 items on admin_level=4, and hundreds on admin_level=6 or 8)
This is further complicated because the bounding box around the country takes in all other countries sharing a border with the one "in-box".
So this seems not to be the way - what technique would I use to get all nodes on all ways that form administrative boundaries of a country?
What I am currently trying is this
<osm-script>
<query type="relation">
<bbox-query s="49" w="2" n="52" e="7"/>
<has-kv k="boundary" v="administrative"/>
<has-kv k="admin_level" v="2"/>
</query>
<print mode="meta"/>
</osm-script>
This is a bounding box for Belgium (say mid-size), and restricted to admin_level=2. Also, this does not yet deliver node information.
I know how to change that - but this will deliver all neighbouring countries too. What is a better way of doing this?
Thanks!
Ralf
I would recommend to look at OSM Boundaries map instead, a dedicated application for worldwide boundary on just about any level you can think of: https://wambachers-osm.website/boundaries
In any case, there's quite some load on the main overpass api instance and it is likely that you will hit a HTTP 429 error when running the following query to fetch all ways and nodes for Germany:
(rel(51477);>;);out meta;
or:
(rel[boundary=administrative][admin_level=2][name="Deutschland"];>;);out meta;
To get all boundary=administrative relations with all of their ways and nodes in Luxemburg, the following query should do (careful: returns 47 MB of data, including Germany, Rhineland-Palatine and Saarland).
rel(2171347);
map_to_area;
(rel[boundary=administrative](area);>;);
out meta;
Maybe try this alternative as well.
Related
Using Acumatica ERP 2018R (V18.112.0019) which is the latest version as of this question.
I've been trying to find a way to download contact address records in batch -- not one at a time, and it seems that everything I try, I get an error from the optimizer about a BQL delegate.
I've tried using the Default/17.200.001 endpoint to go after the Contact entity and use the $expand=Address parameter.
Often suggested as a workaround is to create a Generic Inquiry, so I've also tried extending the endpoint and get the results from a GI, but no matter how basic I make that GI, I get the same BQL delegate error. For the DAC, I've tried using PX.Objects.CR.Address, PX.Objects.CR.ContactExtAddress, PX.Objects.AR.ARAddress, and went so far as to return just the AddressID field. No other tables, relations, parameters, conditions, etc. Just a very basic GI to try and get to an address. Even if the GI worked, there's issues with being able to page the results (I get a method not allowed error when trying to do a Put to a GI, but that could just be me doing it wrong).
Looking that differences between the old endpoint Contract versions, it looks like V1 was more forgiving with sub-optimal queries than V3? Maybe the REST API needs to have a URL parameter or a way to specify to allow a non-optimized query to run? Or if it has that feature, I can't seem to locate it.
Any insight or examples that anyone could add would be greatly appreciated. I can't imagine having to download contacts and/or contact address records one at a time - that would be a ridiculous number of round trips/requests and would be much slower and more of a resource hog than a single non-optimized query.
I think I was able to solve my own problem by expanding the Contacts in the following manner:
Customer?$expand=ShippingContact/Address,MainContact/Address,Contacts,Contacts/Contact/Address
You should be able to do PUT to a Generic Inquiry entity, and in that way you should not get a delegate-related optimization error.
I am wondering if it is possible to adhere to REST principles when creating what will essentially amount to a BI tool.
In my scenario I have high data volume with 100,000's of IDs (frankly more than this but for the sake of this example let's go with that.). These are presented in a traditional table that allows for necessary features when accessing large data sets such as pagination. The user also has the ability to filter by one, or many of these ID's to drill down the data set as they see fit.
It is theoretically possible that the user would want to filter on 100 of the ID's, thus rendering a GET URI incredibly long. Which as I understand it would kind of break the resource identification principle of a REST API. Not to mention could potentially bump into the character limit in a GET request for certain browsers since the ID's may be quite long. Normally I would just use a POST since I can add all of the applied filters in the body and generate a where clause on the server.
Since a POST in a REST API is supposed to
Create a new entry in the collection.
By definition it would appear, at least to me that generating a complex query for something like this would mean that a REST API is not possible. Or does this perhaps mean that I am approaching the solution wrong (totally plausible).
It would seem that in my scenario using a GET simply isn't possible due to the potential length of the parameters. Thus I am forced to use a POST. Though using a POST as I am seems to violate REST style, which isn't the end of the world. I mostly just wanted to double check that I am not missing something and there is not a solution using a GET.
To follow the resources principle, make a search like resource. POST your ids in a body wto this endpoint and it will prepare a list of results for you and redirect you to searchresults/{id}.
See for example: https://gooroo.io/GoorooTHINK/Article/16583/HTTP-Patterns---Bouncer/25829#.W3aBsugzaUk
OpenStreetMap supports different name translations for places. For example "Moscow" in English and "Москва" in Russian for the same node. But there are also French, German, Bahasa translations and so on. Is there a way to get all these translations in a single request? I know I can use "accept_language" property on a "reverse" request, but in will only return a single translation, and I'd like to get all available translations in one request. Fast googling didn't return any results. Thank you.
Use the osm_type and osm_id fields to retrieve the original OSM element.
Example: The Nominatim query for Moscow will return osm_type: relation and osm_id: 2555133 for the first result. Based on this information we can look at the original OSM element or perform an OSM API query for this element. This will include all name:<lang> tags.
Note: Make sure to look both at the Nominatim usage policy and OSM API usage policy before running automated queries!
It seems to me that REST has clean and clear semantics for basic CRUD and for listing resources, but I've seen no discussion of how to handle large lists of resources. It doesn't scale to dump an entire database table over the network in a resource-oriented architecture (imagine a customer table with a million customers!), especially if you only need a few items. So it seems that some semantics should exist to filter, map and reduce a list of resources on the server-side.
So, do you know any tried and true ways to do the following kinds of requests in REST:
1) Retrieve just the count of the resources?
I could imagine doing something like GET /api/customer?result=count
Is that how it's usually done?
I could also imagine modifying the URL (/api/count/customer or /api/customer/count, for example), but that seems to either break the continuity of the resource paths or inflict an ugly hack on the expected ID field.
2) Filter the results on the server-side?
I could imagine using query parameters for this, in a context-specific way (such as GET /api/customer?country=US&state=TX).
It seems tricky to do it in a flexible way, especially if you need to join other tables (for example, get customers who purchased in the last 6 months).
I could imagine using the HTTP OPTIONS method to return a JSON string specifying possible filters and their possible values.
Has anyone tried this sort of thing?
If so, how complex did you get (for example, retrieving the items purchased year-to-date by female customers between 18 and 45 years old in Massachussetts, etc.)?
3) Mapping to just get a limited set of fields or to add fields from joined tables?
4) More complicated reductions than count (such as average, sum, etc.)?
EDIT: To clarify, I'm interested in how the request is formulated rather than how to implement it on the server-side.
I think the answer to your question is OData! OData is a generic protocol for querying and interacting with information. OData is based on REST but extends the semantics to include programatic elemements similar to SQL.
OData is not always URL-based only as it use JSON payloads for some scenarios. But it is a standard (OASIS) so it well structured and supported by many APIs.
A few general links:
https://en.wikipedia.org/wiki/Open_Data_Protocol
http://www.odata.org/
The most common ways of handling large data sets in GET requests are (afaict) :
1) Pagination. The request would be something like GET /api/customer?country=US&state=TX&firstResult=0&maxResults=50. This way the client has the freedom to choose the size of the data chunk he needs (this is often useful for UI-based clients).
2) Exposing a size service, so that the client gets to know how large the data set is before actually requesting it. The service would be something like
GET /api/customer/size?country=US&state=TX
Obviously the two can (and imho should) be used together, so that when/if a client (be it mobile or web or whatever) waints to fill its UI with content, he can choose what's the best data chunk size based also on the size of whole data set (e.g. to avoid creating 100 pages for the user to navigate).
I have already gone through this
How best to design a REST API with multiple filters?
This does help when you have say 3 or 4 filtering criteria and you can accomodate that in the query String.
However let's take this example
You want to get call details about 20 telephone numbers, between a certain startdate and enddate.
Now I do agree ideally one should be advised to make individual queries for each number and then on the client side collate all data.
However for certain Live systems that would mean 20 rounds of queries on the switches or cdr databases. That is 20 request-response cycles plus the client having to collate and order them again based on time. While in the database level it would have been a simple single query that can return an ordered data and transformed back into a REST xml response that the client can embed on their system.
If we are to use GET the query string will get really confusing and has a limit as well.
Any suggestions to get around this issue.
Of course we can send a POST request with an xml having all numbers in it but that is against REST Get principles.
In case of GET use OData queries. For example when your start and end dates represented as numbers (unix time) URI could look like:
GET http://operatorcalls.com/Calls/Details?$filter=Date le 1342699200 and Date gt 1342526400
What you seem to be missing is an important concept of REST, caching. This can be done, as an example, in the browser, for a single client. Or it can be done as a shared cache between all the clients and the live production system (whatever it may be). Thus reducing queries against a live production system, or in your example, actual switches.
You should really take some time to read Fieldings thesis, and understand that REST is an architectural style.
I found a solution here Handling multiple parameters in a URI (RESTfully) in Java
but not quite happy with it.
So in effect we will end up using /cdr?numbers=number1,number2,number3 ...
However not too pleased with it as there is a limit to Query String in the url and also doesn't really seem to be an elegant solution. Anyone found any solution to this in their own implementation?
Basically not using POST for this kind of Fetch requests and also not using cumbresome and lengthy Query Strings.
We are using Jersey but also open to using CXF or Spring REST