Since some of the open data has to do with property and each city has a unique identifier for a property, is there an easy way to map those unique identifiers in Socrata?
Visually, given a list of property unique identifiers in the open data, I'd like to see the properties on a map in one step.
More examples: If you go to http://api.us.socrata.com/api/catalog/v1?column_names=bbl you will see hundreds of references because this is what New York City calls it's unique identifier for a property.
What would be helpful is if we could easily see these properties on a map. We know that datasets should have latitude and longitude, but they usually do not. Plus for a property it is better to see the outline of the property and not just a point on a map.
In order to make it easier to do this, there is a tool at https://map.tidalforce.org/ it allows you to paste a comma delimited NYC BBLs (borough, block and lot) list. For example, https://map.tidalforce.org/?1pNFsY5rnHcsFEpLW2CJ this maps the socrata dataset at https://data.cityofnewyork.us/City-Government/Open-Article-7-Petitions/aht6-vxai
Is there a way to integrate with Socrata to make it easier to map properties for each area, city, state, etc. Or does this functionality already exist.
Related
We have a set of tables shown below we use for our other tables to reference for location data. Some examples are:
Find all companies within X miles of X City
Create a company profile's location as X City
We solve the problem of multiple cities with similar names by matching with State as well, but now we ran into a different set of problems. We use Google's Place Autocomplete for both Geocoding and matching up a users query with our Cities. This works fairly well until Google's format deviates from ours.
Example:
St. Louis !== Saint Louis and
Ameca del Torro !== Ameca Torro
Is there a way to fuzzy match cities in our queries?
Our query to match cities now looks like:
SELECT c.id
FROM city c
INNER JOIN state s
ON s.id = c.state_id
WHERE c.name = 'Los Angeles' AND s.short_name = 'CA'
I've also considered the denormalizing city and simply storing coordinates to still accomplish the radius search. We have around 2 million rows in our company table now so a radius search would be performed on that rather than by city table with a JOIN on company. This would also mean we wouldn't be able to create custom regions (simply anyway) for cities, and add other attributes to cities in the future.
I found this answer but it is basically affirming our way of normalizing input is a good method, but not how we match to our local Table (unless Google offers a City Name export I don't know about).
The short answer is that you can use Postgres's full text search functionality, with a customized search configuration.
Since your dealing with place names, your probably want to avoid stemming, so you can use the simple configuration as a starting point. You can also add stop-words that make sense for place names (with the examples above, you can probably consider "St.", "Saint", and "del" as stop-words).
A pretty basic outline of setting up your customized is below:
Create a stopwords file and put it in your $SHAREDIR/tsearch_data Postgres directory. See https://www.postgresql.org/docs/9.1/static/textsearch-dictionaries.html#TEXTSEARCH-STOPWORDS.
Create a dictionary that uses this stopwords list (you can probably use the pg_catalog.simple as your template dictionary). See https://www.postgresql.org/docs/9.1/static/textsearch-dictionaries.html#TEXTSEARCH-SIMPLE-DICTIONARY.
Create a search configuration for place names. See https://www.postgresql.org/docs/9.1/static/textsearch-configuration.html.
Alter your search configuration to use the dictionary you created in Step 2 (cf. the link above).
Another consideration is how to consider internationalization. It seems that the issue for your second example (Ameca del Torro vs. Ameca Torro) might be a Spanish vs. English representation of the name. If that's the case, you could also consider storing both a "localized" and "universal" (e.g. English) version of the city name.
At the end, your query (using full-text search) might look like this (where the 'places' is the name of your search configuration):
SELECT cities."id"
FROM cities
INNER JOIN "state" ON "state".id = cities.state_id
WHERE
"state".short_name = 'CA'
AND TO_TSVECTOR('places', cities.name) ## TO_TSQUERY('places', 'Los & Angeles')
so my company wants me to learn ABAP for SAP and I have started on the road to learn this. My background is mainly VB.net and sqlserver with T-SQL but also have experience in c#.
With ABAP though I am needing some clarification or confirmation on the understanding of Data Types and Domain. If anyone can help.
My understanding currently is we have a table, in the table we have fields and the fields have data types and lengths if needed. Example: We have a table Customer, I could have a customerNumber field with the data type of char(10). To me this mean in the table customer we have a field called CustomerNumber that will have 10 characters.
However with ABAP we have Domains, Data elements then the field, does this mean we have a field named whatever we want. As the field could mean anything we assign a data element which has the descriptions of the sort of data stored within the field. However to store the format and data type we need to assign the Domain to the Data element.
For example I call a field ZCUSNO, currently this means nothing however if I assign the ZCTNMR (with description of customer number) Data element this tells us that the field ZCUSNO is ZCTNMR so ZCUSNO is a customer number field.
Now within the data elements we would have a domain and for our example ZCTNMR data element (the customer number) we could assign ZCTDOM as the domain which would be what I recognise as the data types so Char 20, Char 100 or integer field etc.
Is my understanding correct on this? and could someone give me a clear indication of what the difference between a Domain > Data Element is against what I would know as data types in sqlserver.
Thanks
I don't know if it's 100% correct, but that's is the way I use, like you say.
You can reuse the Domain, If you don't plan to reuse you can use direct the Data Element and refer this to a built-in-type.
Data Element is to define semantic of the field, like label, translation, etc
Domain is to define techinical info of the field, like Type, conversions, predefined Values,e tc
E.G.
Domain:
DOM_VALUE you define it's 10 position and 2 Decimals
Data Element:
UNIT_VAL you refer it to DOM_VALUE and define label as "Unit Value"
TOTAL_VAL you refer it to DOM_VALUE and define label as "Total Value"
Your understanding is pretty correct and not much can be added here.
You should clearly get the main thing.
Domains store technical data (decimal points, length, type, predefined values and so on)
Data elements store semantic data (labels, texts, search help binding, etc.)
Not every table field has data element (they can possess builtin type) but every field has type (either primitive or wrapped in data element).
If you wanna use your field in screens (Dynpros), ALV grids or other reports, then create data elements that will bear business meaning of your field.
If you use this field just for calculations or other utility internal tasks, then don't bother yourself.
As usual table date field (type of variable) uses data element which uses domain.
When you create fields in table and use predefined types instead of data elements you will have some problems in future, when you'll need to see the data on alv_grid.
Actually, you will see that you have some problems even before this (when you will try to make a maintenance view the header will have something like "+" symbol).
And of course we usually try to create 1 domain for 2 and more Data Elements.
In domain you talk about main logic.
In Data Element I always talk about Field label settings (how it'll show in future and some other things)
Final: Actually, the good practice, as I think to create a domain for data element, it may help you in future.
I hope that it helps you. Good luck!
Short description of the situation:
We're running a forked version of Sulu 1.5.2, PHP 7.1, Windows server environment, db connection with PostgreSQL
We have a website structure/tree where we have house templates at the top level; each house has one house_rooms and one house_occupants template; each house_rooms template has N house_rooms_room templates, and each house_occupants template has N house_occupants_occupant templates. This represents an actual House that has N Rooms and N Occupants.
Now I'd like to know if there is a way to specifically get, for instance, all the house_occupants_occupant content that follows a certain pattern of attributes (for instance: their gender attribute having value 'female' and their date_of_birth parameter being >= 1990/01/01), without having to load each house, then find its house_occupantspage among the children, and then loop over that template's house_occupants_occupant children and filter the thus begotten content according to their gender and date of birth attributes.
I already found that there is a ContentRepository class that can ::findAll() and ::findByUuids(), but there doesn't seem to be a way to filter on specific attributes (like template type, template attributes, ...). So I took a roundabout way of creating my own "repository" that does direct PDO queries on the phpcr_nodes table in the database, to specifically scan the props attribute for the occurence of a certain template name:
$this->pdo->query("SELECT identifier, props FROM phpcr_nodes WHERE props LIKE '%>house_occupants_occupant<%'");
I can see that the propscontains a string value representing an XML document that somehow translates into the entire template with attribute-value pairs, however it is obscured regarding tag-levels and how certain attributes relate to certain values. So in theory I could use a specific XML parser to turn this into something human-readable, so that for my house_occupants_occupant data I could get something like:
// what I would get after putting the props through a certain XML parser:
$xmlHumanReadableData = [
'<the_uuid_of_occupant_1>' => [
...
'gender' => 'female',
'date_of_birth' => '1992-05-18T00:00:00.000+00:00',
...
],
... //etcetera etcetera
];
When I would have that, I could filter the readable data to ascertain which content I want to keep, add the node-uuid to some $theUuids variable, and then retrieve the actual content using Sulu's ContentRepository::findByUuids($theUuids) method. That would "only" require 2 queries and some PHP array filtering in between, which is a great deal better than looping over all the children content starting from a certain parent and doing this until you've traversed all the parents and all their children... (Certainly, the overhead would increase if you'd want to search for, for instance, all house nodes where at least one of its house_occupants_occupant nodes represents a child less than 10 years old, since you'd need extra queries to "set up" the filterdata used in the final query. But still: a great deal better than looping everything... ;-) )
So my question is sort-of twofold:
What is the Sulu-specific XML parser I can use to turn the XML string value in this props column into something human-readable, with proper attribute-value pairs?
And/or, hopefully: is there a way I can avoid all this nonsense and just use a less low-level way of retrieving content of a specific template type with specific values for specific attributes ?
The ContentRepository you've found is already an abstraction to some of our requirements for pages. Your requirements are already quite specific, so you should write your own query using SQL-2, the query language for PHPCR.
This should enable you to write a query which matches your requirements.
I'm designing a REST API where you can search for data in different countries, but since you can search for the same thing, at the same time, in different countries (max 4), am I unsure of the best/correct way to do it.
This would work to start with to get data (I'm using cars as an example):
/api/uk,us,nl/car/123
That request could return different ids for the different countries (uk=1,us=2,nl=3), so what do I do when data is requested for those 3 countries?
For a nice structure I could get the data one at the time:
/api/uk/car/1
/api/us/car/2
/api/nl/car/3
But that is not very efficient since it hits the backend 3 times.
I could do this:
/api/car/?uk=1&us=2&nl=3
But that doesn't work very well if I want to add to that path:
/api/uk/car/1/owner
Because that would then turn into:
/api/car/owner/?uk=1&us=2&nl=3
Which doesn't look good.
Anyone got suggestions on how to structure this in a good way?
I answered a similar question before, so I will stick to that idea:
You have a set of elements -cars- and you want to filter it in some way. My advice is add any filter as a field. If the field is not present, then choose one country based on the locale of the client:
mydomain.com/api/v1/car?countries=uk,us,nl
This field should dissapear when you look for a specific car or its owner
mydomain.com/api/v1/car/1/owner
because the country is not needed (unless the car ID 1 is reused for each country)
Update:
I really did not expect the id of the car can be shared by several cars, an ID should be unique (like a primary key in a database). Then, it makes sense to keep the country parameter with the owner's search:
mydomain.com/api/v1/car/1/owner?countries=uk,us
This should return a list of people who own a car with the id 1... but for me this makes little sense as a functionality, in this search I'll only allow one country:
mydomain.com/api/v1/car/1/owner?country=uk
I want to write a code that has the Countrycode and Postcode as an input and the ouput are the streets that are in the given postcode using some apis that use GSM.
My tactic is as follows:
I need to get the relation Id of the district. For Example 1991416 is the relation id for the third district in Vienna - Austria. It's provided by the nominatim api: http://nominatim.openstreetmap.org/details.php?place_id=158947085
Put the id in this api url: http://polygons.openstreetmap.fr/get_wkt.py?id=1991416¶ms=0
After downloading the polygon I can put the gathered polygon in this query on the overpass api
(
way
(poly: "polygone data")
["highway"~"^(primary|secondary|tertiary|residential)$"]
["name"];
);
out geom;
And this gives me the streets of the searched district. My two problems with this solution are
1. that it takes quite a time, because asking three different APIs per request isn't that easy on ressources and
2. I don't know how to gather the relation Id from step one automatically. When I enter a Nominatim query like http:// nominatim.openstreetmap.org/search?format=json&country=austria&postalcode=1030 I just get various point in the district, but not the relation id of the searched district in order to get the desired polygone.
So my questions are if someone can tell my how I can get the relation_Id in order to do the mentioned workflow or if there is another, maybe better way to work this issue out.
Thank you for your help!
Best Regards
Daniel
You can simplify your approach quite a bit, down to a single Overpass API call, assuming you define some relevant tags to match the relation in question. In particular, you don't have to resort to using poly at all, i.e. there's no need to convert a relation to a list of lat/lon pairs. Nowadays the concept of an area can be used instead to query for certain objects in a polygon defined by a way or relation. Please check out the documentation for more details on areas.
To get the matching area for relation 1991416, I have used postal_code=1030 and boundary=administrative as filter criteria. Using that area you can then search for ways in this specific polygon:
//uncomment the following line, if you need csv output
//[out:csv(::id, ::type, name)];
//adjust area to your needs, filter critera are the same as for relations
area[postal_code=1030][boundary=administrative]->.a;
// Alternative: {{geocodeArea:name}} -> see
// http://wiki.openstreetmap.org/wiki/Overpass_turbo/Extended_Overpass_Queries
way(area.a)["highway"~"^(primary|secondary|tertiary|residential)$"]["name"];
(._;>;);out meta;
// just for checking if we're looking at the right area
rel(pivot.a);out geom;
Try it on overpass turbo link: http://overpass-turbo.eu/s/6uN
Note: not all ways/relations have a corresponding area, i.e. some area generation rules apply (see wiki page above). For your particular use case you should be ok, however.