PostgreSQL DB location query with Fluent 4 - swift

I have an application that stores items with a latitude and longitude, I wanted to create a query that filters the items and paginates them by their distance to a given lat/lon pair.
I have read online and a lot of the solutions don't seem feasible within the constraints of fluent.

You'll want to use PostGIS for any kind of Geometric/Geographic query. Luckily there's a package for that! https://github.com/brokenhandsio/fluent-postgis
You can do filterGeograghyWithin() to find items within a certain distance. Note the geographic queries are better suited for these kind of transformations (and more accurate) as they take into account the curvature of the Earth.
The downside is you'll need to convert your lat/lon columns to a GeographicPoint2D which is the native PostGIS type, but given PostGIS is the de facto standard for this kind of work it's worth doing anyway

The best you could do is to calculate the angular distance (ang) from your given latitude (lat) and longitude (lng), and then select the square region that will include the circle. Then use Pythagoras to filter those lying within the required distance:
let ang2 = pow(ang, 2)
Point.query(on: req.db)
.filter(\.$lng >= lng - ang).filter(\.$lng <= lng + ang)
.filter(\.$lng >= lat - ang).filter(\.$lat <= lat + ang).all().flatMap { points in
let closePoints = points.filter { pow($0.lat - lat, 2) * pow($0.lng - lng, 2) <= ang2) }
// continue processing
}
}
EDIT: Following the OPs disclosure that he wants to paginate the results!
Capture the id values of the final set of Points and then filter on this. Something like:
let ang2 = pow(ang, 2)
Point.query(on: req.db)
.filter(\.$lng >= lng - ang).filter(\.$lng <= lng + ang)
.filter(\.$lng >= lat - ang).filter(\.$lat <= lat + ang).all().flatMap { points in
return points.filter { pow($0.lat - lat, 2) * pow($0.lng - lng, 2) <= ang2) }.map { $0.id }
}.flatMap { ids in
Point.query(on: req.db).filter(\.$id ~~ ids).paginate().flatMap { points in
// continue processing
}
}
}

Related

Looking for advice on improving a custom function in AnyLogic

I'm estimating last mile delivery costs in an large urban network using by-route distances. I have over 8000 customer agents and over 100 retail store agents plotted in a GIS map using lat/long coordinates. Each customer receives deliveries from its nearest store (by route). The goal is to get two distance measures in this network for each store:
d0_bar: the average distance from a store to all of its assigned customers
d1_bar: the average distance between all customers common to a single store
I've written a startup function with a simple foreach loop to assign each customer to a store based on by-route distance (customers have a parameter, "customer.pStore" of Store type). This function also adds, in turn, each customer to the store agent's collection of customers ("store.colCusts"; it's an array list with Customer type elements).
Next, I have a function that iterates through the store agent population and calculates the two average distance measures above (d0_bar & d1_bar) and writes the results to a txt file (see code below). The code works, fortunately. However, the problem is that with such a massive dataset, the process of iterating through all customers/stores and retrieving distances via the openstreetmap.org API takes forever. It's been initializing ("Please wait...") for about 12 hours. What can I do to make this code more efficient? Or, is there a better way in AnyLogic of getting these two distance measures for each store in my network?
Thanks in advance.
//for each store, record all customers assigned to it
for (Store store : stores)
{
distancesStore.print(store.storeCode + "," + store.colCusts.size() + "," + store.colCusts.size()*(store.colCusts.size()-1)/2 + ",");
//calculates average distance from store j to customer nodes that belong to store j
double sumFirstDistByStore = 0.0;
int h = 0;
while (h < store.colCusts.size())
{
sumFirstDistByStore += store.distanceByRoute(store.colCusts.get(h));
h++;
}
distancesStore.print((sumFirstDistByStore/store.colCusts.size())/1609.34 + ",");
//calculates average of distances between all customer nodes belonging to store j
double custDistSumPerStore = 0.0;
int loopLimit = store.colCusts.size();
int i = 0;
while (i < loopLimit - 1)
{
int j = 1;
while (j < loopLimit)
{
custDistSumPerStore += store.colCusts.get(i).distanceByRoute(store.colCusts.get(j));
j++;
}
i++;
}
distancesStore.print((custDistSumPerStore/(loopLimit*(loopLimit-1)/2))/1609.34);
distancesStore.println();
}
Firstly a few simple comments:
Have you tried timing a single distanceByRoute call? E.g. can you try running store.distanceByRoute(store.colCusts.get(0)); just to see how long a single call takes on your system. Routing is generally pretty slow, but it would be good to know what the speed limit is.
The first simple change is to use java parallelism. Instead of using this:
for (Store store : stores)
{ ...
use this:
stores.parallelStream().forEach(store -> {
...
});
this will process stores entries in parallel using standard Java streams API.
It also looks like the second loop - where avg distance between customers is calculated doesn't take account of mirroring. That is to say distance a->b is equal to b->a. Hence, for example, 4 customers will require 6 calculations: 1->2, 1->3, 1->4, 2->3, 2->4, 3->4. Whereas in case of 4 customers your second while loop will perform 9 calculations: i=0, j in {1,2,3}; i=1, j in {1,2,3}; i=2, j in {1,2,3}, which seems wrong unless I am misunderstanding your intention.
Generally, for long running operations it is a good idea to include some traceln to show progress with associated timing.
Please have a look at above and post results. With more information additional performance improvements may be possible.

How to find documents in an x mile radius using geohashes without filtering on client?

So currently I am using geohashes to do location based queries as such (following this stackoverflow post: Finding geohashes of certain length within radius from a point)
public extension CLLocationCoordinate2D {
func boundingBox(radius: CLLocationDistance) -> (max: CLLocationCoordinate2D, min: CLLocationCoordinate2D) {
// 0.0000089982311916 ~= 1m
let offset = 0.0000089982311916 * radius
let latMax = self.latitude + offset
let latMin = self.latitude - offset
// 1 degree of longitude = 111km only at equator
// (gradually shrinks to zero at the poles)
// So need to take into account latitude too
let lngOffset = offset * cos(self.latitude * .pi / 180.0)
let lngMax = self.longitude + lngOffset
let lngMin = self.longitude - lngOffset
let max = CLLocationCoordinate2D(latitude: latMax, longitude: lngMax)
let min = CLLocationCoordinate2D(latitude: latMin, longitude: lngMin)
return (max, min)
}
func isWithin(min: CLLocationCoordinate2D, max: CLLocationCoordinate2D) -> Bool {
return
self.latitude > min.latitude &&
self.latitude < max.latitude &&
self.longitude > min.longitude &&
self.longitude < max.longitude
}
}
func getGeohashPrefix(){
let loc = CLLocationCoordinate2D(latitude: lat!, longitude: long!)
MBR = loc.boundingBox(radius: 16093.4) //16093.4 meters = 10 miles
//corners = [NorthWest, SouthWest, SouthEast, NorthEast] in lat n long
let corners = [CLLocationCoordinate2D(latitude: MBR.0.latitude,longitude: MBR.1.longitude),
MBR.1, CLLocationCoordinate2D(latitude: MBR.1.latitude, longitude: MBR.0.longitude),
MBR.0]
var geohashes_of_corners: [String] = []
for corner in corners {
geohashes_of_corners.append(corner.geohash(length: 12))
}
geohashes_prefix = geohashes_of_corners.longestCommonPrefix()
}
var query: Query = db.collection("Users").whereField("geohash",isGreaterThanOrEqualTo: geohashes_prefix).whereField("geohash",isLessThanOrEqualTo: geohashes_prefix + "~").order(by: "geohash", descending: false)
query.getDocuments { (querySnapshot, err) in
if err != nil{
print("error getting da documents")
}else{
if querySnapshot!.isEmpty{
return completion(arr_of_people)
}
for document in querySnapshot!.documents {
let d = document.data()
let isPersonWithin = CLLocationCoordinate2D(latitude: (d["loc"] as! GeoPoint).latitude, longitude: (d["loc"] as! GeoPoint).longitude).isWithin(min: self.MBR.1, max: self.MBR.0)
if !isPersonWithin{
continue
}
arr_of_people.append([d["firstName"] as! String, d["lastName"] as! String])
}
return completion(arr_of_people)
}
}
As you can see, I am querying for documents with a specific prefix and then filtering those documents AGAIN on the client. Is that safe? If not, what is the workaround? Use cloud functions, different algorithm (suggest one if you have it), or something else?
A query on geohashes returns points within a certain range of geohashes, which are (somewhat) rectangular regions.
A geoquery on a central point and a distance returns points that are in a circle.
Since the two shapes are different, your code uses a client-side check to cut off the points that are outside the circle, but inside the rectangles. This is a normal step when performing geoqueries for a max distance around a point when using geohashes.
Here's an example of this on a map:
The green pins are in a 250km circle around San Francisco, which is what I queried for. The red pins are outside of that circle, but within the set of geohash ranges ([["9q0","9qh"],["9nh","9n~"],["9r0","9rh"],["9ph","9p~"]] here) that needed to be queried to be sure we had all points in range.
As said: this so-called oversampling is inherent to using geohashes to perform point-and-distance queries. In my experience you'll end up reading between 2x and 8x too many documents.
It might be possible to reduce the number of false positives by querying for more-but-smaller ranges, but I don't know of any geo-libraries for Firestore that do that.
I mapped the extra cost as a mental model: finding documents within a certain distance from a point costs me 2x to 8x more than a regular document read.
Moving the operation to Cloud Functions on the server makes no difference on the number of documents that needs to be read, it just changes where they are read. So you can perform the operation on a server to reduce the bandwidth of transferring the documents from the database to the client. But it won't make a difference in the number of documents that need to be read.
As discussed in comments: performing the query on the server does allow you to remove client-side access from the data, so that you can ensure the application code will never see documents that are not within the requested range. So if you're worried about document access, performing the query in a trusted environment (such as a server you control or Cloud Functions) is a good option.
To not pay for the extra document reads, consider finding a service that natively supports geoqueries with a pricing model based on the number of results it returns - instead of the number of results it has to consider. Such a service will (very likely) still consider too many points, but if the pricing model matches what you want that might be worth it.

How to remove all array items except at indexes at multiples of x - Swift

I have an array of coordinates (965 in total). I want to use these coordinates in Google Roads API however the limit is 100.
I have a function that determines how many items in the array and then gets the value to use.
round(Double(userCoordinatesHardCoded.count / 100))
produces 9.
I would like to remove ALL items that are not at indexes that are multiples of, in this case, 9. So in theory I will only ever have no more than 100 items in the array.
If possible, I would like to keep the first and last array item.
I know this has already been answered, but this is a great use case for using the filter function built into Swift:
let newCoordinates = oldCoordinates.filter { coord in
return (coord % 9 != 0)
}
If space is not a problem, you can create another array with the desired multiples.
var oldCoordinates // Imagine it having all those elements
var newCoordinates : GoogleCoordinates = [] // This one will have new ones
newCoordinates.append(oldCoordinates[0])
var x = 1
for (x; x < oldCoordinates.count ; x++ ) {
if (x % 5 == 0) {
newCoordinates.append(oldCoordinates[x])
}
}
if (x != (oldCoordinates.count - 1)) {
newCoordinates.append(oldCoordinates[oldCoordinates.count - 1])
}

MongoDb 2.6.1 Error: 17444 - "Legacy point is out of bounds for spherical query"

After upgrading MongoDb to 2.6.1 in my System n i get sometimes the following error:
Legacy point is out of bounds for spherical query
ErrorCode 17444
Here: https://github.com/mongodb/mongo/blob/master/src/mongo/db/geo/geoquery.cpp#L73
I can see that this is raised by mongo db due to some invalid data.
// The user-provided point can be flat. We need to make sure that it's in bounds.
if (isNearSphere) {
uassert(17444,
"Legacy point is out of bounds for spherical query",
centroid.flatUpgradedToSphere || (SPHERE == centroid.crs));
}
But currently i can't figure why and how to prevent it.
My Code looks like this:
IEnumerable<BsonValue> cids = companyIds.ToBsonValueArray();
return Collection.Find(
Query.And(
Query.In("CompanyId", cids),
Query.Near("Location", location.Geography.Longitude, location.Geography.Latitude, location.Radius / 6371000, true))).ToList();
Stacktrace:
QueryFailure flag was Legacy point is out of bounds for spherical
query (response was { "$err" : "Legacy point is out of bounds for
spherical query", "code" : 17444 }).
at MongoDB.Driver.Internal.MongoReplyMessage1.ReadFrom(BsonBuffer buffer, IBsonSerializationOptions serializationOptions)
at MongoDB.Driver.Internal.MongoConnection.ReceiveMessage[TDocument](BsonBinaryReaderSettings
readerSettings, IBsonSerializer serializer, IBsonSerializationOptions
serializationOptions)
at MongoDB.Driver.Operations.QueryOperation1.GetFirstBatch(IConnectionProvider
connectionProvider)
You're using MongoDB 2.6.1 or higher because the code you're looking at was added as a fix for a JIRA-13666 issue.
The problem was that some $near queries would crash MongoDB server when called with legacy coordinates that are out of range.
You're probably sending coordinates that are out of range. The part of the code that checks longitude and latitude when doing $near queries with max distance (GeoParser::parsePointWithMaxDistance method in geoparser.cpp):
bool isValidLngLat(double lng, double lat) {
return lat >= -90 && lat <= 90 && lng >= -180 && lng <= 180;
}
If the coordinates are out of range centroid.flatUpgradedToSphere will be false and that will cause the error you're receiving.
You should either change your coordinates to be in range or set spherical parameter to false to avoid getting this error.
Query.Near("Location", location.Geography.Longitude,
location.Geography.Latitude, location.Radius / 6371000, false)

MKMapView- How to get & display the annotation pins for Near by me addresses/locations from the list of existing address

I was very new to map kit MKMapView application,
I have a list of 1000 addresses of members in an organization.
I have get the coordinates (lat, long) all these members according to their addresses,
using - (CLLocationCoordinate2D) geoCodeUsingAddress:(NSString *)address method
by Referring this link
And display annotations of all the 1000 members in map view
By referring this link
But now my requirement is to display only Near by me Current location memebers locations only in 2 ways,
Top 20 members nearer by me
List of all members surrounded by 50KM around me
How to implement these two scenarios,
You have to find the distance between each member and your current location by
CLLocation *memberlocation = [[CLLocation alloc] initWithLatitude:memberlat longitude:memberlong];
CLLocation *yourcurrentlocation = [[CLLocation alloc] initWithLatitude:currentlocationlat longitude:currentlocationlong];
CLLocationDistance distance = [memberlocation distanceFromLocation:yourcurrentlocation];
and then using them you can find the nearest members, and using your own logic you have code for your required solution.
If you think comparing 1000 address with self location will took time , then send the self location lat-long to server webservice where you have all the members lat-long , there in the webservice you have to use this following query to find the nearest 20 members.
SELECT top (20) id, ( 6371 * acos( cos( radians(13.0610) ) * cos( radians( lat ) ) * cos( radians( lng ) - radians(80.2404) ) + sin( radians(13.0610) ) * sin( radians( lat ) ) ) ) AS distance
FROM markers ORDER BY distance;
Here instead of 13.0610 you have to pass your self location latitude, and 80.2404 is longitude