Related
I'm trying to build isochrones for some schools, to understand the accessibility. My mode of travelling is "walk" and I wanted to create a list of at least 2 travel times, 30 minutes and 1 hour.
Here below is my code:
# import required libraries
import pandas as pd
import geopandas as gpd
import matplotlib.pyplot as plt
import networkx as nx
import osmnx as ox
from descartes import PolygonPatch
from shapely.geometry import Point, LineString, Polygon
from IPython.display import IFrame
import folium
import itertools # will be used to perform feature joining
# %matplotlib inline
ox.__version__
# function to get isochrones
def get_isochrone(lon, lat, walk_time=[30,60], speed=4.5):
loc = (lat, lon)
G = ox.graph_from_point(loc, simplify=True, network_type='walk')
# Create nodes geodataframe from Graph network (G)
gdf_nodes = ox.graph_to_gdfs(G, edges=False)
x, y = gdf_nodes['geometry'].unary_union.centroid.xy
center_node = ox.get_nearest_node(G, (y[0], x[0]))
meters_per_minute = speed * 1000 / 60 #km per hour to m per minute
for u, v, k, data in G.edges(data=True, keys=True):
data['time'] = data['length'] / meters_per_minute
# get one color for each isochrone
iso_colors = ox.plot.get_colors(n=len(walk_time), cmap="plasma", start=0, return_hex=True)
# color the nodes according to isochrone then plot the street network
node_colors = {}
for walks_time, color in zip(sorted(walk_time, reverse=True), iso_colors):
subgraph = nx.ego_graph(G, center_node, radius=walks_time, distance="time")
for node in subgraph.nodes():
node_colors[node] = color
nc = [node_colors[node] if node in node_colors else "none" for node in G.nodes()]
ns = [15 if node in node_colors else 0 for node in G.nodes()]
# make isochrone polygons
isochrone_polys = []
for trip_times in sorted(walk_time, reverse=True):
subgraph = nx.ego_graph(G, center_node, radius=trip_times, distance="time")
node_points = [Point(data['x'], data['y']) for node, data in subgraph.nodes(data=True)]
polys = gpd.GeoSeries(node_points).unary_union.convex_hull
isochrone_polys.append(polys)
# adding color
for polygon, fc in zip(isochrone_polys, iso_colors):
patch = PolygonPatch(polygon, fc=fc, ec="none", alpha=0.6, zorder=-1)
# isochrone_polys.add_patch(patch)
return isochrone_polys
When I call the get_isochrone function:
# calling the function with the coordinates
map = coordinates.apply(lambda x: get_isochrone(x.longitude, x.latitude), axis=1)
What is returned is a list of polygons with each school point having polygons equal to the number of items in the travel times list.
I however noticed that the polygons returned for each point are exactly the same.
Just to verify: This script shows that they are the same:
for i in map:
print(i[0])
print(i[1])
print('\n')
Here are the results:
As you can see, each school point returns two polygons with exactly the same coordinates, for different travel times. I have reviewed the python function several times, even changed the travel time to one item in the travel time list like 100, and I still get exactly the same coordinates. I may have missed something and I'll appreciate any help. Thanks!
In addition, I also realized that the add_patch() method in the function adds color to the plot_graph() only. How can I add the colors to the polygons in the Folium map?
it appears that you have used this example: https://github.com/gboeing/osmnx-examples/blob/main/notebooks/13-isolines-isochrones.ipynb
if all you want is polygons, then there is no need to do PolygonPatch
you have refactored incorrectly, for multiple walk times. You only need to generate edges once
pulling it all together:
source some schools
get_isochrone() refactored to your use case. Have changed do it returns a dict that has index and name of point / school being investigated.
generate a geopandas data frame of isochrones
visualise it
data sourcing
import osmnx as ox
import pandas as pd
import warnings
import networkx as nx
import geopandas as gpd
from shapely.geometry import Point
warnings.simplefilter(action="ignore", category=FutureWarning)
warnings.simplefilter(action="ignore", category=PendingDeprecationWarning)
ox.config(use_cache=True, log_console=False)
# get some cities
cities = ["Hereford"] # ["Hereford", "Worcester", "Gloucester"]
cities = ox.geocode_to_gdf([{"city": c, "country": "UK"} for c in cities])
# get some schools
tags = {"amenity": "school"}
schools = pd.concat(
[
ox.geometries.geometries_from_polygon(r["geometry"], tags)
for i, r in cities.iterrows()
]
)
schools = (
schools.loc["way"].dropna(axis=1, thresh=len(schools) / 4).drop(columns=["nodes"])
)
# change polygon to point
schools["geometry"] = schools.to_crs(schools.estimate_utm_crs())[
"geometry"
].centroid.to_crs(schools.crs)
get_isochrone()
# function to get isochrones
def get_isochrone(
lon, lat, walk_times=[15, 30], speed=4.5, name=None, point_index=None
):
loc = (lat, lon)
G = ox.graph_from_point(loc, simplify=True, network_type="walk")
gdf_nodes = ox.graph_to_gdfs(G, edges=False)
center_node = ox.distance.nearest_nodes(G, lon, lat)
meters_per_minute = speed * 1000 / 60 # km per hour to m per minute
for u, v, k, data in G.edges(data=True, keys=True):
data["time"] = data["length"] / meters_per_minute
polys = []
for walk_time in walk_times:
subgraph = nx.ego_graph(G, center_node, radius=walk_time, distance="time")
node_points = [
Point(data["x"], data["y"]) for node, data in subgraph.nodes(data=True)
]
polys.append(gpd.GeoSeries(node_points).unary_union.convex_hull)
info = {}
if name:
info["name"] = [name for t in walk_times]
if point_index:
info["point_index"] = [point_index for t in walk_times]
return {**{"geometry": polys, "time": walk_times}, **info}
integration
WT = [5, 10, 15]
SCHOOLS = 5
# build geopandas data frame of isochrone polygons for each school
isochrones = pd.concat(
[
gpd.GeoDataFrame(
get_isochrone(
r["geometry"].x,
r["geometry"].y,
name=r["name"],
point_index=i,
walk_times=WT,
),
crs=schools.crs,
)
for i, r in schools.head(SCHOOLS).iterrows()
]
)
visualise
warnings.filterwarnings("ignore")
gdf = isochrones.set_index(["time", "point_index"]).copy()
# remove shorter walk time from longer walk time polygon to make folium work better
for idx in range(len(WT)-1,0,-1):
gdf.loc[WT[idx], "geometry"] = (
gdf.loc[WT[idx]]
.apply(
lambda r: r["geometry"].symmetric_difference(
gdf.loc[(WT[idx-1], r.name), "geometry"]
),
axis=1,
)
.values
)
m = gdf.reset_index().explore(column="time", height=300, width=500, scheme="boxplot")
schools.head(SCHOOLS).explore(m=m, marker_kwds={"radius": 3, "color": "red"})
merge overlapping polygons
using this technique https://gis.stackexchange.com/questions/334459/how-to-dissolve-overlapping-polygons-using-geopandas
have visualised in different way
import matplotlib.cm as cm
import matplotlib.colors as colors
import folium
# merge overlapping polygons
# https://gis.stackexchange.com/questions/334459/how-to-dissolve-overlapping-polygons-using-geopandas
mergedpolys = gpd.GeoDataFrame(
geometry=isochrones.groupby("time")["geometry"]
.agg(lambda g: g.unary_union)
.apply(lambda g: [g] if isinstance(g, shapely.geometry.Polygon) else g.geoms)
.explode(),
crs=isochrones.crs,
)
# visualize merged polygons
m = None
for i, wt in enumerate(WT[::-1]):
m = mergedpolys.loc[[wt]].explore(
m=m,
color=colors.to_hex(cm.get_cmap("tab20b", len(WT))(i)),
name=wt,
height=300,
width=500,
)
m = schools.head(SCHOOLS).explore(
m=m, marker_kwds={"radius": 3, "color": "red"}, name="schools"
)
folium.LayerControl().add_to(m)
m
im trying to replicate Steel8's answer, but when I run that code I get errors. I think these errors are due to the fact that I am using a different version of osmnx. Could you tell me which version of each package are you using?
import osmnx as ox
import pandas as pd
import warnings
import networkx as nx
import geopandas as gpd
from shapely.geometry import Point
warnings.simplefilter(action="ignore", category=FutureWarning)
warnings.simplefilter(action="ignore", category=PendingDeprecationWarning)
ox.config(use_cache=True, log_console=False)
# get some cities
cities = ["Hereford"] # ["Hereford", "Worcester", "Gloucester"]
cities = ox.geocode_to_gdf([{"city": c, "country": "UK"} for c in cities])
# get some schools
tags = {"amenity": "school"}
schools = pd.concat(
[
ox.geometries.geometries_from_polygon(r["geometry"], tags)
for i, r in cities.iterrows()
]
)
schools = (
schools.loc["way"].dropna(axis=1, thresh=len(schools) / 4).drop(columns=["nodes"])
)
# change polygon to point
schools["geometry"] = schools.to_crs(schools.estimate_utm_crs())[
"geometry"
].centroid.to_crs(schools.crs)
# function to get isochrones
def get_isochrone(
lon, lat, walk_times=[15, 30], speed=4.5, name=None, point_index=None
):
loc = (lat, lon)
G = ox.graph_from_point(loc, simplify=True, network_type="walk")
gdf_nodes = ox.graph_to_gdfs(G, edges=False)
center_node = ox.distance.nearest_nodes(G, lon, lat)
meters_per_minute = speed * 1000 / 60 # km per hour to m per minute
for u, v, k, data in G.edges(data=True, keys=True):
data["time"] = data["length"] / meters_per_minute
polys = []
for walk_time in walk_times:
subgraph = nx.ego_graph(G, center_node, radius=walk_time, distance="time")
node_points = [
Point(data["x"], data["y"]) for node, data in subgraph.nodes(data=True)
]
polys.append(gpd.GeoSeries(node_points).unary_union.convex_hull)
info = {}
if name:
info["name"] = [name for t in walk_times]
if point_index:
info["point_index"] = [point_index for t in walk_times]
return {**{"geometry": polys, "time": walk_times}, **info}
WT = [5, 10, 15]
SCHOOLS = 5
# build geopandas data frame of isochrone polygons for each school
isochrones = pd.concat(
[
gpd.GeoDataFrame(
get_isochrone(
r["geometry"].x,
r["geometry"].y,
name=r["name"],
point_index=i,
walk_times=WT,
),
crs=schools.crs,
)
for i, r in schools.head(SCHOOLS).iterrows()
]
)
warnings.filterwarnings("ignore")
gdf = isochrones.set_index(["time", "point_index"]).copy()
# remove shorter walk time from longer walk time polygon to make folium work better
for idx in range(len(WT)-1,0,-1):
gdf.loc[WT[idx], "geometry"] = (
gdf.loc[WT[idx]]
.apply(
lambda r: r["geometry"].symmetric_difference(
gdf.loc[(WT[idx-1], r.name), "geometry"]
),
axis=1,
)
.values
)
m = gdf.reset_index().explore(column="time", height=300, width=500, scheme="boxplot")
schools.head(SCHOOLS).explore(m=m, marker_kwds={"radius": 3, "color": "red"})
I'm gettin the following error message:
PS C:\Users\IGPOZO\Desktop\INAKI\Isochrones\Python> python test4.py
Traceback (most recent call last):
File "C:\Users\IGPOZO\Anaconda3\envs\isochrones_env\lib\site-packages\pandas\core\indexes\base.py", line 3621, in get_loc
return self._engine.get_loc(casted_key)
File "pandas\_libs\index.pyx", line 136, in pandas._libs.index.IndexEngine.get_loc
File "pandas\_libs\index.pyx", line 144, in pandas._libs.index.IndexEngine.get_loc
File "pandas\_libs\index_class_helper.pxi", line 41, in pandas._libs.index.Int64Engine._check_type
KeyError: 'way'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users\IGPOZO\Desktop\INAKI\Isochrones\Python\test4.py", line 24, in <module>
schools.loc["way"].dropna(axis=1, thresh=len(schools) / 4).drop(columns=["nodes"])
File "C:\Users\IGPOZO\Anaconda3\envs\isochrones_env\lib\site-packages\pandas\core\indexing.py", line 967, in __getitem__
return self._getitem_axis(maybe_callable, axis=axis)
File "C:\Users\IGPOZO\Anaconda3\envs\isochrones_env\lib\site-packages\pandas\core\indexing.py", line 1202, in _getitem_axis
return self._get_label(key, axis=axis)
File "C:\Users\IGPOZO\Anaconda3\envs\isochrones_env\lib\site-packages\pandas\core\indexing.py", line 1153, in _get_label
return self.obj.xs(label, axis=axis)
File "C:\Users\IGPOZO\Anaconda3\envs\isochrones_env\lib\site-packages\pandas\core\generic.py", line 3876, in xs
loc = index.get_loc(key)
File "C:\Users\IGPOZO\Anaconda3\envs\isochrones_env\lib\site-packages\pandas\core\indexes\base.py", line 3623, in get_loc
raise KeyError(key) from err
KeyError: 'way'
I need to compute a numerical (triple) integral, but do not need very high precision on the value, and would therefore like to sacrifice some precision for speed when using nquad. I thought that I might be able to do this by increasing the epsrel and/or epsabs options, but they seem to have no effect. For example (note, this is just an example integrand - I don't actually need to compute this particular integral...):
import numpy as np
from scipy.integrate import nquad
def integrand(l, b, d, sigma=250):
x = d * np.cos(l) * np.cos(b)
y = d * np.sin(l) * np.cos(b)
z = d * np.sin(b)
return np.exp(-0.5 * z**2 / sigma**2) / np.sqrt(2*np.pi * sigma**2)
ranges = [
(0, 2*np.pi),
(0.5, np.pi/2),
(0, 1000.)
]
# No specification of `opts` - use the default epsrel and epsabs:
result1 = nquad(integrand, ranges=ranges, full_output=True)
# Set some `quad` opts:
result2 = nquad(integrand, ranges=ranges, full_output=True,
opts=dict(epsabs=1e-1, epsrel=0, limit=3))
Both outputs are identical:
>>> print(result1)
(4.252394424844468, 1.525272379143154e-12, {'neval': 9261})
>>> print(result2)
(4.252394424844468, 1.525272379143154e-12, {'neval': 9261})
A full example is included here: https://gist.github.com/adrn/b9aa92c236df011dbcdc131aa94ed9f9
Is this not the right approach, or is scipy.integrate ignoring my inputted opts?
From the scipy.integrate.nquad it is stated that opts can only be passed to quad as can be seen here:
https://docs.scipy.org/doc/scipy/reference/generated/scipy.integrate.nquad.html
Example of application:
import numpy as np
from scipy.integrate import quad
def integrand(a, sigma=250):
x = 2 * np.sin(a) * np.cos(a)
return x
# No specification of `opts` - use the default epsrel and epsabs:
result1 = quad(integrand,0, 2*np.pi)
# Set some `quad` opts:
result2 = quad(integrand,0, 4*np.pi,epsabs=1e-6, epsrel=1e-6, limit=40)
returns:
result1: (-1.3690011097614755e-16, 4.4205541621600365e-14)
result2: (-1.7062635631484713e-15, 9.096805257467047e-14)
The reason nquad doesn't complain about the presence of options is because nquad includes quad, dbquad and tplquad.
I am using the standard (string indexer + one hot encoder + randomForest) pipeline in spark, as shown below
labelIndexer = StringIndexer(inputCol = class_label_name, outputCol="indexedLabel").fit(data)
string_feature_indexers = [
StringIndexer(inputCol=x, outputCol="int_{0}".format(x)).fit(data)
for x in char_col_toUse_names
]
onehot_encoder = [
OneHotEncoder(inputCol="int_"+x, outputCol="onehot_{0}".format(x))
for x in char_col_toUse_names
]
all_columns = num_col_toUse_names + bool_col_toUse_names + ["onehot_"+x for x in char_col_toUse_names]
assembler = VectorAssembler(inputCols=[col for col in all_columns], outputCol="features")
rf = RandomForestClassifier(labelCol="indexedLabel", featuresCol="features", numTrees=100)
labelConverter = IndexToString(inputCol="prediction", outputCol="predictedLabel", labels=labelIndexer.labels)
pipeline = Pipeline(stages=[labelIndexer] + string_feature_indexers + onehot_encoder + [assembler, rf, labelConverter])
crossval = CrossValidator(estimator=pipeline,
estimatorParamMaps=paramGrid,
evaluator=evaluator,
numFolds=3)
cvModel = crossval.fit(trainingData)
now after the the fit I can get the random forest and the feature importance using cvModel.bestModel.stages[-2].featureImportances, but this does not give me feature/ column names, rather just the feature number.
What I get is below:
print(cvModel.bestModel.stages[-2].featureImportances)
(1446,[3,4,9,18,20,103,766,981,983,1098,1121,1134,1148,1227,1288,1345,1436,1444],[0.109898803421,0.0967396441648,4.24568235244e-05,0.0369705839109,0.0163489685127,3.2286694534e-06,0.0208192703688,0.0815822887175,0.0466903663708,0.0227619959989,0.0850922269211,0.000113388896956,0.0924779490403,0.163835022713,0.118987129392,0.107373548367,3.35577640585e-05,0.000229569946193])
How can I map it back to some column names or column name + value format?
Basically to get the feature importance of random forest along with the column names.
The transformed dataset metdata has the required attributes.Here is an easy way to do -
create a pandas dataframe (generally feature list will not be huge, so no memory issues in storing a pandas DF)
pandasDF = pd.DataFrame(dataset.schema["features"].metadata["ml_attr"]
["attrs"]["binary"]+dataset.schema["features"].metadata["ml_attr"]["attrs"]["numeric"]).sort_values("idx")
Then create a broadcast dictionary to map. broadcast is necessary in a distributed environment.
feature_dict = dict(zip(pandasDF["idx"],pandasDF["name"]))
feature_dict_broad = sc.broadcast(feature_dict)
You can also look here and here
Hey why don't you just map it back to the original columns through list expansion. Here is an example:
# in your case: trainingData.columns
data_frame_columns = ["A", "B", "C", "D", "E", "F"]
# in your case: print(cvModel.bestModel.stages[-2].featureImportances)
feature_importance = (1, [1, 3, 5], [0.5, 0.5, 0.5])
rf_output = [(data_frame_columns[i], feature_importance[2][j]) for i, j in zip(feature_importance[1], range(len(feature_importance[2])))]
dict(rf_output)
{'B': 0.5, 'D': 0.5, 'F': 0.5}
I was not able to find any way to get the true initial list of the columns back after the ml algorithm, I am using this as the current workaround.
print(len(cols_now))
FEATURE_COLS=[]
for x in cols_now:
if(x[-6:]!="catVar"):
FEATURE_COLS+=[x]
else:
temp=trainingData.select([x[:-7],x[:-6]+"tmp"]).distinct().sort(x[:-6]+"tmp")
temp_list=temp.select(x[:-7]).collect()
FEATURE_COLS+=[list(x)[0] for x in temp_list]
print(len(FEATURE_COLS))
print(FEATURE_COLS)
I have kept a consistent suffix naming across all the indexer (_tmp) & encoder (_catVar) like:
column_vec_in = str_col
column_vec_out = [col+"_catVar" for col in str_col]
indexers = [StringIndexer(inputCol=x, outputCol=x+'_tmp')
for x in column_vec_in ]
encoders = [OneHotEncoder(dropLast=False, inputCol=x+"_tmp", outputCol=y)
for x,y in zip(column_vec_in, column_vec_out)]
tmp = [[i,j] for i,j in zip(indexers, encoders)]
tmp = [i for sublist in tmp for i in sublist]
This can be further improved and generalized, but currently this tedious work around works best
I'd like to create a polyhedron in OpenSCAD by generating its point vectors programmatically. However, assigning a value to a matrix...
p = [1,0,0];
r = [[], [], [], [], [], [] ];
for( i=[0:5] )
{
echo("i=",i);
r[i] = [0,1];
}
... produces a syntax error:
ERROR: Parser error in line 7: syntax error
ERROR: Compilation failed!
Am I making a mistake (which) or are these types really only read-only (no assignment by index)?
It's not possible to modify r after it was assigned.
See https://en.wikibooks.org/wiki/OpenSCAD_User_Manual/List_Comprehensions for expressions that can generate lists.
Example:
points = [ for (a = [0 : 5 : 359]) [ 20 * sin(a), 10 * cos(a) ] ];
polygon(points);
I want to add two matrcies in python 3 but the problem comes when I add input to the program
Here is my code
def addmatrix(a,b):
d=[]
n=0
while n < len (a):
c = []
k = 0
while k < len (a[0]) :
c.append(a[n][k]+b[n][k])
k += 1
n += 1
d.append (c)
return d
def main():
a = input("Enter a Matrix: ")
b = input("Enter another Matrix: ")
print (addmatrix(a,b))
main()
If the input is
Enter a Matrix: [[5,6], [1,2], [2,4]]
Enter another Matrix: [[2,3], [-6,0], [-2, 4]]
The output comes out as [['[['], ['[['], ['52'], [',,'], ['63'], [']]'], [',,'], [' '], ['[['], ['1-'], [',6'], ['2,'], [']0'], [',]'], [' ,'], ['[ '], ['2['], [',-'], ['42'], ['],'], ['] ']]
But if I take out the input from the program and make it so that
def main():
a = [[5,6], [1,2], [2,4]]
b = [[2,3], [-6,0], [-2, 4]]
print (addmatrix(a,b))
main()
The output then comes out as [[7, 9], [-5, 2], [0, 8]] which is correct.
Is there a way I can make my program work so that when a person inputs two matrices they add together? I'm new at python so any help will be appreciated :)
You will have to convert the user input into a Python object. Right now, it's a string.
You can use eval (which should not be used if you don't know what your users will input. I can type in __import__('os').system('rm /some/file.txt') and Python will delete a file):
a = eval(input("Enter a Matrix: "))
Or you can use ast.literal_eval, which is safe:
from ast import literal_eval
...
a = literal_eval(input("Enter a Matrix: "))
Try this:
import ast
def addmatrix(a,b):
return [map(sum, zip(*x)) for x in zip(a,b)]
def main():
a = ast.literal_eval(raw_input("Enter a Matrix: "))
b = ast.literal_eval(raw_input("Enter another Matrix: "))
print addmatrix(a,b)
main()