I'm trying to use clustering techniques which should allow me to find centroids (or medoids) for each group of people inside a density map (of a real photo). I could I reach that? I've already used Kmeans strategy, and maybe the calculated centroids could be also correct. But how could I better view them over the image?
h5 file: density map of a crowd - points are representing people
Download the ".h5" from here: https://drive.google.com/file/d/1C5xvEQELswr4SJ5zhtYtUEVw2FbP2QWo/view?usp=sharing
I obtain the matrix of this h5 file through this code:
import sys
import numpy
import h5py
import matplotlib.pyplot as plt
from PIL import Image as im
with h5py.File('/content/img001001.h5', 'r') as hf:
h5_matrix= hf.get('density')[:]
plt.imshow(h5_matrix)
#print(h5_matrix[:, 1])
print(h5_matrix.shape)
Printed matrix look like this:
https://drive.google.com/file/d/1f376lUPaWT58iBIg5E693uQfC22g5m3U/view?usp=sharing
what I would like to find and have: density map with centroids
How could I afford that?
Related
I have plotted the histogram of network (dataframe), with count of 'k' node connections, like so:
import seaborn as sns
parameter ='k'
sns.histplot(network[parameter])
But now I need to create a modular random graph using above group distribution with:
from networkx.generators.community import random_partition_graph
random_partition_graph(sizes, p_in, p_out, seed=None, directed=False)
And, instead of counts, I need this value p(k), which must be passed as p_in.
p_in (float)
probability of edges with in groups
How do I get p(k) from my network?
This is how I would handle what you described. First, you can normalize your histogram such that the integral of the histogram is equal to 1. This can be done by setting the weights argument of your histogram appropriately. This histogram can then be considered the probability distribution of your degrees. Now that you have this probability distribution, i.e. a list of probability (deg_prob in the code) you can randomly sample from it using np.random.choice(np.arange(np.amin(degrees),np.amax(degrees)+1), p=deg_prob, size=N_sampling). From this random sampling, you can then create a random expected_degree_graph by just passing your samples in the w argument.
You can then compare the degree distribution of your original graph with the one from your random graph.
See below for the code and more details:
import networkx as nx
from networkx.generators.random_graphs import binomial_graph
from networkx.generators.degree_seq import expected_degree_graph
import matplotlib.pyplot as plt
import numpy as np
fig=plt.figure()
N_nodes=1000
G=binomial_graph(n=N_nodes, p=0.01, seed=0) #Creating a random graph as data
degrees = np.array([G.degree(n) for n in G.nodes()])#Computing degrees of nodes
bins_val=np.arange(np.amin(degrees),np.amax(degrees)+2) #Bins
deg_prob,_,_=plt.hist(degrees,bins=bins_val,align='left',weights=np.ones_like(degrees)/N_nodes,
color='tab:orange',alpha=0.3,label='Original distribution')#Histogram
#Sampling from distribution
N_sampling=500
random_sampling=np.random.choice(np.arange(np.amin(degrees),np.amax(degrees)+1), p=deg_prob, size=N_sampling)
#Creating random graph from samples
G_random_sampling=expected_degree_graph(random_sampling,seed=0,selfloops=False)
degrees_random_sampling = np.array([G_random_sampling.degree(n) for n in G_random_sampling.nodes()])
deg_prob_random_sampling,_,_=plt.hist(degrees_random_sampling,bins=bins_val,align='left',
weights=np.ones_like(degrees_random_sampling)/N_sampling,color='tab:blue',label='Sample distribution',alpha=0.3)
#Plotting both histograms
plt.xticks(bins_val)
plt.xlabel('degree')
plt.ylabel('Prob')
plt.legend()
plt.show()
The output then gives:
Suppose you have a graph graph = nx.read_gml("x.gml") and you'd like to drop n edges. Is there any quick way to do so?
Here is one approach using the sample function from the random library. I set k, the number of edges to be sampled to 2.
import networkx as nx
import random
G=nx.Graph()
G.add_edges_from([[1,2],[1,3],[2,3],[2,4],[3,5],[4,5]])
to_remove=random.sample(G.edges(),k=2)
G.remove_edges_from(to_remove)
print(G.edges())
I want to fit a lognormal distribution in Python. My question is why should I use scipy.lognormal.fit instead of just doing the following:
from numpy import log
mu = log(data).mean()
sigma = log(data).std()
which gives the MLE of mu and sigma so that the distribution is lognormal(mu, sigma**2)?
Also, once I get mu and sigma how can I get a scipy object of the distribution lognormal(mu, sigma**2)? The arguments passed to scipy.stats.lognorm are not clear to me.
Thanks
Wrt fitting, you could use scipy.lognormal.fit, you could use scipy.normal.fit applied to log(x), you could do what you just wrote, I believe you should get pretty much the same result.
The only thing I could state, that you have to fit two parameters (mu, sigma), so you have to match two values. Instead of going for mean/stddev, some people might prefer to match peaks, thus getting (mu,sigma) from mode/stddev.
Wrt using lognorm with known mean and stddev
from scipy.stats import lognorm
stddev = 0.859455801705594
mean = 0.418749176686875
dist=lognorm([stddev],loc=mean) # will give you a lognorm distribution object with the mean and standard deviation you specify.
# You can then get the pdf or cdf like this:
import numpy as np
import pylab as pl
x=np.linspace(0,6,200)
pl.plot(x,dist.pdf(x))
pl.plot(x,dist.cdf(x))
pl.show()
I try to fit my data to a poisson distribution:
import seaborn as sns
import scipy.stats as stats
sns.distplot(x, kde = False, fit = stats.poisson)
But I get this error:
AttributeError: 'poisson_gen' object has no attribute 'fit'
Other distribution (gamma, etc) de work well.
The Poisson distribution (implemented in scipy as scipy.stats.poisson) is a discrete distribution. The discrete distributions in scipy do not have a fit method.
I'm not very familiar with the seaborn.distplot function, but it appears to assume that the data comes from a continuous distribution. If that is the case, then even if scipy.stats.poisson had a fit method, it would not be an appropriate distribution to pass to distplot.
The question title is "How to fit a poisson distribution with seaborn?", so for the sake of completeness, here's one way to get a plot of the data and its fit. seaborn is only used for the bar plot, using #mwaskom's suggestion to use seaborn.countplot. The fitting is actually trivial, because the maximum likelihood estimation for the Poisson distribution is simply the mean of the data.
First, the imports:
In [136]: import numpy as np
In [137]: from scipy.stats import poisson
In [138]: import matplotlib.pyplot as plt
In [139]: import seaborn
Generate some data to work with:
In [140]: x = poisson.rvs(0.4, size=100)
These are the values in the x:
In [141]: k = np.arange(x.max()+1)
In [142]: k
Out[142]: array([0, 1, 2, 3])
Use seaborn.countplot to plot the data:
In [143]: seaborn.countplot(x, order=k, color='g', alpha=0.5)
Out[143]: <matplotlib.axes._subplots.AxesSubplot at 0x114700490>
The maximum likelihood estimation of the Poisson parameter is simply the mean of the data:
In [144]: mlest = x.mean()
Use poisson.pmf() to get the expected probability, and multiply by the size of the data set to get the expected counts, and then plot using matplotlib. The bars are the counts of the actual data, and the dots are the expected counts of the fitted distribution:
In [145]: plt.plot(k, poisson.pmf(k, mlest)*len(x), 'go', markersize=9)
Out[145]: [<matplotlib.lines.Line2D at 0x114da74d0>]
I have a 3000x50 feature vector matrix. I obtained a similarity matrix for this using sklearn.metrics.pairwise_distances as 'Similarity_Matrix'. Now I used networkx to create a graph using the similarity matrix generated in the previous step as G=nx.from_numpy_matrix(Similarity_Matrix). I want to perform spectral clustering on this graph G now but several google searches have failed to provide a decent example of scikit learn spectral clustering on this graph :( The official documentation shows how spectral clustering can be done on some image data which is highly unclear at least to a newbie like myself.
Can anyone give me a code sample for this or for graph cuts or graph partitioning using networkx, scikit learn etc.
Thanks a million!
adj_matrix = nx.from_numpy_matrix will help you create an adjacency matrix which will be your affinity matrix. You need to feed this to scikit-learn like this: SpectralClustering(affinity = 'precomputed', assign_labels="discretize",random_state=0,n_clusters=2).fit_predict(adj_matrix)
If you don't have any similarity matrix, you can change the value of 'affinity' param to 'rbf' or 'nearest_neighbors'. An example below explains the entire Spectral Clustering pipeline:
import sklearn
import networkx as nx
import matplotlib.pyplot as plt
'''Graph creation and initialization'''
G=nx.Graph()
G.add_edge(1,2) # default edge weight=1
G.add_edge(3,4,weight=0.2) #weight represents edge weight or affinity
G.add_edge(2,3,weight=0.9)
G.add_edge("Hello", "World", weight= 0.6)
'''Matrix creation'''
adj_matrix = nx.to_numpy_matrix(G) #Converts graph to an adj matrix with adj_matrix[i][j] represents weight between node i,j.
node_list = list(G.nodes()) #returns a list of nodes with index mapping with the a
'''Spectral Clustering'''
clusters = SpectralClustering(affinity = 'precomputed', assign_labels="discretize",random_state=0,n_clusters=2).fit_predict(adj_matrix)
plt.scatter(nodes_list,clusters,c=clusters, s=50, cmap='viridis')
plt.show()