Delete redundant entries within groups - postgresql

Question

I want to remove redundant rows in my database within each group (in this case datasource), which I define as rows which contains strictly less information or different information than some other row.
For example in the table below. Row 1 is redundant as the other row row 0 in its same group contains the exact same information as it but with more data.
For the same reason row 6 is redundant all the other rows 3, 4 and 5 in the group contains more information that it. However I keep both row 4 and 5 as they have some additional different information than the other rows in the group.
datasource city country
0 1 Shallotte US
1 1 None US
2 2 austin US
3 3 Casselberry US
4 3 None AU
5 3 Springfield None
6 3 None None
An example when there are more columns, rows 0 and 1, 4 are different information. However rows 2 and 3(or row 1) contains redundant information.
datasource city country Count
0 1 None US 11
1 1 austin None None
2 1 None None 11
3 1 austin None None
4 1 None CA None
Expected output
datasource city country Count
0 1 None US 11
1 1 austin None None
4 1 None CA None
Is there a simple way which I could achieve such logic in pandas or SQL (PostrgeSQL) for any number of columns?

Answer 1

Here's a different approach using the same basic strategy as Bharath shetty's solution. This way feels a bit neater to me.
First, construct the example data frame:
import pandas as pd
data = {"datasource": [1,1,2,3,3,3,3],
"city": ["Shallotte", None, "austin", "Casselberry", None, "Springfield", None],
"country": ["US", "US", "US", "US", "AU", None, None]}
df = pd.DataFrame(data)
df['null'] = df.isnull().sum(axis=1)
print(df)
city country datasource null
0 Shallotte US 1 0
1 None US 1 1
2 austin US 2 0
3 Casselberry US 3 0
4 None AU 3 1
5 Springfield None 3 1
6 None None 3 2
Now make a boolean mask using groupby and apply - we just drop the biggest null values per group:
def null_filter(d):
if len(d) > 1:
return d.null < d.null.max()
return d.null == d.null
mask = df.groupby("datasource").apply(null_filter).values
df.loc(mask).drop("null", 1)
Output:
city country datasource
0 Shallotte US 1
2 austin US 2
3 Casselberry US 3
4 None AU 3
5 Springfield None 3

Answer 2

One of the way is based on None Count and removing rows of maximum None values i.e
#Count the None values across the row
df['Null'] = (df.values == 'None').sum(axis=1)
#Get the maximum of the count based on groupby
df['Max'] = df.groupby('datasource')['Null'].transform(max)
# Get the values are not equal to max and equal to zero and drop the columns
df = df[~((df['Max'] !=0) & (df['Max'] == df['Null']))].drop(['Null','Max'],axis=1)
Output :
datasource city country
0 1 Shallotte US
2 2 austin US
3 3 Casselberry US
4 3 None AU
5 3 Springfield None
Hope it helps