I have 2 dataset as follows:
input province municipality year str5 muni_name population
1 1 2000 AAA 1000
1 2 2000 AAB 5000
2 1 2000 AAC 1500
2 2 2000 AAA 3000
3 1 2000 AAA 5600
end
input province municipality year str5 muni_name population
1 1 2010 AAA 2000
1 2 2010 AAB 6000
10 5 2010 AAC 3500
10 6 2010 AAA 7000
11 1 2010 AAA 6000
end
In each dataset, observations are uniquely identified by the combination of province and municipality. However, the value of the province or municipality may change depending on the year. Technically, province 2 municipality 1 in year 2000 is the same as province 10 municipality 5 in year 2010. We can tell this by the same muni_name. However, the challenge arises because muni_name does not uniquely identify the observation. In fact, there are municipality name that is found in multiple province (AAA is found in province 1, 2 and 3 in year 2000).
I'd like to have the final merged dataset as follows:
input id year str5 id_name population
1 1 2000 AAA 1000
1 1 2010 AAA 2000
1 2 2000 AAB 5000
1 2 2010 AAB 6000
10 5 2000 AAC 1500
10 5 2010 AAC 3500
10 6 2000 AAA 3000
10 6 2010 AAA 7000
11 1 2000 AAA 5600
11 1 2010 AAA 6000
end
I'd like to have the merged data to be uniquely identified by province, municipality and year. For the conflicting province municipality code, I'd like to replace the province municipality with the most recent year.
What is the best way to do this? My current idea is as follows: Ideally, I'd like to consider a match to be 'perfect match' if province, municipality and muni_name all coincides (AAA and AAB in province 1 corresponds to this). Among the observation that are not 'perfect match', I'd like to match by non-duplicate muni_name (let's call this semi-perfect match). In this example, AAC corresponds to this.
Lastly, for duplicated muni_name (AAA) that is not 'perfect match', I'll match based on the province of other perfect match or semi-perfect match. Note that AAC and one of the AAA is in the same province. Since AAC is matched to province 10 in 2010, AAA in the same province should also be matched to province 10 in 2010.
How can I code the following match strategy in Stata?
Related
I have a data structure issue. I have a problem where I need to roll up my data within tableau so that aggregated numbers do not skew in a certain manner.
Example of current data
ID Model_Number Value
123 fff 2
123 ggg 2
123 hhh 2
123 uuu 2
124 yyy 5
124 qqq 5
124 eee 5
Avg: NA 3.28
Ideal state of data and aggregation
ID Value
123 2
124 5
Avg 3.5
As you see since the data is at two different grains the aggregated number (avg) will be different. I would like to roll up my numbers to the distinct value of ID and then calculate my average which will result in a different (correct in my context) aggergated number.
Here is one calculated field that could help.
{ FIXED [ID] : AVG([Value]) }
This will give you the avg value by ID. You can then use a grand total(avg) to get the 3.5
I have the data in tableau like this
City Number of office Total Offices
Chicago 5 20
Houston 6 20
Dallas 4 20
I want to create a calculated field which has the difference of value [Total Offices] -(minus) Sum([Number of Office]). It should look like this
(20 - (5+6+4)) = 5
City Number of office Total Offices Difference(Calculated Field)
Chicago 5 20 5
Houston 6 20 5
Dallas 4 20 5
How can achieve this
simply this
[Total Offices] - {sum([Number of Office])}
BACKGROUND
I have three large tables (employee_info, driver_info, school_info) that I have joined together on common attributes using a series of LEFT OUTER JOIN operations. After each join, the resulting number of records increased slightly, indicating that there are duplicate IDs in the data. To try and find all of the duplicates in the IDs, I dumped the ID columns into a temp table like so:
Original Dump of ID Columns
first_name
last_name
employee_id
driver_id
school_id
Mickey
Mouse
1234
abcd
wxyz
Donald
Duck
2423
heca
qwer
Mary
Poppins
1111
acbe
aaaa
Wiley
Cayote
1234
strf
aaaa
Daffy
Duck
1256
acbe
pqrs
Bugs
Bunny
9999
strf
yxwv
Pink
Panther
2222
zzzz
zzaa
Michael
Archangel
0000
rstu
aaaa
In this overly simplified example, you will see that IDs 1234 (employee_id), strf (driver_id), and aaaa (school_id) are each duplicated at least once. I would like to add a count column for each of the ID columns, and populate them with the count for each ID used, like so:
ID Columns with Counts
first_name
last_name
employee_id
employee_id_count
driver_id
driver_id_count
school_id
school_id_count
Mickey
Mouse
1234
2
abcd
1
wxyz
1
Donald
Duck
2423
1
heca
1
qwer
1
Mary
Poppins
1111
1
acbe
1
aaaa
3
Wiley
Cayote
1234
2
strf
2
aaaa
3
Daffy
Duck
1256
1
acbe
1
pqrs
1
Bugs
Bunny
9999
1
strf
2
yxwv
1
Pink
Panther
2222
1
zzzz
1
zzaa
1
Michael
Archangel
0000
1
rstu
1
aaaa
3
You can see that IDs 1234 and strf each have 2 in the count, and aaaa has 3. After generating this table, my goal is to pull out all records where any of the counts are greater than 1, like so:
All Records with One or More Duplicate IDs
first_name
last_name
employee_id
employee_id_count
driver_id
driver_id_count
school_id
school_id_count
Mickey
Mouse
1234
2
abcd
1
wxyz
1
Mary
Poppins
1111
1
acbe
1
aaaa
3
Wiley
Cayote
1234
2
strf
2
aaaa
3
Bugs
Bunny
9999
1
strf
2
yxwv
1
Michael
Archangel
0000
1
rstu
1
aaaa
3
Real World Perspective
In my real-world work, the JOIN'd table contains 100 columns, 15 different ID fields and over 30,000 records, and the final table came out to be 28 more than the original. This may seem like a small amount, but each of the 28 represent a broken link that we must fix.
Is there a simple way to get the counts populated like in the second table above? I have been wrestling with this for hours already, and have not been able to make this work. I tried some aggregate functions, but they cannot be used in table UPDATE operations.
The COUNT function, when used as an analytic function, can do what you want here, e.g.
WITH cte AS (
SELECT *,
COUNT(employee_id) OVER (PARTITION BY employee_id) employee_id_count,
COUNT(driver_id) OVER (PARTITION BY driver_id) driver_id_count,
COUNT(school_id) OVER (PARTITION BY school_id) school_id_count
FROM yourTable
)
SELECT *
FROM cte
WHERE
employee_id_count > 1
driver_id_count > 1
school_id_count > 1;
I have two table 1000 of record given below.
My first table is USER table.
ID Name DateOfBirth
1 John 1980-11-20 00:00:00.000
2 Denial 1940-04-10 00:00:00.000
3 Binney 1995-12-25 00:00:00.000
4 Sara 1960-11-20 00:00:00.000
5 Poma 1980-11-20 00:00:00.000
6 Cameroon 1980-11-20 00:00:00.000
.....
.....
And my second table is CHANNEL_WATCH_DURATION_BY_USER
userid duration channelname
1 100 SAB
2 200 zee Tv
1 400 axn
2 0 star 1
3 800 star 2
3 700 star 3
4 200 star 4
.....
.....
I need to write the POSTGRES SQL Query to display different age groups contain duration with each channel.
under 18 20-30 age 30-40 age chaneel
10 40 100 star 1
20 0 200 star 2
30 79 0 zee
40 80 30 axn
.....
.....
SELECT
SUM(IF(DATEDIFF(NOW(),DateOfBirth)<18,1,0)) AS under18,
SUM(IF(DATEDIFF(NOW(),DateOfBirth) BETWEEN 20 AND 30,1,0)) as 20_to_30_age,
SUM(IF(DATEDIFF(NOW(),DateOfBirth)BETWEEN 30 AND 40,1,0)) as 30_to_40_age,
channelname as chaneel from
USER a,CHANNEL_WATCH_DURATION_BY_USER b where a.ID=b.USERID GROUP BY channelname
I have created a Crystal report whose sample output is below:
C_name P_name Code P. Rev F. rev C.rev
ABC AAA ABC-1-1 100 1100 1100
ABC AAA ABC-1-2 200 1200 1200
ABC AAA ABC-1-3 300 1300 2300
XYZ BBB XYZ-1-1 200 1200 2200
XYZ BBB XYZ-1-2 150 1150 3150
DEF CCC DEF-5-1 400 1400 1400
DEF CCC DEF-5-6 100 1100 2100
DEF CCC DEF-5-9 200 1200 4200
DEF DDD DEF-8-11 300 1300 2300
DEF DDD DEF-8-12 400 1400 400
Now, I want to add up the values for max value of Code. For example, ABC have 3 codes out of which ABC-1-3 is the latest code. So I want to dsiplay one record for these 3 records and add up the revenue values for 3 records and display it in one row only. The final output should look like below:
ABC AAA ABC-1-3 600 3600 4600
XYZ BBB XYZ-1-2 350 2350 5350
DEF CCC DEF-5-9 700 3700 7700
DEF DDD DEF-8-12 700 2700 2700
Please help..
Thanks
you can insert Group and P_name is the Group field which you select to create your group then in Running Total Fields and create new field for each of P.Rev F.rev C.rev in Evaluate and Reset part click on change of group and chose Type of summary Sum and for the Code field do the same,just chose Type of summary maximum. now you can add new fields to your Group and see the result.