Power Query - creating a tracker for multiple studies - group-by

New to Power Query and trying to learn how I can use it more to optimize workflows.
Here, I have multiple tables (1 for each study), and am trying to create a tracker that can identify the subject, whether or not they are participating in a specific study, and other data (e.g., study ID). Each table has:
Unique subject identifier (which could be used to link them across studies)
Study-specific identifier
Other misc info (dates of test1, test2, etc)
I am trying to make something that groups the columns of all my tables by the unique subject identifier, and then could have columns for "Study1", "Study 2", "Study 3" with a simple Y/N to show whether or not they are participating in that study. If Y, then also show the study-specific identifier. However, I am stuck.
I've appended all 3 tables into 1 master table - which results in duplicates as a participant can be in multiple studies. When I try the "Group By" function, I group by "Unique subject identifier" with new columns Study1, Study2, Study3, etc and operation "All rows". When I expand these however, it creates multiple duplicate rows, which defeats the purpose of the "Group by" function.
Would appreciate any suggestions you may have.
Example
Each table has some variation of these columns:
Unique participant identifier
Unique Study identifier
Date of Test 1, etc
A
100
01-Apr-2022
B
101
02-Apr-2022
C
102
03-Apr-2022
Let's say Participant A is on study 1 only. Participant B is on Study 2 (study ID 201) and 3 (study ID 301). Participant C is on all 3 (study IDs 102, 202, and 302 respectively).
I am trying to make a table that will show:
Unique participant identifier
Study 1
Study 1 Identifier
Study 2
Study 2 Identifier
Study 3
Study 3 Identifier
A
Y
100
B
Y
101
Y
201
Y
301
C
Y
102
Y
302
Alongside the dates of tests (not shown, but same concept). These tables are updated as we go along, so power query would draw these data from the tables to create a "live" tracker.
The current code I have in the advanced editor is:
let
Source = Table.Combine({STUDY1, STUDY2, STUDY3}),
#"Grouped Rows" = Table.Group(#"Removed Columns", {"Unique participant identifier"}, {{"STUDY12", each _, type table}, {"STUDY22", each _, type table}, {"STUDY32", each _, type table}}),
#"Expanded STUDY12" = Table.ExpandTableColumn(#"Grouped Rows", "STUDY12", {"STUDY1-Study ID", "STUDY1-ABC Study ID", "STUDY1"}, {"STUDY12.STUDY1-Study ID", "STUDY12.STUDY1-ABC Study ID", "STUDY12.STUDY1"}),
#"Expanded STUDY22" = Table.ExpandTableColumn(#"Expanded STUDY12", "STUDY22", {"STUDY2-Study ID", "STUDY2-OC Study ID", "STUDY2"}, {"STUDY22.STUDY2-Study ID", "STUDY22.STUDY2-ABC Study ID", "STUDY22.STUDY2"}),
#"Expanded STUDY32" = Table.ExpandTableColumn(#"Expanded STUDY22", "STUDY32", {"STUDY3-Study ID", "STUDY3"}, {"STUDY32.STUDY3-Study ID", "STUDY32.STUDY3"}),
#"Reordered Columns" = Table.ReorderColumns(#"Expanded STUDY32",{" Unique participant identifier ", "STUDY12.STUDY1-Study ID", "STUDY12.STUDY1", "STUDY12.STUDY1-ABC Study ID", "STUDY22.STUDY2", "STUDY22.STUDY2-Study ID", "STUDY22.STUDY2-ABC Study ID", "STUDY32.STUDY3", "STUDY32.STUDY3-Study ID"})
in
#"Reordered Columns"
Done via the "Group by" function. This creates multiple rows for each participant though.

Here's one method. Your sample data is not that great. You also dont mention how to divide the Unique Study identifier into buckets. I assume it is just the integer portion of the number divided 100. This is a bit long to attempt to make the steps obvious
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Unique participant identifier", type text}, {"Unique Study identifier", Int64.Type}, {"Date of Test", type date}}),
// there needs to be some way to identify groups of studies. I am using the hundreds place to find that
#"Added Custom" = Table.AddColumn(#"Changed Type", "StudyNumber", each Number.RoundDown([Unique Study identifier]/100)),
// first table -- studies only
#"Removed Columns" = Table.RemoveColumns(#"Added Custom",{"Date of Test"}),
#"Pivoted Column" = Table.Pivot(Table.TransformColumnTypes(#"Removed Columns", {{"StudyNumber", type text}}, "en-US"), List.Distinct(Table.TransformColumnTypes(#"Removed Columns", {{"StudyNumber", type text}}, "en-US")[StudyNumber]), "StudyNumber", "Unique Study identifier", List.Sum),
// second tabe -- study dates only
#"Removed Columns2" = Table.RemoveColumns(#"Added Custom",{"Unique Study identifier"}),
#"Pivoted Column1" = Table.Pivot(Table.TransformColumnTypes(#"Removed Columns2", {{"StudyNumber", type text}}, "en-US"), List.Distinct(Table.TransformColumnTypes(#"Removed Columns2", {{"StudyNumber", type text}}, "en-US")[StudyNumber]), "StudyNumber", "Date of Test"),
RenamedColumns = Table.TransformColumnNames(#"Pivoted Column1", each if _="Unique participant identifier" then _ else _&" Date"),
#"Removed Columns1" = Table.RemoveColumns(RenamedColumns,{"Unique participant identifier"}),
// combine the tables
combined = Table.FromColumns(Table.ToColumns(#"Pivoted Column")&Table.ToColumns( #"Removed Columns1"),Table.ColumnNames(#"Pivoted Column") &Table.ColumnNames(#"Removed Columns1"))
in combined
code version for yes/no
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Unique participant identifier", type text}, {"Unique Study identifier", Int64.Type}, {"Date of Test", type date}}),
#"Added Custom" = Table.AddColumn(#"Changed Type", "StudyNumber", each Number.RoundDown([Unique Study identifier]/100)),
// first table -- studies only
#"Removed Columns" = Table.RemoveColumns(#"Added Custom",{"Date of Test"}),
#"Rounded Off" = Table.TransformColumns(#"Removed Columns",{{"StudyNumber", each Text.From(_) &" Identifier"}}),
Part1 = Table.Pivot(Table.TransformColumnTypes(#"Rounded Off", {{"StudyNumber", type text}}, "en-US"), List.Distinct(Table.TransformColumnTypes(#"Rounded Off", {{"StudyNumber", type text}}, "en-US")[StudyNumber]), "StudyNumber", "Unique Study identifier", List.Sum),
// 2nd table -- YN only
#"Change to Y" = Table.TransformColumns(#"Removed Columns",{{"Unique Study identifier",each "Y" }}),
#"Rounded Off1" = Table.TransformColumns(#"Change to Y",{{"StudyNumber", each "Study " &Text.From(_)}}),
Pivot = Table.Pivot(Table.TransformColumnTypes(#"Rounded Off1", {{"StudyNumber", type text}}, "en-US"), List.Distinct(Table.TransformColumnTypes(#"Rounded Off1", {{"StudyNumber", type text}}, "en-US")[StudyNumber]), "StudyNumber", "Unique Study identifier"),
Part2 = Table.RemoveColumns(Pivot,{"Unique participant identifier"}),
// 3rd table -- study dates only
#"Removed Columns2" = Table.RemoveColumns(#"Added Custom",{"Unique Study identifier"}),
#"Pivoted Column" = Table.Pivot(Table.TransformColumnTypes(#"Removed Columns2", {{"StudyNumber", type text}}, "en-US"), List.Distinct(Table.TransformColumnTypes(#"Removed Columns2", {{"StudyNumber", type text}}, "en-US")[StudyNumber]), "StudyNumber", "Date of Test"),
RenamedColumns3 = Table.TransformColumnNames(#"Pivoted Column", each if _="Unique participant identifier" then _ else _&" Date"),
Part3 = Table.RemoveColumns(RenamedColumns3,{"Unique participant identifier"}),
// combine the tables
combined = Table.FromColumns(Table.ToColumns(Part1)&Table.ToColumns( Part2)&Table.ToColumns( Part3),Table.ColumnNames(Part1) &Table.ColumnNames(Part2)&Table.ColumnNames(Part3))
in combined

Related

Power Query how to get listed rows from the table for the Min or Max Date?

Power Query how to get listed rows from the table for the Min or Max Date?
Here is the code that is currently not working
let Source = Excel.CurrentWorkbook(){[Name="Table3"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Project ID", type text}, {"Activity ID", type text}, {"Activity Name", type text}, {"System Locator-g", type text}, {"Milestone-g", type text}, {"Original Duration", Int64.Type}, {"Test Work Type-g", type text}, {"MXO_WONUM", type text}, {"Start", type any}, {"Finish", type any}, {"Activity Status", type text}, {"MXO_EQUIPTAGS", type any}, {"Discipline-g", type any}, {"Restraints", type any}, {"Empty row", type any}}),
#"Merged Queries" = Table.NestedJoin(#"Changed Type", {"MXO_WONUM"}, #"Append1 Loc Max Dump & Loc Max Dump", {"wonum"}, "Append1 Loc Max Dump & Loc Max Dump", JoinKind.LeftOuter),
#"Merged Queries1" = Table.NestedJoin(#"Renamed Columns", {"Milestone-g"}, #"Milestone Sequence", {"Milestone-g"}, "Milestone Sequence", JoinKind.LeftOuter),
#"Expanded Milestone Sequence" = Table.ExpandTableColumn(#"Merged Queries1", "Milestone Sequence", {"Milestone Sequence"}, {"Milestone Sequence.Milestone Sequence"}),
#"Renamed Columns1" = Table.RenameColumns(#"Expanded Milestone Sequence",{{"Milestone Sequence.Milestone Sequence", "Milestone Sequence"}}),
#"Merged Queries2" = Table.NestedJoin(#"Changed Type3", {"Component"}, #"CTM P6 Update Tab (2) - unpivoted", {"Component"}, "CTM P6 Update Tab (2) - unpivoted", JoinKind.LeftOuter),
#"Expanded CTM P6 Update Tab (2) - unpivoted" = Table.ExpandTableColumn(#"Merged Queries2", "CTM P6 Update Tab (2) - unpivoted", {"Test Status "}, {"CTM P6 Update Tab (2) - unpivoted.Test Status "}),
#"Renamed Columns2" = Table.RenameColumns(#"Expanded CTM P6 Update Tab (2) - unpivoted",{{"CTM P6 Update Tab (2) - unpivoted.Test Status ", "Test Status"}}),
#"Grouped Rows" = Table.Group(#"Renamed Columns2", {"Component", "Milestone-g", "Milestone Sequence", "Activity Status", "Activity ID", "Activity Name"}, {{"Grouped", each Table.FirstN(Table.Sort(#"Renamed Columns2",{"Grouped",Order.Ascending}),1), type table}})
in #"Grouped Rows"
Are you trying to get the rows that are either the min or max date of the Grouped column?
#"Filtered Rows" = Table.SelectRows(#"YourPriorStepNameGoesHere", each ([Grouped] = List.Min(#"YourPriorStepNameGoesHere"[Grouped]) or [Grouped] =List.Max(#"YourPriorStepNameGoesHere"[Grouped])))
Edit per question:
to do this by Component, best way is to group on Component then expand. I am assuming one row per date, otherwise I'd have to do it another way
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Component", type text}, {"Grouped", type date}}),
#"Grouped Rows" = Table.Group(#"Changed Type", {"Component"}, {{"data", each
Table.FirstN(Table.Sort(_,{{"Grouped", Order.Ascending}}),1) &
Table.LastN(Table.Sort(_,{{"Grouped", Order.Ascending}}),1)
, type table }}),
#"Expanded data" = Table.ExpandTableColumn(#"Grouped Rows", "data", {"Grouped"}, {"Grouped"})
in #"Expanded data"

How to create a measure that show the count on previous month (PowerBI)

Hi everyone,
I want to create a column in the matrix table to show the count of OK for previous month, I named the measure as Prev Month OK count as shown in the screenshot above. However, the output is not what I want based on what I tried. This is my formula:
Prev Month OK count = CALCULATE([OK count], DATEADD('Aggregate'[Intake],-1,MONTH))
Even I change the MONTH parameter to QUARTER in DATEADD, it doesn't work as well, there is only a value in the Total which I'm not sure what does the value means.
The screenshot above is the expected output that I want. Take note that [OK count] is also a measure.
Any help or advise will be greatly appreciated!
Screenshot for Date Column
You can do it in Power Query and create a new table using this following code-
let your table name is your_old_table_name
let you have these 2 column there - date and value
Now create a new table **your_new_table_name" as below-
let
Source = your_old_table_name,
#"Grouped Rows" = Table.Group(Source, {"date"}, {{"_sum", each List.Sum([value]), type nullable number}}),
#"Sorted Rows" = Table.Sort(#"Grouped Rows",{{"date", Order.Ascending}}),
#"Added Index" = Table.AddIndexColumn(#"Sorted Rows", "Index", 1, 1, Int64.Type),
#"Added Custom" = Table.AddColumn(#"Added Index", "Custom", each [Index] + 1),
#"Merged Queries" = Table.NestedJoin(#"Added Custom", {"Index"}, #"Added Custom", {"Custom"}, "Added Custom", JoinKind.LeftOuter),
#"Expanded Added Custom" = Table.ExpandTableColumn(#"Merged Queries", "Added Custom", {"_sum"}, {"Added Custom._sum"}),
#"Removed Columns" = Table.RemoveColumns(#"Expanded Added Custom",{"Index", "Custom"}),
#"Renamed Columns" = Table.RenameColumns(#"Removed Columns",{{"Added Custom._sum", "previous_date_sum"}})
in
#"Renamed Columns"
You will have an output similar as below-

Merge queries matching on dates and leave null if no match found

I have got two tables below (examples):
Table SFID
Sales Force ID
Type
Name
Assistant
From
To
123
ABC
Store A
Ben
01/04/2020
30/04/2020
123
ABC
Store A
Jen
01/05/2020
31/05/2020
123
ABC
Store A
Ben
01/06/2020
21/06/2020
126
DEF
Store B
Tim
01/04/2020
30/04/2020
126
DEF
Store B
Tim
01/04/2020
null
and
Table Activity
Transaction ID
Date
Sales Force ID
1
03/05/2020
123
2
03/06/2020
200
3
01/01/2021
123
4
02/01/2021
126
I want my end result to be
Transaction ID
Date
Sales Force ID
Type
Name
Assistant
1
03/05/2020
123
ABC
Store A
Jen
2
03/06/2020
200
null
null
null
3
01/01/2021
123
null
null
null
4
02/01/2021
126
DEF
Store B
Tim
To do this, the best solution was the one posted in here with some modifications (allow both To and From to be null). However, only the row on transaction ID 2 disappears because that Sales Force had already had assistant entries (they get wiped out on the filtered row step). I also tried the solution presented in here but it takes ages to load.
I would like to know if there was a way to guarantee all transactions appear without having to introduce lines to table SFID for periods they don't have assistants and without making the query a really slow one.
This is my code:
let
Source = Source,
#"Merged Queries" = Table.NestedJoin(Source,{"Sales Force ID"},SFID,{"SFID"},"SFID",JoinKind.LeftOuter),
#"Expanded SFID" = Table.ExpandTableColumn(#"Merged Queries", "SFID", {"Type", "Name", "Assistant", "From", "To"}, {"Type", "Name", "Assistant", "From", "To"}),
#"Changed Type" = Table.TransformColumnTypes(#"SFID",{{"Date", type date}, {"From", type date}, {"To", type date}}),
FilteredRows = Table.SelectRows(#"Changed Type", each ([Date] >= [From] and [Date] <= [To]) or ([Date] >= [From] and [To] = null)or ([From] = null and [To] = null)),
#"Removed Columns" = Table.RemoveColumns(FilteredRows,{"From", "To"})
in
#"Removed Columns"
So after almost 2 weeks with no response and an unjustified downgrade, I found a solution!
I created a query that would basically produce this table with the above mentioned code.
Transaction ID
Date
Assistant
1
01/01/2021
Jen
4
02/01/2021
Tim
let
Source = Source,
#"Merged Queries" = Table.NestedJoin(Source,{"Sales Force ID"},SFID,{"SFID"},"SFID",JoinKind.LeftOuter),
#"Expanded SFID" = Table.ExpandTableColumn(#"Merged Queries", "SFID", {"Type", "Name", "Assistant", "From", "To"}, {"Type", "Name", "Assistant", "From", "To"}),
#"Changed Type" = Table.TransformColumnTypes(#"Expanded SFID",{{"Date", type date}, {"From", type date}, {"To", type date}}),
FilteredRows = Table.SelectRows(#"Changed Type", each ([Date] >= [From] and [Date] <= [To]) or ([Date] >= [From] and [To] = null)),
#"Removed Columns" = Table.RemoveColumns(FilteredRows,{"From", "To"})
in
#"Removed Columns"
And left-joined it on my initial version of Table Activity
I kept the information of type Type and Name in a separate Query (as they don't change) and then again left-joined it on the Table Activity.
Final query looks like this:
let
Source = Source,
#"Merged Queries1" = Table.NestedJoin(Source,{"Sales Force Code"},Info,{"SFID"},"Info",JoinKind.LeftOuter),
#"Expanded Info" = Table.ExpandTableColumn(#"Merged Queries1", "Info", {"Type", "Name"}, {"Type", "Name"}),
#"Merged Queries" = Table.NestedJoin(#"Expanded Info",{"ID"},IDvsAssistant,{"ID"},"IDvsAssistant",JoinKind.LeftOuter),
#"Expanded IDvsAssistant" = Table.ExpandTableColumn(#"Merged Queries", "IDvsAssistant", {"Assistant"}, {"Assistant"})
in
#"Expanded IDvsAssistant"

Remove first column after "group by All rows" operationin Power Query

I have an issue when trying to group by first column Podrocje_dela (PHASE 1). I use Group by All rows where second column has a name Aktivnosti.
PHASE 1
as the result is in two column format (PHASE 2), but I would like only last column, so the one marked with blue pen should be removed.
PHASE 2
Why this is a problem? I need a first column items as column Names (red), but this names also appear as column values (blue) combined from 1st and 2nd column (PHASE 3), comma separated. I would need only second column values as shown in last image (PHASE 4).
PHASE 3
End result should be like is shown in image below (PHASE 4). This result is accomplished by .AfterDelimiter operation but as the number of different first column items (PHASE 1) can change it is not flexible enough.
PHASE 4
For the end, I need result as shown in PHASE 4 from PHASE 1. Maybe my approach is not the best and I would be happy for any suggestions.
My current code is
let
Source = Access.Database(File.Contents("\\fs-srv01\PROJEKTI\S-DELOVNI-PROCES\DELOVNE URE\MS_Access\Podatkovne_baze\Organizacija.accdb"), [CreateNavigationProperties=true]),
_tblPodrocja_dela = Source{[Schema="",Item="tblPodrocja_dela"]}[Data],
#"Removed Columns" = Table.RemoveColumns(_tblPodrocja_dela,{"tblPodrocjaDela_Aktivnosti"}),
#"Merged Queries" = Table.NestedJoin(#"Removed Columns",{"ID_Podrocje_dela"},tblPodrocjaDela_Aktivnosti_M2M,{"ID_Podrocje_dela"},"tblPodrocjaDela_Aktivnosti_M2M",JoinKind.FullOuter),
#"Expanded tblPodrocjaDela_Aktivnosti_M2M" = Table.ExpandTableColumn(#"Merged Queries", "tblPodrocjaDela_Aktivnosti_M2M", {"ID_Aktivnost"}, {"tblPodrocjaDela_Aktivnosti_M2M.ID_Aktivnost"}),
#"Merged Queries1" = Table.NestedJoin(#"Expanded tblPodrocjaDela_Aktivnosti_M2M",{"tblPodrocjaDela_Aktivnosti_M2M.ID_Aktivnost"},tblAktivnostiVsakodnevne,{"ID_aktivnost"},"tblAktivnosti",JoinKind.LeftOuter),
#"Expanded tblAktivnosti" = Table.ExpandTableColumn(#"Merged Queries1", "tblAktivnosti", {"Aktivnost"}, {"tblAktivnosti.Aktivnost"}),
#"Removed Columns1" = Table.RemoveColumns(#"Expanded tblAktivnosti",{"ID_Podrocje_dela", "tblPodrocjaDela_Aktivnosti_M2M.ID_Aktivnost"}),
#"Grouped Rows" = Table.Group(#"Removed Columns1", {"Podrocje_dela"}, {{"Aktivnosti", each _, type table}}),
toList = Table.TransformColumns(#"Grouped Rows", {"Aktivnosti", Table.ToList}),
output = #table(toList[Podrocje_dela],List.Zip(toList[Aktivnosti])),
#"Extracted Text After Delimiter" = Table.TransformColumns(output, {{"Finance", each Text.AfterDelimiter(_, ","), type text}, {"IT", each Text.AfterDelimiter(_, ","), type text}, {"Kadrovska", each Text.AfterDelimiter(_, ","), type text}, {"Komerciala", each Text.AfterDelimiter(_, ","), type text}, {"Podporne službe", each Text.AfterDelimiter(_, ","), type text}, {"Proizvodnja", each Text.AfterDelimiter(_, ","), type text}, {"Raziskave in razvoj", each Text.AfterDelimiter(_, ","), type text}, {"Vodstvo", each Text.AfterDelimiter(_, ","), type text}})
Thank you all!
If anybody has the same problem. I found a solution to my problem. To have only single column grouped I specified which column to be specified:
from
#"Grouped Rows" = Table.Group(#"Removed Columns1", {"Podrocje_dela"}, {{"Aktivnosti", each _, type table}}),
to
#"Grouped Rows" = Table.Group(#"Removed Columns1", {"Podrocje_dela"}, {{"Aktivnosti", each _[Aktivnosti], type table}}),
The Power Query code with Ascending sort where nulls are at the end:
let
Source = Access.Database(File.Contents("\\fs-srv01\PROJEKTI\S-DELOVNI-PROCES\DELOVNE URE\MS_Access\Podatkovne_baze\Organizacija.accdb"), [CreateNavigationProperties=true]),
_tblPodrocjaDela_Aktivnosti = Source{[Schema="",Item="tblPodrocjaDela_Aktivnosti"]}[Data],
#"Removed Columns" = Table.RemoveColumns(_tblPodrocjaDela_Aktivnosti,{"ID_Podrocje_dela", "ID_Aktivnost"}),
#"Reordered Columns" = Table.ReorderColumns(#"Removed Columns",{"tblPodrocja_dela", "tblAktivnosti"}),
#"Expanded tblPodrocja_dela" = Table.ExpandRecordColumn(#"Reordered Columns", "tblPodrocja_dela", {"Podrocje_dela"}, {"Podrocje_dela"}),
#"Expanded tblAktivnosti" = Table.ExpandRecordColumn(#"Expanded tblPodrocja_dela", "tblAktivnosti", {"Aktivnost", "Vsakodnevna"}, {"Aktivnosti", "tblAktivnosti.Vsakodnevna"}),
#"Filtered Rows" = Table.SelectRows(#"Expanded tblAktivnosti", each true),
#"Sorted Rows" = Table.Sort(#"Filtered Rows",{{"Aktivnosti", Order.Ascending}}),
#"Removed Columns1" = Table.RemoveColumns(#"Sorted Rows",{"tblAktivnosti.Vsakodnevna"}),
#"Grouped Rows" = Table.Group(#"Removed Columns1", {"Podrocje_dela"}, {{"Aktivnosti", each _[Aktivnosti], type table}}),
toList = Table.Sort(#"Grouped Rows",{{"Podrocje_dela", Order.Ascending}}),
MyColumnNames = toList[Podrocje_dela],
arrangedDataFromAccess = #table(MyColumnNames ,List.Zip(toList[Aktivnosti])),
SortAllColumns = Table.FromColumns(List.Generate(()=>
[x=0,y=Table.Column(arrangedDataFromAccess ,MyColumnNames{x})],
each [x] < List.Count(MyColumnNames),
each [x=[x]+1,y=Table.Column(arrangedDataFromAccess ,MyColumnNames{x})],
each List.Sort([y] ,(xx,yy)=>if xx&yy<> null then Value.Compare(xx ,yy) else Value.Compare(yy ,xx))),MyColumnNames),
CountItemsInColumns = List.Generate(()=>
[x=0,y=Table.Column(arrangedDataFromAccess ,MyColumnNames{x})],
each [x] < List.Count(MyColumnNames),
each [x=[x]+1,y=Table.Column(arrangedDataFromAccess ,MyColumnNames{x})],
each List.NonNullCount([y])),
CountItemsInColumnsTransposed = Table.Transpose(Table.FromList(CountItemsInColumns,Splitter.SplitByNothing()), MyColumnNames),
finalDataFromAccess = Table.Combine({CountItemsInColumnsTransposed, SortAllColumns}) in finalDataFromAccess

Show Visual Studio Online Work Item Cycle Times

Ideally, I would like to calculate the cycle time of the whole value stream represented by Product Backlog Items in Visual Studio Online via Power BI. (I would then love to get time per status, i.e. how long was it sat in the "New" state, or how long was it stuck in the "Committed" state.)
To start with, I'm interested in using the calculated value that represents the time between the created date and the closed date of the product backlog item. After this, I'm keen on getting the distribution of values found.
product backlog item time between created timestamp to closed timestamp
Would be a starting point, but this shows "queued duration minutes by title", of course.
Most other attempts result in:
Can’t determine relationships between the fields
Is there a way to get some indication of cycle times?
Assuming each rows has a created date and closed date, you could use the power query duration function to get the difference between the two dates.
https://msdn.microsoft.com/en-us/library/mt296613.aspx
Update: After looking into it (see my comments below) I realize that it takes some Olympic-level power query skills to build the query you need. So I created it for you :). The below works in Power BI desktop, but doesn't refresh in PowerBI.com for some reason (I'm checking into that). To make this work, you need to put your VSO account and the Query ID in the ConnectionInfo record.
Recommend setting up an alternate credential rather than using BASIC auth.
You can get a QueryID by saving a shared query to your VSO and then using curl to retrieve the id using the query:
//Replace YOURACCOUNT and YOURPROJECT with correct values below
curl -u username:password https://YOURACCOUNT.visualstudio.com/DefaultCollection/YOURPROJECT/_apis/wit/queries?$depth=1&api-version=1.0
Then you can use the following query (create a blank query and paste it into the advanced editor in Power BI Desktop). You might need to adjust some of the steps starting at "workitems =" since your query might have different fields than mine did. If you return created, resolved and closed dates, the duration calculations I used should work for you. You'll set the authentication to "Basic" when challenged in Power BI Desktop and supply the required credentials.
let
//TODO: replace YOURACCOUNT and YOURQUERYID below
ConnectionInfo = [account = "YOURACCOUNT.visualstudio.com", queryID="YOURQUERYID"],
account = Record.FieldValues(ConnectionInfo){0},
rootQuery = let
queryID = Record.FieldValues(ConnectionInfo){1},
query = "https://" & account & "/DefaultCollection/PowerBIClients/_apis/wit/wiql/" & queryID,
Source = Json.Document(Web.Contents(query))
in
Source,
#"INT-columns" = let
Source = rootQuery,
columns = Source[columns],
#"Converted to Table" = Table.FromList(columns, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Expanded Column1" = Table.ExpandRecordColumn(#"Converted to Table", "Column1", {"referenceName", "name", "url"}, {"referenceName", "name", "url"})
in
#"Expanded Column1",
#"RAW-workitems" = let
Source = rootQuery,
workItems = Source[workItems],
#"Converted to Table" = Table.FromList(workItems, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Expanded Column1" = Table.ExpandRecordColumn(#"Converted to Table", "Column1", {"id", "url"}, {"id", "url"})
in
#"Expanded Column1",
#"INT-columnClause" = let
Source = #"INT-columns",
colNames = Table.RemoveColumns(Source,{"name", "url"}),
list = Table.ToList(colNames),
joined = Text.Combine(list, ",")
in
joined,
#"INT-workitemsToGet" = let
Source = #"RAW-workitems",
col = Table.RemoveColumns(Source,{"url"}),
#"Changed Type" = Table.TransformColumnTypes(col,{{"id", type text}}),
list = Table.ToList(#"Changed Type"),
l = List.Count(list),
limits = List.Generate(()=>0, each _ < l, each _ + 100),
#"Converted to Table" = Table.FromList(limits, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Renamed Columns" = Table.RenameColumns(#"Converted to Table",{{"Column1", "iterations"}}),
#"Added Custom" = Table.AddColumn(#"Renamed Columns", "itemsToGet", each List.Range(list, [iterations], 100)),
#"Added Custom1" = Table.AddColumn(#"Added Custom", "itemsToGetString", each Text.Combine([itemsToGet], ",")),
#"Removed Columns" = Table.RemoveColumns(#"Added Custom1",{"iterations", "itemsToGet"})
in
#"Removed Columns",
#"INT-workitemRequests" = let
Source = #"INT-workitemsToGet",
#"Added Custom" = Table.AddColumn(Source, "requests", each "https://" & account & "/DefaultCollection/_apis/wit/workitems?ids=" & [itemsToGetString] & "&fields=" & #"INT-columnClause"),
#"Removed Columns" = Table.RemoveColumns(#"Added Custom",{"itemsToGetString"})
in
#"Removed Columns",
workitems = let
#"INT-workitemsRequests (2)" = let
Source = #"INT-workitemRequests",
results = Table.AddColumn(Source, "Results", each Json.Document(Web.Contents([requests]))),
out = 1
in
results,
#"Removed Columns" = Table.RemoveColumns(#"INT-workitemsRequests (2)",{"requests"}),
#"Expanded Results" = Table.ExpandRecordColumn(#"Removed Columns", "Results", {"count", "value"}, {"Results.count", "Results.value"}),
#"Expanded Results.value" = Table.ExpandListColumn(#"Expanded Results", "Results.value"),
#"Expanded Results.value1" = Table.ExpandRecordColumn(#"Expanded Results.value", "Results.value", {"id", "fields", "url"}, {"Results.value.id", "Results.value.fields", "Results.value.url"}),
#"Expanded Results.value.fields" = Table.ExpandRecordColumn(#"Expanded Results.value1", "Results.value.fields", {"System.Id", "System.WorkItemType", "System.State", "System.AssignedTo", "System.CreatedDate", "System.Title", "Microsoft.VSTS.Common.ResolvedDate", "Microsoft.VSTS.Common.ClosedDate"}, {"System.Id", "System.WorkItemType", "System.State", "System.AssignedTo", "System.CreatedDate", "System.Title", "Microsoft.VSTS.Common.ResolvedDate", "Microsoft.VSTS.Common.ClosedDate"}),
#"Removed Columns1" = Table.RemoveColumns(#"Expanded Results.value.fields",{"Results.count", "Results.value.id"}),
#"Renamed Columns" = Table.RenameColumns(#"Removed Columns1",{{"Results.value.url", "WorkItemUrl"}}),
#"Changed Type" = Table.TransformColumnTypes(#"Renamed Columns",{{"Microsoft.VSTS.Common.ResolvedDate", type datetime}, {"Microsoft.VSTS.Common.ClosedDate", type datetime}, {"System.CreatedDate", type datetime}}),
#"Added Custom" = Table.AddColumn(#"Changed Type", "DurationToResolved", each Duration.TotalDays([Microsoft.VSTS.Common.ResolvedDate] - [System.CreatedDate])),
#"Added Custom1" = Table.AddColumn(#"Added Custom", "DurationToClosed", each Duration.TotalDays([Microsoft.VSTS.Common.ClosedDate] - [System.CreatedDate])),
#"Changed Type1" = Table.TransformColumnTypes(#"Added Custom1",{{"DurationToResolved", type number}, {"DurationToClosed", type number}})
in
#"Changed Type1"
in workitems