Power Query - Subtract non-consecutive rows and create a new row - append

I am looking for help with the following problem:
My data set contains values for several variables (earnings, opex, volumes) by region and by country. There is a region total for each variable, but not every country is included in the data set, and as a result the sum of the countries does not add up to the region total. I would like to create a new calculated line "All Other Countries" for each variable that equals the sum of values for all non-included countries. The calculation is e.g.
Earnings All Other = Earnings Europe - Earnings Germany - Earnings UK - Earnings France.
The solutions I found so far were typically creating new columns for subtractions, and often based on some time-based order that used an index to link a previous row / period with a new row / period. That would not work for my problem, because I want to append the missing country rows to my table and my countries are not sorted.
Below a sample table as the data is now
enter image description here
enter image description here
and below the new rows I'd like to add / create
enter image description here
enter image description here

This seems to work in PowerQuery. Next time please paste machine readable numbers
Unpivot
I assume that when Region=BU, thats the total. So using a custom column to test that
Filter to get the non-totals and Group. That gives me the sum to subtract
Filter for the total. Merge in the non-total as a column and subtract the values. Thats the leftover. Change the BU column to say Other and rename some columns. Append to original data
Re-pivot
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Unpivoted Other Columns" = Table.UnpivotOtherColumns(Source, {"Region", "BU", "Analysis", "IS GRP", "Unit"}, "Attribute", "Value"),
#"Added Custom" = Table.AddColumn(#"Unpivoted Other Columns", "TopLevel", each if [Region]=[BU] then 1 else 0),
#"Filtered Rows" = Table.SelectRows(#"Added Custom", each ([TopLevel] = 0)),
#"Grouped Rows" = Table.Group(#"Filtered Rows", {"Region", "Analysis", "IS GRP", "Unit", "Attribute"}, {{"Value", each List.Sum([Value]), type number}}),
#"Filtered Rows2" = Table.SelectRows(#"Added Custom", each ([TopLevel] = 1)),
#"Merged Queries" = Table.NestedJoin(#"Filtered Rows2",{"Region", "Analysis", "IS GRP", "Unit", "Attribute"},#"Grouped Rows",{"Region", "Analysis", "IS GRP", "Unit", "Attribute"},"Filtered Rows2",JoinKind.LeftOuter),
#"Expanded Filtered Rows2" = Table.ExpandTableColumn(#"Merged Queries", "Filtered Rows2", {"Value"}, {"Value.1"}),
#"Added Custom1" = Table.AddColumn(#"Expanded Filtered Rows2", "Other.Value", each [Value]-[Value.1]),
#"ChangeNametoOther" = Table.TransformColumns(#"Added Custom1",{{"BU", each "Other", type text}}),
#"Removed Columns" = Table.RemoveColumns(#"ChangeNametoOther",{"Value", "TopLevel", "Value.1"}),
#"Renamed Columns" = #"Unpivoted Other Columns" & Table.RenameColumns(#"Removed Columns",{{"Other.Value", "Value"}}),
#"Pivoted Column" = Table.Pivot(#"Renamed Columns", List.Distinct(#"Renamed Columns"[Attribute]), "Attribute", "Value", List.Sum)
in #"Pivoted Column"

Related

Is it possible to iterate through a single API call to a SharePoint list using Power Query?

Strange use case here. I have a list with approximately 18000 rows. I have specific lookup columns that show up as expected when I use the SharePoint Online list connector in Power BI. This method is horribly slow though. I stumbled upon a guide by a Power BI user Hoosier BI that outlines a way to get list items using REST API. When I use a REST API call, however, results return but the same columns I want are suspiciously absent.
I can hit the list in a web browser and see that when using https://mysharepointurl/mysharepointsite/_api/web/lists/GetByID('mylistid')/Items
all 18000 rows are accounted for but the columns I want are not there. This changes when I call the item number specifically and view field values as text
https://mysharepointurl/mysharepointsite/_api/web/lists/GetByID('mylistid')/Items(1)/FieldValuesAsText
All columns are available and expandable, including the lookup columns that were previously missing.
If using Power Query, is there a way to incrementally iterate through each item in my list and return results based on ViewFieldsAsText call?
If not, am I missing something simple here? I've taken a look at the list in SharePoint, changed the default view to include all columns I want, made sure those columns are indexed, and more that I can't recall off the top of my head.
Thoughts?
For reference, this is the M query I was using to retrieve all rows initially:
sitename = "", // if a subsite use "Site/SubSite"
listname = "BigList",
baseurl = "https:///sites/"
& sitename
& "/_api/web/lists/GetByTitle('"
& listname
& "')/",
itemcount = Json.Document(
Web.Contents(baseurl & "ItemCount", [Headers = [Accept = "application/json"]])
)[value],
skiplist = List.Numbers(0, Number.RoundUp(itemcount / 5000), 5000),
#"Converted to Table" = Table.FromList(
skiplist,
Splitter.SplitByNothing(),
null,
null,
ExtraValues.Error
),
#"Renamed Columns" = Table.RenameColumns(#"Converted to Table", {{"Column1", "Skip"}}),
#"Changed Type" = Table.TransformColumnTypes(#"Renamed Columns", {{"Skip", type text}}),
fieldselect = "&$top=5000", // all fields with no expansion
//fieldselect = "&$top=5000&$select = Id,Title,Person,Date", // list desired fields (no expansion)
//fieldselect = "&$top=5000&$select=Id,Title,Choice,LookupColumn/Title,LookupColumn/Project,LookupColumn/ProjectStatus,Date,Person/LastName,Person/FirstName,Person/EMail&$expand=LookupColumn,Person",
Custom1 = Table.AddColumn(
#"Changed Type",
"Items",
each Json.Document(
Web.Contents(
baseurl & "/items?$skipToken=Paged=TRUE%26p_ID=" & [Skip] & fieldselect,
[Headers = [Accept = "application/json"]]
)
)
),
#"Expanded Items" = Table.ExpandRecordColumn(Custom1, "Items", {"value"}, {"value"}),
#"Expanded value" = Table.ExpandListColumn(#"Expanded Items", "value")
in
#"Expanded value"```

Power Query: Create New Columns from Nested Table in Grouped Column

I have a Table A with two columns: a unique ID and a column with a nested table with two rows and one column of interest for each unique ID. I want to add two new columns (Min, Max) to Table A that have the minimum and the maximum value for the column of interested broken out into those two new columns Min, Max. How would I do that?
I need to replicate what is done with this in DAX:
NewColumn_MIN = CALCULATE(
MIN(
Table[Column_A]),
FILTER(Table,Table[id]=EARLIER(Table[id]) ) )
If your set up looks like this
then you could have added the Min columns row-by-row with Add Column ... Custom Column... using formula
= List.Min([Custom][Column1])
and added the Max columns with Add Column ... Custom Column... using formula
= List.Max([Custom][Column1])
full code:
#"Added Custom1" = Table.AddColumn(TableA, "Min", each List.Min([Custom][Column1])),
#"Added Custom2" = Table.AddColumn(#"Added Custom1", "Max", each List.Max([Custom][Column1]))
in #"Added Custom2"
If you are trying to group on the unique ID and grab Min/Max for that ID across all the nested tables for that ID, without changing the data, you could try the below, which is expand, group, then merge
#"Expanded Custom" = Table.ExpandTableColumn(TableA, "Custom", {"Column1"}, {"Column1"}),
#"Grouped Rows" = Table.Group(#"Expanded Custom", {"ID"}, {{"Min", each List.Min([Column1]), type number}, {"Max", each List.Max([Column1]), type number}}),
#"Merged Queries" = Table.NestedJoin(#"Added Custom", {"ID"}, #"Grouped Rows", {"ID"}, "Table2", JoinKind.LeftOuter),
#"Expanded Table2" = Table.ExpandTableColumn(#"Merged Queries", "Table2", {"Min", "Max"}, {"Min", "Max"})
in #"Expanded Table2"

All combinations of two table in Powerquery

This is a question about Power Query/ M.
I have two tables: one with unique objects and one table with two columns (year and €).
I want to merge the two tables to one table with rows for all the object and years.
How can I do that?
Picture about input and expected result
In Table2 add column ... custom column ... using formula:
= Table1
Use arrows atop new column to [x] select and expand
That will give you every combination of Table1 and Table2
let Source = Excel.CurrentWorkbook(){[Name="Table2"]}[Content],
#"Added Custom" = Table.AddColumn(Source, "Custom", each Table1),
#"Expanded Custom" = Table.ExpandTableColumn(#"Added Custom", "Custom", {"Column1"}, {"Column1.1"})
in #"Expanded Custom"

PowerBI-How to use a date table in a dataset with date columns or change a shape map data selecting another columns in the same table

I have added a CSV file in PowerBi with this structure:
In each mau_audience column I have values for different dates for each country in the table.
In the report I have a shape map like this:
I'm trying to change the data on this map inside the report, selecting each mau_audience column, I am a beginner in PowerBi, and I have already tried to create a date table for this table without success.
I expect to be able to select a different mau_audience column to visualize the different values in the map.
Any help or guidance will be very appreciated.
Your source data has no date value. Each column is considered separately.
You can edit your query to unpivot your source data, and calculate a date from the header names:
let
Source = Csv.Document(File.Contents("C:\temp\south_america_data.csv"),[Delimiter=",", Columns=10]),
ReportYear = "2018",
#"Promoted Headers" = Table.PromoteHeaders(Source, [PromoteAllScalars=true]),
#"Unpivoted Other Columns" = Table.UnpivotOtherColumns(#"Promoted Headers", {"admin"}, "Attribute", "Value"),
#"Changed Type" = Table.TransformColumnTypes(#"Unpivoted Other Columns",{{"Value", type number}}),
#"Split Column by Delimiter" = Table.SplitColumn(#"Changed Type", "Attribute", Splitter.SplitTextByEachDelimiter({"_"}, QuoteStyle.Csv, true), {"Field", "Date"}),
#"Split Column by Position" = Table.SplitColumn(#"Split Column by Delimiter", "Date", Splitter.SplitTextByPositions({0, 3}, false), {"Month Name", "Day"}),
#"Inserted Merged Column" = Table.AddColumn(#"Split Column by Position", "Date", each Date.FromText(Text.Combine({[Day], [Month Name], ReportYear}, " ")), type date),
#"Removed Columns" = Table.RemoveColumns(#"Inserted Merged Column",{"Month Name", "Day"}),
#"Pivoted Column" = Table.Pivot(#"Removed Columns", List.Distinct(#"Removed Columns"[Field]), "Field", "Value", List.Sum)
in
#"Pivoted Column"
Now you can create a relationship to your Date table, and build a measure based on the single mau_audience column.

Merging Columns with null values PowerQuery

I have:
Name Value
A null
B null
null 5
null 10
and I need:
Name Value
A 5
B 10
Thanks a lot for providing a solution.
I had asked a similar question, to which #MarcelBeug provided a very helpful response; which, in turn, I'm using as the basis for my answer to you for your specific table.
This requires you to use Power Query (Power BI's query editor).
For your situation, I...
added a column named "Group", with the word "Group" in every one of its rows
then I used "Group By" on that new Group column, using sum aggregation for both the Name and Value columns
then I edited the code that was generated in step 2...changing the occurences of List.Sum to List.RemoveNulls
then I added a column with an embedded table from the two lists that resulted from step 3
then I deleted all columns other than the Tabled column
then I expanded the Tabled column, which gave me this:
Here's the M code:
let
Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Name", type text}, {"Value", Int64.Type}}),
#"Added Custom" = Table.AddColumn(#"Changed Type", "Group", each "Group"),
#"Grouped Rows" = Table.Group(#"Added Custom", {"Group"}, {{"NameList", each List.RemoveNulls([Name]), type text}, {"ValueList", each List.RemoveNulls([Value]), type number}}),
#"Added Custom.1" = Table.AddColumn(#"Grouped Rows", "Tabled", each Table.FromColumns({[NameList],[ValueList]},{"Name","Value"})),
#"Removed Other Columns" = Table.SelectColumns(#"Added Custom.1",{"Tabled"}),
#"Expanded Tabled" = Table.ExpandTableColumn(#"Removed Other Columns", "Tabled", {"Name", "Value"}, {"Name", "Value"})
in
#"Expanded Tabled"