Power BI create column from nested Record values - bing-maps

I'm trying to create a new column from one of the records in a List type column. The value is the country that corresponds with the Latitude and Longitude fields. The information is retrieved from the Bing Map API, which got using Get Data from Web (following the tutorial here: https://sqldusty.com/2016/04/26/power-bi-and-the-bing-maps-api/).
Basically I need List.Record[1].address.countryRegion. Is it possible to make a column that holds this specific value without doing "Expand to new rows"? The issue is that some of the columns come back with France, and the number of rows increases to over 1000 but there should only be around 250.
Here is how I got to the point of having the column of Lists:
1. Get data from the web
2. Used Basic option and pasted working API request for a location with my bing maps key. Then click ok.
http://dev.virtualearth.net/REST/v1/Locations/point=40,-122?&key=BingMapsKey
3. Navigated to the Advanced editor in View > Advanced editor.
4. Made a function that uses Latitude and Longitude as input
let getCountry = (Latitude as text, Longitude as text) =>
let
Source = Json.Document(Web.Contents("http://dev.virtualearth.net/REST/v1/Locations/point="& Latitude &","& Longitude &"?&key=BingMapsKey"))
in
Source
in
getCountry
5. Renamed the function to GetCountry, then navigated to the desired table to add the column to (with Latitude and Longitude)
6. In the target table, Navigate to Add Column > Invoke Custom Function
7. Chose GetCountry from the list of functions, changed the type to column name and assigned the inputs to respective column names (latitude and longitude). Then clicked OK.
8. The column shows up on the right. I filtered out all columns besides 'resourceSets' because that has the address values.
EDIT I found a way to reduce the number of lists that are returned in the request, which is to only request the Country/Region as a query parameter:
http://dev.virtualearth.net/REST/v1/Locations/40,-122?key=BingMapsKey&includeEntityTypes=CountryRegion
This works for my needs for now, but maybe it's a good idea to keep this open to see if someone knows how to make a table from the nested table values? Thanks for all your help!

This sounds like an XY Problem to me. Let me try to clarify:
The Table.ExpandListColumn function expands records to multiple rows because there are indeed multiple rows of records returned from the API endpoint.
Unless you apply filter logic afterwards, there are no ways for the code to understand which row of records to choose.
There shouldn't be multiple rows of records returned from the API. (Find the countryRegion for a given (lat, long))
So after reading through the question, the real problem lies in the API endpoint you're using.
It should be
http://dev.virtualearth.net/REST/v1/Locations/40,-122?key=yourAPIKey
instead of
http://dev.virtualearth.net/REST/v1/Locations/point=40,-122?key=yourAPIKey
The point= is not needed. (Yes, the documentation is slightly confusing)
So you can update your GetCountry function as follows:
let getCountry = (Latitude as text, Longitude as text) =>
let
Source = Json.Document(Web.Contents("http://dev.virtualearth.net/REST/v1/Locations/"& Latitude &","& Longitude &"?key=yourAPIKey"))
in
Source
in
getCountry
(Side note: better not to expose your API Key to public :) )
As a result, there should only be one countryRegion for each place.
My query for your reference:
let
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45Wcs/PT89JLchJrVDSUTI21zMxMjIwMACydQ2NjPQMLEwMTM2VYnWilRwLgIoUPPPSMvMyS1IVfPLzCyA6jI0NzczN4DqMDQwtLJViYwE=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type text) meta [Serialized.Text = true]) in type table [Place = _t, Lat = _t, Long = _t]),
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Place", type text}}),
#"Invoked Custom Function" = Table.AddColumn(#"Changed Type", "GetCountry", each GetCountry([Lat], [Long])),
#"Expanded GetCountry" = Table.ExpandRecordColumn(#"Invoked Custom Function", "GetCountry", {"resourceSets"}, {"GetCountry.resourceSets"}),
#"Expanded GetCountry.resourceSets" = Table.ExpandListColumn(#"Expanded GetCountry", "GetCountry.resourceSets"),
#"Expanded GetCountry.resourceSets1" = Table.ExpandRecordColumn(#"Expanded GetCountry.resourceSets", "GetCountry.resourceSets", {"resources"}, {"resources"}),
#"Expanded resources" = Table.ExpandListColumn(#"Expanded GetCountry.resourceSets1", "resources"),
#"Expanded resources1" = Table.ExpandRecordColumn(#"Expanded resources", "resources", {"address"}, {"address"}),
#"Expanded address" = Table.ExpandRecordColumn(#"Expanded resources1", "address", {"countryRegion"}, {"countryRegion"})
in
#"Expanded address"
Hope it helps :)

did you use a Parameter to filter the row you want ?
Have a look there: Get only the data I want from a db but keep structure

Related

Is there a way to expand dynamically tables found in multiple columns using Power Query?

I have used the List.Accumulate() to merge mutliple tables. This is the output I've got in this simple example:
Now, I need a solution to expand all these with a formula, because in real - world I need to merge multiple tables that keep increasing in number (think Eurostat tables, for instance), and modifying the code manually wastes much time in these situations.
I have been trying to solve it, but it seems to me that the complexity of syntax easily becomes the major limitation here. For instance, If I make a new step where I nest in another List.Accumulate() the Table.ExpandTableColumns(), I need to pass inside a column name of an inner table, as a text. Fine, but to drill it down actually, I first need to pass a current column name in [] in each iteration - for instance, Column 1 - and it triggers an error if I store column names to a list because these are between "". I also experimented with TransformColumns() but didn't work either.
Does anyone know how to solve this problem whatever the approach?
See https://blog.crossjoin.co.uk/2014/05/21/expanding-all-columns-in-a-table-in-power-query/
which boils down to this function
let Source = (TableToExpand as table, optional ColumnNumber as number) =>
//https://blog.crossjoin.co.uk/2014/05/21/expanding-all-columns-in-a-table-in-power-query/
let ActualColumnNumber = if (ColumnNumber=null) then 0 else ColumnNumber,
ColumnName = Table.ColumnNames(TableToExpand){ActualColumnNumber},
ColumnContents = Table.Column(TableToExpand, ColumnName),
ColumnsToExpand = List.Distinct(List.Combine(List.Transform(ColumnContents, each if _ is table then Table.ColumnNames(_) else {}))),
NewColumnNames = List.Transform(ColumnsToExpand, each ColumnName & "." & _),
CanExpandCurrentColumn = List.Count(ColumnsToExpand)>0,
ExpandedTable = if CanExpandCurrentColumn then Table.ExpandTableColumn(TableToExpand, ColumnName, ColumnsToExpand, NewColumnNames) else TableToExpand,
NextColumnNumber = if CanExpandCurrentColumn then ActualColumnNumber else ActualColumnNumber+1,
OutputTable = if NextColumnNumber>(Table.ColumnCount(ExpandedTable)-1) then ExpandedTable else ExpandAll(ExpandedTable, NextColumnNumber)
in OutputTable
in Source
alternatively, unpivot all the table columns to get one column, then expand that value column
ColumnsToExpand = List.Distinct(List.Combine(List.Transform(Table.Column(#"PriorStepNameHere", "ValueColumnNameHere"), each if _ is table then Table.ColumnNames(_) else {}))),
#"Expanded ColumnNameHere" = Table.ExpandTableColumn(#"PriorStepNameHere", "ValueColumnNameHere",ColumnsToExpand ,ColumnsToExpand ),

Is it possible to iterate through a single API call to a SharePoint list using Power Query?

Strange use case here. I have a list with approximately 18000 rows. I have specific lookup columns that show up as expected when I use the SharePoint Online list connector in Power BI. This method is horribly slow though. I stumbled upon a guide by a Power BI user Hoosier BI that outlines a way to get list items using REST API. When I use a REST API call, however, results return but the same columns I want are suspiciously absent.
I can hit the list in a web browser and see that when using https://mysharepointurl/mysharepointsite/_api/web/lists/GetByID('mylistid')/Items
all 18000 rows are accounted for but the columns I want are not there. This changes when I call the item number specifically and view field values as text
https://mysharepointurl/mysharepointsite/_api/web/lists/GetByID('mylistid')/Items(1)/FieldValuesAsText
All columns are available and expandable, including the lookup columns that were previously missing.
If using Power Query, is there a way to incrementally iterate through each item in my list and return results based on ViewFieldsAsText call?
If not, am I missing something simple here? I've taken a look at the list in SharePoint, changed the default view to include all columns I want, made sure those columns are indexed, and more that I can't recall off the top of my head.
Thoughts?
For reference, this is the M query I was using to retrieve all rows initially:
sitename = "", // if a subsite use "Site/SubSite"
listname = "BigList",
baseurl = "https:///sites/"
& sitename
& "/_api/web/lists/GetByTitle('"
& listname
& "')/",
itemcount = Json.Document(
Web.Contents(baseurl & "ItemCount", [Headers = [Accept = "application/json"]])
)[value],
skiplist = List.Numbers(0, Number.RoundUp(itemcount / 5000), 5000),
#"Converted to Table" = Table.FromList(
skiplist,
Splitter.SplitByNothing(),
null,
null,
ExtraValues.Error
),
#"Renamed Columns" = Table.RenameColumns(#"Converted to Table", {{"Column1", "Skip"}}),
#"Changed Type" = Table.TransformColumnTypes(#"Renamed Columns", {{"Skip", type text}}),
fieldselect = "&$top=5000", // all fields with no expansion
//fieldselect = "&$top=5000&$select = Id,Title,Person,Date", // list desired fields (no expansion)
//fieldselect = "&$top=5000&$select=Id,Title,Choice,LookupColumn/Title,LookupColumn/Project,LookupColumn/ProjectStatus,Date,Person/LastName,Person/FirstName,Person/EMail&$expand=LookupColumn,Person",
Custom1 = Table.AddColumn(
#"Changed Type",
"Items",
each Json.Document(
Web.Contents(
baseurl & "/items?$skipToken=Paged=TRUE%26p_ID=" & [Skip] & fieldselect,
[Headers = [Accept = "application/json"]]
)
)
),
#"Expanded Items" = Table.ExpandRecordColumn(Custom1, "Items", {"value"}, {"value"}),
#"Expanded value" = Table.ExpandListColumn(#"Expanded Items", "value")
in
#"Expanded value"```

Azure Data Factory: Dataset Dynamic DB Table name not resolving in Data Wrangling Flow

I created a DataSet which points to a table in my database. The name of the table is set as dynamic content: #concat(dataset().db_prefix, '_Baseline_CIs'). This works when checking in the Dataset through 'Preview Data'. The table contents are shown.
BUT: When using the dataset in the Data Warngling Flow, the M-query fails with the following error:
Expression.Error: The key didn't match any rows in the table.
AdfDoc = Sql.Database("oedudigital.database.windows.net", "IntegratedEnvironments"),
InputTable = AdfDoc{[Schema = "dbo", Item = "undefined"]}[Data]
As you can see, the table name concatenation has returned 'undefined'. Is this a bug?
BR, Denis
If I understand it right you have the DataSet which is parameter , at least that was the case on my side . Under the AdFResouce you will see the dataset name . You will have to pass the table name as
AdfDoc{[Schema = "dbo", Item = "TableName"]}[Data]
and then it will bring in the records .

Accessing the ATTR field troubleshooting in Tableau?

I am working on Tableau server but I believe the problem I am facing does not correspond to tableau server specifically.
I am using two data sources ds1 and ds2 which are joined using the dimension Id . ds1 has a field city and ds2 has a field district. There is only 1 city corresponding to each Id but there can be multiple district corresponding to an Id .
I have created a calculated field Points in ds2 which is described in the code segment.
I have researched from different sites and blogs (including tableau support). I came to a close possible reason behind this and I might be wrong . The ATTR function which works on row level and identify if a row is unique then it outputs the dimension otherwise it outputs '*' . I think when I joined those two tables the district dimension in ds2 might have the '*' instead of actual district values, so it might not be able to compare the conditions in if statements of Point .
//Point//
IF [city] == "Delhi"
AND [district] == "Dist1"
AND [district] == "Dist2"
THEN "100 Section"
ELSEIF [city] == "Mumbai"
AND [district] == "Dist11"
AND [district] == "Dist12"
THEN "200 Section"
ELSE "Other Section"
END
When I insert data which satisfy the conditions in calculated field, it is going in Other section of the Point .I want it to go in desired section.
For instance
Id = 19
city = Delhi
district = Dist1
district = Dist2
district = Dist3
It should go in 100 Section but it is going in Other Section . What modifications should I do or add to make the Point work properly ?

Randomizing SQL data with data from table function

I trying to make a sql script that will randomize the city, state, and zip code of a "members" table. I have made a table function that returns a single row with columns "city", "state" and "zip" taken from another database at random (via a view). This ensures that I get a city, state, and zip that actually correlate to each other in the real world.
From there I am trying to do something like this:
update t
set
t.City = citystate.city,
t.State = citystate.state,
t.PostalCode = citystate.zip
from
(select
City,
State,
PostalCode from DATABASE.dbo.Member) t,
DATABASE.dbo.getRandomCityState() citystate
Problem is, this only calls my function once, and puts the same city, state, and zip into every row of the table. Is there some way to call my function once for every row in the table?
Use a CROSS APPLY
update t
set
t.City = citystate.city,
t.State = citystate.state,
t.PostalCode = citystate.zip
from
(select
City,
State,
PostalCode from DATABASE.dbo.Member) t
CROSS APPLY
DATABASE.dbo.getRandomCityState() citystate
Ok so thanks to one of my co-workers, we found a solution. It would seem that since the function didn't take any parameters, SQL Server decided that the result would never change. So we tricked the server into thinking it will be different every time by passing a parameter to the function that was different: the id of each row. Since we were passing a different parameter each time, it called the function for every row.
update t
set
t.City = citystate.city,
t.State = citystate.state,
t.PostalCode = citystate.zip
from
(select top 10
City,
State,
PostalCode from TrajectoryServicesTest.dbo.Member) t
cross apply SanitizePhi.dbo.getRandomCityState(t.MemberID) citystate
Kinda hacky, but it worked. Thanks to Joe for the help.