Is it possible to iterate through a single API call to a SharePoint list using Power Query? - rest

Question

Strange use case here. I have a list with approximately 18000 rows. I have specific lookup columns that show up as expected when I use the SharePoint Online list connector in Power BI. This method is horribly slow though. I stumbled upon a guide by a Power BI user Hoosier BI that outlines a way to get list items using REST API. When I use a REST API call, however, results return but the same columns I want are suspiciously absent.
I can hit the list in a web browser and see that when using https://mysharepointurl/mysharepointsite/_api/web/lists/GetByID('mylistid')/Items
all 18000 rows are accounted for but the columns I want are not there. This changes when I call the item number specifically and view field values as text
https://mysharepointurl/mysharepointsite/_api/web/lists/GetByID('mylistid')/Items(1)/FieldValuesAsText
All columns are available and expandable, including the lookup columns that were previously missing.
If using Power Query, is there a way to incrementally iterate through each item in my list and return results based on ViewFieldsAsText call?
If not, am I missing something simple here? I've taken a look at the list in SharePoint, changed the default view to include all columns I want, made sure those columns are indexed, and more that I can't recall off the top of my head.
Thoughts?
For reference, this is the M query I was using to retrieve all rows initially:
sitename = "", // if a subsite use "Site/SubSite"
listname = "BigList",
baseurl = "https:///sites/"
& sitename
& "/_api/web/lists/GetByTitle('"
& listname
& "')/",
itemcount = Json.Document(
Web.Contents(baseurl & "ItemCount", [Headers = [Accept = "application/json"]])
)[value],
skiplist = List.Numbers(0, Number.RoundUp(itemcount / 5000), 5000),
#"Converted to Table" = Table.FromList(
skiplist,
Splitter.SplitByNothing(),
null,
null,
ExtraValues.Error
),
#"Renamed Columns" = Table.RenameColumns(#"Converted to Table", {{"Column1", "Skip"}}),
#"Changed Type" = Table.TransformColumnTypes(#"Renamed Columns", {{"Skip", type text}}),
fieldselect = "&$top=5000", // all fields with no expansion
//fieldselect = "&$top=5000&$select = Id,Title,Person,Date", // list desired fields (no expansion)
//fieldselect = "&$top=5000&$select=Id,Title,Choice,LookupColumn/Title,LookupColumn/Project,LookupColumn/ProjectStatus,Date,Person/LastName,Person/FirstName,Person/EMail&$expand=LookupColumn,Person",
Custom1 = Table.AddColumn(
#"Changed Type",
"Items",
each Json.Document(
Web.Contents(
baseurl & "/items?$skipToken=Paged=TRUE%26p_ID=" & [Skip] & fieldselect,
[Headers = [Accept = "application/json"]]
)
)
),
#"Expanded Items" = Table.ExpandRecordColumn(Custom1, "Items", {"value"}, {"value"}),
#"Expanded value" = Table.ExpandListColumn(#"Expanded Items", "value")
in
#"Expanded value"```