Getting data from a website using Invoke-WebRequest - powershell

I have a website http://www.rnbtop99.com/chart which I am trying to capture the list of songs, by artist and title.
I have the this simple PowerShell script as a start:
$site = Invoke-WebRequest -Uri 'http://www.rnbtop99.com/chart'
Start-Sleep -Seconds 5
However, I am confused of all the different Tag/Class/ID, how to match them up on the site to using PowerShell's functions and arrangements that I need to use as the various options to capture the data, e.g.
$site.ParsedHtml.body.getElementsByTagName('div')
$site.ParsedHtml.body.getElementsByClassName('?')
Intellisense tells me I can use these options above, Tag/ClassName etc.
In Chrome using the inspect option it says for the artist - class="artist-name ng-binding" however if I run $site.ParsedHtml.body.getElementsByClassName('artist-name ng-binding') then it returns nothing.
Would be good to know also for future the best way for any site, where I can use either the Edge/Chrome the inspect option to get the correct element ID/Tag/ClassName then match up with the correct options to use in PowerShell to capture any data e.g. Document/ParsedHtml/All/Body/InnerHTML/InnerText etc etc.

try this :
$r=iwr http://www.rnbtop99.com/api/chart/charts.json
$j=$r.Content |convertfrom-json
$j.Previous.Tracks |%{"$($_.track.title) BY $($_.track.artist.name)"}

This is my final script pieced together with the help of all in this post. Just 4 lines, very nice. Small, but very effective.
$r=Invoke-WebRequest "http://www.rnbtop99.com/api/chart/charts.json"
#Invoke-WebRequest seems to treat application/json as a byte stream
$j=-join($r.Content-as[char[]]) |convertfrom-json
$j.Previous.Tracks |%{"$($_.track.artist.name) - $($_.track.title)"}

Related

PowerShell AzureAD odata v3.0 filter

So I am trying to fetch all sign-in logs that fails a particular Conditional Access that have been set in Report-Only mode.
The cmdlet is in preview and is unable to fetch all logs and then filtering using piping and powershell alone, so I am trying to query with a filter instead.
I currently have this query that runs successfully and returns lots of SignIn logs, but the results does not contains CA's with the result of "reportOnlyFailure" so something is wrong:
Get-AzureADAuditSignInLogs -Filter "AppliedConditionalAccessPolicies/any(c:c/id eq 'xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxx' and c/Result eq 'reportOnlyFailure')"
I found your post, because I have the exact same problem.
My Powershell skills are pretty low but I may have found one problem, even if I have no idean how to fix it.
IsnĀ“t the part "and c/Result eq 'reportOnlyFailure'" searching for the result of all ConditionalAccessPolicies and maybe failing because of that?
Whould it be possible to do it like you would with a nested Where-Object? Something like this:
$($.AppliedConditionalAccessPolicies | Where-Object {$.id -eq 'XXX' -or $_.id -eq 'XXX'}).result -eq "reportOnlyFailure"
I dont know the full syntax for the filter but maybe you could replace
AppliedConditionalAccessPolicies/any(...
with something like
AppliedConditionalAccessPolicies/(id eq 'xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxx')(...
I hope maybe this is usefull or you already found a solution.
If you got a solution I would be very thankfull if you could post ist.
Have a nice day,
Christian

O365 Powershell | Breaking up a long list into two sets of 100

I am looking to create a rule in Office 365 applied to all of the members in our org.
I would like this rule to append a warning on all incoming email from outside the organization with the same Display Names as our users.
When I attempt to apply it to all of the users in our org I get an error stating that the rule is too long.
In order to solve that I pulled a group, but I am still about 1000 characters over the limit.
I would like to make two variables, that each hold one half of the list, created by this command:
(Get-DistibutionGroupMember -Identity email#contoso.com -ResultSize Unlimited).DisplayName
I have attempted to modify the ResultSize parameter, but what I would need is result 1-100 and then 100-200 from the same list.
Another caveat to this problem is that the list cannot be static. It is something that the script will have to update every time it is run.
There is a sub-string command that you can use on a particular username that I have utilized when I made something for AD, but I am not aware of any way to break up a list like this.
If anyone has any other ways to solve this issue I would be more than open to any suggestion.
Thanks for taking the time to read this!
There are many ways of doing it. I found it very readable.
My favorite one is this one:
$ObjectList = 1..1000
$Step = 100
$counter = [pscustomobject] #{ Value = 0 }
$ObjectListSplitted = $ObjectList | Group-Object -Property { math]::Floor($counter.Value++ / $step) }
Then if you want to show the third subset just use this format :
$ObjectListSplitted[3].Group
Have a look to this solution already explained.
As a note other languages are capable of slicing an array of object with a start, stop and a step, have a look here if you're curious.

powershell google sheets restmethod post

I'm working on a powershell script to get information from a google sheet, change data and then update the info in the google sheet. I have it working were i can get the data, but i'm having a issues posting the data back. Right now im only requesting one row as of now. it comes back as a array of array object in $file. To change the data i do $file[0][2]. This will change the data in column two to what ever i want.
What does the body need to look like? i have tried different things but keep getting a bad request error. My message body looks like this;
$body = #{
"values" = $File
}
EDIT: So i guess use you need to add the "?valueInputOption=RAW" to the end of the URL. So i'm able to change the data, but i still cant use a array to update the whole row. What is the best way to update the whole row?
Use spreadsheets.values.update to update values in a spreadsheet. Check the Writing guide docs for more info and code samples.
Did you try to convert the data you return to JSON format? Most API's (as far as i've seen them) mostly like there data back in JSON.
try this:
$data = $body | ConvertTo-Json
and return $data to google sheets.
i might be wrong but have only done this a few times too.

HTMLDocumentClass and getElementsByClassName not working

Last year I had powershell (v3) script that parsed HTML of one festival page (and generate XML for my Windows Phone app).
I also was asking a question about it here and it worked like a charm.
But when I run the script this year, it is not working. To be specific - the method getElemntsByClassName is not returning anything. I tried that method also on other web pages with no luck.
Here is my code from last year, that is not working now:
$tmpFile_bandInfo = "C:\band.txt"
Write-Host "Stahuji kapelu $($kap.Nazev) ..." -NoNewline
Invoke-WebRequest http://www.colours.cz/ucinkujici/the-asteroids-galaxy-tour/ -OutFile $tmpFile_bandInfo
$content = gc $tmpFile_bandInfo -Encoding utf8 -raw
$ParsedHtml = New-Object -com "HTMLFILE"
$ParsedHtml.IHTMLDocument2_write($content)
$ParsedHtml.Close()
$bodyK = $ParsedHtml.body
$bodyK.getElementsByClassName("body four column page") # this returns NULL
$page = $page.item(0)
$aside = $page.getElementsByTagName("aside").item(0)
$img = $aside.getElementsByTagName("img").item(0)
$imgPath = $img.src
this is code I used to workaround this:
$sec = $bodyK.getElementsByTagName("section") | ? ClassName -eq "body four column page"
# but now I have no innerHTML, only the lonely tag SECTION
# so I am walking through siblings
$img = $sec.nextSibling.nextSibling.nextSibling.getElementsByTagName("img").item(0)
$imgPath = $img.src
This works, but this seems silly solution to me.
Anyone knows what I am doing wrong?
I actually solved this problem by abandoning Invoke-WebRequest cmdlet and by adopting HtmlAgilityPack.
I transformed my former sequential HTML parsing into few XPath queries (everything stayed in powershell script). This solution is much more elegant and HtmlAgilityPack is real badass ;) It is really honour to work with project like this!
The issue is not a bug but rather that the return where you're seeing NULL is because it's actually a reference to a proxy HTMLFile COM call to the DOM model.
You can force this to operate and return the underlying strings by boxing it into an array #() as such:
#($mybody.getElementsByClassName("body four column page")).textContent
If you do a Select-Object on it, that also automatically happens and it will unravel it via COM and return it as a string
$mybody.getElementsByClassName("body four column page") | Select-Object -Property TextContent

Auto Complete User Input PowerShell 2.0

I have a large list of data (over 1000 different values) and I want the user to be able to select certain values from the list from a PowerShell console.
What is the easiest way from within the console to allow the user to quickly select values?
I would like to do something like tab completion or the ability to use the arrow keys to scroll through the values but I am not sure how to do either of these things.
Any advice would be greatly appreciated.
PowerShell tab completion can be extended to custom parameters and parameter values (in v3). However, this is a property of advanced functions. You can use the ValidateSetAttribute to do that.
Check the Technet help topic on advanced functions: http://technet.microsoft.com/en-us/library/hh847806.aspx
You can replace the tabexpansion (v2) and tabexpansion2 (v3) function in PowerShell to auto complete parameter values outside of advanced functions. You can get a basic definition of this in PowerShell v3 by running
Get-Content function:TabExpansion2
Here is an example of showing custom tab expansion function.
http://www.powershellmagazine.com/2012/11/29/using-custom-argument-completers-in-powershell-3-0/
But, if you want to the user to be able to auto complete values for a Read-Host kind of input, you need to write a proxy for Read-Host to achieve that.
You can, optionally, look at PowerTab module at http://powertab.codeplex.com/
For folks who are looking for a way to do this and are fortunate enough to be using PS v3 (and my apologies for all those required to stay with V2):
The easiest way to achieve this is using the "ValidateSet" option in your input parameters.
function Show-Hello {
param (
[ValidateSet("World", "Galaxy", "Universe")]
[String]$noun
)
$greetingString = "Hello, " + $noun + "!"
Write-Host "`t=>`t" $greetingString "`t<="
}
ValidateSet throws an error if a user attempts to use any other input:
Show-Hello "Solar System"
Show-Hello : Cannot validate argument on parameter 'noun'. The argument `
"Solar System" does not belong to the set "World,Galaxy,Universe" specified `
by the ValidateSet attribute. Supply an argument that is in the set and `
then try the command again.
It also adds tab-completion to your function for that parameter. And if it is the FIRST parameter for your function, you don't even have to type in "-noun" for the tab-complete to make suggestions for its value.