Im attempting to scrape a website.
When testing I can get the values I'm looking for by doing...
$WebResponse = Invoke-WebRequest -Uri https://www.livesquawk.com/latest-news
($WebResponse.ParsedHtml.DocumentElement.GetElementsByTagName('div') | Where { $_.ClassName -match 'latest_news_each_title'}).InnerText
($WebResponse.ParsedHtml.DocumentElement.GetElementsByTagName('div') | Where { $_.ClassName -match 'latest_news_each_time'}).InnerText
But when I try to put it into a two column return, I get errors and multiple lines of same data?
What am I doing wrong? I'm out of ideas. Thanks in advance.
$WebResponse = Invoke-WebRequest -Uri https://www.livesquawk.com/latest-news
$lists = $WebResponse.ParsedHtml.DocumentElement.GetElementsByTagName('div')
$nodes = $lists[0].childNodes
$r = $lists | % {
[pscustomobject]#{
Time = $_.getElementsByClassName('latest_news_each_time')[0].innerText.Substring(0) | Where-Object { $_ -match "\d+.*" } | foreach { $Matches.Values }
News = $_.getElementsByClassName('latest_news_each_title')[0].innerText
}
}
$R
Since they aren't really grouped together and you already have the logic to get each, you could two arrays and a simple for loop to build the objects.
$WebResponse = Invoke-WebRequest -Uri https://www.livesquawk.com/latest-news
$title = ($WebResponse.ParsedHtml.DocumentElement.GetElementsByTagName('div') | Where { $_.ClassName -match 'latest_news_each_title'}).InnerText
$time = ($WebResponse.ParsedHtml.DocumentElement.GetElementsByTagName('div') | Where { $_.ClassName -match 'latest_news_each_time'}).InnerText
for($i = 0; $i -le $title.count; $i++)
{
[PSCustomObject]#{
Time = $time[$i]
News = $title[$i]
}
}
Related
I am currently trying to create a foreach loop to take each value from a list and search for it in return data from an API. For some reason I am getting NULL data when I try to loop it.
$agentList = (Import-CSV -Path 'O:\Test\AgentList.txt')
$Params = #{
"Uri" = "https://10.245.55.88:8834/agents"
"Method" = "GET"
"Headers" = #{
"X-ApiKeys" = "accessKey=$($AccessKey); secretKey=$($SecretKey)"
}
}
$agentsNotInSC = Invoke-Restmethod #Params
$agentID = foreach ($agent in $agentList) {
$agentID = $agentsNotInSC.agents | Where-Object { $_.name -eq $agent }
$agentID.id
}
$agentID | Export-Csv -Path "O:\Test\IDAgent.CSV" -NoTypeInformation
trying to get all the return data and export it to a CSV
Essentially what I'm after is the results of rest API Gateways - Get Datasource Users but retaining the ID (in this example $Line.id from my imported CSV file).
The end result should be a CSV with the following fields -
ID, emailAddress, datasourceAccessRight, displayName, identifier, principalType
I'm new to PowerShell and surprised I got this far but can't figure out this final bit.
Cheers
$webclient=New-Object System.Net.WebClient
$webclient.Proxy.Credentials = [System.Net.CredentialCache]::DefaultNetworkCredentials
$Dir = "C:\pbi_pro_user_logs\"
Login-PowerBI
$GateWayFile = Import-CSV -Path "C:\pbi_pro_user_logs\Gateway_Detail.csv"
$Output = #()
foreach ($Line in $GateWayFile){
$Item = $Line.id
$url = "https://api.powerbi.com/v1.0/myorg/gateways/HIDDEN/datasources/"+$Item+"/users"
$Output += (Invoke-PowerBIRestMethod -Url $url -Method Get | ConvertFrom-Json)
}
$Result = $Output.value
$Result | Export-Csv $Dir"GateWay_users.csv" -NoTypeInformation
Try this, using a calculated property from Select-Object:
$GateWayFile = Import-CSV -Path "C:\pbi_pro_user_logs\Gateway_Detail.csv"
$Output = Foreach ($Line in $GateWayFile){
$url = "https://api.powerbi.com/v1.0/myorg/gateways/HIDDEN/datasources/"+$Line.id+"/users"
$Item = (Invoke-PowerBIRestMethod -Url $url -Method Get | ConvertFrom-Json)
# output all properties of the item, plus the ID:
$ItemWithID = $Item | Select *,#{l='Id';e={$line.id}}
Write-Output $ItemWithID
}
# This depends on how you want your csv structured, but for example:
$Result = $Output | Select Id,Value
Or, if Value is a whole object that ID should be assigned inside of, then change the selection lines:
$ItemWithID = $Item.Value | Select *,#{l='Id';e={$line.id}}
$Result = $Output
I want to export what I already filtered in ForEach-Object. The problem is that I can't export the filtered data.
I tried the following:
$getTapes.rows | Export-Csv C:\\123\\123456.txt but this has exported all the information without filter.
$getTapes = Invoke-RestMethod -Method GET -ContentType $content -Uri $Uri -Headers #{'Authorization' = $Authorization}
$today = Get-Date
$getTapes.rows | ForEach-Object {
$tape = $_;
if ( $tape.custom_fields.Ueberschreibschutz.value -ge $today ) {
Write-Host "Treffer ID=" $tape.asset_tag " Name=" $tape.name " SNR=" $tape.serial " Mediensatz=" $tape.custom_fields.Mediensatz.value
}
}
$getTapes.rows |export-Csv C:\\123\\123456.txt
I expect:
Treffer ID= 1 Name= 12 SNR= 12345 Mediensatz= M
Treffer ID= 2 Name= 32 SNR= 54321 Mediensatz= W
You should not use Write-Host to collect data. That's only to output pixels on the screen. Instead you should create a custom object you can use as you want later on ... like this:
$Result = $getTapes.rows | ForEach-Object {
if ( $_.custom_fields.Ueberschreibschutz.value -ge $today ) {
[PSCustomObject]#{
TrefferID = $_.asset_tag
Name = $_.name
SNR = $_.serial
Mediensatz = $_.custom_fields.Mediensatz.value
}
}
}
$Result | Export-Csv -Path C:\123\123456.csv -NoTypeInformation
Write-host do nothing except it shows you the result in the console, so it will not modify or delete the things you don't want in $getTapes.rows.
Instead you can define a variable $result and iterate over the $getTapes.rows using Foreach-Object, and add the result if it meets your if condition.
Try this:
$getTapes = Invoke-RestMethod -Method GET -ContentType $content -Uri $Uri -Headers #{'Authorization' = $Authorization}
$today = Get-Date
$getTapes.rows | ForEach-Object -begin {$result = "" } {
$tape = $_;
if ( $tape.custom_fields.Ueberschreibschutz.value -ge $today ) {
$result += "Treffer ID= $($tape.asset_tag) Name= $($tape.name) SNR= $($tape.serial) Mediensatz= $($tape.custom_fields.Mediensatz.value)`n"
}
} -end {$result | export-Csv C:\123\123456.txt}
I would like to change my PowerShell script to define a $spWeb value to a particular site straight away in a site collection, instead of going through the multiple foreach loops.
Is it possible?
Code sample:
$url = "http://testweb.com"
$siteCollection = Get-SPSite $url
$WebApp = $siteCollection.WebApplication
$url = $url+"/sites/site1"
foreach ($Site in $WebApp.Sites)
{
foreach($spWeb in $Site.AllWebs)
{
if ($spWeb.URL -eq $url)
{
//Skip previous foreach loops and declare $spWeb value with a fixed URL here
$users = $spWeb.SiteUserInfoList
$groups = $users.Items
$RList = $spWeb.Lists["Records"]
$RCount = 0
#GET Records
#$outputnewwebtext = "GET Records in $($spWeb.URL)"
#$outputnewwebtext |Tee-Object MRCE_Records.txt -Append
$query = New-Object Microsoft.SharePoint.SPQuery
$query.Query = '<Where><Eq><FieldRef Name="ContentType" /><Value Type="Computed">Item</Value></Eq></Where>'
$query.ViewAttributes = "Scope='RecursiveAll'";
$records = $Rlist.GetItems($query)
for ($intIndex =$records.Count - 1; $intIndex -gt -1; $intIndex--)
{
$RCount ++
}
$outputnewwebtext = "Records in $($spWeb.URL): $RCount`n"
}
}
}
Try Get-SPWeb command.
Get-SPWeb http://sp10/sites/team1
My code is below, it's pretty much complete, however a tiny part I am stuck on.
Basically what the code does is, based on a Song, the script goes out to a couple of websites on the internet to bring back the song label, composer & year that the song was released.
Only thing is, with the particular song Alicia Keys - Girl On Fire, if you manually go to the website http://staff.australian-charts.com/showitem.asp?interpret=Alicia+Keys+feat%2E+Nicki+Minaj&titel=Girl+On+Fire&cat=s you will notice that there is more than one Composer listed under the Music/Lyrics section.
In the event the song has more than one composer, my script below in it's current state will grab the first composer listed only.
What I need is for the script to grab all the composers. If there's one composer, or if there are multiple composers, I need them captured in the format of "Composer1, Composer2, Composer3, Composer4" etc (Commas included)
I am thinking of changing the Invoke-Webrequest to get TABLES back and particular tables with rows etc, not sure....
$song = "Alicia Keys - Girl On Fire"
Write-Host $song
$SearchSong = $song -replace '\(' -replace '\)' -replace '&' -replace ' - ', ' ' -replace '\s','+'
$MatchSong = $song -replace ' - ', '&titel=' -replace '\s','\+'
#Check iTunes for music Label information
$uri = "https://itunes.apple.com/search?term=$SearchSong&country=au&entity=song"
$x = Invoke-WebRequest -Uri $uri
$iTunesResults = ($x.Content | ConvertFrom-Json).results
$y = Invoke-WebRequest -Uri $iTunesResults[0].trackViewUrl
$iTunesSongCopyright = ($y.ParsedHtml.getElementsByTagName('li') | ? {$_.ClassName -eq 'copyright'}).innerText -replace '℗ '
$iTunesSongLabel = $iTunesSongCopyright -replace '.*\d\s'
#The check australian-charts for Composer & Year infomation
$domain = 'http://staff.australian-charts.com/'
$uri = $domain + "search.asp?search=$SearchSong&cat=s"
$x = Invoke-WebRequest -Uri $uri
$x.AllElements[462].outerHTML -match 'A.href="(.*)"';$resultURL = $domain + $Matches[1]
$resultURL = $resultURL -replace("&","&") -replace('"','"')
$y = Invoke-WebRequest -Uri $resultURL
$Element = ($y.AllElements | ? {$_.tagName -eq 'HTML'})
if($Element.innerText -match 'Music\/Lyrics:(.*)')
{
$Element.innerText -match 'Music\/Lyrics:(.*)'
$Composer = $Matches[1]
Write-Host $Composer
} else {
$Composer = $null
}
if($Element.innerText -match 'Year:(.*)')
{
$Element.innerText -match 'Year:(.*)'
$Year = $Matches[1]
Write-Host $Year
} else {
$Year = $null
}
Write-Host $iTunesSongLabel
you can use this for getting composer list :
if($Element.innerText -match 'Music\/Lyrics:(.*)')
{
$startpos = $Element.innertext.IndexOf("Lyrics:") + 7
$endpos = $Element.innertext.IndexOf("Producer:") -1
$composer=$Element.innertext.substring($startpos,($endpos - $startpos))
#even the below line will give the same result as the above line if uncommented
#$composer = $Element.innertext[$startpos..$endpos] -join ""
Write-Host $Composer
}