I'm trying to build a script that will grab some text from a website and place it into an excel sheet, but I'm having trouble with using a variable in the URL, when using a variable my output is http://#{variable=xyz}.com, I'm trying to get the xyz only,
any help would be greatly appreciated
my code:
$ip = get-content c:\..
foreach ($ip in $ipadd){
$url = http://$ip/mywebsite.com
$html = invoke-webrequest -uri $url
$elements = $html.Allelements | where class -eq "class" | select -expandproperty innertext
}
I ended up using plain text and it worked just fine.
but id appreciate if anyone could tell me why I'm getting this http://#{variable=xyz}.com output from a csv file
Related
I am new to azure data explorer and Kusto Queries. I am learning from below online sample
https://dataexplorer.azure.com/clusters/help/databases/Samples
Here is the query which i am getting results in Data Explorer but unable to display in power shell
StormEvents
| where DamageProperty >0
| limit 2
| project StormSummary.TotalDamages
Below is reference link for code which i am trying to run query in Powershell (Example2 in below link page)
https://learn.microsoft.com/en-us/azure/data-explorer/kusto/api/powershell/powershell
I had changed only "$Query" and modified last code line as like below
$dataView | Format-Table -AutoSize
i am getting output as
StormSummary_TotalDamages
-------------------------
{}
I tried modifying query without "TotalDamages" in "StormSummary.TotalDamages" but resulting dataview which i am unable to get again "TotalDamages" .
StormSummary
------------
{TotalDamages, StartTime, EndTime, Details}
Some one helped me to fix my issue. I am posting it to helps others.
Explanation:
Query result is importing to Json string, converting it from json and converting the columns&rows data-layout back into individual PSObjects really helped my issue.
Code:
As per the Example 2 mentioned in my question . we are calling
$reader = $queryProvider.ExecuteQuery($query, $crp)
After this I removed existing code and modified like below:
Modified Code to get Projected Field data(TotalDamages):
$json = [Kusto.Cloud.Platform.Data.ExtendedDataReader]::ToJsonString($reader)
$data = $json | ConvertFrom-Json
$columns = #{}
$count = 0
foreach ($column in $data.Tables[0].Columns) {
$columns[$column.ColumnName] = $count
$count++
}
$items = foreach ($row in $data.Tables[0].Rows) {
$hash = #{}
foreach ($property in $columns.Keys){
$hash[$property] = $row[$columns[$property]]
}
[PSCustomObject]$hash
}
foreach($item in $items)
{
Write-Host "TotalDamages: "$item.StormSummary.TotalDamages
}
Output:
TotalDamages: 6200000
TotalDamages: 2000
I have a script which I'm trying to modify, which adds objects to a firewall via powershell, using Invoke-RestMethod
The current script has the following code;
#Import CSV and set variables
$csv = import-csv C:\Powershell\groups.csv
# RESTful API Call
$csv | ForEach-Object {
$Name = $_.name
$Member1 =$_.member1
$Member2 =$_.member2
Invoke-RestMethod -Uri https://172.16.16.16:4444/webconsole/APIController?reqxml=<Request><Login><Username>admin</Username><Password>password</Password></Login><Set%20operation=%27add%27><IPHostGroup><Name>$Name</Name><IPFamily>IPv4</IPFamily><HostList><Host>$Member1</Host><Host>$Member2</Host></IPHostGroup></Set></Request>
}
I am wanting to import the hostgroups via groups.csv which (in my test) has 3 columns as follows;
Name,Member1,Member2
TestGroup,TestHost1,TestHost2
TestGroup2,TestHost3,TestHost4
etc.
My problem is that in the real data, there are varying amount of hosts in each group, some have hundreds. I'm not sure how to get these into the command without defining a variable for each possible member. Even then, say I created $Member(s) all the way to 200 (be gentle, I'm not a real coder!) and then imported them in manually one by one in the Invoke-Restmethod command (Might as well do it by hand at that point!) I'm not sure the command would handle the blank inputs in the cases where there were only a few hosts in the group.
(In other words if my csv had the following entries;)
Name,Member1,Member2,Member3,Member4
TestGroup,TestHost1,TestHost2,TestHost3,TestHost4
TestGroup2,TestHost5,TestHost6
TestGroup3,TestHost7
And I did;
# RESTful API Call
$csv | ForEach-Object {
$Name = $_.name
$Member1 =$_.member1
$Member2 =$_.member2
$Member3 =$_.member3
$Member4 =$_.member4
The Rest call for the third group would end up running as;
Invoke-RestMethod -Uri https://172.16.16.16:4444/webconsole/APIController?reqxml=<Request><Login><Username>admin</Username><Password>password</Password></Login><Set%20operation=%27add%27><IPHostGroup><Name>TestGroup3</Name><IPFamily>IPv4</IPFamily><HostList><Host>TestHost7</Host><Host></Host><Host></Host><Host></Host></IPHostGroup></Set></Request>
Can anyone point me in the direction of a better way of doing this?
You can get all member names using .PSObject.Properties.Name
Example:
$Csv = Import-Csv -Path 'C:\Powershell\groups.csv'
# Request XML template
$RequestTpl = #'
<Request>
<Login>
<Username>admin</Username>
<Password>password</Password>
</Login>
<Set%20operation=%27add%27>
<IPHostGroup>
<Name>{0}</Name>
<IPFamily>IPv4</IPFamily>
<HostList>
{1}
</HostList>
</IPHostGroup>
</Set>
</Request>
'#
# Host list XML template
$RequestHostListTpl = '<Host>{0}</Host>'
$Csv | ForEach-Object {
<#
Get names of all the properties in the current object
Leave only those that don't match '^Name$' regex.
-match, when operates on collections, returns matched items
You can use
$_.PSObject.Properties.Name | Where-Object {$_ -ne 'Name'}
but it's a bit slower.
#>
$Members = #($_.PSObject.Properties.Name) -notmatch '^Name$'
# Build list of hosts
$RequestHostList = foreach ($item in $Members) {
# Only add item if it's not empty
if ($_.$item) {
$RequestHostListTpl -f $_.$item
}
}
# Build request XML
$Request = $RequestTpl -f $_.Name, -join $RequestHostList
# Remove newlines to make it one long string
$Request = $Request -replace '\r|\n'
# Show resulting url
"Invoke-RestMethod -Uri https://172.16.16.16:4444/webconsole/APIController?reqxml=$Request"
# Uncomment to actually invoke API call
#Invoke-RestMethod -Uri "https://172.16.16.16:4444/webconsole/APIController?reqxml=$Request"
}
I encountered a problem today while trying to get a value from a website using PowerShell.
This is the website.
I am trying to get the number "90" here, that in the webpage itself is the value of "Downloaded" (this number might be a little bigger when you look at it if there are more downloads):Screenshot of the element i am trying to return
<span title="Downloads" class="mod-card-info-tag" data-reactid=".0.0.0.2.0.0.2.2">
<div class="mod-card-info-tag-label" data-reactid=".0.0.0.2.0.0.2.2.0">90</div>
This is the PowerShell code i used to try and get the number "90" from the element above (i know i should use ".innertext" in the end, i just used the get-member to see if any object was found):
$URI = "https://mods.factorio.com/mods/TpTheGreat/TpTheGreats%20Large%20Roboport%20Logistics%20Area"
$HTML = Invoke-WebRequest -Uri $URI
($HTML.ParsedHtml.getElementsByTagName("div") | Where{ $_.className -eq ‘mod-card-info-tag-label’ }) | Get-Member
When calling for the element by tag name like in my code above, I get an empty object.
I tried lots of things without any success.
It'll be really great if any of you could take a look and check if you are able to solve my problem.
Thanks a lot!!!
How about another approach:
$URI = "https://mods.factorio.com/mods/TpTheGreat/TpTheGreats%20Large%20Roboport%20Logistics%20Area"
$HTML = Invoke-WebRequest -Uri $URI
$arr = $HTML.AllElements.Item(9).innerHTML -split ' = '
$myObj = $arr[1].replace("`n"," ")
$myObj = $myObj.replace(";","") | ConvertFrom-Json
$myObj.mod.mod.downloads_count
I've encountered a situation that I cannot seem to find a solution to. I am scraping a website using Invoke-WebRequest and when I look at my output from my array, several of the properties are System.Objects. I need to find a way to have them be strings so that when I Export-Csv I can actually see the values. Here is my code:
$params = #{api_id='';api_key='';page_size='100';site_id=''}
$stats = Invoke-WebRequest https://my.incapsula.com/api/visits/v1 -Method Post -Body $params
$s = $stats
$s = $s | ConvertFrom-Json
$s = $s.visits
Here are what my results look like:
My solution was to create a new custom object and use the following syntax on the properties that were an object themselves:
($_ | select -expandproperty 'propertyname')
At the moment I am working on a script to automate a process in IE to add Computer Names and their MACs so we can image them. The page has two fields one for MAC and one for a computer name then a add new button. I had to come to a pretty sloppy solution for avoiding a popup from the page by just quitting out of the com object after submitting.
I don't have much experience with Powershell yet and none with working with CSVs so I'm having a bit of trouble making this work. My goal is to have the script read two entries from a row fill out the correct field then submit it then move to the next row and repeat.
Right now what it does is fills out the fields with undefined in both fields, then submits and repeats.
EDIT: I have edited my code slightly just so it confirms what is trying to read.This is what the results look like. I believe #WalterMitty is on to something that something is wrong with $ie.document.getElementsByName lines, I just tried $ie.document.getElementById but that didn't fill out any fields. It seems it has no problem reading the CSV, but it does have a problem entering the information it reads into the fields properly.
This is an example of what the CSV would look like.
NewComputerName,NewMACAddress
ComputerName1,111122223333
ComputerName2,112233446677
ComputerName3,AAAABBBBCCCC
ComputerName4,AABBCCDDEEFF
This is what my code currently looks like.
cls
$URL = ""
$iterator = 1;
$csv = Get-Content C:\example1.csv
foreach($row in $csv)
{
#starts IE
$ie = new-object -ComObject "InternetExplorer.Application"
$ie.visible = $true
$ie.navigate($URL)
while($ie.Busy -eq $true) { start-sleep -Milliseconds 100 }
($ie.document.getElementsByName("mc_id") |select -first 1).value = $_.NewComputerName;
($ie.document.getElementsByName("mac_id") |select -first 1).value = $_.NewMACAddress;
$ie.document.forms | Select -First 1| % { $_.submit() };
$ie.quit()
$iterator++
write-host "$iterator new ID(s) added"
write-host $row.NewComputerName - $row.NewMACAddress
}
$URL = ""
$iterator = 1
# use Import-Csv for CSV files
$csv = Import-Csv "C:\example1.csv" -Delimiter ","
foreach($row in $csv) {
Write-Host "$iterator new ID(s) added"
#starts IE
$ie = New-Object -ComObject "InternetExplorer.Application"
$ie.Visible = $true
$ie.Navigate($URL)
while ($ie.Busy -eq $true) { Start-Sleep -Milliseconds 100 }
# $_ is not defined in foreach blocks, you have to use $row here
($ie.Document.getElementsByName("mc_id") | Select-Object -First 1).Value = $row.NewComputerName
($ie.Document.getElementsByName("mac_id") | Select-Object -First 1).Value = $row.NewMACAddress
$ie.Document.Forms | Select-Object -First 1 | ForEach-Object { $_.submit() }
$ie.Quit()
$iterator++
}
I'm having similar issues but I just found this may help you guys - I don't have 50 reputation to comment sorry :/....
I messed around with the -Path of the Import-CSV command but I just couldn’t make it work. Apparently this has nothing to do with the path of the CSV file. The Warlock posted this on his blog:
Long story short, the error came from having trailing blank columns in
my CSV. Import-Csv uses the first row in the CSV as names for the
columns (unless you specify otherwise) and when you have blank columns
(or at least multiple blank columns) it causes this error as it
doesn’t have a valid name for them.
Instead of changing the file, I changed my import command to include the headers as per Dale's comment and it worked perfectly:
$data = import-csv "C:\Sharepoint.csv" -header("Department","AD Group","Members","Notes")
The Warlock and Dale saved me lots of time, please stop by the Warlock’s blog and give them a big Thanks
Consider using Import-Csv and Invoke-WebRequest in combination, e.g. like this:
import-csv .\example.csv | %{ iwr http://someurl.local -body #{mc_id=$_.NewComputerName; mac_id=$_.NewMACAddress} -Method POST }
It will read the csv file, iterate over the records and create a application/x-www-form-urlencoded POST request with the values from each record.
When you use iwr (Invoke-WebRequest) and pass a hash table as the "body" it will act as if it is a form being submitted. The POST method will submit the form values as application/x-www-form-urlencode. Without the POST method it would submit the form as if it was a GET, i.e. pass the values in the url.
If you need authentication, session support etc. then read the documentation for Invoke-WebRequest.
Using IE to automate web requests is brittle and error prone.