I am trying to genrate URLs using a bunch of strings and an anchor URL
strings in s.txt are
123
234
345
anchor URL is https://testurl.com/prod/hostdetails.php?qs=
The code I am using
$ur = gc C:\temp\s.txt
foreach($u in $ur) {
$test = invoke-webrequest -uri "https://testurl.com/prod/hostdetails.php?qs=$u" -UseDefaultCredentials
}
$test
but it returns data only for
https://testurl.com/prod/hostdetails.php?qs=345
Its because you reassign $test each time within your loop. I suggest this solution:
$test = 'https://testurl.com/prod/hostdetails.php?{0}' -f ((C:\temp\s.txt) -join '&')
You edited your question. This is probably what you are looking for. Please notice the += operator to concat the results:
$result = #()
Get-Content C:\temp\s.txt | ForEach-Object {
$result += invoke-webrequest -uri "https://testurl.com/prod/hostdetails.php?qs=$($_)"
}
Related
I am trying to create a Powershell script, using the Cylance API, which:
Find all device older than 45 days
Store these devices as Name: Date into a .csv file
Delete all of these devices
I have a Perl background, but I am new to Powershell, and so I figure I could do it this way:
Find each device older than 45 days, store them as Name -> Date in a hashtable, print the hashtable to a .csv file, and then delete the devices stored in the hash table.
However, I am only getting a few devices, and the csv file is blank. This is what is stored in the HashTable(I know for a fact there should be WAY more devices stored in this HashTable).
Name Value
---- -----
MPB8NTVR 2018-12-11T17:25:38.259
FENAPPTESTNICK 2018-12-11T15:50:31.924
SERSENM2 2018-12-11T14:57:50.109
FENWAPNMSP40 2018-12-07T20:26:04.005
FENXENAPPD01 2018-12-13T16:42:34.517
FENXENAPPD02 2018-12-13T16:43:54.908
M5CG7193HR4 2018-12-14T17:10:20.588
param (
[string]$AccessTokenPath = "Documents\Cylance\",
[string]$CylanceGenerator = "CylanceGeneratorTest.exe",
[int]$PageNumber = 1,
[int]$PageSize = 200,
[string]$CylanceURI = "https://protectapi.cylance.com/devices/v2?page=$PageNumber&page_size=$PageSize",
[string]$CylanceDeviceURI = "https://protectapi.cylance.com/devices/v2/",
[string]$OutFile = "$PWD\testfile.txt",
$TodaysDate = (Get-Date -Format 'yyyy-MM-ddTHH:mm:ss.fff'),
$date = 45
)
$AccessToken = & $AccessTokenPath\$CylanceGenerator
$CSVHash = #{}
do {
[string]$CylanceURI = "https://protectapi.cylance.com/devices/v2?page=$PageNumber&page_size=$PageSize"
try {
$CylanceRequest = Invoke-RestMethod -Method Get -Uri $CylanceURI -Header #{"Authorization" = "Bearer $AccessToken"}
}
catch {
$CylanceError = $_
}
$response = $CylanceRequest | Select-object -ExpandProperty page_items | Select-Object -Property id, name, date_first_registered, date_offline
$response | ForEach-Object {
try {
$uri = $CylanceDeviceURI + $_.id
$CylanceDeviceReq = Invoke-RestMethod -Method Get -Uri $uri -Header #{"Authorization" = "Bearer $AccessToken"}
$DaysOffline = $CylanceDeviceReq.date_offline
if ($DaysOffline) {
"Days offline = " + $DaysOffline
$TimeDiff = New-TimeSpan -Start $TodaysDate -End $DaysOffline
"TimeDiff.Days = " + $TimeDiff.Days
}
if ($DaysOffline -and ($TimeDiff.Days -le -$date)) {
"Adding " + $_.name + " " + $DaysOffline
$CSVHash.Add($_.name, $DaysOffline)
}
$temp = $_ | ForEach-Object {
if ($_.date_offline) {
New-Object psobject -Property #{
"Name" = $_.name
"ID" = $_.id
"Date Offline" = $CylanceDeviceReq.date_offline
}
}
$temp | Export-csv -Append $PWD\raw.csv -NoTypeInformation
}
}
catch {
$CylanceError = $_
}
$PageNumber++
}
} while ($PageNumber -le $PageSize)
#$CompHash
$CSVHash
What stands out to me as strange is the line where you assign value to $response. I'm not familiar with the object you're dealing with, but at first glance I suspect a misuse of -ExpandProperty on that line. That would return a property as a string. You then pipe to second Select-Object, also unusual. Try something like $CylanceRequest.page_items | Select-Object....
I could be mistaken, but that's the bit that jumps out at me as iffy.
As for …
I have a Perl background, but I am new to Powershell
Are you saying, you've done some ramp up on PowerShell before you started down this path and that you've already leveraged the Cylance modules / scripts from the MS PowerShellGallery, GitHub and could not leveraged them directly or tweak them to your needs before trying what you've posted?
Cylance-API-PS
https://github.com/Maliek/Cylance-API-PS
CylanceDSCScript.ps1
GetNewThreatsAndDeviceLastDay.ps1
GetOldDevices.ps1
GetOldDevicesAndRemove.ps1
GetRansom.ps1
GetSubClassificationsThreats.ps1
GetThreatsAndDevice.ps1
README.md
SendMailLatestThreatsAndDevices.ps1
UploadMultipleHashes.ps1
hashes.txt
Cylance-API-Powershell-example
https://github.com/regexninja826/Cylance-API-Powershell-example
https://www.powershellgallery.com/packages/CyCLI/0.5.6/Content/CyAPI.ps1
https://github.com/jan-tee/cycli
Hash tables do not restrict the amount of data that is returned.
So, pull the raw dataset natively first to make sure you are getting the count you'd expect, that work out the formatting.
I encountered a problem today while trying to get a value from a website using PowerShell.
This is the website.
I am trying to get the number "90" here, that in the webpage itself is the value of "Downloaded" (this number might be a little bigger when you look at it if there are more downloads):Screenshot of the element i am trying to return
<span title="Downloads" class="mod-card-info-tag" data-reactid=".0.0.0.2.0.0.2.2">
<div class="mod-card-info-tag-label" data-reactid=".0.0.0.2.0.0.2.2.0">90</div>
This is the PowerShell code i used to try and get the number "90" from the element above (i know i should use ".innertext" in the end, i just used the get-member to see if any object was found):
$URI = "https://mods.factorio.com/mods/TpTheGreat/TpTheGreats%20Large%20Roboport%20Logistics%20Area"
$HTML = Invoke-WebRequest -Uri $URI
($HTML.ParsedHtml.getElementsByTagName("div") | Where{ $_.className -eq ‘mod-card-info-tag-label’ }) | Get-Member
When calling for the element by tag name like in my code above, I get an empty object.
I tried lots of things without any success.
It'll be really great if any of you could take a look and check if you are able to solve my problem.
Thanks a lot!!!
How about another approach:
$URI = "https://mods.factorio.com/mods/TpTheGreat/TpTheGreats%20Large%20Roboport%20Logistics%20Area"
$HTML = Invoke-WebRequest -Uri $URI
$arr = $HTML.AllElements.Item(9).innerHTML -split ' = '
$myObj = $arr[1].replace("`n"," ")
$myObj = $myObj.replace(";","") | ConvertFrom-Json
$myObj.mod.mod.downloads_count
I have a few URLs which would need to cut and separate the first part of the each URL, i.e example1.com, example2.com, example3.com from each line and store in a variable
Contents in url.csv
https://example1.com/v1/test/f3de-a8c6-464f-8166-9fd4
https://example2.com/v1/test/14nf-d7jc-54lf-fd90-fds8
https://example3.com/v1/test/bd38-17gd-2h65-0j3b-4jf6
Script:
$oldurl = Import-CSV "url.csv"
$newurl = $oldurl.list -replace "https://"
This would replace https://, however the rest of each cannot be hard coded as those values can change.
What could be change code change required to cut anything from and after /v1/ along with https://?
$list = #(
"https://example1.com/v1/test/f3de-a8c6-464f-8166-9fd4",
"https://example2.com/v1/test/14nf-d7jc-54lf-fd90-fds8",
"https://example3.com/v1/test/bd38-17gd-2h65-0j3b-4jf6"
)
$result = $list | %{
$uri = [System.Uri] $_
$uri.Authority
}
$result
See System.Uri properties to potentially assemble the information you need in your result list.
This will cut off anything after "/v1/" and it self. Is that what you want?
$string = "https://example1.com/v1/test/f3de-a8c6-464f-8166-9fd4"
$string = $string -replace "https://"
$pos = $string.IndexOf("/v1/")
$result = $string.Substring(0, $pos)
$result
Output: example1.com
I've encountered a situation that I cannot seem to find a solution to. I am scraping a website using Invoke-WebRequest and when I look at my output from my array, several of the properties are System.Objects. I need to find a way to have them be strings so that when I Export-Csv I can actually see the values. Here is my code:
$params = #{api_id='';api_key='';page_size='100';site_id=''}
$stats = Invoke-WebRequest https://my.incapsula.com/api/visits/v1 -Method Post -Body $params
$s = $stats
$s = $s | ConvertFrom-Json
$s = $s.visits
Here are what my results look like:
My solution was to create a new custom object and use the following syntax on the properties that were an object themselves:
($_ | select -expandproperty 'propertyname')
I'm just recently starting to learn powershell and made a webcrawler. Im trying to get the redirected url to job listings.
$links=#();
For ($i=0; $i -lt 50; $i=$i+10) {
$arr=((Invoke-WebRequest –Uri (‘http://www.indeed.com/jobs?q=software+engineer+entry+level+%2460%2C000&l=San+Diego%2C+CA&jt=fulltime&start=’+$i) -MaximumRedirection 2).Links
| Where data-tn-element -eq “jobTitle”).href;
foreach ($arr in $arr) {
$foo="https://www.indeed.com"+$arr -replace "mp;a","";
$bar=Invoke-WebRequest –Uri ($foo) -MaximumRedirection 10;
$test=$bar;
#im trying to get the url of $bar
$links+= $test;
}
}
foreach($links in $links){
echo $links;
}
With some editing i can get the links that indeed uses, but im trying to get the url where the actual job description is listed
If you get a redirection, the result should be visible response of your invoke-webrequest object:
$bar.BaseResponse.ResponseUri
So:
$test = $bar.BaseResponse.ResponseUri