I have a Web API that returns an HTML string which I want to compare with a html file on my local machine.
To do so, I have the following code
$Result = (Invoke-WebRequest `
-Uri "{uri}" `
-Headers #{"some-header", "some-value"}).Content
$TestContent = Get-Content -Path ($RepositoryLocation + "index.html") -Raw
$Equal = Compare-Object -ReferenceObject $TestContent -DifferenceObject $Result
When I now use Write-Hosts $Equal it displays me that the whole content is different
When I use Write-Host $Equal.SideIndicator it displays me => <= which also indicates that the complete file is different
Furthermore, using the command with -IncludeEqual -ExcludeDifferent displays empty result, so like I said, no lines are the same.
So what I tried next was to save the Content of $Result into a text file and compare them then, but still, it told me, that the whole file is different.
I then used diffchecker.com as well as JetBrains IDEs integrated comparison tool, to check for differences. Both tools told me that the content is identical. I'm losing my mind, why does PowerShell tell me they have complete different content?
Sadly, I cannot post the content of the API response as well as the content of the index.html
What I thought maybe could be the reason is
Encoding, however both are UTF8
Line endings, however no diff if I use CL, CL RF or RF on the file
Some hidden characters (tabs instead of spaces) but I activated to see that on JetBrains IDE and they still are identical.
How do I know what's causing this issue here?
Not sure if Compare-Object is the right choice here. As the name says, it's for objects. Why not use the equals-operator -eq or string.Equals?
$equal = $testContent -eq $result
# or
$equal = $testContent.Equals($result)
Compare-Object does not do line-by-line-comparison if both are just single strings. So your strings are not equal, as long as there's any tiny difference, as much as just an extra line-feed at the end, etc.
You could try several things:
# trim
$equal = $testContent.Trim() -eq $result.Trim()
# case-insensitive comparison
$equal = $testContent.Equals($result, 'OrdinalIgnoreCase')
# or both
$equal = $testContent.Trim().Equals($result.Trim(), 'OrdinalIgnoreCase')
Btw: If you do want a line-by-line-comparison, you have to split up the strings into lines first, e.g.:
Compare-Object ($testContent -split "`r`n") ($result -split "`r`n")
Related
I'm trying to find an efficient way to read the value of a string variable in a PowerShell .ps1 file and then update the same variable/value in another .ps1 file. In my specific case, I would update a variable for the version # on script one and then I would want to run a script to update it on multiple other .ps1 files. For example:
1_script.ps1 - Script I want to read variable from
$global:scriptVersion = "v1.1"
2_script.ps1 - script I would want to update variable on (Should update to v1.1)
$global:scriptVersion = "v1.0"
I would want to update 2_script.ps1 to set the variable to "v1.1" as read from 1_script.ps1. My current method is using get-content with a regex to find a line starting with my variable, then doing a bunch of replaces to get the portion of the string I want. This does work, but it seems like there is probably a better way I am missing or didn't get working correctly in my tests.
My Modified Regex Solution Based on Answer by #mklement0 :
I slightly modified #mklement0 's solution because dot-sourcing the first script was causing it to run
$file1 = ".\1_script.ps1"
$file2 = ".\2_script.ps1"
$fileversion = (Get-Content $file1 | Where-Object {$_ -match '(?m)(?<=^\s*\$global:scriptVersion\s*=\s*")[^"]+'}).Split("=")[1].Trim().Replace('"','')
(Get-Content -Raw $file2) -replace '(?m)(?<=^\s*\$global:scriptVersion\s*=\s*")[^"]+',$fileversion | Set-Content $file2 -NoNewLine
Generally, the most robust way to parse PowerShell code is to use the language parser. However, reconstructing source code, with modifications after parsing, may situationally be hampered by the parser not reporting the details of intra-line whitespace - see this answer for an example and a discussion.[1]
Pragmatically speaking, using a regex-based -replace solution is probably good enough in your simple case (note that the value to update is assumed to be enclosed in "..." - but matching could be made more flexible to support '...' quoting too):
# Dot-source the first script in order to obtain the new value.
# Note: This invariably executes *all* top-level code in the script.
. .\1_script.ps1
# Outputs to the display.
# Append
# | Set-Content -Encoding utf8 2_script.ps1
# to save back to the input file.
(Get-Content -Raw 2_script.ps1) -replace '(?m)(?<=^\s*\$global:scriptVersion\s*=\s*")[^"]+', $global:scriptVersion
For an explanation of the regex and the ability to experiment with it, see this regex101.com page.
[1] Syntactic elements are reported in terms of line and column position, and columns are character-based, meaning that spaces and tabs are treated the same, so that a difference of, say, 3 character positions can represent 3 spaces, 3 tabs, or any mix of it - the parser won't tell you. However, if your approach allows keeping the source code as a whole while only removing and splicing in certain elements, that won't be a problem, as shown in iRon's helpful answer.
To compliment the helpful answer from #mklement0. In case your do go for the PowerShell abstract syntax tree (AST) class, you might use the Extent.StartOffset/Extent.EndOffset properties to reconstruct your script:
Using NameSpace System.Management.Automation.Language
$global:scriptVersion = 'v1.1' # . .\Script1.ps1
$Script2 = { # = Get-Content -Raw .\Script2.ps1
[CmdletBinding()]param()
begin {
$global:scriptVersion = "v1.0"
}
process {
$_
}
end {}
}.ToString()
$Ast = [Parser]::ParseInput($Script2, [ref]$null, [ref]$null)
$Extent = $Ast.Find(
{
$args[0] -is [AssignmentStatementAst] -and
$args[0].Left.VariablePath.UserPath -eq 'global:scriptVersion' -and
$args[0].Operator -eq 'Equals'
}, $true
).Right.Extent
-Join (
$Script2.SubString(0, $Extent.StartOffset),
$global:scriptVersion,
$Script2.SubString($Extent.EndOffset)
) # |Set-Content .\Script2.ps1
Basically I have a huge csv of phishing links and I'm trying to trim off https://www. and anything after .com .edu etc. so basically the ideal ouput of the powershell script would be a long list of urls all of which look something like google.com or microsoft.com so far I have imported the csv but everything I have tried either doesn't work or leaves the www on the beggining. Any help would be great. The csv im using is this: http://data.phishtank.com/data/online-valid.csv
$urls = Import-Csv -Path .\online-valid.csv | select -ExpandProperty "url"
The below will take your CSV and do magic for you. Have a play around with [Uri], it is very useful when parsing web links.
$csv = import-csv C:\temp\verified_online.csv
Foreach($Site in $csv) {
$site | Add-Member -MemberType NoteProperty -Name "Host" -Value $(([Uri]$Site.url).Host -replace '^www\.')
}
$csv | Export-Csv C:\temp\verified_online2.csv -NoTypeInformation
Adjusted based on recommendation from Mklement0.
A concise and fast alternative to Drew's helpful answer based on casting the URL strings directly to an array of [uri] (System.Uri) instances, and then trimming prefix www., if present, from their .Host (server name) property:
([uri[]] (Import-Csv .\online-valid.csv).url).Host -replace '^www\.'
Note that the -replace operator is regex-based, and regex ^www\. makes sure what www is only replaced at the start (^) of the string, and only if followed by a literal . (\.), in which case this prefix is removed (replaced with the implied empty string); if no such prefix is present, the input string is passed through as-is.
The solution reads the entire CSV file into memory at once, for convenience and speed, and outputs just the trimmed server names, as an array of strings.
I'm trying to test out a web filtering solution so i have a powershell which loop throughs a list of URLs and returns the webresponse. The problem is that often times you hit cdns or other sites that maybe unauthorized 403 or 404 not found and you need to find the root domain.
The only logical solution from what i've found is to cross reference it against the publicsuffix list. The only language it doesn't operate well with from what i've seen is PowerShell. I'm wondering if anyone has come across this or has a solution.
While your solution works, there is an alternative that is both more concise and much faster:
$url = 'https://publicsuffix.org/list/public_suffix_list.dat'
(Invoke-RestMethod $url) -split "`n" -match '^[^/\s]' |
Set-Content .\public_suffix_list.dat
Invoke-RestMethod $url returns the text file at the specified URL as a single string.
-split "`n" splits the string into an array of lines
-match '^[^/\s]' matches those lines that start with (^) a character (from the set enclosed in [...]) that is not (^) a literal / and not a whitespace character (/s), which effectively filters out comment / (hypothetical) non-data lines.
The above saves the data-lines-only array to a file, as in your solution.
Note that determining whether a given URL has a public suffix involves more than just suffix matching against the data lines, because the latter have wildcard labels (*) and involve exceptions (lines starting with !) - see https://publicsuffix.org/list/
# You can use whatever directory
$workingdirectory = "C:\"
# Downloads the public suffix list
Invoke-WebRequest -Uri "https://publicsuffix.org/list/public_suffix_list.dat" -OutFile "$workingdirectory\public_suffix_list.dat"
# Gets the content of the file, removes the empty spaces, removes all the
# comments that has // and outputs it to a file
(gc $workingdirectory\public_suffix_list.dat) |
? { $_.Trim() -ne "" } |
Select-String -Pattern "//" -NotMatch |
Set-Content "$workingdirectory\public_suffix_list.dat"
I'm far away from being an expert in PowerShell, so I'll be my best to explain here.
I was able to add a column, but now I want to add stuff in a column (already there) using a separate script.
For example, the following CSV file:
WhenToBuyThings,ThingsToBuy
BuyNow ,Bed
BuyNow ,Desk
BuyNow ,Computer
BuyNow ,Food
BuyLater ,
BuyLater ,
BuyLater ,
I have the array:
$BuyStuffLater = "Books","Toys","ShinnyStuff"
So the end result of the file should look like this
BuyNow ,Bed
BuyNow ,Desk
BuyNow ,Computer
BuyNow ,Food
BuyLater ,Books
BuyLater ,Toys
BuyLater ,ShinnyStuff
Any help with how to do this in code would be much appreciated. Also, we can't use delimiter ",". Because in the real script some values will have commas.
I got it after a few hours of fiddling...
$myArray = "Books","Toys","ShinnyStuff"
$i = 0
Import-Csv "C:\Temp\test.csv" |
ForEach-Object {if($_.WhenToBuyThings -eq "BuyLater"){$_.ThingsToBuy = $myArray[$i];$i++}return $_} |
Export-Csv C:\Temp\testtemp.csv -NoTypeInformation
All is well now...
I am new to powershell, too. Here's what I found. This searches and returns all lines that fit. I'm not sure it can pipe.
$BuyStuffLater = "Books","Toys","ShinnyStuff"
$x = 0
Get-Content -Path .\mydata.txt | select-string -pattern "BuyLater" #searches and displays
# Im not sure about this piping. (| foreach {$_ + $BuyStuffLater[$x];$x++} | Set-Content .\outputfile.csv)
This filter will work, though I still have to work on the piping. The other answer might be better.
I don't see a point to iterating through each object to see if it is a WhenToBuyThings is "BuyLater". If anything what you are doing could be harmful if you run multiple passes adding to the list. It could remove previous things you wanted to by. If "Kidney Dialysis Machine" was listed as a "BuyLater" under WhenToBuyThings then you would overwrite it with dire consequences.
What we can do is build two lists and merge into new csv file. First list is your original file minus any entry where a "BuyLater" has a blank ThingsToBuy. The second list is an object array built from your $BuyStuffLater. Add these lists together and export.
Also there is zero need to worry about using a comma delimiter when using Export-CSV. The data is quoted so commas in data do not affect the data structure. If this was still a concern you could use -Delimiter ";". I noticed in your answer that you did not attempt to account for commas either (not that it matters based on what I just said).
$path = "C:\Temp\test.csv"
$ListOfItemsToBuy = "Books","Toys","ShinnyStuff: Big, ShinyStuff"
$BuyStuffLater = $ListOfItemsToBuy | ForEach-Object{
[pscustomobject]#{
WhenToBuyThings = "BuyLater"
ThingsToBuy = $_
}
}
$CurrentList = Import-Csv $path | Where-Object{$_.WhenToBuyThings -ne "BuyLater" -and ![string]::IsNullOrWhiteSpace($_.ThingsToBuy)}
$CurrentList + $BuyStuffLater | Export-Csv $path -NoTypeInformation
Since you have 3.0 we can use [pscustomobject]#{} to build the new object very easily. Combine both arrays simply by adding them together and export back to the original file.
You should notice I used slightly different input data. One includes a comma. I did that so you can see what the output file looks like.
"BuyLater","Books"
"BuyLater","Toys"
"BuyLater","ShinnyStuff: Big, ShinyStuff"
I've looked all around this site and can't quite seem to find anything that fits my situation. Basically, I am trying to write an addition to the NETLOGON file that will replace text in a text file on all of our users' desktops. The current text is static across the board.
The text I want it changed to will be unique to each user. I want to change the current text (user1) to the users AD username (i.e. johnd, janed, etc.). I am using Windows Server 2008 R2 and all the workstations are Windows 7 Professional SP1 64 bit.
Here's what I have tried so far (with a few variables, which none have worked for one reason or the other):
gc c:\Users\%USERNAME%\desktop\VPN.txt' -replace "user1",$env:username | out-file c:\Users\%USERNAME%\desktop\VPN.txt
I didn't get an error, but it also did not go back to the normal "PS C:>" prompt, just ">>>" and the file did not change as anticipated.
If that is how you have the code exactly then I suppose it is because you have an opening single quote without a closing one. You are still going to have two other problems and you have one answer in your code. The >>> is the line continuation characters because the parser knows that the code is not complete and giving you the option to continue with the code. If you were purposely coding a single line on multiple lines you would consider this a feature.
$path = "c:\Users\$($env:username)\desktop\VPN.txt"
(Get-Content $path) -replace "user1",$env:username | out-file $path
Closed the path in quotes and used a variable since you called the path twice.
%name% is used in command prompt. Environment variables in PowerShell use the $env: provider which you did you once in your snippet.
-replace is a regex replaced tool that can work against Get-Content but you need to capture the result in a sub expression first.
Secondly with -replace is for regex and your string is not regex based you could just use .Replace() as well.
Set-Content is generally preferred over Out-File for performance reasons.
All that being said...
you could also try something like this.
$path = "c:\Users\$($env:username)\desktop\VPN.txt"
(Get-Content $path).Replace("user1",$env:username) | Set-Content $path
Do you want to only replace the first occurrence?
You could use a little regex here with a tweak in how you get the use Get-Content
$path = "c:\Users\$($env:username)\desktop\VPN.txt"
(Get-Content $path | Out-String) -replace "(.*?)user1(.*)",('$1{0}$2' -f $env:username) | out-file $path
Regex will match the entire file. There are two groups which it captures.
(.*?) - Up until the first "user1"
(.*) - Everything after that
Then we use the format operator to sandwich the new username in between those capture groups.
Use:
(Get-Content $fileName) | % {
if ($_.ReadCount -eq 1) {
$_ -replace "$original", "$content"
}
else {
$_
}
} | Set-Content $fileName