Comparing two text files and output the differences in Powershell - powershell

So I'm new to the Powershell scripting world and I'm trying to compare a list of IPs in text file against a database of IP list. If an IP from (file) does not exist in the (database) file put it in a new file, let's call it compared.txt. When I tried to run the script, I didn't get any result. What am I missing here?
$file = Get-Content "C:\Users\zack\Desktop\file.txt"
$database = Get-Content "C:\Users\zack\Desktop\database.txt"
foreach($line1 in $file){
$check = 0
foreach($line2 in $database)
{
if($line1 != $line2)
{
$check = 1
}
else
{
$check = 0
break
}
}
if ($check == 1 )
{
$line2 | Out-File "C:\Users\zack\Desktop\compared.txt"
}
}

There is a problem with your use of PowerShell comparison operators unlike in C#, equality and inequality are -eq and -ne, and since PowerShell is a case insensitive language, there is also -ceq and -cne.
There is also a problem with your code's logic, a simple working version of it would be:
$database = Get-Content "C:\Users\zack\Desktop\database.txt"
# iterate each line in `file.txt`
$result = foreach($line1 in Get-Content "C:\Users\zack\Desktop\file.txt") {
# iterate each line in `database.txt`
# this happens on each iteration of the outer loop
$check = foreach($line2 in $database) {
# if this line of `file.txt` is the same as this line of `database.txt`
if($line1 -eq $line2) {
# we don't need to keep checking, output this boolean
$true
# and break the inner loop
break
}
}
# if above condition was NOT true
if(-not $check) {
# output this line, can be `$line1` or `$line2` (same thing here)
$line1
}
}
$result | Set-Content path\to\comparisonresult.txt
However, there are even more simplified ways you could achieve the same results:
Using containment operators:
$database = Get-Content "C:\Users\zack\Desktop\database.txt"
$result = foreach($line1 in Get-Content "C:\Users\zack\Desktop\file.txt") {
if($line1 -notin $database) {
$line1
}
}
$result | Set-Content path\to\comparisonresult.txt
Using Where-Object:
$database = Get-Content "C:\Users\zack\Desktop\database.txt"
Get-Content "C:\Users\zack\Desktop\file.txt" | Where-Object { $_ -notin $database } |
Set-Content path\to\comparisonresult.txt
Using a HashSet<T> and it's ExceptWith method (Note, this will also get rid of duplicates in your file.txt):
$file = [System.Collections.Generic.HashSet[string]]#(
Get-Content "C:\Users\zack\Desktop\file.txt"
)
$database = [string[]]#(Get-Content "C:\Users\zack\Desktop\database.txt")
$file.ExceptWith($database)
$file | Set-Content path\to\comparisonresult.txt

Related

Remove the need to use out-file only to import the file immediately using PowerShell just to convert the base type

I am attempting to turn the file below into one that contains no comments '#', no blank lines, no unneeded spaces, and only one entry per line. I'm unsure how to run the following code without the need to output the file and then reimport it. There should be code that doesn't require that step but I can't find it. The way I wrote my script also doesn't look right to me even though it works. As if there was a more elegant way of doing what I'm attempting but I just don't see it.
Before File Change: TNSNames.ora
#Created 9_27_16
#Updated 8_30_19
AAAA.world=(DESCRIPTION =(ADDRESS_LIST =
(ADDRESS =
(COMMUNITY = tcp.world)
(PROTOCOL = TCP)
(Host = www.url1111.com)
(Port = 1111)
)
)
(CONNECT_DATA = (SID = SID1111)
)
)
#Created 9_27_16
BBBB.world=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(COMMUNITY=tcp.world)(PROTOCOL=TCP)(Host=url2222.COM)(Port=2222))(ADDRESS=(COMMUNITY=tcp.world)(PROTOCOL=TCP)(Host=url22222.COM)(Port=22222)))(CONNECT_DATA=(SID=SID2222)))
CCCC.world=(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(Host=url3333.COM)(Port=3333))(CONNECT_DATA=(SID=SID3333)))
DDDD.url =(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(COMMUNITY=tcp.world)(PROTOCOL=TCP)(Host=URL4444 )(Port=4444))(ADDRESS=(COMMUNITY=TCP.world)(PROTOCOL=TCP)(Host=URL44444 )(Port=44444)))(CONNECT_DATA=(SID=SID4444 )(GLOBAL_NAME=ASDF.URL)))
#Created 9_27_16
#Updated 8_30_19
After File Change:
AAAA.world=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(COMMUNITY=tcp.world)(PROTOCOL=TCP)(Host=www.url1111.com)(Port=1111)))(CONNECT_DATA=(SID=SID1111)))
BBBB.world=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(COMMUNITY=tcp.world)(PROTOCOL=TCP)(Host=url2222.COM)(Port=2222))(ADDRESS=(COMMUNITY=tcp.world)(PROTOCOL=TCP)(Host=url22222.COM)(Port=22222)))(CONNECT_DATA=(SID=SID2222)))
CCCC.world=(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(Host=url3333.COM)(Port=3333))(CONNECT_DATA=(SID=SID3333)))
DDDD.url=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(COMMUNITY=tcp.world)(PROTOCOL=TCP)(Host=URL4444)(Port=4444))(ADDRESS=(COMMUNITY=TCP.world)(PROTOCOL=TCP)(Host=URL44444)(Port=44444)))(CONNECT_DATA=(SID=SID4444)(GLOBAL_NAME=ASDF.URL)))
Code:
# Get the file
[System.IO.FileInfo] $File = 'C:\temp\TNSNames.ora'
[string] $data = (Get-Content $File.FullName | Where-Object { !$_.StartsWith('#') }).ToUpper()
# Convert the data. This part is where any (CONNECT_DATA entry ends up on it's own line.
$Results = $data.Replace(" ", "").Replace("`t", "").Replace(")))", ")))`n")
# Convert $Results from BaseType of System.Object to System.Array
$Path = '.\.vscode\StringResults.txt'
$Results | Out-File -FilePath $Path
$Results = Get-Content $Path
# Find all lines that start with '(CONNECT_DATA'
for ($i = 0; $i -lt $Results.Length - 1; $i++) {
if ($Results[$i + 1].StartsWith("(CONNECT_DATA")) {
# Add the '(CONNECT_DATA' line to the previous line
$Results[$i] = $Results[$i] + $Results[$i + 1]
# Blank out the '(CONNECT_DATA' line
$Results[$i + 1] = ''
}
}
# Remove all blank lines
$FinalForm = $null
foreach ($Line in $Results) {
if ($Line -ne "") {
$FinalForm += "$Line`n"
}
}
$FinalForm
So the crux of your problem is that you have declared $data as a [string] which is fine because probably some of your replace operations work better as a single string. Its just that $Results also then ends up being a string so when you try to index into $Results near the bottom these operations fail. You can however easily turn your $Results variable into a string array using the -split operator this would eliminate the need to save the string to disk and import back in just to accomplish the same. See comments below.
# Get the file
[System.IO.FileInfo] $File = 'C:\temp\TNSNames.ora'
[string] $data = (Get-Content $File.FullName | Where-Object { !$_.StartsWith('#') }).ToUpper()
# Convert the data. This part is where any (CONNECT_DATA entry ends up on it's own line.
$Results = $data.Replace(' ', '').Replace("`t", '').Replace(')))', ")))`n")
# You do not need to do this next section. Essentially this is just saving your multiline string
# to a file and then using Get-Content to read it back in as a string array
# Convert $Results from BaseType of System.Object to System.Array
# $Path = 'c:\temp\StringResults.txt'
# $Results | Out-File -FilePath $Path
# $Results = Get-Content $Path
# Instead split your $Results string into multiple lines using -split
# this will do the same thing as above without writing to file
$Results = $Results -split "\r?\n"
# Find all lines that start with '(CONNECT_DATA'
for ($i = 0; $i -lt $Results.Length - 1; $i++) {
if ($Results[$i + 1].StartsWith('(CONNECT_DATA')) {
# Add the '(CONNECT_DATA' line to the previous line
$Results[$i] = $Results[$i] + $Results[$i + 1]
# Blank out the '(CONNECT_DATA' line
$Results[$i + 1] = ''
}
}
# Remove all blank lines
$FinalForm = $null
foreach ($Line in $Results) {
if ($Line -ne '') {
$FinalForm += "$Line`n"
}
}
$FinalForm
Also, for fun, try this out
((Get-Content 'C:\temp\tnsnames.ora' |
Where-Object {!$_.StartsWith('#') -and ![string]::IsNullOrWhiteSpace($_)}) -join '' -replace '\s' -replace '\)\s?\)\s?\)', ")))`n" -replace '\r?\n\(Connect_data','(connect_data').ToUpper()

Powershell - Store hash table in file and read its content

As follow-up, suggested by Doug, on my previous question on anonymizing file (
PowerShell - Find and replace multiple patterns to anonymize file) I need to save all hash tables values in single file "tmp.txt" for further processing.
Example: after processing the input file with string like:
<requestId>>qwerty-qwer12-qwer56</requestId>
the tmp.txt file contains:
qwerty-qwer12-qwer56 : RequestId-1
and this is perfect. The problem is when working with many strings, in the tmp.txt file there are more pairs than there should be. In my example below in tmp.txt I should see 4 times the "RequestId-x" but there are 6. Also when there are 2 or more "match" on the same line, only the first is updated/replaced. Any idea from where these extra lines comes from? Any why the script doesn't continue to check till the end of the same line?
Here is my test code:
$log = "C:\log.txt"
$tmp = "C:\tmp.txt"
Clear-Content $log
Clear-Content $tmp
#'
<requestId>qwerty-qwer12-qwer56</requestId>qwertykeyId>Qwd84lPhjutf7Nmwr56hJndcsjy34imNQwd84lPhjutZ7Nmwr56hJndcsjy34imNPozDr5</ABC reportId>poGd56Hnm9q3Dfer6Jh</msg:reportId>
<requestId>zxcvbn-zxcv12-zxcv56</requestId>
<requestId>qwerty-qwer12-qwer56</requestId>abcde reportId>plmkjh8765FGH4rt6As</msg:reportId>
<requestId>1234qw-12qw12-12qw56</requestId>
keyId>Qwd84lPhjutf7Nmwr56hJndcsjy34imNQwd84lPhjutZ7Nmwr56hJndcsjy34imNPozDr5</
keyId>Qwd84lPhjutf7Nmwr56hJndcsjy34imNQwd84lPhjutZ7Nmwr56hJndcsjy34imNPozDr5</
keyId>Zdjgi76Gho3sQw0ib5Mjk3sDyoq9zmGdZdjgi76Gho3sQw0ib5Mjk3sDyoq9zmGdLkJpQw</
reportId>plmkjh8765FGH4rt6As</msg:reportId>
reportId>plmkjh8765FGH4rt6As</msg:reportId>
reportId>poGd56Hnm9q3Dfer6Jh</msg:reportId>
'# | Set-Content $log -Encoding UTF8
$requestId = #{
Count = 1
Matches = #()
}
$keyId = #{
Count = 1
Matches = #()
}
$reportId = #{
Count = 1
Matches = #()
}
$output = switch -Regex -File $log {
'(\w{6}-\w{6}-\w{6})' {
if(!$requestId.matches.($matches.1))
{
$req = $requestId.matches += #{$matches.1 = "RequestId-$($requestId.count)"}
$requestId.count++
$req.keys | %{ Add-Content $tmp "$_ : $($req.$_)" }
}
$_ -replace $matches.1,$requestId.matches.($matches.1)
}
'keyId>(\w{70})</' {
if(!$keyId.matches.($matches.1))
{
$kid = $keyId.matches += #{$matches.1 = "keyId-$($keyId.count)"}
$keyId.count++
$kid.keys | %{ Add-Content $tmp "$_ : $($kid.$_)" }
}
$_ -replace $matches.1,$keyId.matches.($matches.1)
}
'reportId>(\w{19})</msg:reportId>' {
if(!$reportId.matches.($matches.1))
{
$repid = $reportId.matches += #{$matches.1 = "Report-$($reportId.count)"}
$reportId.count++
$repid.keys | %{ Add-Content $tmp "$_ : $($repid.$_)" }
}
$_ -replace $matches.1,$reportId.matches.($matches.1)
}
default {$_}
}
$output | Set-Content $log -Encoding UTF8
Get-Content $log
Get-Content $tmp
If you don't care about the order in which they were found, which I assume you wouldn't if you don't want duplicates, just export them all at the end. I would still keep them in an "object" form so you can easily import/export them. Csv would be an ideal candidate for the data.
$requestId,$keyid,$reportid | Foreach-Object {
foreach($key in $_.matches.keys)
{
[PSCustomObject]#{
Original = $key
Replacement = $_.matches.$key
}
}
}
The data output to console for this example
Original Replacement
-------- -----------
qwerty-qwer12-qwer56 RequestId-1
zxcvbn-zxcv12-zxcv56 RequestId-2
1234qw-12qw12-12qw56 RequestId-3
Qwd84lPhjutf7Nmwr56hJndcsjy34imNQwd84lPhjutZ7Nmwr56hJndcsjy34imNPozDr5 keyId-1
Zdjgi76Gho3sQw0ib5Mjk3sDyoq9zmGdZdjgi76Gho3sQw0ib5Mjk3sDyoq9zmGdLkJpQw keyId-2
poGd56Hnm9q3Dfer6Jh Report-1
plmkjh8765FGH4rt6As Report-2
Just pipe it into Export-Csv
$requestId,$keyid,$reportid | Foreach-Object {
foreach($key in $_.matches.keys)
{
[PSCustomObject]#{
Original = $key
Replacement = $_.matches.$key
}
}
} | Export-Csv $tmp -NoTypeInformation

Writing an output on a .txt file on Powershell

I found a little script to get all the local groups and members and it's working perfectly but I need to write the output on PowerShell.
Trap {"Error: $_"; Break;}
function EnumLocalGroup($LocalGroup) {
$Group = [ADSI]"WinNT://$strComputer/$LocalGroup,group"
"`r`n" + "Group: $LocalGroup"
$Members = #($Group.psbase.Invoke("Members"))
foreach ($Member In $Members) {
$Name = $Member.GetType().InvokeMember("Name", 'GetProperty', $Null, $Member, $Null)
$Name
}
}
$strComputer = gc env:computername
"Computer: $strComputer"
$computer = [adsi]"WinNT://$strComputer"
$objCount = ($computer.PSBase.Children | Measure-Object).Count
$i = 0
foreach ($adsiObj in $computer.PSBase.Children) {
switch -regex ($adsiObj.PSBase.SchemaClassName) {
"group" {
$group = $adsiObj.Name
EnumLocalGroup $group
}
}
$i++
}
I already tried this:
function EnumLocalGroup($LocalGroup) | Out-File -FilePath "E:\PS\Malik\group.txt"
But the code won't start if I do that. I also tried to use this whole Out-File line at the end of the code after the } but doesn't work either and this is the only solution I find on Internet.
If you want to incorporate logging into a function you need to put it into the function body, e.g.
function EnumLocalGroup($LocalGroup) {
....
$foo = 'something'
$foo # output returned by function
$foo | Add-Content 'log.txt' # output to log file
...
}
or
function EnumLocalGroup($LocalGroup) {
...
$foo = 'something'
$foo | Tee-Object 'log.txt' -Append # output goes to log file and StdOut
...
}
Otherwise you have to do the logging when you call the function:
EnumLocalGroup $group | Add-Content 'C:\log.txt'

Why Isn't This Counting Correctly | PowerShell

Right now, I have a CSV file which contains 3,800+ records. This file contains a list of server names, followed by an abbreviation stating if the server is a Windows server, Linux server, etc. The file also contains comments or documentation, where each line starts with "#", stating it is a comment. What I have so far is as follows.
$file = Get-Content .\allsystems.csv
$arraysplit = #()
$arrayfinal = #()
[int]$windows = 0
foreach ($thing in $file){
if ($thing.StartsWith("#")) {
continue
}
else {
$arraysplit = $thing.Split(":")
$arrayfinal = #($arraysplit[0], $arraysplit[1])
}
}
foreach ($item in $arrayfinal){
if ($item[1] -contains 'NT'){
$windows++
}
else {
continue
}
}
$windows
The goal of this script is to count the total number of Windows servers. My issue is that the first "foreach" block works fine, but the second one results in "$Windows" being 0. I'm honestly not sure why this isn't working. Two example lines of data are as follows:
example:LNX
example2:NT
if the goal is to count the windows servers, why do you need the array?
can't you just say something like
foreach ($thing in $file)
{
if ($thing -notmatch "^#" -and $thing -match "NT") { $windows++ }
}
$arrayfinal = #($arraysplit[0], $arraysplit[1])
This replaces the array for every run.
Changing it to += gave another issue. It simply appended each individual element. I used this post's info to fix it, sort of forcing a 2d array: How to create array of arrays in powershell?.
$file = Get-Content .\allsystems.csv
$arraysplit = #()
$arrayfinal = #()
[int]$windows = 0
foreach ($thing in $file){
if ($thing.StartsWith("#")) {
continue
}
else {
$arraysplit = $thing.Split(":")
$arrayfinal += ,$arraysplit
}
}
foreach ($item in $arrayfinal){
if ($item[1] -contains 'NT'){
$windows++
}
else {
continue
}
}
$windows
1
I also changed the file around and added more instances of both NT and other random garbage. Seems it works fine.
I'd avoid making another ForEach loop for bumping count occurrences. Your $arrayfinal also rewrites everytime, so I used ArrayList.
$file = Get-Content "E:\Code\PS\myPS\2018\Jun\12\allSystems.csv"
$arrayFinal = New-Object System.Collections.ArrayList($null)
foreach ($thing in $file){
if ($thing.StartsWith("#")) {
continue
}
else {
$arraysplit = $thing -split ":"
if($arraysplit[1] -match "NT" -or $arraysplit[1] -match "Windows")
{
$arrayfinal.Add($arraysplit[1]) | Out-Null
}
}
}
Write-Host "Entries with 'NT' or 'Windows' $($arrayFinal.Count)"
I'm not sure if you want to keep 'Example', 'example2'... so I have skipped adding them to arrayfinal, assuming the goal is to count "NT" or "Windows" occurrances
The goal of this script is to count the total number of Windows servers.
I'd suggest the easy way: using cmdlets built for this.
$csv = Get-Content -Path .\file.csv |
Where-Object { -not $_.StartsWith('#') } |
ConvertFrom-Csv
#($csv.servertype).Where({ $_.Equals('NT') }).Count
# Compatibility mode:
# ($csv.servertype | Where-Object { $_.Equals('NT') }).Count
Replace servertype and 'NT' with whatever that header/value is called.

Monitor a command and wait for it to complete before proceeding to next command?

I have written a PowerShell script that will:
grab all txt files from a directory
perform a line-by-line assessment of the first file (grabbing headers and appending, appending data to each line in file, saving to an output file)
for subsequent files, grab body (excluding header), append data, then add to output file
The problem is in the use of Add-Content where the process hangs so certain files don't get written because the output file is in use. I added a function (based on recommendations found in various places on StackExchange) that test the output file to determine if it is available for read-write. This seems like a 'brute-force' approach.
Is there a way to monitor the actual Add-Content process launched by PowerShell to identify when it is complete? Or is there some other way to disaggregate the code as written to use the process control commands in PowerShell?
Sample:
function IsFileAccessible([String]$FullFileName) {
[Boolean]$IsAccessible = $false
try {
[IO.File]::OpenWrite($FullFileName).Close();
$IsAccessible = $true
} catch {
$IsAccessible = $false
}
return $IsAccessible
}
cd '[filepath]'
del old_output.type
$filearray = #()
$files = Get-ChildItem '[filepath]' -Filter "*.txt"
$outfile = 'new_output.type'
for ($i=0; $i -lt $files.Count; $i++) {
# Define variables
$lastWriteTime = $files[$i].LastWriteTime
# Define process steps for appending data
filter Add-Time {"$_$lastWriteTime"}
if ($i -eq 0) {
$lines = Get-Content $files[$i]
for ($j=0;$j -lt $lines.Count; $j++) {
if ($j -eq 0) {
$appended_txt = 'New_Header'
filter Add-Header{"$_$appended_txt"}
$lines[$j] | Add-Header | Add-Content $outfile
} else {
$lines[$j] | Add-Time | Add-Content $outfile
}
}
} else {
do {
$ErrorActionPreference = 'SilentlyContinue'
$test = IsFileAccessible('[filepath-new_output.type]')
echo 'file open'
} until ($test -eq 'True')
$ErrorActionPreference = 'Continue'
echo 'okay'
(Get-Content $files[$i].FullName | Select-Object -Skip 1) |
Add-Time | Add-Content $outfile
}
}