Right now, I have a CSV file which contains 3,800+ records. This file contains a list of server names, followed by an abbreviation stating if the server is a Windows server, Linux server, etc. The file also contains comments or documentation, where each line starts with "#", stating it is a comment. What I have so far is as follows.
$file = Get-Content .\allsystems.csv
$arraysplit = #()
$arrayfinal = #()
[int]$windows = 0
foreach ($thing in $file){
if ($thing.StartsWith("#")) {
continue
}
else {
$arraysplit = $thing.Split(":")
$arrayfinal = #($arraysplit[0], $arraysplit[1])
}
}
foreach ($item in $arrayfinal){
if ($item[1] -contains 'NT'){
$windows++
}
else {
continue
}
}
$windows
The goal of this script is to count the total number of Windows servers. My issue is that the first "foreach" block works fine, but the second one results in "$Windows" being 0. I'm honestly not sure why this isn't working. Two example lines of data are as follows:
example:LNX
example2:NT
if the goal is to count the windows servers, why do you need the array?
can't you just say something like
foreach ($thing in $file)
{
if ($thing -notmatch "^#" -and $thing -match "NT") { $windows++ }
}
$arrayfinal = #($arraysplit[0], $arraysplit[1])
This replaces the array for every run.
Changing it to += gave another issue. It simply appended each individual element. I used this post's info to fix it, sort of forcing a 2d array: How to create array of arrays in powershell?.
$file = Get-Content .\allsystems.csv
$arraysplit = #()
$arrayfinal = #()
[int]$windows = 0
foreach ($thing in $file){
if ($thing.StartsWith("#")) {
continue
}
else {
$arraysplit = $thing.Split(":")
$arrayfinal += ,$arraysplit
}
}
foreach ($item in $arrayfinal){
if ($item[1] -contains 'NT'){
$windows++
}
else {
continue
}
}
$windows
1
I also changed the file around and added more instances of both NT and other random garbage. Seems it works fine.
I'd avoid making another ForEach loop for bumping count occurrences. Your $arrayfinal also rewrites everytime, so I used ArrayList.
$file = Get-Content "E:\Code\PS\myPS\2018\Jun\12\allSystems.csv"
$arrayFinal = New-Object System.Collections.ArrayList($null)
foreach ($thing in $file){
if ($thing.StartsWith("#")) {
continue
}
else {
$arraysplit = $thing -split ":"
if($arraysplit[1] -match "NT" -or $arraysplit[1] -match "Windows")
{
$arrayfinal.Add($arraysplit[1]) | Out-Null
}
}
}
Write-Host "Entries with 'NT' or 'Windows' $($arrayFinal.Count)"
I'm not sure if you want to keep 'Example', 'example2'... so I have skipped adding them to arrayfinal, assuming the goal is to count "NT" or "Windows" occurrances
The goal of this script is to count the total number of Windows servers.
I'd suggest the easy way: using cmdlets built for this.
$csv = Get-Content -Path .\file.csv |
Where-Object { -not $_.StartsWith('#') } |
ConvertFrom-Csv
#($csv.servertype).Where({ $_.Equals('NT') }).Count
# Compatibility mode:
# ($csv.servertype | Where-Object { $_.Equals('NT') }).Count
Replace servertype and 'NT' with whatever that header/value is called.
Related
I have a LARGE list of hashes. I need to find out which ones only appear once as most are duplicates.
EX: the last line 238db2..... only appears once
ac6b51055fdac5b92934699d5b07db78
ac6b51055fdac5b92934699d5b07db78
7f5417a85a63967d8bba72496faa997a
7f5417a85a63967d8bba72496faa997a
1e78ba685a4919b7cf60a5c60b22ebc2
1e78ba685a4919b7cf60a5c60b22ebc2
238db202693284f7e8838959ba3c80e8
I tried the following that just listed one of each of the doubles, not just identifying the one that only appeared once
foreach ($line in (Get-Content "C:\hashes.txt" | Select-Object -Unique)) {
Write-Host "Line '$line' appears $(($line | Where-Object {$_ -eq $line}).count) time(s)."
}
You could use a Hashtable and a StreamReader.
The StreamReader reads the file line-by-line and the Hashtable will store that line as Key and in its Value state $true (if this is a duplicate) or $false (if it is unique)
$reader = [System.IO.StreamReader]::new('D:\Test\hashes.txt')
$hash = #{}
while($null -ne ($line = $reader.ReadLine())) {
$hash[$line] = $hash.ContainsKey($line)
}
# clean-up the StreamReader
$reader.Dispose()
# get the unique line(s) by filtering for value $false
$result = $hash.Keys | Where-Object {-not $hash[$_]}
Given your example, $result will contain 238db202693284f7e8838959ba3c80e8
Given that you're dealing with a large file, Get-Content is best avoided.
A switch statement with the -File parameter allows efficient line-by-line processing, and given that duplicates appear to be grouped together already, they can be detected by keeping a running count of identical lines.
$count = 0 # keeps track of the count of identical lines occurring in sequence
switch -File 'C:\hashes.txt' {
default {
if ($prevLine -eq $_ -or $count -eq 0) { # duplicate or first line.
if ($count -eq 0) { $prevLine = $_ }
++$count
}
else { # current line differs from the previous one.
if ($count -eq 1) { $prevLine } # non-duplicate -> output
$prevLine = $_
$count = 1
}
}
}
if ($count -eq 1) { $prevLine } # output the last line, if a non-duplicate.
$values = Get-Content .\hashes.txt # Read the values from the hashes.txt file
$groups = $values | Group-Object | Where-Object { $_.Count -eq 1 } # Group the values by their distinct values and filter for groups with a single value
foreach ($group in $groups) {
foreach ($value in $group.Values) {
Write-Host "$value" # Output the value of each group
}
}
To handle very large files you could try this.
$chunkSize = 1000 # Set the chunk size to 1000 lines
$lineNumber = 0 # Initialize a line number counter
# Use a do-while loop to read the file in chunks
do {
# Read the next chunk of lines from the file
$values = Get-Content .\hashes.txt | Select-Object -Skip $lineNumber -First $chunkSize
# Group the values by their distinct values and filter for groups with a single value
$groups = $values | Group-Object | Where-Object { $_.Count -eq 1 }
foreach ($group in $groups) {
foreach ($value in $group.Values) {
Write-Host "$value" # Output the value of each group
}
}
# Increment the line number counter by the chunk size
$lineNumber += $chunkSize
} while ($values.Count -eq $chunkSize)
Or this
# Create an empty dictionary
$dict = New-Object System.Collections.Hashtable
# Read the file line by line
foreach ($line in Get-Content .\hashes.txt) {
# Check if the line is already in the dictionary
if ($dict.ContainsKey($line)) {
# Increment the value of the line in the dictionary
$dict.Item($line) += 1
} else {
# Add the line to the dictionary with a count of 1
$dict.Add($line, 1)
}
}
# Filter the dictionary for values with a count of 1
$singles = $dict.GetEnumerator() | Where-Object { $_.Value -eq 1 }
# Output the values of the single items
foreach ($single in $singles) {
Write-Host $single.Key
}
So I'm new to the Powershell scripting world and I'm trying to compare a list of IPs in text file against a database of IP list. If an IP from (file) does not exist in the (database) file put it in a new file, let's call it compared.txt. When I tried to run the script, I didn't get any result. What am I missing here?
$file = Get-Content "C:\Users\zack\Desktop\file.txt"
$database = Get-Content "C:\Users\zack\Desktop\database.txt"
foreach($line1 in $file){
$check = 0
foreach($line2 in $database)
{
if($line1 != $line2)
{
$check = 1
}
else
{
$check = 0
break
}
}
if ($check == 1 )
{
$line2 | Out-File "C:\Users\zack\Desktop\compared.txt"
}
}
There is a problem with your use of PowerShell comparison operators unlike in C#, equality and inequality are -eq and -ne, and since PowerShell is a case insensitive language, there is also -ceq and -cne.
There is also a problem with your code's logic, a simple working version of it would be:
$database = Get-Content "C:\Users\zack\Desktop\database.txt"
# iterate each line in `file.txt`
$result = foreach($line1 in Get-Content "C:\Users\zack\Desktop\file.txt") {
# iterate each line in `database.txt`
# this happens on each iteration of the outer loop
$check = foreach($line2 in $database) {
# if this line of `file.txt` is the same as this line of `database.txt`
if($line1 -eq $line2) {
# we don't need to keep checking, output this boolean
$true
# and break the inner loop
break
}
}
# if above condition was NOT true
if(-not $check) {
# output this line, can be `$line1` or `$line2` (same thing here)
$line1
}
}
$result | Set-Content path\to\comparisonresult.txt
However, there are even more simplified ways you could achieve the same results:
Using containment operators:
$database = Get-Content "C:\Users\zack\Desktop\database.txt"
$result = foreach($line1 in Get-Content "C:\Users\zack\Desktop\file.txt") {
if($line1 -notin $database) {
$line1
}
}
$result | Set-Content path\to\comparisonresult.txt
Using Where-Object:
$database = Get-Content "C:\Users\zack\Desktop\database.txt"
Get-Content "C:\Users\zack\Desktop\file.txt" | Where-Object { $_ -notin $database } |
Set-Content path\to\comparisonresult.txt
Using a HashSet<T> and it's ExceptWith method (Note, this will also get rid of duplicates in your file.txt):
$file = [System.Collections.Generic.HashSet[string]]#(
Get-Content "C:\Users\zack\Desktop\file.txt"
)
$database = [string[]]#(Get-Content "C:\Users\zack\Desktop\database.txt")
$file.ExceptWith($database)
$file | Set-Content path\to\comparisonresult.txt
I hope you can help me out. I work with two ArrayLists:
array1 is filled with log.csv (contains header and logon-data, values of column 'pc' are unique). It's defined as ArrayList because I want to add entries.
#pc,name,date,city
#pc-x,name-1,2017-01-01,berlin
#pc-y,name-1,2017-01-02,berlin
#pc-z,name-2,2017-01-02,munich
[System.Collections.ArrayList]$array1 = Import-Csv log.csv
array2 is filled during runtime
$array2=[System.Collections.ArrayList]#()
... ForEach-Object {
$array2.Add([PSCustomObject]#{ pc=$pcname
name=$loginname
date=$logindate }) >$null }
What I want to do:
Update array1.date=array2.date where array1.pc=array2.pc
If no entry found in array1 I want to add it:
$array1.Add([PSCustomObject]#{ pc=$pcname
name=$loginname
date=$logindate
city='[unknown]' }) >$null
Finally array1 is exported again:
$array1 | Export-Csv log.csv -Encoding UTF8 -NoTypeInformation
So the question is: how can I find the entry in array1 and update it? Trying hard for days now ...
try Something like this:
$array1=import-csv "C:\temp\log.csv"
$array2=import-csv "C:\temp\log2.csv"
#modify founded and output not founded
$toadd=$array2 | %{
$current=$_
$founded=$array1 | where pc -eq $current.pc | %{$_.date=$current.date;$_}
if ($founded -eq $null)
{
$current.city='UNKNOW'
$current
}
}
#output of $array1 modified and elements to add
$array1, $toadd
Here is a sample I created that might help. Note: I use List types instead of ArrayList ones. Also, it assumes only one possible matching PC name in the data to be updated. You'll have to alter it to update the file since it merely updates the first List variable. Let me know how it goes.
[PSCustomObject]
{
[string] $pc,
[string] $name,
[string] $date,
[string] $city
}
[System.Collections.Generic.List[PSCustomObject]] $list1 = Import-Csv "C:\SOSamples\log.csv";
[System.Collections.Generic.List[PSCustomObject]] $list2 = Import-Csv "C:\SOSamples\log2.csv";
[PSCustomObject] $record = $null;
[PSCustomObject] $match = $null;
foreach($record in $list2)
{
# NOTE: This only retrieves the FIRST MATCHING item using a CASE-INSENSITIVE comparison
$match = $list1 | Where { $_.pc.ToLower() -eq $record.pc.ToLower() } | Select -First 1;
if($match -eq $null)
{
Write-Host "Not Found!";
$list1.Add($record);
}
else
{
Write-Host "Found!";
$match.date = $record.date;
}
}
Write-Host "--------------------------------------------------------------------"
foreach($record in $list1)
{
Write-Host $record
}
In my current powershell script I have hash table with values. Am using
this syntax
$x = $f.contains("$k")
but I figured recently that am having problems with this approach I was wondering if powershell has something that says "starts with," or related, that would search thru the hash table with "starts with" instead of contains
Example of the hash table:
"bio.txt" = "server1\datafiles\bio";
etc.......
EDIT Sample from comments
foreach ($key in $filehash.keys) {
$path = $filehash.get_Item($key)
$filecount = 0
foreach ($file in $FileArray) {
if ($file.LastWriteTime -lt($(GetDate).adddays(-1))) {
[string] $k = $key.ToLower()
[string] $f = $file.name.ToLower()
if ($x = $f.contains("$k")) { }
}
}
}
Try using -like to check if a string starts with yourvalue. I rewrote your sample in the comments to use it:
$filehash.GetEnumerator() | foreach {
#$_ is now current object from hashtable(like foreach)
#$_.key is key and $_.value is path
$filecount = 0
foreach ($file in $FileArray) {
if ( ($file.LastWriteTime -lt $((Get-Date).AddDays(-1))) -and ($file.name -like "$($_.Key)*") ) {
#process file
}
}
}
I'm facing the problem of moving and copying some items on the file system with PowerShell.
I know by experiments the fact that, even with PowerShell v3, the cmdlet Copy-Item, Move-Item and Delete-Item cannot handle correctly reparse point like junction and symbolic link, and can lead to disasters if used with switch -Recurse.
I want to prevent this evenience. I have to handle two or more folder each run, so I was thinking to something like this.
$Strings = #{ ... }
$ori = Get-ChildItem $OriginPath -Recurse
$dri = Get-ChildItem $DestinationPath -Recurse
$items = ($ori + $dri) | where { $_.Attributes -match 'ReparsePoint' }
if ($items.Length -gt 0)
{
Write-Verbose ($Strings.LogExistingReparsePoint -f $items.Length)
$items | foreach { Write-Verbose " $($_.FullName)" }
throw ($Strings.ErrorExistingReparsePoint -f $items.Length)
}
This doen't work because $ori and $dri can be also single items and not arrays: the op-Addition will fail. Changing to
$items = #(#($ori) + #($dri)) | where { $_.Attributes -match 'ReparsePoint' }
poses another problem because $ori and $dri can also be $null and I can end with an array containing $null. When piping the join resutl to Where-Object, again, I can end with a $null, a single item, or an array.
The only apparently working solution is the more complex code following
$items = $()
if ($ori -ne $null) { $items += #($ori) }
if ($dri -ne $null) { $items += #($dri) }
$items = $items | where { $_.Attributes -match 'ReparsePoint' }
if ($items -ne $null)
{
Write-Verbose ($Strings.LogExistingReparsePoint -f #($items).Length)
$items | foreach { Write-Verbose " $($_.FullName)" }
throw ($Strings.ErrorExistingReparsePoint -f #($items).Length)
}
There is some better approch?
I'm interested for sure if there is a way to handle reparse point with PowerShell cmdlets in the correct way, but I'm much more interested to know how to join and filters two or more "PowerShell collections".
I conclude observing that, at present, this feature of PowerShell, the "polymorphic array", doen't appear such a benefit to me.
Thanks for reading.
Just add a filter to throw out nulls. You're on the right track.
$items = #(#($ori) + #($dri)) | ? { $_ -ne $null }
I've been on Powershell 3 for a while now but from what I can tell this should work in 2.0 as well:
$items = #($ori, $dri) | %{ $_ } | ? { $_.Attributes -match 'ReparsePoint' }
Basically %{ $_ } is a foreach loop that unrolls the inner arrays by iterating over them and passing each inner element ($_) down the pipeline. Nulls will automatically be excluded from the pipeline.