Problem assign data from a file read to variables - powershell

This is a PowerShell question, sorry if it wound up in the wrong place.
I have a data file which I will call PS_Upgrade_Data. It has 6 items in it listed like this:
item1=item-2=item.3=item 4=item5=item6
Please note that items 2 through 4 are written that way due to the data coming in over which I have no control. There is a dash in item2, a period in intem3 and a space in item4 just to be clear. Items 1, 5, and 6 have nothing between the word item and the number.
I am using the follow PS line to read the data in from the file:
Get-Content "P:\PS_Upgrade_Data.txt" | Where-Object {$_.length -gt 0} | Where-Object {!$_.StartsWith("#")} | Foreach-Object -begin {$count1=0} -process {$var = $_.Split('=',6).Trim()}
This does read the data from the file in ok.
I want to take that data and drop it into six (6) different variables and I tried a foreach loop like the one below:
$count1 = 0
ForEach ($item in $var) {
Write-Host "`n"
Write-Host "count = " , $count1
if($count1 = 0) {
Write-Host "UserInit"
$UserInit = $item
}
elseif ($count1 = 1) {
Write-Host "UserInit"
$TicketNumber = $item
}
elseif ($count1 = 2) {
Write-Host "UserInit"
$UpgradeVersion = $item
}
elseif ($count1 = 3) {
Write-Host "UserInit"
$ClientName = $item
}
elseif ($count1 = 4) {
Write-Host "UserInit"
$InstName = $item
}
elseif ($count1 = 5) {
Write-Host "UserInit"
$Coffee = $item
}
$count1 +=1
}
The problem is that the counter is jumping from 0 (zero) to two (2) and then wont increase and I have no idea why.
What stupid thing am I doing or missing?
Thanks for any help.

PowerShell's assignment operator = supports multiple assignment targets per operation, so you can skip the counter and foreach loop and instead, simply do:
Get-Content "P:\PS_Upgrade_Data.txt" | Where-Object {$_.length -gt 0} | Where-Object {!$_.StartsWith("#")} | Foreach-Object {
$UserInit,$TicketNumber,$UpgradeVersion,$ClientName,$InstName,$Coffee = $_.Split('=',6).Trim()
# do with your variables what you want here
}

Related

how to find unique line in a txt file?

I have a LARGE list of hashes. I need to find out which ones only appear once as most are duplicates.
EX: the last line 238db2..... only appears once
ac6b51055fdac5b92934699d5b07db78
ac6b51055fdac5b92934699d5b07db78
7f5417a85a63967d8bba72496faa997a
7f5417a85a63967d8bba72496faa997a
1e78ba685a4919b7cf60a5c60b22ebc2
1e78ba685a4919b7cf60a5c60b22ebc2
238db202693284f7e8838959ba3c80e8
I tried the following that just listed one of each of the doubles, not just identifying the one that only appeared once
foreach ($line in (Get-Content "C:\hashes.txt" | Select-Object -Unique)) {
Write-Host "Line '$line' appears $(($line | Where-Object {$_ -eq $line}).count) time(s)."
}
You could use a Hashtable and a StreamReader.
The StreamReader reads the file line-by-line and the Hashtable will store that line as Key and in its Value state $true (if this is a duplicate) or $false (if it is unique)
$reader = [System.IO.StreamReader]::new('D:\Test\hashes.txt')
$hash = #{}
while($null -ne ($line = $reader.ReadLine())) {
$hash[$line] = $hash.ContainsKey($line)
}
# clean-up the StreamReader
$reader.Dispose()
# get the unique line(s) by filtering for value $false
$result = $hash.Keys | Where-Object {-not $hash[$_]}
Given your example, $result will contain 238db202693284f7e8838959ba3c80e8
Given that you're dealing with a large file, Get-Content is best avoided.
A switch statement with the -File parameter allows efficient line-by-line processing, and given that duplicates appear to be grouped together already, they can be detected by keeping a running count of identical lines.
$count = 0 # keeps track of the count of identical lines occurring in sequence
switch -File 'C:\hashes.txt' {
default {
if ($prevLine -eq $_ -or $count -eq 0) { # duplicate or first line.
if ($count -eq 0) { $prevLine = $_ }
++$count
}
else { # current line differs from the previous one.
if ($count -eq 1) { $prevLine } # non-duplicate -> output
$prevLine = $_
$count = 1
}
}
}
if ($count -eq 1) { $prevLine } # output the last line, if a non-duplicate.
$values = Get-Content .\hashes.txt # Read the values from the hashes.txt file
$groups = $values | Group-Object | Where-Object { $_.Count -eq 1 } # Group the values by their distinct values and filter for groups with a single value
foreach ($group in $groups) {
foreach ($value in $group.Values) {
Write-Host "$value" # Output the value of each group
}
}
To handle very large files you could try this.
$chunkSize = 1000 # Set the chunk size to 1000 lines
$lineNumber = 0 # Initialize a line number counter
# Use a do-while loop to read the file in chunks
do {
# Read the next chunk of lines from the file
$values = Get-Content .\hashes.txt | Select-Object -Skip $lineNumber -First $chunkSize
# Group the values by their distinct values and filter for groups with a single value
$groups = $values | Group-Object | Where-Object { $_.Count -eq 1 }
foreach ($group in $groups) {
foreach ($value in $group.Values) {
Write-Host "$value" # Output the value of each group
}
}
# Increment the line number counter by the chunk size
$lineNumber += $chunkSize
} while ($values.Count -eq $chunkSize)
Or this
# Create an empty dictionary
$dict = New-Object System.Collections.Hashtable
# Read the file line by line
foreach ($line in Get-Content .\hashes.txt) {
# Check if the line is already in the dictionary
if ($dict.ContainsKey($line)) {
# Increment the value of the line in the dictionary
$dict.Item($line) += 1
} else {
# Add the line to the dictionary with a count of 1
$dict.Add($line, 1)
}
}
# Filter the dictionary for values with a count of 1
$singles = $dict.GetEnumerator() | Where-Object { $_.Value -eq 1 }
# Output the values of the single items
foreach ($single in $singles) {
Write-Host $single.Key
}

Powershell - Exchange JSON output without needing to write to a file

EDIT: Added Setupconfigfiles.ps1
I'm a bit new to detailed scripting so please bear with me.
I have two Powershell scripts
Setupconfigfiles.ps1 generates JSON output to be fed to an API.
Script2 uses that JSON data to execute API commands.
Script 2 can call setupconfigfiles.ps1 as indicated below and use the output data.
.\SetupConfigFiles.ps1 -type $Type -outfile .\Templist.json
$servers = Get-Content -Raw -Path .\templist.json | ConvertFrom-Json
setupconfigfiles.ps1:
param (
# If this parameter is set, format the output as csv.
# If this parameter is not set, just return the output so that the calling program can use the info
[string]$outfile,
# this parameter can be 'production', 'development' or 'all'
[string]$type
)
enum MachineTypes {
production = 1
development = 2
all = 3
}
$Servers = Get-ADObject -Filter 'ObjectClass -eq "computer"' -SearchBase 'Obfuscated DSN' | Select-Object Name
$output = #()
$count = 0
# Set this to [MachineTypes]::production or [MachineTypes]::development or [MachineTypes]::all
if ($type -eq "all") {
$server_types = [MachineTypes]::all
}
ElseIf ($type -eq "production") {
$server_types = [MachineTypes]::production
}
else {
$server_types = [MachineTypes]::development
}
ForEach ($Server in $Servers)
{
$count = $count + 1
$this_server = #{}
$this_server.hostname = $Server.Name
$this_server.id = $count
$this_server."site code" = $this_server.hostname.substring(1,3)
$this_server."location code" = $this_server.hostname.substring(4,2)
if ($this_server.hostname.substring(7,1) -eq "P") {
$this_server.environment = "Production"
}
ElseIf ($this_server.hostname.substring(7,1) -eq "D") {
$this_server.environment = "Development"
}
Else {
$this_server.environment = "Unknown"
}
if (($server_types -eq [MachineTypes]::production ) -and ($this_server.environment -eq "Production")) {
$output += $this_server
}
ElseIf (($server_types -eq [MachineTypes]::development ) -and ($this_server.environment -eq "Development")) {
$output += $this_server
}
Else {
if ($server_types -eq [MachineTypes]::all ) {
$output += $this_server
}
}
}
if ($outfile -eq "")
{
ConvertTo-Json $output
}
else {
ConvertTo-Json $output | Out-File $outfile
}
How can I do it without needing to write to the Templist.json file?
I've called this many different ways. The one I thought would work is .\SetupConfigFiles.ps1 $servers
Y'all are great. #Zett42 pointed me in a direction and #Mathias rounded it out.
The solution was to change:
"ConvertTo-Json $output" to "Write-Output $output"
Then it's handled in the calling script.
thanks!

Speeding up Test-Path and GCI on large data set

I have a large list of Customer IDs (40,000). Files for each Customer have been saved in many different locations. I first run a Test-Path to see if the directory exists, if it does then I run Get-ChildItem and filter down the results to find the file I need (Any file matching string 'Contract'). I am hoping to educate myself on how to speed this up, I have attempted to introduce another variable 'Trigger' to try and prevent some of the excess code from running if the matching file is found. It is taking a very long time to loop this through 40,000 times so if there is a better way any help greatly appreciated, many thanks.
Here is the code I'm currently using
ForEach ($Customer in $CustomerList)
{
$Trigger = 0
$Result1 = Test-Path "$Location1\$Customer"
$Result2 = Test-Path "$Location2\$Customer"
$Result3 = Test-Path "$Location3\$Customer"
IF ($Result1 -eq $True)
{
$1Files = GCI "$Location1\$Customer" -Recurse
ForEach ($File IN $1Files)
{
IF ($Trigger -eq 0)
{
$FileName = $File.Name
$FileLocation = $File.FullName
IF ($FileName -match 'Contract')
{
$Report += "$FileName $FileLocation"
$Trigger = 1
}
}
}
}
ELSEIF ($Result2 -eq True)
{
Same as result 1 codeblock but using $Location2
}
ELSEIF ($Result3 -eq True)
{
Same as result 1 codeblock but using $Location3
}
}

Powershell export an imported CSV as fixed record length with no delimiter?

I have a pretty standard CSV file which could look something like this:
heading1,heading2,heading3
aaaaaaaaa,bb,ccccccc
d,eeeeeeee,ff
gggggggg,hh,iiiiiiiiiii
This file is imported to an object using import-csv. I now want to export that object to a file which has a fixed record length with no delimiters and no table headers. If the imported values are too long for the fixed file, they should be cut off. It the imported values are too short, the values should be left aligned and filled with spaces.
Assuming the width of:
heading1 is 5
heading2 is 2
heading3 is 10
basically the output should look like this:
aaaaabbccccccc
d eeff
ggggghhiiiiiiiiii
Please note the spaces at the end of the values in column3.
The algorithms should not be completely inefficient - it will be used for converting a 300MB csv file.
I searched on Stackoverflow and googled for while and could find some related questions using solutions like a custom table format and format-table, but these solutions did not seem to be easily adaptable for me to this specific problem.
As to why: this very ugly/unusual format is required by a niche COTS software.
## Q:\Test\2018\07\10\SO_51265871.ps1
$SPC = ' ';
Import-Csv .\Input.csv |
ForEach-Object {"[{0}{1}{2}]" -f ($_.heading1+$SPC).Substring(0,5),
($_.heading2+$SPC).Substring(0,2),
($_.heading3+$SPC).Substring(0,10)
} | Set-Content .\Output.rec
The [] in the format string are just to show the length including trailing spaces.
Sample output:
PS> Get-Content .\Output.rec
[aaaaabbccccccc ]
[d eeff ]
[ggggghhiiiiiiiiii]
EDIT: a variant a bit more general, feeding column width from an array - same output
$CW = #(5,2,10) # array CW = ColumnWidth
Import-Csv .\Input.csv | ForEach-Object { $i = 0
"[{0}{1}{2}]" -f `
($_.heading1).PadRight($CW[$i]).Substring(0,$CW[$i++]),
($_.heading2).PadRight($CW[$i]).Substring(0,$CW[$i++]),
($_.heading3).PadRight($CW[$i]).Substring(0,$CW[$i])
} #| Set-Content .\Output.rec
I don't really know why you would want to do this but hey you do so fair play.
I think I catch what youa re trying to do and this will give you an array with each line being that processed data into the format you wanted, you can then loop that array into a txt file, log file, whatever you want.
$spacerCSVFile = "spacers.csv"
$prinspacerCSV = Import-Csv $spacerCSVFile -header "1","2","3"
$processedArray = New-Object System.Collections.Generic.List[System.Object]
foreach($row in $prinspacerCSV) {
if ($row.1 -like "heading*") {
# do nothing for headings
} else {
$item1 = $row.1
$item2 = $row.2
$item3 = $row.3
while ($item1.length -lt 6) {
$item1 += " "
}
while ($item2.length -lt 3) {
$item2 += " "
}
while ($item3.length -lt 11) {
$item3 += " "
}
if ($item1.length -gt 5) {
$item1 = $item1.substring(0,5)
}
if ($item2.length -gt 2) {
$item2 = $item2.substring(0,2)
}
if ($item3.length -gt 10) {
$item3 = $item3.substring(0,10)
}
$processedArray += ,"$item1$item2$item3"
}
}
foreach ($item in $processedArray) {
write-host $item
}
I would advise however that it is worth giving it a go and coming here with some code in place first as the idea of stack overflow is to help with your code not provide it.
This is very ugly and might be more complicated than what it needs to be but this is working from what I can tell.
Hope this gives you either an idea or this just helps
$file = "C:\Logs\Test.csv"
$data = Import-Csv $file
$properties = $data |
Get-Member |
Where-Object MemberType -EQ NoteProperty |
Select Name
foreach ($line in $data){
[string]$dataout = $null
foreach ($property in $properties) {
$dataout = $dataout + $line.($property.Name)
}
if($dataout.Length -eq 20){
$dataout
}
elseif ($dataout.Length -lt 20) {
Do{
$dataout = $dataout + " "
}
Until($dataout.Length -eq 20)
$dataout
}
else {
$dataout = ($dataout.Substring(0,20))
$dataout
}
}

Why Isn't This Counting Correctly | PowerShell

Right now, I have a CSV file which contains 3,800+ records. This file contains a list of server names, followed by an abbreviation stating if the server is a Windows server, Linux server, etc. The file also contains comments or documentation, where each line starts with "#", stating it is a comment. What I have so far is as follows.
$file = Get-Content .\allsystems.csv
$arraysplit = #()
$arrayfinal = #()
[int]$windows = 0
foreach ($thing in $file){
if ($thing.StartsWith("#")) {
continue
}
else {
$arraysplit = $thing.Split(":")
$arrayfinal = #($arraysplit[0], $arraysplit[1])
}
}
foreach ($item in $arrayfinal){
if ($item[1] -contains 'NT'){
$windows++
}
else {
continue
}
}
$windows
The goal of this script is to count the total number of Windows servers. My issue is that the first "foreach" block works fine, but the second one results in "$Windows" being 0. I'm honestly not sure why this isn't working. Two example lines of data are as follows:
example:LNX
example2:NT
if the goal is to count the windows servers, why do you need the array?
can't you just say something like
foreach ($thing in $file)
{
if ($thing -notmatch "^#" -and $thing -match "NT") { $windows++ }
}
$arrayfinal = #($arraysplit[0], $arraysplit[1])
This replaces the array for every run.
Changing it to += gave another issue. It simply appended each individual element. I used this post's info to fix it, sort of forcing a 2d array: How to create array of arrays in powershell?.
$file = Get-Content .\allsystems.csv
$arraysplit = #()
$arrayfinal = #()
[int]$windows = 0
foreach ($thing in $file){
if ($thing.StartsWith("#")) {
continue
}
else {
$arraysplit = $thing.Split(":")
$arrayfinal += ,$arraysplit
}
}
foreach ($item in $arrayfinal){
if ($item[1] -contains 'NT'){
$windows++
}
else {
continue
}
}
$windows
1
I also changed the file around and added more instances of both NT and other random garbage. Seems it works fine.
I'd avoid making another ForEach loop for bumping count occurrences. Your $arrayfinal also rewrites everytime, so I used ArrayList.
$file = Get-Content "E:\Code\PS\myPS\2018\Jun\12\allSystems.csv"
$arrayFinal = New-Object System.Collections.ArrayList($null)
foreach ($thing in $file){
if ($thing.StartsWith("#")) {
continue
}
else {
$arraysplit = $thing -split ":"
if($arraysplit[1] -match "NT" -or $arraysplit[1] -match "Windows")
{
$arrayfinal.Add($arraysplit[1]) | Out-Null
}
}
}
Write-Host "Entries with 'NT' or 'Windows' $($arrayFinal.Count)"
I'm not sure if you want to keep 'Example', 'example2'... so I have skipped adding them to arrayfinal, assuming the goal is to count "NT" or "Windows" occurrances
The goal of this script is to count the total number of Windows servers.
I'd suggest the easy way: using cmdlets built for this.
$csv = Get-Content -Path .\file.csv |
Where-Object { -not $_.StartsWith('#') } |
ConvertFrom-Csv
#($csv.servertype).Where({ $_.Equals('NT') }).Count
# Compatibility mode:
# ($csv.servertype | Where-Object { $_.Equals('NT') }).Count
Replace servertype and 'NT' with whatever that header/value is called.