Related
I have a text file that contains a string I want to modify.
Example text file contents:
abc=1
def=2
ghi=3
If I run this code:
$file = "c:\test.txt"
$MinX = 100
$MinY = 100
$a = (Get-Content $file) | %{
if($_ -match "def=(\d*)"){
if($Matches[1] -gt $MinX){$_ -replace "$($Matches[1])","$($MinX)" }
}
}
$a
The result is:
def=100
If I omit the greater-than check like so:
$a = (Get-Content $file) | %{
if($_ -match "def=(\d*)"){
$_ -replace "$($Matches[1])","$($MinX)"
}
}
$a
The result is correct:
abc=1
def=100
ghi=3
I don't understand how a simple integer comparison before doing the replace could screw things up so badly, can anyone advise what I'm missing?
The comparison operator -gt will never get you a value of $true because you need to
cast the $matches[1] string value to int first so it compares two integer numbers
2 is never greater than 100.. Change the operator to -lt instead.
Your code outputs only one line, because you forgot to also output unchanged lines that do not match the regex
$file = 'c:\test.txt'
$MinX = 100
$MinY = 100
$a = (Get-Content $file) | ForEach-Object {
if ($_ -match '^def=(\d+)'){
if([int]$matches[1] -lt $MinX){ $_ -replace $matches[1],$MinX }
}
else {
$_
}
}
$a
Or use switch (is also faster than using Get-Content):
$file = 'c:\test.txt'
$MinX = 100
$MinY = 100
$a = switch -Regex -File $file {
'^def=(\d+)' {
if([int]$matches[1] -lt $MinX){ $_ -replace $matches[1],$MinX }
}
default { $_ }
}
$a
Output:
abc=1
def=100
ghi=3
That's because the expression ($Matches[1] -gt $MinX) is a string comparison. In Powershell, the left-hand side of a comparison dictates the comparison type and since that is of type [string], Powershell has to cast/convert the right-hand side of the expression to [string] also. You expression, therefore, is evaluated as ([string]$Matches[1] -gt [string]$MinX).
ProcessName UserName PSComputerName
AnyDesk NT-AUTORITÄT\SYSTEM localhost
csrss dc-01
ctfmon SAD\Administrator rdscb-01
SAD\Administrator srv-01
Remove the second and last row here
Based on your comments, if $data is read from a CSV file and contains custom objects, you can do the following:
$data | where { $_.PsObject.Properties.Value -notcontains $null -and $_.PsObject.Properties.Value -notcontains '' }
This will apply to every property and won't require supplying named properties.
There are more elegant ways, but, here is a kind of ugly answer, to illustrate this...
$Data = #"
"ProcessName","UserName","PSComputerName"
"AnyDesk","NT-AUTORITÄT\SYSTEM","localhost"
"csrss","","dc-01"
"ctfmon","SAD\Administrator","rdscb-01"
"","SAD\Administrator","srv-01"
"# | Out-File -FilePath 'D:\Temp\ProcData.csv'
$headers = (
(Get-Content -Path 'D:\Temp\ProcData.csv') -replace '"','' |
select -First 1
) -split ','
$data = Import-Csv -Path 'D:\Temp\ProcData.csv'
$colCnt = $headers.count
$lineNum = 0
:newline
foreach ($line in $data)
{
$lineNum++
for ($i = 0; $i -lt $colCnt; $i++)
{
# test to see if contents of a cell is empty
if (-not $line.$($headers[$i]))
{
Write-Warning -Message "$($lineNum): $($headers[$i]) is blank"
continue newline
}
}
"$($lineNum): OK"
# Perform other actions with good data
}
<#
# Results
1: OK
WARNING: 2: UserName is blank
3: OK
WARNING: 4: ProcessName is blank
#>
param([string]$roles,[string]$members)
Suppose I am passing input on the command line like this:
PS> role1,role2,role3,role4 member1,member2,,,,member3,,member4
The array I expect for this would be:
$array = #(
#('role1', 'member1,member2'),
#('role2', ''),
#('role3', 'member3'),
#('role4', 'member4')
)
I know to turn string to array:
$roles = 'role1,role2,role3,role4' -split ','
$members = 'member1,member2,,,,member3,,member4' -split ',,'
Now how do I combine $roles with $members so that each role will be associated with member(s)? and how wouldIi generate the array dynamically?
Pseudocode:
$array = #()
($roles+$members) | %{
$role = $_.roles
if ($_.members) {
$_.members -split ',,' | ForEach-Object { $array += $role $_ }
} else {
$array += $role
}
}
Note: I am splitting members as an index of its own for each double comma because apparently semicolons aren't accepted on a command line because they break the command line, so I have to use double comma as delimiter.
Note 2: notice the 4 commas: ,,,, this indicates that role2 does not have members to add, so in essence it means between the 4 commas is no input for member to that index/item (role2), i.e. ,,EMPTY,,.
If you really want to stick with this parameter format, you can create the desired output array as follows:
$roles = 'role1,role2,role3,role4' -split ','
$members = 'member1,member2,,,,member3,,member4' -split ',,'
$i = 0
$array = #(foreach ($role in $roles) {
, ($role, $members[$i++])
})
Note that if you pass your arguments from PowerShell, you need to quote them, as PowerShell will otherwise parse them as an array.
And with quoting you're free to use ; in lieu of ,,, for instance, to separate the member groups.
A better way to represent the argument data for later processing is to create an array of custom objects rather than a nested array:
$roles = 'role1,role2,role3,role4' -split ','
$members = 'member1,member2,,,,member3,,member4' -split ',,'
$i = 0
$array = #(foreach ($role in $roles) {
[pscustomobject] #{
Role = $role
Members = $members[$i++] -split ','
}
})
Each object in $array now has a .Role and a .Members property, the latter containing the individual members as a an array of strings.
Alternatively, you could create a[n ordered] hashtable from the input, keyed by role name, but that is only necessary if you need to access roles by name or if you wanted to rule out duplicate roles having been specified.
Here's an alternative argument format that is easier to understand:
$rolesAndMembers = 'role1 = member1,member2 ; role2= ; role3=member3 ; role4=member4'
$array = #(foreach ($roleAndMembers in ($rolesAndMembers -replace ' ' -split ';')) {
$role, $members = $roleAndMembers -split '='
[pscustomobject] #{
Role = $role
Members = $members -split ','
}
})
Your parameter format is rather bizarre, but here's one way:
$roles = 'role1,role2,role3,role4' -split ','
$members = 'member1,member2,,,,member3,,member4' -split ',,'
$result = #()
for ( $i = 0; $i -lt $roles.Count; $i++ ) {
$result += ,#($roles[$i],$members[$i])
}
I would recommend redesigning the script to use standard PowerShell parameters (the engineering effort would be worth it, IMO).
I'd strongly recommend using hashtables/dictionaries to pass these role mappings:
param(
[System.Collections.IDictionary]$RoleMembers
)
# now we can access each mapping by role name:
$RoleMembers['role1'] # member1, member2
# or iterate over them like an array:
foreach($role in $RoleMembers.Keys){
$RoleMembers[$role]
}
You could use one of the construct the input argument from your current input strings:
$roles = 'role1,role2,role3,role4' -split ','
$members = 'member1,member2,,,,member3,,member4' -split ','
$roleMembers = #{}
for ($i = 0; $i -lt $roles.Count; $i++) {
# `Where Length -ne 0` to filter out empty strings
$roleMembers[$roles[$i]] = $members[($i*2)..($i*2+1)] |Where Length -ne 0
}
I have a source file with the below contents:
0
ABC
1
181.12
2
05/07/16
4
Im4thData
5
hello
-1
0
XYZ
1
1333.21
2
02/02/16
3
Im3rdData
5
world
-1
...
The '-1' in above lists is the record separator which indicates the start of the next record. 0,1,2,3,4,5 etc are like column identifiers (or column names).
This is my code below.
$txt = Get-Content 'C:myfile.txt' | Out-String
$txt -split '(?m)^-1\r?\n' | ForEach-Object {
$arr = $_ -split '\r?\n'
$indexes = 1..$($arr.Count - 1) | Where-Object { ($_ % 2) -ne 0 }
$arr[$indexes] -join '|'
}
The above code creates output like below:
ABC|181.12|05/07/16|Im4thData|hello
XYZ|1333.21|02/02/16|Im3rdData|World
...
But I need output like below. When there are no columns in the source file, then their row data should have blank pipe line (||) like below in the output file. Please advise the change needed in the code.
ABC|181.12|05/07/16||Im4thData|hello ← There is no 3rd column in the source file. so blank pipe line (||).
XYZ|1333.21|02/02/16|Im3rdData||World ← There is no 4th column column in the source file. so blank pipe line (||).
...
If you know the maximum number of columns beforehand you could do something like this:
$cols = 6
$txt = Get-Content 'C:myfile.txt' | Out-String
$txt -split '(?m)^-1\r?\n' | ForEach-Object {
# initialize array of required size
$row = ,$null * $cols
$arr = $_ -split '\r?\n'
for ($n = 0; $n -lt $arr.Count; $n += 2) {
$i = [int]$arr[$n]
$row[$i] = $arr[$n+1]
}
$row -join '|'
}
Otherwise you could do something like this:
$txt = Get-Content 'C:myfile.txt' | Out-String
$txt -split '(?m)^-1\r?\n' | ForEach-Object {
# create empty array
$row = #()
$arr = $_ -split '\r?\n'
$k = 0
for ($n = 0; $n -lt $arr.Count; $n += 2) {
$i = [int]$arr[$n]
# if index from record ($i) is greater than current index ($k) append
# required number of empty fields
for ($j = $k; $j -lt $i-1; $j++) { $row += $null }
$row += $arr[$n+1]
$k = $i
}
$row -join '|'
}
Needs quite a bit of processing. There might be a more efficient way to do this, but the below does work.
$c = Get-Content ".\file.txt"
$rdata = #{}
$data = #()
$i = 0
# Parse the file into an array of key-value pairs
while ($i -lt $c.count) {
if($c[$i].trim() -eq '-1') {
$data += ,$rdata
$rdata = #{}
$i++
continue
}
$field = $c[$i].trim()
$value = $c[++$i].trim()
$rdata[$field] = $value
$i++
}
# Check if there are any missing values between 0 and the highest value and set to empty string if so
foreach ($row in $data) {
$top = [int]$($row.GetEnumerator() | Sort-Object Name -descending | select -First 1 -ExpandProperty Name)
for($i = 0; $i -lt $top; $i++) {
if ($row["$i"] -eq $null) {
$row["$i"] = ""
}
}
}
# Sort each hash by field order and join with pipe
$data | ForEach-Object { ($_.GetEnumerator() | Sort-Object -property Name | Select-Object -ExpandProperty Value) -join '|' }
In the while loop, we are just iterating over each line of the file. The field number an value are separated by a value of one, so each iteration we take both values and add them to the hash.
If we encounter -1 then we know we have a record separator, so add the hash to an array, reset it, bump the counter to the next record and continue to the next iteration.
Once we've collected everything we need to check if there are any missing field values, so we grab the highest number from each hash, loop over it from 0 and fill any missing values with an empty string.
Once that is done you can then iterate the array, sort each hash by field number and join the values.
I have two csv files. They both have SamAccountName in common. User records may or may not have a match found for every record between both files (THIS IS VERY IMPORTANT TO NOTE).
I am trying to basically just merge all columns (and their values) into one file (based from the SamAccountNames found in the first file...).
If the SamAccountName is not found in the 2nd file, it should add all null values for that user record in the merged file (since the record was found in the first file).
If the SamAccountName is found in the 2nd file, but not in the first, it should ignore merging that record.
Number of columns in each file may vary (5, 10, 2, so forth...).
Function MergeTwoCsvFiles
{
Param ([String]$baseFile, [String]$fileToBeMerged, [String]$columnTitleLineInFileToBeMerged)
$baseFileCsvContents = Import-Csv $baseFile
$fileToBeMergedCsvContents = Import-Csv $fileToBeMerged
$baseFileContents = Get-Content $baseFile
$baseFileContents[0] += "," + $columnTitleLineInFileToBeMerged
$baseFileCsvContents | ForEach-Object {
$matchFound = $False
$baseSameAccountName = $_.SamAccountName
[String]$mergedLineInFile = $_
[String]$lineMatchFound = $fileToBeMergedCsvContents | Where-Object {$_.SamAccountName -eq $baseSameAccountName}
Write-Host '$mergedLineInFile =' $mergedLineInFile
Write-Host '$lineMatchFound =' $lineMatchFound
Exit
}
}
The problem is, the record in the file is being written as a hash table instead of a string like line (if you were to view it as .txt). So I'm not really sure how to do this...
Adding results csv example files...
First CSV File
"SamAccountName","sn","GivenName"
"PBrain","Pinky","Brain"
"JSteward","John","Steward"
"JDoe","John","Doe"
"SDoo","Scooby","Doo"
Second CSV File
"SamAccountName","employeeNumber","userAccountControl","mail"
"KYasunori","678213","546","KYasunori#mystuff.com"
"JSteward","43518790","512","JSteward#mystuff.com"
"JKibogabi","24356","546","JKibogabi#mystuff.com"
"JDoe","902187u4","1114624","JDoe#mystuff.com"
"CStrife","54627","512","CStrife#mystuff.com"
Expected Merged CSV File
"SamAccountName","sn","GivenName","employeeNumber","userAccountControl","mail"
"PBrain","Pinky","Brain","","",""
"JSteward","John","Steward","43518790","512","JSteward#mystuff.com"
"JDoe","John","Doe","902187u4","1114624","JDoe#mystuff.com"
"SDoo","Scooby","Doo","","",""
Note: This will be part of a loop process in merging multiple files, so I would like to avoid hardcoding the title names (with $_.SamAccountName as an exception)
Trying suggestion from "restless 1987" (Not Working)
$baseFileCsvContents = Import-Csv 'D:\Scripts\Powershell\Tests\base.csv'
$fileToBeMergedCsvContents = Import-Csv 'D:\Scripts\Powershell\Tests\lookup.csv'
$resultsFile = 'D:\Scripts\Powershell\Tests\MergedResults.csv'
$resultsFileContents = #()
$baseFileContents = Get-Content 'D:\Scripts\Powershell\Tests\base.csv'
$recordsMatched = compare-object $baseFileCsvContents $fileToBeMergedCsvContents -Property SamAccountName
switch ($recordsMatched)
{
'<=' {}
'=>' {}
'==' {$resultsFileContents += $_}
}
$resultsFileCsv = $resultsFileContents | ConvertTo-Csv
$resultsFileCsv | Export-Csv $resultsFile -NoTypeInformation -Force
Output gives a blank file :(
The code below outputs the desired results based on the inputs you provided.
function CombineSkip1($s1, $s2){
$s3 = $s1 -split ','
$s2 -split ',' | select -Skip 1 | % {$s3 += $_}
$s4 = $s3 -join ', '
$s4
}
Write-Output "------Combine files------"
# content
$c1 = Get-Content D:\junk\test1.csv
$c2 = Get-Content D:\junk\test2.csv
# users in both files, could be a better way to do this
$t1 = $c1 | ConvertFrom-Csv
$t2 = $c2 | ConvertFrom-Csv
$users = $t1 | Select SamAccountName
# generate final, combined output
$combined = #()
$combined += CombineSkip1 $c1[0] $c2[0]
$c2PropCount = ($c2[0] -split ',').Count - 1
$filler = (', ""' * $c2PropCount)
for ($i = 1; $i -lt $c1.Count; $i++){
$user = $c1[$i].Split(',')[0]
$u2 = $c2 | where {([string]$_).StartsWith($user)}
if ($u2)
{
$combined += CombineSkip1 $c1[$i] $u2
}
else
{
$combined += ($c1[$i] + $filler)
}
}
# write to output and file
Write-Output $combined
$combined | Set-Content -Path D:\junk\test3.csv -Force
You can use compare-object for that purpose. Use -property samaccountname with it. For example:
$a = 1,2,3,4,5
$b = 4,5,6,7
$side = compare-object $a $b
switch ($side){
'<=' {is not in $a}
'=>' {is not in $b}
'==' { is on both sides}
}
When you have all the data in your output-variable, trow it at convertto-csv and write it in a file
After an entire day, I finally came up with something that works...
...
Edit
Reason: breaking the inner loop and removing the found element from the array will be much faster when merging files with thousands of records...
Function GetTitlesFromFileToBeMerged
{
Param ($csvFile)
[String]$fileToBeMergedTitles = Get-Content $fileToBeMerged -TotalCount 1
[String[]]$fileToBeMergedTitles = ($fileToBeMergedTitles -replace "`",`"", "|").Trim()
[String[]]$fileToBeMergedTitles = ($fileToBeMergedTitles -replace "`"", "").Trim()
[String[]]$fileToBeMergedTitles = ($fileToBeMergedTitles -replace "SamAccountName", "").Trim()
[String[]]$listOfColumnTitles = $fileToBeMergedTitles.Split('|',[System.StringSplitOptions]::RemoveEmptyEntries)
Write-Output $listOfColumnTitles
}
$baseFile = 'D:\Scripts\Powershell\Tests\base.csv'
$fileToBeMerged = 'D:\Scripts\Powershell\Tests\lookup.csv'
$baseFileCsvContents = Import-Csv $baseFile
$baseFileContents = Get-Content $baseFile
$fileToBeMergedCsvContents = Import-Csv $fileToBeMerged
[System.Collections.Generic.List[System.Object]]$fileToBeMergedContents = Get-Content $fileToBeMerged
$resultsFile = 'D:\Scripts\Powershell\Tests\MergedResults.csv'
$resultsFileContents = #()
[String]$baseFileTitles = $baseFileContents[0]
[String]$fileToBeMergedTitles = (Get-Content $fileToBeMerged -TotalCount 1) -replace "`"SamAccountName`",", ""
$resultsFileContents += $baseFileTitles + "," + $fileToBeMergedTitles
[String]$lineMatchNotFound = ""
$arrayFileToBeMergedTitles = GetTitlesFromFileToBeMerged $fileToBeMerged
For ($valueNum = 0; $valueNum -lt $arrayFileToBeMergedTitles.Length; $valueNum++)
{
$lineMatchNotFound += ",`"`""
}
$baseLineCounter = 1
$baseFileCsvContents | ForEach-Object {
$baseSameAccountName = $_.SamAccountName
[String]$baseLineInFile = $baseFileContents[$baseLineCounter]
$lineMatchCounter = 1
$lineMatchFound = ""
:inner
ForEach ($line in $fileToBeMergedContents) {
If ($line -like "*$baseSameAccountName*") {
[String]$lineMatchFound = "," + ($line -replace '^"[^"]*",', "")
$fileToBeMergedContents.RemoveAt($lineMatchCounter)
break inner
}; $lineMatchCounter++
}
If (!($lineMatchFound))
{
[String]$lineMatchFound = $lineMatchNotFound
}
$mergedLine = $baseLineInFile + $lineMatchFound
$resultsFileContents += $mergedLine
$baseLineCounter++
}
ForEach ($line in $resultsFileContents)
{
Write-Host $line
}
$resultsFileContents | Set-Content $resultsFile -Force
I'm very sure this is not the best approach and there is something better that would handle this much faster. If anyone has any ideas, I'm open to them. Thanks.