Replace statement creating duplicate lines - powershell

I'm using the code below
$content | ForEach-Object {
$addresseeName = $_.Substring(11,50)
$example = 'X' * 50
if ($_.Substring(0,10) -eq 'ADDRESSEE0')
{
$_ -replace $addresseeName, $example
}
else
{
$_
}
$addresseeName2 = $_.Substring(62,50)
$example = 'Y' * 50
if ($_.Substring(0,10) -eq 'ADDRESSEE0')
{
$_ -replace $addresseeName2, $example
}
else
{
$_
}
Text file before:
ABSINVEST2|Y|N|
ADDRESSEE0|Name Name Letters Letters & Financial | |19 Greemount House |
ADDRSDETAS| | |
Text File After:
ABSINVEST2|Y|N|
ADDRESSEE0|XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX| |19 Greemount House
ADDRESSEE0|Samp less Independent Examples & Exampless s |XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX|19 Greemount House
ADDRSDETAS| | |
I am attempting to mask particular fields (positions). Can anyone help me as to why the line is being duplicated for every IF Else statement?
Updating code I got working:
$content |
ForEach-Object {
$addresseeName = $_.Substring(11,50)
$addresseeName2 = $_.Substring(62,50)
$example = 'X' * 50
if ($_.Substring(0,10) -eq 'ADDRESSEE0')
{
$_.replace($addresseeName, $example).replace($addresseeName2, $example)
}

The short answer is, because you are duplicating it (your current script outputs it twice). Try this instead:
$content | ForEach-Object {
$addresseeName = $_.Substring(11,50)
$example = 'X' * 50
if ($_.Substring(0,10) -eq 'ADDRESSEE0')
{
$_ = $_ -replace $addresseeName, $example
}
$addresseeName2 = $_.Substring(62,50)
$example = 'Y' * 50
if ($_.Substring(0,10) -eq 'ADDRESSEE0')
{
$_ = $_ -replace $addresseeName2, $example
}
$_
}
This modifies the current line variable $_ by setting it when performing the replace and then only outputting it once at the end (whether it has changed or not).

Related

How would you generate a unique sequence from 000 to 9ZZ

How would you generate a unique sequence from 000 to 9ZZ. Lastly, my export-csv is not working. Please see the data and export output below.
The alphanumeric sequence starts from 0 through 9 and then A through Z.
Please be advised, my PowerShell skill are a bit new. :)
$i = #()
$a = 0..9
$b = 65..90 | Foreach{[Char]$_}
$i = $a + $b
For($d = 0; $d -le $i.count; $d++){
$g = $i[$d]
For($e = 0; $e -le $i.count; $e++){
$h = $i[$e]
For($f = 0; $f -le $i.count; $f++){
$j = $i[$f]
$k = "{0}{1}{2}" -f $g, $h, $j
$k #| Export-Csv -Path .\List.csv -NoTypeInformation -Append
If($k -eq '9ZZ'){
Break
}
}
}
}
Data Output:
000
001
.
.
|
V
009
00A
.
.
00Z
00 <-- I don't get this.
010
Export:
Length
3 <-- I don't get this either.
.
.
|
v
3
Any and all help is appreciated. Thank you in Advanced. ;)
I would do it this way, with 3 foreach loops and a labeled break:
$digits = 0..9
$chars = (65..90).ForEach([char])
$dict = $digits + $chars
$result = :outer foreach($i in $dict)
{
foreach($x in $dict)
{
foreach($z in $dict)
{
'{0}{1}{2}' -f $i,$x,$z
if($i -eq 9 -and $x -eq 'Z' -and $z -eq 'Z')
{
break outer
}
}
}
}
$result | Export-Csv ...
This is a classic off-by-1 error - your loops run from 0 through $i.count (included), which is exactly 1 longer than $i.
Change -le $i.count to -lt $i.count in all 3 loop conditions and it'll work.
You could simplify your code by generating the full range of digits/characters up front a little differently, and then use 3 nested foreach loops instead:
$digits = '0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ'.ToCharArray()
$ranges =
:outerLoop
foreach($a in $digits){
foreach($b in $digits){
foreach($c in $digits){
# save and output new value
($label = "${a}${b}${c}")
# exit the label generation completely if we've reached the desired upper boundary
if($label -eq '9ZZ'){ break outerLoop }
}
}
}
$ranges now contain the correct range of labels from 000 through 9ZZ
To complement the helpful existing answers with a generalized solution for producing permutations of characters with a given number of places, using recursive helper function Get-Permutations:
function Get-Permutations {
param(
[Parameter(Mandatory)]
[string] $Chars, # e.g. '0123456789'
[Parameter(Mandatory)]
[uint] $NumPlaces # e.g. 2, to produce '00', '01', ..., '99'
)
switch ($NumPlaces) {
0 { return }
1 { [string[]] $Chars.ToCharArray() }
default {
Get-Permutations -Chars $Chars -NumPlaces ($NumPlaces-1) | ForEach-Object {
foreach ($c in $chars.ToCharArray()) { $_ + $c }
}
}
}
}
You'd call it as follows (to get '000', '001', ..., but all the way up to 'ZZZ' - you can post-filter with ... | Where-Object { $_ -le '9ZZ' })
Get-Permutations '0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ' 3
Note:
In PowerShell (Core) 7+ you can create the string of characters with
-join ('0'..'9' + 'A'..'Z')
In Windows PowerShell, where you cannot use [char] instances with the range operator, .., you can use the following, more concise alternative to your approach:
-join ([char[]] (48..57) + [char[]] (65..90)), based on the chars.' code points obtained with, e.g. [int] [char] 'A'
similarly you could cheat since '009' -le '00A'
$sb = [System.Text.StringBuilder]::new()
function inc([string]$s){
[byte[]]$v = $s.ToCharArray()
$radix = $v.Count-1
$carry = $true
while($carry -and $radix -ge 0){
$carry = $false
switch(++$v[$radix]){
91{$v[$radix--] = 48; $carry = $true}
58{$v[$radix] = 65}
}
}
$null = $sb.clear()
if($carry){$null = $sb.Append('1')}
return $sb.Append([char[]]$v).ToString()
}
Describe 'Test inc function' {
It 'increments 000' {inc '000' | should be '001'}
It 'increments 001' {inc '001' | should be '002'}
It 'increments 008' {inc '008' | should be '009'}
It 'increments 009' {inc '009' | should be '00A'}
It 'increments 00A' {inc '00A' | should be '00B'}
It 'increments 00Y' {inc '00Y' | should be '00Z'}
It 'increments 00Z' {inc '00Z' | should be '010'}
It 'increments 010' {inc '010' | should be '011'}
It 'increments 0ZZ' {inc '0ZZ' | should be '100'}
It 'increments 1ZZ' {inc '1ZZ' | should be '200'}
It 'increments ZZZ' {inc 'ZZZ' | should be '1000'}
}
measure-command {for($i = '000'; $i -le '9ZZ'; $i = inc $i){$i}}
&{for($i = '000'; $i -le '9ZZ'; $i = inc $i){
$i
}} | set-content -Path .\List.csv
Edit: unnecessarily faster
Inspired by mklement0's awesome answer
$cache = #{}
function Get-Permutations{
param(
[Parameter(Mandatory=$true)][string]$Alphabet # e.g. '0123456789'
,[Parameter(Mandatory=$true)][uint32]$Length # e.g. 2, to produce '00', '01', ..., '99'
)
if($Length -eq 0)
{return}
if($Length -eq 1)
{return $Alphabet.ToCharArray()}
if($Alphabet -NotIn $cache.Keys){
$cache.$Alphabet = new-object String[][] $Length
$cache.$Alphabet[1] = $Alphabet.ToCharArray()
}elseif($cache.$Alphabet.Count -lt $Length){
$tmp = new-object String[][] $Length
[Array]::Copy($cache.$Alphabet,$tmp,$cache.$Alphabet.Count)
$cache.$Alphabet = $tmp
}
Get-Permutations-Helper $Alphabet $Length
}
function Get-Permutations-Helper{
param(
[string]$Alphabet
,[uint32]$Length
)
$TailLength = $Length-1
if($cache.$Alphabet[$TailLength] -eq $null){
$cache.$Alphabet[$TailLength] = Get-Permutations-Helper $Alphabet $TailLength
}
foreach($head in $Alphabet.ToCharArray()){
foreach($tail in $cache.$Alphabet[$TailLength]){
$head + $tail
}
}
}
Describe 'Test Get-Permutations function' {
It 'Makes an ordered ABC,2 permutation' {
$valid = #('AA','AB','AC','BA','BB','BC','CA','CB','CC')
$cache = #{}; $i = 0
Get-Permutations 'ABC' 2 | foreach-object {
$_ | should be $valid[$i++]
}
}
It 'Extends the cache' {
$cache = #{}
$null = Get-Permutations 'ABC' 2
$cache.'ABC'[1][0] | should be 'A'
$cache.'ABC'[1][0] = 'test'
$results = Get-Permutations 'ABC' 3
$cache.'ABC'[2][0] | should be 'Atest'
$results[0] | should be 'AAtest'
}
}
$digits = '0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ'
$spin = 10
$ttl=$null;for($i=0;$i -lt $spin;$i++){
$cache = #{}
$ttl+=measure-command {
Get-Permutations $digits 3
}
};$ttl;$ttl.TotalMilliseconds/$spin
$ttl=$null;for($i=0;$i -lt $spin;$i++){
$cache = #{}
$ttl+=measure-command {
$enu = (Get-Permutations $digits 3).GetEnumerator();
while($enu.MoveNext() -and ($v = $enu.Current) -le '9ZZ'){$v}
}
};$ttl;$ttl.TotalMilliseconds/$spin
$enu = (Get-Permutations $digits 3).GetEnumerator();
&{while($enu.MoveNext() -and ($v = $enu.Current) -le '9ZZ'){
$v
}} | set-content -Path .\List.csv

Powershell - Store hash table in file and read its content

As follow-up, suggested by Doug, on my previous question on anonymizing file (
PowerShell - Find and replace multiple patterns to anonymize file) I need to save all hash tables values in single file "tmp.txt" for further processing.
Example: after processing the input file with string like:
<requestId>>qwerty-qwer12-qwer56</requestId>
the tmp.txt file contains:
qwerty-qwer12-qwer56 : RequestId-1
and this is perfect. The problem is when working with many strings, in the tmp.txt file there are more pairs than there should be. In my example below in tmp.txt I should see 4 times the "RequestId-x" but there are 6. Also when there are 2 or more "match" on the same line, only the first is updated/replaced. Any idea from where these extra lines comes from? Any why the script doesn't continue to check till the end of the same line?
Here is my test code:
$log = "C:\log.txt"
$tmp = "C:\tmp.txt"
Clear-Content $log
Clear-Content $tmp
#'
<requestId>qwerty-qwer12-qwer56</requestId>qwertykeyId>Qwd84lPhjutf7Nmwr56hJndcsjy34imNQwd84lPhjutZ7Nmwr56hJndcsjy34imNPozDr5</ABC reportId>poGd56Hnm9q3Dfer6Jh</msg:reportId>
<requestId>zxcvbn-zxcv12-zxcv56</requestId>
<requestId>qwerty-qwer12-qwer56</requestId>abcde reportId>plmkjh8765FGH4rt6As</msg:reportId>
<requestId>1234qw-12qw12-12qw56</requestId>
keyId>Qwd84lPhjutf7Nmwr56hJndcsjy34imNQwd84lPhjutZ7Nmwr56hJndcsjy34imNPozDr5</
keyId>Qwd84lPhjutf7Nmwr56hJndcsjy34imNQwd84lPhjutZ7Nmwr56hJndcsjy34imNPozDr5</
keyId>Zdjgi76Gho3sQw0ib5Mjk3sDyoq9zmGdZdjgi76Gho3sQw0ib5Mjk3sDyoq9zmGdLkJpQw</
reportId>plmkjh8765FGH4rt6As</msg:reportId>
reportId>plmkjh8765FGH4rt6As</msg:reportId>
reportId>poGd56Hnm9q3Dfer6Jh</msg:reportId>
'# | Set-Content $log -Encoding UTF8
$requestId = #{
Count = 1
Matches = #()
}
$keyId = #{
Count = 1
Matches = #()
}
$reportId = #{
Count = 1
Matches = #()
}
$output = switch -Regex -File $log {
'(\w{6}-\w{6}-\w{6})' {
if(!$requestId.matches.($matches.1))
{
$req = $requestId.matches += #{$matches.1 = "RequestId-$($requestId.count)"}
$requestId.count++
$req.keys | %{ Add-Content $tmp "$_ : $($req.$_)" }
}
$_ -replace $matches.1,$requestId.matches.($matches.1)
}
'keyId>(\w{70})</' {
if(!$keyId.matches.($matches.1))
{
$kid = $keyId.matches += #{$matches.1 = "keyId-$($keyId.count)"}
$keyId.count++
$kid.keys | %{ Add-Content $tmp "$_ : $($kid.$_)" }
}
$_ -replace $matches.1,$keyId.matches.($matches.1)
}
'reportId>(\w{19})</msg:reportId>' {
if(!$reportId.matches.($matches.1))
{
$repid = $reportId.matches += #{$matches.1 = "Report-$($reportId.count)"}
$reportId.count++
$repid.keys | %{ Add-Content $tmp "$_ : $($repid.$_)" }
}
$_ -replace $matches.1,$reportId.matches.($matches.1)
}
default {$_}
}
$output | Set-Content $log -Encoding UTF8
Get-Content $log
Get-Content $tmp
If you don't care about the order in which they were found, which I assume you wouldn't if you don't want duplicates, just export them all at the end. I would still keep them in an "object" form so you can easily import/export them. Csv would be an ideal candidate for the data.
$requestId,$keyid,$reportid | Foreach-Object {
foreach($key in $_.matches.keys)
{
[PSCustomObject]#{
Original = $key
Replacement = $_.matches.$key
}
}
}
The data output to console for this example
Original Replacement
-------- -----------
qwerty-qwer12-qwer56 RequestId-1
zxcvbn-zxcv12-zxcv56 RequestId-2
1234qw-12qw12-12qw56 RequestId-3
Qwd84lPhjutf7Nmwr56hJndcsjy34imNQwd84lPhjutZ7Nmwr56hJndcsjy34imNPozDr5 keyId-1
Zdjgi76Gho3sQw0ib5Mjk3sDyoq9zmGdZdjgi76Gho3sQw0ib5Mjk3sDyoq9zmGdLkJpQw keyId-2
poGd56Hnm9q3Dfer6Jh Report-1
plmkjh8765FGH4rt6As Report-2
Just pipe it into Export-Csv
$requestId,$keyid,$reportid | Foreach-Object {
foreach($key in $_.matches.keys)
{
[PSCustomObject]#{
Original = $key
Replacement = $_.matches.$key
}
}
} | Export-Csv $tmp -NoTypeInformation

How can I transpose and parse a large vertical text file into a CSV file with headers?

I have a large text file (*.txt) in the following format:
; KEY 123456
; Any Company LLC
; 123 Main St, Anytown, USA
SEC1 = xxxxxxxxxxxxxxxxxxxxx
SEC2 = xxxxxxxxxxxxxxxxxxxxx
SEC3 = xxxxxxxxxxxxxxxxxxxxx
SEC4 = xxxxxxxxxxxxxxxxxxxxx
SEC5 = xxxxxxxxxxxxxxxxxxxxx
SEC6 = xxxxxxxxxxxxxxxxxxxxx
This is repeated for about 350 - 400 keys. These are HASP keys and the SEC codes associated with them. I am trying to parse this file into a CSV file with KEY and SEC1 - SEC6 as the headers, with the rows being filled in. This is the format I am trying to get to:
KEY,SEC1,SEC2,SEC3,SEC4,SEC5,SEC6
123456,xxxxxxxxxx,xxxxxxxxxxx,xxxxxxxxxx,xxxxxxxxxx,xxxxxxxxxx,xxxxxxxxxx
456789,xxxxxxxxxx,xxxxxxxxxx,xxxxxxxxxx,xxxxxxxxxx,xxxxxxxxxx,xxxxxxxxxx
I have been able to get a script to export to a CSV with only one key in the text file (my test file), but when I try to run it on the full list, it only exports the last key and sec codes.
$keysheet = '.\AllKeys.txt'
$holdarr = #{}
Get-Content $keysheet | ForEach-Object {
if ($_ -match "KEY") {
$key, $value = $_.TrimStart("; ") -split " "
$holdarr[$key] = $value }
elseif ($_ -match "SEC") {
$key, $value = $_ -split " = "
$holdarr[$key] = $value }
}
$hash = New-Object PSObject -Property $holdarr
$hash | Export-Csv -Path '.\allsec.csv' -NoTypeInformation
When I run it on the full list, it also adds a couple of extra columns with what looks like properties instead of values.
Any help to get this to work would be appreciated.
Thanks.
Here's the approach I suggest:
$output = switch -Regex -File './AllKeys.txt' {
'^; KEY (?<key>\d+)' {
if ($o) {
[pscustomobject]$o
}
$o = #{
KEY = $Matches['key']
}
}
'^(?<sec>SEC.*?)\s' {
$o[$Matches['sec']] = ($_ | ConvertFrom-StringData)[$Matches['sec']]
}
default {
Write-Warning -Message "No match found: $_"
}
}
# catch the last object
$output += [pscustomobject]$o
$output | Export-Csv -Path './some.csv' -NoTypeInformation
This would be one approach.
& {
$entry = $null
switch -Regex -File '.\AllKeys.txt' {
"KEY" {
if ($entry ) {
[PSCustomObject]$entry
}
$entry = #{}
$key, $value = $_.TrimStart("; ") -split " "
$entry[$key] = [int]$value
}
"SEC" {
$key, $value = $_ -split " = "
$entry[$key] = $value
}
}
[PSCustomObject]$entry
} | sort KEY | select KEY,SEC1,SEC2,SEC3,SEC4,SEC5,SEC6 |
Export-Csv -Path '.\allsec.csv' -NoTypeInformation
Lets leverage the strength of ConvertFrom-StringData which
Converts a string containing one or more key and value pairs to a hash table.
So what we will do is
Split into blocks of text
edit the "; Key" line
Remove an blank lines or semicolon lines.
Pass to ConvertFrom-StringData to create a hashtable
Convert that to a PowerShell object
$path = "c:\temp\keys.txt"
# Split the file into its key/sec collections. Drop any black entries created in the split
(Get-Content -Raw $path) -split ";\s+KEY\s+" | Where-Object{-not [string]::IsNullOrWhiteSpace($_)} | ForEach-Object{
# Split the block into lines again
$lines = $_ -split "`r`n" | Where-Object{$_ -notmatch "^;" -and -not [string]::IsNullOrWhiteSpace($_)}
# Edit the first line so we have a full block of key=value pairs.
$lines[0] = "key=$($lines[0])"
# Use ConvertFrom-StringData to do the leg work after we join the lines back as a single string.
[pscustomobject](($lines -join "`r`n") | ConvertFrom-StringData)
} |
# Cannot guarentee column order so we force it with this select statement.
Select-Object KEY,SEC1,SEC2,SEC3,SEC4,SEC5,SEC6
Use Export-CSV to your hearts content now.

String trim and split

I have a text file that I read and I need to get the values from.
Example text file:
[Site 01]
DBServer=LocalHost
DBName=Database01
Username=admin
Password=qwerty
[Site 02]
DBServer=192.168.0.10
DBName=Database02
Username=admin
Password=qwerty
Currently my code reads through the file and places each each as an array entry for each line DBServer= that is found and this text file can have many sites:
$NumOfSites = Get-Content $Sites |
Select-String -Pattern "DBServer=" -Context 0,3
$i = 0
$NumOfSites | ForEach-Object {
$svr = $NumOfSites[$i] -isplit "\n" |
% { ($_ -isplit 'DBServer=').Trim()[1] }
$db = $NumOfSites[$i] -isplit "\n" |
% { ($_ -isplit 'DBName='.Trim())[1] }
$uid = $NumOfSites[$i] -isplit "\n" |
% { ($_ -isplit 'Username='.Trim())[1] }
$pswd = $NumOfSites[$i] -isplit "\n" |
% { ($_ -isplit 'Password='.Trim())[1] }
$i = $i+1
}
I can't get each attribute to split out properly without some extra spaces or something nicely as a string variable.
I just need to extract the info to put into an SQL connection line as variables from the format of the file example I have.
Other than the record headers (i.e. [Site 01]) the rest can be handled by ConvertFrom-StringData just fine. We can just convert the records to objects directly splitting on the header row more or less. ConvertFrom-StringData turns a multi-line string into a hashtable, and you can just cast that as a [PSCustomObject] and viola, you have objects that are easy to use.
$NumOfSites = Get-Content $Sites -raw
$SiteObjects = $NumOfSites -split '\[.+?\]'|%{[PSCustomObject](ConvertFrom-StringData -StringData $_)}
Then you can manipulate $SiteObjects however you see fit (output to CSV if you want, or filter on any property using Select-Object). Or, if you're looking to make connections you can loop through it building your connections as needed...
ForEach($Connection in $SiteObjects){
$ConStr = "Server = {0}; Database = {1}; Integrated Security = False; User ID = {2}; Password = {3};" -f $Connection.DBServer.Trim(), $Connection.DBName.Trim(), $Connection.Username.Trim(), $Connection.Password.Trim()
<Do stuff with SQL>
}
Edit: Updating my answer since the sample text was changed to add <pre> and </pre>. We just need to remove those, and since the OP is getting errors about methods on null values we'll filter for null as well.
$NumOfSites = Get-Content $Sites -raw
$SiteObjects = $NumOfSites -replace '<.*?>' -split '\[.+?\]' | ?{$_} |%{[PSCustomObject](ConvertFrom-StringData -StringData $_)}
ForEach($Connection in $SiteObjects){
$svr = $Connection.DBServer.Trim()
$db = $Connection.DBName.Trim()
$uid = $Connection.Username.Trim()
$pwd = $Connection.Password.Trim()
}
Here's a suggestion if you only care about getting the value after the equals:
Get-Content Example.txt |
ForEach-Object {
Switch -Regex ($_) {
'dbs.+=' { $svr = ($_ -replace '.+=').Trim()
.. etc ..
}
}
Get-Content piped to ForEach-Object will interpret each line as its own object.
Edit:
You were most of the way there, but it's unnecessary to -split the lines
$NumOfSites = Get-Content $Sites | Select-String -pattern "DBServer=" -Context 0,3
$NumOfSites | ForEach-Object {
Switch -Wildcard ($_) {
'DBS*=' { $svr = ($_ -replace '.+=').Trim() }
'DBN*=' { $db = ($_ -replace '.+=').Trim() }
'U*=' { $uid = ($_ -replace '.+=').Trim() }
'P*=' { $pw = ($_ -replace '.+=').Trim() }
}
}

How to merge all contents in two csv files where records match off 1 column

I have two csv files. They both have SamAccountName in common. User records may or may not have a match found for every record between both files (THIS IS VERY IMPORTANT TO NOTE).
I am trying to basically just merge all columns (and their values) into one file (based from the SamAccountNames found in the first file...).
If the SamAccountName is not found in the 2nd file, it should add all null values for that user record in the merged file (since the record was found in the first file).
If the SamAccountName is found in the 2nd file, but not in the first, it should ignore merging that record.
Number of columns in each file may vary (5, 10, 2, so forth...).
Function MergeTwoCsvFiles
{
Param ([String]$baseFile, [String]$fileToBeMerged, [String]$columnTitleLineInFileToBeMerged)
$baseFileCsvContents = Import-Csv $baseFile
$fileToBeMergedCsvContents = Import-Csv $fileToBeMerged
$baseFileContents = Get-Content $baseFile
$baseFileContents[0] += "," + $columnTitleLineInFileToBeMerged
$baseFileCsvContents | ForEach-Object {
$matchFound = $False
$baseSameAccountName = $_.SamAccountName
[String]$mergedLineInFile = $_
[String]$lineMatchFound = $fileToBeMergedCsvContents | Where-Object {$_.SamAccountName -eq $baseSameAccountName}
Write-Host '$mergedLineInFile =' $mergedLineInFile
Write-Host '$lineMatchFound =' $lineMatchFound
Exit
}
}
The problem is, the record in the file is being written as a hash table instead of a string like line (if you were to view it as .txt). So I'm not really sure how to do this...
Adding results csv example files...
First CSV File
"SamAccountName","sn","GivenName"
"PBrain","Pinky","Brain"
"JSteward","John","Steward"
"JDoe","John","Doe"
"SDoo","Scooby","Doo"
Second CSV File
"SamAccountName","employeeNumber","userAccountControl","mail"
"KYasunori","678213","546","KYasunori#mystuff.com"
"JSteward","43518790","512","JSteward#mystuff.com"
"JKibogabi","24356","546","JKibogabi#mystuff.com"
"JDoe","902187u4","1114624","JDoe#mystuff.com"
"CStrife","54627","512","CStrife#mystuff.com"
Expected Merged CSV File
"SamAccountName","sn","GivenName","employeeNumber","userAccountControl","mail"
"PBrain","Pinky","Brain","","",""
"JSteward","John","Steward","43518790","512","JSteward#mystuff.com"
"JDoe","John","Doe","902187u4","1114624","JDoe#mystuff.com"
"SDoo","Scooby","Doo","","",""
Note: This will be part of a loop process in merging multiple files, so I would like to avoid hardcoding the title names (with $_.SamAccountName as an exception)
Trying suggestion from "restless 1987" (Not Working)
$baseFileCsvContents = Import-Csv 'D:\Scripts\Powershell\Tests\base.csv'
$fileToBeMergedCsvContents = Import-Csv 'D:\Scripts\Powershell\Tests\lookup.csv'
$resultsFile = 'D:\Scripts\Powershell\Tests\MergedResults.csv'
$resultsFileContents = #()
$baseFileContents = Get-Content 'D:\Scripts\Powershell\Tests\base.csv'
$recordsMatched = compare-object $baseFileCsvContents $fileToBeMergedCsvContents -Property SamAccountName
switch ($recordsMatched)
{
'<=' {}
'=>' {}
'==' {$resultsFileContents += $_}
}
$resultsFileCsv = $resultsFileContents | ConvertTo-Csv
$resultsFileCsv | Export-Csv $resultsFile -NoTypeInformation -Force
Output gives a blank file :(
The code below outputs the desired results based on the inputs you provided.
function CombineSkip1($s1, $s2){
$s3 = $s1 -split ','
$s2 -split ',' | select -Skip 1 | % {$s3 += $_}
$s4 = $s3 -join ', '
$s4
}
Write-Output "------Combine files------"
# content
$c1 = Get-Content D:\junk\test1.csv
$c2 = Get-Content D:\junk\test2.csv
# users in both files, could be a better way to do this
$t1 = $c1 | ConvertFrom-Csv
$t2 = $c2 | ConvertFrom-Csv
$users = $t1 | Select SamAccountName
# generate final, combined output
$combined = #()
$combined += CombineSkip1 $c1[0] $c2[0]
$c2PropCount = ($c2[0] -split ',').Count - 1
$filler = (', ""' * $c2PropCount)
for ($i = 1; $i -lt $c1.Count; $i++){
$user = $c1[$i].Split(',')[0]
$u2 = $c2 | where {([string]$_).StartsWith($user)}
if ($u2)
{
$combined += CombineSkip1 $c1[$i] $u2
}
else
{
$combined += ($c1[$i] + $filler)
}
}
# write to output and file
Write-Output $combined
$combined | Set-Content -Path D:\junk\test3.csv -Force
You can use compare-object for that purpose. Use -property samaccountname with it. For example:
$a = 1,2,3,4,5
$b = 4,5,6,7
$side = compare-object $a $b
switch ($side){
'<=' {is not in $a}
'=>' {is not in $b}
'==' { is on both sides}
}
When you have all the data in your output-variable, trow it at convertto-csv and write it in a file
After an entire day, I finally came up with something that works...
...
Edit
Reason: breaking the inner loop and removing the found element from the array will be much faster when merging files with thousands of records...
Function GetTitlesFromFileToBeMerged
{
Param ($csvFile)
[String]$fileToBeMergedTitles = Get-Content $fileToBeMerged -TotalCount 1
[String[]]$fileToBeMergedTitles = ($fileToBeMergedTitles -replace "`",`"", "|").Trim()
[String[]]$fileToBeMergedTitles = ($fileToBeMergedTitles -replace "`"", "").Trim()
[String[]]$fileToBeMergedTitles = ($fileToBeMergedTitles -replace "SamAccountName", "").Trim()
[String[]]$listOfColumnTitles = $fileToBeMergedTitles.Split('|',[System.StringSplitOptions]::RemoveEmptyEntries)
Write-Output $listOfColumnTitles
}
$baseFile = 'D:\Scripts\Powershell\Tests\base.csv'
$fileToBeMerged = 'D:\Scripts\Powershell\Tests\lookup.csv'
$baseFileCsvContents = Import-Csv $baseFile
$baseFileContents = Get-Content $baseFile
$fileToBeMergedCsvContents = Import-Csv $fileToBeMerged
[System.Collections.Generic.List[System.Object]]$fileToBeMergedContents = Get-Content $fileToBeMerged
$resultsFile = 'D:\Scripts\Powershell\Tests\MergedResults.csv'
$resultsFileContents = #()
[String]$baseFileTitles = $baseFileContents[0]
[String]$fileToBeMergedTitles = (Get-Content $fileToBeMerged -TotalCount 1) -replace "`"SamAccountName`",", ""
$resultsFileContents += $baseFileTitles + "," + $fileToBeMergedTitles
[String]$lineMatchNotFound = ""
$arrayFileToBeMergedTitles = GetTitlesFromFileToBeMerged $fileToBeMerged
For ($valueNum = 0; $valueNum -lt $arrayFileToBeMergedTitles.Length; $valueNum++)
{
$lineMatchNotFound += ",`"`""
}
$baseLineCounter = 1
$baseFileCsvContents | ForEach-Object {
$baseSameAccountName = $_.SamAccountName
[String]$baseLineInFile = $baseFileContents[$baseLineCounter]
$lineMatchCounter = 1
$lineMatchFound = ""
:inner
ForEach ($line in $fileToBeMergedContents) {
If ($line -like "*$baseSameAccountName*") {
[String]$lineMatchFound = "," + ($line -replace '^"[^"]*",', "")
$fileToBeMergedContents.RemoveAt($lineMatchCounter)
break inner
}; $lineMatchCounter++
}
If (!($lineMatchFound))
{
[String]$lineMatchFound = $lineMatchNotFound
}
$mergedLine = $baseLineInFile + $lineMatchFound
$resultsFileContents += $mergedLine
$baseLineCounter++
}
ForEach ($line in $resultsFileContents)
{
Write-Host $line
}
$resultsFileContents | Set-Content $resultsFile -Force
I'm very sure this is not the best approach and there is something better that would handle this much faster. If anyone has any ideas, I'm open to them. Thanks.