How to convert text files in a folder into a CSV file - powershell

Question

I am trying to write a script to take a bunch of text files in a folder (which are all in the same format) and output them to a csv file. Each file has the same "header" information. I have been able to get information in a more easily usable format (removing the first and last lines, which aren't needed), but am having some trouble after that.
Here is the beginning of the text file, though there will be more than just these 7 lines, there will be a total of 36 lines per file:
TYPE VOID
DOB 20200131
DATE 20200131
TIME 21:19:42
TERMINAL 3
ORGTERM 2
EMPLOYEE 1234 John Doe
And here is what I have so far, though I know that it doesn't work:
$currentdir = '.\'
$results = #()
$outputfilename = 'data.csv'
foreach ($req in Get-ChildItem($currentdir)) {
(Get-Content $req)[1..((Get-Content $req).count - 2)] |
ForEach-Object {
$header = $_[0] -split '`t'
$data = $_[1] -split '`t'
$results = $header, $data
}
}
The final product would look something like this:
A B C D E F G
1 TYPE DOB DATE TIME TERMINAL ORGTERM EMPLOYEE
2 VOID 20200131 20200131 21:19:42 3 2 1234 John Doe
3 AUTHORIZE 20200131 20200131 23:29:22 2 4678 Jane Doe
Full sample of VOID file:
BEGIN
TYPE VOID
DOB 20200131
DATE 20200131
TIME 21:19:42
TERMINAL 3
ORGTERM 2
EMPLOYEE 1234 Jane Doe
TABLE TBL 101
CHECK 20030
PAYMENT 20029
AUTHAMT 20.68
BATCHAMT 20.68
CARDTYPE MASTERCARD
CARDMASK XXXXXXXXXXXXXXXXX
{XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX}
EXP 0423
REF 482
STANDALONE YES
PINDEX 1
APPROVEAMT 20.68
LOGTIME 21:07:01
FOHFEATS 10000000000000000000000000000000
TERMCAPS 00000000000000000000000000000000
FOHVERSION 15.1.34.2.97
ACTIONCODE 000
LASTSEND 1580585993
ORIGDATE 20200131
ORIGTIME 21:02:11
ORIGTYPE AUTHORIZE
ORIGREF 482
ORGREFTIME 21:02:11
TENDER_NUM 12
CRCY 840
VPD Sequence #: 107
REVID 2
REVNAME 712 Bar
END
Sample AUTHORIZE file:
BEGIN
TYPE AUTHORIZE
DOB 20200131
DATE 20200131
TIME 23:29:22
TERMINAL 2
EMPLOYEE 1234 Jane Doe
TABLE Table 121
CHECK 20045
PAYMENT 20038
AUTHAMT 72.42
BATCHAMT 72.42
CARDTYPE VISA
CARDMASK XXXXXXXXXXXXXXXX
{XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX}
EXP 0124
REF 485900
STANDALONE YES
PINDEX 1
LOGTIME 23:29:22
FOHFEATS 10000000000000000000000000000000
TERMCAPS 00000000000000000000000000000000
FOHVERSION 15.1.34.2.97
LASTSEND 1580586235
TENDER_NUM 13
CRCY 840
REVID 1
REVNAME 712 Restaurant
COMMERROR TRUE
END
Sample adjust file:
BEGIN
TYPE ADJUST
DOB 20200131
DATE 20200131
TIME 22:18:27
TERMINAL 8
ORGTERM 8
EMPLOYEE 789 Judy Garland
TABLE BAR GUEST
CHECK 80161
PAYMENT 80036
BATCHAMT 30.43
BATCHTIP 6
CARDTYPE MASTERCARD
CARDMASK XXXXXXXXXXXX8699
{XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX}
EXP 0323
REF 1504602
STANDALONE YES
PINDEX 1
LOGTIME 22:18:27
FOHFEATS 10000000000000000000000000000000
TERMCAPS 00000000000000000000000000000000
FOHVERSION 15.1.34.2.97
LASTSEND 1580638928
TENDER_NUM 12
CRCY 840
REVID 4
REVNAME 712 Second Bar
END

Answer 1

here's one way to merge those text files into a CSV. it presumes the files are in a specific dir and can be loaded by matching the names OR by simply grabbing all the files.
what it does ...
sets the source dir
sets the file filter
grabs all the matching files
iterates thru the file list
loads each file into a $Var
uses the way that PoSh handles a collection on the LEFT side of a match
that gives you the matching item, not the usual [bool].
builds a PSCustomObject
it does that by matching the line with the target word, getting the 1st item in the returned array, replaces the unwanted part of the line with nothing, and finally assigns that value to the desired property.
this is rather inefficient, but i can't think of a better way. [blush]
sends the PSCO out to the $Results collection
shows what is in $Results on the screen
exports $Results to a CSV file
here's the code ...
$SourceDir = $env:TEMP
$Filter = 'harlan_*.txt'
$FileList = Get-ChildItem -LiteralPath $SourceDir -Filter $Filter -File
$Results = foreach ($FL_Item in $FileList)
{
$Lines = Get-Content -LiteralPath $FL_Item.FullName
[PSCustomObject]#{
Type = ($Lines -match '^type')[0] -replace '^type\s{1,}'
DOB = ($Lines -match '^dob')[0] -replace '^dob\s{1,}'
Date = ($Lines -match '^date')[0] -replace '^date\s{1,}'
Time = ($Lines -match '^time')[0] -replace '^time\s{1,}'
Terminal = ($Lines -match '^terminal')[0] -replace '^terminal\s{1,}'
OrgTerm = ($Lines -match '^orgterm')[0] -replace '^orgterm\s{1,}'
Employee = ($Lines -match '^employee')[0] -replace '^employee\s{1,}'
}
}
# show on screen
$Results
# save to CSV
$Results |
Export-Csv -LiteralPath "$SourceDir\Harlan_-_MergedFiles.csv" -NoTypeInformation
display on screen ...
Type : ADJUST
DOB : 20200131
Date : 20200131
Time : 22:18:27
Terminal : 8
OrgTerm : 8
Employee : 789 Judy Garland
Type : AUTHORIZE
DOB : 20200131
Date : 20200131
Time : 23:29:22
Terminal : 2
OrgTerm :
Employee : 1234 Jane Doe
Type : VOID
DOB : 20200131
Date : 20200131
Time : 21:19:42
Terminal : 3
OrgTerm : 2
Employee : 1234 Jane Doe
content of the csv file ...
"Type","DOB","Date","Time","Terminal","OrgTerm","Employee"
"ADJUST","20200131","20200131","22:18:27","8","8","789 Judy Garland"
"AUTHORIZE","20200131","20200131","23:29:22","2","","1234 Jane Doe"
"VOID","20200131","20200131","21:19:42","3","2","1234 Jane Doe"

Answer 2

To capture all fields in the files without hardcoding the headers and combine them into a CSV file, the below code should do it.
Snag is that there is one line in each file that does not have a 'Header', it is just a string {XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX}.
I'm guessing that should be the Card Number, so I'm manually inserting the header CARDNUMBER there. If this is something else, please change that in the code.
$files = Get-ChildItem -Path 'D:\Test' -File
$result = foreach($file in $files) {
$obj = [PsCustomObject]#{}
Get-Content -Path $file.FullName | Where-Object { $_ -notmatch '^(BEGIN|END)$' } | ForEach-Object {
# There is a line without 'header' name. Is this the card number?
if ($_ -like '{*}') {
$name = 'CARDNUMBER' # <-- add your own preferred header name here
$value = $_
}
else {
$name,$value = $_ -split '\s+', 2
}
$obj | Add-Member -MemberType NoteProperty -Name $name -Value $value
}
# output the object for this file to be colected in the $result variable
$obj
}
# output on screen
$result
#output to CSV file
$result | Export-Csv -Path 'D:\output.csv' -NoTypeInformation
You need to set the paths for Get-ChildItem and Export-CSV to match your own situation of course

Answer 3

If I'm reading this correctly you have some files each has a single record of data delimited between the aptly positions words "BEGIN" & "END" You want each file to be translated into a single CSV file?
I think I've cooked up something worth while. Though I'm sure it's not perfect.
$Select = 'TYPE','DOB','DATE','TIME','TERMINAL','ORGTERM','EMPLOYEE'
ForEach( $InputFile in (Get-ChildItem $CurrentDirectory) )
{
$OutputFile = $InputFile.BaseName + '.csv'
$Table = Get-Content $InputFile
$TempHash = [Ordered]#{}
ForEach( $Column in $Table )
{
If( $Column -notmatch '(^BEGIN$|^END$)' )
{
$TempArr = $Column.Split( ' ', 2, [System.StringSplitOptions]::RemoveEmptyEntries ) | ForEach{$_.Trim()}
If( $Select -contains $TempArr[0] )
{
$TempHash.Add($TempArr[0], $TempArr[1] )
}
}
}
#Now $TempHash should have enough to create the object and export to CSV
[PSCustomObject]$TempHash | Export-Csv -Path $OutputFile -NoTypeInformation
}
A few points:
I'm ignoring the lines BEGIN & END
I'm manipulating each line thereafter into an array, which for the
most part should be 2 elements.
If the first element [0] is in the collection of fileds your looking
for I'll add as a key/value pair to the hash. Otherwise do nothing.
After processing the lines Convert the object to a PSCustomObject and
export to a CSV file.
I only tested it on a single file I created from your question. I wrapped up the outer loop just as pseudo code.
This works, but the output looks a little choppy, like numbers being strings and such. That said, as a rev one I think we've got something to work with.
If misread your comment, and you want a single output CSV file the adjustment is just to declare the filename before the loop and use the append param on the Export-CSV cmdlet. See below, though I didn't test it any further:
$OutputFile = 'YourOutput.csv'
$Select = 'TYPE','DOB','DATE','TIME','TERMINAL','ORGTERM','EMPLOYEE'
ForEach( $InputFile in (Get-ChildItem $CurrentDirectory) )
{
$Table = Get-Content $InputFile
$TempHash = [Ordered]#{}
ForEach( $Column in $Table )
{
If( $Column -notmatch '(^BEGIN$|^END$)' )
{
$TempArr = $Column.Split( ' ', 2, [System.StringSplitOptions]::RemoveEmptyEntries ) | ForEach{$_.Trim()}
If( $Select -contains $TempArr[0] )
{
$TempHash.Add($TempArr[0], $TempArr[1] )
}
}
}
#Now $TempHash should have enough to create the object and export to CSV
[PSCustomObject]$TempHash | Export-Csv -Path $OutputFile -NoTypeInformation -Append
}
Sorry about the variable names, that could obviously use a refactor...
Let me know what you think.