I am trying to compare the string of two CSV files. If the string from the 2nd CSV file occurs in the 1st CSV file, the corresponding line in the 1st CSV file should be marked with a label (e.g.: "TestLabel") after the semicolon. The strings contain a lot of special characters. By and large, the comparison already works, I can also already add the label.
Since Powershell is still new to me and this is my first script, the following question still arises. How can I set my text "TestLabel" to a certain place in an uncomplicated way? Here, for example, in the next empty field between the semicolons?
CSV1 contains:
Testdefinition;Stichwörter;Stichwörter;Stichwörter;Stichwörter;Stichwörter
It is just a normal text (with round brackets).Test: success;ExistingLabel;;;;
This is a second text;;;
Another text;ExistingLabel;;;;
One more text for the testing - success;ExistingLabel;;;;
CSV2 contains:
Testdefinition;Stichwörter;Stichwörter;Stichwörter;Stichwörter;Stichwörter
It is just a normal text (with round brackets).Test: success
One more text for the testing - success
My script so far:
$header='Testdefinition', 'Stichwörter1', 'Stichwörter2', 'Stichwörter3', 'Stichwörter4', 'Stichwörter5'
$exportheader="Testdefinition;Stichwörter;Stichwörter;Stichwörter;Stichwörter;Stichwörter"
$path1='D:\data\.....test.csv'
$path2='D:\data\.....test_failed.csv'
$wfile='temp1.csv'
$wfile2='temp2.csv'
Get-Content $path1 | Select-Object -Skip 1 | Set-Content $wfile -Encoding UTF8
Get-Content $path2 | Select-Object -Skip 1 | Set-Content $wfile2 -Encoding UTF8
$file1=Import-CSV -Path $wfile -Delimiter ";" -Header $header
$file2=Import-CSV -Path $wfile2 -Delimiter ";" -Header $header
$exportfile='test.csv'
#$exportfile=$file1
$file1 | Get-Member
$file2 | Get-Member
$file1 | Format-Table
$file2 | Format-Table
Write-Output ""
Write-Output "Searching for failed results"
Set-Content $exportfile -Value $exportheader
$file1.Testdefinition | ForEach-Object {
Write-Output "The Testdefinition is: $_ "
$testSearch = $_
$testlinecontent = $file2.Testdefinition | Select-String $testSearch
$testlinenumber = $testlinecontent.LineNumber
if("$_" -eq "$testlinecontent")
{
Write-Output "Testline found: $testlinecontent in Line $testlinenumber"
Write-Output "$_ = $testlinecontent"
$testlineexport = "$_;$testlinenumber;TestLabel"
Write-Output $testlineexport
$testlineexport | Add-Content -Path $exportfile
}
else
{
Write-Output "Testline not found"
$testlineexport = "$_;$testlinenumber;NULL"
Write-Output $testlineexport
$testlineexport | Add-Content -Path $exportfile
}
Write-Output ""
}
$exportCsv = Import-Csv $exportfile -Delimiter ";" -Header $header
$exportCsv | Format-Table
Remove-Item -Path $wfile
Remove-Item -Path $wfile2
I hope you can give me a hint. Thanks in advance!
Assuming the files aren't too big, you can use the following approach based on Compare-Object, which is conceptually clear and relatively simple:
# Read the CSV files into their header row and the array of data rows, as strings.
$header, $rows1 = Get-Content $path1
$null, $rows2 = Get-Content $path2
# Initialize the export file by writing its header
Set-Content -Encoding utf8 $exportfile -Value $exportheader
# Compare the data rows by their first ";"-separated field.
# If the fields match, append ";TestLabel" to the LHS data row before
# passing it through, otherwise pass it as-is, and append to the
# export file.
Compare-Object -PassThru $rows1 $rows2 -IncludeEqual -Property { $_.Split(';')[0] } |
ForEach-Object { if ($_.SideIndicator -eq '==') { $_ + ';TestLabel' } else { $_ } } |
Add-Content $exportfile
Note:
For brevity I've omitted the code to also add a line number.
As you are already aware, PowerShell doesn't support CSV files whose headers contain duplicate column names, given that the column names become property names on import, and must therefore be unique.
Related
I am trying to compare the string of two CSV files. If the string from the 2nd CSV file occurs in the 1st CSV file, the corresponding line in the 1st CSV file should be marked with a label (e.g.: "TestLabel") after the semicolon. The strings contain a lot of special characters. By and large, the comparison already works, I can also already add the label.
Since Powershell is still new to me and this is my first script, the following question still arises. How can I set my text "TestLabel" to a certain place in an uncomplicated way? Here, for example, in the next empty field between the semicolons?
CSV1 contains:
Testdefinition;Stichwörter;Stichwörter;Stichwörter;Stichwörter;Stichwörter
It is just a normal text (with round brackets).Test: success;ExistingLabel;;;;
This is a second text;;;
Another text;ExistingLabel;;;;
One more text for the testing - success;ExistingLabel;;;;
CSV2 contains:
Testdefinition;Stichwörter;Stichwörter;Stichwörter;Stichwörter;Stichwörter
It is just a normal text (with round brackets).Test: success
One more text for the testing - success
My script so far:
$header='Testdefinition', 'Stichwörter1', 'Stichwörter2', 'Stichwörter3', 'Stichwörter4', 'Stichwörter5'
$exportheader="Testdefinition;Stichwörter;Stichwörter;Stichwörter;Stichwörter;Stichwörter"
$path1='D:\data\.....test.csv'
$path2='D:\data\.....test_failed.csv'
$wfile='temp1.csv'
$wfile2='temp2.csv'
Get-Content $path1 | Select-Object -Skip 1 | Set-Content $wfile -Encoding UTF8
Get-Content $path2 | Select-Object -Skip 1 | Set-Content $wfile2 -Encoding UTF8
$file1=Import-CSV -Path $wfile -Delimiter ";" -Header $header
$file2=Import-CSV -Path $wfile2 -Delimiter ";" -Header $header
$exportfile='test.csv'
#$exportfile=$file1
$file1 | Get-Member
$file2 | Get-Member
$file1 | Format-Table
$file2 | Format-Table
Write-Output ""
Write-Output "Searching for failed results"
Set-Content $exportfile -Value $exportheader
$file1.Testdefinition | ForEach-Object {
Write-Output "The Testdefinition is: $_ "
$testSearch = $_
$testlinecontent = $file2.Testdefinition | Select-String $testSearch
$testlinenumber = $testlinecontent.LineNumber
if("$_" -eq "$testlinecontent")
{
Write-Output "Testline found: $testlinecontent in Line $testlinenumber"
Write-Output "$_ = $testlinecontent"
$testlineexport = "$_;$testlinenumber;TestLabel"
Write-Output $testlineexport
$testlineexport | Add-Content -Path $exportfile
}
else
{
Write-Output "Testline not found"
$testlineexport = "$_;$testlinenumber;NULL"
Write-Output $testlineexport
$testlineexport | Add-Content -Path $exportfile
}
Write-Output ""
}
$exportCsv = Import-Csv $exportfile -Delimiter ";" -Header $header
$exportCsv | Format-Table
Remove-Item -Path $wfile
Remove-Item -Path $wfile2
I hope you can give me a hint. Thanks in advance!
Assuming the files aren't too big, you can use the following approach based on Compare-Object, which is conceptually clear and relatively simple:
# Read the CSV files into their header row and the array of data rows, as strings.
$header, $rows1 = Get-Content $path1
$null, $rows2 = Get-Content $path2
# Initialize the export file by writing its header
Set-Content -Encoding utf8 $exportfile -Value $exportheader
# Compare the data rows by their first ";"-separated field.
# If the fields match, append ";TestLabel" to the LHS data row before
# passing it through, otherwise pass it as-is, and append to the
# export file.
Compare-Object -PassThru $rows1 $rows2 -IncludeEqual -Property { $_.Split(';')[0] } |
ForEach-Object { if ($_.SideIndicator -eq '==') { $_ + ';TestLabel' } else { $_ } } |
Add-Content $exportfile
Note:
For brevity I've omitted the code to also add a line number.
As you are already aware, PowerShell doesn't support CSV files whose headers contain duplicate column names, given that the column names become property names on import, and must therefore be unique.
Below is one of the file data I have in text file
B97SW | CHANGED | rc=0 >>
Server Name";"SystemFolderPath";"IdenityReference";"FileSystemRights";"Vulnerable
B97SW;C:\Windows\system32;CREATOR OWNER;268435456;No
B97SW;C:\Windows\system32;NT AUTHORITY\SYSTEM;268435456;No
B97SW;C:\Windows\system32;NT AUTHORITY\SYSTEM;Modify, Synchronize;No
........
I am trying to replace ";" with "," and write to csv.
Below is the code I wrote but it is not writing the data in csv.
$FileList = Get-ChildItem -Path "C:\Files"
$props=[ordered]#{
ServerName=''
SystemFolderPath=''
IdenityReference=''
FileSystemRights=''
Vulnerable=''
}
New-Object PsObject -Property $props |
Export-Csv C:\2021.csv -NoTypeInformation
$FinalData = #()
foreach($n_file in $FileList)
{
$FileName = $n_file.FullName
$FileContent = Get-Content -Path $FileName | Select-Object -Skip 2
foreach($line in $FileContent)
{
$line = $line -replace(";",",")
$line | Export-Csv -Path C:\2021.csv -Append -NoTypeInformation -Force
}
}
output I am getting
"ServerName","SystemFolderPath","IdenityReference","FileSystemRights","Vulnerable"
"","","","",""
,,,,
,,,,
Please let me know what is wrong I am doing here.
$line | Export-Csv -Path C:\2021.csv -Append -NoTypeInformation -Force
This doesn't work because Export-Csv expects object(s) with properties, but $line is just a string. You need to parse it into an object first, using ConvertFrom-Csv.
Try this:
$FileList = Get-ChildItem -Path "C:\Files"
foreach($n_file in $FileList)
{
$FileName = $n_file.FullName
Get-Content -Path $FileName |
Select-Object -Skip 2 |
ConvertFrom-Csv -Delimiter ';' -Header ServerName, SystemFolderPath, IdenityReference, FileSystemRights, Vulnerable |
Export-Csv -Path C:\2021.csv -Append -NoTypeInformation -Force
}
As we have skipped the original headers, we have to supply these through the -Header parameter of ConvertFrom-Csv.
Your CSV file is goofed up in two ways. First, there is a line of garbage before the header line. Second, in the header line the semi-colons are surrounded by double quotes. The correct form would be to surround the header names with quotes instead.
Once these format errors are fixed, you can read the csv file with this:
Import-Csv myfile.csv -delimiter ";"
Or if you want to produce a comma delimited csv file, try this:
Import-Csv myfile.csv -delimiter ";" | Export-Csv newfile.csv
The result will be correct but it will have a lot of unnecessary double quotes.
I have a script that I am working on that reads in some text files and converts them to .csv and changes some values. I have two different file sources. One is a tab delimited .txt file and the other is a comma separated .txt file. Is there a way to determine which type of delimiter is being used to determine which export function is appropriate?
get-childitem $workingDir -filter *.txt -Recurse| ForEach-Object {
$targetfile = $_.Name
$targetFile = $_.FullName.Substring(0,$_.FullName.Length-4)
$targetFile = $targetfile += ".csv"
if( Get-Content -Delimiter = `t ){
Write-Host "The file is tab-delimited"
Get-Content -path $_.FullName
ForEach-Object {$_ -replace “`t”,”,” } |
Out-File -filepath $targetFile -Encoding utf8
}
else {
Write-Host "The file is comma-separated"
Get-Content -path $_.FullName |
Out-File -filepath $targetFile -Encoding utf8
}
}
Another approach would be to use Select-String to check for tab character and set delimiter.
if(Get-Content $csvfile -First 1 | Select-String -Pattern "`t")
{
$delim = "`t"
}
else
{
$delim = ','
}
Import-Csv $csvfile -Delimiter $delim
Assuming that the comma-separated files never contain tabs (which would then be data), the most efficient approach is to inspect only the first line of each file for the presence of tab characters, which is most easily done with (Get-Content -First 1 $_.FullName) -match "`t" - see Get-Content and -match, the regular-expression matching operator.
# Determine the arguments to pass to Set-Content - later, via splatting -
# for writing the output file.
$setContentArgs = #{
LiteralPath = $_.BaseName + '.csv'
Encoding = 'utf8'
}
# Check the 1st line for containing a tab.
# (This assumes that the comma-separated files contain not tabs as data.)
if ((Get-Content -First 1 $_.FullName) -match "`t") {
Write-Host "The file is tab-delimited."
# Read line by line, replace tabs with commas, and write with UTF-8 encoding.
Get-Content $_.FullName | ForEach-Object { $_ -replace "`t", ',' } |
Set-Content #setContentArgs
}
else {
Write-Host "The file is comma-separated."
# Just read lines as-is and write with UTF-8 encoding.
Get-Content $_.FullName |
Set-Content #setContentArgs
}
Note the use of the .BaseName property on the input [System.IO.FileInfo], which conveniently reports the file name without its extension, which allows you to simply append the new extension.
Since you're dealing with text (strings) only, Set-Content, which is slightly more efficient, is preferable to Out-File.
For the technique of passing arguments via a hashtable (#{ ... }), see about_Splatting
If the files are smallish (easily fit into memory as a whole (possibly twice) each), you can significantly speed up processing by reading each file as a whole with -Raw and using
-NoNewLine (PSv5+) to write that (possibly modified) string as-is, without appending a trailing newline, to the output file.
Since you're then reading the entire file anyway, you can get away with a single Get-Content call and apply -replace "`t", ',' blindly, given that for comma-separated files this will simply be a (fast) no op.
(Get-Content -Raw $_.FullName) -replace "`t", ',' |
Set-Content ($_.BaseName + '.csv') -Encoding Utf8 -NoNewLine
I will use Import-Csv for this:
If(Import-Csv "File path to test if Tab-delimited file" -Delimiter "`t" -Ea SilentlyContinue){
"File is tab-delimited"
}
If(Import-Csv "File path to test if Comma-CSV file" -Ea SilentlyContinue){
"File is a comma-separated CSV"
}
I would like remove all quotations character in my exported csv file, it's very annoying when i generated a new csv file and i need to manually to remove all the quotations that include in the string. Could anyone provide me a Powershell script to overcome this problem? Thanks.
$File = "c:\programfiles\programx\file.csv"
(Get-Content $File) | Foreach-Object {
$_ -replace """, ""
} | Set-Content $File
Next time you make one, export-csv in powershell 7 has a new option you may like:
export-csv -UseQuotes AsNeeded
It seems many of us have already explained that quotes are sometimes needed in CSV files. This is the case when:
the value contains a double quote
the value contains the delimiter character
the value contains newlines or has whitespace at the beginning or the end of the string
With PS version 7 you have the option to use parameter -UseQuotes AsNeeded.
For older versions I made this helper function to convert to CSV using only quotes when needed:
function ConvertTo-CsvNoQuotes {
# returns a csv delimited string array with values unquoted unless needed
[OutputType('System.Object[]')]
[CmdletBinding(DefaultParameterSetName = 'ByDelimiter')]
param (
[Parameter(Mandatory = $true, ValueFromPipeline = $true, ValueFromPipelineByPropertyName = $true, Position = 0)]
[PSObject]$InputObject,
[Parameter(Position = 1, ParameterSetName = 'ByDelimiter')]
[char]$Delimiter = ',',
[Parameter(ParameterSetName = 'ByCulture')]
[switch]$UseCulture,
[switch]$NoHeaders,
[switch]$IncludeTypeInformation # by default, this function does NOT include type information
)
begin {
if ($UseCulture) { $Delimiter = (Get-Culture).TextInfo.ListSeparator }
# regex to test if a string contains a double quote, the delimiter character,
# newlines or has whitespace at the beginning or the end of the string.
# if that is the case, the value needs to be quoted.
$needQuotes = '^\s|["{0}\r\n]|\s$' -f [regex]::Escape($Delimiter)
# a boolean to check if we have output the headers or not from the object(s)
# and another to check if we have output type information or not
$doneHeaders = $doneTypeInfo = $false
}
process {
foreach($item in $InputObject) {
if (!$doneTypeInfo -and $IncludeTypeInformation) {
'#TYPE {0}' -f $item.GetType().FullName
$doneTypeInfo = $true
}
if (!$doneHeaders -and !$NoHeaders) {
$row = $item.PsObject.Properties | ForEach-Object {
# if needed, wrap the value in quotes and double any quotes inside
if ($_.Name -match $needQuotes) { '"{0}"' -f ($_.Name -replace '"', '""') } else { $_.Name }
}
$row -join $Delimiter
$doneHeaders = $true
}
$item | ForEach-Object {
$row = $_.PsObject.Properties | ForEach-Object {
# if needed, wrap the value in quotes and double any quotes inside
if ($_.Value -match $needQuotes) { '"{0}"' -f ($_.Value -replace '"', '""') } else { $_.Value }
}
$row -join $Delimiter
}
}
}
}
Using your example to remove the unnecessary quotes in an existing CSV file:
$File = "c:\programfiles\programx\file.csv"
(Import-Csv $File) | ConvertTo-CsvNoQuotes | Set-Content $File
keeping in mind that this may trash your data if you have embedded double quotes in your data, here is yet another variation on the idea ... [grin]
what it does ...
defines the input & output full file names
grabs the *.tmp files from the temp dir
filters for the 1st three files & only three basic properties
creates the file to work with
loads the file content
replaces the double quotes with nothing
saves the cleaned file to the 2nd file name
displays the original & cleaned versions of the file
the code ...
$TestCSV = "$env:TEMP\Ted.Xiong_-_Test.csv"
$CleanedTestCSV = $TestCSV -replace 'Test', 'CleanedTest'
Get-ChildItem -LiteralPath $env:TEMP -Filter '*.tmp' -File |
Select-Object -Property Name, LastWriteTime, Length -First 3 |
Export-Csv -LiteralPath $TestCSV -NoTypeInformation
(Get-Content -LiteralPath $TestCSV) -replace '"', '' |
Set-Content -LiteralPath $CleanedTestCSV
Get-Content -LiteralPath $TestCSV
'=' * 30
Get-Content -LiteralPath $CleanedTestCSV
output ...
"Name","LastWriteTime","Length"
"hd4130E.tmp","2020-03-13 5:23:06 PM","0"
"hd418D4.tmp","2020-03-12 11:47:59 PM","0"
"hd41F7D.tmp","2020-03-13 5:23:09 PM","0"
==============================
Name,LastWriteTime,Length
hd4130E.tmp,2020-03-13 5:23:06 PM,0
hd418D4.tmp,2020-03-12 11:47:59 PM,0
hd41F7D.tmp,2020-03-13 5:23:09 PM,0
As above, the quotations are valid for csv, but to remove them you need to escape the quote mark in the replace operation as is a special character:
$File = "c:\programfiles\programx\file.csv"
(Get-Content $File) | Foreach-Object {
$_ -replace "`"", ""
} | Set-Content $File
Why are you manually in a text editor read Csv files?
You exported them to that format for a reason. To read them, just import them back in and view them on screen and or Read them back in and send the readout to notepad for reading.
Export-Csv -Path D:\temp\book1.csv
Import-Csv -Path D:\temp\book1.csv |
Clip |
Notepad # then press crtl+v, then save the notepad file with a new name.
If you don't want Csv, then don't export as Csv, just output as a flat-file, using Out-File instead.
Update
Since your last comment to me indicated your final use case. CSV into SQL is a very common thing. A quick web search will show you how even provide you with a script. You should also be looking at the PowerShell DBATools module.
How to import data from .csv in SQL Server using PowerShell?
Importing CSV files into a Microsoft SQL DB using PowerShell
ImportingCSVsIntoSQLv1.zip
Four Easy Ways to Import CSV Files to SQL Server with PowerShell
Find-Module -Name '*dba*'
<#
Version Name Repository Description
------- ---- ---------- -----------
1.0.101 dbatools PSGallery The community module that enables SQL Server Pros to automate database development and server administration
...
#>
Update
You mean this...
Get-Content 'D:\temp\book1.csv'
<#
# Results
"Site","Dept"
"Main","aaa,bbb,ccc"
"Branch1","ddd,eee,fff"
"Branch2","ggg,hhh,iii"
#>
Get-ChildItem -Path 'D:\temp' -Filter 'book1.csv' |
ForEach {
$NewFile = New-Item -Path 'D:\Temp' -Name "$($PSItem.BaseName).txt"
Get-Content -Path $PSItem.FullName |
ForEach-Object {
Add-Content -Path $NewFile -Value ($PSItem -replace '"') -WhatIf
}
}
<#
What if: Performing the operation "Add Content" on target "Path: D:\Temp\book1.txt".
What if: Performing the operation "Add Content" on target "Path: D:\Temp\book1.txt".
What if: Performing the operation "Add Content" on target "Path: D:\Temp\book1.txt".
What if: Performing the operation "Add Content" on target "Path: D:\Temp\book1.txt"
#>
Get-ChildItem -Path 'D:\temp' -Filter 'book1.csv' |
ForEach {
$NewFile = New-Item -Path 'D:\Temp' -Name "$($PSItem.BaseName).txt"
Get-Content -Path $PSItem.FullName |
ForEach-Object {
Add-Content -Path $NewFile -Value ($PSItem -replace '"')
}
}
Get-Content 'D:\temp\book1.txt'
<#
# Results
Site,Dept
Main,aaa,bbb,ccc
Branch1,ddd,eee,fff
Branch2,ggg,hhh,iii
#>
Of course, you need to use a wildcard for the csv files and use the -Resurse to get all directories and an error handler to make sure you don't have file name collisions.
One solution for dont remove the double quote into the string quoted :
$delimiter=","
$InputFile="c:\programfiles\programx\file.csv"
$OutputFile="c:\programfiles\programx\resultfile.csv"
#import file in variable (not necessary if your faile is big repeat this import where i use $ContentFile)
$ContentFile=import-csv $InputFile -Delimiter $delimiter -Encoding utf8
#list of property of csv file
$properties=($ContentFile | select -First 1 | Get-Member -MemberType NoteProperty).Name
#write header into new file
$properties -join $delimiter | Out-File $OutputFile -Encoding utf8
#write data into new file
$ContentFile | %{
$RowObject=$_ #==> get row object
$Line=#() #==> create array
$properties | %{$Line+=$RowObject."$_"} #==> Loop on every property, take value (without quote) inot row object
$Line -join $delimiter #==> join array for get line with delimer and send to standard outut
} | Out-File $OutputFile -Encoding utf8 -Append #==> export result to output file
An extra double quote can be used to escape a double quote in a string:
$File = "c:\programfiles\programx\file.csv"
(Get-Content $File) | Foreach-Object { $_ -replace """", "" } | Set-Content $File
After you have exported the CSV file with Export-CSV, you can use Get-Content to load the CSV file into an array of strings, then use Set-Content and replace to remove the quotation marks:
Set-Content -Path sample.csv -Value ((Get-Content -Path sample.csv) -replace '"')
As mklement0 helpfully pointed out, this could potentially corrupt the CSV if some lines need quoting. This solution simply goes through the whole file and replaces every quote with ''.
You could also speed this up with using the -Raw switch with Get-Content, which returns a whole string with the newlines preserved, instead of an array of newline delimited strings:
Set-Content -NoNewline -Path sample.csv -Value ((Get-Content -Raw -Path sample.csv) -replace '"')
I have a CSV file which is structured like this:
"SA1";"21020180123155514000000000000000002"
"SA2";"21020180123155514000000000000000002";"210"
"SA4";"21020180123155514000000000000000002";"210";"200000001"
"SA5";"21020180123155514000000000000000002";"210";"200000001";"140000001";"ZZ"
"SA1";"21020180123155522000000000000000002"
"SA2";"21020180123155522000000000000000002";"210"
"SA4";"21020180123155522000000000000000002";"210";"200000001"
"SA5";"21020180123155522000000000000000002";"210";"200000001";"140000671";"ZZ"
"SA1";"21020180123155567000000000000000002"
"SA2";"21020180123155567000000000000000002";"210"
"SA4";"21020180123155567000000000000000002";"210";"200000001"
"SA5";"21020180123155567000000000000000002";"210";"200000001";"140000001";"ZZ"
So the Value in the second field (separator ';') marks the data which belongs together and value 140000001 or 140000671 is the trigger.
So the result should be:
1st file: 140000001.txt
"SA1";"21020180123155514000000000000000002"
"SA2";"21020180123155514000000000000000002";"210"
"SA4";"21020180123155514000000000000000002";"210";"200000001"
"SA5";"21020180123155514000000000000000002";"210";"200000001";"140000001";"ZZ"
"SA1";"21020180123155567000000000000000002"
"SA2";"21020180123155567000000000000000002";"210"
"SA4";"21020180123155567000000000000000002";"210";"200000001"
"SA5";"21020180123155567000000000000000002";"210";"200000001";"140000001";"ZZ"
2nd file: 140000671.txt
"SA1";"21020180123155522000000000000000002"
"SA2";"21020180123155522000000000000000002";"210"
"SA4";"21020180123155522000000000000000002";"210";"200000001"
"SA5";"21020180123155522000000000000000002";"210";"200000001";"140000671";"ZZ"
For now I found a snippet which splits the big file by the second field:
$src = "C:\temp\ORD001.txt"
$dstDir = "C:\temp\files\"
Remove-Item -Path "$dstDir\\*"
$header = Get-Content -Path $src | select -First 1
Get-Content -Path $src | select -Skip 1 | foreach {
$file = "$(($_ -split ";")[1]).txt"
Write-Verbose "Wrting to $file"
$file = $file.Replace('"',"")
if (-not (Test-Path -Path $dstDir\$file))
{
Out-File -FilePath $dstDir\$file -InputObject $header -Encoding ascii
}
$file -replace '"', ""
Out-File -FilePath $dstDir\$file -InputObject $_ -Encoding ascii -Append
}
For the rest I'm standing in the dark.
Please help.
The Import-CSV cmdlet will work here, if you don't already know about it. I would use that, as it returns all the rows as different objects in an array, with the properties being the column values. And you don't have to manually remove the quotes and such. Assuming the second column is a date time value, and should be unique for each group of 4 consecutive rows, then this will work:
$src = "C:\temp\ORD001.txt"
$dstDir = "C:\temp\files\"
Remove-Item -Path "$dstDir\*"
$csv = Import-CSV $src -Delimiter ';'
$DateTimeGroups = $csv | Group-Object -Property 'ColumnTwoHeader'
foreach ($group in $DateTimeGroups) {
$filename = $group.Group.'ColumnFiveHeader' | select -Unique
$group.Group | Export-CSV "$dstDir\$filename.txt" -Append -NoTypeInformation
}
However, this will break if two of those "groups of 4 consecutive rows" have the same value for the second column and the fifth column. There isn't a way to fix this unless you are certain that there will always be 4 consecutive rows in each time group. In which case:
$src = "C:\temp\ORD001.txt"
$dstDir = "C:\temp\files\"
Remove-Item -Path "$dstDir\*"
$csv = Import-CSV $src -Delimiter ';'
if ($csv.count % 4 -ne 0) {
Write-Error "CSV does not have a proper number of rows. Attempting to continue will be bad :)"
return
}
for ($i = 0 ; $i -lt $csv.Count ; $i=$i+4) {
$group = $csv[$i..($i+4)]
$group | Export-Csv "$dstDir\$($group[3].'ColumnFiveHeader').txt" -Append -NoTypeInformation
}
Just be sure to replace Column2Header and Column5Header with the appropriate values.
If performance is not a concern, combining Import-Csv / Export-Csv with Group-Object allows the most concise, direct expression of your intent, using PowerShell's ability to convert CSV to objects and back:
$src = "C:\temp\ORD001.txt" # Input CSV file
$dstDir = "C:\temp\files" # Output directory
# Delete previous output files, if necessary.
Remove-Item -Path "$dstDir\*" -WhatIf
# Import the source CSV into custom objects with properties named for the columns.
# Note: The assumption is that your CSV header line defines columns "Col1", "Col2", ...
Import-Csv $src -Delimiter ';' |
# Group the resulting objects by column 2
Group-Object -Property Col2 |
ForEach-Object { # Process each resulting group.
# Determine the output filename via the group's last row's column 5 value.
$outFile = '{0}\{1}.txt' -f $dstDir, $_.Group[-1].Col5
# Append the group at hand to the target file.
$_.Group | Export-Csv -Append -Encoding Ascii $outFile -Delimiter ';' -NoTypeInformation
}
Note:
The assumption - in line with your sample data - is that it is always the last row in a group of lines sharing the same column-2 value whose column 5 contains the root of the output filename (e.g., 140000001)
Sorry but I don't have a Header Column. It's a semikolon seperated txt file for an interface
You can simply read the file with Get-Content, and then search for the trigger in the line.
I hope this small example can help:
$file = Get-Content CSV_File.txt
$140000001 = #()
$140000671 = #()
$bTrig = #()
foreach($line in $file){
$bTrig += $line
if($line -match ';"140000001";'){
$140000001 += $bTrig
$bTrig = #()
}
elseif($line -match ';"140000671";'){
$140000671 += $bTrig
$bTrig = #()
}
}
if($bTrig.Count -ne 0){Write-Warning "No trigger for $bTrig"}
$140000001 | Out-File 140000001.txt -Encoding ascii
$140000671 | Out-File 140000671.txt -Encoding ascii