spilt text files to tsv UTF-8 with headers in PowerShell - powershell

Question

Im using the below code for splitting the huge file into 20K TSV UTF-8 files..
However, i need every split files should have the header in the 20k count, how we can do it?
$sourceFile = "C:\Users\lingaguru.c3\Desktop\Test\DE.txt"
$partNumber = 1
$batchSize = 20000
$pathAndFilename = "C:\Users\lingaguru.c3\Desktop\Test\Temp part $partNumber file.tsv"
[System.Text.Encoding]$enc = [System.Text.Encoding]::GetEncoding(65001) # utf8 this one
$fs=New-Object System.IO.FileStream ($sourceFile,"OpenOrCreate", "Read", "ReadWrite",8,"None")
$streamIn=New-Object System.IO.StreamReader($fs, $enc)
$streamout = new-object System.IO.StreamWriter $pathAndFilename
$line = $streamIn.readline()
$counter = 0
while ($line -ne $null)
{
$streamout.writeline($line)
$counter +=1
if ($counter -eq $batchsize)
{
$partNumber+=1
$counter =0
$streamOut.close()
$pathAndFilename = "C:\Users\lingaguru.c3\Desktop\Test\Temp part $partNumber file.tsv"
$streamout = new-object System.IO.StreamWriter $pathAndFilename
}
$line = $streamIn.readline()
}
$streamin.close()
$streamout.close()

Answer 1

Without altering your code too much, you need to capture the header line in a variable and write that out first thing on every new file:
$sourceFile = "C:\Users\lingaguru.c3\Desktop\Test\DE.txt"
# using a template filename saves writing
$pathOut = "C:\Users\lingaguru.c3\Desktop\Test\Temp part {0} file.tsv"
$partNumber = 1
$batchSize = 20000 # max number of data lines to write in each part
# construct the output filename using the template $pathOut
$pathAndFilename = $pathOut -f $partNumber
$enc = [System.Text.Encoding]::UTF8
$fs = [System.IO.FileStream]::new($sourceFile,"Open", "Read") # don't need write access on source file
$streamIn = [System.IO.StreamReader]::new($fs, $enc)
$streamout = [System.IO.StreamWriter]::new($pathAndFilename)
# assuming the first line contains the headers
$header = $streamIn.ReadLine()
# write out the header on the first part
$streamout.WriteLine($header)
$counter = 0
while (($line = $streamIn.ReadLine()) -ne $null) {
$streamout.WriteLine($line)
$counter++
if ($counter -ge $batchsize) {
$partNumber++
$counter = 0
$streamOut.Flush()
$streamOut.Dispose()
$pathAndFilename = $pathOut -f $partNumber
$streamout = [System.IO.StreamWriter]::new($pathAndFilename)
# write the header on this new part
$streamout.WriteLine($header)
}
}
$streamin.Dispose()
$streamout.Dispose()
$fs.Dispose()