Merge columns of data from multiple text files by row from each seperate file using Powershell - powershell

Question

I have output from a numerical modelling code. I needed to extract a specific value from a series of files. I used the following code to get it (I derived this from an example that would extract IP addresses from logfiles):
$input_path = ‘C:\_TEST\Input_PC\out5.txt’
$output_file = ‘C:\_TEST\Output_PC_All\out5.txt’
$regex = ‘\bHEAD(.+)\s+[\-]*\d{1,3}\.\d{6,6}\s?\b’
select-string -Path $input_path -Pattern $regex -AllMatches | % { $_.Matches } | % { $_.Value } > $output_file
So I now have got a number of text files which contain measurements (the number of files may be variable, currently there are 50) with one column of numeric data (with a number of rows which currently equals 7302 but which may vary depending on the length of the time series modelled) and which may be positive or negative as per the example data below.
Note a semicolon preceding the text indicates that what follows is a comment I am using to explain the order of the dataset and does not appear in the data to be processed...
out1.txt
-1.000000 ; 1st line of out1.txt
2.000000 ; 2nd line of out1.txt
-3.000000 ; 3rd line of out1.txt
...
5.000000 ; nth line of out1.txt
out2.txt
-1.200000 ; 1st line of out2.txt
-2.200000 ; 2nd line of out2.txt
3.200000 ; 3rd line of out2.txt
...
-5.20000 ; nth line of out2.txt
outn.txt
1.300000 ; 1st line of outn.txt
-2.300000 ; 2nd line of outn.txt
-3.300000 ; 3rd line of outn.txt
...
10.300000 ; nth line of outn.txt
I need to merge them into a single text file (for this example lets call it "Combined_Output.txt") using Powershell with the data ordered so that the first row of values from the differing output files appear first, then repeat this for row 2 and so on as below:
Combined_Output.txt
-1.000000 ; 1st line of out1.txt
-1.200000 ; 1st line of out2.txt
1.300000 ; 1st line of outn.txt
2.000000 ; 2nd line of out1.txt
-2.200000 ; 2nd line of out2.txt
-2.300000 ; 2nd line of outn.txt
-3.000000 ; 3rd line of out1.txt
3.200000 ; 3rd line of out2.txt
-3.300000 ; 3rd line of outn.txt
...
5.000000 ; nth line of out1.txt
-5.200000 ; nth line of out2.txt
10.300000 ; nth line of outN.txt
Just to say that I'm very new to this sort of thing so I hope that the explanation above makes sense and also to say any help that you can provide would be much appreciated.
EDIT
Having now run the models, when using this code for the large data files created, there seems to be an issue of sorting of the imported data. This seems to occur primarily when there are repeated values for example the second row of data from each outfile has been combined in the following order by the script. It looks like there is some sorting based on the value of the data and not just based on the out file name:
Value ; out file text number
-1.215809 ; 1
-0.480543 ; 18
-0.480541 ; 19
-0.48054 ; 2
-0.480539 ; 20
-0.480538 ; 21
-0.480537 ; 22
-0.480536 ; 23
-0.480535 ; 24
-0.480534 ; 25
-0.480534 ; 26
-0.480688 ; 10
-0.480533 ; 27
-0.480532 ; 3
-0.480776 ; 4
-0.48051 ; 5
-0.48051 ; 6
-0.48051 ; 7
-0.48051 ; 8
-0.48051 ; 9
-0.48051 ; 11
-0.48051 ; 12
-0.48051 ; 13

Answer 1

I feel like I might have over complicated this answer but lets see how we do. Consider the following dummy data similar to your samples
Out1.txt Out2.txt Out3.txt
-0.40000 0.800000 4.100000
3.500000 0.300000 -0.90000
-2.60000 0.800000 2.200000
0.500000 1.800000 -1.40000
3.600000 1.800000 1.400000
40000000 -0.70000 1.500000
The file contents are arranged side by side for answer brevity and to help understand the output. The code is as follows:
$allTheFiles = #()
Get-ChildItem c:\temp\out*.txt | ForEach-Object{
$allTheFiles += ,(Get-Content $_.FullName)
}
For ($lineIndex=0; $lineIndex -lt $allTheFiles[0].Count; $lineIndex++){
For($fileIndex=0; $fileIndex -lt $allTheFiles.Count; $fileIndex++){
$allTheFiles[$fileIndex][$lineIndex]
}
} | Out-File -FilePath c:\temp\file.txt -Encoding ascii
Gather all out*.txt files the code creates an array of arrays which are the file contents themselves. Using nested For loops cycle though each single file outputting one line from each file at a time. While I am having a hard time being clear on what is happening if you compare the sample data to the output you should see that the first line or every file is outputted together followed by the next line...etc.
This code will produce the following output
-0.40000
0.800000
4.100000
3.500000
0.300000
-0.90000
-2.60000
0.800000
2.200000
0.500000
1.800000
-1.40000
3.600000
1.800000
1.400000
40000000
-0.70000
1.500000
Caveats
The code assumes that all files are of the same size. The number of lines is determined by the first file. If other files contain more data it would be lost in this model.