How to parse empty columns getting displaced within TSV? - perl

Question

I have a TSV file that I want to parse. There are empty fields in all columns resulting in displacement of the order of columns, so that not all the values I get using a certain column number actually come from that column.
Some fields contain long strings with empty space inside them. Also, some columns contain potential delimiters like ; | :
Input file
columnA columnB columnC columnD
A1 B1 C1 D1
B2 C2 D2
A3 D3
A4 B4 D4
Desired output
columnA columnB columnC columnD
A1 B1 C1 D1
B2 C2 D2
A3 D3
A4 B4 D4
$file myfile
`ASCII English text, with very long lines`
$awk '-F\t' '{print NF}' myfile | sort | uniq -c | tail -n
`247871 136`
I have found this code posted in reply to a similar question (https://unix.stackexchange.com/questions/29023/how-to-display-tsv-csv-in-console-when-empty-cells-are-missed-by-column-t), but I cannot make this work for my file:
sed ':x s/\(^\|\t\)\t/\1 \t/; t x' < file.tsv | column -t -s $'\t'
(The problem persists after importing into Excel.)

Answer 1

FieldEmpty=' '
FieldSize=${#FieldEmpty}
sed "
s/A/&/
t B
s/^ */ ${FieldEmpty}/
t B
: B
s/B/&/
t C
s/^ .\{${FieldSize}\}/&${FieldEmpty}/
t C
: C
s/C/&/
t D
s/^ \(.\{${FieldSize}\}\)\{2\}/&${FieldEmpty}/
t D
: D
s/D/&/
t
s/^ \(.\{${FieldSize}\}\)\{3\}/&${FieldEmpty}/
" YourFile
If more column are used, an iterative way is to be used instead (same concept of test/"insert")
On my AIX/KSH (so should be the same as with --posix -e on GNU sed)
$ cat YourFile
columnA columnB columnC columnD
A1 B1 C1 D1
B2 C2 D2
A3 D3
A4 B4 D4
$ FieldEmpty=' ';FieldSize=${#FieldEmpty};echo $FieldSize
11
$sed "..." YourFile
columnA columnB columnC columnD
A1 B1 C1 D1
B2 C2 D2
A3 D3
A4 B4 D4

Answer 2

If your file is tab separated, you should use tab as a field separator in awk. Like
$ column -t -s $'\t' file
columnA columnB columnC columnD
A1 1 B1 2 C1 3 D1 4
B2 2 C2 4 D2 4
A3 1 D3 4
A4 1 B4 2 D4 4
$xxd file
0000000: 636f 6c75 6d6e 4109 636f 6c75 6d6e 4209 columnA.columnB.
0000010: 636f 6c75 6d6e 4309 636f 6c75 6d6e 440a columnC.columnD.
0000020: 4131 2031 0942 3120 3209 4331 2033 0944 A1 1.B1 2.C1 3.D
0000030: 3120 340a 0942 3220 3209 4332 2034 0944 1 4..B2 2.C2 4.D
0000040: 3220 340a 4133 2031 0909 0944 3320 340a 2 4.A3 1...D3 4.
0000050: 4134 2031 0942 3420 3209 0944 3420 340a A4 1.B4 2..D4 4.
$ awk -F'\t' '{
for (i=1; i<=NF; i++) {
printf "%-8s ", $i
}
print ""
}'
columnA columnB columnC columnD
A1 1 B1 2 C1 3 D1 4
B2 2 C2 4 D2 4
A3 1 D3 4
A4 1 B4 2 D4 4