sed command not working ...how to strip a pipe out - postgresql

Question

I have a data file I am trying to import into redshift (postgress mpp database). I am trying to import into postgres with a '|' delimiter. But some data, has the '|' in the string data itself, for example :
73779087|"UCGr4c0_zShyHbctxJJrJ03w"|"ItsMattSpeaking | SuPra"
So I tried this sed command :
sed -i -E "s/(.+|)(.+|)|/\1\2\\|/g" inputfile.txt >outputfile.txt
Any ideas on what is wrong with the sed command, to replace the | in the last string with a \| escape character so that Redshift will not view this as a delimiter ? Any help is appreciated.

Answer 1

This might work for you (GNU sed):
sed -r ':a;s/^([^"]*("[^"|]*"[^"]*)*"[^"|]*)\|/\1/g;ta' file
This removes | within double quotes however it does not cater for quoted quotes, so beware!

Answer 2

There are somethings you don't use SED for, I'd say this is one of those things. Try using a python script with the re library or just plain string manipulation.

Answer 3

I think this C++ code does what you need.
// $ g++ -Wall -Wextra -std=c++11 main.cpp
#include <iostream>
int main(int, char*[]) {
bool str = false;
char c;
std::ios_base::sync_with_stdio(false);
std::cin.tie(nullptr);
while (std::cin.get(c)) {
if (c == '|') {
if (str) {
std::cout << '\\'; } }
else if (c == '"') {
// Toggle string parsing.
str = !str; }
else if (c == '\\') {
// Skip escaped chars.
std::cout << c;
std::cin.get(c); }
std::cout << c; }
return 0; }
The problem with sed in this example is you need to know more than the basics in order to track which state you are in (string or not).

Answer 4

Here's a script for translating a file with pipe-separated values, as described, to one following the simpler conventions of a TSV file. It assumes the availability of a PHP interpreter. If the script is saved as psv2tsv and made executable in a Mac or Linux environment, then psv2tsv -h should offer more details.
Example usage (using <TAB> to indicate a TAB in the output):
$ psv2tsv <<< $'73779087|"UCGr4c0_zShyHbctxJJrJ03w"|"ItsMattSpeaking | SuPra"'
73779087<TAB>UCGr4c0_zShyHbctxJJrJ03w<TAB>ItsMattSpeaking | SuPra<TAB>
$ psv2tsv <<< $'a|"b|c\t\d"|"e\n"'
a<TAB>b|c\t\d<TAB>e\n<TAB>
The script:
#!/usr/bin/env php
<?php
# Author: pkoppstein at gmail.com 12/2015
# Use at your own risk.
# Input - pipe-separated values along the lines of CSV.
# Translate embedded newline and tab characters.
function help() {
global $argv;
echo <<<EOT
Syntax: {$argv[0]} [filepath]
Convert a file or stream of records with pipe-separated values to the
TSV (tab-separated value) format. If no argument is specified, or if
filepath is specified as -, then input is taken from stdin.
The input is assumed to be like a CSV file but with pipe characters
(|) used instead of commas. The output follows the simpler
conventions of TSV files.
Note that each tab in the input is translated to "\\t", and each
embedded newline is translated to "\\n". Each translated record is
then written to stdout. See PHP's fgetcsv for further details.
EOT;
}
$file = ($argc > 1 ) ? $argv[1] : 'php://stdin';
if ( $file == "-h" or $file == "--help" ) {
help();
exit;
}
if ( $file == "-" ) $file = 'php://stdin';
$handle = #fopen($file, "r");
if ($handle) {
while (($data = fgetcsv($handle,0,"|")) !== FALSE) {
$num = count($data);
for ($c=0; $c < $num; $c++) {
# str_replace( mixed $search , mixed $replace , mixed $subject [, int &$count ] )
echo str_replace("\t", "\\t", str_replace("\n", "\\n", $data[$c])) . "\t";
}
echo "\n";
}
fclose($handle);
}
else {
echo "{$argv[0]}: unable to fopen $argv[1]\n";
exit(1);
}
?>