Run sed for a range of lines FASTER - sed

Question

I have a file with millions of lines
And in a script I want to use the following line (which removes unprintable characters) in a loop for a given line range
sed -i $'s/[^[:print:]\t]//g' ~/test.txt
How do I do that ? It needs to run very fast
I've tried
sed -i $"{${line1},${line2}}{s/[^[:print:]\t]//g}" ~/test.txt
which is very slow for range of 1000 lines near the end of the file

Answer 1

I'm not sure of the composition of your test file (how many replacements are needed?), so I can't faithfully reproduce your test, but I'm using my /bin/bash (1168776 bytes in 5714 "lines", with 372073 (31.8%) printable/tab characters).
sed
(Baseline for timing purposes)
$ cp /bin/bash sh; time sed -i $'s/[^[:print:]\t]//g' sh
sed -i $'s/[^[:print:]\t]//g' sh 1.66s user 0.01s system 98% cpu 1.687 total
$ cp /bin/bash sh; time sed -i $'s/[^[:print:]\t]//g' sh
sed -i $'s/[^[:print:]\t]//g' sh 1.74s user 0.01s system 89% cpu 1.945 total
$ cp /bin/bash sh; time sed -i $'s/[^[:print:]\t]//g' sh
sed -i $'s/[^[:print:]\t]//g' sh 1.67s user 0.01s system 97% cpu 1.718 total
Mean average of total times = 1.783s 🚶 (it's important to run multiple times to control for caching. I ran four times and dropped the first to account for caching, then averaged to control for externalities like my web browser)
perl
I translated this to perl to see if that would be faster:
$ cp /bin/bash sh; time perl -i -pe $'s/[^[:print:]\t]//g' sh
perl -i -pe $'s/[^[:print:]\t]//g' sh 0.18s user 0.01s system 92% cpu 0.208 total
$ cp /bin/bash sh; time perl -i -pe $'s/[^[:print:]\t]//g' sh
perl -i -pe $'s/[^[:print:]\t]//g' sh 0.18s user 0.01s system 68% cpu 0.271 total
$ cp /bin/bash sh; time perl -i -pe $'s/[^[:print:]\t]//g' sh
perl -i -pe $'s/[^[:print:]\t]//g' sh 0.21s user 0.00s system 81% cpu 0.258 total
Mean average of total times = 0.246s 🏃‍
However, I noticed some differences. GNU sed is struggling, perhaps due to either a different definition of the [:print:] class or (more likely) different handling of control characters:
$ sed $'s/[^[:print:]\t]//g' /bin/bash |head -c64 |hd
00000000 45 4c 46 3e 30 f6 40 48 ce 40 38 40 40 40 40 68 |ELF>0.#H.#8####h|
00000010 68 a8 a8 a8 98 cd 98 cd d0 d0 d0 8d d7 0a 8d d7 |h...............|
00000020 0a b0 b0 b0 30 57 30 57 f0 f0 23 f0 23 b9 a8 55 |....0W0W..#.#..U|
00000030 f0 3c f0 4c f0 4c c4 c4 c4 44 44 50 e5 74 64 30 |.<.L.L...DDP.td0|
00000040
$ perl -pe $'s/[^[:print:]\t]//g' /bin/bash |head -c64 |hd
00000000 45 4c 46 3e 30 40 48 40 38 40 40 40 40 68 68 30 |ELF>0#H#8####hh0|
00000010 57 30 57 23 23 55 3c 4c 4c 44 44 50 74 64 30 49 |W0W##U<LLDDPtd0I|
00000020 30 49 30 49 44 44 51 74 64 52 74 64 23 23 2c 2c |0I0IDDQtdRtd##,,|
00000030 2f 6c 69 62 36 34 2f 6c 64 2d 6c 69 6e 75 78 2d |/lib64/ld-linux-|
00000040
See all those dots in the GNU sed output? Those are failures to replace content. I also observe these in Busybox sed. BSD sed (which is used by Mac OS X) does not appear to have this limitation, but note this for portability purposes as needed.
tr
$ cp /bin/bash sh; time tr -cd $'[ -~\t\n]' < sh > sh-tr && mv sh-tr sh
tr -cd $'[[:print:]\t]' < sh > sh-tr 0.00s user 0.01s system 62% cpu 0.012 total
$ cp /bin/bash sh; time tr -cd $'[ -~\t\n]' < sh > sh-tr && mv sh-tr sh
tr -cd $'[[:print:]\t]' < sh > sh-tr 0.00s user 0.01s system 81% cpu 0.009 total
$ cp /bin/bash sh; time tr -cd $'[ -~\t\n]' < sh > sh-tr && mv sh-tr sh
tr -cd $'[[:print:]\t]' < sh > sh-tr 0.00s user 0.01s system 82% cpu 0.012 total
Mean average of total times = 0.011s ⚡️
I've tested this with GNU tr and Busybox tr (they have equal performance). We're using tr to delete (-d) rather than translate, and we're acting on the complement (-c) of the given class (tr does not use regex, so we can't invert the character class with a caret the way we can in sed).
Busybox tr does not support $'[[:print:]\t]' so I have converted it to a range from space to tilde (all printable lower-ASCII except tab and newline) and I added not just tab but also newline since tr needs to explicitly preserve that character (sed did not). If the lines don't match properly, consider adding \r to the replacement set.
strings is also good here, but it does not preserve lines (it replaces each contiguous string of non-printable characters with a newline)