Most efficient way to replace a line in a text document? - text-processing

Question

I am learning to code in Unix with C. So far I have written the code to find the index of the first byte of the line that I want to replace. The problem is that sometimes, the number of bytes replacing the line might be greater than the number of bytes already on the line. In this case, the code start overwriting the next line. I came up with two standard solutions:
a) Rather than trying to edit the file in-place, I could copy the entire file into memory, edit it by shifting all the bytes if necessary and rewriting it back to file.
b) Only copy the line I want to end-of-file to memory and edit.
Both suggestions doesn't scale well. And I don't want to impose any restrictions on the line size(like every line must be 50 bytes or something). Is there any efficient way to do the line replacement ? Any help would be appreciated.

Answer 1

With text files you always have to "spool" them as the text to delete/insert/replace will nearly always be larger or smaller than what was there.
"Spooling" means to open a temp file in the file's directory, reading the original file and writing it to the temp file, stop where the replace/insert/delete starts, do your thing and copy the remainder to the output. If everything went fine, then unlink the original file and rename the new file to the old file.
P.s.: if you don't want to have restrictions on line size, then you must use fgetc/fputc to process character-by-character (no sweat; C can be pretty fast, your disks permitting).

Answer 2

I actually came across this problem last month with log files that had grown to 30 GB and one line. Utils like sed, perl wanted to consume all available memory to do anything with them at all. So technically, neither of your solutions scale well. But in practice, they're fine, with (b) being preferred. You should use fgets with a buffer size of, say, 8kB and iterate until the last character is a newline or you've reached EOF. In my soultion, I used perl's sysread function and read 16 kB chunks at a time.
From memory:
#define BUF_SZ 16383
char *buf = alloca(BUF_SZ + 1);
infile = fopen(...);
while (!feof(infile) && fgets(buf, BUF_SZ, infile) != NULL) {
readmore = (buf[0] != '\0' && buf[ strlen(buf)-1 ] != '\n');
/* other processing
.
.
*/
if (readmore) {
/* apply different strategies for dealing with buf */
}
}
I think the strategy really depends on what you're trying to do. If you want to remove the line or truncate it, but you need only match the beginning of the line, then it's pretty trivial (no special code). However, if you need to do a long pattern match that might extend past the first 16kB, then you have to do something like move the last n bytes (where n is the maximize size of your search pattern) to the beginning of buf
and do the next read into &buf[ n ].
You will output to a new filehandle, and when everything is done and done correctly, you unlink the first file, and rename the new file to the old one. Also research the mktemp for creating the temporary file in the same directory and the atexit call for cleaning up after in case there was an error.

Answer 3

The most efficient way to replace a line of text in a file depends on a number of things.[1] The primary concern when wanting an efficient search and replace is to minimize the number of file reads/writes as file I/O is generally an order of magnitude slower than memory operations. The trivial case occurs when the search and replacement string have the exact number of characters. In that, and only in that, case can you look to do an in-file replacement of the text without having to write a second (or temporary file).
Given the file I/O efficiencies, the most efficient (fastest) way of performing the search/replace is to either mmap the entire file or use sendfile. Both can make use of kernel-space copying of blocks of text which generally yields significant improvement over user-space copy operations. Neither are that difficult. The next best option is to use a buffered read to read the entire contents of the file into memory and then perform a search on the memory buffer to identify the location (address) where the contents are to be changed. You can then incrementally write the buffer out to a third file writing the replacement text at each required location identified during search of the original buffer.
Given the concerns in your example, there is no need to read the entire file into memory at once (even though files are rarely larger than INT_MAX bytes). With embedded systems, etc. where both memory storage and efficiency are challenges, you can simply set a block size of some arbitrary size that meets your size constraints and then read the file one-block of memory at a time, perform the search/replace (handling corner cases as required, e.g. where less than a search length of characters containing the first part of the search string remain in a given block, etc...)
The key is to minimize the number of times you go back to the drive for more information or write from your buffer to the drive. So generally the larger the block the better.
The following is a short minimum example that will read a given file in block-sized memory chunks as specified by the user (or as limited by the file-size itself if smaller than the requested block size). The code simply reads a block of data from a file, uses memchr to locate each beginning character in the search term. When the beginning character is found, memcmp is used to check a search-term-length of memory beginning at the matched character. Depending on whether the comparison matches or fails, the various indexes are updated and the search continues.
Data is output to the new file (stdout) in this case, each time a matching term is found within a block of memory, or at the end of the block if no matching terms are found. This code can be further optimized and additional sanity check can always be added. Look over the example and let me know if you have any questions. None of it is difficult, but there are a couple of separate indexes necessary (e.g. current buffer position, and last read/write position) to perform the search/replace. The example that follows uses a text file that contains several paragraphs that discus "personal injury" where "hygiene" is substituted for "injury" and the results are output to stdout.
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <limits.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
#define BUFSZ (1 << 20) /* default max block size (1M) */
void *find_rplc_file (char *srch, char *rplc, FILE *ifp, FILE *ofp, long blksz);
int main (int argc, char **argv) {
if (argc != 4) {
fprintf (stderr, "error: insufficient input.\n"
"usage: %s infile <search> <replace>\n", argv[0]);
return 1;
}
FILE *ifp = fopen (argv[1], "rb");
if (!ifp) {
fprintf (stderr, "error: file open failed '%s'.\n", argv[1]);
return 1;
}
if (!find_rplc_file (argv[2], argv[3], ifp, stdout, BUFSZ)) {
fprintf (stderr, "error: find/replace failure.\n");
return 1;
}
putchar ('\n');
fclose (ifp);
return 0;
}
void *find_rplc_file (char *srch, char *rplc, FILE *ifp, FILE *ofp, long blksz)
{
if (!ifp || !srch || !rplc || !blksz) return NULL;
char *fb, *filebuf = NULL;
size_t offset = 0, nbytes = 0, readsz = 0, rlen, slen;
long bytecnt = 0, readpos = 0, size = 0;
rlen = strlen (rplc); /* length of search/replace text */
slen = strlen (srch);
fseek (ifp, 0, SEEK_END);
if ((size = ftell (ifp)) == -1) { /* get file length */
fprintf (stderr, "error: unable to determine file length.\n");
return NULL;
}
fseek (ifp, 0, SEEK_SET);
/* limit blksz to less or INT_MAX or blksz */
blksz = blksz > INT_MAX ? INT_MAX : blksz;
/* validate blksz does not exceed file size */
readsz = blksz > size ? size : blksz;
/* allocate memory for filebuf */
if (!(filebuf = calloc (readsz, sizeof *filebuf))) {
fprintf (stderr, "error: virtual memory exhausted.\n");
return NULL;
}
/* read entire file readsz bytes at a time */
while ((nbytes = fread (filebuf, sizeof *filebuf, readsz, ifp))) {
if (nbytes != readsz) fprintf (stderr, "warning: short read.\n");
readpos = 0; /* initialize read position & pointer */
fb = filebuf;
/* for each occurrence of 1st char of search term */
while ((fb = memchr (fb, *srch, nbytes - offset))) {
/* set current offset in buffer */
offset = fb - filebuf;
/* if less than length of search term remains */
if (offset + slen > nbytes) {
nbytes = offset; /* set nbytes to current offset */
/* reset file pointer to account for nbytes reduction */
fseek (ifp, bytecnt + nbytes, SEEK_SET);
goto getnext; /* read next block from here */
}
/* otherwise compare fb to search term */
if (memcmp (srch, fb, slen) == 0) {
/* if term found, write prior buffer to output file */
fwrite (filebuf + readpos, sizeof *filebuf,
offset - readpos, ofp);
/* write replacement text */
fwrite (rplc, sizeof *rplc, rlen, ofp);
/* set next readpos to 1st char following search term */
readpos = offset + slen;
}
fb++; /* advance fb pointer for next memchr search */
}
getnext:
bytecnt += nbytes; /* increment bytecnt with bytes searched */
/* write remaining buffer to output file */
fwrite (filebuf + readpos, sizeof *filebuf,
nbytes - readpos, ofp);
/* check file complete */
if (bytecnt == size) break;
/* set next read size (either blksz or remaining chars < blksz) */
readsz = size - bytecnt > blksz ? blksz : size - bytecnt;
}
/* validate all bytes successfully read */
if ((long)bytecnt != size) {
fprintf (stderr, "error: file read failed.\n");
return NULL;
}
free (filebuf); /* free filebuf */
return srch; /* return something other than NULL for success */
}
Example Input
$ cat dat/damages.txt
Personal injury damage awards are unliquidated
and are not capable of certain measurement; thus, the
jury has broad discretion in assessing the amount of
damages in a personal injury case. Yet, at the same
time, a factual sufficiency review insures that the
evidence supports the jury's award; and, although
difficult, the law requires appellate courts to conduct
factual sufficiency reviews on damage awards in
personal injury cases. Thus, while a jury has latitude in
assessing intangible damages in personal injury cases,
a jury's damage award does not escape the scrutiny of
appellate review.
Because Texas law applies no physical manifestation
rule to restrict wrongful death recoveries, a
trial court in a death case is prudent when it chooses
to submit the issues of mental anguish and loss of
society and companionship. While there is a
presumption of mental anguish for the wrongful death
beneficiary, the Texas Supreme Court has not indicated
that reviewing courts should presume that the mental
anguish is sufficient to support a large award. Testimony
that proves the beneficiary suffered severe mental
anguish or severe grief should be a significant and
sometimes determining factor in a factual sufficiency
analysis of large non-pecuniary damage awards.
Search "injury" Replace "hygiene"
$ ./bin/fread_blks_min dat/damages.txt "injury" "hygiene"
Personal hygiene damage awards are unliquidated
and are not capable of certain measurement; thus, the
jury has broad discretion in assessing the amount of
damages in a personal hygiene case. Yet, at the same
time, a factual sufficiency review insures that the
evidence supports the jury's award; and, although
difficult, the law requires appellate courts to conduct
factual sufficiency reviews on damage awards in
personal hygiene cases. Thus, while a jury has latitude in
assessing intangible damages in personal hygiene cases,
a jury's damage award does not escape the scrutiny of
appellate review.
Because Texas law applies no physical manifestation
rule to restrict wrongful death recoveries, a
trial court in a death case is prudent when it chooses
to submit the issues of mental anguish and loss of
society and companionship. While there is a
presumption of mental anguish for the wrongful death
beneficiary, the Texas Supreme Court has not indicated
that reviewing courts should presume that the mental
anguish is sufficient to support a large award. Testimony
that proves the beneficiary suffered severe mental
anguish or severe grief should be a significant and
sometimes determining factor in a factual sufficiency
analysis of large non-pecuniary damage awards.
Footnote[1]: the most efficient way is to use one shell tools designed for the task such as sed or awk, which both have years of development and fairly good text handling optimizations built in.