Modify Perl script to output new line every 20 words - perl

Question

I'm using an open-source perl script to create a text corpus based on the English language wikipedia dump. The plain text has been extracted, but various punctuation marks and the like still need to be removed. However, the output of this script essentially creates a 7.2GiB text file containing a single line. Due to my needs, I want to alter the script such that it inserts a new line character every 20 words.
So far, I've tried this:
$wordCount=0;
while (<STDIN>) {
$wordCount++;
//text processing regex commands here
# Remove ellipses
s/\.\.\./ /g;
# Remove dashes surrounded by spaces (e.g. phrase - phrase)
s/\s-+\s/ /g;
# Remove dashes between words with no spaces (e.g. word--word)
s/([A-Za-z0-9])\-\-([A-Za-z0-9])/$1 $2/g;
# Remove dash at a word end (e.g. three- to five-year)
s/(\w)-\s/$1 /g;
# Remove some punctuation
s/([\"\�,;:%�?�!()\[\]{}<>_\.])/ /g;
# Remove trailing space
s/ $//;
# Remove double single-quotes
s/'' / /g;
s/ ''/ /g;
# Replace accented e with normal e for consistency with the CMU pronunciation dictionary
s/�/e/g;
# Remove single quotes used as quotation marks (e.g. some 'phrase in quotes')
s/\s'([\w\s]+[\w])'\s/ $1 /g;
# Remove double spaces
s/ / /g;
chomp($_);
if ($wordCount == 20){
print uc($_) . "\n";
$wordCount=0;
}
print uc($_) . " ";
}
print "\n";
However, this doesn't seem to work, as the raw output has only newlines scattered around arbitrarily. I'd like to have the text formatted so it will fit on a typical 1200px wide monitor without word wrapping.
A sample input text from the file is
The Concise Oxford Dictionary of Politics. Proponents of anarchism
(known as "anarchists") advocate stateless societies as the only moral
form of social organization. There are many types and traditions of
anarchism, not all of which are mutually exclusive. Anarchism as a
social movement has regularly endured fluctuations in popularity. The
term anarchism derives from the Greek ἄναρχος, anarchos, meaning
"without rulers", its use as a synonym is still common outside the
United States. The earliest anarchist themes can be found in the 6th
century BC, among the works of Taoist philosopher Laozi, and in later
centuries by Zhuangzi and Bao Jingyan. The term "anarchist" first
entered the English language in 1642, during the English Civil War, as
a term of abuse, used by Royalists against their Roundhead opponents.
By the time of the French Revolution some, such as the Enragés, began
to use the term positively, in opposition to Jacobin centralisation
of power, seeing "revolutionary government" as oxymoronic. By the
turn of the 19th century, the English word "anarchism" had lost its
initial negative connotation. Modern anarchism sprang from the secular
or religious thought of the Enlightenment, particularly Jean-Jacques
Rousseau's arguments for the moral centrality of freedom. Anarchism",
Encarta Online Encyclopedia 2006 (UK version). From this climate
William Godwin developed what many consider the first expression of
modern anarchist thought. Godwin was, according to Peter Kropotkin,
"the first to formulate the political and economical conceptions of
anarchism, even though he did not give that name to the idea s
developed in his work", while Godwin attached his anarchist ideas to
an early Edmund Burke. The anarcho-communist Joseph Déjacque was the
first person to describe himself as "libertarian". Unlike Proudhon, he
argued that, "it is not the product of his or her labor that the
worker has a right to, but to the satisfaction of his or her needs,
whatever may be t heir nature. Jesus is sometimes considered the first
anarchist in the Christian anarchist tradition. Georges Lechartier
wrote that "The true founder of anarchy was Jesus Christ and . In
Europe, harsh reaction followed the revolutions of 1848, during which
ten countries had experienced brief or long-term social upheaval as
groups carried out nationalis t uprisings. After most of these
attempts at systematic change ended in failure, conservative elements
took advantage of the divided groups of socialists, anarchists,
liberals, and na tionalists, to prevent further revolt. Blanquists,
Philadelphes, English trade unionists, socialists and social
democrats. Due to its links to active workers' movements, the
International became a significant organization. Karl Marx became a
leading figure in the International and a member of its General
Council. Proudhon's followers, the mutualists, opposed Marx's state
socialism, advocating political abstentionism and small property
holdings. In 1868, following their unsuccessful participation in the
League of Peace and Freedom (LPF), Russian revolutionary Mikhail
Bakunin and his collectivist anarchist associa tes joined the First
International (which had decided not to get involved with the LPF). At
first, the collectivists worked with the Marxists to push the First
International in a more revolutionary socialist direction.
Subsequently, the International became polarised into two camps, with
Marx and Bakunin as their respective figureheads. In 1872, the
conflict climaxed with a final split between the two groups at the
Hague Congress, where Bakunin and James Guillaume were expelled from
the International and its headquarters were transferred to New York.
In response, the federalist sections formed their own International at
the St. Imier Congress, adopting a revolutionary anarchist program.
Black Rose Books 2005) ISBN 1-55164-251-4.
There's 7-something gigs worth of text in the file. So using a list or other data structure might be a bit of overkill for these requirements.
What is needed in order to fit my requirements?

Answer 1

Consider using something like Text::Wrap or Text::Autoformat .

Answer 2

open my $in, '<', $inFileName;
open my $out, '>', $outFileName;
my $wordcount = 0;
while(defined( my $line = <$in> )){
$line=~s/\n//g; #remove newline character
#split the words into an array(could use '\W+' instead of ' ')
my #words = split ' ', $line;
foreach my $word (#words){
$wordCount++;
if ($wordCount == 20){
$wordCount = 0;
print $out "\n";
}
else {
print $out uc($word)." ";
}
} # end of foreach line in input
} # end of file while loop
close $in;
close $out;

Answer 3

First, set perl's input record separator to something frequent and useful, like a space:
$/ = ' ';
then loop over the input word by word:
while (<>) {
trim the word:
s/^\s+|\s+$//g;
skip it if it was all space:
$_ or next;
do any other transforms you need
and then add it to a stack, splitting any internal tabs or other space-like characters:
push #words, split /\s+/;
next, check to see if you have 20 words, and if so, print them:
print join(' ' => splice #words, 0, 20), "\n" while #words >= 20;
}
then print anything remaining:
print "#words\n" if #words;

Answer 4

Without knowing more details about this problem, I'd suggest a brute force solution:
slurp the entire entry,
split to an array based on " ",
foreach the array and print "\n" after every 20 elements.

Answer 5

True to Perl, there are various ways to solve this, but one (perverse?!) way to do it is to read the file byte by byte instead of line by line, or slurping the whole thing in. It's rather brute force-ish but it works. Essentially you are trading memory use for disk usage.
#!/usr/bin/perl -w
use strict;
open(IN, "in.txt") or die;
my $rc = 1;
my $wc = 0;
my $new;
while ($rc != 0)
{
# Read a byte - not safe for Unicode or other double-byte environments!
$rc = read IN, $new, 1, 0;
# We're only interested if the byte isn't punctuation (POSIX character class).
if ($new !~ m/[[:punct:]]/)
{
# word boundary?
if ($new =~ m/ /)
{
$wc++;
if ($wc % 20 == 0)
{
print "\n"; # 20th word, time for a new line.
}
}
print $new;
}
# move on to the next byte
seek IN, 0, 1;
}
close(IN);