How can I remove <math></math> multiline sections with Perl? - perl

Question

How can I remove multiline sections with Perl?
I have such wiki test code:
{|
|-
| colspan="2"|
: <math>
[\underbrace{\color{Red}4,2}_{4 > 2},5,1,7] \rightarrow
[2,\underbrace{\color{OliveGreen}4,5}_{4 < 5},1,7] \rightarrow
[2,4,\underbrace{\color{Red}5,1}_{5 > 1},7] \rightarrow
[2,4,1,\underbrace{\color{OliveGreen}5,7}_{5 < 7}]
</math>
|-
|
: <math>
[\underbrace{\color{OliveGreen}2,4}_{2 < 4},1,5,{\color{Blue}7}] \rightarrow
[2,\underbrace{\color{Red}4,1}_{4 > 1},5,{\color{Blue}7}] \rightarrow
[2,1,\underbrace{\color{OliveGreen}4,5}_{4 < 5},{\color{Blue}7}]
</math>
: <math>
[\underbrace{\color{Red}2,1}_{2 > 1},4,{\color{Blue}5},{\color{Blue}7}] \rightarrow
[1,\underbrace{\color{OliveGreen}2,4}_{2 < 4},{\color{Blue}5},{\color{Blue}7}]
</math>
: <math>
[\underbrace{\color{OliveGreen}1,2}_{1 < 2},{\color{Blue}4},{\color{Blue}5},{\color{Blue}7}]
</math>
|}
And I want to remove from this code all how to do it? I have done such code:
cat math-text.txt | perl -e 'while(<>) { s/<math>.+?<\/math>//gs; print $_; }'
It is not works but should since documentation explains that . will much new lines. How to do it?

Answer 1

The following is a python script which I use to extract all the mathematical formula from wikipedia dumps. Rather than using a multi-line regexp it scans for occurrences of <math> </math> and uses the position on the line to work out where the actual position on the line is and uses a finite state machine to find the actual equations, basically with two states determined by inEqn. It does a few other things like find the title and name space and attributes in the maths tags.
As dumps are in the order of 100MB using a line by line approach may well end up being more efficient than multi-line regexps.
import sys
import re
titleRE = re.compile('<title>(.*)</title>')
nsRE = re.compile('<ns>(.*)</ns>')
mathRE = re.compile('</?math(.*?)>')
pageEndRE = re.compile('</page>')
title =""
attr = ""
ns = -1
inEqn = 0
for line in sys.stdin:
m = titleRE.search(line)
if m :
title = m.group(1)
expression = ""
inEqn = 0
m = nsRE.search(line)
if m :
ns = m.group(1)
start = 0
pos = 0
m = mathRE.search(line,pos)
while m :
if m.group().startswith('<math'):
attr = m.group(1)
start = m.end()
pos = start
expression = ""
inEqn = 1
if m.group() == '</math>' :
end = m.start()
expression = ' '.join([expression,line[start:end]])
print title,'\t',attr,'\t',expression.lstrip().replace('<','<').replace('>','>').replace('&','&')
pos = m.end()
expression = ""
start = 0
inEqn = 0
m = mathRE.search(line,pos)
if start > 0 :
expression = line[start:].rstrip()
elif inEqn :
expression = ' '.join([expression,line.rstrip()])
Another option might be to consider an xml parser. A SAX or DOM based parser would be able to find the equations. This might be worth considering if you want to do more sophisticated analysis of the wiki-text.