Creating csv from an XML that has repeating tags - python-3.7

Question

I am trying to normalize an xml file that has a parent tag with repeating child tags. Please see a sample of the data below:
<GeneralQuestions>
<HeaderText>Pain Management Assessment</HeaderText>
<QuestionText>Pain assessment</QuestionText>
<QuestionAnswer>Yes</QuestionAnswer>
<HeaderText>Activities of Daily Living</HeaderText>
<QuestionText>Patient walks</QuestionText>
<QuestionAnswer>With Some Help</QuestionAnswer>
<Score>1</Score>
<HeaderText>Pain Management Assessment</HeaderText>
<QuestionText>Patient consents to having Pain Management Assessment screening completed.</QuestionText>
<QuestionAnswer>Patient accepts</QuestionAnswer>
<HeaderText>Activities of Daily Living</HeaderText>
<QuestionText>Patient gets dressed</QuestionText>
<QuestionAnswer>With Some Help</QuestionAnswer>
<Score>1</Score>
</GeneralQuestions>
You'll observe that the child tags in "GeneralQuestions" are getting repeated with the child tag "Score" being optional. I am trying to convert it to a normalized form in which each set of child tags form a row, like shown below:
HeaderText, QuestionText,QuestionAnswer,Score
HeaderText, QuestionText,QuestionAnswer,Score
HeaderText, QuestionText,QuestionAnswer,Score
If the "Score" is missing, I want null value. I am using python 3.7 and xml.etree.ElementTree.iterparse to parse the data. Please let me know how I can normalize the data.

Answer 1

Here an incomplete answer that can help you using numpy to number the tags. I assumed some structure which is same. Say we have
XML = """<GeneralQuestions>
<HeaderText>Pain Management Assessment</HeaderText>
<QuestionText>Pain assessment</QuestionText>
<QuestionAnswer>Yes</QuestionAnswer>
<HeaderText>Activities of Daily Living</HeaderText>
<QuestionText>Patient walks</QuestionText>
<QuestionAnswer>With Some Help</QuestionAnswer>
<Score>1</Score>
<HeaderText>Pain Management Assessment</HeaderText>
<QuestionText>Patient consents to having Pain Management Assessment screening completed.</QuestionText>
<QuestionAnswer>Patient accepts</QuestionAnswer>
<HeaderText>Activities of Daily Living</HeaderText>
<QuestionText>Patient gets dressed</QuestionText>
<QuestionAnswer>With Some Help</QuestionAnswer>
<Score>1</Score>
</GeneralQuestions>"""
And we create a tree with
import xml.etree.ElementTree as ET
tree = ET.fromstring(XML)
Than with using numpy we can create an index with
import numpy as np
index_of_score = np.cumsum( [ e.tag == 'HeaderText' for e in tree.getchildren() ] )
Now with help of the index_of_score you can create a dictionary with indexed tags, and possible value
{ "{}_{}".format(a.tag,i) : a.text for i,a in zip(index_of_score, tree.getchildren() ) }
which will give you
{'HeaderText_1': 'Pain Management Assessment',
'QuestionText_1': 'Pain assessment',
'QuestionAnswer_1': 'Yes',
'HeaderText_2': 'Activities of Daily Living',
'QuestionText_2': 'Patient walks',
'QuestionAnswer_2': 'With Some Help',
'Score_2': '1',
'HeaderText_3': 'Pain Management Assessment',
'QuestionText_3': 'Patient consents to having Pain Management Assessment screening completed.',
'QuestionAnswer_3': 'Patient accepts',
'HeaderText_4': 'Activities of Daily Living',
'QuestionText_4': 'Patient gets dressed',
'QuestionAnswer_4': 'With Some Help',
'Score_4': '1'}
Depending on your desired output you can cherry pick the values you need. Say the above dictionary is dict_output, python has the nice dict_output.get("Score_1", None) which will give you either the value or in this case None which might help you in processing the data.

Answer 2

I created a context from the XML file:
xmlIter=ET.iterparse('C:\\Users\\ANAND_RA\\Documents\\Project\\XXXXXXX_MA_05042020_0.xml', events=('start','end'))
context=iter(xmlIter)
Then I used a for loop to parse through each tag:
for eachEvent, eachElement in context:
The final step (the most important one) is to process the tag "GeneralQuestions" under the for loop:
if eachElement.tag=='GeneralQuestions' and eachEvent=='start':
GQstart=True
GQcount=0
GQlist=[]
GQDataList=[]
if eachElement.tag=='HeaderText' and GQstart and eachEvent=='start':
if GQcount!=0:
if len(GQlist)<6:
GQlist.extend([None,1,today,today])
else:
GQlist.extend([1,today,today])
GQDataList.append(tuple(GQlist))
GQcount+=1
GQlist=[]
GQlist.extend([nextDLMemberSK,GQcount,eachElement.text])
if eachElement.tag=='QuestionText' and GQstart and eachEvent=='start':
GQlist.append(eachElement.text)
if eachElement.tag=='QuestionAnswer' and GQstart and eachEvent=='start':
GQlist.append(eachElement.text)
if eachElement.tag=='Score' and GQstart and eachEvent=='start':
GQlist.append(eachElement.text)
if eachElement.tag=='GeneralQuestions' and eachEvent=='end' and GQcount!=0:
if len(GQlist)==5:
GQlist.extend([None,1,today,today])
else:
GQlist.extend([1,today,today])
GQDataList.append(tuple(GQlist))
if eachElement.tag=='GeneralQuestions' and eachEvent=='end' and len(GQDataList)>0:
# print(GQDataList)
cur1.executemany(SQL_INS_INOV_GENERALQ,GQDataList)
Processing an XML is quite different from processing a json file. In json, the data inspection is more flexible. However, in xml data you have to inspect the tags as it comes in the for loop.