How to set the FSM configuaration for Textricator PDF OCR reader? - itext

Question

I'm trying to use the PDF document parser called Textricator. It can use 3 different methods for parsing a PDF with some common OCR libraries. (itext5, itext7, pdfbox) The available methods are: text, table and form. Text for normal raw OCR recognition, table to read out structured table data, and form for parsing less structured forms, using a Finite State Machine (FSM).
However, I am not able to use the form parser. Perhaps I simply don't understand how to organize the many configuration states. The documentation is lacking a simple form example, and someone recently posted an attempt to read a very basic table using the form method, but was not able to. I also gave it a shot, but without any success.
Q: Can someone help me configure the state machine in the YML file?
(This is used to parse the demo file from one of that repo's issues, and shown in the copied screenshot below.)
The YML configuration file.
extractor: "pdf.pdfbox"
header:
default: 100
footer:
default: 600
maxRowDistance: 2
rootRecordType: item
recordTypes:
item:
label: "item"
valueTypes:
- item
- date
- description
- order_number
- quantity
- price
valueTypes:
item:
label: "Item"
date:
label: "Date"
description:
label: "Description"
order_number:
label: "OrderNo"
quantity:
label: "Qty"
price:
label: "Price"
initialState: "INIT"
states:
INIT:
transitions:
-
condition: item
nextState: item
item:
startRecord: true
transitions:
-
condition: date
nextState: date
date:
include: true
transitions:
-
condition: description
nextState: description
description:
include: true
transitions:
-
condition: description
nextState: description
-
condition: order_number
nextState: order_number
-
condition: quantity
nextState: quantity
order_number:
include: true
transitions:
-
condition: order_number
nextState: order_number
-
condition: quantity
nextState: quantity
quantity:
include: true
transitions:
-
condition: price
nextState: price
price:
include: true
transitions:
-
condition: end
nextState: end
end:
include: false
transitions:
-
condition: any
nextState: end
conditions:
item: '73 < ulx < 110 and text =~ /(\\d)*/'
date: '110 < ulx < 181 and text =~ /([0-9\-]*)/'
description: '193 < ulx < 366'
# order_number: '12 <= uly_rel <= 16 and text =~ ^.+/((\d{6})\-)((\d{2}))/'
order_number: '12 <= uly_rel <= 16 and text =~ ^.+((\d{6})\-)((\d{2}))'
quantity: '393 < ulx < 459'
price: '459 < ulx < 523'
end: 'text =~ /(Footer)/'
any: "1 = 1"
You may wonder why I am insisting in using the form processor for this simple example, but it is because in my real life document I will have a much more complex sub-structure of child items under the Description field. This can only (?) be processed efficiently by a state-machine, AFAIK.
But, maybe this is not the right tool for the job? So what other options are there?
UPDATE: (2021-05-18)
The author of Textricate has now bumped the libraries used, the documentation and corrected several working examples and user issues. Thanks to user mweber I now have a perfectly working parser and no longer need to use awk to handle weird columns.

Answer 1

As Textricator is kind of a hidden gem for pdf parsing imo, I'm happy to see someone using it and posted a config working with the sample document to the github issue:
extractor: "pdf.pdfbox"
header:
default: 100
footer:
default: 600
maxRowDistance: 2
rootRecordType: item
recordTypes:
item:
label: "item"
valueTypes:
- item
- date
- description
- order_number
- quantity
- price
valueTypes:
item:
label: "Item"
date:
label: "Date"
description:
label: "Description"
order_number:
label: "OrderNo"
quantity:
label: "Qty"
price:
label: "Price"
initialState: "INIT"
states:
INIT:
include: false
transitions:
-
condition: item
nextState: item
- condition: any
nextState: INIT
item:
startRecord: true
transitions:
-
condition: date
nextState: date
date:
include: true
transitions:
-
condition: description
nextState: description
description:
include: true
transitions:
-
condition: description
nextState: description
-
condition: order_number
nextState: order_number
-
condition: quantity
nextState: quantity
-
condition: item
nextState: item
order_number:
include: true
transitions:
-
condition: order_number
nextState: order_number
-
condition: quantity
nextState: quantity
quantity:
include: true
transitions:
-
condition: price
nextState: price
price:
include: true
transitions:
-
condition: end
nextState: end
-
condition: description
nextState: description
-
condition: item
nextState: item
end:
include: false
transitions:
-
condition: any
nextState: end
conditions:
item: '73 < ulx < 110 and text =~ /(\\d)*/'
date: '110 < ulx < 181 and text =~ /([0-9\\-]*)/'
description: '193 < ulx < 366'
order_number: '12 <= uly_rel <= 16 and text =~ /^.+(([0-9]{6})\\-)(([0-9]{2}))/'
quantity: '393 < ulx < 459'
price: '459 < ulx < 523'
end: 'text =~ /(Footer)/'
any: "1 = 1"