Reading Documents Beyond OCR
Reconstruct reading order, tables, and source relationships so a document becomes traceable evidence.
- Author
- Huang Tzu LinFounder
- Published
- Reading time
- 25 min
- Updated
Contents
- 01How This Lesson Connects
- 02Learning plan
- Prerequisites
- Learning Outcomes
- Practice exercise
- Failure exercise
- Completion check
- Check your understanding
- 03Why Plain-Text Retrieval Fails on Real Documents
- 04Choose Direct Text Extraction, OCR, or Image-Based Parsing
- 05Layout and Reading Order Carry Meaning
- 06Tables Are Structural Objects, Not Paragraphs with Delimiters
- 07One File May Contain Many Logical Documents
- 08Within-Document Structure and Across-Document Relationships
- 09Measure Each Reconstruction Stage
- 10Worked Example: Reconstructing an Evidence Packet
- 11Hands-On Exercise: Reconstruct the Mixed Packet
- 12Failure exercise
- 13Completion check
- 14What This Post Is Not
- 15Misconceptions That Cause Bad System Design
- 16Common Failure Modes in Production
- 17Practical Design Guidance
- 18Why This Leads Naturally to Multimodal Evidence Systems
- 19Source Notes
A common document-processing baseline starts with OCR: convert pages into text, index the text, and let retrieval or an LLM answer questions from it.
That baseline can work when flat text preserves the evidence the task needs. Once meaning depends on multi-column layouts, tables, or cross-document references, it needs additional checks.
Real operational files carry structure beyond their sentences. A single PDF packet may bundle a cover sheet, a multi-page seasonal rate schedule with room categories, cancellation clauses, an accessibility appendix, and a blackout-date schedule. Flattening all of that into one undifferentiated text stream strips away the structural cues that tell you what belongs together: which line is a governing heading, which note is a caption or footnote, which cell is a column header, and which nightly rate belongs to which room category and season.
One reason plain-text retrieval can fail on these files is lost structure: documents are structured evidence objects rather than text blobs. Recovering characters is only the first step; downstream systems also need enough layout, table, and hierarchy structure to assemble verifiable evidence.
This lesson starts with OCR as the baseline, then builds the stages production document-intelligence pipelines actually need: layout analysis, reading-order recovery, table-structure reconstruction, logical document splitting, document hierarchies and cross-document links, and evidence extraction with provenance. The running example is a travel-planning copilot for OptiVerse Travel that must reconstruct an inspectable evidence packet across hotel contracts, rate appendices, accessibility specifications, and booking confirmations.
Reconstruct the rate table
Correct words can still be paired with the wrong row.
April rooms
| Room | Rate |
|---|---|
| Standard | $180 |
| Accessible | $240 |
Room notes: reserve an accessible room explicitly.
Terms: rates apply in April only.
room: accessible
rate: 180 USD
period: AprilInspect reading order, cell boundaries, and source coordinates. Recognizing every word is insufficient.
Read this result
- Source address
- page 1 · row 3 · column 2
- Expected rate
- 240 USD
- Extracted rate
- 180 USD
The words can all be recognized while the row association is wrong. Select a room and compare its cell address with the extracted record; the prices alone cannot identify the correct row.
Synthetic document layout and extraction outputs; no OCR model is being run.
How this guide connects to the next
How This Lesson Connects
Bring forward: 03: the permitted-evidence rankings and RAG evaluation packet
Add here: the document reconstruction and provenance scorecard. Keep its assumptions, source IDs, and failure cases with the result.
Use next: 09 builds on this work. See the complete reading route for a sequence that includes prerequisites across topic series. Article IDs are stable labels, not numerical reading order.
Learning goals and practice plan
Tutorial Contract
Prerequisites
Complete baseline RAG. Hybrid Retrieval is useful optional context. All contracts, rates, pages, and inspections below are fictional teaching fixtures.
Learning Outcomes
Choose direct extraction, OCR, or OCR-free parsing by document type; preserve layout and reading order; reconstruct multi-page tables; split physical files into logical documents; link entities across files; evaluate each stage with task-specific metrics.
Build Artifact and Hands-On Exercise
Build a mixed-document evidence packet for KYO-H12: a four-page born-digital contract, a two-page scanned rate appendix, a photographed accessibility addendum, and a booking confirmation. Produce a structured JSON packet with page/cell provenance and a scorecard for text, layout, table, entity-link, and provenance quality.
Deliberately Broken Case
The second table page loses its repeated header, so JPY 25,000 is attached to the standard-room shoulder-season column instead of the accessible-room peak-season column.
Exit Criteria
For the bounded fixture: every critical identifier and amount exactly matches ground truth, all 12 reading-order constraints pass, the reconstructed table preserves every header and span, the three declared cross-document links are 3/3 correct, and provenance coverage is 100%.
Knowledge Check and Answer Key
Does perfect OCR prove that a table was understood? Answer: no. Character recovery, reading order, table structure, entity linking, and provenance are separate evaluation targets.
Why Plain-Text Retrieval Fails on Real Documents
Suppose a travel planner asks:
For contract KYO-H12, extract every room rate and accessibility specification from the contract terms, the rate appendix, the accessibility appendix, and the booking confirmation, then flag any disagreement.
A text-only retrieval pipeline looks tempting on paper:
Extract text from every file.
Chunk the text.
Index the chunks.
Retrieve relevant passages for
KYO-H12.Ask an LLM to summarize the answer.
That pipeline can fail when the query needs document structure that the plain-text representation has lost. This query exposes five such gaps.
First, reading order is often ambiguous. Flattening a two-column contract into a single stream can interleave left- and right-column sentences, splice footnotes into mid-paragraph prose, and sever captions from the tables they explain.
Second, document hierarchy disappears. A heading governs the material below it; once section boundaries are stripped away, retrieval can surface sentences that sound relevant but belong to a different pricing season, appendix, or disclaimer section.
Third, flattening can detach table cells from their row and column headers. Consider this extracted line:
KYO-H12 Standard 18500 22000 Accessible 21000 25000
Those bare values cannot establish a room- and season-specific rate on their own; you still need the table structure to know the season, currency, room type, and whether a figure comes from a base-rate or meal-inclusive column.
Fourth, one file may contain several logical documents. A single uploaded PDF can bundle the contract terms, a rate appendix, and an accessibility appendix; if the system indexes the file as one unit, retrieval and answer synthesis can blur claims that should remain distinct.
Fifth, cross-document matching is fragile. The contract terms may cite KYO-H12, the rate appendix may use a booking reference, the accessibility appendix may use a property code, and the booking confirmation may encode the same hotel in a reservation-system field. A plain-text pipeline rarely resolves those aliases reliably.
Retrieval quality is only part of the failure: the representation may already lack the relationships this task requires. A reliable pipeline recovers structure before it retrieves, normalizes, and compares evidence.
Choose Direct Text Extraction, OCR, or Image-Based Parsing
Many downstream document tasks need machine-readable content, but a standalone OCR pass is only one way to get it. Born-digital PDFs may provide reliable embedded text directly, whereas scanned contracts and photographed pages need OCR or vision-based parsing. OCR-free models such as Donut map page images directly to structured outputs, and they were proposed partly to avoid OCR error propagation (Kim et al., 2022).
Where OCR runs, it solves character recovery rather than full document understanding.
Teams often conflate the two: when extracted text looks legible in a debug view, it is tempting to assume the pipeline understood the page. Character transcription is only one layer of the representation, and correct words can still arrive with scrambled reading order, missing section boundaries, detached headers, or merged footnotes.
This misconception recurs in document systems:
If OCR got the words right, the document has been understood.
The statement is false for the same reason transcription is not analysis: every character on a rate sheet can be transcribed correctly while prices still map to the wrong room category.
OCR also introduces failure modes of its own. Recognition errors can cascade into later steps, especially when downstream logic depends on exact identifiers, units, or prices. If KYO-H12 is misread as KYO-H1Z, or a digit drops out of a nightly rate, retrieval and comparison can fail silently unless the pipeline preserves provenance and validation signals.
Document intelligence is therefore a pipeline or architecture layer, not a single OCR feature.
Layout and Reading Order Carry Meaning
Once characters are recoverable, the pipeline has to determine how blocks are arranged and ordered on the page.
Layout analysis identifies meaningful regions such as titles, body paragraphs, running headers, footnotes, lists, and tables; bounding boxes record where each block sits, and reading order determines the sequence in which those regions should be interpreted.
Page geometry can change legal and commercial meaning.
Picture a hotel contract PDF containing:
a title page and contract summary
a two-column rate schedule section
a table spanning both columns
a footnote clarifying that one rate applies only during cherry-blossom season with a minimum-stay requirement
an accessibility appendix appended to the same PDF
Misordered columns can splice unrelated clauses together; a detached cherry-blossom footnote can strip the season and minimum-stay conditions from a quoted rate; a missed appendix boundary can mix accessibility specifications into the main rate schedule without telling the user.
Layout-aware models reduce those errors by encoding page positions alongside text and visual features: LayoutLMv3 jointly pretrains text, layout, and image representations, while the LayoutReader paper's ReadingBank benchmark evaluates reading-order detection as a separate task (Huang et al., 2022; Wang et al., 2021).
As a rule of thumb, if an answer depends on what sits above, below, beside, or inside another block on the page, you do not have a text-only problem.
Tables Are Structural Objects, Not Paragraphs with Delimiters
Tables deserve separate treatment because they are one of the most common places where document pipelines fail.
A table is more than a compact way to display text. It is a structured object that relates header cells, row labels, column groups, units, merged cells, and often footnotes, and a downstream system needs those relationships intact to answer quantitative questions correctly.
Consider a seasonal rate table split across two pages of a scanned contract appendix: page one carries the header rows and the first room categories, while page two continues the rows, repeats only part of the header, and adds a note on blackout dates and seasonal surcharges. A flat text extractor may produce legible strings while breaking the grid:
repeated headers may be dropped
merged cells may be expanded incorrectly
currencies may be separated from the values they qualify
continuation rows may be treated as a new table
notes may be detached from the cells they modify
Retrieving flat text snippets from that broken output invites the generator to quote the wrong season column, the wrong currency, or the wrong room category.
Table extraction therefore typically involves several distinct tasks, a split that PubTables-1M makes explicit by separating detection, structure recognition, and functional analysis (Smock, Pesala, and Abraham, 2022):
detecting where the table is
reconstructing row and column structure
identifying which cells function as headers
linking notes, currencies, and continuation pages back into the table
Keeping those subtasks separate helps teams avoid mistaking table text extracted for table structure recovered.
For the travel-planning copilot, a request for every room rate and accessibility specification is effectively a request for structured table evidence. Without header ancestry and row-column relationships, the system cannot compare values with confidence.
One File May Contain Many Logical Documents
Another structural trap appears before page parsing even starts: equating one physical file with one logical document.
That assumption is often wrong.
A single PDF can concatenate a master hotel agreement with its seasonal rate schedule; an internal booking packet can bundle cover sheets, flight itineraries, insurance clauses, and accessibility addenda; a vendor scan can merge a cover letter, quote table, and invoice. File containers and logical document boundaries need not match one-to-one.
Document splitting identifies those logical boundaries. In practice it can use page classification, section-transition detection, metadata cues, or workflow rules that inspect titles, recurring headers, or attachment markers.
In our running scenario, one uploaded PDF might pack four distinct artifacts together:
the contract terms for hotel KYO-H12
a rate appendix with the seasonal pricing table
an accessibility appendix with room and facility specifications
a scanned addendum page with handwritten annotations
If those four artifacts stay merged, several downstream behaviors degrade:
retrieval scopes become noisy
provenance becomes harder to explain
answer synthesis can combine incomparable evidence
comparison logic may miss that the same property is being described in different document roles
Plain-text RAG tends to underperform here because text retrieval assumes the chunks already represent sensible semantic units; document intelligence often has to create those units first.
Within-Document Structure and Across-Document Relationships
Recovering pages, blocks, and tables still leaves two distinct relationship structures to resolve.
Within a single logical document, hierarchical links such as these define interpretive scope:
title to section
section to subsection
paragraph to list
section to table
figure to caption
footnote to referenced text
Local scope prevents false equivalences: a nightly rate under Peak Season Rates is not equivalent to a superficially similar value in an appendix describing promotional offers or group discounts.
Across separate logical documents, relationships such as these form a graph rather than a strict tree:
contract terms to rate appendix
contract to accessibility appendix
booking confirmation to itinerary export
main clause to supporting schedule
Because evidence often arrives as a multi-file packet rather than a single canonical record, the pipeline must resolve which files belong to the same agreement, which references point across documents, and which identifier aliases map to the same property or booking.
This is where an evidence packet becomes a useful design pattern. Instead of handing a pile of raw text to a generator, the pipeline first assembles structured claims, rates, specifications, and references with explicit source links, and the answer model works over that packet. Extraction quality, entity matching, and answer quality can then be evaluated independently rather than collapsing into one opaque generation step.
Measure Each Reconstruction Stage
Parse-success rate tracks operational health without proving extracted facts are right. Build labeled ground truth for high-consequence document classes and score each transformation on its own metric family.
Stage | What to compare | Example metrics |
|---|---|---|
Text recognition | recognized text versus reference transcription | CER, WER, exact match for critical IDs/numbers |
Layout | predicted regions/classes versus labeled boxes | precision, recall, F1, mAP/IoU by region class |
Reading order | predicted block sequence versus reference order | pairwise-order accuracy, sequence/BLEU-style score, violated constraints |
Table structure | predicted grid, spans, headers, and cell content versus ground truth | GriTS, TEDS, cell exact match |
Logical splitting | predicted document boundaries versus labeled boundaries | boundary precision/recall/F1 |
Entity linking | predicted cross-document links versus gold links | precision/recall/F1 |
Provenance | delivered fields with inspectable page/region/cell sources | provenance coverage |
For text recognition, normalized edit distance yields reproducible Character Error Rate (CER) and Word Error Rate (WER):
CER or WER = (substitutions + deletions + insertions) / reference units
NIST's TRAIT evaluation defines normalized character- and word-level edit distance for text recognition (NIST IR 8199). A low corpus-wide CER can still hide a corrupted contract ID or a price error, so report exact-match accuracy on critical identifiers and amounts separately from aggregate CER/WER.
Declare text normalization and segmentation rules before scoring: for example, Unicode code points for CER and a specified word segmenter for WER, since English whitespace splitting does not define words in Traditional Chinese. Insertions can push CER or WER above 1.0, and an empty reference string needs an explicit convention. Critical IDs and amounts still need exact comparison against the source, and a blank or low-confidence extraction must never be coerced to numeric zero.
For tables, GriTS scores the predicted table directly as a grid, with separate topology, content, and location variants (Smock, Pesala, and Abraham, 2023), while TEDS derives a similarity score from the normalized tree-edit distance between predicted and reference HTML table trees (Zhong, ShafieiBavani, and Jimeno Yepes, 2020). Match the metric to your parser's output representation, and name the metric and its variant whenever you report a table score.
Worked Example: Reconstructing an Evidence Packet
Suppose an OptiVerse Travel planner requests all reported room rates and accessibility specifications for contract KYO-H12 across the master contract, rate appendix, accessibility addendum, and booking confirmation for itinerary JPN-2026-0417, flagging any discrepancy.
One document-intelligence pipeline for that request has eight inspectable stages:
Inspect each uploaded file.
Determine whether each file is a born-digital PDF, a scan, an image-heavy page set, or a merged packet containing multiple logical documents.
Parse text with the right baseline.
Use OCR where the page is image-based; use direct text extraction where the PDF contains reliable embedded text; optionally use an OCR-free parser where that architecture fits the operating constraints.
Recover page structure.
Identify page regions, reading order, headings, lists, footnotes, tables, and likely document boundaries.
Reconstruct tables.
Detect table regions, preserve header structure, and merge continuation pages into one logical table when appropriate.
Split logical documents.
Separate the contract terms from rate appendices and accessibility appendices when they are bundled into one file.
Match entities across documents.
Link KYO-H12, booking references, property codes, and reservation-system IDs where the evidence indicates they refer to the same hotel or itinerary.
Build a structured evidence packet.
Store extracted values, currencies, qualifiers, confidence signals, and page-level or cell-level provenance.
Compare and synthesize.
Only after the evidence packet exists should a downstream model summarize disagreements or produce a user-facing explanation.
That pipeline has more moving parts than OCR-plus-RAG because it solves a different problem: instead of generating a plausible answer from nearby chunks, it reconstructs evidence faithfully enough that cross-document discrepancies can be surfaced and inspected.
Rather than a bare summary paragraph, the output is an inspectable packet along these lines:
JPY 22,000standard-room peak-season rate from the contract terms, page 4, Clause 4.3, rate schedule rowStandard, columnPeakJPY 25,000accessible-room peak-season rate fromrate-appendix-scan.pdf, local page 2 (packet page 6), rowAccessible, columnPeak, with header ancestry verifiedwheelchair-accessible roll-in showerspecification reported byaccessibility-addendum.jpg, image page 1 (packet page 7), sectionAccessible Room Features; it is a document claim, not an independent inspection by the parsera booking confirmation record linked by itinerary ID
JPN-2026-0417with a timestamp and reserved-room fieldin the broken variant, a warning that the raw amount is readable but its room/season assignment is unverified; retain the raw observation and withhold the interpreted rate until the appendix continuation header is resolved
Because extracted evidence stays separate from the generated explanation, planners and reviewers can check every rate against its exact page, row, and column.
Hands-On Exercise: Reconstruct the Mixed Packet
Prepare four fixture inputs:
contract-kyo-h12.pdf: born-digital, two columns, four pages;rate-appendix-scan.pdf: two local pages, corresponding to packet pages 5–6, containing a continued table;accessibility-addendum.jpg: photographed page with property codeKYO-012;booking-jpn-2026-0417.pdf: born-digital confirmation using itinerary ID rather than contract ID.
Define the intact reference table explicitly. On each appendix page, Room spans two header rows; Nightly rate (JPY) spans the two season columns above Shoulder and Peak. The first page contains Standard; the second repeats the headers and contains Accessible:
Room | Shoulder | Peak | Source local page |
|---|---|---|---|
Standard | 18500 | 22000 | 1 |
Accessible | 21000 | 25000 | 2 |
The footnote states that the rates are per room, per night under the fixture contract; its blackout exclusions and occupancy limits must stay attached to the table. Prices for different room categories are not a conflict: a genuine rate discrepancy requires matching room category, season, currency, inclusions, and contract version.
The master contract explicitly maps KYO-H12 to KYO-012; the rate appendix names KYO-H12; the photographed addendum names KYO-012; and the booking confirmation names both KYO-012 and JPN-2026-0417. Recover the three gold links (contract→rate appendix, contract→accessibility addendum, and contract→booking confirmation) from those explicit identifier joins rather than fuzzy hotel-name similarity.
For reading order, label the target contract page C0–C5 (title, left body, left footnote, right heading, right body, footer), the appendix R0–R5 (first-page group header, season labels, Standard row, second-page repeated header, Accessible row, footnote), the addendum A0–A1 (property code, shower statement), and the confirmation B0–B1 (itinerary ID, property reference). Those four sequences yield 12 adjacent-pair constraints. Placing each column footnote immediately after its column body is a convention of this fixture, not a universal page-reading rule.
Use one-based file-local page numbers alongside packet_page (1–4 for the contract, 5–6 for the appendix, 7 for the addendum, and 8 for the confirmation). Below is one validated field from the intact fixture rather than the complete output:
{
"packet_id": "EP-KYO-H12-v1",
"logical_documents": ["contract", "rate_appendix", "accessibility_addendum", "booking_confirmation"],
"entities": {
"contract_id": "KYO-H12",
"property_code": "KYO-012",
"itinerary_id": "JPN-2026-0417"
},
"critical_fields": [
{
"name": "accessible_room_peak_rate",
"value": 25000,
"currency": "JPY",
"source": {"file": "rate-appendix-scan.pdf", "page": 2, "packet_page": 6, "table": "seasonal_rates", "row": "Accessible", "column": "Peak"}
}
],
"warnings": []
}Check the declared reading-order constraints independently of OCR text quality:
const sequences = [
["C0", "C1", "C2", "C3", "C4", "C5"],
["R0", "R1", "R2", "R3", "R4", "R5"],
["A0", "A1"], ["B0", "B1"],
];
const constraints = sequences.flatMap(s => s.slice(1).map((id, i) => [s[i], id]));
function orderScore(predicted) {
if (new Set(predicted).size !== predicted.length) throw new Error("duplicate block ID");
const positions = new Map(predicted.map((id, i) => [id, i]));
const passed = constraints.filter(([a, b]) => positions.has(a) && positions.has(b)
&& positions.get(a) < positions.get(b)).length;
return { passed, total: constraints.length };
}
const intact = orderScore(sequences.flat());
if (intact.passed !== 12 || intact.total !== 12) throw new Error("intact order");
const broken = sequences.flat();
[broken[7], broken[8]] = [broken[8], broken[7]];
if (orderScore(broken).passed !== 11) throw new Error("shifted reading order");
console.log({ intact, broken: orderScore(broken) });Run the pipeline in visible stages: inspect file type, recover text, detect blocks, derive reading order, reconstruct the two-page table, split logical documents, link identifiers, emit evidence with provenance, and compute the scorecard. Save every intermediate representation so failures can be localized.
Deliberately Broken Case
Remove the repeated Peak header from appendix local page 2 (packet page 6) and shift the second-page columns one position left. A flat parser still extracts every number, but maps JPY 25,000 to the wrong room and season.
The pipeline must catch the error through at least one explicit invariant:
continuation columns must map to appendix local page 1 (packet page 5) headers using labels, positions, and qualifiers; a missing repeated header alone does not imply an error when continuity is otherwise unambiguous;
every rate cell must have one room row, season column, and currency ancestor;
cell-content or field-assignment scores must detect a wrong column association, since a topology-only score can stay unchanged when values move within the same grid;
a critical value with ambiguous column ancestry must be withheld and sent to review.
Also corrupt the OCR output from KYO-H12 to KYO-H1Z. Entity linking must not silently fuzzy-match a critical identifier; it should surface the conflict across contract, property, and itinerary records.
Exit Criteria
The fixture passes when:
critical IDs
KYO-H12,KYO-012, andJPN-2026-0417are exact matches;all four logical documents and three cross-document links are recovered;
all 12 labeled reading-order constraints pass;
the table retains the expected rows, columns, merged headers, continuation link, currency, and cell provenance;
accessible_room_peak_rate = JPY 25,000points to local page 2, packet page 6, rowAccessible, columnPeak;provenance coverage is
delivered critical fields with valid sources / delivered critical fields = 1.0, and required-field completeness is separatelyrecovered required fields / all required fields = 1.0for the intact fixture so withholding everything cannot pass;both the shifted-table and corrupted-ID cases are withheld rather than guessed;
parser version, OCR/model version, source checksums, per-stage latency, and scorecard are recorded.
These gates are specific to the fixture. Set production thresholds by document class and consequence: a rate table and a marketing brochure do not share the same failure tolerance.
What This Post Is Not
Document intelligence is already multimodal whenever it combines text, page images, and layout. This post draws a practical boundary around document reconstruction: layout recovery, table structure, document splitting, and evidence assembly. Post 9 treats photos, figures, floor plans, and cross-modal retrieval as first-class evidence, while retrieval strategy stays in Post 7.
Misconceptions That Cause Bad System Design
Five misconceptions recur in document-heavy AI projects.
First, teams assume OCR text alone is sufficient regardless of what the task requires. A faithful textual or structured representation can preserve the table, layout, and cross-document relationships a task needs, but once those relationships are lost, retrieval over the remaining text cannot recover them on its own.
Second, teams assume a table's cell values preserve its meaning by themselves. Merged headers, continuation rows, currencies, and notes outside the grid may determine how values relate. Flattening a grid without preserving those relationships can destroy its meaning; an explicit serialization that retains them need not.
Third, teams treat one uploaded PDF as one logical document. That assumption fails on contract bundles, appendices, and supplements, all of which are common in travel operations and enterprise workflows.
Fourth, teams assume OCR-free document models remove the need for structure and validation. Even without a separate OCR stage, a system still has to recover document relationships, preserve provenance, and evaluate extraction quality.
Fifth, teams treat document intelligence as a warm-up exercise for multimodality. Structure, hierarchy, and extraction quality make document intelligence its own systems layer, and they matter even before figures and photographs become central.
Common Failure Modes in Production
The most consequential production failures are usually structural.
Broken reading order can interleave multi-column text or detach sidebars from surrounding clauses, and section headings can drift away from the paragraphs they govern. Tables split across page breaks can be reconstructed incorrectly, and dropped header rows can lead parsers to assign rates to the wrong columns. Bundled appendices can stay merged with master agreements, and entity linkers can over-match similar identifiers or miss explicit cross-references. Extracted numbers can also travel downstream without page or cell provenance, letting OCR errors propagate silently into retrieval, normalization, and answer generation.
These failures are dangerous because the resulting answers often look plausible. A planner may receive a clean summary with no sign that the underlying rate table was misread or that a rate came from an appendix rather than the master contract. In high-stakes workflows, the main risk is silent misattribution rather than an obvious parse crash.
Practical Design Guidance
Treat document intelligence as a dedicated architectural layer rather than a preprocessing script bolted onto retrieval.
Start with file inspection. Before indexing or retrieving anything, classify incoming files, distinguish born-digital text from scans, and check whether a single upload bundles multiple logical documents.
Preserve structure before chunking. Recover page regions, reading order, and table boundaries first; flattening pages up front can destroy the spatial cues needed for reliable extraction.
Represent tables explicitly with row and column hierarchy, headers, currencies, and page coordinates. Only reduce a table to plain text if task-level evaluation shows the structural loss is acceptable.
Separate extraction from synthesis. Assemble a validated evidence packet first, then hand that structured packet to a downstream model for comparison or explanation so every stage remains debuggable and auditable.
Make provenance first-class by recording page coordinates and, for tables, cell coordinates wherever feasible. Reviewers cannot adjudicate conflicting numbers if they cannot jump straight to the source location.
Keep entity linking conservative. Match contract IDs, booking references, and property codes through explicit rules or calibrated scoring with validation rather than casual string similarity.
Evaluate each document task independently. Measure table reconstruction, document splitting, identifier linking, and provenance coverage alongside final answer quality so fluent prose cannot mask a broken parser.
Treat OCR as a baseline rather than dogma. Some workflows will stay OCR-first for practical reasons; others may adopt OCR-free parsers for selected document classes. What matters is whether the pipeline preserves the structure the task requires.
Why This Leads Naturally to Multimodal Evidence Systems
Once a team accepts that documents are not text blobs, the next architectural step becomes obvious.
Many important claims in travel contracts do not live only in paragraphs or tables. They also live in floor plans, property photos, accessibility diagrams, captions, legends, and the relationships among them. A system that can reconstruct page layout, table structure, and document hierarchy is already moving beyond plain-text retrieval toward a fuller evidence model.
That is the bridge to multimodal systems.
Post 9 picks up from here. Once figures, charts, and images become first-class evidence alongside text and tables, document intelligence expands into multimodal grounding: linking visual content to captions, surrounding prose, and cross-document claims without losing provenance.
Source Notes
This post draws on the following primary and practitioner sources:
Smith, R. "An Overview of the Tesseract OCR Engine." *ICDAR 2007.* Reference for OCR as character recognition, not full document understanding. research.google/pubs/an-overview-of-the-tesseract-ocr-engine
Xu, Y., Li, M., Cui, L., et al. "LayoutLM: Pre-training of Text and Layout for Document Image Understanding." *KDD 2020.* Reference for the claim that layout and style information matter to document understanding. arxiv.org/abs/1912.13318
Huang, Y., Lv, T., Cui, L., et al. "LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking." *ACM Multimedia 2022.* Joint text/image/layout document representation and downstream document-task evidence. arXiv:2204.08387
Wang, Z., Xu, Y., Cui, L., Shang, J., and Wei, F. "LayoutReader: Pre-training of Text and Layout for Reading Order Detection." *EMNLP 2021.* ReadingBank and reading order as a separately evaluable task. ReadingBank
Li, M., Xu, Y., Cui, L., et al. "DocBank: A Benchmark Dataset for Document Layout Analysis." *COLING 2020.* Reference for fine-grained layout analysis as a distinct task. arxiv.org/abs/2006.01038
Smock, B., Pesala, R., and Abraham, R. "PubTables-1M: Towards Comprehensive Table Extraction From Unstructured Documents." *CVPR 2022.* Reference for table detection, structure recognition, and functional analysis as separable tasks. arxiv.org/abs/2110.00061
Zhong, X., ShafieiBavani, E., and Jimeno Yepes, A. "Image-based Table Recognition: Data, Model, and Evaluation." *ECCV 2020.* Primary source for TEDS as a structure-and-content table metric. arXiv:1911.10683
Kim, G., Hong, T., Yim, M., et al. "OCR-free Document Understanding Transformer." *ECCV 2022.* Reference for OCR-free document-understanding approaches and OCR error-propagation cautions. arxiv.org/abs/2111.15664
National Institute of Standards and Technology. "Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile." NIST AI 600-1, July 2024. Reference for provenance, traceability, and review language in high-stakes extraction workflows. nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence
National Institute of Standards and Technology. "The Text Recognition Algorithm Independent Evaluation (TRAIT)." NISTIR 8199, December 2017. Normalized character- and word-level edit-distance measures for text recognition. NIST IR 8199
Smock, B., Pesala, R., and Abraham, R. "GriTS: Grid Table Similarity Metric for Table Structure Recognition." *ICDAR 2023.* Grid-based table evaluation and its variants. arXiv:2203.12555
Reading order
Complete reading route
Chapter 14 / 25
Huang Tzu Lin
Founder of OptiVerse Technology. Writing about AI system design, evidence, and engineering trade-offs.
Subscribe via RSS
Follow the latest AI systems engineering tutorials in your preferred RSS reader.
Open RSS FeedWorks with any RSS reader — new posts arrive automatically.
