Skip to content
Document & Multimodal Intelligence

Reading Documents Beyond OCR

Reconstruct reading order, tables, and source relationships so a document becomes traceable evidence.

Author
Huang Tzu LinFounder
Published
Reading time
25 min
Updated
Contents
  1. 01How This Lesson Connects
  2. 02Learning plan
  3. Prerequisites
  4. Learning Outcomes
  5. Practice exercise
  6. Failure exercise
  7. Completion check
  8. Check your understanding
  9. 03Why Plain-Text Retrieval Fails on Real Documents
  10. 04Choose Direct Text Extraction, OCR, or Image-Based Parsing
  11. 05Layout and Reading Order Carry Meaning
  12. 06Tables Are Structural Objects, Not Paragraphs with Delimiters
  13. 07One File May Contain Many Logical Documents
  14. 08Within-Document Structure and Across-Document Relationships
  15. 09Measure Each Reconstruction Stage
  16. 10Worked Example: Reconstructing an Evidence Packet
  17. 11Hands-On Exercise: Reconstruct the Mixed Packet
  18. 12Failure exercise
  19. 13Completion check
  20. 14What This Post Is Not
  21. 15Misconceptions That Cause Bad System Design
  22. 16Common Failure Modes in Production
  23. 17Practical Design Guidance
  24. 18Why This Leads Naturally to Multimodal Evidence Systems
  25. 19Source Notes

A common document-processing baseline starts with OCR: convert pages into text, index the text, and let retrieval or an LLM answer questions from it.

That baseline can work when flat text preserves the evidence the task needs. Once meaning depends on multi-column layouts, tables, or cross-document references, it needs additional checks.

Real operational files carry structure beyond their sentences. A single PDF packet may bundle a cover sheet, a multi-page seasonal rate schedule with room categories, cancellation clauses, an accessibility appendix, and a blackout-date schedule. Flattening all of that into one undifferentiated text stream strips away the structural cues that tell you what belongs together: which line is a governing heading, which note is a caption or footnote, which cell is a column header, and which nightly rate belongs to which room category and season.

One reason plain-text retrieval can fail on these files is lost structure: documents are structured evidence objects rather than text blobs. Recovering characters is only the first step; downstream systems also need enough layout, table, and hierarchy structure to assemble verifiable evidence.

This lesson starts with OCR as the baseline, then builds the stages production document-intelligence pipelines actually need: layout analysis, reading-order recovery, table-structure reconstruction, logical document splitting, document hierarchies and cross-document links, and evidence extraction with provenance. The running example is a travel-planning copilot for OptiVerse Travel that must reconstruct an inspectable evidence packet across hotel contracts, rate appendices, accessibility specifications, and booking confirmations.

INTERACTIVE LAB

Reconstruct the rate table

Correct words can still be paired with the wrong row.

KYO-H12 / RATE SHEET

April rooms

RoomRate
Standard$180
Accessible$240

Room notes: reserve an accessible room explicitly.

Terms: rates apply in April only.

Extracted recordroom: accessible rate: 180 USD period: April
Wrong row association

Inspect reading order, cell boundaries, and source coordinates. Recognizing every word is insufficient.

Read this result

Source address
page 1 · row 3 · column 2
Expected rate
240 USD
Extracted rate
180 USD

The words can all be recognized while the row association is wrong. Select a room and compare its cell address with the extracted record; the prices alone cannot identify the correct row.

Synthetic document layout and extraction outputs; no OCR model is being run.

How this guide connects to the next

How This Lesson Connects

Bring forward: 03: the permitted-evidence rankings and RAG evaluation packet

Add here: the document reconstruction and provenance scorecard. Keep its assumptions, source IDs, and failure cases with the result.

Use next: 09 builds on this work. See the complete reading route for a sequence that includes prerequisites across topic series. Article IDs are stable labels, not numerical reading order.

Learning goals and practice plan

Tutorial Contract

Prerequisites

Complete baseline RAG. Hybrid Retrieval is useful optional context. All contracts, rates, pages, and inspections below are fictional teaching fixtures.

Learning Outcomes

Choose direct extraction, OCR, or OCR-free parsing by document type; preserve layout and reading order; reconstruct multi-page tables; split physical files into logical documents; link entities across files; evaluate each stage with task-specific metrics.

Build Artifact and Hands-On Exercise

Build a mixed-document evidence packet for KYO-H12: a four-page born-digital contract, a two-page scanned rate appendix, a photographed accessibility addendum, and a booking confirmation. Produce a structured JSON packet with page/cell provenance and a scorecard for text, layout, table, entity-link, and provenance quality.

Deliberately Broken Case

The second table page loses its repeated header, so JPY 25,000 is attached to the standard-room shoulder-season column instead of the accessible-room peak-season column.

Exit Criteria

For the bounded fixture: every critical identifier and amount exactly matches ground truth, all 12 reading-order constraints pass, the reconstructed table preserves every header and span, the three declared cross-document links are 3/3 correct, and provenance coverage is 100%.

Knowledge Check and Answer Key

Does perfect OCR prove that a table was understood? Answer: no. Character recovery, reading order, table structure, entity linking, and provenance are separate evaluation targets.

Why Plain-Text Retrieval Fails on Real Documents

Suppose a travel planner asks:

For contract KYO-H12, extract every room rate and accessibility specification from the contract terms, the rate appendix, the accessibility appendix, and the booking confirmation, then flag any disagreement.

A text-only retrieval pipeline looks tempting on paper:

  1. Extract text from every file.

  2. Chunk the text.

  3. Index the chunks.

  4. Retrieve relevant passages for KYO-H12.

  5. Ask an LLM to summarize the answer.

That pipeline can fail when the query needs document structure that the plain-text representation has lost. This query exposes five such gaps.

First, reading order is often ambiguous. Flattening a two-column contract into a single stream can interleave left- and right-column sentences, splice footnotes into mid-paragraph prose, and sever captions from the tables they explain.

Second, document hierarchy disappears. A heading governs the material below it; once section boundaries are stripped away, retrieval can surface sentences that sound relevant but belong to a different pricing season, appendix, or disclaimer section.

Third, flattening can detach table cells from their row and column headers. Consider this extracted line:

KYO-H12 Standard 18500 22000 Accessible 21000 25000

Those bare values cannot establish a room- and season-specific rate on their own; you still need the table structure to know the season, currency, room type, and whether a figure comes from a base-rate or meal-inclusive column.

Fourth, one file may contain several logical documents. A single uploaded PDF can bundle the contract terms, a rate appendix, and an accessibility appendix; if the system indexes the file as one unit, retrieval and answer synthesis can blur claims that should remain distinct.

Fifth, cross-document matching is fragile. The contract terms may cite KYO-H12, the rate appendix may use a booking reference, the accessibility appendix may use a property code, and the booking confirmation may encode the same hotel in a reservation-system field. A plain-text pipeline rarely resolves those aliases reliably.

Retrieval quality is only part of the failure: the representation may already lack the relationships this task requires. A reliable pipeline recovers structure before it retrieves, normalizes, and compares evidence.

Choose Direct Text Extraction, OCR, or Image-Based Parsing

Many downstream document tasks need machine-readable content, but a standalone OCR pass is only one way to get it. Born-digital PDFs may provide reliable embedded text directly, whereas scanned contracts and photographed pages need OCR or vision-based parsing. OCR-free models such as Donut map page images directly to structured outputs, and they were proposed partly to avoid OCR error propagation (Kim et al., 2022).

Where OCR runs, it solves character recovery rather than full document understanding.

Teams often conflate the two: when extracted text looks legible in a debug view, it is tempting to assume the pipeline understood the page. Character transcription is only one layer of the representation, and correct words can still arrive with scrambled reading order, missing section boundaries, detached headers, or merged footnotes.

This misconception recurs in document systems:

If OCR got the words right, the document has been understood.

The statement is false for the same reason transcription is not analysis: every character on a rate sheet can be transcribed correctly while prices still map to the wrong room category.

OCR also introduces failure modes of its own. Recognition errors can cascade into later steps, especially when downstream logic depends on exact identifiers, units, or prices. If KYO-H12 is misread as KYO-H1Z, or a digit drops out of a nightly rate, retrieval and comparison can fail silently unless the pipeline preserves provenance and validation signals.

Document intelligence is therefore a pipeline or architecture layer, not a single OCR feature.

Layout and Reading Order Carry Meaning

Once characters are recoverable, the pipeline has to determine how blocks are arranged and ordered on the page.

Layout analysis identifies meaningful regions such as titles, body paragraphs, running headers, footnotes, lists, and tables; bounding boxes record where each block sits, and reading order determines the sequence in which those regions should be interpreted.

Page geometry can change legal and commercial meaning.

Picture a hotel contract PDF containing:

  • a title page and contract summary

  • a two-column rate schedule section

  • a table spanning both columns

  • a footnote clarifying that one rate applies only during cherry-blossom season with a minimum-stay requirement

  • an accessibility appendix appended to the same PDF

Misordered columns can splice unrelated clauses together; a detached cherry-blossom footnote can strip the season and minimum-stay conditions from a quoted rate; a missed appendix boundary can mix accessibility specifications into the main rate schedule without telling the user.

Layout-aware models reduce those errors by encoding page positions alongside text and visual features: LayoutLMv3 jointly pretrains text, layout, and image representations, while the LayoutReader paper's ReadingBank benchmark evaluates reading-order detection as a separate task (Huang et al., 2022; Wang et al., 2021).

As a rule of thumb, if an answer depends on what sits above, below, beside, or inside another block on the page, you do not have a text-only problem.

Tables Are Structural Objects, Not Paragraphs with Delimiters

Tables deserve separate treatment because they are one of the most common places where document pipelines fail.

A table is more than a compact way to display text. It is a structured object that relates header cells, row labels, column groups, units, merged cells, and often footnotes, and a downstream system needs those relationships intact to answer quantitative questions correctly.

Consider a seasonal rate table split across two pages of a scanned contract appendix: page one carries the header rows and the first room categories, while page two continues the rows, repeats only part of the header, and adds a note on blackout dates and seasonal surcharges. A flat text extractor may produce legible strings while breaking the grid:

  • repeated headers may be dropped

  • merged cells may be expanded incorrectly

  • currencies may be separated from the values they qualify

  • continuation rows may be treated as a new table

  • notes may be detached from the cells they modify

Retrieving flat text snippets from that broken output invites the generator to quote the wrong season column, the wrong currency, or the wrong room category.

Table extraction therefore typically involves several distinct tasks, a split that PubTables-1M makes explicit by separating detection, structure recognition, and functional analysis (Smock, Pesala, and Abraham, 2022):

  • detecting where the table is

  • reconstructing row and column structure

  • identifying which cells function as headers

  • linking notes, currencies, and continuation pages back into the table

Keeping those subtasks separate helps teams avoid mistaking table text extracted for table structure recovered.

For the travel-planning copilot, a request for every room rate and accessibility specification is effectively a request for structured table evidence. Without header ancestry and row-column relationships, the system cannot compare values with confidence.

One File May Contain Many Logical Documents

Another structural trap appears before page parsing even starts: equating one physical file with one logical document.

That assumption is often wrong.

A single PDF can concatenate a master hotel agreement with its seasonal rate schedule; an internal booking packet can bundle cover sheets, flight itineraries, insurance clauses, and accessibility addenda; a vendor scan can merge a cover letter, quote table, and invoice. File containers and logical document boundaries need not match one-to-one.

Document splitting identifies those logical boundaries. In practice it can use page classification, section-transition detection, metadata cues, or workflow rules that inspect titles, recurring headers, or attachment markers.

In our running scenario, one uploaded PDF might pack four distinct artifacts together:

  • the contract terms for hotel KYO-H12

  • a rate appendix with the seasonal pricing table

  • an accessibility appendix with room and facility specifications

  • a scanned addendum page with handwritten annotations

If those four artifacts stay merged, several downstream behaviors degrade:

  • retrieval scopes become noisy

  • provenance becomes harder to explain

  • answer synthesis can combine incomparable evidence

  • comparison logic may miss that the same property is being described in different document roles

Plain-text RAG tends to underperform here because text retrieval assumes the chunks already represent sensible semantic units; document intelligence often has to create those units first.

Within-Document Structure and Across-Document Relationships

Recovering pages, blocks, and tables still leaves two distinct relationship structures to resolve.

Within a single logical document, hierarchical links such as these define interpretive scope:

  • title to section

  • section to subsection

  • paragraph to list

  • section to table

  • figure to caption

  • footnote to referenced text

Local scope prevents false equivalences: a nightly rate under Peak Season Rates is not equivalent to a superficially similar value in an appendix describing promotional offers or group discounts.

Across separate logical documents, relationships such as these form a graph rather than a strict tree:

  • contract terms to rate appendix

  • contract to accessibility appendix

  • booking confirmation to itinerary export

  • main clause to supporting schedule

Because evidence often arrives as a multi-file packet rather than a single canonical record, the pipeline must resolve which files belong to the same agreement, which references point across documents, and which identifier aliases map to the same property or booking.

This is where an evidence packet becomes a useful design pattern. Instead of handing a pile of raw text to a generator, the pipeline first assembles structured claims, rates, specifications, and references with explicit source links, and the answer model works over that packet. Extraction quality, entity matching, and answer quality can then be evaluated independently rather than collapsing into one opaque generation step.

Measure Each Reconstruction Stage

Parse-success rate tracks operational health without proving extracted facts are right. Build labeled ground truth for high-consequence document classes and score each transformation on its own metric family.

Stage

What to compare

Example metrics

Text recognition

recognized text versus reference transcription

CER, WER, exact match for critical IDs/numbers

Layout

predicted regions/classes versus labeled boxes

precision, recall, F1, mAP/IoU by region class

Reading order

predicted block sequence versus reference order

pairwise-order accuracy, sequence/BLEU-style score, violated constraints

Table structure

predicted grid, spans, headers, and cell content versus ground truth

GriTS, TEDS, cell exact match

Logical splitting

predicted document boundaries versus labeled boundaries

boundary precision/recall/F1

Entity linking

predicted cross-document links versus gold links

precision/recall/F1

Provenance

delivered fields with inspectable page/region/cell sources

provenance coverage

For text recognition, normalized edit distance yields reproducible Character Error Rate (CER) and Word Error Rate (WER):

CER or WER = (substitutions + deletions + insertions) / reference units

NIST's TRAIT evaluation defines normalized character- and word-level edit distance for text recognition (NIST IR 8199). A low corpus-wide CER can still hide a corrupted contract ID or a price error, so report exact-match accuracy on critical identifiers and amounts separately from aggregate CER/WER.

Declare text normalization and segmentation rules before scoring: for example, Unicode code points for CER and a specified word segmenter for WER, since English whitespace splitting does not define words in Traditional Chinese. Insertions can push CER or WER above 1.0, and an empty reference string needs an explicit convention. Critical IDs and amounts still need exact comparison against the source, and a blank or low-confidence extraction must never be coerced to numeric zero.

For tables, GriTS scores the predicted table directly as a grid, with separate topology, content, and location variants (Smock, Pesala, and Abraham, 2023), while TEDS derives a similarity score from the normalized tree-edit distance between predicted and reference HTML table trees (Zhong, ShafieiBavani, and Jimeno Yepes, 2020). Match the metric to your parser's output representation, and name the metric and its variant whenever you report a table score.

Worked Example: Reconstructing an Evidence Packet

Suppose an OptiVerse Travel planner requests all reported room rates and accessibility specifications for contract KYO-H12 across the master contract, rate appendix, accessibility addendum, and booking confirmation for itinerary JPN-2026-0417, flagging any discrepancy.

One document-intelligence pipeline for that request has eight inspectable stages:

  1. Inspect each uploaded file.

Determine whether each file is a born-digital PDF, a scan, an image-heavy page set, or a merged packet containing multiple logical documents.

  1. Parse text with the right baseline.

Use OCR where the page is image-based; use direct text extraction where the PDF contains reliable embedded text; optionally use an OCR-free parser where that architecture fits the operating constraints.

  1. Recover page structure.

Identify page regions, reading order, headings, lists, footnotes, tables, and likely document boundaries.

  1. Reconstruct tables.

Detect table regions, preserve header structure, and merge continuation pages into one logical table when appropriate.

  1. Split logical documents.

Separate the contract terms from rate appendices and accessibility appendices when they are bundled into one file.

  1. Match entities across documents.

Link KYO-H12, booking references, property codes, and reservation-system IDs where the evidence indicates they refer to the same hotel or itinerary.

  1. Build a structured evidence packet.

Store extracted values, currencies, qualifiers, confidence signals, and page-level or cell-level provenance.

  1. Compare and synthesize.

Only after the evidence packet exists should a downstream model summarize disagreements or produce a user-facing explanation.

That pipeline has more moving parts than OCR-plus-RAG because it solves a different problem: instead of generating a plausible answer from nearby chunks, it reconstructs evidence faithfully enough that cross-document discrepancies can be surfaced and inspected.

Rather than a bare summary paragraph, the output is an inspectable packet along these lines:

  • JPY 22,000 standard-room peak-season rate from the contract terms, page 4, Clause 4.3, rate schedule row Standard, column Peak

  • JPY 25,000 accessible-room peak-season rate from rate-appendix-scan.pdf, local page 2 (packet page 6), row Accessible, column Peak, with header ancestry verified

  • wheelchair-accessible roll-in shower specification reported by accessibility-addendum.jpg, image page 1 (packet page 7), section Accessible Room Features; it is a document claim, not an independent inspection by the parser

  • a booking confirmation record linked by itinerary ID JPN-2026-0417 with a timestamp and reserved-room field

  • in the broken variant, a warning that the raw amount is readable but its room/season assignment is unverified; retain the raw observation and withhold the interpreted rate until the appendix continuation header is resolved

Because extracted evidence stays separate from the generated explanation, planners and reviewers can check every rate against its exact page, row, and column.

Hands-On Exercise: Reconstruct the Mixed Packet

Prepare four fixture inputs:

  1. contract-kyo-h12.pdf: born-digital, two columns, four pages;

  2. rate-appendix-scan.pdf: two local pages, corresponding to packet pages 5–6, containing a continued table;

  3. accessibility-addendum.jpg: photographed page with property code KYO-012;

  4. booking-jpn-2026-0417.pdf: born-digital confirmation using itinerary ID rather than contract ID.

Define the intact reference table explicitly. On each appendix page, Room spans two header rows; Nightly rate (JPY) spans the two season columns above Shoulder and Peak. The first page contains Standard; the second repeats the headers and contains Accessible:

Room

Shoulder

Peak

Source local page

Standard

18500

22000

1

Accessible

21000

25000

2

The footnote states that the rates are per room, per night under the fixture contract; its blackout exclusions and occupancy limits must stay attached to the table. Prices for different room categories are not a conflict: a genuine rate discrepancy requires matching room category, season, currency, inclusions, and contract version.

The master contract explicitly maps KYO-H12 to KYO-012; the rate appendix names KYO-H12; the photographed addendum names KYO-012; and the booking confirmation names both KYO-012 and JPN-2026-0417. Recover the three gold links (contract→rate appendix, contract→accessibility addendum, and contract→booking confirmation) from those explicit identifier joins rather than fuzzy hotel-name similarity.

For reading order, label the target contract page C0–C5 (title, left body, left footnote, right heading, right body, footer), the appendix R0–R5 (first-page group header, season labels, Standard row, second-page repeated header, Accessible row, footnote), the addendum A0–A1 (property code, shower statement), and the confirmation B0–B1 (itinerary ID, property reference). Those four sequences yield 12 adjacent-pair constraints. Placing each column footnote immediately after its column body is a convention of this fixture, not a universal page-reading rule.

Use one-based file-local page numbers alongside packet_page (1–4 for the contract, 5–6 for the appendix, 7 for the addendum, and 8 for the confirmation). Below is one validated field from the intact fixture rather than the complete output:

json
{
  "packet_id": "EP-KYO-H12-v1",
  "logical_documents": ["contract", "rate_appendix", "accessibility_addendum", "booking_confirmation"],
  "entities": {
    "contract_id": "KYO-H12",
    "property_code": "KYO-012",
    "itinerary_id": "JPN-2026-0417"
  },
  "critical_fields": [
    {
      "name": "accessible_room_peak_rate",
      "value": 25000,
      "currency": "JPY",
      "source": {"file": "rate-appendix-scan.pdf", "page": 2, "packet_page": 6, "table": "seasonal_rates", "row": "Accessible", "column": "Peak"}
    }
  ],
  "warnings": []
}

Check the declared reading-order constraints independently of OCR text quality:

javascript
const sequences = [
  ["C0", "C1", "C2", "C3", "C4", "C5"],
  ["R0", "R1", "R2", "R3", "R4", "R5"],
  ["A0", "A1"], ["B0", "B1"],
];
const constraints = sequences.flatMap(s => s.slice(1).map((id, i) => [s[i], id]));
function orderScore(predicted) {
  if (new Set(predicted).size !== predicted.length) throw new Error("duplicate block ID");
  const positions = new Map(predicted.map((id, i) => [id, i]));
  const passed = constraints.filter(([a, b]) => positions.has(a) && positions.has(b)
    && positions.get(a) < positions.get(b)).length;
  return { passed, total: constraints.length };
}
const intact = orderScore(sequences.flat());
if (intact.passed !== 12 || intact.total !== 12) throw new Error("intact order");
const broken = sequences.flat();
[broken[7], broken[8]] = [broken[8], broken[7]];
if (orderScore(broken).passed !== 11) throw new Error("shifted reading order");
console.log({ intact, broken: orderScore(broken) });

Run the pipeline in visible stages: inspect file type, recover text, detect blocks, derive reading order, reconstruct the two-page table, split logical documents, link identifiers, emit evidence with provenance, and compute the scorecard. Save every intermediate representation so failures can be localized.

Deliberately Broken Case

Remove the repeated Peak header from appendix local page 2 (packet page 6) and shift the second-page columns one position left. A flat parser still extracts every number, but maps JPY 25,000 to the wrong room and season.

The pipeline must catch the error through at least one explicit invariant:

  • continuation columns must map to appendix local page 1 (packet page 5) headers using labels, positions, and qualifiers; a missing repeated header alone does not imply an error when continuity is otherwise unambiguous;

  • every rate cell must have one room row, season column, and currency ancestor;

  • cell-content or field-assignment scores must detect a wrong column association, since a topology-only score can stay unchanged when values move within the same grid;

  • a critical value with ambiguous column ancestry must be withheld and sent to review.

Also corrupt the OCR output from KYO-H12 to KYO-H1Z. Entity linking must not silently fuzzy-match a critical identifier; it should surface the conflict across contract, property, and itinerary records.

Exit Criteria

The fixture passes when:

  • critical IDs KYO-H12, KYO-012, and JPN-2026-0417 are exact matches;

  • all four logical documents and three cross-document links are recovered;

  • all 12 labeled reading-order constraints pass;

  • the table retains the expected rows, columns, merged headers, continuation link, currency, and cell provenance;

  • accessible_room_peak_rate = JPY 25,000 points to local page 2, packet page 6, row Accessible, column Peak;

  • provenance coverage is delivered critical fields with valid sources / delivered critical fields = 1.0, and required-field completeness is separately recovered required fields / all required fields = 1.0 for the intact fixture so withholding everything cannot pass;

  • both the shifted-table and corrupted-ID cases are withheld rather than guessed;

  • parser version, OCR/model version, source checksums, per-stage latency, and scorecard are recorded.

These gates are specific to the fixture. Set production thresholds by document class and consequence: a rate table and a marketing brochure do not share the same failure tolerance.

What This Post Is Not

Document intelligence is already multimodal whenever it combines text, page images, and layout. This post draws a practical boundary around document reconstruction: layout recovery, table structure, document splitting, and evidence assembly. Post 9 treats photos, figures, floor plans, and cross-modal retrieval as first-class evidence, while retrieval strategy stays in Post 7.

Misconceptions That Cause Bad System Design

Five misconceptions recur in document-heavy AI projects.

First, teams assume OCR text alone is sufficient regardless of what the task requires. A faithful textual or structured representation can preserve the table, layout, and cross-document relationships a task needs, but once those relationships are lost, retrieval over the remaining text cannot recover them on its own.

Second, teams assume a table's cell values preserve its meaning by themselves. Merged headers, continuation rows, currencies, and notes outside the grid may determine how values relate. Flattening a grid without preserving those relationships can destroy its meaning; an explicit serialization that retains them need not.

Third, teams treat one uploaded PDF as one logical document. That assumption fails on contract bundles, appendices, and supplements, all of which are common in travel operations and enterprise workflows.

Fourth, teams assume OCR-free document models remove the need for structure and validation. Even without a separate OCR stage, a system still has to recover document relationships, preserve provenance, and evaluate extraction quality.

Fifth, teams treat document intelligence as a warm-up exercise for multimodality. Structure, hierarchy, and extraction quality make document intelligence its own systems layer, and they matter even before figures and photographs become central.

Common Failure Modes in Production

The most consequential production failures are usually structural.

Broken reading order can interleave multi-column text or detach sidebars from surrounding clauses, and section headings can drift away from the paragraphs they govern. Tables split across page breaks can be reconstructed incorrectly, and dropped header rows can lead parsers to assign rates to the wrong columns. Bundled appendices can stay merged with master agreements, and entity linkers can over-match similar identifiers or miss explicit cross-references. Extracted numbers can also travel downstream without page or cell provenance, letting OCR errors propagate silently into retrieval, normalization, and answer generation.

These failures are dangerous because the resulting answers often look plausible. A planner may receive a clean summary with no sign that the underlying rate table was misread or that a rate came from an appendix rather than the master contract. In high-stakes workflows, the main risk is silent misattribution rather than an obvious parse crash.

Practical Design Guidance

Treat document intelligence as a dedicated architectural layer rather than a preprocessing script bolted onto retrieval.

Start with file inspection. Before indexing or retrieving anything, classify incoming files, distinguish born-digital text from scans, and check whether a single upload bundles multiple logical documents.

Preserve structure before chunking. Recover page regions, reading order, and table boundaries first; flattening pages up front can destroy the spatial cues needed for reliable extraction.

Represent tables explicitly with row and column hierarchy, headers, currencies, and page coordinates. Only reduce a table to plain text if task-level evaluation shows the structural loss is acceptable.

Separate extraction from synthesis. Assemble a validated evidence packet first, then hand that structured packet to a downstream model for comparison or explanation so every stage remains debuggable and auditable.

Make provenance first-class by recording page coordinates and, for tables, cell coordinates wherever feasible. Reviewers cannot adjudicate conflicting numbers if they cannot jump straight to the source location.

Keep entity linking conservative. Match contract IDs, booking references, and property codes through explicit rules or calibrated scoring with validation rather than casual string similarity.

Evaluate each document task independently. Measure table reconstruction, document splitting, identifier linking, and provenance coverage alongside final answer quality so fluent prose cannot mask a broken parser.

Treat OCR as a baseline rather than dogma. Some workflows will stay OCR-first for practical reasons; others may adopt OCR-free parsers for selected document classes. What matters is whether the pipeline preserves the structure the task requires.

Why This Leads Naturally to Multimodal Evidence Systems

Once a team accepts that documents are not text blobs, the next architectural step becomes obvious.

Many important claims in travel contracts do not live only in paragraphs or tables. They also live in floor plans, property photos, accessibility diagrams, captions, legends, and the relationships among them. A system that can reconstruct page layout, table structure, and document hierarchy is already moving beyond plain-text retrieval toward a fuller evidence model.

That is the bridge to multimodal systems.

Post 9 picks up from here. Once figures, charts, and images become first-class evidence alongside text and tables, document intelligence expands into multimodal grounding: linking visual content to captions, surrounding prose, and cross-document claims without losing provenance.


Source Notes

This post draws on the following primary and practitioner sources:

  • Xu, Y., Li, M., Cui, L., et al. "LayoutLM: Pre-training of Text and Layout for Document Image Understanding." *KDD 2020.* Reference for the claim that layout and style information matter to document understanding. arxiv.org/abs/1912.13318

  • Huang, Y., Lv, T., Cui, L., et al. "LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking." *ACM Multimedia 2022.* Joint text/image/layout document representation and downstream document-task evidence. arXiv:2204.08387

  • Wang, Z., Xu, Y., Cui, L., Shang, J., and Wei, F. "LayoutReader: Pre-training of Text and Layout for Reading Order Detection." *EMNLP 2021.* ReadingBank and reading order as a separately evaluable task. ReadingBank

  • Li, M., Xu, Y., Cui, L., et al. "DocBank: A Benchmark Dataset for Document Layout Analysis." *COLING 2020.* Reference for fine-grained layout analysis as a distinct task. arxiv.org/abs/2006.01038

  • Smock, B., Pesala, R., and Abraham, R. "PubTables-1M: Towards Comprehensive Table Extraction From Unstructured Documents." *CVPR 2022.* Reference for table detection, structure recognition, and functional analysis as separable tasks. arxiv.org/abs/2110.00061

  • Zhong, X., ShafieiBavani, E., and Jimeno Yepes, A. "Image-based Table Recognition: Data, Model, and Evaluation." *ECCV 2020.* Primary source for TEDS as a structure-and-content table metric. arXiv:1911.10683

  • Kim, G., Hong, T., Yim, M., et al. "OCR-free Document Understanding Transformer." *ECCV 2022.* Reference for OCR-free document-understanding approaches and OCR error-propagation cautions. arxiv.org/abs/2111.15664

  • National Institute of Standards and Technology. "The Text Recognition Algorithm Independent Evaluation (TRAIT)." NISTIR 8199, December 2017. Normalized character- and word-level edit-distance measures for text recognition. NIST IR 8199

  • Smock, B., Pesala, R., and Abraham, R. "GriTS: Grid Table Similarity Metric for Table Structure Recognition." *ICDAR 2023.* Grid-based table evaluation and its variants. arXiv:2203.12555

Reading order

Complete reading route

Chapter 14 / 25

You are here56%
  1. 01
  2. 02
  3. 03
  4. 04
  5. 05
  6. 06
  7. 07
  8. 08
  9. 09
  10. 10
  11. 11
  12. 12
  13. 13
  14. 14
  15. 15
  16. 16
  17. 17
  18. 18
  19. 19
  20. 20
  21. 21
  22. 22
  23. 23
  24. 24
  25. 25

Review prerequisites, route branches, and the shared lab →

Share this post
XLinkedIn

Huang Tzu Lin

Founder of OptiVerse Technology. Writing about AI system design, evidence, and engineering trade-offs.

Subscribe via RSS

Follow the latest AI systems engineering tutorials in your preferred RSS reader.

Open RSS Feed

Works with any RSS reader — new posts arrive automatically.

Back to top