Skip to content

Fix text loss at page boundaries during incremental parsing#96

Open
alfarhan wants to merge 1 commit intobigbag:mainfrom
alfarhan:fix/pagination-text-loss
Open

Fix text loss at page boundaries during incremental parsing#96
alfarhan wants to merge 1 commit intobigbag:mainfrom
alfarhan:fix/pagination-text-loss

Conversation

@alfarhan
Copy link
Copy Markdown

Summary

All format parsers (TXT, Markdown, EPUB, FB2) silently drop text when a paragraph spans a page-batch boundary during incremental parsing (hot extend). This manifests as missing sentences/paragraphs when reading — the user turns the page and text that should follow is gone.

Root Cause

When layoutAndExtractLines() is called and the page batch limit (maxPages) is reached mid-paragraph:

  1. The processLine callback stops adding lines to pages (returns or sets a flag)
  2. But the extract loop continuesextractLine() destructively erases words from the ParsedText object
  3. After flushBlock returns false, the ParsedText is destroyed/reset — remaining words are permanently lost
  4. The next parsePages call resumes from a saved file offset that is past the lost text

This affects every parser because they all share the same pattern of calling layoutAndExtractLines() without an abort callback to stop word consumption when the page limit is hit.

The Fix

Part 1 — Stop consuming words when the page limit is hit:

Pass an abort callback to layoutAndExtractLines() that returns true when the page batch limit is reached. The existing abort check at ParsedText.cpp:286-287 already handles this correctly — it returns false without consuming more words. The problem was that no caller was passing this callback.

Part 2 — Preserve unconsumed words for the next batch:

When the page limit interrupts a paragraph mid-layout, the ParsedText object still contains the remaining words. Instead of destroying it, save it as a class member (pendingBlock_ / pendingTextBlock_) and flush it at the start of the next parsePages call.

Parsers Fixed

Parser Changes
PlainTextParser Added pendingBlock_ member, abort callback in flushBlock, resume logic
MarkdownParser Added pendingTextBlock_ member, abort callback in flushTextBlock and memory-flush path
Fb2Parser Added abort callback, conditional currentTextBlock_ reset
ChapterHtmlSlimParser Changed includeLastLine from false to true in emergency split (abort callback was already passed)

Test plan

  • Built and flashed on Xteink X4 (ESP32-C3)
  • Tested with long Arabic RTL markdown files that previously exhibited missing text between pages
  • Verified text flows correctly across all page boundaries after clearing page cache
  • Test with EPUB and FB2 Arabic content
  • Test with LTR content to verify no regression
  • Test under memory pressure (large files on device with limited free heap)

🤖 Generated with Claude Code

All format parsers (TXT, Markdown, EPUB, FB2) share a bug where text is
silently dropped when a paragraph spans a page-batch boundary during
incremental parsing (hot extend).

Root cause: when layoutAndExtractLines() is called and the page batch
limit (maxPages) is reached mid-paragraph, the processLine callback
stops adding lines to pages. However, the extract loop continues
consuming words from the ParsedText — calling extractLine() which
destructively erases words. After flushBlock returns false, the
ParsedText is destroyed or reset, permanently losing those words.
The next parsing session resumes from a saved file offset that is
past the lost text.

The fix has two parts:

1. Pass an abort callback to layoutAndExtractLines() so it stops
   the extract loop when the page limit is hit. This preserves
   unconsumed words in the ParsedText object.

2. Preserve the interrupted ParsedText (as a class member) so the
   remaining words are flushed at the start of the next parsePages
   call instead of being discarded.

Parsers fixed:
- PlainTextParser: added pendingBlock_ member, abort callback
- MarkdownParser: added pendingTextBlock_ member, abort callback,
  also fixed memory-flush path (includeLastLine=false)
- Fb2Parser: added abort callback, conditional textBlock reset
- ChapterHtmlSlimParser: changed includeLastLine from false to true
  in emergency memory-pressure split (the abort callback was already
  passed correctly)

Tested on Xteink X4 (ESP32-C3) with long Arabic RTL markdown files
that previously exhibited missing text between pages.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant