Thai and number blocks are not auto-scaled and get wrong hyphen when using ­ in insert_htmlbox

[Arlencl]

Description of the bug

When using insert_htmlbox to render Thai text or long pure number sequences (which do not have natural word breaks), the following problems occur:

1. No Auto-Scaling:

   • If the input text cannot be split (for example, Thai text without spaces, or a long number like "12345678901234567890"), insert_htmlbox will not auto-scale (shrink) the text to fit the given rectangle. As a result, the content will overflow the boundary and be cut off.

2. Wrong Hyphen for Thai (­ handling):

   • When Thai text is pre-tokenized (e.g., using PyThaiNLP), and then joined with ­ to represent soft line-break opportunities, insert_htmlbox may insert a "-" (hyphen) at the line break. However, adding a hyphen at word breaks in Thai is not in line with Thai writing conventions and is visually/semantically incorrect.

How to reproduce the bug

• Thai case:

  "bbox": [
                    317.98,
                    201.75,
                    641.93,
                    264.3
                ]
  text = '''<span style=\"font-size:60.02pt;color:rgb(255,255,211);\">ค่าธรรมเนียมชำระเมื่อมาถึง</span>'''
  # or: '''<span style=\"font-size:60.02pt;color:rgb(255,255,211);\">ค่าธรรมเนียม&shy;ชำระ&shy;เมื่อ&shy;มาถึง</span>'''
  page.insert_htmlbox(rect, text, scale_low=0)
  
  

  

• Number case:

Expected:

• If the text does not naturally break but overflows the rect, insert_htmlbox should apply auto-scaling so the full content fits.
• When using ­ as a break point (such as in Thai tokenization), do not insert a hyphen character at the break; in Thai, no such symbol should appear.

Observed:

• No auto-shrinking (scaling) for unbreakable blocks.
• A hyphen - is added at Thai ­ line breaks, which is not appropriate for Thai (and similarly for Chinese, Japanese, Korean, etc).

---

PyMuPDF version

1.26.3

Operating system

Windows

Python version

3.10

[Arlencl]

Sorry, I selected the wrong Python version.
The actual Python version I'm using is: Python 3.13

[Arlencl]

After more tests, I found that using the zero width space (​, Unicode \u200b) instead of ­ can prevent the unwanted hyphen with Thai line breaks in insert_htmlbox. This works much better for the Thai language.

So the main issue remaining is:

If the input text (such as Thai or a long number) does not contain any spaces or break points, insert_htmlbox will not auto-scale the text to fit the rectangle, which results in content overflow or being cut off.

Thank you!

[JorjMcKie]

Thank for looking into this and pointing us to zero width space.
I am out of good advice at the moment as to how these findings can be converted into actionable items:
We cannot know beforehand if the text contains overly long strings (with no intervening space, hyphens, optional etc.).
Not to mention, that we neither know what "too long" actually means in any given situation ... taking the rectangle, chosen fonts, ... WTF else into account.

And changing the method in a way such that it microscopically scales down the text until it fits cannot be a suitable solution either, right?

Now what?

[JorjMcKie]

Still waiting for your reaction, but for the time being it seems that the only solution is return the problem to the programmer: When determining long strings, insert zero-width spaces every 5 characters or something like that ...

[Arlencl]

Sorry for the late reply—I missed the notification email.

Thank you for your response! I’d like to further clarify the practical pain points and add some constructive suggestions for insert_htmlbox:

1. Some lines can still overflow, even with break opportunities:

   • Even after adding zero-width spaces or other soft breaks, if a particular “word” or number is itself too long (e.g., an indivisible Thai word or a short number like "100"), it can still overflow the rectangle. This results in part of the word or number being clipped, rather than having the text automatically scaled down to fit the available space.
   • For example, even with good segmentation, if the bounding box is just a little too narrow, the last character of "100" (the "0") may only be half-visible.

2. Auto-scaling is only triggered when breaks are present:

   • In my tests, when there are multiple break opportunities (such as zero-width spaces), the PyMuPDF layout engine is able to do line breaks in optimal places and may auto-scale the content so that every line fits.
   • However, if there are no breaks at all in the text, auto-scaling does not get triggered, and overflow/clipping happens. In these cases, it would be more reasonable to automatically shrink the content (even for a single line) so that the full text fits, even if that means using a smaller font size for that line.

3. Suggestion:

   • If a line (even a single unbreakable “word” or number) would naturally exceed the width of its container, could there be a fallback detection to trigger auto-scaling (font-size reduction) for that line?
   • Even a simple loop that tries smaller font sizes until the content just fits horizontally would produce a much better result than the current hard clipping.
   • At the very least, it would be helpful to document this behavior, including the “best practice” of inserting zero-width spaces for CJK, Thai, or number block auto-wrapping, so that other developers can apply the right workaround for these languages.

[JorjMcKie]

I found a way to detect the width overflow and adjust scaling appropriately. Currently testing this ...

Please me have your Thai example.

[Arlencl]

3619_0.pdf

This example simulates the case of restored overflow.

restored_blocks = [
            {
                "bbox": [
                    317.98,
                    201.75,
                    641.93,
                    264.3
                ],
                "translated_html": "<span style=\"font-size:60.02pt;color:rgb(255,255,255);\">ภาษีและค่าธรรมเนียม</span>"
            },
            {
                "bbox": [
                    352.06,
                    304.99,
                    607.9,
                    330.0
                ],
                "translated_html": "<span style=\"font-size:24.0pt;color:rgb(255,255,255);\">ทีมสนับสนุนภาษีและค่าธรรมเนียม</span>"
            },
            {
                "bbox": [
                    437.14,
                    340.85,
                    522.86,
                    365.88
                ],
                "translated_html": "<span style=\"font-size:24.02pt;color:rgb(255,255,255);\">20230612</span>"
            }
        ]


    input_pdf = '3619_0.pdf'
    doc = fitz.open(input_pdf)
    page = doc[0]
    for block in restored_blocks:
        try:
            block['bbox'][3] = block['bbox'][3] + (block['bbox'][3] - block['bbox'][1]) * 0.3
            rect = fitz.Rect(block['bbox'])
            result = page.insert_htmlbox(rect, block['translated_html'], scale_low=0)
        except Exception as e:
            print(f"insert block {block['id']} fail: {str(e)}")
            continue
    doc.ez_save('result.pdf', garbage=4)  # 使用ez_save方法以更好的性能保存)

[Arlencl]

If appropriate zero width spaces are added, the content will be correctly scaled down.

restored_blocks = [
            {
                "bbox": [
                    317.98,
                    201.75,
                    641.93,
                    264.3
                ],
                "translated_html": "<span style=\"font-size:60.02pt;color:rgb(255,255,255);\">ภาษี&#8203;และ&#8203;ค่าธรรมเนียม</span>"
            },
            {
                "bbox": [
                    352.06,
                    304.99,
                    607.9,
                    330.0
                ],
                "translated_html": "<span style=\"font-size:24.0pt;color:rgb(255,255,255);\">ทีม&#8203;สนับสนุน&#8203;ภาษี&#8203;และ&#8203;ค่า&#8203;ธรรมเนียม</span>"
            },
            {
                "bbox": [
                    437.14,
                    340.85,
                    522.86,
                    365.88
                ],
                "translated_html": "<span style=\"font-size:24.02pt;color:rgb(255,255,255);\">20230612</span>"
            }
        ]