fix(pdf): recover whitespace when pdfminer collapses plain text

[LaplaceYoung]

Summary

Fixes whitespace-collapse behavior for plain-text PDFs in issue #120 without regressing table/form extraction.

What changed

• Added a collapse detector in PdfConverter to identify pathological extraction where words are concatenated with almost no spaces.
• Improved plain-page extraction via pdfplumber with conservative spacing tolerances (x_tolerance=1, y_tolerance=3).
• Kept existing pdfminer fallback for plain PDFs, but when pdfminer output is detected as collapsed, now fallback to the already-collected pdfplumber plain text if it is healthier.
• Tightened form/table column-width guard (adaptive_max_columns) to reduce false-positive table extraction on dense academic layouts.
• Added regression test coverage for the collapse-recovery path.

Files

• packages/markitdown/src/markitdown/converters/_pdf_converter.py
• packages/markitdown/tests/test_pdf_memory.py

Validation

• python -m pytest packages/markitdown/tests/test_pdf_memory.py -q
• python -m pytest packages/markitdown/tests/test_pdf_tables.py::TestPdfTableExtraction::test_borderless_table_extraction packages/markitdown/tests/test_pdf_tables.py::TestPdfFullOutputComparison::test_academic_paper_full_output -q

Issue

Closes #120

[LaplaceYoung]

@microsoft-github-policy-service agree