Repository navigation
Conversation
A .cv file (https://cvfile.org/spec/) is a PDF/A-3u that embeds its canonical Markdown or HTML as PDF Associated Files and names the primary payload in a cv: XMP packet. Convert it from that payload with the Markdown or HTML backend instead of parsing the rendered pages. Add InputFormat.CV, detected from the .cv extension, the application/vnd.cv+pdf MIME type, or the cv: XMP namespace in a PDF. A PDF converts as CV only when InputFormat.CV is allowed, so callers that allow only InputFormat.PDF keep the current behavior. The backend verifies the sha-256 digest from cv:integrity, applies the 16 MiB payload cap of the specification, and warns on a newer major version. pypdf and defusedxml come with the new format-cv extra, which the standard bundle includes. Resolves docling-project#3567 Signed-off-by: Ilan <ilan.ohayon3@gmail.com>
|
✅ DCO Check Passed Thanks @ilanoh, all your commits are properly signed off. 🎉 |
Merge Protections🔴 1 of 2 protections blocking · waiting on 👀 reviews
🔴 Require two reviewer for test updatesWaiting for
This rule is failing.When test data is updated, we require two reviewers
Show 1 satisfied protection🟢 Enforce conventional commitMake sure that we follow https://www.conventionalcommits.org/en/v1.0.0/
|
AI triage (advisory)Checked commit The PR adds CI safety: needs maintainer care before you approve CI.
2 related pull request(s) in the same code
Reference data changes
Assessment: expected. The PR adds a new The |
There was a problem hiding this comment.
AI first review (advisory): no blocker found
This review covers the full diff at 3f772af807. It read the code and did not run it.
The change adds a PDF-to-CV reroute that runs for every PDF in the default converter, including installs that lack pypdf. Check what a slim install does with a PDF that declares the cv: namespace.
A maintainer makes the merge decision. This review never approves the pull request.
| obj = item | ||
| backend_input = item | ||
| format = self._guess_format(obj) | ||
| format = self._resolve_cv_format( |
There was a problem hiding this comment.
question: A .cv file may now fail on installs without format-cv, where it used to convert as a PDF
DocumentConverter() uses list(InputFormat) as allowed_formats by default, and format_to_options gets a CvFormatOption for each of them. So InputFormat.CV is always in format_options, even when pypdf and defusedxml are missing. _resolve_cv_format then maps the file to InputFormat.CV in two cases. In the first, the name ends in .cv. In the second, the XMP packet declares the cv: namespace. CvDocumentBackend.__init__ raises a plain ImportError when the dependencies are missing. InputDocument._init_doc re-raises every error that is not a DocumentLoadError. The conversion therefore aborts instead of returning a FAILURE result. Before this PR, these files converted as ordinary PDFs on a docling-slim[format-pdf] install.
Please confirm that this is intended. If it is not, fall back to InputFormat.PDF in _resolve_cv_format when the CV dependencies are unavailable. The new test test_backend_reports_missing_dependency_with_install_hint only covers the backend in isolation. It does not cover the default converter on a PDF.
There was a problem hiding this comment.
Not intended, thanks. Fixed in e60e66c. AbstractDocumentBackend now has a dependencies_available() classmethod, which returns True by default. The CV, Markdown and HTML backends implement it. _resolve_cv_format resolves to InputFormat.PDF when the CV backend cannot run, so a slim install converts these files as PDFs, as before. test_cv_converts_as_pdf_without_cv_dependencies covers the routing through docs(). I also checked it in a fresh interpreter with pypdf blocked: both the .cv file and the renamed .pdf resolve to pdf.
| stream = stream.get_object() if stream is not None else None | ||
| if not isinstance(stream, StreamObject): | ||
| continue | ||
| data = stream.get_data() |
There was a problem hiding this comment.
suggestion: The 16 MiB cap is checked after the stream is fully decoded
stream.get_data() decodes the whole embedded stream. Only after that does len(data) > _MAX_PAYLOAD_BYTES run. A small compressed payload can still expand to a large buffer in memory before it is rejected. If the stream has a /Length or a /Params /Size entry, check it before decoding. You can also bound the decoder output. I could not read the installed pypdf source, so I did not check its own decompression limits.
There was a problem hiding this comment.
Fixed in e60e66c. The backend now reads /Params /Size, which the .cv specification requires on every payload stream. It rejects a declared size above the cap before it decodes. The check after decoding stays in place for a stream that declares a wrong size. pypdf also bounds the output of its decoders, through pypdf.filters.ZLIB_MAX_OUTPUT_LENGTH (75 MB in 6.21), so memory stays bounded even then.
A default DocumentConverter allows InputFormat.CV, so on an install without the format-cv extra a .cv file, or a PDF that declares the cv: namespace, raised an ImportError where it used to convert as a PDF. Add AbstractDocumentBackend.dependencies_available(), implement it for the CV, Markdown and HTML backends, and resolve to InputFormat.PDF when the CV backend cannot run. Also reject a payload whose declared /Params /Size exceeds the 16 MiB cap before decoding it. Signed-off-by: Ilan <ilan.ohayon3@gmail.com>
There was a problem hiding this comment.
AI first review (advisory): no blocker found
This review covers the changes since 3f772af807 at e60e66c7a7. It read the code and did not run it.
I found nothing important in the new changes. dependencies_available() routes a .cv file or a CV-tagged PDF to the PDF format when the CV, Markdown or HTML packages are missing. The /Params /Size check now runs before decoding the payload.
Earlier findings
- fixed (question): A .cv file may now fail on installs without
format-cv, where it used to convert as a PDF - fixed (suggestion): The 16 MiB cap is checked after the stream is fully decoded
A maintainer makes the merge decision. This review never approves the pull request.
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
Adds
.cvresume files as an input format, as agreed in #3567.A
.cvfile (https://cvfile.org/spec/) is a PDF/A-3u that embeds its canonical Markdown or HTML as PDF Associated Files (ISO 32000-2, 14.13) and names the primary payload in acv:XMP packet. Docling treated these files as ordinary PDFs and ran layout and OCR over the rendered pages. With this change Docling reads the embedded payload and converts it with the existing Markdown or HTML backend, so the text is kept exactly and no model runs.Changes
InputFormat.CVwith thecvextension and theapplication/vnd.cv+pdfMIME type.CvDocumentBackend(docling/backend/cv_backend.py) readscv:primaryPayloadfrom the XMP packet with defusedxml, finds that file in the catalog/AFarray with pypdf, and delegates toMarkdownDocumentBackendorHTMLDocumentBackendaccording to the payload/Subtype. It verifies the sha-256 digest fromcv:integrity, applies the 16 MiB payload cap of the specification (section 7.3), and logs a warning for a newer major version (section 8.3).CvFormatOption(SimplePipeline+CvDocumentBackend), registered as the default option forInputFormat.CV.StandardPdfPipelineis unchanged._DocumentConversionInput:.cvextension maps the PDF MIME type reported byfiletypetoapplication/vnd.cv+pdf, next to the OLE2 extension rule.docs(), a PDF whose XMP declares thehttp://ns.cvfile.org/cv/1.0/namespace converts as CV whenInputFormat.CVis in the format options. A.cvfile converts as PDF when onlyInputFormat.PDFis allowed. Callers that passallowed_formats=[InputFormat.PDF]therefore see no change.format-cvextra (format-web,pypdf,defusedxml), included instandard. The imports are guarded like the email backend, andtest_backend_optional_dependencies.pycovers the missing dependency case.CvFormatOptionin the API reference, and theformat-cvextra in the packaging skill reference.Dependency on docling-core
DocumentOriginrejectsapplication/vnd.cv+pdfuntil docling-project/docling-core#898 is released. I will raise thedocling-corefloor inpyproject.tomlanduv.lockonce that release is out, as was done for AFP. Until then, the two end-to-end tests intests/test_backend_cv.pyfail on the origin MIME type check.Questions for reviewers
format-cvinstandardadds pypdf to the defaultdoclinginstall. pypdfium2 cannot read the catalog/AFarray, so I did not find a way to use it. I can move the extra out ofstandardif you prefer. In that case a slim install without the extra raises the install hint when it receives a PDF that declares thecv:namespace, unless the caller removesInputFormat.CVfromallowed_formats..cvextension relies on the XMP stream being unfiltered, which is the case for the reference producers (Python, TypeScript and Go SDKs). A.cvfile with a compressed XMP stream and a.pdfname converts as a PDF.Issue resolved by this Pull Request:
Resolves #3567
Checklist:
Review evidence:
Before: a
.cvfile, or the same file named.pdf, ran throughStandardPdfPipeline. The fixtures have a blank page, so the result was empty.After: both convert through
SimplePipelinefrom the embedded payload in about 0.03 s with the CLI (docling marie_curie.cv resume.pdf --to md). The Markdown output is identical to the ground truth.docling tests/data/pdf/bookmark_sample.pdfstill runsStandardPdfPipeline.Reference data: new files only, in
tests/data/cv/. The two fixtures were built with the cvfile Python SDK from public facts about Marie Curie on a blank page, so any text in the output comes from the embedded payload. One has a Markdown primary payload with an HTML alternate, the other has only HTML. Ground truth was generated withDOCLING_GEN_TEST_DATA=1.Commands and results, with docling-core#898 installed in editable mode:
make validate: all hooks pass.uv run pytest tests/test_backend_cv.py tests/test_input_doc.py tests/test_invalid_input.py tests/test_backend_markdown.py tests/test_backend_html.py tests/test_backend_email.py tests/test_backend_afp.py tests/test_backend_optional_dependencies.py tests/test_video_format_routing.py: 324 passed, 1 skipped, 1 failed. The failure istest_backend_html.py::test_e2e_html_conversions. It also fails onmainwith docling-core main, and it passes with the locked docling-core 2.100.0.allowed_formats=[InputFormat.CV]. The valid files convert. The integrity mismatch, oversized payload, missing XMP and encrypted files fail with aDocumentLoadErrorthat names the problem. The future major version file converts with a warning.Not run: the full test suite, because I installed only the extras that the touched code needs.