[ktz03]

Summary

exact_duplicates hashes flattened pixel bytes, so a 3×2 image and a 2×3 image containing the same byte sequence are incorrectly grouped as exact duplicates. Include the pixel-array shape in the MD5 input so dimensions and channel layout participate in the comparison.

from pathlib import Path
from tempfile import TemporaryDirectory
from PIL import Image
from cleanvision import Imagelab

with TemporaryDirectory() as folder:
    for name, size in [("wide.png", (3, 2)), ("tall.png", (2, 3))]:
        Image.frombytes("RGB", size, bytes(range(18))).save(Path(folder) / name)
    lab = Imagelab(data_path=folder, verbose=False)
    lab.find_issues(issue_types={"exact_duplicates": {}}, n_jobs=1)
    assert lab.info["exact_duplicates"]["sets"] == []

Impact

Only the MD5 input used for exact-duplicate detection changes. Genuine decoded-pixel duplicates, including copies saved as PNG and TIFF, remain grouped. The five perceptual hash algorithms are unchanged.

Testing

• Full pytest --verbose --cov=src/cleanvision/ --cov-config .coveragerc tests/: 92 passed, compared with 86 passed, 6 failed on the original production code using the same test cases. This includes the existing Hugging Face, Torchvision and anonymous S3 checks.
• Six new public-API regressions cover L/RGB/RGBA images with one or two workers, genuine copies, changed pixels, issue flags, scores and summary counts. All six were rerun after a final test-only Black formatting change; the complete test-file AST is unchanged.
• Full-tree Black, flake8 over src tests, and strict mypy over all 18 source files pass.

Unaddressed cases: local execution was Windows CPU/Python 3.12.14, NumPy 1.26.4 and Pillow 13.0.0. Other platform/Python combinations were not run locally.

References

This retains the decoded-pixel hashing approach introduced in #105, rather than comparing encoded image-file bytes.

Reviewer Notes

Internal MD5 strings change; the current implementation retains duplicate sets rather than exposing per-image hashes. No precomputed-hash API is introduced.

Read the Docs currently fails in the unchanged Hugging Face tutorial while resolving the legacy cats_vs_dogs dataset ID, before duplicate detection runs (build log).

[CLAassistant]

All committers have signed the CLA.