Skip to content

fix(seeder): detect gzipped sitemaps by their content, not the URL - #2542

Open
adityajai119 wants to merge 1 commit into
unclecode:developfrom
adityajai119:fix/sitemap-gzip-detection
Open

adityajai119 wants to merge 1 commit into
unclecode:developfrom
adityajai119:fix/sitemap-gzip-detection

Conversation

@adityajai119

Copy link
Copy Markdown

Summary

Fixes #2541

AsyncUrlSeeder._iter_sitemap() and _iter_sitemap_content() gunzip a sitemap whenever its URL ends with .gz. When the server sends a .gz sitemap with Content-Encoding: gzip (nginx gzip_static, Apache AddEncoding, S3/CloudFront metadata), httpx has already removed that encoding, so gzip.decompress() gets plain XML, raises BadGzipFile ("Not a gzipped file (b'<?')"), and all URLs of that sitemap are dropped. In the other direction, gzipped content at a URL that doesn't end in .gz (e.g. sitemap.xml.gz?v=2) is never decompressed.

Both places now decompress when the body starts with the gzip magic bytes 1f 8b, whatever the URL. XML can't start with those bytes, so a plain sitemap is never passed to gzip.decompress().

List of files changed and why

  • crawl4ai/async_url_seeder.py: decide whether to gunzip by the body's magic bytes instead of the URL suffix, in _iter_sitemap() and _iter_sitemap_content().
  • tests/unit/test_sitemap_namespace_parsing.py: DummyResponse accepts bytes, plus three tests for a sitemap index whose child is a gzipped .xml.gz, a .xml.gz already decoded by the client, and a gzipped child whose URL has a query string after .gz.

How Has This Been Tested?

  • The two new tests for the already-decoded .xml.gz and for the ?v=2 URL fail on develop (assert [] == ['https://example.com/posts/1']) and pass with the fix. The gzipped .xml.gz test passes before and after.
  • pytest tests/unit/test_sitemap_namespace_parsing.py tests/test_bug_batch_1622_1786_1796.py: 23 passed (Windows, Python 3.13).
  • tests/unit/test_egress_policy.py: 13 passed; test_seeder_client_has_no_proxy_by_default fails on develop too on this machine (httpx mounts from local proxy settings).
  • The script from [Bug]: URL seeder drops .gz sitemaps that are served with Content-Encoding: gzip #2541 (httpx MockTransport serving the child with Content-Encoding: gzip): URLs found: [] on 0.9.5 and develop, both URLs with this branch.
  • black reports no changes in the added test code, and ruff check --select E9,F finds nothing new in the two files.

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have added/updated unit tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

AI-assisted: drafted with an AI coding agent; the tests above were run locally.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant