Repository navigation
fix(seeder): detect gzipped sitemaps by their content, not the URL - #2542
Open
adityajai119 wants to merge 1 commit into
Open
adityajai119 wants to merge 1 commit into
adityajai119 wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #2541
AsyncUrlSeeder._iter_sitemap()and_iter_sitemap_content()gunzip a sitemap whenever its URL ends with.gz. When the server sends a.gzsitemap withContent-Encoding: gzip(nginxgzip_static, ApacheAddEncoding, S3/CloudFront metadata), httpx has already removed that encoding, sogzip.decompress()gets plain XML, raisesBadGzipFile("Not a gzipped file (b'<?')"), and all URLs of that sitemap are dropped. In the other direction, gzipped content at a URL that doesn't end in.gz(e.g.sitemap.xml.gz?v=2) is never decompressed.Both places now decompress when the body starts with the gzip magic bytes
1f 8b, whatever the URL. XML can't start with those bytes, so a plain sitemap is never passed togzip.decompress().List of files changed and why
crawl4ai/async_url_seeder.py: decide whether to gunzip by the body's magic bytes instead of the URL suffix, in_iter_sitemap()and_iter_sitemap_content().tests/unit/test_sitemap_namespace_parsing.py:DummyResponseaccepts bytes, plus three tests for a sitemap index whose child is a gzipped.xml.gz, a.xml.gzalready decoded by the client, and a gzipped child whose URL has a query string after.gz.How Has This Been Tested?
.xml.gzand for the?v=2URL fail ondevelop(assert [] == ['https://example.com/posts/1']) and pass with the fix. The gzipped.xml.gztest passes before and after.pytest tests/unit/test_sitemap_namespace_parsing.py tests/test_bug_batch_1622_1786_1796.py: 23 passed (Windows, Python 3.13).tests/unit/test_egress_policy.py: 13 passed;test_seeder_client_has_no_proxy_by_defaultfails ondeveloptoo on this machine (httpx mounts from local proxy settings).MockTransportserving the child withContent-Encoding: gzip):URLs found: []on 0.9.5 anddevelop, both URLs with this branch.blackreports no changes in the added test code, andruff check --select E9,Ffinds nothing new in the two files.Checklist:
AI-assisted: drafted with an AI coding agent; the tests above were run locally.