Skip to content

fix(deep-crawl): don't append external links to result.links["internal"] - #2536

Open
laoyyu wants to merge 1 commit into
unclecode:mainfrom
laoyyu:fix/deep-crawl-link-list-mutation
Open

laoyyu wants to merge 1 commit into
unclecode:mainfrom
laoyyu:fix/deep-crawl-link-list-mutation

Conversation

@laoyyu

@laoyyu laoyyu commented Nov 11, 2026

Copy link
Copy Markdown

What

BFSDeepCrawlStrategy, DFSDeepCrawlStrategy and BestFirstCrawlingStrategy appended the external links of a page to the caller's CrawlResult.links["internal"].

Why

All three strategies start with

links = result.links.get("internal", [])
if self.include_external:
    links += result.links.get("external", [])

result.links["internal"] is the list object owned by the CrawlResult that is handed back to the user after the crawl, and += mutates it in place. With include_external=True every external URL of every crawled page therefore ended up in result.links["internal"], so downstream filtering (links["external"], link counting, storage) saw a result that no longer matched the page.

The fix copies the list before extending it, in all three strategies.

How checked

Added tests/test_deep_crawl_include_external_links.py, which calls link_discovery() on a synthetic result with one internal and one external link and asserts that the external link is returned for crawling but result.links["internal"] is left as it was.

  • On main: 3 failed (links['internal'] == ['https://example.com/a', 'https://other.com/b'])
  • With this change: 3 passed

All three deep-crawl strategies took the internal-link list straight out of
`result.links` and then did `links += external`. That list is the caller's own
`CrawlResult.links["internal"]`, so with `include_external=True` every external
link of the page stayed appended to the result the user receives after the
crawl, inflating `links["internal"]` with URLs from other domains.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant