Skip to content

fix(links): resolve a relative base href against the page URL - #2552

Open
HhH666-ux wants to merge 1 commit into
unclecode:mainfrom
HhH666-ux:fix/base-href-relative
Open

HhH666-ux wants to merge 1 commit into
unclecode:mainfrom
HhH666-ux:fix/base-href-relative

Conversation

@HhH666-ux

Copy link
Copy Markdown

Summary

Fixes #2550.

<base href> is now resolved against the page URL with urljoin, so a relative base (root-relative or document-relative) produces absolute links instead of scheme-less strings such as /sub/page.html. Absolute base values behave exactly as before.

List of files changed and why

  • crawl4ai/content_scraping_strategy.py - resolve base_href with urljoin(url, base_href) before it is used to normalise links.
  • tests/test_base_href_relative_links.py - new regression tests for root-relative, document-relative, absolute and missing <base> values.

How Has This Been Tested?

python -m pytest tests/test_base_href_relative_links.py -q
4 passed

Without the change the two relative-base tests fail (/sub/page.html, sub/page.html); with it they return https://example.com/sub/page.html and https://example.com/app/sub/page.html.

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have added/updated unit tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

A document with `<base href="/sub/">` produced scheme-less links such as
`/sub/page.html`, while the same document without a `<base>` element produced
absolute links. Resolve the base href with urljoin so both paths agree.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Relative <base href> produces scheme-less extracted links

1 participant