Skip to content

fix(adaptive): read head_data with the keys _parse_head actually returns - #2535

Open
laoyyu wants to merge 1 commit into
unclecode:mainfrom
laoyyu:fix/adaptive-head-data-keys
Open

laoyyu wants to merge 1 commit into
unclecode:mainfrom
laoyyu:fix/adaptive-head-data-keys

Conversation

@laoyyu

@laoyyu laoyyu commented Nov 11, 2026

Copy link
Copy Markdown

What

StatisticalStrategy looked the <head> data of a discovered link up under keys that AsyncUrlSeeder._parse_head() never returns, so the title, meta description and meta keywords of every link were silently ignored.

Why

_parse_head() returns {"title": ..., "meta": {"description": ..., "keywords": ...}, ...} (that is how _extract_text_context() reads it too). But the ranking helpers used:

  • link.head_data["meta"]["title"] in _calculate_relevance()
  • link.head_data["description"] and link.head_data["keywords"] in _calculate_novelty()

Neither meta.title nor the two top-level keys exist, so all three lookups always produced ''.

User-visible effect: a link whose only match with the query was its page title scored 0 relevance and was never followed, and a link whose only new terms were in its meta description was reported with the "unknown novelty" fallback (its terms were never counted as already seen).

How checked

Added tests/test_adaptive_link_head_data_keys.py, which builds the head_data with the real _parse_head() and asserts that the title contributes to relevance and that the description/keywords contribute to novelty.

  • On main: 2 failed
  • With this change: 2 passed

`AsyncUrlSeeder._parse_head()` returns `{"title": ..., "meta": {...}, ...}`,
but `_calculate_relevance()` looked the page title up under
`head_data["meta"]["title"]` and `_calculate_novelty()` looked the description
and keywords up under `head_data["description"]` / `["keywords"]`.

None of those three keys exist, so the term extraction always ignored the
`<head>` data: a link whose only match was its page title scored 0 relevance
and was never followed, and a link whose only new terms lived in the meta
description was reported as having unknown novelty.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant