Repository navigation
fix(scrape): distinguish non-Latin companies in fallback job keys - #487
Merged
MadsLorentzen merged 1 commit intoSep 21, 2026
Merged
Conversation
MadsLorentzen
force-pushed
the
fix/job-key-unicode-company
branch
from
September 21, 2026 06:21
32769f8 to
99ea1d1
Compare
Owner
|
Merged. Reproduced through the CLI on master: |
This was referenced Sep 21, 2026
p3ji
pushed a commit
to p3ji/ai-job-search-ca
that referenced
this pull request
Sep 23, 2026
MadsLorentzen
pushed a commit
that referenced
this pull request
Oct 5, 2026
…n fragment (#514) The hash fallbacks from #487 and #502 fired only when the slug was completely empty. A mixed-script name kept its Latin or digit fragment and skipped them: "Программист 1С" and "Аналитик 1С" at one company both keyed as company-6d769a_1, "Python-разработчик" and "Python-аналитик" as ..._python, "Сбер AI" and "Яндекс AI" as ai_... /scrape Step 4 then dropped the second posting as already seen, and because the dict key is overwritten, --audit could not report the collision as a duplicate URL. Treat the fold as lossy whenever it drops a letter outside the Latin script, and let the existing fallbacks take over: the title half uses the portal's numeric id or the URL hash with the fragment kept as a readable prefix, the company half uses the NFC-casefold name hash with the fragment as prefix. Latin letters that also lack a decomposition (ø, æ, ß, ł) are deliberately not counted, so "Ørsted" still keys as "rsted" and a live Danish seen_jobs.json does not re-key. Seven new cases; the four collision cases fail on master.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was wrong
make_key()usesunknown-companywhenever a company name has no ASCII slug. That is appropriate for an absent name, but makes unrelated non-Latin companies indistinguishable:腾讯 / Software Engineerand阿里巴巴 / Software Engineerboth becomeunknown-company_software-engineeron master09435eb.The key is used by
/scrapeto store entries inseen_jobs.json, so these two postings cannot coexist under their independently generated keys. This is a reproduction with supplied CLI inputs, not a claim that a live portal collision was observed.Demonstration
On master both commands print
unknown-company_software-engineer. On this branch each printscompany-<name hash>_software-engineer, with a different hash. An empty company name still printsunknown-company_software-engineer.The new CLI test asserts three distinct, canonical keys for those two names and an absent name. It fails on master (
1 != 3) and passes here. Two control tests pin stable keys across case/canonical Unicode variants and different posting URLs, plus unchanged ASCII and missing-company keys.What changes
tools/job_key.py: when the ASCII company slug is empty but the name is not, hash the stripped, case-folded, NFC-normalized name. Use the existingHASH_LENconvention and keep the output in the canonical ASCII key alphabet. The fallback does not depend on the posting URL.tests/test_job_key.py: three CLI tests covering the collision and controls above.[Unreleased]/ Fixed entry.Existing state is not rewritten.
/scrapeStep 3 already recognizes existing postings by URL regardless of their old key, the continuity rule added during #441. The audit can still report old keys as drift. Names with an existing ASCII slug, including mixed-script names, are outside this change.Verification
09435eb; both control tests pass.python3 -m unittest tests.test_job_key: 19 tests, all pass.python3 -m unittest discover -s tests -t .: 473 tests, all pass.tools/lint_skills.py,tools/check_framework_version.py,tools/security_guards.py, andgit diff --check: all clean.