Repository navigation
job_key stores one freehire posting under two keys when a non-Latin title is re-listed #501
Description
Description
tools/job_key.py stores one freehire posting under two keys when the portal re-lists it with the title altered, because the hash that keys a non-Latin title covers title + url rather than the URL alone.
The numeric-id branch that is meant to handle this never fires for freehire. _JOB_ID requires a run of six digits, and freehire's real slugs are alphanumeric — inzhener-ooo-chen-hlk3qjfg. So for a posting whose title slugifies to nothing, the hash is the only key half left, and it moves whenever the title moves.
freehire-search's own SKILL.md describes it as covering "many countries and markets", and its public API returns Cyrillic and Greek titles today.
Steps to Reproduce
Real values, taken from the public API (/api/v1/agent/jobs/search?q=инженер) rather than constructed:
U=https://freehire.me/jobs/inzhener-ooo-chen-hlk3qjfg
python3 tools/job_key.py --company "ООО Чен" --title "Инженер" --url "$U"
python3 tools/job_key.py --company "ООО Чен" --title "инженер" --url "$U"
python3 tools/job_key.py --company "ООО Чен" --title "Инженер-механик" --url "$U"Expected Behavior
One URL is one posting, so the three commands print one key. /scrape Step 4 keys seen_jobs.json by this helper, and .claude/skills/job-scraper/SKILL.md states the requirement: "The key must be a pure function of the posting, because two runs that slugify differently store the same job twice and defeat the dedup this step exists to provide."
Actual Behavior
Three different keys:
company-3dc110_untitled-ba243d
company-3dc110_untitled-2463a2
company-3dc110_untitled-450848
The company half is a hash too, because the company name is also Cyrillic — that part is by design per the docstring.
Note the second case is only a change of letter case. That is not hypothetical: the same query returns both Инженер and инженер across rows.
Impact
seen_jobs.json accumulates duplicate entries for one posting on any portal whose ids carry no six-digit run, which is exactly the failure class the helper was written to close. /rank builds its exclusion set from that file, so a duplicate can be presented again after it was already seen.
python3 tools/job_key.py --audit already reports this shape under duplicate_urls, so the defect is observable with the tooling in the repo.
Scope is limited to titles that slugify to nothing — Cyrillic, Greek, CJK — on portals without a six-digit id. Latin-titled postings and LinkedIn are unaffected.
I have a fix and tests ready and will open a PR referencing this issue. The shape I went with is hashing the URL alone when one is present, falling back to the title only when there is no URL — the smallest change, and it treats the URL as the posting's identity. Two alternatives I did not take, in case you prefer one: widening _JOB_ID to accept alphanumeric slugs, or transliterating Cyrillic and Greek before slugifying. The first makes the key depend on a slug that can change on re-list; the second needs a transliteration table, which sits against the zero-runtime-dependency norm.
Activity
fix(job_key): hash the URL alone so a re-listed non-Latin title keeps…