Skip to content

job_key stores one freehire posting under two keys when a non-Latin title is re-listed #501

Closed

Description

@ahmedashraffcih

Description

tools/job_key.py stores one freehire posting under two keys when the portal re-lists it with the title altered, because the hash that keys a non-Latin title covers title + url rather than the URL alone.

The numeric-id branch that is meant to handle this never fires for freehire. _JOB_ID requires a run of six digits, and freehire's real slugs are alphanumeric — inzhener-ooo-chen-hlk3qjfg. So for a posting whose title slugifies to nothing, the hash is the only key half left, and it moves whenever the title moves.

freehire-search's own SKILL.md describes it as covering "many countries and markets", and its public API returns Cyrillic and Greek titles today.

Steps to Reproduce

Real values, taken from the public API (/api/v1/agent/jobs/search?q=инженер) rather than constructed:

U=https://freehire.me/jobs/inzhener-ooo-chen-hlk3qjfg
python3 tools/job_key.py --company "ООО Чен" --title "Инженер"         --url "$U"
python3 tools/job_key.py --company "ООО Чен" --title "инженер"         --url "$U"
python3 tools/job_key.py --company "ООО Чен" --title "Инженер-механик" --url "$U"

Expected Behavior

One URL is one posting, so the three commands print one key. /scrape Step 4 keys seen_jobs.json by this helper, and .claude/skills/job-scraper/SKILL.md states the requirement: "The key must be a pure function of the posting, because two runs that slugify differently store the same job twice and defeat the dedup this step exists to provide."

Actual Behavior

Three different keys:

company-3dc110_untitled-ba243d
company-3dc110_untitled-2463a2
company-3dc110_untitled-450848

The company half is a hash too, because the company name is also Cyrillic — that part is by design per the docstring.

Note the second case is only a change of letter case. That is not hypothetical: the same query returns both Инженер and инженер across rows.

Impact

seen_jobs.json accumulates duplicate entries for one posting on any portal whose ids carry no six-digit run, which is exactly the failure class the helper was written to close. /rank builds its exclusion set from that file, so a duplicate can be presented again after it was already seen.

python3 tools/job_key.py --audit already reports this shape under duplicate_urls, so the defect is observable with the tooling in the repo.

Scope is limited to titles that slugify to nothing — Cyrillic, Greek, CJK — on portals without a six-digit id. Latin-titled postings and LinkedIn are unaffected.

I have a fix and tests ready and will open a PR referencing this issue. The shape I went with is hashing the URL alone when one is present, falling back to the title only when there is no URL — the smallest change, and it treats the URL as the posting's identity. Two alternatives I did not take, in case you prefer one: widening _JOB_ID to accept alphanumeric slugs, or transliterating Cyrillic and Greek before slugifying. The first makes the key depend on a slug that can change on re-list; the second needs a transliteration table, which sits against the zero-runtime-dependency norm.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions