fix(job_key): hash the URL alone so a re-listed non-Latin title keeps one key

[ahmedashraffcih]

What changed and why

Closes #501.

make_key's hash for a title that slugifies to nothing covered title + url. The URL identifies the posting; the title does not. So when a portal re-listed the same posting with the title altered, the hash moved and seen_jobs.json gained a second entry for one URL — the failure tools/job_key.py exists to prevent, and the one .claude/skills/job-scraper/SKILL.md states the rule against: "The key must be a pure function of the posting."

The numeric-id branch above it never fires for freehire. _JOB_ID wants a run of six digits and freehire's real slugs are alphanumeric, so for its Cyrillic and Greek titles the hash is the only key half there is.

The change hashes the URL alone when one is present, and falls back to the title only when there is no URL. One line of logic; the rest is the comment explaining why.

I took this shape over two alternatives. Widening _JOB_ID to accept alphanumeric slugs would key on a slug that can itself change on re-list. Transliterating Cyrillic and Greek would give readable keys but needs a transliteration table, which sits against the zero-runtime-dependency norm for portal tooling. Happy to switch if you prefer either.

Failing case / reproduction

Real values from the public API (/api/v1/agent/jobs/search?q=инженер), not constructed. freehire-search's SKILL.md describes the portal as covering "many countries and markets", and that query returns both Инженер and инженер across rows today, so the case-variation below is something the portal really emits.

U=https://freehire.me/jobs/inzhener-ooo-chen-hlk3qjfg
python3 tools/job_key.py --company "ООО Чен" --title "Инженер"         --url "$U"
python3 tools/job_key.py --company "ООО Чен" --title "инженер"         --url "$U"
python3 tools/job_key.py --company "ООО Чен" --title "Инженер-механик" --url "$U"

On master, three keys for one URL:

company-3dc110_untitled-ba243d
company-3dc110_untitled-2463a2
company-3dc110_untitled-450848

On this branch, one:

company-3dc110_untitled-e9df22
company-3dc110_untitled-e9df22
company-3dc110_untitled-e9df22

python3 tools/job_key.py --audit already reports this shape under duplicate_urls, so the defect was observable with the tooling already in the repo.

Verification

Two tests added to tests/test_job_key.py, both using the real slugs above:

• test_one_url_keeps_one_key_when_a_non_latin_title_is_re_listed — fails on master, passes here. This is the one that pins the fix.
• test_distinct_urls_still_get_distinct_keys — passes both ways. It is the guard: collapsing title variants must not collapse two genuinely different postings from one company.

Ran what CI runs:

python3 tools/lint_skills.py              PASS
python3 tools/check_framework_version.py  PASS
python3 tools/security_guards.py          PASS
python3 -m unittest discover -s tests     490 tests, OK

488 before, 490 after. No existing test changed behaviour — the Korean-title, LinkedIn-id and Cyrillic-company cases all still pass untouched.

No CLI was modified, so no bun legs were affected.

Scope

Only titles that slugify to nothing — Cyrillic, Greek, CJK — on portals whose URLs carry no six-digit id. Latin-titled postings are untouched, and LinkedIn still resolves through the numeric-id branch, which the existing test pins.

Existing state files are unaffected in the sense that no key is rewritten; the audit reports legacy entries and never rewrites, as before. New scrapes of an affected posting will key differently from an entry written before this change, which is the same one-time shift any key-derivation fix causes.

[MadsLorentzen]

Reproduced on master with your three commands (three keys for one URL) and on the branch (one key, untitled-e9df22). The pinning test fails against master's job_key.py and passes here; 490 OK on Windows, lint and guards green, CI 18/18. The live API does return Инженер and инженер across rows today, and freehire's slugs carry no six-digit run, so the numeric-id branch really never fires for it. Keying the hash on the URL alone is the right shape: the URL is the identity /scrape stores, and the title was never a stable half.

Two things before merge:

1. No CHANGELOG entry. Please add one under [Unreleased] / ### Fixed naming tools/job_key.py and the test, and say in it that entries written before the fix for a non-Latin title on a portal without a numeric id will re-key once on the next scrape (--audit reports them, never rewrites).
2. Out of interest, not a blocker: did --audit report duplicate_urls on a real seen_jobs.json of yours, or is the same-URL re-list inferred from the case variation across rows? The fix stands either way; I would like to log which it was.

Merge on the push.

[ahmedashraffcih]

CHANGELOG added at 5854215, under [Unreleased] / ### Fixed, naming tools/job_key.py and tests/test_job_key_url_identity.py. The migration consequence is stated in bold: an entry written before this fix, for a non-Latin title on a portal without a numeric id, re-keys once on the next scrape; --audit reports those under keys_not_matching_current_rule and never rewrites them.

I checked that last clause rather than asserting it. A seen_jobs.json holding one entry under its pre-fix key for a Cyrillic-titled freehire URL reports keys_not_matching_current_rule: 1 under the fixed audit.

On your second question — inferred, not observed

Inferred. I want to be precise about it, since you asked to log which.

What I actually observed:

• The live API returning both Инженер and инженер across rows, on the same endpoint in the same window.
• The code path: freehire slugs carry no six-digit run, so normalizeId finds nothing and the title stays in the hash. Both casings slugify to untitled plus a different digest — which is where your untitled-e9df22 comes from.
• The two keys differing for one URL, by running make_key directly.

What I did not observe: duplicate_urls firing on a real accumulated seen_jobs.json. I ran --audit against an 884-entry file just now to be sure before answering:

{
  "entries": 884,
  "duplicate_urls": {},
  "keys_not_matching_current_rule": 0
}

Both zero. Two reasons, and the second is the interesting one: those 884 keys are the company_title form, which is_canonical() excludes from the drift check, so they could not have shown up under either field regardless. And no freehire posting in that file had been re-listed with a different casing during its lifetime.

So the honest characterisation is: a latent defect, demonstrated by mechanism plus live data, not one caught in the wild in a file of mine. It needs a re-list with changed casing between two scrapes of the same posting, which is a narrow window to catch by accident even when the conditions are all present.

If that changes your read of the priority, say so — the fix stands on the mechanism either way, but you should have the real provenance rather than my summary of it.

[ahmedashraffcih]

Rebased onto d7287ec (v1.7.2) and force-pushed — 936c846. The PR is mergeable again.

The conflict was the v1.7.2 cut: it rolled [Unreleased] into the release section, and my entry had been written against the old one, so it landed inside the released 1.7.2 block. Resolved by taking master's side wholesale and re-adding the entry under the now-empty [Unreleased] with a fresh ### Fixed. tools/job_key.py and tests/test_job_key.py are untouched by your six new commits, so only CHANGELOG.md moved.

One correction to flag, since you asked for the test to be named. The first version of that entry named tests/test_job_key_url_identity.py. No such file exists — the test is in tests/test_job_key.py. I invented a plausible filename instead of checking, in the exact line you asked me to add. Caught it while resolving the conflict, and I verified both named paths resolve on disk before pushing this time.

tests/test_job_key.py is 21 passed on the new base.

[ahmedashraffcih]

One heads-up so you are not waiting on a check that cannot start: the run on 936c846 is sitting at action_required (run 36692198261), the fork-PR approval gate. It needs "Approve and run workflows" before anything reports.

For what it is worth, no Actions run ever completed on this PR's earlier heads either — 7506ab1 and 5854215 have none — so the CI 18/18 in your review was presumably your own run rather than this PR's. Locally on 936c846: tests/test_job_key.py 21 passed.

[MadsLorentzen]

Merged, thank you. Re-verified 936c846 before merging: your three commands give one key on the branch and three on master, 505 OK on Windows, lint, guards and the framework check green, the CHANGELOG entry sits under [Unreleased] / Fixed, and CI is 18/18 on this head once I approved the fork gate. For the record, the run on 7506ab1 did complete green on Sep-28 (run 36405113215, the one I quoted); the run on 5854215 never started because the head moved under it.

Provenance logged as you gave it: inferred from mechanism plus live data, with --audit at zero on an 884-entry file. That is the honest characterisation and it does not change the priority. The key was never a pure function of the posting for this class, which is the one rule the helper exists to keep. Thanks also for catching the test-file name yourself. Closes #501.