Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions AGENTS.md

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ Quick Start Example
>>> russian.parse("Сидоров Иван Петрович").family
'Сидоров'
>>> locales.available()
('bn', 'cs', 'es', 'ga', 'gd', 'hi', 'hr', 'ja', 'ne', 'pt', 'pt_br', 'ru', 'sk', 'sl', 'tr_az', 'zh')
('bn', 'cs', 'es', 'es_ar', 'ga', 'gd', 'hi', 'hr', 'ja', 'ne', 'pt', 'pt_br', 'ru', 'sk', 'sl', 'tr_az', 'zh')

Learn more
----------
Expand Down
24 changes: 21 additions & 3 deletions docs/design/decisions/O6-two-family-names.md

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/design/mechanisms.md
Original file line number Diff line number Diff line change
Expand Up @@ -334,7 +334,7 @@ a fork for all ambiguous particles on "Dr. Van Jr.".

**Contract statement.** A locale pack is pure data — a choice of languages, a Lexicon fragment of ADDITIONS and a Policy patch — applied by parser_for, a shipped pack's patch compiled from its language file's `[rules]` table where one is written, each key the `Policy` field of its name: the packs' languages union with the base's recorded choice into one choice, recorded on the `Lexicon`, which selects among the base's shipped readings by the file each sits in, the fragments join in pack order and the caller's lexicon after them, its own readings applying after the choice, and the patches fold in order; packs contain no code paths of their own. A vocabulary pack's deviations are derived from the reading index, the words whose compiled readings its choice changes; a policy or segmentation pack declares its own in its docstring and `DEVIATES`.

**How it works.** What is policy for every language (an order constant) is policy, not pack data; packs are lowercase modules exposing uppercase constants; a pack error is wrapped with the pack's code (rule D2). A choice changes the base's readings only by selection, never by a list of removals: a chosen language's `default = false` readings join; on a word a chosen language holds a vocabulary reading of, other languages' vocabulary readings leave, save `mul.toml`'s; and a chosen language's conjunction loses `may_be_initial`, its clash with the initial being a shape reading a preference outranks, which is how a `pt` choice drops `e` from `conjunctions_ambiguous` (decisions/vocabulary-model, 2026-10-27 #753, items 1, 2 and 7; built 2026-10-29, phase 3b). A strict choice, `parser_for(..., strict=True)`, selects by file alone: only the chosen files, `mul.toml` and native-script vocabulary compile, `und.toml`'s readings leaving with every other file's (decisions/vocabulary-model, 2026-10-30 #753, file-strict; built 2026-10-31, phase 3j; `mul.toml` kept since 2026-11-02 #1005). So a bad default is routed around only by choosing the language whose reading is right, which is why #342-class fixes still change shipped vocabulary. The choice filters the base's shipped readings and the caller's own `add` and `remove` calls apply after it, so a choice never drops a caller's word, and one that clashes with a reading the choice turns on is refused at `parser_for`, naming the edit; a lexicon built from `Lexicon.empty()` holds no shipped reading, so a choice turns none on in it; a pack choosing no language leaves the base's exactly as it was. The choice is recorded on the `Lexicon`, so `parser_for(EN, base=parser_for(PT))` chooses both, as `parser_for(PT, EN)` does (decisions/vocabulary-model, 2026-11-02 #754, the `Lexicon` surface, items 3 and 4; built 2026-11-02, sub-phase 4i-c). Pure data means a pack can be audited: a vocabulary pack's whole content is its tag, but what it changes reaches other files too, prefer dropping their readings of its words (a `de` choice drops `ca.toml`'s conjunction `i`), so the audit is the derived surface, `changed_words`, not the pack's file alone.
**How it works.** What is policy for every language (an order constant) is policy, not pack data; packs are lowercase modules exposing uppercase constants; a pack error is wrapped with the pack's code (rule D2). A choice changes the base's readings only by selection, never by a list of removals: a chosen language's `default = false` readings join; on a word a chosen language holds a vocabulary reading of, other languages' vocabulary readings leave, save `mul.toml`'s; and a chosen language's conjunction loses `may_be_initial`, its clash with the initial being a shape reading a preference outranks, which is how a `pt` choice drops `e` from `conjunctions_ambiguous` (decisions/vocabulary-model, 2026-10-27 #753, items 1, 2 and 7; built 2026-10-29, phase 3b). A strict choice, `parser_for(..., strict=True)`, selects by file alone: only the chosen files, `mul.toml` and native-script vocabulary compile, `und.toml`'s readings leaving with every other file's (decisions/vocabulary-model, 2026-10-30 #753, file-strict; built 2026-10-31, phase 3j; `mul.toml` kept since 2026-11-02 #1005). So a bad default is routed around only by choosing the language whose reading is right, which is why #342-class fixes still change shipped vocabulary. The choice filters the base's shipped readings and the caller's own `add` and `remove` calls apply after it, so a choice never drops a caller's word, and one that clashes with a reading the choice turns on is refused at `parser_for`, naming the edit; a lexicon built from `Lexicon.empty()` holds no shipped reading, so a choice turns none on in it; a pack choosing no language leaves the base's exactly as it was. The choice is recorded on the `Lexicon`, so `parser_for(EN, base=parser_for(PT))` chooses both, as `parser_for(PT, EN)` does (decisions/vocabulary-model, 2026-11-02 #754, the `Lexicon` surface, items 3 and 4; built 2026-11-02, sub-phase 4i-c). Pure data means a pack can be audited: a vocabulary pack's whole content is its tag, but what it changes reaches other files too, prefer dropping their readings of its words (a `de` choice drops `ca.toml`'s conjunction `i`), so the audit is the derived surface, `changed_words`, not the pack's file alone. A country pack is the same data with a region tag: `locales.ES_AR` chooses `es-AR`, which selects `es.toml` by ancestry until a reading is Argentina's own, names its language's `[rules]` (`rules_patch("es")`) rather than inheriting them, and carries its own lexicon fragment, so stacked on its language's pack it reads as itself where its fragment holds the language pack's (decisions/O6, 2026-11-04).

**Lives in.** nameparser/locales/ (packs), nameparser/_parser.py (parser_for, the one applier), nameparser/_lexicon.py (the compile under a recorded choice, `_chosen_pool` and `_built`), nameparser/_reading_index.py (the selection, `choose`, and the derived surface, `changed_words`), nameparser/_locale.py (`rules_patch`, a file's compiled `[rules]` as a patch) and tools/vocab/build.py (`RULE_KEYS`, the rules a file may turn on, and `PACK_RULES`, the files the shipped packs read; decisions/vocabulary-model, 2026-10-29 4f). A rule a file turns on is a `Policy` field; a new rule kind is a core change, never code in a pack: a new `Policy` field (with `ADDED_IN`), a row in build.py's `RULE_KEYS` and in `_locale._RULE_VALUES`, and a mechanisms.md entry of its own for the rule kind.

Expand Down
62 changes: 59 additions & 3 deletions docs/locales.rst
Original file line number Diff line number Diff line change
Expand Up @@ -556,7 +556,7 @@ look one up dynamically by its lowercase code with
.. doctest::

>>> locales.available()
('bn', 'cs', 'es', 'ga', 'gd', 'hi', 'hr', 'ja', 'ne', 'pt', 'pt_br', 'ru', 'sk', 'sl', 'tr_az', 'zh')
('bn', 'cs', 'es', 'es_ar', 'ga', 'gd', 'hi', 'hr', 'ja', 'ne', 'pt', 'pt_br', 'ru', 'sk', 'sl', 'tr_az', 'zh')
>>> locales.get("ru") is locales.RU
True

Expand Down Expand Up @@ -603,6 +603,13 @@ Shipped packs
given name out of the family (``Gabriel García Márquez`` →
family ``García Márquez``, ``José Luis Pérez`` → family
``Pérez``). See `Spanish and Portuguese titles`_.
* - ``es_ar``
- Argentine Spanish — ``es`` with a longer given-name list, one
holding given names that Spain bears often enough as surnames
to leave off ``es``'s list, which Argentine names write before a
single surname (``Pablo Sebastián
Rosales`` → family ``Rosales``, where ``es`` gives ``Sebastián
Rosales``). See `Spanish and Portuguese titles`_.
* - ``ga``
- Irish — chooses ``ga``, which turns on the clan particles the
default leaves off: ``Nic`` (daughter of), the married forms
Expand Down Expand Up @@ -837,7 +844,56 @@ stays out of it:
Among ORCID's researchers in the 11 Spanish-speaking countries
measured, the names that agree in every box with the filer rise from
13.5% (Peru) to 82.5% (Argentina) by default to 89.3% (Argentina) to
97.3% (Chile) under the pack (measured 2026-10-30). A compound given name stays opt-in
97.3% (Chile) under the pack (measured 2026-10-30).

Two family names are less settled in Argentina than in Spain and most
of Latin America, and many of its commonest second given names, ``Sebastián``, ``Nicolás``,
``Matías``, ``Andrés``, are given names Spain also bears as surnames
in a fifth to a quarter of their uses, so ``es``'s list, which keeps
only words given in at least 90% of their uses, leaves them out and the
switch takes them into the family.
``es_ar`` is ``es`` with 5,000 given names from the same census tables,
the ones counted as given names in at least half their uses:

.. doctest::

>>> argentina = parser_for(locales.ES_AR)
>>> spain.parse("Pablo Sebastián Rosales").family
'Sebastián Rosales'
>>> argentina.parse("Pablo Sebastián Rosales").family
'Rosales'
>>> argentina.parse("Gabriel García Márquez").family
'García Márquez'

Measured again on 2026-11-04, after later changes to the parser and
without the records whose given box opens with a title, the Argentine
researchers' names that agree in every box rise from 83.4% by default
and 90.2% under ``es`` to 96.1% under ``es_ar``; among Argentine inventors on US patents, 93.8% by
default, 86.2% under ``es`` and 92.7% under ``es_ar``, since most of
them write one family name. A word Spain bears mostly as a surname,
``Martín``, ``Esteban``, is on neither list, so the switch still takes
it. Choosing ``es`` and
``es_ar`` together reads as ``es_ar``.

The longer list costs a name written with two family names whose first
family name is on it. ``Santiago`` is a given name in 57% of its uses
in Spain, so ``es_ar`` lists it and reads it as a middle name, where
``es`` keeps both family names:

.. doctest::

>>> spain.parse("Carlos Santiago López").family
'Santiago López'
>>> argentina.parse("Carlos Santiago López").family
'López'

That cost is why ``es`` keeps its shorter list: in Spain, Mexico,
Peru, Bolivia, Ecuador, Costa Rica, Guatemala, Cuba, the Dominican
Republic and Puerto Rico, the longer list
reads fewer researchers' names right than ``es``'s does (measured
2026-11-04).

A compound given name stays opt-in
(``two_given_names``): people filling a form with a middle-name box put
the second given name there. ``pt`` turns neither switch on, because it serves every
Portuguese-speaking country: among ORCID's researchers the switch
Expand Down Expand Up @@ -1034,7 +1090,7 @@ by ``tests/v2/test_locales.py``:
one per shape of the vocabulary it ships: single surname, compound
surname, and any spelling variant it means to cover.
- **By a choice of languages** (``ne``, ``hi``, ``bn``, ``cs``, ``sk``,
``sl``, ``hr``, ``ga``, ``gd``, ``es``, ``pt``, ``pt_br``): a name for each word the choice changes, in each script it
``sl``, ``hr``, ``ga``, ``gd``, ``es``, ``es_ar``, ``pt``, ``pt_br``): a name for each word the choice changes, in each script it
is written in and each position where the default reads it
otherwise — ``test_choice_rotators_cover_every_changed_word`` fails
until every word is held.
Expand Down
1 change: 1 addition & 0 deletions docs/release_log.rst
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ Release Log
- **Add locales.NE, a Nepali pack, the first that is a choice of languages.** ``parser_for(locales.NE).parse("Guru Prasad Sharma")`` gives given ``Guru``, middle ``Prasad``, where the default parser gives title ``Guru``, given ``Prasad``; ``श्री`` and ``गुरु`` read the same way, each a name word placed by position (the family name under a family-first order) and still a title inside a title phrase (``His Holiness Guru Prasad``). It reads Hindi and Marathi names too, where ``श्री`` is usually the salutation, so use it for Nepali data. :issue:`753`
- **Add the Irish, Scottish Gaelic and Brazilian Portuguese packs, locales.GA, locales.GD and locales.PT_BR.** ``parser_for(locales.GA).parse("Máire Uí Bhriain")`` gives family ``Uí Bhriain``, where the default, as in 1.4.0 through 2.8.0, gives middle ``Uí``. ``GA`` reads ``Nic``, ``Mhic``, ``Mic`` and ``Uí`` as surname particles, ``GD`` reads ``Nic``, and ``PT_BR`` reads ``SB`` as the agnomen Sobrinho; ``F`` and ``N`` stay the name they spell. A choice settles its language's clashes its way, at a cost: ``John Smith MC`` loses the Military Cross under ``GA`` and ``GD``. See :doc:`locales`. :issue:`753`
- **Add the Spanish and Portuguese packs, locales.ES and locales.PT.** ``parser_for(locales.ES).parse("Lic Juan Pérez")`` gives title ``Lic``, given ``Juan``, where the default, as in 1.4.0 through 2.8.0, gives given ``Lic``, middle ``Juan``. ``ES`` reads ``Lic``, ``Licenciado``, ``Licenciada``, ``Lcdo``, ``Lcda``, ``Ingeniero``, ``Ingeniera``, ``Arq``, ``Mtro``, ``Maestra``, ``Mtra``, ``Doctora`` and ``Doña`` as titles, and ``PT`` (and ``PT_BR`` with it) ``Doutor``, ``Doutora``, ``Professora`` and ``Profa``; ``Don``, ``Dona``, ``Dom`` and ``Maestro`` stay names. ``ES`` reads ``Lady`` as a given name too (``Lady Heidenreich`` gives given ``Lady``), a title only in a phrase there (``Lady Chief Justice``, ``Hon Lady``), so a peeress written with the bare title reads given ``Lady``, her own given name joining the family (``Lady Annabel Goldsmith``). ``ES`` also reads two family names, a list of 1,000 Spanish given names keeping a given name out of the family: ``Gabriel García Márquez`` gives family ``García Márquez`` and ``José Luis Pérez`` family ``Pérez``. ``PT`` reads one family name, as the default does. See :doc:`locales`. :issue:`753`, :issue:`980`, :issue:`913`
- **Add locales.ES_AR, an Argentine Spanish pack.** ``parser_for(locales.ES_AR).parse("Pablo Sebastián Rosales")`` gives middle ``Sebastián``, family ``Rosales``, where ``locales.ES`` gives family ``Sebastián Rosales``; 1.4.0 through 2.8.0 and the default read it as ``ES_AR`` does. It is ``ES`` with a longer given-name list, holding given names that Spain bears often enough as surnames to leave off ``ES``'s list, which Argentine names write before a single surname, and two family names still read as two (``Gabriel García Márquez`` gives family ``García Márquez``). :issue:`1061`
- **Add ParsedName.generation, ParsedName.credentials and ParsedName.honorifics, the suffix split by kind.** ``parse("Smith, John, Jr., MD, Esq.")`` gives generation ``Jr.`` and credentials ``MD, Esq.``, where 1.4.0 through 2.8.0 gave only suffix ``Jr., MD, Esq.``, which is unchanged. They are views like ``family_base``: render keys, not fields of ``as_dict()``. A form of address after a name has a third view, ``honorifics`` (``parse("田中さん, PhD")`` gives credentials ``PhD`` and honorifics ``さん``), while a courtesy such as ``Esq.`` is a credential, and a suffix written by ``replace()``, or in a name unpickled from 2.x, is all credentials. ``Token.reading`` reads ``Jr.`` and ``Sr.`` as a generation and ``さん`` as an honorific, and ``Ambiguity.alternative`` of a lone ``Jr`` names a generation, and of a lone ``जी`` an honorific, where 2.7.0 and 2.8.0 named a post-nominal. :issue:`326`, :issue:`754`
- **Add ParsedName.patronymic, the words naming the bearer's parent.** ``parse("Mahathir bin Mohamad")`` gives patronymic ``bin Mohamad`` beside family ``bin Mohamad``, which 1.4.0 through 2.8.0 gave alone and which is unchanged; under ``locales.RU``, ``Ivan Ivanovich Ivanov`` gives patronymic ``Ivanovich`` and keeps it as the middle name, and under ``locales.TR_AZ`` a marker such as ``oglu`` reads with the name before it. A view like ``family_base``, not a field of ``as_dict()``, read only from a patronymic particle (``bin``, ``binti``, ``binte``, ``ibn``, ``bint``, ``bar``, ``bat`` and their Arabic and Hebrew forms) or a rule a locale turns on, so ``Anwar Ibrahim`` has none, and a name written by ``replace()``, or unpickled from 2.x, has none either. ``Ambiguity.alternative`` of a leading ``bin`` read as a given name names a patronymic where 2.7.0 and 2.8.0 named a particle. :issue:`754`
- **Add ParsedName.surname2, the second of two family names.** ``Parser(policy=Policy(two_family_names=True)).parse("Gabriel García Márquez")`` gives surname2 ``Márquez`` beside family ``García Márquez``, which 2.8.0 gave alone and which is unchanged. A view like ``family_base``, not a field of ``as_dict()``, filled only where ``two_family_names`` reads the family as two, so the default, a family comma (``Lloyd Webber, Andrew``) and a hyphenated family have none, and so does a name unpickled from 2.x or one whose family ``replace()`` rewrote. ``Reading.second_surname`` marks its words; it is not compared. :issue:`754`
Expand Down
9 changes: 5 additions & 4 deletions nameparser/_locale.py
Original file line number Diff line number Diff line change
Expand Up @@ -183,10 +183,11 @@ def rules_patch(tag: str) -> PolicyPatch:

def deviates_by_two_family_names(name: str) -> bool:
"""The DEVIATES declaration of a pack whose patch turns on
``Policy.two_family_names`` (rules/O6), the es, cs and sk packs, and
of the given-name list es carries beside it, which acts only where
O6 does. O6 acts only on a name of three or more name words, so a
name declares that holds three or more words, commas read as spaces,
``Policy.two_family_names`` (rules/O6), the es, es_ar, cs and sk
packs, and of the given-name list each Spanish pack carries beside
it, which acts only where O6 does. O6 acts only on a name of three
or more name words, so a name declares that holds three or more
words, commas read as spaces,
or three or more units counting each run of letters and each word
holding no letter: the tokenizer may split one word into two runs
(a zero-width space, a Hangul name segmented), and a word of no
Expand Down
1 change: 1 addition & 0 deletions nameparser/locales/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@
"BN": ("nameparser.locales.bn", "BN"),
"CS": ("nameparser.locales.cs", "CS"),
"ES": ("nameparser.locales.es", "ES"),
"ES_AR": ("nameparser.locales.es_ar", "ES_AR"),
"GA": ("nameparser.locales.ga", "GA"),
"GD": ("nameparser.locales.gd", "GD"),
"HI": ("nameparser.locales.hi", "HI"),
Expand Down
Loading
Loading