Index of every page on this site: llms.txt
All skills
remsky avatar

/adding-normalizer

@ffa0691
by Jeremyremsky/kokoro-fastapi5.7k stars
951
Run it yourself

Adding a language normalizer to Kokoro-FastAPI: the Normalizer contract, registry, and test expectations. Use when adding or changing text normalization under api/src/services/text_processing/normalization/.

  • 1 file
  • 3.9 KB
  • Updated last week
  • GitHub
Use this Skill: https://skilld.dev/gh/remsky/kokoro-fastapi/adding-normalizer

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

SKILL.md

≈57 tokens for metadata: the name and description. ≈945 when used: this file.

Description uses 2.8% of example budget

Before choosing a Skill, your Agent reads its name and description. All available Skills share that space.

  • A shorter description leaves more room for other Skills. This entry exceeds our 1% size suggestion.

In our Claude Code example, all Skill names and descriptions share 8,000 characters. This Skill uses ≈227 characters, or 2.8%.

The 1% threshold is a size suggestion. Longer descriptions can still fit.

Your model, settings, and other Skills decide how much text your Agent can read.

Example settings and source

The example uses a 200k-token context and default Claude Code settings. The count includes the name, description, separators, and when_to_use when present. Codex also counts local file paths.

Skit's source and limits: Codex 0.160.1, Claude Code 2.1.292.

Read AGENTS.md and CONTRIBUTING.md first, house rules live there.

Adding a language normalizer

Text normalization (numbers, money, time, URLs, etc. spelled out before phonemization) is per language. English and Brazilian Portuguese are implemented; other languages skip normalization and rely on the phonemizer.

Where things live

api/src/services/text_processing/normalization/:

  • base.py: Normalizer, the contract. pass_order() is the pass order, normalize() runs it. The language-neutral passes (quotes, CJK punctuation, whitespace) are implemented; every other pass is a hook that returns its input unchanged until overridden. Override pass_order() if a language needs a different order.
  • english.py: EnglishNormalizer, the reference implementation.
  • english_data.py: the English tables, TLDs, units, symbol readings, currencies.
  • portuguese.py: PortugueseNormalizer, a small second language (numbers, currency, times, units). The closer template for a new one.
  • __init__.py: LANGUAGES, the registry. Lang codes come from lang_codes on each class.

Adding one

  1. Create normalization/<language>.py with a Normalizer subclass. Set lang_codes to the Kokoro pipeline codes it serves (p for Portuguese, e for Spanish, etc.). Override the hooks you need, numbers first (money is spelled out there too); espeak already reads bare digits, so the value is in currency, time, and units. num2words is in the lock and covers cardinals for most languages.
  2. Add an instance to LANGUAGES in __init__.py.
  3. Add api/tests/test_normalizer_<language>.py with a CASES table of (input, expected, options) rows, mirroring test_normalizer_portuguese.py. test_normalizer_contract.py runs the never-raises and idempotence properties on every registered language automatically.

Gotchas

  • Passes run in order (units only with unit_normalization, then times, then numbers), and an early pass hides text from the later ones. Each pattern that takes digits owns its sign and its left boundary ((?<![\w:])), or -2 km keeps a bare - and abc2km / 12:30min get read as units.
  • Reject malformed input whole (a trailing lookahead like (?!\d|[.,]\d)), so it falls through to the plain number pass. A half match reads worse than no match.
  • Bound every digit quantifier (\d{1,30}, (?:\.\d{3}){1,9}). It keeps the regex linear, and long runs are left for the phonemizer.
  • num2words raises OverflowError on large values (from 10^18 in pt_BR, limits vary by language). Catch it and return the digits unchanged, in every pass that calls it.
  • Pass num2words an int. Its Decimal path goes through float, losing precision and the sign between -1 and 0. Spell the fraction digit by digit and add the sign word yourself.
  • num2words has no grammatical gender and punctuates scale words its own way (dois milhões, duzentos). Agreement (duas horas, duzentas libras) is post-processing on its output, so strip punctuation when matching tokens.
  • Decide what happens to the other locale's format ($1.50 in Portuguese text) and pin it in a test.

Testing

  • Keep regexes linear. test_normalizer_redos.py shows the flood inputs that have bitten before. Add a flood unit there for each new pattern, with the option that enables its pass.
  • The contract properties run with default options, so option-gated passes (units) are only covered by rows in the language's own table. For each pattern include a sign, a huge number, and a neighbor that must stay untouched (clock, identifier).
  • uv run pytest api/tests/test_normalizer*.py, then uv run --only-group lint ruff format . and uv run --only-group lint ruff check . --fix before staging.

Source: SKILL.md on GitHub

No rule matched.

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No third-party reports yet.

Signed by skilld at ffa0691. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated last week

README badge

README badge for remsky/kokoro-fastapi/adding-normalizer