# `LangExtract.Alignment.Tokenizer`
[🔗](https://github.com/mdepolli/lang_extract/blob/v0.10.0/lib/lang_extract/alignment/tokenizer.ex#L1)

Regex-based tokenizer that splits text into tokens with byte offsets.

Mirrors upstream langextract's `RegexTokenizer`: letter runs, digit
runs, and same-symbol runs (`...` is one token, `?!` is two) — so
`Tooke’s` tokenizes as `Tooke` · `’` · `s` and the aligner sees the
bare name exactly as upstream does. Whitespace tokens are additionally
preserved (upstream tracks newlines as a token flag instead) for
continuous offset mapping and the chunker's newline rule.
No text normalization is applied.

Internal — no stability guarantees; see the README's "Stability"
section. Documented because it explains how the library works, not
because it is API.

# `tokenize`

```elixir
@spec tokenize(String.t()) :: [LangExtract.Alignment.Token.t()]
```

---

*Consult [api-reference.md](api-reference.md) for complete listing*