# `LangExtract.Alignment.Token`
[🔗](https://github.com/mdepolli/lang_extract/blob/v0.10.0/lib/lang_extract/alignment/token.ex#L1)

A token with its byte position in the source text.

Offsets are byte positions in the UTF-8 binary, matching `Regex.scan/3`
with `return: :index` and consumable by `binary_part/3`.

Internal — no stability guarantees; see the README's "Stability"
section. Documented because it explains how the library works, not
because it is API.

# `t`

```elixir
@type t() :: %LangExtract.Alignment.Token{
  byte_end: non_neg_integer(),
  byte_start: non_neg_integer(),
  text: String.t(),
  type: token_type()
}
```

# `token_type`

```elixir
@type token_type() :: :word | :number | :punctuation | :whitespace
```

---

*Consult [api-reference.md](api-reference.md) for complete listing*