# `LangExtract.Chunker`
[🔗](https://github.com/mdepolli/lang_extract/blob/v0.10.0/lib/lang_extract/chunker.ex#L1)

Splits text into sentence-level chunks using the Alignment.Tokenizer.

Sentence boundary rules (mirroring upstream's `find_sentence_range`):
1. A `:punctuation` token ending in a sentence terminator (`.`, `!`, `?`,
   CJK equivalents — `...` counts, since symbol runs are one token) ends a
   sentence, unless the previous token plus the terminator form a known
   abbreviation (`"Dr" <> "." == "Dr."`).
2. After sentence-ending punctuation, trailing closing punctuation
   (`"`, `'`, `)`, `]`, `}`, `»`, `”`, `’`) is consumed into the same sentence.
3. A `:whitespace` token containing `\n` starts a new sentence unless the
   next token begins lowercase — lines opening with quotes, digits, or
   capitals all break (upstream: "assume break unless lowercase").

# `chunk`

```elixir
@spec chunk(
  String.t(),
  keyword()
) :: [LangExtract.Chunker.Chunk.t()]
```

Splits text into chunks respecting sentence boundaries.

## Options

  * `:max_chunk_chars` — maximum characters per chunk (required).
    Char-denominated to mirror upstream's `max_char_buffer`, so chunk
    boundaries land identically across the two libraries — the
    cross-library benchmarks depend on that. Output offsets are bytes.

---

*Consult [api-reference.md](api-reference.md) for complete listing*