feat(tokenizer): add early exit for truncation

[McPatate]

Start tackling #1573.

Here we skip tokenizing remaining splits after the max_length of the TruncationParams has been reached in the case TruncationDirection::Right.

This leaves further questions as to how we tackle shortening the pre-tokenization pipeline as well as the tokenization. Since here we introduce the bound check on the token count, we still do the full context's pretok before the token count skip.
In the benchmark I've left for reproduction in case that is of interest to anyone, there's a pre-truncated-input-512 bench that is orders of magnitude faster, because it cuts the actual input text before passing it to the tokenizer.

I also only handled right truncation as it's the simplest, need to figure out how to do left tokenization. I think I'll tackle both topics in subsequent PRs.

Benchmark improvement results (if unspecified, direction == right):

Benchmark	Before	After	Change
truncate-to-512 (full corpus)	1.01 s	732 ms	-28%
scaling: 10K chars	1.26 ms	1.06 ms	-16%
scaling: 100K chars	13.2 ms	10.5 ms	-21%
scaling: 500K chars	71.3 ms	56.6 ms	-21%
max_length=128	6.63 ms	5.21 ms	-22%
max_length=512	6.62 ms	5.30 ms	-20%
max_length=2048	6.77 ms	5.46 ms	-20%
max_length=8192	6.74 ms	6.00 ms	-11%
right direction	1.27 ms	1.06 ms	-16%
left direction	1.27 ms	1.27 ms	no change (expected)
no truncation	975 ms	971 ms	no change (expected)

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[McPatate]

So actually implementing left truncation support was pretty easy, yields a perf boost consitent with the right truncation of ~17%:

Benchmark	Before	After	Change
left direction	1.27 ms	1.06 ms	-17%

[ArthurZucker]

Mega nice actually

[ArthurZucker]

If you have 1 split, HUGE ass one -> slow -> chinese don't benefit AFAIK.
If you have many small splits -> normal
If you have many big splits -> huge wins (after 2 splits boom your done).