Make tokenizer per-thread to unblock concurrent Redact

[ericstj]

Summary

PrivacyFilterTokenizer serialized all tokenization behind a global lock, which bottlenecked concurrent Redact calls on a single shared PrivacyFilter — even though ONNX Runtime's Run is already thread-safe. The tokenizer was the last serialization point preventing a shared filter from serving many concurrent redactions at full throughput.

This gives each thread its own TiktokenTokenizer via ThreadLocal<T> instead of locking (the tokenizer keeps internal encode caches that aren't guaranteed safe for concurrent mutation), and disposes it with the filter. Encode logic is otherwise unchanged.

Why not a batch API?

Concurrency beats batching for this model: a padded batch does maxLen work for every row, so throughput comes from driving many concurrent single-string Run calls on one shared instance — which this unblocks — not from a batch entrypoint. The single-string API is unchanged.

Testing

• New EncodeIsThreadSafeUnderConcurrency test hammers a shared tokenizer from all cores (256 iterations) against the tokenizer oracle; mismatched ids/offsets or an exception would reveal unsafe shared state.
• Full suite: 12/12 passing.

[Copilot]

Pull request overview

This PR removes the global tokenizer lock that serialized all tokenization work, enabling a single shared PrivacyFilter instance to handle many concurrent Redact calls without bottlenecking on tokenization.

Changes:

• Replaced global locking in PrivacyFilterTokenizer with a per-thread TiktokenTokenizer via ThreadLocal<T>.
• Ensured PrivacyFilter disposes its tokenizer during Dispose().
• Added a concurrency stress test to validate Encode behavior under parallel usage.

Show a summary per file

File	Description
tests/PrivacyFilter.Net.Tests/TokenizerTests.cs	Adds a parallel stress test to validate thread-safe Encode behavior against the oracle.
src/PrivacyFilter.Net/PrivacyFilterTokenizer.cs	Switches to a per-thread tokenizer instance and adds IDisposable to clean up the ThreadLocal.
src/PrivacyFilter.Net/PrivacyFilter.cs	Disposes the tokenizer when the filter instance is disposed.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

• Files reviewed: 3/3 changed files
• Comments generated: 1
• Review effort level: Low

[Copilot]

Review details

Comments suppressed due to low confidence (1)

tests/PrivacyFilter.Net.Tests/TokenizerTests.cs:33

• PrivacyFilterTokenizer now implements IDisposable, but MatchesTiktokenOracle still constructs it without disposing. This can leave the internal ThreadLocal<T> undisposed across the test run; switch this test to using var as well for symmetry with the new concurrency test.

    [Fact]
    public void EncodeIsThreadSafeUnderConcurrency()
    {
        // The tokenizer is now per-thread (replacing a global lock), so a single
        // shared instance must serve concurrent Encode calls without corruption.
        // Hammer it from many threads against the oracle: mismatched ids/offsets or

• Files reviewed: 3/3 changed files
• Comments generated: 1
• Review effort level: Low