Hoist special-token lookups in wav2vec2 decode paths and drop a dead filter in wav2vec2_phoneme

[ishan-1010]

What this does

Fixes #47556. Finishes the per-token special-token cleanup from #47425 for the wav2vec2 family.

• Wav2Vec2CTCTokenizer.convert_ids_to_tokens: hoist set(self.all_special_ids) out of the id loop. The property rebuilds its list on every access, so the loop paid a rebuild per id.
• Wav2Vec2CTCTokenizer._decode: same hoist for self.all_special_tokens in the token loop.
• Wav2Vec2PhonemeCTCTokenizer._decode: delete a filter loop that never matches. It compares token strings against all_special_ids (ints), and convert_ids_to_tokens was already called with skip_special_tokens passed through, so even a type-corrected check would find nothing to remove.

Numbers

20k ids, 205 special tokens (the MMS shape: one added special per language), M1 CPU:

convert_ids_to_tokens: 751.6 ms -> 1.3 ms
decode:                362.3 ms -> 16.7 ms

Behavior

No output changes anywhere:

• wav2vec2 + wav2vec2_phoneme tokenization suites: identical results on this branch and on main (49 passed, 31 subtests both).
• phoneme decode compared byte-for-byte on main vs branch across skip_special_tokens True/False, with base vocab, delimiter, added and special tokens in the input: identical. That is the proof the deleted loop was dead.

Notes

#46578 (open) touches the same convert_ids_to_tokens line to change word-delimiter handling. This PR does not change that logic either way; the two compose semantically and I am happy to rebase whichever lands second.

AI-assisted; I reviewed every line and ran the tests and benchmarks above.

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: wav2vec2, wav2vec2_phoneme

[github-actions]

---

CI recap

Dashboard: View test results in Grafana
Latest run: 30195672536
Result: success | Grafana metrics are not available yet.

[itazap]

Nice! Thank you!

[itazap]

run-slow: wav2vec2, wav2vec2_phoneme

[github-actions]

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["models/wav2vec2", "models/wav2vec2_phoneme"]
quantizations: []

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[github-actions]

CI Results

Workflow Run ⚙️

Commit Info

Context	Commit	Description
RUN	d172a7ee	workflow commit (merge commit)
PR	d2668e9f	branch commit (from PR)
main	b6d5084f	base commit (on main)

✅ No failing test specific to this PR 🎉 👏 !