[docs] adding audio/video processors

[stevhliu]

follow up to this comment about separate docs for how to add audio and video processors

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[zucchini-nlp]

Nice to see more docs about each modality! The main q I have is about sectioning, since for me video+image belong together more than with audio. The API is also very similar to both

Also left a few more comments on common pain points when reviewing a model

[zucchini-nlp]

not sure this is the right sectioning. Video is also part of vision processing component imo. Prob we need a 1) small dropdown sectioning for multimodal processing and add three separately (images/videos/audio) OR 2) we can have "vision" and "audio" as two big sections

[stevhliu]

lets have image/video as one section, audio as another, and use your docs in #45493 for multimodal processors :)

[zucchini-nlp]

for processors, there is one more thing to mention. The usage of typed dicts and that we can enforce model-specific defaults in there
(now that I think of it, I dont understand why we set defaults in code and not save in configs 🤔 cc @molbap if you remember, or is that for BC?)

They are later merged in

transformers/src/transformers/processing_utils.py

Lines 1205 to 1248 in 7f6419e

	def _merge_kwargs(
	self,
	ModelProcessorKwargs: ProcessingKwargs,
	tokenizer_init_kwargs: dict | None = None,
	**kwargs,
	) -> dict[str, dict]:
	"""
	Method to merge dictionaries of kwargs cleanly separated by modality within a Processor instance.
	The order of operations is as follows:
	1) kwargs passed as before have highest priority to preserve BC.
	```python
	high_priority_kwargs = {"crop_size" = {"height": 222, "width": 222}, "padding" = "max_length"}
	processor(..., **high_priority_kwargs)
	```
	2) kwargs passed as modality-specific kwargs have second priority. This is the recommended API.
	```python
	processor(..., text_kwargs={"padding": "max_length"}, images_kwargs={"crop_size": {"height": 222, "width": 222}}})
	```
	3) kwargs passed during instantiation of a modality processor have fourth priority.
	```python
	tokenizer = tokenizer_class(..., {"padding": "max_length"})
	image_processor = image_processor_class(...)
	processor(tokenizer, image_processor) # will pass max_length unless overridden by kwargs at call
	```
	4) defaults kwargs specified at processor level have lowest priority.
	```python
	class MyProcessingKwargs(ProcessingKwargs, CommonKwargs, TextKwargs, ImagesKwargs, total=False):
	_defaults = {
	"text_kwargs": {
	"padding": "max_length",
	"max_length": 64,
	},
	}
	```
	Args:
	ModelProcessorKwargs (`ProcessingKwargs`):
	Typed dictionary of kwargs specifically required by the model passed.
	tokenizer_init_kwargs (`Dict`, *optional*):
	Dictionary of kwargs the tokenizer was instantiated with and need to take precedence over defaults.
	Returns:
	output_kwargs (`Dict`):
	Dictionary of per-modality kwargs to be passed to each modality-specific processor.

[zucchini-nlp]

oh this doesn't yet work for Processor and FeatureExtractor. I wanted to be sure it doesn't create new issues, and then expand more 😅
Will work on it next week!

[stevhliu]

sounds good! removed for now, but will add it back once its ready :)

[zucchini-nlp]

Also linking #45493 (processing refactor), I will add docs explaining how to add processors with the new API in mind and how to convert from old format to new. Not sure which one gets merged first, so I will tag you for review when the doc is pushed

[zucchini-nlp]

Thanks

I will merge the other PR on processors this week, and I can just rebase on top of yours. Feel free to merge when ready

[zucchini-nlp]

might need a re-do after merging #44394, just linking :)