GH-36593: [Python] Add rename_columns method to pyarrow datasets

[JonatanMartens]

Rationale for this change

See #36593
In particular this change is convenient when the column names stored in a file are different from the logical names associated with the columns (see deltalake column mapping feature as an example).

What changes are included in this PR?

Adds the rename_columns method to datasets in pyarrow.
This mehod allows a user to rename columns in the data returned from a scan before actually creating a scanner object.

Are these changes tested?

This PR also add a test for the new rename_columns method using an InMemoryDataset.

Are there any user-facing changes?

Adds the rename_columns method to pyarrow datasets.

• GitHub Issue: [Python] Add rename_columns to DataSet #36593

[github-actions]

⚠️ GitHub issue #36593 has been automatically assigned in GitHub to PR creator.

[raulcd]

I don't think the examples where bad you just have to add the imports for doctest to work:
>>> import pyarrow.dataset as ds as seen here:

arrow/python/pyarrow/_dataset.pyx

Lines 371 to 391 in b2e8f25

	>>> import pyarrow as pa
	>>> table = pa.table({'year': [2020, 2022, 2021, 2022, 2019, 2021],
	... 'n_legs': [2, 2, 4, 4, 5, 100],
	... 'animal': ["Flamingo", "Parrot", "Dog", "Horse",
	... "Brittle stars", "Centipede"]})
	>>>
	>>> import pyarrow.parquet as pq
	>>> pq.write_table(table, "dataset_scanner.parquet")
	>>> import pyarrow.dataset as ds
	>>> dataset = ds.dataset("dataset_scanner.parquet")
	Selecting a subset of the columns:
	>>> dataset.scanner(columns=["year", "n_legs"]).to_table()
	pyarrow.Table
	year: int64
	n_legs: int64
	----
	year: [[2020,2022,2021,2022,2019,2021]]
	n_legs: [[2,2,4,4,5,100]]

[JonatanMartens]

@rok @raulcd @AlenkaF
It's ready to merge now, could you take a look?

[AlenkaF]

I am not sure about the changes in this PR, mainly because I am not very knowledgable when it comes to Acero and datasets. The functionality seems great to have, but modifying _scan_options for change of column names on read feels a bit hacky.

What do you think @rok ?

[rok]

The change looks good to me in principle.
I do agree with @AlenkaF that changing _scan_options seems a bit forced and could have unexpected consequences elsewhere. Can you check if there is a nicer way?

[JonatanMartens]

Sounds good, I'm now using a new attribute called _columns instead of relying on _scan_options
@rok @AlenkaF

[JonatanMartens]

@rok @AlenkaF Could you review this again?