[C++][Python] Refactor python to arrow conversions based on a reusable conversion API

[asfimport]

We have a lot of technical debt accumulated in the python to arrow conversion code paths including hidden bugs. We need to simplify the implementation.

Reporter: Krisztian Szucs / @kszucs
Assignee: Krisztian Szucs / @kszucs

Related issues:

• [Python] ListArray has trouble with sizes greater than kMaximumCapacity (fixes)
• [Python] Tzinfo - string roundtrip fails on pytz.StaticTzInfo objects (fixes)
• [C++][Python] Auto chunking nested array containing binary-like fields result malformed output (fixes)
• [C++] Dictionary is unset when calling DictionaryArray.GetScalar for null values (fixes)
• [Python] ArrowCapacityError when doing Table.from_pandas with large dataframe (fixes)
• [Python] Support constructing dictionary array directly through pa.array() (fixes)
• [C++][Python] Support constructing StructArray from list of key-value pairs (fixes)
• [Python] Produce chunked arrays for nested types in pyarrow.array (fixes)
• [Python] Add custom pickle reducers for DictionaryScalar (fixes)
• [Python] StructScalar.as_py() fails if the type has duplicate field names (is related to)
• [C++][Python] Do not collect python iterators if not necessary (is related to)
• [Python] Consider to raise or normalize if a timezone aware datetime.time object is encountered during conversion (is related to)
• [C++] Add an Append method to the time builders which validates the input range (is related to)

PRs and other links:

• GitHub Pull Request #8088

_{Note: This issue was originally created as ARROW-9992. Please see the migration documentation for further details.}

[asfimport]

Ben Kietzman / @bkietz:
Issue resolved by pull request 8088
#8088