[Python][C++] pyarrow.repeat returns an invalid array when a chunked array is required.

[daniel-shields]

Describe the bug, including details regarding any error messages, version, and platform.

pyarrow.repeat silently fails and returns an invalid array when the result cannot fit in a single chunk.

import sys
import pyarrow as pa
print(f"{sys.version}")
print(f"{pa.__version__=}")

array = pa.repeat("?", 2**31 - 1)
print(array)
array = pa.repeat("?", 2**31)
print(array)

output:

3.9.12 (main, Jan  1 2020, 00:00:00) 
[GCC 8.3.0]
pa.__version__='12.0.0'
[
  "?",
  "?",
  ...
  "?",
  "?",
]
<Invalid array: Negative offsets in binary array>

Component(s)

Python

[AlenkaF]

I agree the code should give an error when the result doesn't fit into an array.

Arrow format represents arrays of length up to the maximum 32-bit signed integer:
https://arrow.apache.org/docs/format/Columnar.html#array-lengths

and pyarrow.repeat or MakeArrayFromScalar function in C++ returns an Array, not a ChunkedArray:

arrow/python/pyarrow/array.pxi

Line 426 in cd1ed18

arr : Array

arrow/r/src/scalar.cpp

Line 61 in cd1ed18

std::shared_ptr<arrow::Array> MakeArrayFromScalar(

If the length is exceeding the limit, it should raise an informative error on the C++ side.

[AlenkaF]

Oh sorry, the last link to the C++ is wrong. I meant to add this:

arrow/cpp/src/arrow/array/util.cc

Line 851 in fb8760e

Result<std::shared_ptr<Array>> MakeArrayFromScalar(const Scalar& scalar, int64_t length,

After talking to @jorisvandenbossche about this issue I would like to add a proposed fix for it in the C++ here (contributions welcome).

The Negative offsets in binary array message is coming from CreateOffsetsBuffer:

arrow/cpp/src/arrow/array/util.cc

Lines 808 to 817 in fb8760e

	template <typename OffsetType>
	Status CreateOffsetsBuffer(OffsetType value_length, std::shared_ptr<Buffer>* out) {
	TypedBufferBuilder<OffsetType> builder(pool_);
	RETURN_NOT_OK(builder.Resize(length_ + 1));
	OffsetType offset = 0;
	for (int64_t i = 0; i < length_ + 1; ++i, offset += value_length) {
	builder.UnsafeAppend(offset);
	}
	return builder.Finish(out);
	}

where the value_length * (the number of repetitions) exceeds the int64 limit.

We could add a check for the overflow in the RepeatedArrayFactory for binary type:

arrow/cpp/src/arrow/array/util.cc

Lines 638 to 649 in fb8760e

	template <typename T>
	enable_if_base_binary<T, Status> Visit(const T&) {
	std::shared_ptr<Buffer> value =
	checked_cast<const typename TypeTraits<T>::ScalarType&>(scalar_).value;
	std::shared_ptr<Buffer> values_buffer, offsets_buffer;
	RETURN_NOT_OK(CreateBufferOf(value->data(), value->size(), &values_buffer));
	auto size = static_cast<typename T::offset_type>(value->size());
	RETURN_NOT_OK(CreateOffsetsBuffer(size, &offsets_buffer));
	out_ = std::make_shared<typename TypeTraits<T>::ArrayType>(length_, offsets_buffer,
	values_buffer);
	return Status::OK();
	}

using MultiplyWithOverflow, something similar to what we do here:

arrow/python/pyarrow/src/arrow/python/datetime.h

Lines 162 to 163 in fb8760e

	if (MultiplyWithOverflow(result, 1000000LL, &result)) {
	return Status::Invalid("Timedelta too large to fit in 64-bit integer");

[llama90]

@AlenkaF Hello.

I have taken an interest in this issue and upon review, I find some aspects to be ambiguous and would like to ask some questions.

It appears that in order to avoid overflow in this issue, the following two scenarios need to be considered:

1. Initially, is the repetition count(e.g., 2**31 -1) greater than int32_t or less than 0?
2. Does the product of the repeated value (e.g., ?) and repetition count trigger MultiplyWithOverflow?

Moreover, it seems possible to allocate the repetition count within the int64_t range, but when calling the c++ function from python code, is there a type conversion happening, leading the user to discover an error related to int32_t?

I wanted to test the cases exceeding int32_t, but I encountered an error like "mimalloc: warning: unable to allocate aligned OS memory directly, fall back to over-allocation (8598323200 bytes, address: 0x300800000, alignment: 67108864, commit: 1)" and was unable to proceed beyond this point.

Considering these points, should the range for repeat size be restricted to int32_t or should it be limited to int64_t?

[AlenkaF]

It appears that in order to avoid overflow in this issue, the following two scenarios need to be considered:

1. Initially, is the repetition count(e.g., 2**31 -1) greater than int32_t or less than 0?
2. Does the product of the repeated value (e.g., ?) and repetition count trigger MultiplyWithOverflow?

The main thing I would check is that value size multiplied with number of repetitions passes MultiplyWithOverflow when creating offsets buffer for the array with repeated values.

Moreover, it seems possible to allocate the repetition count within the int64_t range, but when calling the c++ function from python code, is there a type conversion happening, leading the user to discover an error related to int32_t?

I do not think there are any conversions happening. The int64_t range is the limit when creating the offset buffer and not a limit to the number of repetitions. Restricting the number of repetitions will not work, I think. For example if the string has a bigger value (length) then the limit for the number of repetitions will be even lower?

[rustyconover]

Hit this from a different angle in pyarrow 24.0.0 — small N, large scalar, where N * len(scalar) overflows int32 even though N itself is well below 2**31. Posting in case it's useful for the fix discussion (in particular: confirms @AlenkaF's read that the check belongs on value_size * N, not on N).

import struct, pyarrow as pa
N, SIZE = 2048, 1 << 20                  # 2 GiB total
arr = pa.repeat(pa.scalar("z" * SIZE), N)

# Surface API silently looks fine:
print(len(arr), arr.null_count)          # 2048 0
print(arr[0].as_py()[:4], arr[-1].as_py()[:4])  # zzzz zzzz

# Offsets buffer wraps:
print(struct.unpack("<3i", bytes(arr.buffers()[1])[-12:]))
# (2145386496, 2146435072, -2147483648)

arr.validate()
# pyarrow.lib.ArrowInvalid: Negative offsets in binary array

pa.RecordBatch.from_arrays([arr], names=["x"])
# pyarrow.lib.ArrowInvalid: In column 0: Invalid: Negative offsets in binary array

Threshold is exactly at value_size * N >= 2**31:

N	value_size	total	validate()
2047	1 MiB	2 146 435 072	✅
2048	1 MiB	2 147 483 648	❌

Two characteristics worth flagging:

1. Failure point is far from the cause. pa.repeat returns "successfully", and len() / __getitem__ work because the element accessor uses wider arithmetic. The corruption only surfaces at validate() (and any consumer that calls it — RecordBatch.from_arrays, IPC writers, etc.). In the wild this manifests as ArrowInvalid: In column 0: Invalid: Negative offsets in binary array from RecordBatch.from_arrays with no hint that pa.repeat was the actual culprit. Hit this in a real consumer: https://github.com/duckdb/duckdb-python/... (DuckDB→PyArrow conversion fixture in VGI).

2. An overflow-on-the-product check (per @AlenkaF's sketch in RepeatedArrayFactory) catches both this case and the original pa.repeat("?", 2**31) case — bounds-checking just N would miss this one.

Workaround for callers that know the data shape: build with pa.large_string() (int64 offsets) explicitly, e.g.

pa.array([scalar.as_py()] * N, type=pa.large_string())