DPL Analysis: avoid Arrow's Slice API by aalkin · Pull Request #15613 · AliceO2Group/AliceO2

aalkin · 2026-07-20T08:06:24Z

Slicing arrow tables is a very common operation within Analysis Framework and currently it goes through the standard Arrow API that involves calling the Slice method on every ChunkedArray backing a column in a given table. This involves allocating internal buffers for the new instances of the ChunkedArray and, while it is not a heavy operation by itself, it is performed thousands of times, or even hundreds of thousands of times in case of derived data. This change replaces usage of Arrow's API with an internal representation that consists of a shared pointer to an Arrow table and a range defined by its starting offset and size. Slicing operation only changes the range, and all the relevant table handling code was updated to take into account externally imposed offset and size.

alibuild · 2026-07-20T08:30:36Z

Error while checking build/O2/fullCI_slc9 for c845be7 at 2026-07-20 10:30:

## sw/BUILD/O2-latest/log
/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/Framework/Core/include/Framework/Concepts.h:20:11: error: 'uint32_t' has not been declared
/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/Framework/Core/include/Framework/Concepts.h:200:141: error: template argument 1 is invalid
/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/Framework/Core/include/Framework/Concepts.h:200:48: error: template argument 1 is invalid
/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/Framework/Core/include/Framework/Concepts.h:201:92: error: template argument 1 is invalid
/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/Framework/Core/include/Framework/Concepts.h:201:94: error: '<declaration error>' is not a template [-fpermissive]
/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/Framework/Core/include/Framework/Concepts.h:241:92: error: template argument 1 is invalid
/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/Framework/Core/include/Framework/Concepts.h:241:30: error: template argument 1 is invalid
/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/Framework/Core/include/Framework/Concepts.h:241:176: error: template argument 1 is invalid
/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/Framework/Core/include/Framework/Concepts.h:241:104: error: template argument 1 is invalid
/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/Framework/Core/include/Framework/Concepts.h:253:122: error: template argument 1 is invalid
/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/Framework/Core/include/Framework/Concepts.h:253:29: error: template argument 1 is invalid
/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/Framework/Core/include/Framework/Concepts.h:253:237: error: template argument 1 is invalid
/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/Framework/Core/include/Framework/Concepts.h:253:239: error: '<declaration error>' is not a template [-fpermissive]
/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/Framework/Core/include/Framework/Concepts.h:253:134: error: template argument 1 is invalid
/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/Framework/Core/include/Framework/Concepts.h:257:134: error: template argument 1 is invalid
/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/Framework/Core/include/Framework/Concepts.h:257:41: error: template argument 1 is invalid
/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/Framework/Core/include/Framework/Concepts.h:257:262: error: template argument 1 is invalid
/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/Framework/Core/include/Framework/Concepts.h:257:264: error: '<declaration error>' is not a template [-fpermissive]
/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/Framework/Core/include/Framework/Concepts.h:257:146: error: template argument 1 is invalid
/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/Framework/Core/include/Framework/ASoA.h:280:126: error: template argument 1 is invalid
/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/Framework/Core/include/Framework/ASoA.h:280:31: error: expected primary-expression before '{' token
/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/Framework/Core/include/Framework/ASoA.h:280:31: error: expected ';' before '{' token
/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/Framework/Core/include/Framework/ASoA.h:280:130: error: '<expression error>::metadata' has not been declared
/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/Framework/Core/include/Framework/ASoA.h:280:169: error: expected primary-expression before '>' token
/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/Framework/Core/include/Framework/ASoA.h:280:171: error: expected primary-expression before ')' token
/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/Framework/Core/include/Framework/ASoA.h:280:179: error: expected ';' before '...' token
/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/Framework/Core/include/Framework/ASoA.h:281:4: error: parameter packs not expanded with '...':
/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/Framework/Core/include/Framework/ASoA.h:1263:26: error: '<expression error>' in namespace 'o2::aod' does not name a type
/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/Framework/Core/include/Framework/ASoA.h:1264:69: error: 'H' was not declared in this scope
/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/Framework/Core/include/Framework/ASoA.h:1264:103: error: template argument 1 is invalid
/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/Framework/Core/include/Framework/ASoA.h:1264:103: error: template argument 2 is invalid
/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/Framework/Core/include/Framework/ASoA.h:1307:116: error: template argument 1 is invalid
/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/Framework/Core/include/Framework/ASoA.h:1307:118: error: '<declaration error>' is not a template [-fpermissive]
/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/Framework/Core/include/Framework/ASoA.h:1307:128: error: expected template-argument before '::' token
/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/Framework/Core/include/Framework/ASoA.h:1307:128: error: expected '>' before '::' token
/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/Framework/Core/include/Framework/ASoA.h:1327:82: error: template argument 1 is invalid
/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/Framework/Core/include/Framework/ASoA.h:1327:84: error: '<declaration error>' is not a template [-fpermissive]
/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/Framework/Core/include/Framework/ASoA.h:1327:86: error: '<expression error>::metadata' has not been declared
/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/Framework/Core/include/Framework/ASoA.h:1333:82: error: template argument 1 is invalid
/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/Framework/Core/include/Framework/ASoA.h:1333:84: error: '<declaration error>' is not a template [-fpermissive]
/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/Framework/Core/include/Framework/ASoA.h:1333:86: error: '<expression error>::metadata' has not been declared
/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/Framework/Core/include/Framework/ASoA.h:1674:98: error: template argument 1 is invalid
/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/Framework/Core/include/Framework/ASoA.h:1674:39: error: template argument 1 is invalid
/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/Framework/Core/include/Framework/ASoA.h:1677:65: error: template argument 1 is invalid
/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/Framework/Core/include/Framework/ASoA.h:1677:67: error: '<declaration error>' is not a template [-fpermissive]
/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/Framework/Core/include/Framework/ASoA.h:1677:69: error: '<expression error>::metadata' has not been declared
/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/Framework/Core/include/Framework/ASoA.h:2192:100: error: template argument 1 is invalid
/sw/SOURCES/O2/slc9_x86-64-slc9_x86-64/0/Framework/Core/include/Framework/ASoA.h:2192:100: error: template argument 2 is invalid
[0 more errors; see full log]

Full log here.

alibuild · 2026-07-20T08:39:57Z

Error while checking build/O2/fullCI_slc9 for dfa07ab at 2026-07-20 10:39:

## sw/BUILD/O2-latest/log
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.

Full log here.

Co-authored-by: Anton Alkin <anton.alkin@cern.ch>

alibuild · 2026-07-20T11:38:23Z

Error while checking build/O2/fullCI_slc9 for 6b31c6e at 2026-07-20 18:32:

## sw/BUILD/O2Physics-latest/log
/sw/SOURCES/O2Physics/slc9_x86-64-slc9_x86-64/0/EventFiltering/PWGUD/diffractionBCFilter.cxx:351:90: error: no matching function for call to 'o2::soa::Join<o2::soa::Table<o2::aod::Hash<15601895>, o2::aod::Hash<4180572075>, o2::aod::Hash<2286545062> >, o2::soa::Table<o2::aod::Hash<1815461783>, o2::aod::Hash<197928243>, o2::aod::Hash<2286545062> > >::Join(<brace-enclosed initializer list>)'
ninja: build stopped: subcommand failed.


## sw/BUILD/o2checkcode-latest/log
--
========== List of errors found ==========
++ GRERR=0
++ grep -v clang-diagnostic-error error-log.txt
++ grep ' error:'
grep: error-log.txt: binary file matches
++ GRERR=1
++ [[ 1 == 0 ]]
++ mkdir -p /sw/INSTALLROOT/fbee9dc089f6b1edadf90aa6ea5f9a393d974c26/slc9_x86-64/o2checkcode/1.0-local294/etc/modulefiles
++ alibuild-generate-module
+ HAS_BIN=
+ HAS_LIB=
+ HAS_CMAKE=
+ HAS_ROOT=
+ '[' 0 '!=' 0 ']'
+ cat
+ printf 'if ![ is-loaded '\''BASE/1.0'\'' ] {\n module load BASE/1.0\n}'
+ echo 'BUILD_REQUIRES=CMake alibuild-recipe-tools defaults-release'
BUILD_REQUIRES=CMake alibuild-recipe-tools defaults-release
+ FULL_BUILD_REQUIRES='MPFR O2-customization ninja-fortran defaults-release date googlebenchmark UUID safe_int Alice-GRID-Utils double-conversion alibuild-recipe-tools ninja json-c pytorch_cpuinfo bz2 Python-modules-list FairCMakeModules CMake'
+ echo 'FULL_BUILD_REQUIRES=MPFR O2-customization ninja-fortran defaults-release date googlebenchmark UUID safe_int Alice-GRID-Utils double-conversion alibuild-recipe-tools ninja json-c pytorch_cpuinfo bz2 Python-modules-list FairCMakeModules CMake'
FULL_BUILD_REQUIRES=MPFR O2-customization ninja-fortran defaults-release date googlebenchmark UUID safe_int Alice-GRID-Utils double-conversion alibuild-recipe-tools ninja json-c pytorch_cpuinfo bz2 Python-modules-list FairCMakeModules CMake
++ env
++ cut -f1 -d=
++ grep -v '^DEFAULT_'
++ grep -v PKGREVISION
++ grep -v ALIBUILD_RECIPE_TOOLS
++ grep REVISION
++ sed -e s/_REVISION//
+ for x in $(env | cut -f1 -d= | grep -v "^DEFAULT_" | grep -v PKGREVISION | grep -v ALIBUILD_RECIPE_TOOLS | grep REVISION | sed -e 's/_REVISION//')
++ eval 'echo $LIBFFI_REVISION'
+++ echo 10
+ REVISION_VALUE=10
++ eval 'echo $LIBFFI_VERSION'
+++ echo v3.2.1-alice1
+ VERSION_VALUE=v3.2.1-alice1
++ eval 'echo $LIBFFI_ROOT'
+++ echo /sw/slc9_x86-64/libffi/v3.2.1-alice1-10
+ ROOT_PATH_VALUE=/sw/slc9_x86-64/libffi/v3.2.1-alice1-10
+ '[' -z 10 ']'
+ echo 'MPFR O2-customization ninja-fortran defaults-release date googlebenchmark UUID safe_int Alice-GRID-Utils double-conversion alibuild-recipe-tools ninja json-c pytorch_cpuinfo bz2 Python-modules-list FairCMakeModules CMake'
+ tr '[:lower:]' '[:upper:]'
+ tr - _
[0 more errors; see full log]

Full log here.

Co-authored-by: Anton Alkin <anton.alkin@cern.ch>

aalkin added 21 commits July 13, 2026 09:49

initial

64f7f35

move concepts out of ASoA.h

ba2666a

fixup! move concepts out of ASoA.h

b4a5254

move concepts out of AnalysisHelpers.h

8a5b495

address FIXMEs

c37f9ac

fixup! address FIXMEs

6fb94f9

start

a81daa3

use table ref for soa::Table

08e66fd

minor fixes

c0b5e59

join/concat using ranges

6e8bcc6

cover more cases

066edd8

update join

ea105c7

rework constructors and operators

948ac94

start adapting slicing

0204587

use imposed size

c0883a7

make it compile

9740d9a

make it pass tests

bd7f342

make O2Physics compile

3d7ced3

minor fix

0b9f6bd

bug with filtered slice

d8f5fa0

add test

c845be7

aalkin requested a review from a team as a code owner July 20, 2026 08:06

alibuild mentioned this pull request Jul 20, 2026

Please consider the following formatting changes to #15613 aalkin/AliceO2#146

Merged