[SYSTEMDS-3900] Stream aware write operation for OOC

[j143]

• The existing CP WriteInstruction (VariableCPInstruction) was modified to be "stream-aware."

• It now checks if its input MatrixObject has an OOC stream handle. If a stream exists, it acts as a synchronous terminal consumer, reading blocks from the stream and writing them to separate part-files in an output directory.

• Multi-Block Write: The OOC write logic was made robust to handle multi-block matrices by writing to separate part-files, which is the standard for distributed systems.

[j143]

Hi @mboehm7 , could you please check if this direction is ok! (implementation is not yet complete)

1. using the existing write itself
2. using matrix partition files in the directory

[mboehm7]

Sorry for the delay, and thanks for getting started on this task @j143.

1. Integration: Instead of integrating this write into the VariableCPInstruction (where non-binary formats are written), I would recommend to integrate the OOC write into the individual writers (with support for only binary) with a new method which is called if an MatrixObject has indeed an existing OOC stream of blocks.

2. Core Write Logic: In order to yield the same output files as a normal (single-threaded) write, I recommend to not create part files for every single block, but stream all these blocks into a single file. Once you extend the binary write you see that this approach is even easier and result in files that can be processed much faster (not too many files which can be an issue on distributed file systems).