[SYSTEMDS-3900] Stream aware write operation for OOC (option 2)

[j143]

..

I've chosen two pass method, write data separately and hold metadata in memory and concat at the end.

[j143]

Due to recompilation, this is visited the second time. Since the stream is null the second time, it fails.

So, I tried setting the values explicitly. But, recompilation still happening.

mo.updateDataCharacteristics(mc);
HDFSTool.copyFileOnHDFS(fname, mo.getFileName());
mo.setDirty(false);

[mboehm7]

I would recommend to simple probe if a stream exists and not consume the matrix if it does.

[j143]

Hi, I've not understood this part! :)

[j143]

Hi @mboehm7 , I would like to talk about the design choices here?

1. Since part files are not preferred. I choose two file format (one data file, one light metadata and concat in the end)

   a. file directory

UnaryTest/
in/
.X.crc
.X.mtd.crc
X
X.mtd
out/
.res._data.crc
.res._header.crc
.res.crc
.res.mtd.crc
res
res._data
res._header
res.mtd

b. In my machine, x86-64 Windows 11 - HDFS concat doesn't seem to work. I had winutils and other Junit tests work

2. Facing issue with recompilation step, due to which the stream = null and the if condition fails

3. If I try to explicitly set matrixCharacterstics to avoid recompilation - it still doesn't work.

---

could you let me know about the choices so far and which ones or better.

[mboehm7]

You don't have to invent any new things here: we take the stream of blocks and directly write them into a sequence file (like our existing binary writer, which actually takes the overall matrix and chunks them off into 1k-1k blocks). The resulting sequence files should be identical, and there is no file-system merge needed.

[j143]

Hi @mboehm7 , thanks for review and feedback on the design.

I've tested this on
int rows = 1000, cols = 1000;
5000, 1000
5000, 5000

[j143]

Hi @mboehm7 , could you please review this one. I've tried to address most of the comments. thank you.

I have the matrixvector multiplication task in progress, will shortly raise a PR built on the top of the write operation for testing.

[mboehm7]

LGTM - thanks for the patch @j143. The revised code looked already pretty good, I just moved the core stream write into the export logic, and fixed a few smaller issues.