Fileworker should observe temp store

[aldeed]

Fileworker observe can be changed to watch the temp store's collection. When combined chunk sizes for a file equals the file size, kick off storage. This is more efficient than watching all FS.Collections.

[aldeed]

Also, the temp store collection (and temp files) is likely to accumulate junk documents (chunks that were uploaded and stored but we never received enough chunks to move on to the storage phase). We can add the upload date to this collection and then have the fileworker delete chunks that were uploaded more than N minutes ago.

[raix]

Good idea, maybe configurable via FS.Config({});

[aldeed]

@raix, your changes to tempstore mean that it's not doing any tracking in storeCollection now. That means we're stuck with having the worker watch all of the FS.Collections. Maybe add back minimal tracking in storeCollection in the temp store, just so that we have something much smaller for the worker to observe? I think it could still be an in-memory minimongo collection that we rebuild on restart.

[raix]

True I did remove a collection in tempStore... The worker could have its own collection of tasks?
The tempstore adds a task to the worker when file is uploaded - and gets called when temp file is no longer needed - this would work really great with later sync.

[raix]

yep I did consider the in memory db and reloading on startup - but I removed all the code again, thinking it was too much db.

[raix]

(It was a bit sad - I did actually create a buffer 1/8 size of chunks - and each bit represented a chunk... It wrote changes to a track.db in the file folder and loaded it again on startup - it wasn't observable so really a beautiful wast of time. Well I half'ed the code by removing it again)

[raix]

Hmm, I can see that I went the opposite direction of the title of this issue - even though we have been discussing this - I'm sorry about it - just feels better this way - thinking about it the temp store resembles a SA very much - chunk'a stream. It could actually emit an upload event?

[aldeed]

OK, I guess any way of communicating between temp store and worker is fine with me. The worker is just saying "tell me when there's new file data for a file" and we need to tell it one way or another. Like you said, worker could perhaps expose an addTask method, and if tempstore doesn't see a worker loaded, then instead stream direct to store (potentially through a before-save transform).

So the flow would be:

TempStore (last chunk received) --------------------------------------------
                 |                                                         |
                 |                                                         |
                 V                                                         |
add FileWorker task (optional)                                             |
                 |                                                         | (no worker found)
                 |                                                         |
                 V                                                         |
(per store) merge chunk streams and (optionally) pass through transforms <--
                 |
                 |
                 V
stream to store

[raix]

Yep, i guess we would support transforms blindly, we either send to worker or direct. Not sure direct is scaleable, but having the plugin mechanism is important.

[raix]

Closing this, its tracked in new issue