[1.2.0] Update the cfs-worker

[raix]

The file worker makes sure that we are not storing all uploaded files to all stores at once.

• We do this by tracking the uploaded files and queue them up in tasks - one task pr. storage adapter.
• We throttle the worker queue and failed storage attempts are retried later.
• If a storage adapter wants to synchronize its data with the rest of the storage adapters on a file it simply store the file into Temp storage and the worker will get notified - This action does not involve conflict resolution.

Pull structure https://github.com/CollectionFS/Meteor-cfs-gridfs/issues/4#issuecomment-37139912
WIP https://github.com/CollectionFS/DEPRECATING-cfs-worker/blob/pull-arc/fileWorker.js

Tasks:

• The temp store can loose the file - We have to accept that as a worker and cancel the task - FS.Tempstore.createWriteStream cannot access temporary filesystem #209
• The file worker should observe or listen to upload events on the Tempstore Fileworker should observe temp store #186
• The transformWrite/transformRead should run async New CFS: allow beforeSave to run async #156
• Use power-queue to handle tasks New CFS: Use PowerQueue for server file worker #126
• Maybe have an internal worker collection for tracking tasks - could persist a spinal queue for power-queue
• Worker should be optional
• Worker should be able to be installed on a separate server and use one or more separate server processes. It should actually probably be a Meteor app as opposed to a Meteor package. (Could potentially even be a nodejs app, but I don't think it matters much.)
• Worker should have queue/throttle options, and maybe a simple admin UI page to monitor and adjust (which would go along with making it an app.)
• Does not run within the Meteor app process.
• Connects to the same MONGO_URL
• Safe to run multiple instances of the worker app at once for scaling
• Assume the temp store is using GridFS. Could add support for any accessible SA in the future, such as S3.
• All dependencies rigged in a cfs imply package (no inter-package dependencies)
• cfs:worker does not observe any fsCollections directly, it only interacts with the queue
• ability to scale individual SA store copies
• control over the job-collection cleanup schedule, default to remove on completion?

[aldeed]

Since we want to support both worker and no worker, maybe it's better to handle the logic directly in the temp store (instead of listening to upload event)?

We can have logic in temp store when upload is done:

If worker pkg added, call FS.FileWorker.addFile(fileObj).

Else call fileObj.createReadStream().pipe(fileObj.createWriteStream(storeName)) for each defined store.

[raix]

I was thing to have the collectionFS package have a small code stamp to hook up the worker to the tempStore events - We could have it route directly to SA if no worker package found?
I'll push some code in a moment - hope it makes more sense to have the tempstore simply emit events - not caring about who is listening.

[nkraich]

Apologies if I don't quite understand the bigger picture here; I'll do my best!

From the above, it looks like it might now be possible to disable workers (which seem to be the cause of an error for me on meteor.com deployments.) I tried deleting the cfs-worker package from my app, but I got a bunch of "error building package" messages from the various cfs packages when I started Meteor.

I tried removing cfs-worker from the collectionFS package.js, and Meteor did start after removing it in that way, although files will not upload. It doesn't give any errors, but creates a file that links to /null.

Is it currently possible to disable the cfs-worker package? Sorry if I jumped the gun or misunderstood anything; just thought I'd give it a try.

For reference, my issue was #209, the temp store "losing the file".

[aldeed]

No, you currently need the worker, but we plan to make it optional. @raix, is rewriting the worker and related code within the next week, I think, and after that, you should be able to remove that package.

[nkraich]

OK, thank you!

[raix]

I think your problem is related to missing rights to the filesystem (os temp folder) perhaps?

The concept of fileWorkers is:

1. Throttling cpu / io
2. scalablility - you could have some transform streams that could be heavy eg. data analyze or something - so we want to be able to handle uploaded files even if we are on multiple Meteor instances

@aldeed I'm on this - but I don't think it'll make it into the 1.0.0
Maybe we could:

1. have cfs work without file workers pr. default
2. have a minor release that allows TempStore to use any SA - e.g. GridFS, allowing scaling and systems where os temp folder is not accessible
3. have a minor release that complete the fileWorker for scalability - I like to have some tests and benchmarks on this

Its not api changing stuff - just changes the way cfs works internally

[aldeed]

Sounds good. If we at least remove the worker dependency for 1.0.0, that's fine with me. I might even be able to do that this weekend if you don't get to it.

[nkraich]

This would be a big help for me if you go in this direction. My assumption is that this would allow me to use CollectionFS on meteor.com again, which is a lot more important to me than scalability (right now, anyway).

Regardless, thank you very much for the explanation.