S3/Dropbox directory synchronization/observing

[michaelrokosh]

When someone removes or inserts files into the remote storage (S3/Dropbox) outside of the meteor app it doesn't affect the FS.Collection.
Do you have any ideas about this? I'm not sure if there is an efficient way to do so.

E.g.

Meteor.startup(function () {
  FS.Store.Dropbox.sync({
    folder: 'photos'
  })
});

[rhyslbw]

Check this list for other issues relating to SA Sync:

https://github.com/CollectionFS/Meteor-CollectionFS/milestones/1.8.0%20Allow%20Storage%20Adapters%20to%20sync%20back%20into%20CFS%20

I guess we want to start with the SA interface API then for Dropbox could use https://www.dropbox.com/developers/webhooks/docs to be notified of the changes when the app is running, and on startup run a diff over the mappings.

Solution would require multi-instance system support (check worker changes). Similar issue to using collection observers in the existing fileworker package, and may need to be isolated to allow for a single instance to be listening for notifications to create jobs for the workers to then complete.

A syncing map could be passed into the constructor of the SA to encapsulate the logic?

[raix]

The sync is just like an upload it does the same thing: trigger storage of the file to the other SA's - We could also just copy the file to tempstorage and have the worker take it from there (the dropbox SA would already be completed and should not rerun) - a resync is basically the same except we are resetting all other copies of the file.

The big question is about the fileKey - we need some way of making filenaming more flexible - we should prop. do a hackpad on the naming subject alone, theres alot of aspects to it:

• security settings eg. having a home folder pr. user
• giving the user freedom to name files - making this work on all systems
• having an easy way of referencing files (collectionName, id [store name])

https://meteorcfs.hackpad.com/Paths-and-filenames-tqdiNfhnj8h please join in add concerns etc.

[michaelrokosh]

@rhyslbw, thanks. I was playing with Dropbox webhooks but I can't generate a proper sha256 key to validate the request. May be that's because JSON.stringify(this.request.body) is not the same as request.data JSON in their example.

if (Meteor.isServer) {
  crypto = Meteor.npmRequire('crypto');
  Router.route('/webhook', function () {
    var hmac;
    hmac = crypto.createHmac('sha256', Meteor.settings.dropboxSecretKey)
                 .update(JSON.stringify(this.request.body)).digest('hex');

    if (this.request.headers['x-dropbox-signature'] === hmac) {
      console.log('delta', this.request.body.delta);
      // retrieve files and create mondodb records
    }

    this.response.end(this.params.query.challenge);
  }, {where: 'server'});
}

[rhyslbw]

Might be an issue with Iron Router. What's expected to be in the data object as per the Dropbox docs?

Also there's an internal SHA package you might be able to use.

[raix]

@praxie if the code is part of cfs we should use http-methods or better just plain webapp / connect adding the hook - you could add the parseBody middleware on the connect handle for the route /hooks - we don't depend on iron router in cfs.

[michaelrokosh]

@rhyslbw, in their example request.data contains a json.
Example:

"{
    "delta": {
        "users": [
            12345678,
            23456789
        ]
    }
}"

[rhyslbw]

Did you get any further with this @praxie ?