Unhandled stream error in pipe. production

[for2ne]

Hi!

Often there is an error on the server and the server crashes:

error: Script restart attempt stream.js:94 throw er; // Unhandled stream error in pipe. ^ Error: ENOENT, open '/opt/bling/cfs/files/_tempstore/propertyImages-ACsFQLq38qsy97oL8-0.chunk' error: Forever detected script exited with code: 8

so the problem is solved for a while, but then again there:
db['cfs._tempstore.chunks'].drop()

I also often see such errors:
SA FileSystem: Could not delete /opt/bling/cfs/files/proListing/propertyImages-ddfa3HPTm2bqvtHAT-bling.jpg because the file was not found.

Why is this happening?
all code by adding images from the official documentation

Whether it can occur if the file is not downloaded a huge until the end? and how to fight it?

Meteor list:
...
aldeed:collection2 2.3.3 Automatic validation of insert and update operations on the client and server.
cfs:filesystem 0.1.2 Filesystem storage adapter for CollectionFS
cfs:graphicsmagick 0.0.18 Addsgmto scope with the ability to perform GraphicsMagick or ImageMagick manipulation
cfs:standard-packages 0.5.7 Filesystem for Meteor, collectionFS
...

[rhyslbw]

Are you running just a single app instance?

[for2ne]

@rhyslbw yes. I understand a little bit when an error occurs specifically. I upload images and resize them in transformWrite function. I've seen this error when using the function transformWrite for incorrect images (text/plain instead of image/*)

once it was so:
(server crash on wrong file. restart and crash again loop until you do db['cfs._tempstore.chunks'].drop() on mongo shell )

transformWrite: function(fileObj, readStream, writeStream) {
    gm(readStream, fileObj.name()).resize('600', '400').stream().pipe(writeStream);
 },

now:
(now server crash on wrong file, but not crash after restart)

transformWrite: function(fileObj, readStream, writeStream)
      {
        var Fiber = Npm.require('fibers');
        Fiber(function() {
          var magic = Meteor.npmRequire('stream-mmmagic');

          magic(readStream, Meteor.bindEnvironment(function (err, mime, output)
          {
            if ((mime.type === 'image/jpeg') || (mime.type === 'image/png'))
            {

              gm(output, fileObj.name()).autoOrient().quality(80).resize('600', '400').stream().pipe(writeStream);
            }
            else
            {
              // Error occurs here. I do not know how to delete the wrong file in another way
              fileObj.remove(); 

              output.pipe(writeStream);
            }

          }));

        }).run();
      },

[mxab]

@for2ne thanks for the tip with the db['cfs._tempstore.chunks'].drop() it solved my issue, I imported a database dump and after I started to insert more files I also got the same error, I guess the fileworked was then looking for temp files to import which were not available on the imported system

[rhyslbw]

I'll close this as it's currently not possible to run multiple app instances. See #667

[mxab]

@rhyslbw I wasn't running multiple app instances. My setup is one app - one db (replica set, hosted on compose.io)

[mxab]

but if I guess dumping/restoring the db to another instance+db has more or less the same effect.

[rhyslbw]

Thanks for confirming, but yes with the current released version cfs._tempstore.chunks takes the role of a queue. My changes are all about separating responsibilities, of which covers this scenario too

[steve-ross]

This is how I handled transforms incase the wrong file type made it up to the server:

var resizeAndCropImageHorizontal = function(fileObj, readStream, writeStream, width, height){
    var g = gm(readStream);
    // NOTE: this loads the file into memory... might be an issue if there is high load or massive images
    g.identify({ bufferStream: true }, Meteor.bindEnvironment(function (err, data) {
        if (err) {
          console.warn('GraphicsMagick resizeAndCropImageHorizontal() failed to identify file', fileObj);
          fileObj.remove();
        } else {
          g.autoOrient().resize(width, width + '>').gravity('Center').crop(width, height).stream('PNG').pipe(writeStream);
        }
      })
    );
};

[bluepuma77]

I have the same issue. Two apps using one DB and file system. But one is for uploading, the other one is just serving the images.

Is there a way to get a try-catch block around this? It's kind of annoying that a single-problem-file crashes the whole app.

[jackkav]

@bluepuma77 I would strongly advise against using this in your scenario. It would appear that more than on app instance or multiple apps accessing the same file store just wasn't considered in the design. Which is shame because it is a really great idea to have multiple file store endpoints.