Unable to read file from location: asset meta on s3

[goellner]

Bug description

Getting hit with a lot of Unable to read file from location: media/1/.meta/xxx.mp4.yaml errors.

When we upload images directly to S3 Statamic does not generate the .meta files, which is to be expected. Would be good if Statamic tries to create the .meta files if they don't exist instead of throwing an error and making the asset browser unusable.

Also php please assets:meta does not work, throws the same error

How to reproduce

Setup S3 asset container
upload image directly
open asset browser and enter folder where a new asset has been uploaded

Logs

No response

Environment

Environment
Application Name: SWAG SWAG SWAG
Laravel Version: 10.20.0
PHP Version: 8.1.22
Composer Version: 2.5.8
Environment: production
Debug Mode: ENABLED
URL: app.asdf.com
Maintenance Mode: OFF

Cache
Config: NOT CACHED
Events: NOT CACHED
Routes: NOT CACHED
Views: CACHED

Drivers
Broadcasting: log
Cache: redis
Database: mysql
Logs: prod / mezmo, flare
Mail: smtp
Queue: redis
Scout: meilisearch
Session: redis

Statamic
Addons: 3
Antlers: runtime
Stache Watcher: Enabled
Static Caching: Disabled
Version: 4.19.0 PRO

Statamic Addons
aerni/advanced-seo: 2.1.4
aryehraber/statamic-uuid: 2.2.0
statamic/eloquent-driver: 2.6.1

Statamic Eloquent Driver
Asset Containers: file
Assets: eloquent
Blueprints: file
Collection Trees: file
Collections: eloquent
Entries: eloquent
Forms: eloquent
Global Sets: file
Global Variables: file
Navigation Trees: file
Navigations: eloquent
Revisions: eloquent
Taxonomies: eloquent
Terms: eloquent

Installation

Fresh statamic/statamic site via CLI

Antlers Parser

None

Additional details

No response

[duncanmcclean]

This might have the same cause as #7350, which is thrown when using Glide.

[FrankPeters]

We are using an SFTP driver using the official Spatie's package and we are also getting these .meta not found errors quite frequently. I thought it was related to the visibility not being public in order for the folders to be created with the correct rights, but that does not seem to solve the problem.

Example of our config:

      'assets_main' => [
          'driver' => 'sftp',
          'host' => '[...]',
          'username' => '*',
          'privateKey' => '/home/forge/[...]/sftp_ssh_key',
          'root' => '[...]/public/assets/main',
          'visibility' => 'public',
          'throw' => true,
          'url' => '[...]/assets/main',
      ],

Maybe unrelated, but we also notice that there's a huge delay after uploading a file that ends with the file being uploaded but the .meta error being triggered (sometimes on a different file that was already on the file system but misses the .meta file), so the user never gets to see a success message.

[FrankPeters]

It seemed to be caused by the 'throw' => true,. After disabling that, the issue went away.

[ryanmitchell]

@goellner are you still seeing this issue? If so could you provide a full stack trace?

[goellner]

@ryanmitchell sorry, didn't see the replys. I think it was, because I manually uploaded assets to the S3 bucket. Not sure if this still happens, but Statamic didn't try to create the metadata files, it just threw an error

[goellner]

@ryanmitchell just running into this again. I am on 4.58.2 now and every asset I upload to S3 breaks the asset viewer. Have to manually create a corresponding yaml file. The asset is uploaded. I saw that there are a some fixes for meta on the v5 branch, just can't update yet because of some dependencies

[goellner]

if this could be related, i have a custom root folder in filesystems.php

AWS_ACCESS_KEY_ID=xxx
AWS_SECRET_ACCESS_KEY=xxx
AWS_DEFAULT_REGION=eu-central-1
AWS_BUCKET=xxx
AWS_USE_PATH_STYLE_ENDPOINT=false
AWS_URL=https://assets.example.com

'cms' => [
    'driver' => 's3',
    'key' => env('AWS_ACCESS_KEY_ID'),
    'secret' => env('AWS_SECRET_ACCESS_KEY'),
    'region' => env('AWS_DEFAULT_REGION'),
    'bucket' => env('AWS_BUCKET'),
    'url' => env('AWS_URL'),
    'endpoint' => env('AWS_ENDPOINT'),
    'use_path_style_endpoint' => env('AWS_USE_PATH_STYLE_ENDPOINT', false),
    'throw' => true,
    'root' => 'cms',
],

[stuartcusackie]

This problem keeps resurfacing for me too. Major pain. Statamic can't seem to handle certain images that my client downloads from stock photo websites. Once one gets uploaded it breaks the whole asset browser and it takes quite a while to figure out.

My last workaround was to manually create the .meta file and upload it via digital oceans browser.

[ryanmitchell]

@stuartcusackie could it be a file size issue? that over a certain file size the meta isn't being created?

[duncanmcclean]

@stuartcusackie Out of curiosity, I just uploaded the images you previously linked and I didn't get any errors. I'm also using an S3 filesystem. 🤔

The largest file in the zip was 5.1MB.

[stuartcusackie]

@ryanmitchell Yes, I think you are correct. This is very strange, here's what happens:

• Upload a large image (15.8mb)
• Error occurs: Unable to read file from location: .meta/img_20200527_182048.jpg.yaml
• The meta file does not exist on Digital ocean
• The error keeps occurring on the Statamic asset browser.
• Deleting the file through DO control panel doesn't solve the problem
• Clearing the stache doesn't solve the problem
• The only thing that works is manually creating the .yaml file and uploading it through the DO interface

[ryanmitchell]

Is it to do with the max_upload_size and post_max_size you have specified in PHP (1.7mb seems suspiciously close to the 2MB default)

[stuartcusackie]

@duncanmcclean

I wonder if you try this file:
https://limewire.com/d/IM44l#HQ4XtE9ew0

[stuartcusackie]

@ryanmitchell

I think it is to do with filesize. I could increase the server limit but I think Statamic should handle this one better and reject the file, or else be able to detect bad / partially uploaded files somehow and skip / remove them.

Weird though, because my forge server has a 32mb upload limit.