previewgenerator doesn't scan all folders

[Marco-Total]

Hi,

I have two questions.

When I get the command

sudo -u www-data /var/www/nextcloud/ occ preview:create-all "myuserid"

only the first 83 folders are scanned. In the main directory, however, there are many other subdirectories?

When I get the command

sudo -u www-data /var/www/nextcloud/ occ preview: generate-all

without specifying a userid execution, the scan is performed only for one user. Is this a desired behavior?

Thanks!
Marco

[krypton9208]

I have the same issue.
I run
sudo -u www-data php /var/www/nextcloud/occ preview:generate-all
and scanned was only first few folders, even not all of admin account and no one of other users.

[Marco-Total]

Hi,

I have solved my problem. The Preview Generator needs a few times to scan realy all folders at first full scan...i have generate a cronjob for the first days:
Sudo -u www-daata crontab -e

• 23 * * * /var/www/nextcloud/occ preview:generate-all >> /var/www/nextcloud/log/thumbnail.log

Cheers, Marco

[rullzer]

Do you actually get errors when scanning? Because we just traverse the whole tree. Also running it multiple times sounds weird. As we should get the same list back each time.

Long story short. What is the output of the command?

I have the same problem.

It looks like this

Scanning folder /maalik/files
Scanning folder /maalik/files/Auto
...
Scanning folder /maalik/files/***/***/MR

DONE

The folder MR contained 3 images, one had 45MB. After removing the folder, the task continued a little bit longer

Scanning folder /maalik/files
Scanning folder /maalik/files/Auto
...
Scanning folder /maalik/files/***/***/MR
...
libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile
Scanning folder /maalik/files/***/***/***/***/***

DONE

After that, it finishes. But there a lot more folders to do. Doesn't matter, how many times i restart the command, it always finishes at the same point.

[matt01]

I just had the same issue, running a generate-all would stop on the same directory every time I ran it.

The directory is nothing special in my case but I did notice the following error in my nextcloud.log ...

{"reqId":"ruqim6GgDk4jf6ReWyxq","level":3,"time":"2018-01-22T07:37:54+00:00","remoteAddr":"","user":"--","app":"PHP","method":"--","url":"--","message":"Allowed memory size of 134217728 bytes exhausted (tried to allocate 28672 bytes) at \/var\/www\/localhost\/htdocs\/lib\/private\/legacy\/image.php#566","userAgent":"--","version":"12.0.3.3"}

I'm running gentoo, which has different php.ini files for cgi, cli and fpm and the cgi and fpm php.ini files had been set to reasonable values but the cli ini file was at the default 128M.

Changing the ini file and re running the generate-all command has now progressed further then before.

My log shows the same:
Allowed memory size of 134217728 bytes exhausted (tried to allocate 24576 bytes) at /volume1/web/nextcloud/lib/private/legacy/image.php#585

@matt01 I have a Synology DS918+. Can you try to describe me, what value I have to put higher? Already set memory_limit to 256M but that didn't work.
Nevermind, I changed it for PHP5.6 while the command was running under PHP7
Works fine now!

I am also experiencing the same issue.
Mine appears to jump in leaps and bounds (if pre-generating) and as of late, one file each minute (I execute each minute while trying to troubleshoot the issue).

I was running an encrypted instance so I downloaded all the files via the sync tool and then deleted the old instance, created a new one and then kicked off the preview generation.

Is there anyway to get an advanced log?

Below is the output from a pre-generate. A generate all never moves past the first file it scans.

2018-02-19T22:52:01+00:00 Generating previews for //files/InstantUpload/Camera/20170402_144128.jpg
2018-02-19T22:54:01+00:00 Generating previews for //files/InstantUpload/Camera/20170408_203527.jpg
2018-02-19T22:55:02+00:00 Generating previews for //files/InstantUpload/Camera/20170424_215341.jpg
2018-02-19T22:56:02+00:00 Generating previews for //files/InstantUpload/Camera/20170427_183050.jpg
2018-02-19T22:57:01+00:00 Generating previews for //files/InstantUpload/Camera/20170507_190634.jpg
2018-02-19T22:58:01+00:00 Generating previews for //files/InstantUpload/Camera/20170513_165847.jpg
2018-02-19T22:59:02+00:00 Generating previews for //files/InstantUpload/Camera/20170514_134256.jpg
2018-02-19T23:00:02+00:00 Generating previews for //files/InstantUpload/Camera/20170514_201006.jpg
2018-02-19T23:01:02+00:00 Generating previews for //files/InstantUpload/Camera/20170517_125120.jpg
2018-02-19T23:02:02+00:00 Generating previews for //files/InstantUpload/Camera/20170515_203726.jpg
2018-02-19T23:03:02+00:00 Generating previews for //files/InstantUpload/Camera/20170520_040533.jpg
2018-02-19T23:04:02+00:00 Generating previews for //files/InstantUpload/Camera/20170525_134437.jpg
2018-02-19T23:05:02+00:00 Generating previews for //files/InstantUpload/Camera/20170525_134453.jpg
2018-02-19T23:06:02+00:00 Generating previews for //files/InstantUpload/Camera/20170525_134458.jpg
2018-02-19T23:07:02+00:00 Generating previews for //files/InstantUpload/Camera/20170525_174706.jpg
2018-02-19T23:08:02+00:00 Generating previews for //files/InstantUpload/Camera/20170527_174449.jpg
2018-02-19T23:09:01+00:00 Generating previews for //files/InstantUpload/Camera/20170528_123749.jpg
2018-02-19T23:10:02+00:00 Generating previews for //files/InstantUpload/Camera/20170528_134532.jpg
2018-02-19T23:11:02+00:00 Generating previews for //files/InstantUpload/Camera/20170528_144107.jpg
2018-02-19T23:12:01+00:00 Generating previews for //files/InstantUpload/Camera/20170528_144111.jpg
2018-02-19T23:13:01+00:00 Generating previews for //files/InstantUpload/Camera/20170528_144309.jpg
2018-02-19T23:14:02+00:00 Generating previews for //files/InstantUpload/Camera/20170528_144311.jpg
2018-02-19T23:15:01+00:00 Generating previews for //files/InstantUpload/Camera/20170528_144331.jpg
2018-02-19T23:16:01+00:00 Generating previews for //files/InstantUpload/Camera/20170528_144726.jpg
2018-02-19T23:17:02+00:00 Generating previews for //files/InstantUpload/Camera/20170528_163809.jpg
2018-02-19T23:18:02+00:00 Generating previews for //files/InstantUpload/Camera/20170531_183805.jpg
2018-02-19T23:19:01+00:00 Generating previews for //files/InstantUpload/Camera/20170528_163813.jpg
2018-02-19T23:20:02+00:00 Generating previews for //files/InstantUpload/Camera/20170531_183816.jpg
2018-02-19T23:21:02+00:00 Generating previews for //files/InstantUpload/Camera/20170531_183846.jpg
2018-02-19T23:22:01+00:00 Generating previews for //files/InstantUpload/Camera/20170531_183854.jpg
2018-02-19T23:23:01+00:00 Generating previews for //files/InstantUpload/Camera/20170531_183905.jpg
2018-02-19T23:24:02+00:00 Generating previews for //files/InstantUpload/Camera/20170531_183911.jpg
2018-02-19T23:25:01+00:00 Generating previews for //files/InstantUpload/Camera/20170601_162629.jpg
2018-02-19T23:26:01+00:00 Generating previews for //files/InstantUpload/Camera/20170609_075343.jpg
2018-02-19T23:27:02+00:00 Generating previews for //files/InstantUpload/Camera/20170607_184201.jpg
2018-02-19T23:28:02+00:00 Generating previews for //files/InstantUpload/Camera/20170610_183520.jpg
2018-02-19T23:29:01+00:00 Generating previews for //files/InstantUpload/Camera/20170611_191432.jpg
2018-02-19T23:30:02+00:00 Generating previews for //files/InstantUpload/Camera/20170611_191433.jpg
2018-02-19T23:31:02+00:00 Generating previews for //files/InstantUpload/Camera/20170616_164441.jpg
2018-02-19T23:32:01+00:00 Generating previews for //files/InstantUpload/Camera/20170611_191436.jpg
2018-02-19T23:33:01+00:00 Generating previews for //files/InstantUpload/Camera/20170617_171510.jpg
2018-02-19T23:34:01+00:00 Generating previews for //files/InstantUpload/Camera/20170619_131416.jpg
2018-02-19T23:35:02+00:00 Generating previews for //files/InstantUpload/Camera/20170617_171514.jpg
2018-02-19T23:36:01+00:00 Generating previews for //files/InstantUpload/Camera/20170623_174438.jpg
2018-02-19T23:37:01+00:00 Generating previews for //files/InstantUpload/Camera/20170629_112745.jpg
2018-02-19T23:38:02+00:00 Generating previews for //files/InstantUpload/Camera/20170627_200946.jpg
2018-02-19T23:39:01+00:00 Generating previews for //files/InstantUpload/Camera/20170709_144417.jpg
2018-02-19T23:40:02+00:00 Generating previews for //files/InstantUpload/Camera/20170709_154047.jpg
2018-02-19T23:41:02+00:00 Generating previews for //files/InstantUpload/Camera/20170709_144644.jpg
2018-02-19T23:42:01+00:00 Generating previews for //files/InstantUpload/Camera/20170709_165213.jpg
2018-02-19T23:43:01+00:00 Generating previews for //files/InstantUpload/Camera/20170710_184034.jpg
2018-02-19T23:44:02+00:00 Generating previews for //files/InstantUpload/Camera/20170711_204132.jpg
2018-02-19T23:45:02+00:00 Generating previews for //files/InstantUpload/Camera/20170709_165216.jpg
2018-02-19T23:46:01+00:00 Generating previews for //files/InstantUpload/Camera/20170713_215358.jpg
2018-02-19T23:47:02+00:00 Generating previews for //files/InstantUpload/Camera/20170714_125510.jpg
2018-02-19T23:48:02+00:00 Generating previews for //files/InstantUpload/Camera/20170713_230323.jpg
2018-02-19T23:49:01+00:00 Generating previews for //files/InstantUpload/Camera/20170720_193927.jpg

[JulienDevillers]

Same for me. I can provide more info on request.
Nothing specific on the command line output.

For me the only way was to reduce files over 20mb and the ones bigger were usually created on my Samsung device using the panorama function.

The last file in the log is the problematic one and I just ended up moving them into a separate folder with the .nomedia file.

I had tried a script which hinted that the issue is with the panorama tag on the photo was incorrect l. It didn't work for me as none of the files were detected.

[XA21X]

Seems like this is almost always caused by #108 which can be mitigated by increasing memory_limit. It's been a while, but from what I remember, if a manual execution of preview:generate-all encounters OOM, it will silently exit (not sure if it has a special exit code), giving the impression that it "doesn't scan all folders" etc. Whereas if it was invoked by uploading a new image (and therefore run by a background job - cron), then you might find an error logged in nextcloud.log.

[joshtrichards]

Likely made better via nextcloud/server#22088 for #108.

Since this issue hasn't seen any new traffic in >5 years I'm closing it out. Anyone experiencing this today should create a new issue so it can be evaluated against today's implementation.