lot of memory used when uploading a file with nextcloud client #3077
Description
Steps to reproduce
- add or update a file in a folder synced by nextcloud client (same problem with owncloud client also)
Expected behaviour
The file should by uploaded, then the nextcloud client should display that the folder is now synched.
Actual behaviour
- see the progress bar go to the end (saying that it uploaded 1 file on 1)
- wait
- wait
- client say that the server send a 500 error
- reconnect
- check for updated file in the folder
- display that the folder is now synched
details
I've update to nextcloud 11 at Xmas, got no problems to update or add new file a that point. Next week I activate the preview generator plugin and, in a screen launched an occ preview:delete_old followed by an occ preview:generate-all which lasted several days.
at one point, I had a power failure and after restarting the server I launched again occ preview:generate-all that seems start again from the point it was before the power failure.
Now, each time I try to update a file(even a 10k text file), I can see that the file is uploaded correctly by the client. but the client seems to be stuck at the end.
Meanwhile, on the server, I can see the mysql use cpu doing the following request:
SELECT `fileid`, `storage`, `path`, `parent`, `name`,
`mimetype`, `mimepart`, `size`, `mtime`, `encrypted`,
`etag`, `permissions`, `checksum`
FROM `oc_filecache`
WHERE `storage` = '4'
AND `name` COLLATE UTF8_general_ci LIKE '%%'
and after that, I see an apache process memory grew until it reach the maximum value allowed in nextcloud conf file then crash.
I had to increase the max memory allowed for php to 1Go to be able to complete an upload without crashing but even for modifying a 10Ko text file, it require 1Go memory and several minute to complete. and if several files are trying to be updated simultaneously, each apache process eat 1Go, force the server to swap and time out.
It seems that the request above is taking time and memory. The storage 4 is the root of nextcloud data:
select * from oc_storages where numeric_id=4;
+----------------------------+------------+-----------+--------------+
| id | numeric_id | available | last_checked |
+----------------------------+------------+-----------+--------------+
| local::/var/owncloud-data/ | 4 | 1 | NULL |
+----------------------------+------------+-----------+--------------+
and is filed with lot of preview files like:
select storage, path, name from oc_filecache WHERE `storage` = '4' limit 1;
+---------+----------------------------------------------------+---------------+
| storage | path | name |
+---------+----------------------------------------------------+---------------+
| 4 | appdata_XXX/preview/141197/1278-2048.png | 1278-2048.png |
+---------+----------------------------------------------------+---------------+
select count(*) from oc_filecache WHERE `storage` = '4' AND `name` COLLATE UTF8_general_ci LIKE '%%';
+----------+
| count(*) |
+----------+
| 437166 |
+----------+
1 row in set (27.45 sec)
Is this request is part of the normal process and thus I need to get rid of all preview files to get an acceptable behaviour or could it come from a bad database value and in that case how can I find and fix it without spending days to regenerate preview files?
Edit:
After further test, I noticed that I can create or update a csv or xls file without problem.
I can also create a gif, png or jpg file then update it without problems, but as soon a I have executed occ preview:pre-generate if I try to update those gif, png or jpg file, I got the memory issue.
There is no problems to delete them.
The problem also occur with 11.0.1 .
Server configuration
Operating system: Mageia 5
Web server: apache-2.4
Database: mariadb-10.0.28
PHP version: php-5.6.29
Nextcloud version: Nextcloud 11.0.0 (stable)
Updated from an older Nextcloud/ownCloud or fresh install: owncloud 8.2 -> nextcloud 9 -> nextcloud 10 -> nextcloud 11
Where did you install Nextcloud from: rpm packages
List of activated apps:
App list
Enabled: - activity: 2.4.1 - admin_audit: 1.1.0 - bookmarks: 0.9.1 - calendar: 1.4.1 - contacts: 1.5.2 - dav: 1.1.1 - federatedfilesharing: 1.1.1 - federation: 1.1.1 - files: 1.6.1 - files_pdfviewer: 1.0.1 - files_sharing: 1.1.1 - files_texteditor: 2.2 - files_trashbin: 1.1.0 - files_videoplayer: 1.0.0 - gallery: 16.0.0 - logreader: 2.0.0 - lookup_server_connector: 1.0.0 - mail: 0.6.2 - nextcloud_announcements: 1.0 - notes: 2.2.0 - notifications: 1.0.1 - password_policy: 1.1.0 - previewgenerator: 1.0.1 - provisioning_api: 1.1.0 - serverinfo: 1.1.1 - tasks: 0.9.4 - templateeditor: 0.2 - theming: 1.1.1 - twofactor_backupcodes: 1.0.0 - updatenotification: 1.1.1 - workflowengine: 1.1.1 Disabled: - comments - documents - encryption - external - files_accesscontrol - files_automatedtagging - files_external - files_retention - files_versions - firstrunwizard - galleryplus - sharebymail - survey_client - systemtags - user_external - user_ldap - user_samlThe content of config/config.php:
Config report
true, 'dbtype' => 'mysql', 'dbname' => 'OwnCloud', 'dbuser' => 'XXXXX', 'dbpassword' => 'XXXXXX', 'dbhost' => 'localhost', 'dbtableprefix' => 'oc_', 'passwordsalt' => 'XXXXXX', 'forcessl' => true, 'trusted_domains' => array ( 0 => 'XXXXXX', ), 'theme' => '', 'overwritewebroot' => '/nextcloud', 'default_language' => 'fr', 'datadirectory' => '/var/owncloud-data', 'maintenance' => false, 'instanceid' => 'XXXXXX', 'version' => '11.0.0.10', 'maxZipInputSize' => 838860800, 'allowZipDownload' => true, 'defaultapp' => 'calendar', 'secret' => 'XXXXXXX', 'overwrite.cli.url' => 'https://XXXXXXX/nextcloud', 'mail_from_address' => 'nextcloud', 'mail_smtpmode' => 'php', 'mail_domain' => 'sandalphon', 'logtimezone' => 'Europe/Paris', 'log_rotate_size' => 10485760, 'loglevel' => 2, 'memcache.local' => '\\OC\\Memcache\\APCu', 'trashbin_retention_obligation' => 'auto, 90', 'integrity.check.disabled' => true, );Are you using external storage, if yes which one: no
Are you using encryption: no
Are you using an external user-backend, if yes which one: no
Client configuration
Browser:
nextcloud client 2.2.4
Operating system:
Mageia cooker