Upgrading from 13.0.1 to 13.0.5 using CLI gets stuck

[hostingnuggets]

Steps to reproduce

1. Have a Nextcloud 13.0.1 installation
2. Download Nextcloud 13.0.5 for upgrade
3. Manually upgrade using php /var/www/nextcloud/occ upgrade

Expected behaviour

Successful upgrade from 13.0.1 to 13.0.5

Actual behaviour

The occ upgrade gets stuck after the following step and does nothing:

...
[owncloud][updater][1] \OC\Repair::step: Repair step: Clear frontend caches
[owncloud][updater][1] \OC\Repair::info: Repair info: Image cache cleared

Server configuration

Operating system: Debian 9.4

Web server: nginx 1.10.3

Database: MariaDB 10.1.26

PHP version: 7.0.30

Nextcloud version: 13.0.1

Updated from an older Nextcloud/ownCloud or fresh install: Yes update from older Nextcloud

Where did you install Nextcloud from: nextcloud.com

Signing status:

Signing status

No errors have been found.

List of activated apps:

App list

Enabled:
  - activity: 2.6.1
  - admin_audit: 1.3.0
  - bruteforcesettings: 1.1.0
  - calendar: 1.6.1
  - comments: 1.3.0
  - contacts: 2.1.5
  - dav: 1.4.6
  - drop_account: 0.0.11
  - encryption: 2.0.0
  - federatedfilesharing: 1.3.1
  - federation: 1.3.0
  - files: 1.8.0
  - files_pdfviewer: 1.2.1
  - files_sharing: 1.5.0
  - files_texteditor: 2.5.1
  - files_trashbin: 1.3.0
  - files_videoplayer: 1.2.0
  - gallery: 18.0.0
  - logreader: 2.0.0
  - lookup_server_connector: 1.1.0
  - nextcloud_announcements: 1.2.0
  - notes: 2.3.2
  - notifications: 2.1.2
  - oauth2: 1.1.0
  - password_policy: 1.3.0
  - provisioning_api: 1.3.0
  - quota_warning: 1.2.0
  - serverinfo: 1.3.0
  - sharebymail: 1.3.0
  - survey_client: 1.1.0
  - tasks: 0.9.6
  - theming: 1.4.5
  - twofactor_backupcodes: 1.2.3
  - twofactor_totp: 1.4.1
  - updatenotification: 1.3.0
  - workflowengine: 1.3.0
Disabled:
  - files_external
  - files_versions
  - firstrunwizard
  - systemtags
  - user_external
  - user_ldap

Nextcloud configuration:

Config report

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "13.0.1.1",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "forcessl": true,
        "enable_previews": true,
        "enable_avatars": false,
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpmode": "smtp",
        "log_authfailip": true,
        "loglevel": 2,
        "maintenance": false,
        "trusted_domains": [
            "cloud.mydomain.tld"
        ],
        "share_folder": "\/Shared",
        "overwritewebroot": "\/",
        "overwriteprotocol": "https",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "singleuser": false,
        "memcache.local": "\\OC\\Memcache\\APCu",
        "trashbin_retention_obligation": "auto,30",
        "activity_expire_days": 14,
        "logtimezone": "Europe\/Berlin",
        "log_type": "errorlog",
        "skeletondirectory": "\/var\/www\/nextcloud\/themes\/mydomain\/core\/skeleton",
        "overwrite.cli.url": "https:\/\/cloud.mydomain.tld",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpauthtype": "LOGIN"
    }
}

Are you using external storage, if yes which one: local (NFS mount)

Are you using encryption: yes

Are you using an external user-backend, if yes which one: no

Client configuration

Browser: not relevant

Operating system: not relevant

Logs

Web server error log

Web server error log

2018/07/24 10:04:04 [error] 20978#20978: *10 FastCGI sent in stderr: "PHP message: [owncloud][core][0] Scss is disabled for /var/www/nextcloud/core/css/jquery-ui-fixes.scss, ignoring
PHP message: [owncloud][core][0] Scss is disabled for /var/www/nextcloud/core/css/server.scss, ignoring
PHP message: [owncloud][core][0] Scss is disabled for /var/www/nextcloud/core/css/jquery.ocdialog.scss, ignoring" while reading response header from upstream, client: 192.168.1.112, server: cloud.mydomain.tld, request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.sock:", host: "cloud.mydomain.tld"

Nextcloud log (data/nextcloud.log)

Nextcloud log

Nextcloud or one of the apps require upgrade - only a limited number of commands are available
You may use your browser or the occ upgrade command to do the upgrade
Set log level to debug
[owncloud][core][0] starting upgrade from 13.0.1.1 to 13.0.5.2
[owncloud][updater][1] \OC\Repair::step: Repair step: Repair MySQL collation
[owncloud][updater][1] \OC\Repair::info: Repair info: All tables already have the correct collation -> nothing to do
[owncloud][updater][1] \OC\Repair::step: Repair step: Repair SQLite autoincrement
[owncloud][updater][1] \OC\Repair::step: Repair step: Copy data from accounts table when migrating from ownCloud
[owncloud][updater][1] \OC\Repair::step: Repair step: Drop account terms table when migrating from ownCloud
Updating database schema
[owncloud][updater][1] \OC\Updater::dbUpgradeBefore: Updating database schema
Updated database
[owncloud][updater][1] \OC\Updater::dbUpgrade: Updated database
Updating <oauth2> ...
[owncloud][updater][1] \OC\Updater::appUpgradeStarted: Updating <oauth2> ...
[owncloud][updater][1] \OC\Repair::step: Repair step: Update OAuth token expiration times
Updated <oauth2> to 1.1.1
[owncloud][updater][1] \OC\Updater::appUpgrade: Updated <oauth2> to 1.1.1
Updating <dav> ...
[owncloud][updater][1] \OC\Updater::appUpgradeStarted: Updating <dav> ...
[owncloud][updater][1] \OC\Repair::step: Repair step: Fix component of birthday calendars
[owncloud][updater][1] \OC\Repair::info: Repair info: 3000 birthday calendars updated.
[owncloud][updater][1] \OC\Repair::step: Repair step: Fix broken values of calendar objects
Fix broken values of calendar objects

 Starting ...
 Done
    0/0 [>---------------------------]   0%
[owncloud][updater][1] \OC\Repair::finishProgress
[owncloud][updater][1] \OC\Repair::step: Repair step: Registering building of calendar search index as background job
[owncloud][updater][1] \OC\Repair::info: Repair info: Repair step already executed
Updated <dav> to 1.4.7
[owncloud][updater][1] \OC\Updater::appUpgrade: Updated <dav> to 1.4.7
Checking for update of app activity in appstore
[owncloud][updater][1] \OC\Updater::checkAppStoreAppBefore: Checking for update of app "activity" in appstore
Checked for update of app "activity" in appstore 
[owncloud][updater][1] \OC\Updater::checkAppStoreApp: Checked for update of app "activity" in appstore
Checking for update of app admin_audit in appstore
[owncloud][updater][1] \OC\Updater::checkAppStoreAppBefore: Checking for update of app "admin_audit" in appstore
Checked for update of app "admin_audit" in appstore 
[owncloud][updater][1] \OC\Updater::checkAppStoreApp: Checked for update of app "admin_audit" in appstore
Checking for update of app bruteforcesettings in appstore
[owncloud][updater][1] \OC\Updater::checkAppStoreAppBefore: Checking for update of app "bruteforcesettings" in appstore
Checked for update of app "bruteforcesettings" in appstore 
[owncloud][updater][1] \OC\Updater::checkAppStoreApp: Checked for update of app "bruteforcesettings" in appstore
Checking for update of app calendar in appstore
[owncloud][updater][1] \OC\Updater::checkAppStoreAppBefore: Checking for update of app "calendar" in appstore
Checked for update of app "calendar" in appstore 
[owncloud][updater][1] \OC\Updater::checkAppStoreApp: Checked for update of app "calendar" in appstore
Checking for update of app comments in appstore
[owncloud][updater][1] \OC\Updater::checkAppStoreAppBefore: Checking for update of app "comments" in appstore
Checked for update of app "comments" in appstore 
[owncloud][updater][1] \OC\Updater::checkAppStoreApp: Checked for update of app "comments" in appstore
Checking for update of app contacts in appstore
[owncloud][updater][1] \OC\Updater::checkAppStoreAppBefore: Checking for update of app "contacts" in appstore
Checked for update of app "contacts" in appstore 
[owncloud][updater][1] \OC\Updater::checkAppStoreApp: Checked for update of app "contacts" in appstore
Checking for update of app dav in appstore
[owncloud][updater][1] \OC\Updater::checkAppStoreAppBefore: Checking for update of app "dav" in appstore
Checked for update of app "dav" in appstore 
[owncloud][updater][1] \OC\Updater::checkAppStoreApp: Checked for update of app "dav" in appstore
Checking for update of app drop_account in appstore
[owncloud][updater][1] \OC\Updater::checkAppStoreAppBefore: Checking for update of app "drop_account" in appstore
Checked for update of app "drop_account" in appstore 
[owncloud][updater][1] \OC\Updater::checkAppStoreApp: Checked for update of app "drop_account" in appstore
Checking for update of app encryption in appstore
[owncloud][updater][1] \OC\Updater::checkAppStoreAppBefore: Checking for update of app "encryption" in appstore
Checked for update of app "encryption" in appstore 
[owncloud][updater][1] \OC\Updater::checkAppStoreApp: Checked for update of app "encryption" in appstore
Checking for update of app federatedfilesharing in appstore
[owncloud][updater][1] \OC\Updater::checkAppStoreAppBefore: Checking for update of app "federatedfilesharing" in appstore
Checked for update of app "federatedfilesharing" in appstore 
[owncloud][updater][1] \OC\Updater::checkAppStoreApp: Checked for update of app "federatedfilesharing" in appstore
Checking for update of app federation in appstore
[owncloud][updater][1] \OC\Updater::checkAppStoreAppBefore: Checking for update of app "federation" in appstore
Checked for update of app "federation" in appstore 
[owncloud][updater][1] \OC\Updater::checkAppStoreApp: Checked for update of app "federation" in appstore
Checking for update of app files in appstore
[owncloud][updater][1] \OC\Updater::checkAppStoreAppBefore: Checking for update of app "files" in appstore
Checked for update of app "files" in appstore 
[owncloud][updater][1] \OC\Updater::checkAppStoreApp: Checked for update of app "files" in appstore
Checking for update of app files_pdfviewer in appstore
[owncloud][updater][1] \OC\Updater::checkAppStoreAppBefore: Checking for update of app "files_pdfviewer" in appstore
Checked for update of app "files_pdfviewer" in appstore 
[owncloud][updater][1] \OC\Updater::checkAppStoreApp: Checked for update of app "files_pdfviewer" in appstore
Checking for update of app files_sharing in appstore
[owncloud][updater][1] \OC\Updater::checkAppStoreAppBefore: Checking for update of app "files_sharing" in appstore
Checked for update of app "files_sharing" in appstore 
[owncloud][updater][1] \OC\Updater::checkAppStoreApp: Checked for update of app "files_sharing" in appstore
Checking for update of app files_texteditor in appstore
[owncloud][updater][1] \OC\Updater::checkAppStoreAppBefore: Checking for update of app "files_texteditor" in appstore
Checked for update of app "files_texteditor" in appstore 
[owncloud][updater][1] \OC\Updater::checkAppStoreApp: Checked for update of app "files_texteditor" in appstore
Checking for update of app files_trashbin in appstore
[owncloud][updater][1] \OC\Updater::checkAppStoreAppBefore: Checking for update of app "files_trashbin" in appstore
Checked for update of app "files_trashbin" in appstore 
[owncloud][updater][1] \OC\Updater::checkAppStoreApp: Checked for update of app "files_trashbin" in appstore
Checking for update of app files_videoplayer in appstore
[owncloud][updater][1] \OC\Updater::checkAppStoreAppBefore: Checking for update of app "files_videoplayer" in appstore
Checked for update of app "files_videoplayer" in appstore 
[owncloud][updater][1] \OC\Updater::checkAppStoreApp: Checked for update of app "files_videoplayer" in appstore
Checking for update of app gallery in appstore
[owncloud][updater][1] \OC\Updater::checkAppStoreAppBefore: Checking for update of app "gallery" in appstore
Checked for update of app "gallery" in appstore 
[owncloud][updater][1] \OC\Updater::checkAppStoreApp: Checked for update of app "gallery" in appstore
Checking for update of app logreader in appstore
[owncloud][updater][1] \OC\Updater::checkAppStoreAppBefore: Checking for update of app "logreader" in appstore
Checked for update of app "logreader" in appstore 
[owncloud][updater][1] \OC\Updater::checkAppStoreApp: Checked for update of app "logreader" in appstore
Checking for update of app lookup_server_connector in appstore
[owncloud][updater][1] \OC\Updater::checkAppStoreAppBefore: Checking for update of app "lookup_server_connector" in appstore
Checked for update of app "lookup_server_connector" in appstore 
[owncloud][updater][1] \OC\Updater::checkAppStoreApp: Checked for update of app "lookup_server_connector" in appstore
Checking for update of app nextcloud_announcements in appstore
[owncloud][updater][1] \OC\Updater::checkAppStoreAppBefore: Checking for update of app "nextcloud_announcements" in appstore
Checked for update of app "nextcloud_announcements" in appstore 
[owncloud][updater][1] \OC\Updater::checkAppStoreApp: Checked for update of app "nextcloud_announcements" in appstore
Checking for update of app notes in appstore
[owncloud][updater][1] \OC\Updater::checkAppStoreAppBefore: Checking for update of app "notes" in appstore
Checked for update of app "notes" in appstore 
[owncloud][updater][1] \OC\Updater::checkAppStoreApp: Checked for update of app "notes" in appstore
Checking for update of app notifications in appstore
[owncloud][updater][1] \OC\Updater::checkAppStoreAppBefore: Checking for update of app "notifications" in appstore
Checked for update of app "notifications" in appstore 
[owncloud][updater][1] \OC\Updater::checkAppStoreApp: Checked for update of app "notifications" in appstore
Checking for update of app oauth2 in appstore
[owncloud][updater][1] \OC\Updater::checkAppStoreAppBefore: Checking for update of app "oauth2" in appstore
Checked for update of app "oauth2" in appstore 
[owncloud][updater][1] \OC\Updater::checkAppStoreApp: Checked for update of app "oauth2" in appstore
Checking for update of app password_policy in appstore
[owncloud][updater][1] \OC\Updater::checkAppStoreAppBefore: Checking for update of app "password_policy" in appstore
Checked for update of app "password_policy" in appstore 
[owncloud][updater][1] \OC\Updater::checkAppStoreApp: Checked for update of app "password_policy" in appstore
Checking for update of app provisioning_api in appstore
[owncloud][updater][1] \OC\Updater::checkAppStoreAppBefore: Checking for update of app "provisioning_api" in appstore
Checked for update of app "provisioning_api" in appstore 
[owncloud][updater][1] \OC\Updater::checkAppStoreApp: Checked for update of app "provisioning_api" in appstore
Checking for update of app quota_warning in appstore
[owncloud][updater][1] \OC\Updater::checkAppStoreAppBefore: Checking for update of app "quota_warning" in appstore
Checked for update of app "quota_warning" in appstore 
[owncloud][updater][1] \OC\Updater::checkAppStoreApp: Checked for update of app "quota_warning" in appstore
Checking for update of app serverinfo in appstore
[owncloud][updater][1] \OC\Updater::checkAppStoreAppBefore: Checking for update of app "serverinfo" in appstore
Checked for update of app "serverinfo" in appstore 
[owncloud][updater][1] \OC\Updater::checkAppStoreApp: Checked for update of app "serverinfo" in appstore
Checking for update of app sharebymail in appstore
[owncloud][updater][1] \OC\Updater::checkAppStoreAppBefore: Checking for update of app "sharebymail" in appstore
Checked for update of app "sharebymail" in appstore 
[owncloud][updater][1] \OC\Updater::checkAppStoreApp: Checked for update of app "sharebymail" in appstore
Checking for update of app survey_client in appstore
[owncloud][updater][1] \OC\Updater::checkAppStoreAppBefore: Checking for update of app "survey_client" in appstore
Checked for update of app "survey_client" in appstore 
[owncloud][updater][1] \OC\Updater::checkAppStoreApp: Checked for update of app "survey_client" in appstore
Checking for update of app tasks in appstore
[owncloud][updater][1] \OC\Updater::checkAppStoreAppBefore: Checking for update of app "tasks" in appstore
Checked for update of app "tasks" in appstore 
[owncloud][updater][1] \OC\Updater::checkAppStoreApp: Checked for update of app "tasks" in appstore
Checking for update of app theming in appstore
[owncloud][updater][1] \OC\Updater::checkAppStoreAppBefore: Checking for update of app "theming" in appstore
Checked for update of app "theming" in appstore 
[owncloud][updater][1] \OC\Updater::checkAppStoreApp: Checked for update of app "theming" in appstore
Checking for update of app twofactor_backupcodes in appstore
[owncloud][updater][1] \OC\Updater::checkAppStoreAppBefore: Checking for update of app "twofactor_backupcodes" in appstore
Checked for update of app "twofactor_backupcodes" in appstore 
[owncloud][updater][1] \OC\Updater::checkAppStoreApp: Checked for update of app "twofactor_backupcodes" in appstore
Checking for update of app twofactor_totp in appstore
[owncloud][updater][1] \OC\Updater::checkAppStoreAppBefore: Checking for update of app "twofactor_totp" in appstore
Checked for update of app "twofactor_totp" in appstore 
[owncloud][updater][1] \OC\Updater::checkAppStoreApp: Checked for update of app "twofactor_totp" in appstore
Checking for update of app updatenotification in appstore
[owncloud][updater][1] \OC\Updater::checkAppStoreAppBefore: Checking for update of app "updatenotification" in appstore
Checked for update of app "updatenotification" in appstore 
[owncloud][updater][1] \OC\Updater::checkAppStoreApp: Checked for update of app "updatenotification" in appstore
Checking for update of app workflowengine in appstore
[owncloud][updater][1] \OC\Updater::checkAppStoreAppBefore: Checking for update of app "workflowengine" in appstore
Checked for update of app "workflowengine" in appstore 
[owncloud][updater][1] \OC\Updater::checkAppStoreApp: Checked for update of app "workflowengine" in appstore
[owncloud][updater][1] \OC\Repair::step: Repair step: Repair MySQL collation
[owncloud][updater][1] \OC\Repair::info: Repair info: All tables already have the correct collation -> nothing to do
[owncloud][updater][1] \OC\Repair::step: Repair step: Repair mime types
[owncloud][updater][1] \OC\Repair::step: Repair step: Clean tags and favorites
[owncloud][updater][1] \OC\Repair::info: Repair info: 264 tags of deleted users have been removed.
[owncloud][updater][1] \OC\Repair::info: Repair info: 0 tags for delete files have been removed.
[owncloud][updater][1] \OC\Repair::info: Repair info: 0 tag entries for deleted tags have been removed.
[owncloud][updater][1] \OC\Repair::info: Repair info: 37 tags with no entries have been removed.
[owncloud][updater][1] \OC\Repair::step: Repair step: Repair invalid shares
[owncloud][updater][1] \OC\Repair::step: Repair step: Remove shares of a users root folder
[owncloud][updater][1] \OC\Repair::step: Repair step: Move .step file of updater to backup location
[owncloud][updater][1] \OC\Repair::step: Repair step: Add move avatar background job
[owncloud][updater][1] \OC\Repair::info: Repair info: Avatars are disabled
[owncloud][updater][1] \OC\Repair::step: Repair step: Add preview cleanup background jobs
[owncloud][updater][1] \OC\Repair::step: Repair step: Fix potential broken mount points
[owncloud][updater][1] \OC\Repair::info: Repair info: No mounts updated
[owncloud][updater][1] \OC\Repair::step: Repair step: Repair language codes
[owncloud][updater][1] \OC\Repair::step: Repair step: Install new core bundle components
[owncloud][updater][1] \OC\Repair::step: Repair step: Repair invalid paths in file cache
[owncloud][updater][1] \OC\Repair::step: Repair step: Rename folder with user specific keys
[owncloud][updater][1] \OC\Repair::step: Repair step: Add log rotate job
[owncloud][updater][1] \OC\Repair::step: Repair step: Clear frontend caches
[owncloud][updater][1] \OC\Repair::info: Repair info: Image cache cleared

Browser log

Browser log

not relevant

[nextcloud-bot]

GitMate.io thinks possibly related issues are #10350 (13.0.5), #9183 (Upgrade from 11.0.5 to 13.0.1 fails on oc_share table), #8912 (Upgrade from 12.0.6 to 13.0.1 failed), #8902 (Upgrade issue from 12.0.4 to 13.0.1 with docker image), and #5515 (Upgrade fails 10.0.5 to 11.0.3 ).

[MorrisJobke]

It should then continue with this:

\OC\Repair::info: Repair info: SCSS cache cleared
\OC\Repair::info: Repair info: JS cache cleared
\OC\Updater::startCheckCodeIntegrity: Starting code integrity check...
\OC\Updater::finishedCheckCodeIntegrity: Finished code integrity check
\OC\Updater::updateEnd: Update successful
\OC\Updater::maintenanceActive: Maintenance mode is kept active
\OC\Updater::resetLogLevel: Reset log level to Debug(0)

@rullzer @juliushaertl Any idea what could cause this?

[juliusknorr]

@hostingnuggets How long has it been stuck? I guess you have cancelled it at some point?

[hostingnuggets]

@juliushaertl the first time I waited 17 minutes (used the time command) until i CTRL-C it because nothing was happening (no database requests, nothing going on watching occ upgrade with strace) and the whole server seemed to do nothing really. Then I tried 2-3 more times to run the upgrade and waited 5 to 10 minutes maximum, still nothing was happening. It is my understanding that this step should be quite fast as it should just delete a bunch of cached files from the assets in the appdata_<instance_id> sub-directory of the data directory, or am I wrong here?

[MorrisJobke]

It is my understanding that this step should be quite fast as it should just delete a bunch of cached files from the assets in the appdata_<instance_id> sub-directory of the data directory, or am I wrong here

This is correct. I have still no idea why it was idling nevertheless. :/

[juliusknorr]

@hostingnuggets Can you check if running occ maintenance:repair before running the upgrade works fine?

[hostingnuggets]

@juliushaertl this is a production server, could you confirm me that this command is harmless and should not generate any downtime and/or break anything? and do I turn on maintenance mode before?

[juliusknorr]

this is a production server, could you confirm me that this command is harmless and should not generate any downtime and/or break anything? and do I turn on maintenance mode before?

It will have some downtime, because it enables the maintenance mode.

[hostingnuggets]

I now ran occ maintenance:repair on the 13.0.1 original installation and this worked fine. As this worked fine I attempted a second try at upgrading to 13.0.5 manually and this time everything worked. The occ upgrade only took 15 seconds to proceed through the whole upgrade. When I initially tried to upgrade there was a higher load on the NFS server as well as on the Nextcloud server but it is hard to say this is the cause of the initial problem, especially that it hanged at a specific stage of the repair process after the upgrade. I also tried on a test system with a similar setup (NFS server) beforehand and was not able to reproduce this problem. If nobody reports a similar issue I guess this was a one off...

[prodigy7]

13.0.1.1 showing production 13.0.4 available. Try to upgrade over web ui but page stays blank. Try it with occ maintenance:repair and occ upgrade but got message "Nextcloud is already latest version"

[MorrisJobke]

Try to upgrade over web ui but page stays blank. Try it with occ maintenance:repair and occ upgrade but got message "Nextcloud is already latest version"

If you want to do this on the CLI you need to go into the folder updater/ and call the updater.phar. https://docs.nextcloud.com/server/13/admin_manual/maintenance/update.html