CRC error with zip files downloaded from web UI

[kiv57]

Steps to reproduce

1. login to web UI
2. select a folder or multiple files, clic "Download"
3. unzip the downloaded zip with your desktop zip handler

Expected behaviour

Unzipped files should have correct CRC and be valid

Actual behaviour

• Errors about CRC and headers (message details depending on the unzip program used)
• Some files are invalid and can't be opened, depending on the file type

Server configuration

Operating system: Debian 8.6

Web server: Nginx 1.6.2

Database: MySQL 5.5.50-0+deb8u1

PHP version: 5.6.24

Nextcloud version: 10.0.1

Updated from an older Nextcloud/ownCloud or fresh install: OC 8.2.7 (debian repos) -> manual upgrade to NC 9.0.54 -> upgrade to NC 10.0.1

Where did you install Nextcloud from: https://download.nextcloud.com/server/releases/

Signing status: No errors have been found.

List of activated apps:

App list

Enabled:
  - activity: 2.3.2
  - admin_audit: 1.0.0
  - calendar: 1.4.1
  - comments: 1.0.0
  - contacts: 1.4.0.0
  - dav: 1.0.1
  - federatedfilesharing: 1.0.1
  - federation: 1.0.1
  - files: 1.5.2
  - files_mv: 0.8.2
  - files_pdfviewer: 0.8.1
  - files_sharing: 1.0.0
  - files_texteditor: 2.1
  - files_trashbin: 1.0.0
  - files_versions: 1.3.0
  - files_videoplayer: 0.9.8
  - firstrunwizard: 1.1
  - gallery: 15.0.0
  - notes: true
  - notifications: 0.3.0
  - password_policy: 1.0.0
  - passwords: 20.1
  - provisioning_api: 1.0.0
  - qownnotesapi: true
  - serverinfo: 1.1.1
  - survey_client: 0.1.5
  - systemtags: 1.0.2
  - tasks: true
  - templateeditor: 0.1
  - theming: 1.0.1
  - updatenotification: 1.0.1
  - workflowengine: 1.0.1
Disabled:
  - encryption
  - external
  - files_accesscontrol
  - files_automatedtagging
  - files_external
  - files_retention
  - user_external
  - user_ldap
  - user_saml

The content of config/config.php:

Config report

{
    "system": {
        "instanceid": "xxxxxxxxxxxx",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "cloud.domain.tld",
            "local.ip.behind.reverse.proxy"
        ],
        "datadirectory": "\/home\/www-data\/data",
        "overwrite.cli.url": "https:\/\/cloud.domain.tld",
        "default_language": "fr",
        "dbtype": "mysql",
        "version": "9.1.1.5",
        "dbname": "owncloud",
        "dbhost": "localhost",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "mail_from_address": "me",
        "mail_smtpmode": "smtp",
        "mail_domain": "domain.tld",
        "loglevel": 2,
        "memcache.local": "\\OC\\Memcache\\APCu",
        "theme": "",
        "maintenance": false,
        "remember_login_cookie_lifetime": 1296000,
        "session_lifetime": 86400,
        "session_keepalive": true,
        "updatechecker": false,
        "trashbin_retention_obligation": "auto",
        "mail_smtpsecure": "ssl",
        "mail_smtpauthtype": "LOGIN",
        "mail_smtphost": "mail.xxx.yyy",
        "mail_smtpport": "465",
        "mail_smtpauth": 1,
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "appstore.experimental.enabled": true
    }
}

Are you using external storage, if yes which one: no

Are you using encryption: no

Are you using an external user-backend, if yes which one: no

Client configuration

Browser: firefox 50.0

Operating system: Windows 7

Logs

Web server error log

Web server error log (dates don't fit with zip download dates)

2016/11/24 07:37:31 [error] 853#0: *7379 FastCGI sent in stderr: "PHP message: PHP Fatal error:  Uncaught exception 'Exception' with message 'Session has been closed - no further changes to the session are allowed' in /var/www/owncloud/lib/private/Session/Internal.php:154
Stack trace:
#0 /var/www/owncloud/lib/private/Session/Internal.php(64): OC\Session\Internal->validateSession()
#1 /var/www/owncloud/lib/private/Session/CryptoSessionData.php(164): OC\Session\Internal->set('encrypted_sessi...', '43893e78d74e0a2...')
#2 /var/www/owncloud/lib/private/Session/CryptoSessionData.php(67): OC\Session\CryptoSessionData->close()
#3 [internal function]: OC\Session\CryptoSessionData->__destruct()
#4 {main}
  thrown in /var/www/owncloud/lib/private/Session/Internal.php on line 154" while reading upstream, client: 10.0.3.1, server: cloud.domain.tld, request: "GET /index.php/login HTTP/1.0", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "cloud.domain.tld"

2016/11/27 13:26:43 [error] 850#0: *46841 open() "/var/www/owncloud/favicon.ico" failed (2: No such file or directory), client: 10.0.3.1, server: cloud.domain.tld, request: "GET /favicon.ico HTTP/1.0", host: "cloud.domain.tld"

Nextcloud log (data/nextcloud.log)

Nextcloud log (just one repeated error I didn't investigate yet)

Error	core	Could not get application: cURL error 60: SSL certificate problem: unable to get local issuer certificate	2016-11-23T19:00:08+00:00

Browser log

Not sure it's useful in this case, if so please ask.

[nickvergessen]

Any information which files are affected? Because it seems to work just fine here with images and documents.

[kiv57]

The error messages always appear, with any file type :

• "Erreur en-têtes" with 7zip ("headers error" should be the translation ?)
• "There is a CRC mistake on the file (...) Continue ?" with power archiver (translated from french to).I have to clic "Yes" for each file in the zip.
  (test with the same .zip of course)

That said, some files are corrupted and can't be opened, some others seem valid and I can open them. The fun part is that it depends on the program which did the unzip :

• pdf and docx : valid with 7zip, corrupted with power archiver
• xlsx : valid with both
• CREO files (CAD) : always corrupted
  I must admit I didn't test every possible file type...
  Moreover, users of the download feature don't have always the choice of the unzip program.

Please feel free to ask for more tests if necessary.

[nickvergessen]

Do zips that only have english characters work?

[kiv57]

No : I already tested files without accents in file name, the problem persists.

[nerdissimo-de]

I can confirm the error. When trying to upload to Gmail it doesn't allow the upload because it thinks there is a virus (there definitely isn't). When trying to open the file with 7zip or Nemo Fileroller it's also not possible. Funny thing is: When I use the commandline unzip it works fine.

Edit: doesn't matter which filetypes are in the archives. Happens for images and documents here.

[JarbyDev]

I have the same problem Here using the microsoft default unzip software

and using 7zip, i was able to extracted the content, but after i tried to open the file i get the file is currupted. happen with pdf, excel, image files

[qwertygc]

I have the same problem with NC 11.

[jospoortvliet]

Same issue here, zip files are often not working in some unzip software but OK in others. On my desktop - Dolphin can't open them as folders but Ark is OK with 'em.

[Idlebawb]

I am having a similar Issue. I did not get any CRC Errors or completely unreadable files, but Headers Error on 7-zip. The Main issue with that is that there is a gateway / reverse proxy between the server and the clients and that doesn't let any of those zips through. So nobody can download folders or multiple files, which is a big problem.

See Errors in attached screenshots.

Server: SUSE Enterprise 12
NextCloud Version: tried 10.0.3 & 11.0.1
Client: Only tried on Windows 10 Pro x64 (1511) but I guess its not a Client OS issue.
Browser: tried FF 50.1, Chrome 56 & IE11 - no difference

What I did:
Select a Folder & Download it
Select multiple Files at once & Download

The Files were: a single line txt file and a new .xlsx with just random numbers in it.
No special characters in either folder or filenames.

I don't know how I would look at the headers in 7zip, the most information I can get out the zip file is from zipinfo -v, but the Version on SUSE is from 2009, so I don't know how helpful that will be. But here it is.

Output of zipinfo -v

Archive: /home/it-department/scripts/New folder.zip There is no zipfile comment.

End-of-central-directory record:

Zip archive file size: 8451 (0000000000002103h)
Actual end-cent-dir record offset: 8353 (00000000000020A1h)
Expected end-cent-dir record offset: 8353 (00000000000020A1h)
(based on the length of the central directory and its expected offset)

This zipfile constitutes the sole disk of a single-part archive; its
central directory contains 3 entries.
The central directory is 328 (0000000000000148h) bytes long,
and its (expected) offset in bytes from the beginning of the zipfile
is 8025 (0000000000001F59h).

Central directory entry #1:

New folder/

offset of local header from start of archive: 0
(0000000000000000h) bytes
file system or operating system of origin: Unix
version of encoding software: 4.5
minimum file system compatibility required: MS-DOS, OS/2 or NT FAT
minimum software version required to extract: 4.5
compression method: none (stored)
file security status: not encrypted
extended local header: no
file last modified on (DOS date/time): 2017 Feb 15 08:50:18
32-bit CRC value (hex): 00000000
compressed size: 0 bytes
uncompressed size: 0 bytes
length of filename: 11 characters
length of extra field: 32 bytes
length of file comment: 0 characters
disk number on which file begins: disk 1
apparent file type: binary
Unix file attributes (040755 octal): drwxr-xr-x
MS-DOS file attributes (10 hex): dir

The central-directory extra field contains:

• A subfield with ID 0x0001 (PKWARE 64-bit sizes) and 28 data bytes. The first
  20 are: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00.

There is no file comment.

Central directory entry #2:

New folder/Microsoft Excel-Arbeitsblatt (neu).xlsx

offset of local header from start of archive: 73
(0000000000000049h) bytes
file system or operating system of origin: Unix
version of encoding software: 4.5
minimum file system compatibility required: MS-DOS, OS/2 or NT FAT
minimum software version required to extract: 4.5
compression method: none (stored)
file security status: not encrypted
extended local header: yes
file last modified on (DOS date/time): 2017 Feb 15 08:50:18
32-bit CRC value (hex): 163d861c
compressed size: 7694 bytes
uncompressed size: 7694 bytes
length of filename: 50 characters
length of extra field: 32 bytes
length of file comment: 0 characters
disk number on which file begins: disk 1
apparent file type: binary
Unix file attributes (100644 octal): -rw-r--r--
MS-DOS file attributes (00 hex): none

The central-directory extra field contains:

• A subfield with ID 0x0001 (PKWARE 64-bit sizes) and 28 data bytes. The first
  20 are: 0e 1e 00 00 00 00 00 00 0e 1e 00 00 00 00 00 00 49 00 00 00.

There is no file comment.

Central directory entry #3:

There are an extra 20 bytes preceding this file.

New folder/Neues Textdokument.txt

offset of local header from start of archive: 7899
(0000000000001EDBh) bytes
file system or operating system of origin: Unix
version of encoding software: 4.5
minimum file system compatibility required: MS-DOS, OS/2 or NT FAT
minimum software version required to extract: 4.5
compression method: none (stored)
file security status: not encrypted
extended local header: yes
file last modified on (DOS date/time): 2017 Feb 15 08:50:18
32-bit CRC value (hex): 4a17b156
compressed size: 11 bytes
uncompressed size: 11 bytes
length of filename: 33 characters
length of extra field: 32 bytes
length of file comment: 0 characters
disk number on which file begins: disk 1
apparent file type: binary
Unix file attributes (100644 octal): -rw-r--r--
MS-DOS file attributes (00 hex): none

The central-directory extra field contains:

• A subfield with ID 0x0001 (PKWARE 64-bit sizes) and 28 data bytes. The first
  20 are: 0b 00 00 00 00 00 00 00 0b 00 00 00 00 00 00 00 db 1e 00 00.

There is no file comment.

Edit: Just some additional info.
I tried to create some 64bit Zips of the same file with 7zip and dotnetzip.
7zip wont let me create any 64bit zips unless the source files are bigger than 4GB
dotnetzip with -64 option creates the same "Header Error", but always useses deflate, couldnt get it to "Store"
32Bit Zips were fine.

EDIT2:
Disabled zip64 through a change in Streamer.php line 45

$this->streamerInstance = new ZipStreamer(['zip64' => PHP_INT_SIZE !== 4]);

I just set straight up false for zip64. I guess I will get errors on bigger files, but I am willing to trade not working at all (for me) for possible occasional errors. Hope this will get a real fix in a future release.

[andyboeh]

Same issue for me (NC 11.0.2). I can also confirm that disabling Zip64 fixes the problem (I do not have any files larger than 4GB anyway) by setting
$this->streamerInstance = new ZipStreamer(['zip64' => false]);
in lib/private/Streamer.php

[McNetic]

Hi there. As some may already know, I'm the author of the ZipStreamer library responsible for creating zip files in OC and NC.
That said, I'm still using OC in production and can't seem to reproduce the problem in my installation. So, a few questions to those who can:

1. Is the problem reproducible with any file, or only with specific files? (if the latter, could you provide some of the problematic files?)
2. Are you running nextcloud on a 32 bit or a 64 bit machine?
3. Although the responsible code seems to still be the same: has anyone been able to reproduce the problem in Owncloud?

Thank you for your help.

[andyboeh]

Sure, thanks for the reply:

1. It seems to be easy to reproduce as it happens nearly every time I share something. I'll send you a problematic file directly.
2. I'm on a 64-bit Server, Apache 2.4, PHP 7.1.1, NextCloud 11.0.2