Enabled encryption, encrypted all files, corrupted uploads, can't disable encryption, dangling `.part` files everywhere that can't be deleted

[posita]

I am evaluating an instance I have created using files_primary_s3 for use at my company. I have onboarded a handful of users who have been using the system for several weeks. They have migrated their contacts, calendars, etc. Recently, after performing what appeared to be a successful upgrade to 10.8.0.4, I enabled server-side encryption per these instructions and did occ encryption:encrypt-all. Shortly thereafter, users started complaining of corruption with newly-synced files. (Sources include desktop and iOS app). I attempted to decrypt-all so that I could disable encryption, which failed on the post-encryption corrupted files. In addition, several newly created file entries appeared as a result of the decrypt-all step alongside those corrupt files with suffixes like .decrypted.<timestamp>.part. These showed up in users' folders (at least on web). Users were able to delete their own post-encryption corrupted files from their instances, but not the newly-created .part files in their home directories because apparently .part is a reserved suffix (but they could remove parent folders where .part files resided without issue).

Now I am stuck. I can't decrypt-all because of the corrupted .part files (even with the originals gone), which means I can't turn off encryption. Users can't sync new content because after turning on encryption, all files coming from desktop and iOS apps get corrupted.

[posita]

UPDATE: I have managed to disable encryption by doing UPDATE `oc_filecache` SET encrypted = 0 long enough for ./occ encryption:disable to "work" (or at least complete successfully). Users are now able to sync files from their desktops and iOS apps again. They were able to restore most of the corrupted uploads from local copies. But some users still have .part files in their home directories (visible on web). How can my users get rid of them without me blowing away their user accounts? Also, how do I ensure that there are no dangling files in my S3 buckets? My users are prepared to resync all of their content from their local desktops if necessary, but restoring contacts, calendars, etc., would be a real pain.

[posita]

[JammingBen]

These part-files are part of the chunking process ownCloud uses to process uploads. With that being said, I never experienced a scenario where those files suddenly show up in the file list. Did you enable the maintenance mode before running all of those encryption commands?

There is a cleanup chunks command and/or a file scan command which could repair these entries. I wouldn't bet on it though, this is a super strange scenario. The cleanest way would definitely be to re-create those users. If you want to try these commands, be sure to backup your database and data first and use maintenance mode!

Edit: Apparently the decrypt-all command also creates such part files. It seems your instance runs into some error during the copy-rename-process (https://github.com/owncloud/core/blob/master/lib/private/Encryption/DecryptAll.php#L266).

[posita]

These part-files are part of the chunking process ownCloud uses to process uploads. With that being said, I never experienced a scenario where those files suddenly show up in the file list. Did you enable the maintenance mode before running all of those encryption commands?

I followed the instructions as written. It's possible I could have made an error, but being careful was a priority for me. Has anyone tested server-side encrypting with files_primary_s3?

There is a cleanup chunks command and/or a file scan command which could repair these entries. … If you want to try these commands, be sure to backup your database and data first and use maintenance mode!

The first notes in both sets of instructions you referenced read:

These commands are not available in single-user (maintenance) mode.

I ran them anyway. Nothing helped:

$ ./occ dav:cleanup-chunks
Cleaning chunks older than 2 days(2021-09-11T12:09:24+00:00)
Cleaning chunks for admin
    0 [->--------------------------]
Cleaning chunks for …
    0 [->--------------------------]
Cleaning chunks for …
    0 [->--------------------------]
…
$ ./occ files:scan --all

Scanning files for … users
Starting scan for user 1 out of … (admin)
Starting scan for user 2 out of … (…)
Starting scan for user 3 out of … (…)
Starting scan for user 4 out of … (…)
…

+---------+-------+--------------+------------------+
| Folders | Files | Elapsed time | Items per second |
+---------+-------+--------------+------------------+
| 0       | 0     | 00:00:00     | 0                |
+---------+-------+--------------+------------------+
% ./occ files:cleanup
0 orphaned file cache entries deleted

Where do the .part entries live? What table(s)? What manages them? Is there (or should there be) an ownCloud analogy to fsck (8) that detects and repairs all internal inconsistencies (like the ones I'm experiencing)?

[posita]

It's probably useful to not lose sight of the forest for the trees here. The .part files are symptomatic of corruption for files synced (via desktop and mobile) after encryption was enabled. encryption:decrypt-all failed on only those files, and resulted in all these .part files. Each attempt created more.

This was after my first attempt (that failed on File1.png:

File1.png
File1.png.decrypted.1631241567.part

After my second:

File1.png
File1.png.decrypted.1631241567.part
File1.png.decrypted.1631241567.part.decrypted.1631243784.part
File1.png.decrypted.1631243783.part

After my third:

File1.png
File1.png.decrypted.1631241567.part
File1.png.decrypted.1631241567.part.decrypted.1631243784.part
File1.png.decrypted.1631241567.part.decrypted.1631247922.part
File1.png.decrypted.1631241567.part.decrypted.1631243784.part.decrypted.1631247923.part
File1.png.decrypted.1631243783.part
File1.png.decrypted.1631243783.part.decrypted.1631247923.part
File1.png.decrypted.1631247922.part

And so on. It's pretty clear to see that not only was the decrypter choking on the corrupted File1.png, but also each of the .part files it created (and left dangling) in the process.

The results of SELECT * FROM `oc_filecache` WHERE encrypted != 0 appeared limited to the corrupted files (i.e., those synced after turning on encryption). I treated those as unsalvageable (encrypted or not). My plan was to allow encryption:decrypt-all to proceed without error by doing UPDATE `oc_filecache` SET encrypted = 0 before running it again, because I needed server-side encryption turned off. (People couldn't sync or upload new files with it on.)

This worked, for the most part. encryption:disable proceeded without error. Users were left with .part files in places alongside corrupted files. The silver lining was that it identified which files were corrupted and which were okay. Users could not delete or move the .part files, but they could move good files out of folders where .part files lived and delete the enclosing folders. This appeared to address most of the cases. (I don't have a way of verifying whether artifacts exist for those .part files whose parent folders were deleted.) However, there is not straightforward way of doing this with home folders.

If I have to blow away users to repair this, I'll probably be eliminating ownCloud as a solution. While it would would take a couple hours for the small number of users I have under this evaluation, I couldn't do that at scale. For our case:

1. (Perhaps channeling RFC 1925), encryption has to work. It doesn't appear to with our setup. I'm willing to leave that aside to test another day, but not without…
2. I have to have a way to address ownCloud file inconsistencies if and when they arise in the future without having to blow away and recreate users.

Is it possible to change the parent folders of the .part files in users' homes to something else?

[posita]

The .part files do not sync to the desktop client, but they're visible on web and mobile. If you try to download a single one (e.g., via https://…/remote.php/webdav/File1.png.decrypted.1631241567.part?downloadStartSecret=…), it results in a Bad Request error.

However, it is possible to select and batch download these .part files via a tar file (e.g., via https://…/index.php/apps/files/ajax/download.php?dir=%2F&files[]=File1.png.decrypted.1631241567.part&files[]=File1.png.decrypted.1631241567.part.decrypted.1631243784.part&downloadStartSecret=…). (They're, unsurprisingly, a bunch of zero byte files.)

[JammingBen]

I digged a bit into this topic and found out that encryption is indeed not fully compatible with the S3 object storage. The docs mentioned this in the past, but the information got lost over time for some unknown reason...

[posita]

Yikes! Okay. That seems pretty important. (Filed separately as #39263.) How do I get rid of the dangling .part files? Is there a way to rename in the data store so they can be removed?

[JammingBen]

I'm assuming these part-files are in your oc_filecache table, too? I still don't know how this happened, this shouldn't happen under any circumstances. AFAIK OC provides no "nice" way to deal with this scenario.

I'd suggest you backup the database and run a SQL-query on oc_filecache to remove all entries which include .decrypted. and end with .part in both the columns path and name. This way, the files shouldn't appear in your user's file lists. Additionally, you might delete those files on your storage (if you haven't already).

[JammingBen]

@posita I'm not a S3 object storage expert, but it seems that it already brings encryption. You can specify serversideencryption in your config.php alongside your object storage configuration.

We're currently working on the docs to bring some more light into this whole topic.

[posita]

I don't see any reference to serversideencryption on owncloud.com, but if you're talking about this, then that's precisely what led to this predicament. If you're talking about this, I would be really reluctant to turn that one without understanding more about what it does, how it operates, recovery modes, etc. This, for example hardly mentions it.

[JammingBen]

I'm not talking about the Encryption documentation. This is ownCloud's encryption mechanism which is, as stated above, not compatible with the S3 object storage unfortunately. I meant the serversideencryption setting that you've found already. Again, we're currently working on the documentation for this. I'll keep you updated once we gathered all the missing information in our docs.

[posita]

I'm assuming these part-files are in your oc_filecache table, too? I still don't know how this happened, this shouldn't happen under any circumstances. AFAIK OC provides no "nice" way to deal with this scenario.

I'd suggest you backup the database and run a SQL-query on oc_filecache to remove all entries which include .decrypted. and end with .part in both the columns path and name. This way, the files shouldn't appear in your user's file lists. Additionally, you might delete those files on your storage (if you haven't already).

I see a lot of entries with the path set to something like files_encryption/keys/files/File1.png.decrypted.1631241567.part.fake or files/File1.png.decrypted.1631241567.part.fake (where both share a name like File1.png.decrypted.1631241567.part). Should I remove all of those? Also, how do I map back to my bucket entries? Those all have names like urn:oid:12345. Is the number portion the same as the fileid?

UPDATE: Is there a way to get a list of "active" (legit) OIDs from the MySQL database? What I'd like to do is get a list of objects in my S3 bucket, and set-subtract the objects still referenced by ownCloud to get a list of orphaned S3 objects, but I don't know how to get the list from ownCloud.