[stable9.1] Integration test encryption homedir by PVince81 · Pull Request #26993 · owncloud/core

PVince81 · 2017-01-20T14:48:11Z

Backport of #26844 to stable9.1

⚠️ I added an extra commit to temporarily enable the new tests on this PR, needs to be removed before merge ! ⚠️

@DeepDiver1975 @SergioBertolinSG

SergioBertolinSG · 2017-01-23T09:23:28Z

I've run that failing test locally and it works fine. - With OC_TEST_ENCRYPTION_ENABLED=1.

The problem happens when running with OC_TEST_ALT_HOME=1.

SergioBertolinSG · 2017-01-23T09:37:04Z

#26920 should be backported first. To avoid that failure.

SergioBertolinSG · 2017-01-23T10:32:21Z

And that PR requires #26865 also.

PVince81 · 2017-01-23T19:33:06Z

I added the commits of #27003 here before the previous ones.

Let's let Jenkins do the testing for hopefully better results.

PVince81 · 2017-01-24T07:59:11Z

Ran the tests locally (but had to cherry-pick #27008):

OC_TEST_ENCRYPTION_ENABLED=1 OC_TEST_ALT_HOME=1
OC_TEST_ENCRYPTION_ENABLED=1 (after removing ext storage test for encryption)
OC_TEST_ALT_HOME=1

So far, tests passed.

DeepDiver1975 · 2017-01-24T08:32:31Z

⚠️ I added an extra commit to temporarily enable the new tests on this PR, needs to be removed before merge ! ⚠️

ah - this is why I'm getting notifications about broken build 😉

SergioBertolinSG · 2017-01-24T08:34:54Z

Running with OC_TEST_ALT_HOME=1 there are two failures:

build/integration/features/sharing-v1.feature:860
build/integration/features/webdav-related.feature:425

Our old friend the blinking merging shares tests:
"Merging shares for recipient when shared from outside with group then user and recipient renames in between"

Then as "user1" gets properties of folder "/merge-test-outside-groups-renamebeforesecondshare-renamed" with                                        # FeatureContext::asGetsPropertiesOfFolderWith()
      | {http://owncloud.org/ns}permissions |
      HTTP error: 404 (Sabre\DAV\Exception)
And the single response should contain a property "{http://owncloud.org/ns}permissions" with value "SRDNVCK"

And "Creating a folder with special chars"

When as "user0" gets properties of folder "/test_folder:5" with                                       # FeatureContext::asGetsPropertiesOfFolderWith()
      | {DAV:}resourcetype |
      HTTP error: 404 (Sabre\DAV\Exception)
    Then the single response should contain a property "{DAV:}resourcetype" with value "{DAV:}collection"

PVince81 · 2017-01-24T08:47:36Z

For me locally all tests passed with OC_TEST_ALT_HOME=1.
Note that I ran the tests with a Mysql instance.

@SergioBertolinSG what's yours ?

SergioBertolinSG · 2017-01-24T08:49:42Z

MySQL as well.

PVince81 · 2017-01-24T10:40:21Z

And Jenkins only has a single failure:

/var/lib/jenkins/workspace/owncloud-core_core_PR-26993-XV2N62E24JPAXCGLBGNNXV2DF7ZQXMJUZ24SGHXEFGTTYNVVLLZQ/build/integration/features/external-storage.feature:7

PVince81 · 2017-01-24T10:40:35Z

Consistency, where are thou ?

PVince81 · 2017-01-24T11:15:03Z

I'm out of ideas and I can't debug this locally since it all works fine.

@SergioBertolinSG can you debug the failures on your side, I suggest you start with the one that you have in common with Jenkins. Thanks.

SergioBertolinSG · 2017-01-24T11:16:07Z

@PVince81 sure, I'm trying in a new environment to see if those errors happen again.

SergioBertolinSG · 2017-01-24T11:23:59Z

Ok in my new environment all has passed correctly. 👍 (using OC_TEST_ALT_HOME=1)

SergioBertolinSG · 2017-01-24T11:31:37Z

The remaining problem happens only using OC_TEST_ENCRYPTION_ENABLED=1 and it happens locally as well.

SergioBertolinSG · 2017-01-24T12:32:45Z

The remaining error is also happening in master. So I guess it would be better to fix it in another PR for master and add the backport here.

Running locally OC_TEST_ENCRYPTION_ENABLED=1 ./run.sh features/external-storage.feature

It passes when the server is fresh installed. A second run makes it fail.

SergioBertolinSG · 2017-01-24T12:34:00Z

Error logs:

[Tue Jan 24 12:24:52 2017] fclose() expects parameter 1 to be resource, string given at /var/www/html/encryption_vincent/lib/private/Files/Storage/Wrapper/Encryption.php#749
[Tue Jan 24 12:24:52 2017] Exception: {"Message":"multikeydecrypt with share key failed:error:0E06D06C:configuration file routines:NCONF_get_string:no value","Exception":"OCA\\Encryption\\Exceptions\\MultiKeyDecryptException","Code":0,"Trace":"#0 \/var\/www\/html\/encryption_vincent\/apps\/encryption\/lib\/KeyManager.php(423): OCA\\Encryption\\Crypto\\Crypt->multiKeyDecrypt('\\x93\\t\\x8E\\xBA~\\xA4.\\x9D\\xCE2\\xFC\\x8A\\xF5G\\x97...', '\\xA1-\\xAC\\x12\\f\\xF6o\\x84\\xE9Z[Bh\\xC51...', '-----BEGIN PRIV...')\n#1 \/var\/www\/html\/encryption_vincent\/apps\/encryption\/lib\/Crypto\/Encryption.php(186): OCA\\Encryption\\KeyManager->getFileKey('\/user0\/files\/lo...', 'user0')\n#2 \/var\/www\/html\/encryption_vincent\/lib\/private\/Files\/Stream\/Encryption.php(257): OCA\\Encryption\\Crypto\\Encryption->begin('\/user0\/files\/lo...', 'user0', 'w', Array, Array)\n#3 [internal function]: OC\\Files\\Stream\\Encryption->stream_open('ocencryption:\/\/', 'w', 0, NULL)\n#4 \/var\/www\/html\/encryption_vincent\/lib\/private\/Files\/Stream\/Encryption.php(197): fopen('ocencryption:\/\/', 'w', false, Resource id #63)\n#5 \/var\/www\/html\/encryption_vincent\/lib\/private\/Files\/Stream\/Encryption.php(179): OC\\Files\\Stream\\Encryption::wrapSource(Resource id #62, Resource id #63, 'ocencryption', 'OC\\\\Files\\\\Stream...', 'w')\n#6 \/var\/www\/html\/encryption_vincent\/lib\/private\/Files\/Storage\/Wrapper\/Encryption.php(459): OC\\Files\\Stream\\Encryption::wrap(Resource id #62, 'foo\/textfile0.t...', '\/user0\/files\/lo...', Array, 'user0', Object(OCA\\Encryption\\Crypto\\Encryption), Object(OC\\Files\\Storage\\Local), Object(OC\\Files\\Storage\\Wrapper\\Encryption), Object(OC\\Encryption\\Util), Object(OC\\Encryption\\File), 'w', 0, 0, 0, true)\n#7 \/var\/www\/html\/encryption_vincent\/lib\/private\/Files\/Storage\/Wrapper\/Encryption.php(743): OC\\Files\\Storage\\Wrapper\\Encryption->fopen('foo\/textfile0.t...', 'w')\n#8 \/var\/www\/html\/encryption_vincent\/lib\/private\/Files\/Storage\/Wrapper\/Encryption.php(613): OC\\Files\\Storage\\Wrapper\\Encryption->copyBetweenStorage(Object(OCA\\Files_Trashbin\\Storage), 'files\/textfile0...', 'foo\/textfile0.t...', true, true)\n#9 \/var\/www\/html\/encryption_vincent\/lib\/private\/Files\/Storage\/Wrapper\/Wrapper.php(568): OC\\Files\\Storage\\Wrapper\\Encryption->moveFromStorage(Object(OCA\\Files_Trashbin\\Storage), 'files\/textfile0...', 'foo\/textfile0.t...')\n#10 \/var\/www\/html\/encryption_vincent\/lib\/private\/Files\/View.php(789): OC\\Files\\Storage\\Wrapper\\Wrapper->moveFromStorage(Object(OCA\\Files_Trashbin\\Storage), 'files\/textfile0...', 'foo\/textfile0.t...')\n#11 \/var\/www\/html\/encryption_vincent\/apps\/dav\/lib\/Connector\/Sabre\/ObjectTree.php(262): OC\\Files\\View->rename('\/textfile0.txt', '\/local_storage\/...')\n#12 \/var\/www\/html\/encryption_vincent\/lib\/composer\/sabre\/dav\/lib\/DAV\/CorePlugin.php(642): OCA\\DAV\\Connector\\Sabre\\ObjectTree->move('textfile0.txt', 'local_storage\/f...')\n#13 [internal function]: Sabre\\DAV\\CorePlugin->httpMove(Object(Sabre\\HTTP\\Request), Object(Sabre\\HTTP\\Response))\n#14 \/var\/www\/html\/encryption_vincent\/lib\/composer\/sabre\/event\/lib\/EventEmitterTrait.php(105): call_user_func_array(Array, Array)\n#15 \/var\/www\/html\/encryption_vincent\/lib\/composer\/sabre\/dav\/lib\/DAV\/Server.php(479): Sabre\\Event\\EventEmitter->emit('method:MOVE', Array)\n#16 \/var\/www\/html\/encryption_vincent\/lib\/composer\/sabre\/dav\/lib\/DAV\/Server.php(254): Sabre\\DAV\\Server->invokeMethod(Object(Sabre\\HTTP\\Request), Object(Sabre\\HTTP\\Response))\n#17 \/var\/www\/html\/encryption_vincent\/apps\/dav\/appinfo\/v1\/webdav.php(58): Sabre\\DAV\\Server->exec()\n#18 \/var\/www\/html\/encryption_vincent\/remote.php(164): require_once('\/var\/www\/html\/e...')\n#19 {main}","File":"\/var\/www\/html\/encryption_vincent\/apps\/encryption\/lib\/Crypto\/Crypt.php","Line":653,"User":"user0"}
[Tue Jan 24 12:24:52 2017] 127.0.0.1:37838 [500]: /remote.php/webdav/textfile0.txt
    And User "user0" moved file "/textfile0.txt" to "/local_storage/foo/textfile0.txt" # FeatureContext::userMovedFile()
      Server error response [url] http://localhost:8080/remote.php/webdav/textfile0.txt [status code] 500 [reason phrase] Internal Server Error (GuzzleHttp\Exception\ServerException)

PVince81 · 2017-01-24T13:18:23Z

Let's investigate if the env is properly cleant up, maybe something remains between runs.

PVince81 · 2017-01-24T13:43:33Z

Known issue with ext storages and encryption: deleting an ext storage leaves stray keys: #10371
What was not known is that such keys could cause further issues when recreating the storage.
To be addressed separately.

Plan: disable enc tests with local storage on master and cherry-pick the fix to this PR and the stable9 one

PVince81 · 2017-01-24T14:54:09Z

Cherry-picked #27015, hopefully all will pass on Jenkins at least.

SergioBertolinSG · 2017-01-24T17:30:53Z

I've run locally the remaining case using only encryption. All tests have passed.

Added filter to disable some tests when encryption is enabled.

…removes it.

Because encryption leaves stray keys that sabotage subsequent runs. Needs to be fixed separately.

PVince81 · 2017-01-25T13:06:48Z

and you've removed the "TEMP" commit now, thanks.

Merging.

lock · 2019-08-04T01:01:51Z

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

PVince81 added the 2 - Developing label Jan 20, 2017

PVince81 added this to the 9.1.4 milestone Jan 20, 2017

PVince81 mentioned this pull request Jan 20, 2017

Integration test encryption homedir #26844

Merged

SergioBertolinSG added the dev:integration-tests label Jan 23, 2017

PVince81 force-pushed the stable9.1-integration-test-encryption-homedir branch from 4829120 to 0874ed3 Compare January 23, 2017 19:32

PVince81 mentioned this pull request Jan 23, 2017

Stable9.1 tests md5 homedir #27003

Closed

PVince81 mentioned this pull request Jan 23, 2017

[stable9] Integration test encryption homedir #26996

Merged

PVince81 assigned SergioBertolinSG Jan 24, 2017

PVince81 mentioned this pull request Jan 24, 2017

Disable encryption integration tests with external storage #27015

Merged

SergioBertolinSG and others added 17 commits January 25, 2017 10:11

Added the option to run tests using md5 home

6eb0991

Start using prefixes

def0b2b

Added alt home execution without makefiles for stable9.1

3858744

Added support for homes which doesn't have user's name

cdba956

Changed data directory for work one

5cd9d0f

Using propfind instead of accessing the file system

a0345c9

Added phpdoc for getUserHome

f2a76cd

fixed alt_home functions names

eb67495

Fixed federation tests

36b6971

Do not take into account testing app in enabled apps scenario

0a39ecf

Add integration test stage with encryption enabled

2fb9b5f

Add no_encryption filter for incompatible int tests

280cb22

Added filter to disable some tests when encryption is enabled.

Make sure we could create the work folder in int tests

01e4f1b

Fix for Reshared files can be still accessed if a user in the middle …

9b8104a

…removes it.

Fixed scenario getting all shares of a user using that user

46346b1

Fixed scenario getting share info of a share

29c831c

Disable encryption integration tests with external storage

df6e2a5

Because encryption leaves stray keys that sabotage subsequent runs. Needs to be fixed separately.

SergioBertolinSG force-pushed the stable9.1-integration-test-encryption-homedir branch from 5a0a2f5 to df6e2a5 Compare January 25, 2017 10:11

PVince81 merged commit 335a398 into stable9.1 Jan 25, 2017

PVince81 deleted the stable9.1-integration-test-encryption-homedir branch January 25, 2017 13:06

lock Bot locked as resolved and limited conversation to collaborators Aug 4, 2019

Uh oh!

Conversation

PVince81 commented Jan 20, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SergioBertolinSG commented Jan 23, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SergioBertolinSG commented Jan 23, 2017

Uh oh!

SergioBertolinSG commented Jan 23, 2017

Uh oh!

PVince81 commented Jan 23, 2017

Uh oh!

PVince81 commented Jan 24, 2017 • edited by SergioBertolinSG Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DeepDiver1975 commented Jan 24, 2017

Uh oh!

SergioBertolinSG commented Jan 24, 2017

Uh oh!

PVince81 commented Jan 24, 2017

Uh oh!

SergioBertolinSG commented Jan 24, 2017

Uh oh!

PVince81 commented Jan 24, 2017

Uh oh!

PVince81 commented Jan 24, 2017

Uh oh!

PVince81 commented Jan 24, 2017

Uh oh!

SergioBertolinSG commented Jan 24, 2017

Uh oh!

SergioBertolinSG commented Jan 24, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SergioBertolinSG commented Jan 24, 2017

Uh oh!

SergioBertolinSG commented Jan 24, 2017

Uh oh!

SergioBertolinSG commented Jan 24, 2017

Uh oh!

PVince81 commented Jan 24, 2017

Uh oh!

PVince81 commented Jan 24, 2017

Uh oh!

PVince81 commented Jan 24, 2017

Uh oh!

SergioBertolinSG commented Jan 24, 2017

Uh oh!

PVince81 commented Jan 25, 2017

Uh oh!

lock Bot commented Aug 4, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

PVince81 commented Jan 20, 2017 •

edited

Loading

SergioBertolinSG commented Jan 23, 2017 •

edited

Loading

PVince81 commented Jan 24, 2017 •

edited by SergioBertolinSG

Loading

SergioBertolinSG commented Jan 24, 2017 •

edited

Loading