Enable testing env for alternative home folder formats by PVince81 · Pull Request #26846 · owncloud/core

PVince81 · 2016-12-20T11:44:16Z

This is to see if integration tests pass when the user's home is md5'd.

A similar behavior would happen when selecting a home directory attribute with LDAP in which case the mounted home folder in ownCloud's virtual FS doesn't match the real filesystem.

There could be some bugs where initMountPoint() was forgotten but it did work correctly with regular homes due to one to one name matching. This PR will tell us if there are other affected code paths...
One known affected code path is #26820

Let's see how it goes...

Eventually I think we need an app or config.php option to be able to set/simulate this behavior when running unit tests and integratin tests. @DeepDiver1975 @owncloud/qa

mention-bot · 2016-12-20T11:44:18Z

@PVince81, thanks for your PR! By analyzing the history of the files in this pull request, we identified @DeepDiver1975, @xoen and @butonic to be potential reviewers.

PVince81 · 2016-12-20T14:15:49Z

Boom 💥 boom 💥


    /var/lib/jenkins/workspace/owncloud-core_core_PR-26846-XG37F5ZPIKDMJCLOLP7CE5XC7QBMUABMV7UOJUV3KVW5ZE3635QA/build/integration/features/checksums.feature:27
    /var/lib/jenkins/workspace/owncloud-core_core_PR-26846-XG37F5ZPIKDMJCLOLP7CE5XC7QBMUABMV7UOJUV3KVW5ZE3635QA/build/integration/features/favorites.feature:121
    /var/lib/jenkins/workspace/owncloud-core_core_PR-26846-XG37F5ZPIKDMJCLOLP7CE5XC7QBMUABMV7UOJUV3KVW5ZE3635QA/build/integration/features/sharing-v1.feature:474
    /var/lib/jenkins/workspace/owncloud-core_core_PR-26846-XG37F5ZPIKDMJCLOLP7CE5XC7QBMUABMV7UOJUV3KVW5ZE3635QA/build/integration/features/sharing-v1.feature:494
    /var/lib/jenkins/workspace/owncloud-core_core_PR-26846-XG37F5ZPIKDMJCLOLP7CE5XC7QBMUABMV7UOJUV3KVW5ZE3635QA/build/integration/features/sharing-v1.feature:554
    /var/lib/jenkins/workspace/owncloud-core_core_PR-26846-XG37F5ZPIKDMJCLOLP7CE5XC7QBMUABMV7UOJUV3KVW5ZE3635QA/build/integration/features/sharing-v1.feature:566
    /var/lib/jenkins/workspace/owncloud-core_core_PR-26846-XG37F5ZPIKDMJCLOLP7CE5XC7QBMUABMV7UOJUV3KVW5ZE3635QA/build/integration/features/sharing-v1.feature:646
    /var/lib/jenkins/workspace/owncloud-core_core_PR-26846-XG37F5ZPIKDMJCLOLP7CE5XC7QBMUABMV7UOJUV3KVW5ZE3635QA/build/integration/features/sharing-v1.feature:655
    /var/lib/jenkins/workspace/owncloud-core_core_PR-26846-XG37F5ZPIKDMJCLOLP7CE5XC7QBMUABMV7UOJUV3KVW5ZE3635QA/build/integration/features/sharing-v1.feature:667
    /var/lib/jenkins/workspace/owncloud-core_core_PR-26846-XG37F5ZPIKDMJCLOLP7CE5XC7QBMUABMV7UOJUV3KVW5ZE3635QA/build/integration/features/sharing-v1.feature:679
    /var/lib/jenkins/workspace/owncloud-core_core_PR-26846-XG37F5ZPIKDMJCLOLP7CE5XC7QBMUABMV7UOJUV3KVW5ZE3635QA/build/integration/features/sharing-v1.feature:691
    /var/lib/jenkins/workspace/owncloud-core_core_PR-26846-XG37F5ZPIKDMJCLOLP7CE5XC7QBMUABMV7UOJUV3KVW5ZE3635QA/build/integration/features/sharing-v1.feature:857
    /var/lib/jenkins/workspace/owncloud-core_core_PR-26846-XG37F5ZPIKDMJCLOLP7CE5XC7QBMUABMV7UOJUV3KVW5ZE3635QA/build/integration/features/sharing-v1.feature:879
    /var/lib/jenkins/workspace/owncloud-core_core_PR-26846-XG37F5ZPIKDMJCLOLP7CE5XC7QBMUABMV7UOJUV3KVW5ZE3635QA/build/integration/features/sharing-v1.feature:921
    /var/lib/jenkins/workspace/owncloud-core_core_PR-26846-XG37F5ZPIKDMJCLOLP7CE5XC7QBMUABMV7UOJUV3KVW5ZE3635QA/build/integration/features/sharing-v1.feature:996
    /var/lib/jenkins/workspace/owncloud-core_core_PR-26846-XG37F5ZPIKDMJCLOLP7CE5XC7QBMUABMV7UOJUV3KVW5ZE3635QA/build/integration/features/sharing-v1.feature:1013
    /var/lib/jenkins/workspace/owncloud-core_core_PR-26846-XG37F5ZPIKDMJCLOLP7CE5XC7QBMUABMV7UOJUV3KVW5ZE3635QA/build/integration/features/sharing-v1.feature:1030
    /var/lib/jenkins/workspace/owncloud-core_core_PR-26846-XG37F5ZPIKDMJCLOLP7CE5XC7QBMUABMV7UOJUV3KVW5ZE3635QA/build/integration/features/sharing-v1.feature:1047
    /var/lib/jenkins/workspace/owncloud-core_core_PR-26846-XG37F5ZPIKDMJCLOLP7CE5XC7QBMUABMV7UOJUV3KVW5ZE3635QA/build/integration/features/sharing-v1.feature:1064
    /var/lib/jenkins/workspace/owncloud-core_core_PR-26846-XG37F5ZPIKDMJCLOLP7CE5XC7QBMUABMV7UOJUV3KVW5ZE3635QA/build/integration/features/sharing-v1.feature:1079
    /var/lib/jenkins/workspace/owncloud-core_core_PR-26846-XG37F5ZPIKDMJCLOLP7CE5XC7QBMUABMV7UOJUV3KVW5ZE3635QA/build/integration/features/webdav-related.feature:57
    /var/lib/jenkins/workspace/owncloud-core_core_PR-26846-XG37F5ZPIKDMJCLOLP7CE5XC7QBMUABMV7UOJUV3KVW5ZE3635QA/build/integration/features/webdav-related.feature:75
    /var/lib/jenkins/workspace/owncloud-core_core_PR-26846-XG37F5ZPIKDMJCLOLP7CE5XC7QBMUABMV7UOJUV3KVW5ZE3635QA/build/integration/features/webdav-related.feature:93
    /var/lib/jenkins/workspace/owncloud-core_core_PR-26846-XG37F5ZPIKDMJCLOLP7CE5XC7QBMUABMV7UOJUV3KVW5ZE3635QA/build/integration/features/webdav-related.feature:111
    /var/lib/jenkins/workspace/owncloud-core_core_PR-26846-XG37F5ZPIKDMJCLOLP7CE5XC7QBMUABMV7UOJUV3KVW5ZE3635QA/build/integration/features/webdav-related.feature:152
    /var/lib/jenkins/workspace/owncloud-core_core_PR-26846-XG37F5ZPIKDMJCLOLP7CE5XC7QBMUABMV7UOJUV3KVW5ZE3635QA/build/integration/features/webdav-related.feature:161
    /var/lib/jenkins/workspace/owncloud-core_core_PR-26846-XG37F5ZPIKDMJCLOLP7CE5XC7QBMUABMV7UOJUV3KVW5ZE3635QA/build/integration/features/webdav-related.feature:179
    /var/lib/jenkins/workspace/owncloud-core_core_PR-26846-XG37F5ZPIKDMJCLOLP7CE5XC7QBMUABMV7UOJUV3KVW5ZE3635QA/build/integration/features/webdav-related.feature:197
    /var/lib/jenkins/workspace/owncloud-core_core_PR-26846-XG37F5ZPIKDMJCLOLP7CE5XC7QBMUABMV7UOJUV3KVW5ZE3635QA/build/integration/features/webdav-related.feature:207
    /var/lib/jenkins/workspace/owncloud-core_core_PR-26846-XG37F5ZPIKDMJCLOLP7CE5XC7QBMUABMV7UOJUV3KVW5ZE3635QA/build/integration/features/webdav-related.feature:308
    /var/lib/jenkins/workspace/owncloud-core_core_PR-26846-XG37F5ZPIKDMJCLOLP7CE5XC7QBMUABMV7UOJUV3KVW5ZE3635QA/build/integration/features/webdav-related.feature:322
    /var/lib/jenkins/workspace/owncloud-core_core_PR-26846-XG37F5ZPIKDMJCLOLP7CE5XC7QBMUABMV7UOJUV3KVW5ZE3635QA/build/integration/features/webdav-related.feature:336
    /var/lib/jenkins/workspace/owncloud-core_core_PR-26846-XG37F5ZPIKDMJCLOLP7CE5XC7QBMUABMV7UOJUV3KVW5ZE3635QA/build/integration/features/webdav-related.feature:348
    /var/lib/jenkins/workspace/owncloud-core_core_PR-26846-XG37F5ZPIKDMJCLOLP7CE5XC7QBMUABMV7UOJUV3KVW5ZE3635QA/build/integration/federation_features/federated.feature:123
    /var/lib/jenkins/workspace/owncloud-core_core_PR-26846-XG37F5ZPIKDMJCLOLP7CE5XC7QBMUABMV7UOJUV3KVW5ZE3635QA/build/integration/federation_features/federated.feature:138

PVince81 · 2016-12-20T15:01:38Z

So far it looks like the tests do not properly clean up. Maybe the code that deletes the folders isn't actually deleting the users but trying to delete the matching folder in the data folder. In the case of the md5 home the name wouldn't match.

Investigation continues...

PVince81 · 2016-12-20T15:10:15Z

Found something: looks like the provisioning API doesn't manage to properly delete the user's home.
Somehow getHome() doesn't get called, something wrong with implementsActions.

PVince81 · 2016-12-20T15:11:36Z

Okay, here's the WTF: getHome() fails because the user is already deleted at the time we want to delete their home... So we need to call getHome() earlier...

PVince81 · 2016-12-20T15:13:43Z

Fixed in ce3ae8b, I'll make a separate PR for this later.

PVince81 · 2016-12-20T15:33:03Z

I've put the fix into a separate PR: #26848

PVince81 · 2016-12-20T17:01:09Z

Remaining failures:

    /var/lib/jenkins/workspace/owncloud-core_core_PR-26846-XG37F5ZPIKDMJCLOLP7CE5XC7QBMUABMV7UOJUV3KVW5ZE3635QA/build/integration/features/webdav-related.feature:197
    /var/lib/jenkins/workspace/owncloud-core_core_PR-26846-XG37F5ZPIKDMJCLOLP7CE5XC7QBMUABMV7UOJUV3KVW5ZE3635QA/build/integration/features/webdav-related.feature:207
    /var/lib/jenkins/workspace/owncloud-core_core_PR-26846-XG37F5ZPIKDMJCLOLP7CE5XC7QBMUABMV7UOJUV3KVW5ZE3635QA/build/integration/federation_features/federated.feature:123
    /var/lib/jenkins/workspace/owncloud-core_core_PR-26846-XG37F5ZPIKDMJCLOLP7CE5XC7QBMUABMV7UOJUV3KVW5ZE3635QA/build/integration/federation_features/federated.feature:138

PVince81 · 2016-12-20T17:25:08Z

@SergioBertolinSG this is wrong: https://github.com/owncloud/core/blob/v9.1.3/build/integration/features/bootstrap/WebDav.php#L438

It assumes that $user is the name of the folder, but it could be something else like the md5.

It looks like the provisioning API does not return the user's home folder which is bad, so the tests have no way to properly clear it. Maybe there's a workaround ?

PVince81 · 2016-12-20T17:32:33Z

@SergioBertolinSG also don't put arbitrary files inside the "data" folder, that one should be kept only for the tested ownCloud instance. Use the "work" folder for that instead.

Please have a look. There must be a way to test this differently. Use Webdav to check if the file was properly uploaded.

PVince81 · 2016-12-20T17:55:32Z

PR to add a oc_preferences setting to override the home folder when using the database backend: #26851
This would make it possible to setup tests automatically without the MD5 hack.

PVince81 · 2016-12-21T08:51:09Z

This PR adds the "home" in the provisioning API response: #26853

PVince81 · 2016-12-22T08:35:59Z

Next steps:

fix tests to not directly access data folder as per Enable testing env for alternative home folder formats #26846 (comment)
adjust this PR to use the non-hacky approach from Testing md5home #26854.
adjust Jenkins file to have a new test case to use the md5 home: the run.sh script might need a flag to enable the testing app with occ app:enable and occ config:... to set the AlternativeUserBackend`.

Later on we'll also add a flag for enabling encryption. I have the feeling that we should revamp run.sh to only be about running tests and use another external method/script to setup the envs. Maybe we could have separate scripts for each env:

env/env_setupExtStorage.sh
env/env_enableMd5Home.sh
env/env_enableEncryption.sh

Thoughts ?

SergioBertolinSG · 2016-12-22T12:28:56Z

Perhaps we can just improve run.sh using functions and more arguments when calling it.

PVince81 · 2016-12-22T12:35:58Z

Perhaps we can just improve run.sh using functions and more arguments when calling it.

Yes, that would also work. Note that in some cases preparing an env and setting behat filters can take several lines of code like the ones from encryption, so need to take this into account.

SergioBertolinSG · 2016-12-22T13:06:23Z

the testing app with occ app:enable

Would that affect capabilities feature? it is enabled there in each scenario.

PVince81 · 2016-12-22T13:20:55Z

@SergioBertolinSG it won't I think. The user backend is only enabled if you specify it in config.php.

PVince81 · 2016-12-23T06:17:44Z

@SergioBertolinSG in a chat, @DeepDiver1975 pointed out that one could create a config file "test.config.php" and it would get integrated automatically. But this means that the integration tests need to fiddle within the ownCloud instance automatically without using APIs. I already don't like the fact that integration tests are writing stuff into the skeleton folder.

(the reason I dislike is because I run Apache as another user than my dev user, so I need to sudo -u wwwrun ./run.sh when running the tests so that the written files receive the correct perms).

Here's a PR to be able to override with a single oc_preferences #26867.

PVince81 · 2016-12-23T09:30:50Z

Since we'll likely have a lot of functions in the future for many scenarios, I suggest we start using prefixed.
For example: env_althome_enable, env_althome_clear. And in my other PR I could do the same with env_encryption_enable.

PVince81 · 2016-12-23T09:31:31Z

I suggest to use prefixed for env variables as well, how about OC_TEST_ALT_STORAGE=1 (and in my other PR it would be OC_TEST_ENCRYPTION=1

PVince81 · 2016-12-23T09:41:23Z

TEST_ALTHOME to be consistent with the functions

SergioBertolinSG · 2016-12-23T10:06:34Z

Last commit will bring conflicts to #26586.

PVince81 · 2016-12-23T11:02:30Z

Please rebase, the other PR was merged. But I can't explain why there is a conflict...

PVince81 · 2017-01-09T15:36:06Z

Now @DeepDiver1975 has moved the integration tests to run only on stable/master branches, so need to rebase and adjust Jenkinsfile.

But this also means that we won't see in this PR whether the tests pass.

PVince81 · 2017-01-09T15:36:24Z

@SergioBertolinSG please rebase

SergioBertolinSG · 2017-01-10T11:17:11Z

Integration tests passed locally.

PVince81

Please adjust Jenkinsfile then this is good to go 👍

I expect this to clash with the encryption integration tests but that's ok. We'll need to rebase that other PR to use a similar approach (functions) for setting up the encryption env.

PVince81 · 2017-01-10T11:27:38Z

            }
-     }
+
+            executeAndReport('build/integration/output/*.xml') {


Please move this into the isOnReleaseBranch() block

It is already there, isn't?

core/Jenkinsfile

Line 119 in 774028d

if (isOnReleaseBranch()) {

Indeed... I misparsed the diff and saw a wrong indent.
Please ignore my comment

PVince81 · 2017-01-10T11:29:56Z

@SergioBertolinSG can you also rebase and delete the two first commits ?

SergioBertolinSG · 2017-01-10T11:33:31Z

@SergioBertolinSG can you also rebase and delete the two first commits ?

Sure.

PVince81 · 2017-01-10T12:18:48Z

Please remove the transifex commit

PVince81 · 2017-01-10T16:18:39Z

The tests passed before, merging now

PVince81 · 2017-01-10T17:56:30Z

 $OCC files_external:option $ID_STORAGE enable_sharing true

+if test "$OC_TEST_ALT_HOME" = "1"; then
+	env_althome_enable


this cannot work, the proper name is env_alt_home_enable.

@SergioBertolinSG but you said you ran the tests locally and they passed ? 😦

had to fix it in my other PR: 49ed380.

PVince81 · 2017-01-10T18:42:44Z

Let's backport this together with #26844 once #26844 is merged

lock · 2019-08-04T04:02:32Z

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

PVince81 added the dev:integration-tests label Dec 20, 2016

PVince81 self-assigned this Dec 20, 2016

PVince81 mentioned this pull request Dec 20, 2016

Integration test encryption homedir #26844

Merged

PVince81 mentioned this pull request Dec 20, 2016

Get user home folder before deletion #26848

Merged

9 tasks

PVince81 assigned SergioBertolinSG Dec 20, 2016

PVince81 mentioned this pull request Dec 20, 2016

Set user home path in oc_preferences #26851

Closed

9 tasks

PVince81 mentioned this pull request Dec 21, 2016

Add home storage location in provisioning API response #26853

Merged

9 tasks

PVince81 mentioned this pull request Dec 21, 2016

Testing md5home #26854

Merged

9 tasks

PVince81 changed the title ~~[TEMP] md5 the user home for testing~~ Enable testing env for alternative home folder formats Dec 22, 2016

PVince81 added this to the 10.0 milestone Dec 22, 2016

PVince81 added the backport-request label Dec 22, 2016

SergioBertolinSG mentioned this pull request Dec 22, 2016

Integration tests allowing md5 home #26865

Merged

PVince81 commented Dec 23, 2016

View reviewed changes

Comment thread build/integration/run.sh Outdated

PVince81 Dec 23, 2016

Copy link
Copy Markdown

Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TEST_ALTHOME to be consistent with the functions

SergioBertolinSG force-pushed the tests-md5-homedir branch from 8b30182 to 5b8c2a1 Compare December 23, 2016 10:08

SergioBertolinSG force-pushed the tests-md5-homedir branch from 5b8c2a1 to 9abc40c Compare December 23, 2016 11:33

SergioBertolinSG force-pushed the tests-md5-homedir branch from 9abc40c to 774028d Compare January 10, 2017 07:57

PVince81 commented Jan 10, 2017

View reviewed changes

SergioBertolinSG force-pushed the tests-md5-homedir branch from 774028d to fe5a9b7 Compare January 10, 2017 11:40

PVince81 removed their assignment Jan 10, 2017

SergioBertolinSG added 3 commits January 10, 2017 12:29

Added the option to run tests using md5 home

0441f89

Start using prefixes

bd877f3

Added a new execution on jenkins using alternative homes

2fc9276

SergioBertolinSG force-pushed the tests-md5-homedir branch from fe5a9b7 to 2fc9276 Compare January 10, 2017 12:30

PVince81 merged commit 59e3a5f into master Jan 10, 2017

PVince81 deleted the tests-md5-homedir branch January 10, 2017 16:18

PVince81 commented Jan 10, 2017

View reviewed changes

SergioBertolinSG mentioned this pull request Jan 23, 2017

Stable9.1 tests md5 homedir #27003

Closed

PVince81 removed the backport-request label Mar 24, 2017

lock Bot locked as resolved and limited conversation to collaborators Aug 4, 2019

Uh oh!

Conversation

PVince81 commented Dec 20, 2016

Uh oh!

mention-bot commented Dec 20, 2016

Uh oh!

PVince81 commented Dec 20, 2016

Uh oh!

PVince81 commented Dec 20, 2016

Uh oh!

PVince81 commented Dec 20, 2016

Uh oh!

PVince81 commented Dec 20, 2016

Uh oh!

PVince81 commented Dec 20, 2016

Uh oh!

PVince81 commented Dec 20, 2016

Uh oh!

PVince81 commented Dec 20, 2016

Uh oh!

PVince81 commented Dec 20, 2016

Uh oh!

PVince81 commented Dec 20, 2016

Uh oh!

PVince81 commented Dec 20, 2016

Uh oh!

PVince81 commented Dec 21, 2016

Uh oh!

PVince81 commented Dec 22, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SergioBertolinSG commented Dec 22, 2016

Uh oh!

PVince81 commented Dec 22, 2016

Uh oh!

SergioBertolinSG commented Dec 22, 2016

Uh oh!

PVince81 commented Dec 22, 2016

Uh oh!

PVince81 commented Dec 23, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SergioBertolinSG commented Dec 23, 2016

Uh oh!

PVince81 commented Dec 23, 2016

Uh oh!

PVince81 commented Jan 9, 2017

Uh oh!

PVince81 commented Jan 9, 2017

Uh oh!

SergioBertolinSG commented Jan 10, 2017

Uh oh!

PVince81 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PVince81 commented Jan 10, 2017

Uh oh!

SergioBertolinSG commented Jan 10, 2017

Uh oh!

PVince81 commented Jan 10, 2017

Uh oh!

PVince81 commented Jan 10, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PVince81 commented Jan 10, 2017

Uh oh!

PVince81 commented Dec 22, 2016 •

edited

Loading