Integration test encryption homedir by PVince81 · Pull Request #26844 · owncloud/core

PVince81 · 2016-12-20T09:30:19Z

Extension of #26586 but where the home directory name is different from the user id.

It currently uses a hack to md5 the user's home.

Goal is to reproduce the issue from #26820 automatically.

Let's see how it goes...

mention-bot · 2016-12-20T09:30:20Z

@PVince81, thanks for your PR! By analyzing the history of the files in this pull request, we identified @DeepDiver1975, @SergioBertolinSG and @jnweiger to be potential reviewers.

PVince81 · 2016-12-20T11:38:10Z

Ouch, many fails even before we even started enabling encryption:

    checksums.Moving file with checksum should return the checksum in the download header
    sharing.Share a file by multiple channels
    sharing.Delete all group shares
    sharing.Allow modification of reshare
    sharing.Do not allow reshare to exceed permissions
    sharing.Correct webdav share-permissions for received folder with all permissions
    sharing.Correct webdav share-permissions for received folder with all permissions but edit
    sharing.Correct webdav share-permissions for received folder with all permissions but create
    sharing.Correct webdav share-permissions for received folder with all permissions but delete
    sharing.Correct webdav share-permissions for received folder with all permissions but share
    sharing.Merging shares for recipient when shared from outside with user then group and recipient renames in between
    sharing.orphaned shares
    webdav-related.Moving a file to overwrite a file in a folder with no permissions
    webdav-related.Copying a file
    webdav-related.Copying a file to a folder with no permissions
    webdav-related.Copying a file to overwrite a file into a folder with no permissions
    webdav-related.Retrieving folder quota when quota is set
    webdav-related.Retrieving folder quota of shared folder with quota when no quota is set for recipient
    webdav-related.Uploading a file as recipient using webdav having quota
    webdav-related.Retrieving folder quota when quota is set and a file was uploaded
    webdav-related.Retrieving folder quota when quota is set and a file was recieved
    webdav-related.A file that is shared to a user has a share-types property
    webdav-related.A file that is shared to a group has a share-types property
    webdav-related.A file that is shared by link has a share-types property
    webdav-related.A file that is shared by user,group and link has a share-types property
    federated.Overwrite a federated shared file as recipient
    federated.Overwrite a federated shared folder as recipient

It could however be a side-effect.
I'll rerun the tests this time without the encryption-related changes and only with the md5'd home.

PVince81 · 2016-12-20T11:44:44Z

I made a separate PR where only the home is MD5'd: #26846
This will at least make it possible to work on other fixes until we decide how to make this testing fully automatic.

PVince81 · 2016-12-20T15:41:08Z

Requires Get user home folder before deletion #26848 for having proper test cleanups

SergioBertolinSG · 2016-12-20T17:48:23Z

I didn't see it finish in my local environment:

Scenario: Federate share a file with another server                                                    # /var/www/html/master_20-12-16/build/integration/federation_features/federated.feature:5
    Given Using server "REMOTE"                                                                          # FederationContext::usingServer()
./run.sh: line 40: 28695 Killed                  vendor/bin/behat --strict -f junit -f pretty $SCENARIO_TO_RUN
./run.sh: line 46: 28685 Terminated              php -S localhost:$PORT -t ../..
./run.sh: line 46: 28686 Terminated              php -S localhost:$PORT_FED -t ../..
files_external disabled
{"reqId":"wOVIiDwNAXh1kccM+Dsu","remoteAddr":"127.0.0.1","app":"core","message":"Login failed: 'user0' (Remote IP: '127.0.0.1')","level":2,"time":"2016-12-20T17:43:22+00:00","method":"DELETE","url":"\/remote.php\/webdav\/myFileToTag.txt","user":"--"}
{"reqId":"UqvcPZXoz78bKSyiVeIg","remoteAddr":"127.0.0.1","app":"core","message":"Login failed: 'user0' (Remote IP: '127.0.0.1')","level":2,"time":"2016-12-20T17:43:24+00:00","method":"DELETE","url":"\/remote.php\/webdav\/myFileToComment.txt","user":"--"}
{"reqId":"Dg2BJNO41njfpVz+w7ek","remoteAddr":"127.0.0.1","app":"core","message":"Login failed: 'user0' (Remote IP: '127.0.0.1')","level":2,"time":"2016-12-20T17:43:24+00:00","method":"DELETE","url":"\/remote.php\/webdav\/myFileToTag.txt","user":"--"}
{"reqId":"QJ7icWu7ynRA7DrPYqAm","remoteAddr":"127.0.0.1","app":"webdav","message":"Exception: {\"Message\":\"HTTP\\\/1.1 503 OC\\\\User\\\\LoginException: User disabled\",\"Exception\":\"Sabre\\\\DAV\\\\Exception\\\\ServiceUnavailable\",\"Code\":0,\"Trace\":\"#0 \\\/var\\\/www\\\/html\\\/master_20-12-16\\\/lib\\\/composer\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Auth\\\/Plugin.php(199): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\Auth->check(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#1 \\\/var\\\/www\\\/html\\\/master_20-12-16\\\/lib\\\/composer\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Auth\\\/Plugin.php(150): Sabre\\\\DAV\\\\Auth\\\\Plugin->check(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#2 [internal function]: Sabre\\\\DAV\\\\Auth\\\\Plugin->beforeMethod(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#3 \\\/var\\\/www\\\/html\\\/master_20-12-16\\\/lib\\\/composer\\\/sabre\\\/event\\\/lib\\\/EventEmitterTrait.php(105): call_user_func_array(Array, Array)\\n#4 \\\/var\\\/www\\\/html\\\/master_20-12-16\\\/lib\\\/composer\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(466): Sabre\\\\Event\\\\EventEmitter->emit('beforeMethod', Array)\\n#5 \\\/var\\\/www\\\/html\\\/master_20-12-16\\\/lib\\\/composer\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(254): Sabre\\\\DAV\\\\Server->invokeMethod(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#6 \\\/var\\\/www\\\/html\\\/master_20-12-16\\\/apps\\\/dav\\\/appinfo\\\/v1\\\/webdav.php(57): Sabre\\\\DAV\\\\Server->exec()\\n#7 \\\/var\\\/www\\\/html\\\/master_20-12-16\\\/remote.php(164): require_once('\\\/var\\\/www\\\/html\\\/m...')\\n#8 {main}\",\"File\":\"\\\/var\\\/www\\\/html\\\/master_20-12-16\\\/apps\\\/dav\\\/lib\\\/Connector\\\/Sabre\\\/Auth.php\",\"Line\":153,\"User\":false}","level":4,"time":"2016-12-20T17:43:26+00:00","method":"GET","url":"\/remote.php\/webdav\/welcome.txt","user":"--"}
{"reqId":"aWxuucVr1aKMiFuGoBYb","remoteAddr":"127.0.0.1","app":"core","message":"Login failed: 'user0' (Remote IP: '127.0.0.1')","level":2,"time":"2016-12-20T17:43:26+00:00","method":"DELETE","url":"\/remote.php\/webdav\/myFileToComment.txt","user":"--"}
{"reqId":"O3p1TlNtXEmmTbihW4ld","remoteAddr":"127.0.0.1","app":"core","message":"Login failed: 'user0' (Remote IP: '127.0.0.1')","level":2,"time":"2016-12-20T17:43:27+00:00","method":"DELETE","url":"\/remote.php\/webdav\/myFileToTag.txt","user":"--"}
{"reqId":"aeV9rQNHhfXwR\/Tn\/zaB","remoteAddr":"127.0.0.1","app":"core","message":"Login failed: 'user0' (Remote IP: '127.0.0.1')","level":2,"time":"2016-12-20T17:43:28+00:00","method":"DELETE","url":"\/remote.php\/webdav\/myFileToComment.txt","user":"--"}
{"reqId":"OjrXH+Pgjm\/aXg0SnDm4","remoteAddr":"127.0.0.1","app":"core","message":"Login failed: 'user0' (Remote IP: '127.0.0.1')","level":2,"time":"2016-12-20T17:43:28+00:00","method":"DELETE","url":"\/remote.php\/webdav\/myFileToTag.txt","user":"--"}
{"reqId":"BrFOS61rGknOutgMafa0","remoteAddr":"127.0.0.1","app":"core","message":"Login failed: 'user0' (Remote IP: '127.0.0.1')","level":2,"time":"2016-12-20T17:43:30+00:00","method":"DELETE","url":"\/remote.php\/webdav\/myFileToComment.txt","user":"--"}
{"reqId":"nlzlhXdq3lMerdQn6pUX","remoteAddr":"127.0.0.1","app":"core","message":"Login failed: 'user0' (Remote IP: '127.0.0.1')","level":2,"time":"2016-12-20T17:43:30+00:00","method":"DELETE","url":"\/remote.php\/webdav\/myFileToTag.txt","user":"--"}
runsh: Exit code: 137

But I didn't add #26848, is it related?

PVince81 · 2016-12-20T17:56:26Z

Let me add that PR here. I hoped we could quickly get #26848 merged and rebase.

PVince81 · 2016-12-20T17:56:58Z

also I'd prefer if we could get #26586 merged first which introduces the encryption tests in the first place.

PVince81 · 2016-12-20T17:59:39Z

I've added #26848 now here.

Ideally we shouldn't see more failures than in the encryption-less test here #26846. But I expect we should see more failures, some being related to #26820. If they don't, we'll need to add more tests to make sure to cover that PR's scope.

PVince81 · 2017-01-10T16:33:04Z

Rebased and adjusted.

See how the test matrix for integration tests is exploding

PVince81 · 2017-01-10T17:09:15Z

Argh, I need to remove the hack commit

PVince81 · 2017-01-10T17:11:04Z

Fixes owncloud/QA#324

PVince81 · 2017-01-10T17:12:54Z

Now I hope that there is a failing test that matches the scenario from #26820

PVince81 · 2017-01-10T18:25:19Z

Aha, finally some results after running OC_TEST_ENCRYPTION_ENABLED=1 OC_TEST_ALT_HOME=1 ./run.sh features/sharing-v1.feature locally:

    /srv/www/htdocs/owncloud/build/integration/features/sharing-v1.feature:269
    /srv/www/htdocs/owncloud/build/integration/features/sharing-v1.feature:453

For the first one:

[Tue Jan 10 19:23:25 2017] 127.0.0.1:34530 [204]: /remote.php/webdav/textfile0%20(2).txt
    When User "user1" deletes file "/textfile0 (2).txt"                              # FeatureContext::userDeletesFile()
    And As an "user3"                                                                # FeatureContext::asAn()
[Tue Jan 10 19:23:26 2017] Can not decrypt this file, probably this is a shared file. Please ask the file owner to reshare the file with you.
[Tue Jan 10 19:23:26 2017] Exception: {"Message":"Can not decrypt this file, probably this is a shared file. Please ask the file owner to reshare the file with you.","Exception":"OC\\Encryption\\Exceptions\\DecryptionFailedException","Code":0,"Trace":"#0 \/srv\/www\/htdocs\/owncloud\/lib\/private\/Files\/Stream\/Encryption.php(459): OCA\\Encryption\\Crypto\\Encryption->decrypt('hONVoITLn+\/yEvL...', '0end')\n#1 \/srv\/www\/htdocs\/owncloud\/lib\/private\/Files\/Stream\/Encryption.php(290): OC\\Files\\Stream\\Encryption->readCache()\n#2 [internal function]: OC\\Files\\Stream\\Encryption->stream_read(23)\n#3 \/srv\/www\/htdocs\/owncloud\/apps\/files_external\/3rdparty\/icewind\/streams\/src\/Wrapper.php(83): fread(Resource id #559, 8192)\n#4 \/srv\/www\/htdocs\/owncloud\/apps\/files_external\/3rdparty\/icewind\/streams\/src\/CallbackWrapper.php(91): Icewind\\Streams\\Wrapper->stream_read(8192)\n#5 [internal function]: Icewind\\Streams\\CallbackWrapper->stream_read(8192)\n#6 \/srv\/www\/htdocs\/owncloud\/lib\/composer\/sabre\/http\/lib\/Sapi.php(78): stream_copy_to_stream(Resource id #563, Resource id #573, '7')\n#7 \/srv\/www\/htdocs\/owncloud\/lib\/composer\/sabre\/dav\/lib\/DAV\/Server.php(498): Sabre\\HTTP\\Sapi::sendResponse(Object(Sabre\\HTTP\\Response))\n#8 \/srv\/www\/htdocs\/owncloud\/lib\/composer\/sabre\/dav\/lib\/DAV\/Server.php(254): Sabre\\DAV\\Server->invokeMethod(Object(Sabre\\HTTP\\Request), Object(Sabre\\HTTP\\Response))\n#9 \/srv\/www\/htdocs\/owncloud\/apps\/dav\/appinfo\/v1\/webdav.php(57): Sabre\\DAV\\Server->exec()\n#10 \/srv\/www\/htdocs\/owncloud\/remote.php(164): require_once('\/srv\/www\/htdocs...')\n#11 {main}","File":"\/srv\/www\/htdocs\/owncloud\/apps\/encryption\/lib\/Crypto\/Encryption.php","Line":360,"User":"user3"}
[Tue Jan 10 19:23:26 2017] 127.0.0.1:34552 [500]: /remote.php/webdav/textfile0%20(2).txt
    And Downloading file "/textfile0 (2).txt" with range "bytes=1-7"                 # FeatureContext::downloadFileWithRange()
      Server error response [url] http://localhost:8080/remote.php/webdav/textfile0%20(2).txt [status code] 500 [reason phrase] Internal Server Error (GuzzleHttp\Exception\ServerException)
    Then Downloaded content should be "wnCloud"                                      # FeatureContext::downloadedContentShouldBe()

and the second one:

[Tue Jan 10 19:24:27 2017] 127.0.0.1:35744 [201]: /remote.php/webdav/textfile0.txt
    And User "user1" moved file "/textfile0.txt" to "/common/textfile0.txt"                                     # FeatureContext::userMovedFile()
[Tue Jan 10 19:24:28 2017] Could not encrypt file for user2: Public Key missing for user: user2
[Tue Jan 10 19:24:28 2017] 127.0.0.1:35766 [201]: /remote.php/webdav/common/textfile0.txt
    And User "user1" moved file "/common/textfile0.txt" to "/common/sub/textfile0.txt"                          # FeatureContext::userMovedFile()
    And As an "user2"                                                                                           # FeatureContext::asAn()
[Tue Jan 10 19:24:28 2017] Can not decrypt this file, probably this is a shared file. Please ask the file owner to reshare the file with you.
[Tue Jan 10 19:24:28 2017] Exception: {"Message":"Can not decrypt this file, probably this is a shared file. Please ask the file owner to reshare the file with you.","Exception":"OC\\Encryption\\Exceptions\\DecryptionFailedException","Code":0,"Trace":"#0 \/srv\/www\/htdocs\/owncloud\/lib\/private\/Files\/Stream\/Encryption.php(459): OCA\\Encryption\\Crypto\\Encryption->decrypt('lGceeUTJeo\/pGEs...', '0end')\n#1 \/srv\/www\/htdocs\/owncloud\/lib\/private\/Files\/Stream\/Encryption.php(290): OC\\Files\\Stream\\Encryption->readCache()\n#2 [internal function]: OC\\Files\\Stream\\Encryption->stream_read(15)\n#3 \/srv\/www\/htdocs\/owncloud\/apps\/files_external\/3rdparty\/icewind\/streams\/src\/Wrapper.php(83): fread(Resource id #561, 8192)\n#4 \/srv\/www\/htdocs\/owncloud\/apps\/files_external\/3rdparty\/icewind\/streams\/src\/CallbackWrapper.php(91): Icewind\\Streams\\Wrapper->stream_read(8192)\n#5 [internal function]: Icewind\\Streams\\CallbackWrapper->stream_read(8192)\n#6 \/srv\/www\/htdocs\/owncloud\/lib\/composer\/sabre\/http\/lib\/Sapi.php(78): stream_copy_to_stream(Resource id #565, Resource id #575, '9')\n#7 \/srv\/www\/htdocs\/owncloud\/lib\/composer\/sabre\/dav\/lib\/DAV\/Server.php(498): Sabre\\HTTP\\Sapi::sendResponse(Object(Sabre\\HTTP\\Response))\n#8 \/srv\/www\/htdocs\/owncloud\/lib\/composer\/sabre\/dav\/lib\/DAV\/Server.php(254): Sabre\\DAV\\Server->invokeMethod(Object(Sabre\\HTTP\\Request), Object(Sabre\\HTTP\\Response))\n#9 \/srv\/www\/htdocs\/owncloud\/apps\/dav\/appinfo\/v1\/webdav.php(57): Sabre\\DAV\\Server->exec()\n#10 \/srv\/www\/htdocs\/owncloud\/remote.php(164): require_once('\/srv\/www\/htdocs...')\n#11 {main}","File":"\/srv\/www\/htdocs\/owncloud\/apps\/encryption\/lib\/Crypto\/Encryption.php","Line":360,"User":"user2"}
[Tue Jan 10 19:24:28 2017] 127.0.0.1:35788 [500]: /remote.php/webdav/common/sub/textfile0.txt
    When Downloading file "/common/sub/textfile0.txt" with range "bytes=9-17"                                   # FeatureContext::downloadFileWithRange()
      Server error response [url] http://localhost:8080/remote.php/webdav/common/sub/textfile0.txt [status code] 500 [reason phrase] Internal Server Error (GuzzleHttp\Exception\ServerException)
    Then Downloaded content should be "test text"                                                               # FeatureContext::downloadedContentShouldBe()
    And Downloaded content when downloading file "/textfile0.txt" with range "bytes=9-17" should be "test text" # FeatureContext::downloadedContentWhenDownloadindShouldBe()
    And user "user2" should see following elements                                                              # FeatureContext::checkElementList()
      | /common/sub/textfile0.txt |

Now let's see if these pass with #26824

PVince81 · 2017-01-10T18:28:05Z

Yes, the tests from above pass if I cherry-pick the changes from #26824 to this PR. This is what I wanted to cover.

PVince81 · 2017-01-10T18:29:46Z

Fixed the Jenkinsfile to use the correct env variable.

@SergioBertolinSG please review this PR.

PVince81 · 2017-01-10T18:30:36Z

argh, missed this one

PVince81 · 2017-01-13T16:04:11Z

#26938 was merged. I have rebased to include it.

@SergioBertolinSG rerun all tests locally ? Hopefully they'll all pass now.

SergioBertolinSG · 2017-01-17T11:04:40Z

Test run locally, they worked fine. (Both with encryption and alt home and without).

PVince81 · 2017-01-17T16:12:29Z

Rebased for CI

PVince81 · 2017-01-17T17:54:17Z

WTF, now unrelated unit tests fail

Test\Encryption\Keys\StorageTest::testGetUserKeyWhenKeyStorageIsOutsideHome
mounting was not necessary
Failed asserting that true is false.

/var/lib/jenkins/workspace/owncloud-core_core_PR-26844-75FHI25UJEAZQY24FCAWRPW7SHNMBAWRD4CUOZ4EEGK72KF2R3HA/tests/lib/Encryption/Keys/StorageTest.php:310

PVince81 · 2017-01-17T17:58:55Z

Reproduceable locally, I'll have a look

Added filter to disable some tests when encryption is enabled.

…removes it.

PVince81 · 2017-01-18T11:13:05Z

Removed old bogus encryption fix that we cherry-picked onto this branch for testing but that is obsolete and fixed differently on master.

👍 for @SergioBertolinSG's changes.

Let's get this in.

PVince81 · 2017-01-20T14:49:09Z

stable9.1: #26993
stable9: #26996

SergioBertolinSG · 2017-01-23T10:45:45Z

Requires to backport first #26846, #26920 and #26865

lock · 2019-08-04T01:02:39Z

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

PVince81 added the dev:integration-tests label Dec 20, 2016

PVince81 added this to the 10.0 milestone Dec 20, 2016

PVince81 self-assigned this Dec 20, 2016

PVince81 mentioned this pull request Dec 20, 2016

[stable9] Setupfs before access a users keys #26820

Merged

9 tasks

PVince81 force-pushed the integration-test-encryption-homedir branch from b74d120 to 03c4ac3 Compare December 20, 2016 11:45

PVince81 force-pushed the integration-test-encryption-homedir branch from a0cf8f7 to 50c3e40 Compare January 10, 2017 16:32

PVince81 force-pushed the integration-test-encryption-homedir branch from 50c3e40 to 1e0bb7b Compare January 10, 2017 17:10

PVince81 mentioned this pull request Jan 10, 2017

Add integration test stage with encryption enabled #26586

Closed

10 tasks

PVince81 force-pushed the integration-test-encryption-homedir branch from 49ed380 to ad0c2b6 Compare January 10, 2017 18:29

PVince81 commented Jan 10, 2017

View reviewed changes

Comment thread build/integration/Makefile Outdated

PVince81 Jan 10, 2017

Copy link
Copy Markdown

Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

argh, missed this one

PVince81 force-pushed the integration-test-encryption-homedir branch from ad0c2b6 to f4ffcdd Compare January 10, 2017 18:31

PVince81 mentioned this pull request Jan 10, 2017

[stable9.1] Setupfs before access a users keys #26824

Merged

PVince81 requested a review from SergioBertolinSG January 10, 2017 18:41

PVince81 added the backport-request label Jan 10, 2017

PVince81 mentioned this pull request Jan 10, 2017

Enable testing env for alternative home folder formats #26846

Merged

PVince81 force-pushed the integration-test-encryption-homedir branch from de8c79a to 6b704cc Compare January 13, 2017 16:04

PVince81 force-pushed the integration-test-encryption-homedir branch from 6b704cc to c92ab57 Compare January 17, 2017 16:12

SergioBertolinSG approved these changes Jan 17, 2017

View reviewed changes

PVince81 added the 2 - Developing label Jan 17, 2017

Vincent Petry and others added 6 commits January 18, 2017 12:12

Add integration test stage with encryption enabled

e558602

Add no_encryption filter for incompatible int tests

c5cabed

Added filter to disable some tests when encryption is enabled.

Make sure we could create the work folder in int tests

e0a3fd8

Fix for Reshared files can be still accessed if a user in the middle …

87dcdff

…removes it.

Fixed scenario getting all shares of a user using that user

3dd462d

Fixed scenario getting share info of a share

1bc2364

PVince81 force-pushed the integration-test-encryption-homedir branch from c92ab57 to 1bc2364 Compare January 18, 2017 11:12

PVince81 added 4 - To release and removed 2 - Developing labels Jan 18, 2017

PVince81 merged commit ded3ba3 into master Jan 19, 2017

PVince81 deleted the integration-test-encryption-homedir branch January 19, 2017 08:59

This was referenced Jan 19, 2017

[Jenkins] Move each test into its own step #26873

Merged

[stable9.1] Integration test encryption homedir #26993

Merged

PVince81 mentioned this pull request Jan 20, 2017

[stable9] Integration test encryption homedir #26996

Merged

PVince81 removed the backport-request label Jan 20, 2017

PVince81 mentioned this pull request Aug 24, 2018

<file_target> of a auto-renamed file is not correct directly after sharing #32322

Closed

lock Bot locked as resolved and limited conversation to collaborators Aug 4, 2019

Uh oh!

Conversation

PVince81 commented Dec 20, 2016

Uh oh!

mention-bot commented Dec 20, 2016

Uh oh!

PVince81 commented Dec 20, 2016

Uh oh!

PVince81 commented Dec 20, 2016

Uh oh!

PVince81 commented Dec 20, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SergioBertolinSG commented Dec 20, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

PVince81 commented Dec 20, 2016

Uh oh!

PVince81 commented Dec 20, 2016

Uh oh!

PVince81 commented Dec 20, 2016

Uh oh!

PVince81 commented Jan 10, 2017

Uh oh!

PVince81 commented Jan 10, 2017

Uh oh!

PVince81 commented Jan 10, 2017

Uh oh!

PVince81 commented Jan 10, 2017

Uh oh!

PVince81 commented Jan 10, 2017

Uh oh!

PVince81 commented Jan 10, 2017

Uh oh!

PVince81 commented Jan 10, 2017

Uh oh!

PVince81 Jan 10, 2017

Choose a reason for hiding this comment

Uh oh!

PVince81 commented Jan 13, 2017

Uh oh!

SergioBertolinSG commented Jan 17, 2017

Uh oh!

PVince81 commented Jan 17, 2017

Uh oh!

PVince81 commented Jan 17, 2017

Uh oh!

PVince81 commented Jan 17, 2017

Uh oh!

PVince81 commented Jan 18, 2017

Uh oh!

PVince81 commented Jan 20, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SergioBertolinSG commented Jan 23, 2017

Uh oh!

lock Bot commented Aug 4, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

PVince81 commented Dec 20, 2016 •

edited

Loading

SergioBertolinSG commented Dec 20, 2016 •

edited

Loading

PVince81 commented Jan 20, 2017 •

edited

Loading