./occ files:scan --all -v and federated shares and users accessing this share via groups

[mmattel]

Steps to reproduce

1. Create some local users
2. Create some local groups
3. Have a federated incoming share to one of the users who is granted groupadmin for some groups
4. Grant some folders of the fd-share to a group
5. Assign some users to the above group
6. The other users except the owncloud admin can now access the granted fd-share folders
7. Run sudo -uwww-data ./occ files:scan --all -v

Following I am only talking about the users who have access to the the fd-share via the group, not the user who owns the fd-share.

Expected behaviour

Either quick scanning including a printout of the fd-share data or no printout and even quicker passing thru.

Actual behaviour

Local files of the users are quickly scanned, but it takes ages while it goes thru the fd-share, but nothing is printed out. You see with tcptrack the connection to the fd-share and a very slow data transfer rate.

Info

Scanning the user who owns the fd-share shows all the files and folders of the fd-share.

Server configuration

Operating system: Ubuntu 14.04

Web server: nginx 1.9.14

Database: mysql

PHP version: 5.5.9

ownCloud version: (see ownCloud admin page)
ownCloud 9.0.2 RC1 (daily) Build:2016-04-28T03:03:12+00:00 795f321

Updated from an older ownCloud or fresh install:
This was a fresh setup from tar and updated once, daily channel

Where did you install ownCloud from:
tar

Signing status (ownCloud 9.0 and above):

Datei nicht gefunden
Das ausgewählte Dokument wurde auf dem Server nicht gefunden.
Sie können zur Rückkehr zu ownCloud hier klicken.

List of activated apps:

Enabled:
  - activity: 2.2.1
  - comments: 0.2
  - dav: 0.1.6
  - federatedfilesharing: 0.1.0
  - federation: 0.0.4
  - files: 1.4.4
  - files_external: 0.5.2
  - files_pdfviewer: 0.8.1
  - files_sharing: 0.9.1
  - files_texteditor: 2.1
  - files_trashbin: 0.8.0
  - files_versions: 1.2.0
  - files_videoplayer: 0.9.8
  - firstrunwizard: 1.1
  - gallery: 14.5.0
  - notifications: 0.2.3
  - provisioning_api: 0.4.1
  - systemtags: 0.2
  - templateeditor: 0.1
  - updatenotification: 0.1.0
Disabled:
  - encryption
  - external
  - user_external
  - user_ldap

The content of config/config.php:

{
    "system": {
        "instanceid": "ock982ae3c5s",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "xxxx"
        ],
        "datadirectory": "\/var\/www\/owncloud\/data",
        "overwrite.cli.url": "https:\/\/xxxx",
        "dbtype": "mysql",
        "version": "9.0.2.0",
        "dbname": "owncloud",
        "dbhost": "localhost",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "logtimezone": "Europe\/Vienna",
        "installed": true,
        "filelocking.enabled": "true",
        "memcache.local": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "localhost",
            "port": 6379,
            "timeout": 0,
            "dbindex": 0
        },
        "assetdirectory": "\/var\/www\/owncloud",
        "asset-pipeline.enabled": true,
        "filesystem_check_changes": 1,
        "preview_max_scale_factor": 1,
        "enabledPreviewProviders": [
            "OC\\Preview\\PNG",
            "OC\\Preview\\JPEG",
            "OC\\Preview\\GIF",
            "OC\\Preview\\Illustrator",
            "OC\\Preview\\Postscript",
            "OC\\Preview\\Photoshop",
            "OC\\Preview\\TIFF",
            "OC\\Preview\\Raw"
        ],
        "loglevel": 2,
        "maintenance": false,
        "mail_smtpmode": "smtp",
        "mail_smtpsecure": "tls",
        "mail_from_address": "owncloud",
        "mail_domain": "xxxx",
        "mail_smtpauthtype": "LOGIN",
        "mail_smtpauth": 1,
        "mail_smtphost": "xxxx",
        "mail_smtpport": "587",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***"
    }
}

Are you using external storage, if yes which one: local/smb/sftp/...
SMB

Are you using encryption: yes/no
No

Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...
No

Client configuration

Browser: Opera

Operating system: W7x64

Logs

owncloud.log

No corresponding entry

[PVince81]

Federated shares are technically like mounted external shares and might in some cases contain unscanned changes, so it might be acceptable that the scanner goes in there.

@icewind1991 @rullzer

[mmattel]

@PVince81 it is absolutely ok that the scanner goes into fd-shares, but it is a big difference:

• if there is a user who has mounted a incoming fd- share (in this case you see the scanning progress and it is very quick)
• while users who are granted to access the fd-share via a group (where there is no scanning progress shown and it is very slow)

There is always a "root" user behind a eg incoming fd-share when it get´s reshared, in this case via a group.
The question is, how are users assigned to groups where the groups are granted to access a fd-share treated during scanning, as everything should be owned by the "root" user - or? This is the question we have on other issues too: ownership when resharing.
A user beeing part of a group where the group gets access to a folder is not the owner of the share he sees and therefore it should not be rescanned as the owner is already defined earlier. Same when user reshares with user.

[PVince81]

Ah yes, you are right. We should change the CLI scanner to not recurse into received local shares, but fed shares are ok.

This is because for local shares it would rescan the same thing for every recipient, and for local shares there is one owner for which the scan will happen anyway.

[mmattel]

This would be a great improvement, as the current behaviour of recursion takes very long for each user. I thought my system crashed and was close to reboot it. The shell was also unresponsive between the user scanning, no ctrl-c possible during the recursion process...

[PVince81]

This might be a matter of adding a check $storage->instanceOfStorage('xxx').

@rullzer @icewind1991 is there a good way to detect local shared storages (and skip them) and differenciate it from federated share storages ?

[rullzer]

The trouble here is that reshared federated shares are a very strange thing. @schiessle had (or has?) a PR to have the new resharing in federated shares as well. That should solve this.

In order to get info like this you would need to get to the root storage... because for the scanning user it is just a normal incomming share. Only the storage below that is a federated share. I don't know if instanceOfStorage is smart enough for that.

[icewind1991]

I don't think going into fed shares is a good idea, the remote server should take care of things being scanned

[PVince81]

@icewind1991 no, because the remote server has its own cache. But the local server also has an own version of the cache, which might not be up to date in some rare cases.

[PVince81]

Hmm, I just tried with v9.1.2 and it seems it doesn't scan into local shared folders, which is expected, but it also doesn't even scan the federated shares here.

My scenario:

1. Share a folder "Photos" from a remote server to the current test instance (admin@localhost/owncloud)
2. Login as admin
3. Accept the share
4. Create three users "user1", "user2", "user3" and add them to group "group1"
5. Share the received fed share "Photos" with "group1".
6. Login as "user1" and "user2", just in case. (but not user3)

± % sudo -uwwwrun ./occ files:scan --all -vvv

Scanning files for 4 users

Starting scan for user 1 out of 4 (admin)
        Folder /admin/
        Folder /admin/thumbnails
        Folder /admin/thumbnails/13
        File   /admin/thumbnails/13/32-32.png
        File   /admin/thumbnails/13/1024-683-max.png
        Folder /admin/thumbnails/14
        File   /admin/thumbnails/14/32-32.png
        File   /admin/thumbnails/14/1024-768-max.png
        Folder /admin/thumbnails/12
        File   /admin/thumbnails/12/32-32.png
        File   /admin/thumbnails/12/1024-683-max.png
        Folder /admin/thumbnails/3
        File   /admin/thumbnails/3/32-32.png
        File   /admin/thumbnails/3/2048-2048-max.png
        Folder /admin/cache
        Folder /admin/files
        File   /admin/files/welcome.txt

Starting scan for user 2 out of 4 (user1)
        Folder /user1/
        Folder /user1/thumbnails
        Folder /user1/thumbnails/27
        File   /user1/thumbnails/27/32-32.png
        File   /user1/thumbnails/27/2048-2048-max.png
        Folder /user1/files
        File   /user1/files/welcome.txt
        Folder /user1/cache

Starting scan for user 3 out of 4 (user2)
        Folder /user2/
        Folder /user2/thumbnails
        Folder /user2/thumbnails/35
        File   /user2/thumbnails/35/32-32.png
        File   /user2/thumbnails/35/2048-2048-max.png
        Folder /user2/files
        File   /user2/files/welcome.txt
        Folder /user2/cache

Starting scan for user 4 out of 4 (user3)

+---------+-------+--------------+
| Folders | Files | Elapsed time |
+---------+-------+--------------+
| 18      | 15    | 00:00:05     |
+---------+-------+--------------+

Hmm, it didn't show "Photos" but there is a small delay between user scan.

[PVince81]

Right... the federated share storage scanner works differently: it only requests the root entry and would only recurse in case of etag change. That's why there is no output.

[PVince81]

Argh... now I understand...

All along I thought that it was never scanning into locally received shares, but that was based on the verbose CLI output. But that output doesn't include results from scanners from shared storage, because the hooks aren't forwarded through.

So it turns out, "occ files:scan" does scan all received storages, fed share or local share.

Now for a fix: allow scanning received fed share (when direct) but skip received local shares. These should be scanned by scanning the owner's home.