introduce callForSeenUsers and countSeenUsers by butonic · Pull Request #26361 · owncloud/core

butonic · 2016-10-12T21:22:13Z

I was running into the upgrade trying to iterate over 170000 zombie users in my two test ldap servers, when only 6 of them ever logged in. Instead of waiting 5-6 hours or maybe even longer I decided to take the time and find a better solution.

This pr introduces callForSeenUsers() and countSeenUsers() to allow only executing a callback for users that logged in at least once and as a result have a home directory that has been initialized. It also helps prevent out of memory errors.

cc @PVince81 @DeepDiver1975 there are at least two further occurences (SyncService and SyncBirthdayCalendar) of callForAllUsers in the dav app, but they seem to be related to calendar and contacts. We might add other callFor???User methods.

Backport to 9 strongly requested ❗ ❕ !!!111einself

mention-bot · 2016-10-12T21:22:14Z

@butonic, thanks for your PR! By analyzing the history of the files in this pull request, we identified @PVince81, @rullzer and @DeepDiver1975 to be potential reviewers.

PVince81 · 2016-10-13T09:19:31Z

won't work here because even users who haven't logged in might have broken "files_target" and duplicate shares due to the fact that some sharing API calls would set up their FS...

PVince81 · 2016-10-13T09:20:54Z

please add a unit test for this. It will further confirm that it works with all DBs.

PVince81 · 2016-10-13T09:21:05Z

Good idea !

PVince81 · 2016-10-13T09:21:25Z

interface change, can't backport directly

would it be easier to simply add a flag $onlySeen = false to callForAllUsers to avoid changing the interface ?

hm, that still changes the return type of countUsers() to int if $onlySeen is true. I left the new methods in the interface ... should I remove them from IUserManager completely? I dislike munching everything into a single call. We can backport it that way, but in the future I want a clean API.

The callForSeenUsers call eg, does not support searching for users because we only have the userid in the preferences. callForAllUsers iterates over the user backends and the LDAP backend may search displayname or emaul in addition to the userid.

So what to do with the interface breakage ?

In the backport you'd remove the methods from the interface ? Ok.
But then we also need to add checks everywhere if (is_callable($userManager, 'callForSeenUsers')) then call it, else fallback to the old way.

I'm fine adding these methods in 9.2.

Actually adding methods in 9.2 would also require a new interface IUserManager2 that extends IUserManager.

Unless we assume that no one ever bothered to implement their own user manager... there is currently no config for easy overriding without hacking server.php ...

@DeepDiver1975 thoughts ?

Somewhere (maybe in another comment), @DeepDiver1975 answered that it is currently not possible anyway to override the interface, so changing it is ok in 9.2

DeepDiver1975 · 2016-10-14T06:20:54Z

I was thinking about this : With the user account table I want the call for uses to iterate only over known users.
Accessing not yet logged in users is a matter of search only from my pov.

butonic · 2016-10-17T08:23:24Z

@DeepDiver1975 What is the status af the user account table?

For this PR the tests are passing and it is a clear improvement. @PVince81 @DeepDiver1975 can we merge so I can backport to 9.0.?

PVince81 · 2016-10-17T09:32:29Z

+	 * @since 9.2.0
+	 */
+	public function countSeenUsers() {
+		$limit = 10000;


hard-coded limit ? should we have this as argument ?

If you pass that as an argument, you will have different hard-coded values for each use. Making it harder to test. If this should be variable, we should inject it as a configuration value with propel, but i think it shouldn't.

PVince81 · 2016-10-17T09:33:05Z

+	 * @since 9.0.0
+	 */
+	public function callForSeenUsers (\Closure $callback) {
+		$limit = 1000;


hardcoded limit ? pass as argument ?

Also, in the count method you hard-coded to 10000 while here only 1000. Any reason ?

gut feeling. count only iterates. call executes a callback ... who knows what is done in there and how much memory is leaked

butonic · 2016-10-18T08:45:32Z

@PhilippSchaffrath could you review, pls?

phisch · 2016-10-18T09:27:20Z

+	 *                   if $onlySeen is true only an int is returned
 	 */
-	public function countUsers() {
+	public function countUsers($onlySeen = false) {


The domain is a little inconsistent here, i would like $onlySeen to be called something like $hasLoggedIn.

phisch · 2016-10-18T09:27:34Z

-	public function countUsers() {
+	public function countUsers($onlySeen = false) {
+		if ($onlySeen) {
+			return $this->countSeenUsers();


countHasLoggedInUsers()? looks ugly but clarifies, hm k.

https://github.com/owncloud/windows_network_drive/pull/51 has already been merged we can change the name after getting this in.

I don't want to block this, it's just a reminder to keep an easy and consistent domain so it's more easy to understand the code.

phisch · 2016-10-18T09:30:43Z

-						continue;
+	public function callForAllUsers(\Closure $callback, $search = '', $onlySeen = false) {
+		if ($onlySeen) {
+			$this->callForSeenUsers($callback);


I would like to see a separation here. callForAllUsers should always call a callback for all users, not sometimes if parameter x is set to y only a subset of all users. Since callForSeenUsers already exists, why does it need to be cascaded in here?

@PVince81 and @DeepDiver1975 wanted a param to make backporting easier since we don't backport API changes ...

phisch · 2016-10-18T09:39:20Z

+		$user4->delete();
+	}
+
+	public function testCallForSeenUsers() {


smart test, like it 👍

DeepDiver1975

Actually adding methods in 9.2 would also require a new interface IUserManager2 that extends IUserManager.

This is only necessary for interfaces where we allow/expect alternative implementations out there.
I see no reason why an alternative implementation would make sense - furthermore there is no way to register one.

DeepDiver1975 · 2016-10-13T10:52:22Z

@butonic seems you missed this one

DeepDiver1975 · 2016-10-13T10:53:21Z

butonic · 2016-10-18T18:52:53Z

I addressed most issues and also removed the sql hack for oracle because it was unnecessary to order the results in fhe first place.

@DeepDiver1975 @PVince81 @PhilippSchaffrath please review

PVince81 · 2016-10-18T19:36:35Z

Looks good to me, also tested with 200 LDAP users where only 4 logged in and saw that only 4 got processed for the avatar move.

@butonic please adjust the "since" tag from above and this is good from my POV 👍

butonic · 2016-10-18T20:11:27Z

@PVince81 done

DeepDiver1975 · 2016-10-19T08:03:26Z

👍

PVince81 · 2016-12-07T11:52:51Z

stable9.1: #27248
stable9: #26415

* introduce callForSeenUsers and countSeenUsers * add tests * oracle should support not null on clob * since 9.2.0 Conflicts: lib/private/Repair/MoveAvatarOutsideHome.php lib/private/User/Manager.php tests/lib/User/ManagerTest.php

* introduce callForSeenUsers and countSeenUsers * add tests * oracle should support not null on clob * since 9.2.0

lock · 2019-08-04T06:02:27Z

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

butonic added Type:Bug 3 - To Review labels Oct 12, 2016

butonic added this to the 9.2 milestone Oct 12, 2016

butonic added the backport-request label Oct 12, 2016

butonic force-pushed the callforusersthatloggedin branch 2 times, most recently from 438a6f5 to e388086 Compare October 13, 2016 08:59

PVince81 reviewed Oct 13, 2016

View reviewed changes

butonic force-pushed the callforusersthatloggedin branch from e388086 to 8f38ca8 Compare October 13, 2016 19:31

butonic force-pushed the callforusersthatloggedin branch 3 times, most recently from ac6b18b to d57f314 Compare October 14, 2016 12:32

PVince81 reviewed Oct 17, 2016

View reviewed changes

phisch reviewed Oct 18, 2016

View reviewed changes

butonic added 2 commits October 18, 2016 14:34

introduce callForSeenUsers and countSeenUsers

1c549b5

add tests

34f8cb5

DeepDiver1975 reviewed Oct 18, 2016

View reviewed changes

oracle should support not null on clob

8513f2a

butonic force-pushed the callforusersthatloggedin branch from d57f314 to 8513f2a Compare October 18, 2016 13:51

since 9.2.0

5a110e4

DeepDiver1975 merged commit 0f8bf00 into master Oct 19, 2016

DeepDiver1975 deleted the callforusersthatloggedin branch October 19, 2016 08:03

butonic mentioned this pull request Oct 19, 2016

[WIP] backport callforseenusers to stable9 #26415

Closed

MorrisJobke mentioned this pull request Oct 24, 2016

introduce callForSeenUsers and countSeenUsers nextcloud/server#1888

Merged

jvillafanez mentioned this pull request Feb 24, 2017

Callforusersthatloggedin stable9.1 #27248

Merged

9 tasks

elie195 pushed a commit to elie195/core that referenced this pull request Mar 23, 2017

introduce callForSeenUsers and countSeenUsers (owncloud#26361)

649500a

* introduce callForSeenUsers and countSeenUsers * add tests * oracle should support not null on clob * since 9.2.0

PVince81 removed the backport-request label Mar 24, 2017

lock Bot locked as resolved and limited conversation to collaborators Aug 4, 2019

Uh oh!

Conversation

butonic commented Oct 12, 2016

Uh oh!

mention-bot commented Oct 12, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PVince81 commented Oct 13, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DeepDiver1975 commented Oct 14, 2016

Uh oh!

butonic commented Oct 17, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

butonic commented Oct 18, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DeepDiver1975 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

butonic commented Oct 18, 2016

Uh oh!

PVince81 commented Oct 18, 2016

Uh oh!

butonic commented Oct 18, 2016

Uh oh!

DeepDiver1975 commented Oct 19, 2016

Uh oh!

PVince81 commented Dec 7, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lock Bot commented Aug 4, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

PVince81 commented Dec 7, 2016 •

edited

Loading