getSectionChanges can be slow for very large transactions

[chrisballinger]

When doing a batch transaction with ~7000 updates, I get only one YapDatabaseModifiedNotification for which I call getSectionChanges:rowChanges:forNotifications:withMappings: as seen here.

This method takes ~10-15 seconds to complete, and I am calling it on the main thread. Would you consider it safe to call this method or otherwise updating mappings from a background queue? (as long as the following UI updates are triggered on the main thread, of course)

It would probably be a nicer user experience if I split the batch transaction into smaller chunks for faster feedback anyway, but I was curious if there were any side effects to updating mappings in the background. I feel like I'd have to be more careful about synchronization to prevent reading from the mappings while they were being updated in the background, so it would be rather complex.

[mackross]

@chrisballinger I've used dispatch_async on a serial queue to do this which stopped the interface from freezing. I used a serial queue to make sure that if a second notification comes in on the main thread that getSectionChanges:rowChanges:forNotifications:withMappings: wouldn't run out of order.

[mackross]

Also if I had over 100 row changes or 10 section changes, I decided to reload the tableview as otherwise the animation would pretty much lock up an iPhone 4 for 5-10 seconds.

[chrisballinger]

@mackross I took another approach, which was to split the updates into 10 transactions. This way if there are other important database changes they can jump in line to be processed. Perhaps we should add a note about this in the wiki or header docs.

[chrisballinger]

I took another (another) approach, which was to not store the distance to the database like an idiot. However, it would still be useful to have an async version of this method that maintains an internal serial dispatch queue with a completionBlock that fires on the main thread to prevent blocking the UI when there are lots of changes all at once.

[robbiehanson]

The 2.5 branch has several performance optimizations for the getSectionChanges:rowChanges:forNotifications:withMappings: routine

• commit 94ac39e
• commit ed9e1db
• commit 11f80b4

If I remember correctly, I was testing against a multi-thousand change-set that originally took around 10 seconds to process, and I was able to reduce it to under a half second.

There may yet me more performance improvements I can make, but I at least killed the biggest bottlenecks.

I feel like I'd have to be more careful about synchronization to prevent reading from the mappings while they were being updated in the background, so it would be rather complex.

This is exactly the problem. As getSectionChanges:rowChanges:forNotifications:withMappings: is actually modifying the mappings that you pass it.

I've used dispatch_async on a serial queue to do this [...]

I'm not sure this is entirely safe. For one, you'll need to watch out for what @chrisballinger said. It would likely be better to first make a copy of the mappings. ([mappings copy]) And then update this copy in your dispatch_async. And swamp in the updated mappings over the old mappings, back on the main thread, after the dispatch_async completes.

However, this isn't going to solve all the problems with this approach. Because, although this approach keeps the mappings stable, the databaseConnection has already been migrated to the latest commit.

Here's what I mean: when you get the yapDatabaseModified notification, you move your databaseConnection to the latest commit. This is how you get the array of notifications. But if you don't update your UI at this moment, then you're stuck in a situation in which your storage backing (your databaseConnection) isn't in sync with your UI. So, for example, if you were scrolling at this moment, then fetching the item at a particular indexPath may return different objects than before. Or it may return nil (if items have been deleted, and the group is smaller).

if I had over 100 row changes or 10 section changes, I decided to reload the tableview as otherwise the animation would pretty much lock up an iPhone 4 for 5-10 seconds.

This is probably the best course of action.

However there is a small complication here. In order to know the exact amount of rowChanges / sectionChanges, one must call getSectionChanges:rowChanges:forNotifications:withMappings:. But if this method is overly slow, then we still have a problem.

Hopefully my performance improvements go a long way here. But if they don't (~7,000 changes), then there's another option (until I can optimized the crap out of that routine). You can look at the notifications themselves, and see how many changes in general you're dealing with. And if the number is just huge, then you may skip the fetching changes & animation routine, and just reload the tableView instead. It would look something like this:

if (changeSetLooksHuge)
{
    [databaseConnection readWithBlock:^(YapDatabaseReadTransaction *transaction){
        [mappings updateWithTransaction:transaction];
    }];
    [tableView reloadData];
    return;
}

I'll add a method to allow you to get a rough estimate of the number of "raw" changes that affect a YapDatabaseView.

[robbiehanson]

New method available in 2.5 branch for this:

NSUInteger sizeEstimate = [[databaseConnection ext:@"myView"] numberOfRawChangesForNotifications:notifications];
if (sizeEstimate > 150)
{
    [databaseConnection readWithBlock:^(YapDatabaseReadTransaction *transaction){
        [mappings updateWithTransaction:transaction];
    }];
    [tableView reloadData];
    return;
}

[robbiehanson]

Closing this issue. And I have a line in my notes to add a corresponding section to the wiki.

[chrisballinger]

:) !!

[mackross]

Great thanks! I was actually using a different mapping on the serial queue to do the estimate. This changes means I can drop all that code though :).