Memory corruption

[diegosanchezr]

Looks like YapDatabaseConnection may end up using a freed wal_file and corrupt memory. It happens very rarely but it could explain some weird crashes we are getting. Please take a look at the AddressSanitizer log below:

==22357==ERROR: AddressSanitizer: heap-use-after-free on address 0x612000e61ff8 at pc 0x00010ccc99ce bp 0x7fff5ac796f0 sp 0x7fff5ac796e8
WRITE of size 8 at 0x612000e61ff8 thread T0
    #0 0x10ccc99cd in -[YapDatabaseConnection postReadTransaction:] YapDatabaseConnection.m:2269
    #1 0x10ccd46f3 in __52-[YapDatabaseConnection endLongLivedReadTransaction]_block_invoke YapDatabaseConnection.m:3201
    #2 0x10ccd4384 in -[YapDatabaseConnection endLongLivedReadTransaction] YapDatabaseConnection.m:3224
    #3 0x10ccd37ff in __54-[YapDatabaseConnection beginLongLivedReadTransaction]_block_invoke YapDatabaseConnection.m:3157
    #4 0x1103343ea in _dispatch_client_callout (libdispatch.dylib+0x223ea)
    #5 0x110318ef4 in _dispatch_barrier_sync_f_invoke (libdispatch.dylib+0x6ef4)
    #6 0x10ccd34e7 in -[YapDatabaseConnection beginLongLivedReadTransaction] YapDatabaseConnection.m:3186
    #7 0x10863dd9b in -[MyAppYapFetchedResultsController yapDatabaseModifiedNotification:] MyAppYapFetchedResultsController.m:121
    #8 0x10f77dc8b in __CFNOTIFICATIONCENTER_IS_CALLING_OUT_TO_AN_OBSERVER__ (CoreFoundation+0x116c8b)
    #9 0x10f77d9ca in _CFXRegistrationPost (CoreFoundation+0x1169ca)
    #10 0x10f77d731 in ___CFXNotificationPost_block_invoke (CoreFoundation+0x116731)
    #11 0x10f7c61e1 in -[_CFXNotificationRegistrar find:object:observer:enumerator:] (CoreFoundation+0x15f1e1)
    #12 0x10f675678 in _CFXNotificationPost (CoreFoundation+0xe678)
    #13 0x10cccfabb in __50-[YapDatabaseConnection postReadWriteTransaction:]_block_invoke.381 YapDatabaseConnection.m:2793
    #14 0x106479643 in __wrap_dispatch_async_block_invoke (libclang_rt.asan_iossim_dynamic.dylib+0x45643)
    #15 0x110313d9c in _dispatch_call_block_and_release (libdispatch.dylib+0x1d9c)
    #16 0x1103343ea in _dispatch_client_callout (libdispatch.dylib+0x223ea)
    #17 0x11031c1ee in _dispatch_main_queue_callback_4CF (libdispatch.dylib+0xa1ee)
    #18 0x10f70d0f8 in __CFRUNLOOP_IS_SERVICING_THE_MAIN_DISPATCH_QUEUE__ (CoreFoundation+0xa60f8)
    #19 0x10f6ceb98 in __CFRunLoopRun (CoreFoundation+0x67b98)
    #20 0x10f6ce0f7 in CFRunLoopRunSpecific (CoreFoundation+0x670f7)
    #21 0x110939ad1 in GSEventRunModal (GraphicsServices+0xcad1)
    #22 0x10b6ddf08 in UIApplicationMain (UIKit+0x25f08)
    #23 0x105357cf5  (MyApp+0x1003d5cf5)
    #24 0x11036992c in start (libdyld.dylib+0x292c)
    #25 0x0  (<unknown module>)

0x612000e61ff8 is located 184 bytes inside of 296-byte region [0x612000e61f40,0x612000e62068)
freed by thread T61 here:
    #0 0x10647b4d7 in __sanitizer_mz_free (libclang_rt.asan_iossim_dynamic.dylib+0x474d7)
    #1 0x10ef6f504 in sqlite3WalClose (libsqlite3.dylib+0x2e504)
    #2 0x10ef6b9aa in sqlite3PagerClose (libsqlite3.dylib+0x2a9aa)
    #3 0x10ef7a911 in sqlite3BtreeClose (libsqlite3.dylib+0x39911)
    #4 0x10ef4cb22 in sqlite3LeaveMutexAndCloseZombie (libsqlite3.dylib+0xbb22)
    #5 0x10ef58da9 in sqlite3Close (libsqlite3.dylib+0x17da9)
    #6 0x10cc739dc in -[YapDatabase handleConnectionPoolTimerFire] YapDatabase.m:2521
    #7 0x10cc73750 in __39-[YapDatabase resetConnectionPoolTimer]_block_invoke YapDatabase.m:2476
    #8 0x106479d73 in __wrap_dispatch_source_set_event_handler_block_invoke (libclang_rt.asan_iossim_dynamic.dylib+0x45d73)
    #9 0x1103343ea in _dispatch_client_callout (libdispatch.dylib+0x223ea)
    #10 0x1103277e4 in _dispatch_source_latch_and_call (libdispatch.dylib+0x157e4)
    #11 0x11032276f in _dispatch_source_invoke (libdispatch.dylib+0x1076f)
    #12 0x11031a69e in _dispatch_queue_drain (libdispatch.dylib+0x869e)
    #13 0x110319d4c in _dispatch_queue_invoke (libdispatch.dylib+0x7d4c)
    #14 0x11031c995 in _dispatch_root_queue_drain (libdispatch.dylib+0xa995)
    #15 0x11031c404 in _dispatch_worker_thread3 (libdispatch.dylib+0xa404)
    #16 0x1106884dd in _pthread_wqthread (libsystem_pthread.dylib+0x34dd)
    #17 0x110686340 in start_wqthread (libsystem_pthread.dylib+0x1340)

previously allocated by thread T0 here:
    #0 0x10647ae5d in __sanitizer_mz_malloc (libclang_rt.asan_iossim_dynamic.dylib+0x46e5d)
    #1 0x11053d3ec in malloc_zone_malloc (libsystem_malloc.dylib+0x113ec)
    #2 0x10f002168 in sqlite3MemMalloc (libsqlite3.dylib+0xc1168)
    #3 0x10ef74247 in pagerOpenWal (libsqlite3.dylib+0x33247)
    #4 0x10ef73faf in sqlite3PagerSharedLock (libsqlite3.dylib+0x32faf)
    #5 0x10ef4a5b1 in sqlite3BtreeBeginTrans (libsqlite3.dylib+0x95b1)
    #6 0x10f02ff12 in sqlite3InitOne (libsqlite3.dylib+0xeef12)
    #7 0x10ef5d5a1 in sqlite3Init (libsqlite3.dylib+0x1c5a1)
    #8 0x10efd79ca in selectExpander (libsqlite3.dylib+0x969ca)
    #9 0x10efd1840 in sqlite3WalkSelect (libsqlite3.dylib+0x90840)
    #10 0x10efd6d54 in sqlite3SelectPrep (libsqlite3.dylib+0x95d54)
    #11 0x10efb1335 in sqlite3Select (libsqlite3.dylib+0x70335)
    #12 0x10ef9e57e in yy_reduce (libsqlite3.dylib+0x5d57e)
    #13 0x10ef9d35e in sqlite3Parser (libsqlite3.dylib+0x5c35e)
    #14 0x10ef563fb in sqlite3RunParser (libsqlite3.dylib+0x153fb)
    #15 0x10ef9ca60 in sqlite3Prepare (libsqlite3.dylib+0x5ba60)
    #16 0x10ef54e7e in sqlite3LockAndPrepare (libsqlite3.dylib+0x13e7e)
    #17 0x10ef54ada in sqlite3_prepare_v2 (libsqlite3.dylib+0x13ada)
    #18 0x10ccbd661 in -[YapDatabaseConnection yapGetDataForKeyStatement] YapDatabaseConnection.m:901
    #19 0x10ccd1c02 in -[YapDatabaseConnection readSnapshotFromDatabase] YapDatabaseConnection.m:3008
    #20 0x10ccc87f5 in __44-[YapDatabaseConnection preReadTransaction:]_block_invoke YapDatabaseConnection.m:2147
    #21 0x1103343ea in _dispatch_client_callout (libdispatch.dylib+0x223ea)
    #22 0x110318ef4 in _dispatch_barrier_sync_f_invoke (libdispatch.dylib+0x6ef4)
    #23 0x10ccc7c81 in -[YapDatabaseConnection preReadTransaction:] YapDatabaseConnection.m:2081
    #24 0x10ccd3945 in __54-[YapDatabaseConnection beginLongLivedReadTransaction]_block_invoke YapDatabaseConnection.m:3161
    #25 0x1103343ea in _dispatch_client_callout (libdispatch.dylib+0x223ea)
    #26 0x110318b73 in _dispatch_barrier_sync_f_slow (libdispatch.dylib+0x6b73)
    #27 0x10ccd34e7 in -[YapDatabaseConnection beginLongLivedReadTransaction] YapDatabaseConnection.m:3186
    #28 0x10863c56f in -[MyAppYapFetchedResultsController initWithDatabase:mappings:collection:cellConfiguration:] MyAppYapFetchedResultsController.m:36
    #29 0x10556f023  (MyApp+0x1005ed023)

Thread T61 created by T60 here:
    <empty stack>

Thread T60 created by T52 here:
    <empty stack>

Thread T52 created by T31 here:
    <empty stack>

Thread T31 created by T27 here:
    <empty stack>

Thread T27 created by T0 here:
    <empty stack>

SUMMARY: AddressSanitizer: heap-use-after-free YapDatabaseConnection.m:2269 in -[YapDatabaseConnection postReadTransaction:]
Shadow bytes around the buggy address:
  0x1c24001cc3a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x1c24001cc3b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x1c24001cc3c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x1c24001cc3d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x1c24001cc3e0: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
=>0x1c24001cc3f0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd[fd]
  0x1c24001cc400: fd fd fd fd fd fd fd fd fd fd fd fd fd fa fa fa
  0x1c24001cc410: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x1c24001cc420: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x1c24001cc430: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x1c24001cc440: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Heap right redzone:      fb
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack partial redzone:   f4
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==22357==ABORTING
AddressSanitizer report breakpoint hit. Use 'thread info -s' to get extended information about the report.
(lldb) 

[robbiehanson]

Thanks for reporting this. I'm currently working on a fix, and should have it done soon.

[robbiehanson]

I pushed a fix to the "v2.9" branch: 8688746

Can you give it a try and let me know if it doesn't fix the issue.

[diegosanchezr]

@robbiehanson thanks for taking a look. Unfortunately, the issue is still there :(

As far as I've seen, the problem is that YapDatabaseConnection once that it takes a pointer to the wal_file, it will keep it forever, even after sqlite deallocs it. What I don't fully understand is why sqlite deallocs the wal_file if there's a YapDatabaseConnection alive.

[robbiehanson]

Thanks for letting me know. Will you test a theory for me:
If you set database.maxConnectionPoolCount = 0; (when you first initialize the database), does the problem go away?

[diegosanchezr]

Yes, it does go away with maxConnectionPoolCount = 0

[diegosanchezr]

@robbiehanson In case it helps I was able to replicate the problem consistently in a small project: https://github.com/diegosanchezr/YapDatabase-Example/tree/toy-project-for-issue-307

[robbiehanson]

Thanks! I've been able to replicate it as well. But I'll be sure to also test with the project you posted.

[robbiehanson]

I think we've got a fix. Try out the above commit.

Your test project worked nicely. After cloning & running, I'd get a crash almost right away. After updating the pod to use the above commit, I can't get it to crash anymore.

[diegosanchezr]

Awesome! I can't get it to crash either, nor our "real" app :D

Do you have any release planned including this fix soon? Or do you recommend us using this commit in production?

Thank you!

[robbiehanson]

Do you have any release planned including this fix soon?

I'm hoping to release v2.9 branch very soon. Just want to spend another day or so stress testing it within my own projects.

Or do you recommend us using this commit in production?

There have been quite a few changes in v2.9. Some of these have been great bug fixes for issues that exist in previous versions. But I've also inadvertently introduced some other bugs recently...

I'm hoping most of the bugs have been shaken out as of commit 0fa554c (latest v2.9 branch), but I won't know for sure until I've spent some more time stress testing it.

Since you're currently experiencing crashes from this issue, I'd probably recommend using commit 0fa554c. (As opposed to v2.8.3, where you're going to continue to experience the same crash.)

Let me know if you run into any other issues.

[robbiehanson]

Update: v2.9 has been released.