Mac_ios microbenchmarks_ios is 7.00% flaky. XCode build concurrent build failure.

[fluttergithubbot]

The post-submit test builder Mac_ios microbenchmarks_ios had a flaky ratio 7.00% for the past (up to) 100 commits, which is above our 2.00% threshold.

One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_ios%20microbenchmarks_ios/5886
Commit: 43fa2c0

Flaky builds:
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_ios%20microbenchmarks_ios/5886
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_ios%20microbenchmarks_ios/5860
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_ios%20microbenchmarks_ios/5848
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_ios%20microbenchmarks_ios/5841
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_ios%20microbenchmarks_ios/5838
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_ios%20microbenchmarks_ios/5806
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_ios%20microbenchmarks_ios/5801

Recent test runs:
https://flutter-dashboard.appspot.com/#/build?taskFilter=Mac_ios%20microbenchmarks_ios

Please follow https://github.com/flutter/flutter/wiki/Reducing-Test-Flakiness#fixing-flaky-tests to fix the flakiness and enable the test back after validating the fix (internal dashboard to validate: go/flutter_test_flakiness).

[cyanglaz]

5886 is flutter run time out:

[2023-05-06 02:31:46.252076] [STDOUT] [STDOUT] [        ] Application launched on the device. Waiting for Dart VM Service url.
[2023-05-06 02:32:16.262792] [STDOUT] [STDOUT] [   +7 ms] (lldb) process interrupt
[2023-05-06 02:32:16.263283] [STDOUT] [STDOUT] [        ] error: Failed to halt process: Halt timed out. State = running
[2023-05-06 02:32:17.191989] [STDOUT] [STDOUT] [ +928 ms] (lldb) Process 534 exited with status = -1 (0xffffffff) lost connection

It looks similar to the observatory timeout in #121231 but with some additional logs:
This particular failure has Failed to halt process: Halt timed out. State = running and Process 534 exited with status = -1 (0xffffffff) lost connection

cc @vashworth @christopherfujino

Update: this is tracked in #120808

[cyanglaz]

5860 crash

[2023-05-05 09:12:06.059964] [STDOUT] [STDOUT]                ===== CRASH =====
[2023-05-05 09:12:06.059988] [STDOUT] [STDOUT]                si_signo=Segmentation fault: 11(11), si_code=0, si_addr=0x0
[2023-05-05 09:12:06.060011] [STDOUT] [STDOUT]                version=3.1.0-80.0.dev (dev) (Fri May 5 00:27:27 2023 -0700) on "macos_x64"
[2023-05-05 09:12:06.060037] [STDOUT] [STDOUT]                pid=31810, thread=7171, isolate_group=kernel-service(0x7fa1b880e000), isolate=(nil)(0x0)
[2023-05-05 09:12:06.060060] [STDOUT] [STDOUT]                os=macos, arch=x64, comp=no, sim=no
[2023-05-05 09:12:06.060083] [STDOUT] [STDOUT]                isolate_instructions=108b0d000, vm_instructions=101585960
[2023-05-05 09:12:06.060108] [STDOUT] [STDOUT]                fp=7000013f2d60, sp=7000013f2d30, pc=1018a70d7
[2023-05-05 09:12:06.060131] [STDOUT] [STDOUT]                  pc 0x00000001018a70d7 fp 0x00007000013f2d60 dart::GCMarker::MarkObjects(dart::PageSpace*)+0x19d7
[2023-05-05 09:12:06.060156] [STDOUT] [STDOUT]                  pc 0x000000010180808d fp 0x00007000013f2d90 dart::UntaggedObject::VisitPointersPredefined(dart::ObjectPointerVisitor*, long)+0x52d
[2023-05-05 09:12:06.060182] [STDOUT] [STDOUT]                  pc 0x00000001018a5208 fp 0x00007000013f2dd0 dart::GCMarker::IncrementalMarkWithSizeBudget(dart::PageSpace*, long)+0x2e8
[2023-05-05 09:12:06.060206] [STDOUT] [STDOUT]                  pc 0x00000001018a65ab fp 0x00007000013f2e70 dart::GCMarker::MarkObjects(dart::PageSpace*)+0xeab
[2023-05-05 09:12:06.060231] [STDOUT] [STDOUT]                  pc 0x00000001018a6c53 fp 0x00007000013f2ea0 dart::GCMarker::MarkObjects(dart::PageSpace*)+0x1553
[2023-05-05 09:12:06.060254] [STDOUT] [STDOUT]                  pc 0x000000010187a8eb fp 0x00007000013f2f20 dart::ThreadPool::WorkerLoop(dart::ThreadPool::Worker*)+0x14b
[2023-05-05 09:12:06.060280] [STDOUT] [STDOUT]                  pc 0x000000010187ad49 fp 0x00007000013f2f50 dart::ThreadPool::Worker::Main(unsigned long)+0x79
[2023-05-05 09:12:06.060304] [STDOUT] [STDOUT]                  pc 0x00000001017f1a2e fp 0x00007000013f2fb0 dart::OSThread::GetMaxStackSize()+0xde
[2023-05-05 09:12:06.060329] [STDOUT] [STDOUT]                  pc 0x00007ff807f4c4e1 fp 0x00007000013f2fd0 _pthread_start+0x7d
[2023-05-05 09:12:06.060352] [STDOUT] [STDOUT]                  pc 0x00007ff807f47f6b fp 0x00007000013f2ff0 thread_start+0xf
[2023-05-05 09:12:06.060376] [STDOUT] [STDOUT]                -- End of DumpStackTrace
[2023-05-05 09:12:06.060399] [STDOUT] [STDOUT]                /opt/s/w/ir/x/w/recipe_cleanup/tmpfxas5lcm/flutter sdk/packages/flutter_tools/bin/xcode_backend.sh: line 29: 31810 Abort trap: 6           "$DART" "$BIN_DIR/xcode_backend.dart" "$@"
[2023-05-05 09:12:06.060425] [STDOUT] [STDOUT]                Command PhaseScriptExecution failed with a nonzero exit code
[2023-05-05 09:12:06.060447] [STDOUT] [STDOUT] 
[2023-05-05 09:12:06.060469] [STDOUT] [STDOUT] 
[2023-05-05 09:12:06.060493] [STDOUT] [STDOUT]                Result bundle written to path:
[2023-05-05 09:12:06.060516] [STDOUT] [STDOUT]                	/opt/s/w/ir/x/t/flutter_tools.HIAXTK/flutter_ios_build_temp_dirA4t5kp/temporary_xcresult_bundle
[2023-05-05 09:12:06.060541] [STDOUT] [STDOUT] 
[2023-05-05 09:12:06.060563] [STDOUT] [STDOUT] 
[2023-05-05 09:12:06.060595] [STDOUT] [STDOUT] [   +5 ms] "flutter run" took 18,101ms.
[2023-05-05 09:12:06.064982] [STDOUT] [STDOUT] [  +10 ms] ensureAnalyticsSent: 0ms
[2023-05-05 09:12:06.065037] [STDOUT] [STDOUT] [        ] Running 1 shutdown hook
[2023-05-05 09:12:06.065587] [STDOUT] [STDOUT] [        ] Shutdown hooks complete
[2023-05-05 09:12:06.065782] [STDOUT] [STDOUT] [        ] exiting with code 2
[2023-05-05 09:12:06.074775] [STDOUT] Taking screenshot of working device 00008030-00062432346B402E at /opt/s/w/ir/x/w/recipe_cleanup/flutter_logs_dir/device-screenshot-2023-05-05T09:12:06.072117.png
[2023-05-05 09:12:06.188979] [STDOUT] Checking for reboot
[2023-05-05 09:12:06.192401] [STDOUT] rebooting
[2023-05-05 09:12:12.975182] [STDOUT] Failed to take screenshot. Continuing.
[microbenchmarks_ios] Process terminated with exit code 0.
Task result:

tracked in #126164

also:

 Task runner system failed with exception!
ext.cocoonRunTask: (-32000) Server error
Executable "/opt/s/w/ir/x/w/recipe_cleanup/tmpfxas5lcm/flutter sdk/bin/flutter" failed with exit code -6.
#0      fail (package:flutter_devicelab/framework/utils.dart:102:3)
#1      _execute (package:flutter_devicelab/framework/utils.dart:371:5)
<asynchronous suspension>
#2      eval (package:flutter_devicelab/framework/utils.dart:430:3)
<asynchronous suspension>
#3      IosDeviceDiscovery.discoverDevices (package:flutter_devicelab/framework/devices.dart:911:47)
<asynchronous suspension>
#4      IosDeviceDiscovery.chooseWorkingDevice (package:flutter_devicelab/framework/devices.dart:882:41)
<asynchronous suspension>
#5      IosDeviceDiscovery.workingDevice (package:flutter_devicelab/framework/devices.dart:872:7)
<asynchronous suspension>
#6      _TaskRunner._getWorkingDeviceIfAvailable (package:flutter_devicelab/framework/framework.dart:96:14)
<asynchronous suspension>
#7      _TaskRunner.run (package:flutter_devicelab/framework/framework.dart:153:30)
<asynchronous suspension>
#8      new _TaskRunner.<anonymous closure> (package:flutter_devicelab/framework/framework.dart:77:33)
<asynchronous suspension>

#0      new _OutstandingRequest (package:vm_service/src/vm_service.dart:1865:45)
#1      VmService._call (package:vm_service/src/vm_service.dart:2417:21)
#2      VmService.callServiceExtension (package:vm_service/src/vm_service.dart:2388:14)
#3      runTask (package:flutter_devicelab/framework/runner.dart:231:73)
<asynchronous suspension>
#4      rerunTask (package:flutter_devicelab/framework/runner.dart:113:29)
<asynchronous suspension>
#5      runTasks (package:flutter_devicelab/framework/runner.dart:51:16)
<asynchronous suspension>
#6      TestCommand.run (package:flutter_devicelab/command/test.dart:72:5)
<asynchronous suspension>
#7      CommandRunner.runCommand (package:args/command_runner.dart:212:13)
<asynchronous suspension>

[microbenchmarks_ios] Terminating process...

[cyanglaz]

5848, 5841, 5838, 5801 are app crash on main thread:

(lldb) process interrupt
[2023-05-04 19:40:42.272362] [STDOUT] [STDOUT] [ +369 ms] (lldb) Process 521 stopped
[2023-05-04 19:40:42.272855] [STDOUT] [STDOUT] [        ] * thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
[2023-05-04 19:40:42.272917] [STDOUT] [STDOUT] [        ]     frame #0: 0x00000001cc798aa8 libsystem_kernel.dylib`mach_msg2_trap + 8
[2023-05-04 19:40:42.273051] [STDOUT] [STDOUT] [        ] libsystem_kernel.dylib`mach_msg2_trap:
[2023-05-04 19:40:42.273190] [STDOUT] [STDOUT] [        ] ->  0x1cc798aa8 <+8>: ret    
[2023-05-04 19:40:42.273312] [STDOUT] [STDOUT] [        ] libsystem_kernel.dylib`macx_swapon:
[2023-05-04 19:40:42.273412] [STDOUT] [STDOUT] [        ]     0x1cc798aac <+0>: mov    x16, #-0x30
[2023-05-04 19:40:42.273506] [STDOUT] [STDOUT] [        ]     0x1cc798ab0 <+4>: svc    #0x80
[2023-05-04 19:40:42.273618] [STDOUT] [STDOUT] [        ]     0x1cc798ab4 <+8>: ret    
[2023-05-04 19:40:42.273680] [STDOUT] [STDOUT] [        ] Target 0: (Runner) stopped.
[2023-05-04 19:40:42.273855] [STDOUT] [STDOUT] [        ] thread backtrace all
[2023-05-04 19:40:42.274117] [STDOUT] [STDOUT] [        ] process continue
[2023-05-04 19:40:42.329324] [STDOUT] [STDOUT] [  +55 ms] * thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
[2023-05-04 19:40:42.329398] [STDOUT] [STDOUT] [        ]   * frame #0: 0x00000001cc798aa8 libsystem_kernel.dylib`mach_msg2_trap + 8
[2023-05-04 19:40:42.329452] [STDOUT] [STDOUT] [        ]     frame #1: 0x00000001cc7aafc4 libsystem_kernel.dylib`mach_msg2_internal + 80
[2023-05-04 19:40:42.329478] [STDOUT] [STDOUT] [        ]     frame #2: 0x00000001cc7ab204 libsystem_kernel.dylib`mach_msg_overwrite + 388
[2023-05-04 19:40:42.329686] [STDOUT] [STDOUT] [        ]     frame #3: 0x00000001cc798fec libsystem_kernel.dylib`mach_msg + 24
[2023-05-04 19:40:42.329731] [STDOUT] [STDOUT] [        ]     frame #4: 0x000000018ee9ead4 CoreFoundation`__CFRunLoopServiceMachPort + 160
[2023-05-04 19:40:42.329792] [STDOUT] [STDOUT] [        ]     frame #5: 0x000000018ee9fd18 CoreFoundation`__CFRunLoopRun + 1232
[2023-05-04 19:40:42.329822] [STDOUT] [STDOUT] [        ]     frame #6: 0x000000018eea4ec0 CoreFoundation`CFRunLoopRunSpecific + 612
[2023-05-04 19:40:42.329863] [STDOUT] [STDOUT] [        ]     frame #7: 0x00000001c8efb368 GraphicsServices`GSEventRunModal + 164
[2023-05-04 19:40:42.329959] [STDOUT] [STDOUT] [        ]     frame #8: 0x000000019139a86c UIKitCore`-[UIApplication _run] + 888
[2023-05-04 19:40:42.330024] [STDOUT] [STDOUT] [        ]     frame #9: 0x000000019139a4d0 UIKitCore`UIApplicationMain + 340
[2023-05-04 19:40:42.330106] [STDOUT] [STDOUT] [        ]     frame #10: 0x0000000102637d9c Runner`main(argc=<unavailable>, argv=<unavailable>) at main.m:10:16 [opt]
[2023-05-04 19:40:42.330138] [STDOUT] [STDOUT] [        ]     frame #11: 0x00000001ad6c6960 dyld`start + 2528

[cyanglaz]

5806 seems to be a timeout caused by libobjc.A.dylib is being read from process memory

[2023-05-03 00:16:17.252475] [STDOUT] [STDOUT] [        ] Application launched on the device. Waiting for Dart VM Service url.
[2023-05-03 00:16:20.574900] [STDOUT] [STDOUT] [+3322 ms] (lldb) warning: libobjc.A.dylib is being read from process memory. This indicates that LLDB could not find the on-disk shared cache for this device. This will likely reduce debugging performance.
[2023-05-03 00:16:47.255866] [STDOUT] [STDOUT] [   +2 ms] process interrupt

It looks similar to the observatory timeout in #121231 but with some additional logs.

cc @vashworth @christopherfujino

[vashworth]

@cyanglaz I also saw some (lldb) warning: libobjc.A.dylib is being read from process memory. issues on other test flakes today. I also ran into this issue locally when I updated one of my devices.

You can read a little about it here: #119147 (comment)

But seems to me that this issue happens when there's something wrong with the symbols. The ios_debug_symbol_doctor step tries to check to see if the xcdevice list info mentions the symbols are being fetched (flutter/cocoon#2421). However, I've found that to be unreliable.

I'm adding this check to help diagnose it: #126437

As for the Process 534 exited with status = -1 errors, that's tracked in this issue: #120808

And for Dart VM hanging forever and not being discovered is being tracked in this issue: #121231

[vashworth]

As for 5848, 5841, 5838, 5801 are app crash on main thread, they're not actually crashes. It's being paused and resumed, if you scroll to the bottom of the stack trace, it should say something like "(lldb) Process 521 resuming"

flutter/packages/flutter_tools/lib/src/ios/devices.dart

Line 549 in a3257ca

iosDeployDebugger?.pauseDumpBacktraceResume();

So those are instances of the Dart VM discovery hanging forever

[cyanglaz]

So those are instances of the Dart VM discovery hanging forever

Are these also #121231?

[vashworth]

So those are instances of the Dart VM discovery hanging forever

Are these also #121231?

Yes