[WIP] Checking Master with <XunitShowProgress>true</XunitShowProgress> Outerloop by Drawaes · Pull Request #24781 · dotnet/corefx

Drawaes · 2017-10-20T21:20:29Z

No description provided.

Drawaes · 2017-10-20T21:31:33Z

@dotnet-bot Test Outerloop Linux x64 Release Build

danmoseley · 2017-10-21T03:39:45Z

  <Import Project="$([MSBuild]::GetDirectoryNameOfFileAbove($(MSBuildThisFileDirectory), dir.props))\dir.props" />
  <PropertyGroup>
    <ProjectGuid>{A55A2B9A-830F-4330-A0E7-02A9FB30ABD2}</ProjectGuid>
+    <XunitShowProgress>true</XunitShowProgress>


I think you should add <XunitMaxThreads>1</XunitMaxThreads> also to make the output not be interleaved.

stephentoub · 2017-10-21T15:38:49Z

2017-10-21 04:45:44,685: INFO: proc(54): run_and_log_output: Output:    System.Net.Security.Tests.SslStreamStreamToStreamTest_BeginEnd.SslStream_StreamToStream_Authentication_IncorrectServerName_Fail [STARTING]
2017-10-21 04:45:44,797: INFO: proc(54): run_and_log_output: Output: 
2017-10-21 04:45:44,798: INFO: proc(54): run_and_log_output: Output: Unhandled Exception: System.NullReferenceException: Object reference not set to an instance of an object.
2017-10-21 04:45:44,798: INFO: proc(54): run_and_log_output: Output:    at System.Net.LazyAsyncResult.get_CurrentThreadContext() in /mnt/j/workspace/dotnet_corefx/master/linux-TGroup_netcoreapp+CGroup_Release+AGroup_x64+TestOuter_false_prtest/src/Common/src/System/Net/LazyAsyncResult.cs:line 26
2017-10-21 04:45:44,798: INFO: proc(54): run_and_log_output: Output:    at System.Net.LazyAsyncResult.Complete(IntPtr userToken) in /mnt/j/workspace/dotnet_corefx/master/linux-TGroup_netcoreapp+CGroup_Release+AGroup_x64+TestOuter_false_prtest/src/Common/src/System/Net/LazyAsyncResult.cs:line 404
2017-10-21 04:45:44,798: INFO: proc(54): run_and_log_output: Output:    at System.Net.Security.SslState.FinishHandshake(Exception e, AsyncProtocolRequest asyncRequest) in /mnt/j/workspace/dotnet_corefx/master/linux-TGroup_netcoreapp+CGroup_Release+AGroup_x64+TestOuter_false_prtest/src/System.Net.Security/src/System/Net/Security/SslState.cs:line 1454
2017-10-21 04:45:44,798: INFO: proc(54): run_and_log_output: Output:    at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest) in /mnt/j/workspace/dotnet_corefx/master/linux-TGroup_netcoreapp+CGroup_Release+AGroup_x64+TestOuter_false_prtest/src/System.Net.Security/src/System/Net/Security/SslState.cs:line 839
2017-10-21 04:45:44,798: INFO: proc(54): run_and_log_output: Output:    at System.Net.Security.SslState.WriteCallback(IAsyncResult transportResult) in /mnt/j/workspace/dotnet_corefx/master/linux-TGroup_netcoreapp+CGroup_Release+AGroup_x64+TestOuter_false_prtest/src/System.Net.Security/src/System/Net/Security/SslState.cs:line 1107
2017-10-21 04:45:44,798: INFO: proc(54): run_and_log_output: Output:    at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) in /root/coreclr/src/mscorlib/shared/System/Threading/ExecutionContext.cs:line 151
2017-10-21 04:45:44,799: INFO: proc(54): run_and_log_output: Output:    at System.Threading.Tasks.AwaitTaskContinuation.RunCallback(ContextCallback callback, Object state, Task& currentTask) in /root/coreclr/src/mscorlib/src/System/Threading/Tasks/TaskContinuation.cs:line 714
2017-10-21 04:45:44,799: INFO: proc(54): run_and_log_output: Output: --- End of stack trace from previous location where exception was thrown ---
2017-10-21 04:45:44,799: INFO: proc(54): run_and_log_output: Output:    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() in /root/coreclr/src/mscorlib/src/System/Runtime/ExceptionServices/ExceptionServicesCommon.cs:line 130
2017-10-21 04:45:44,799: INFO: proc(54): run_and_log_output: Output:    at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) in /root/coreclr/src/mscorlib/shared/System/Threading/ExecutionContext.cs:line 151
2017-10-21 04:45:44,799: INFO: proc(54): run_and_log_output: Output:    at System.Threading.ThreadPoolWorkQueue.Dispatch() in /root/coreclr/src/mscorlib/src/System/Threading/ThreadPool.cs:line 588

stephentoub · 2017-10-21T15:39:55Z

@dotnet-bottest Outerloop Linux x64 Debug Build
@dotnet-bottest Outerloop Linux x64 Release Build

stephentoub · 2017-10-21T15:45:24Z

@jkotas, could this failure be related to your recent ThreadStatic changes? The stack trace shows:

2017-10-21 04:45:44,798: INFO: proc(54): run_and_log_output: Output: Unhandled Exception: System.NullReferenceException: Object reference not set to an instance of an object.
2017-10-21 04:45:44,798: INFO: proc(54): run_and_log_output: Output:    at System.Net.LazyAsyncResult.get_CurrentThreadContext() in /mnt/j/workspace/dotnet_corefx/master/linux-TGroup_netcoreapp+CGroup_Release+AGroup_x64+TestOuter_false_prtest/src/Common/src/System/Net/LazyAsyncResult.cs:line 26
2017-10-21 04:45:44,798: INFO: proc(54): run_and_log_output: Output:    at System.Net.LazyAsyncResult.Complete(IntPtr userToken) in /mnt/j/workspace/dotnet_corefx/master/linux-TGroup_netcoreapp+CGroup_Release+AGroup_x64+TestOuter_false_prtest/src/Common/src/System/Net/LazyAsyncResult.cs:line 404

but get_CurrentThreadContext is:

corefx/src/Common/src/System/Net/LazyAsyncResult.cs

Lines 22 to 35 in ba63583

    
           private static ThreadContext CurrentThreadContext 
        
           { 
        
               get 
        
               { 
        
                   ThreadContext threadContext = t_threadContext; 
        
                   if (threadContext == null) 
        
                   { 
        
                       threadContext = new ThreadContext(); 
        
                       t_threadContext = threadContext; 
        
                   } 
        
                   return threadContext; 
        
               } 
        
           }

and I don't see how that could result in a null reference exception if everything was working properly.

Drawaes · 2017-10-21T15:55:07Z

I have the dumpling working thanks to @danmosemsft useful instructions. I can't really see anything obvious yet.

jkotas · 2017-10-21T16:11:09Z

This looks like https://github.com/dotnet/corefx/issues/24722 . The managed thread statics get corrupted in System.Net.Security tests on Linux. @janvorli and @Priya91 have been looking into it.

It is interesting that we are only seeing the problem in System.Net.Security tests on Linux so far. I would expect managed thread static regression to show up in many different tests ...

stephentoub · 2017-10-21T16:21:40Z

Same crash just happened here:
#24789
https://ci3.dot.net/job/dotnet_corefx/job/master/job/linux-TGroup_netcoreapp+CGroup_Release+AGroup_x64+TestOuter_false_prtest/4596/
https://mc.dot.net/#/user/stephentoub/pr~2Fjenkins~2Fdotnet~2Fcorefx~2Fmaster~2F/test~2Ffunctional~2Fcli~2F/b428e1ace35a75c97d0c3d370ec345c299f1daa1/workItem/System.Net.Security.Tests/wilogs

2017-10-21 15:43:58,660: INFO: proc(54): run_and_log_output: Output: Unhandled Exception: System.NullReferenceException: Object reference not set to an instance of an object.
2017-10-21 15:43:58,660: INFO: proc(54): run_and_log_output: Output:    at System.Net.LazyAsyncResult.get_CurrentThreadContext() in /mnt/j/workspace/dotnet_corefx/master/linux-TGroup_netcoreapp+CGroup_Release+AGroup_x64+TestOuter_false_prtest/src/Common/src/System/Net/LazyAsyncResult.cs:line 26
2017-10-21 15:43:58,661: INFO: proc(54): run_and_log_output: Output:    at System.Net.LazyAsyncResult.Complete(IntPtr userToken) in /mnt/j/workspace/dotnet_corefx/master/linux-TGroup_netcoreapp+CGroup_Release+AGroup_x64+TestOuter_false_prtest/src/Common/src/System/Net/LazyAsyncResult.cs:line 404
2017-10-21 15:43:58,661: INFO: proc(54): run_and_log_output: Output:    at System.Net.Security.SslState.FinishHandshake(Exception e, AsyncProtocolRequest asyncRequest) in /mnt/j/workspace/dotnet_corefx/master/linux-TGroup_netcoreapp+CGroup_Release+AGroup_x64+TestOuter_false_prtest/src/System.Net.Security/src/System/Net/Security/SslState.cs:line 1454
2017-10-21 15:43:58,661: INFO: proc(54): run_and_log_output: Output:    at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest) in /mnt/j/workspace/dotnet_corefx/master/linux-TGroup_netcoreapp+CGroup_Release+AGroup_x64+TestOuter_false_prtest/src/System.Net.Security/src/System/Net/Security/SslState.cs:line 839
2017-10-21 15:43:58,661: INFO: proc(54): run_and_log_output: Output:    at System.Net.Security.SslState.WriteCallback(IAsyncResult transportResult) in /mnt/j/workspace/dotnet_corefx/master/linux-TGroup_netcoreapp+CGroup_Release+AGroup_x64+TestOuter_false_prtest/src/System.Net.Security/src/System/Net/Security/SslState.cs:line 1107
2017-10-21 15:43:58,661: INFO: proc(54): run_and_log_output: Output:    at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) in /root/coreclr/src/mscorlib/shared/System/Threading/ExecutionContext.cs:line 151
2017-10-21 15:43:58,661: INFO: proc(54): run_and_log_output: Output:    at System.Threading.Tasks.AwaitTaskContinuation.RunCallback(ContextCallback callback, Object state, Task& currentTask) in /root/coreclr/src/mscorlib/src/System/Threading/Tasks/TaskContinuation.cs:line 714
2017-10-21 15:43:58,661: INFO: proc(54): run_and_log_output: Output: --- End of stack trace from previous location where exception was thrown ---
2017-10-21 15:43:58,663: INFO: proc(54): run_and_log_output: Output:    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() in /root/coreclr/src/mscorlib/src/System/Runtime/ExceptionServices/ExceptionServicesCommon.cs:line 130
2017-10-21 15:43:58,663: INFO: proc(54): run_and_log_output: Output:    at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) in /root/coreclr/src/mscorlib/shared/System/Threading/ExecutionContext.cs:line 151
2017-10-21 15:43:58,663: INFO: proc(54): run_and_log_output: Output:    at System.Threading.ThreadPoolWorkQueue.Dispatch() in /root/coreclr/src/mscorlib/src/System/Threading/ThreadPool.cs:line 588
2017-10-21 15:44:00,468: INFO: proc(54): run_and_log_output: Output: /home/helixbot/dotnetbuild/work/a8f6c3e1-882c-409f-bd91-7fff541d620e/Work/5b625004-0718-4f0c-9e30-8112fb60b460/Unzip/RunTests.sh: line 87:  1725 Aborted                 (core dumped) $RUNTIME_PATH/dotnet xunit.console.netcore.exe System.Net.Security.Tests.dll -xml testResults.xml -notrait Benchmark=true -notrait category=nonnetcoreapptests -notrait category=nonlinuxtests -notrait category=OuterLoop -notrait category=failing
2017-10-21 15:44:58,704: INFO: proc(54): run_and_log_output: Output: processing dump file /home/helixbot/dotnetbuild/work/a8f6c3e1-882c-409f-bd91-7fff541d620e/Work/5b625004-0718-4f0c-9e30-8112fb60b460/Unzip/core
2017-10-21 15:44:58,705: INFO: proc(54): run_and_log_output: Output: creating dumpling dump 8712226244430cb9a35b4aba164ca646a2b813ab

Drawaes · 2017-10-22T16:45:37Z

ref #24799

Added <XunitShowProgress>true</XunitShowProgress>

26b56e9

dnfclas added the cla-already-signed label Oct 20, 2017

Drawaes changed the title ~~WIP Checking Master with <XunitShowProgress>true</XunitShowProgress> Outerloop~~ [WIP] Checking Master with <XunitShowProgress>true</XunitShowProgress> Outerloop Oct 20, 2017

davidsh added the area-System.Net.Security label Oct 20, 2017

danmoseley reviewed Oct 21, 2017

View reviewed changes

Single proc tests

523ab8b

danmoseley approved these changes Oct 21, 2017

View reviewed changes

Drawaes closed this Oct 22, 2017

karelz added this to the 2.1.0 milestone Oct 28, 2017

Drawaes deleted the TEST branch November 3, 2017 22:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[WIP] Checking Master with <XunitShowProgress>true</XunitShowProgress> Outerloop - #24781

[WIP] Checking Master with <XunitShowProgress>true</XunitShowProgress> Outerloop#24781
Drawaes wants to merge 2 commits into
dotnet:masterfrom
Drawaes:TEST

Drawaes commented Oct 20, 2017

Uh oh!

Drawaes commented Oct 20, 2017

Uh oh!

danmoseley Oct 21, 2017 •

edited

Loading

Uh oh!

stephentoub commented Oct 21, 2017

Uh oh!

stephentoub commented Oct 21, 2017

Uh oh!

stephentoub commented Oct 21, 2017

Uh oh!

Drawaes commented Oct 21, 2017

Uh oh!

jkotas commented Oct 21, 2017 •

edited

Loading

Uh oh!

stephentoub commented Oct 21, 2017 •

edited

Loading

Uh oh!

Drawaes commented Oct 22, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Uh oh!

Conversation

Drawaes commented Oct 20, 2017

Uh oh!

Drawaes commented Oct 20, 2017

Uh oh!

danmoseley Oct 21, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stephentoub commented Oct 21, 2017

Uh oh!

stephentoub commented Oct 21, 2017

Uh oh!

stephentoub commented Oct 21, 2017

Uh oh!

Drawaes commented Oct 21, 2017

Uh oh!

jkotas commented Oct 21, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stephentoub commented Oct 21, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Drawaes commented Oct 22, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

danmoseley Oct 21, 2017 •

edited

Loading

jkotas commented Oct 21, 2017 •

edited

Loading

stephentoub commented Oct 21, 2017 •

edited

Loading