Skip to content
This repository was archived by the owner on Jan 23, 2023. It is now read-only.

Many optimizations for System.Linq.Enumerable. - #2318

Closed
jamesqo wants to merge 14 commits into
dotnet:masterfrom
jamesqo:patch-2
Closed

Many optimizations for System.Linq.Enumerable.#2318
jamesqo wants to merge 14 commits into
dotnet:masterfrom
jamesqo:patch-2

Conversation

@jamesqo

@jamesqo jamesqo commented Jul 11, 2015

Copy link
Copy Markdown
Contributor

Just sat down for 5 hours looking at every method in Enumerable.cs to make every optimization I possibly could.

Here's a summary of the changes:

  • WhereEnumerableIterator<TSource> and WhereSelectEnumerableIterator<TSource> no longer check if IEnumerator<T> implements IDisposable (it does)
  • SkipWhileIterator: remove redundant checks on yielding
  • RangeIterator has been optimized to avoid double adding
  • Avoid null checks on int? and long? and etc. before calling GetValueOrDefault() (you can call a method on null if it's a nullable)
  • Optimize every single Min and Max call:
    • No more first-time loop checking.
    • Nullables like int? and long? avoid many null-checks after "scrolling forward" to the first non-null element in the collection.
    • float? and double? avoid another null-check with a trick using GetValueOrDefault()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will now blow up if _enumerator is null. It should really be:

if (_enumerator != null)
    _enumerator.Dispose();

Note that no cast at all is needed.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a check for null before disposing the enumerator, but not casting to IDisposable may cause backwards compatibility issues if it has an explicit interface implementation.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you provide an example for this case where it would cause an issue?

@jamesqo jamesqo changed the title Many, many optimizations for System.Linq.Enumerable. Many optimizations for System.Linq.Enumerable. Jul 11, 2015

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: would be nice to follow general CoreFX repo convention of using string over String here and change these to double.IsNaN instead (like you used above).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jasonwilliams200OK Since I reverted all my changes to Min and Max, the notation has gone back to System.Single.NaN and System.Double.NaN, from the original implementer.

@stephentoub

Copy link
Copy Markdown
Member

cc: @VSadov

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm still unclear what case you're trying to protect against by explicitly casting the IEnumerator<T> to IDisposable<T>. Can you provide an example? This change is also no longer calling Dispose on the base in the case where MoveNext() was never called to initialize _enumerator.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@stephentoub Fixed this in the latest commit; the cast is no longer there, and base.Dispose() has moved outside the block. I was keeping the cast before because I had seen it in other places throughout the BCL and there seem to be a few questions specifically about this, but I wasn't able to find any problems.

@JonHanna

Copy link
Copy Markdown
Contributor
if (_enumerator is IDisposable) ((IDisposable)_enumerator).Dispose();

This was no doubt a hang-over of somebody being used to the fact that IEnumerable doesn't derive from IDisposable and one of the three reasons that the later IEnumerable<T> does is this check had to be done and also that people might forget it (the third being that people might forget to implement it in cases where they should).

It seems the compiler already does the smart thing of producing the IL that would come from:

if (_enumerator != null) _enumerator.Dispose();

So the change doesn't help performance, but I do think changing from the former to the latter is a nice readability improvement.

@jamesqo

jamesqo commented Jul 12, 2015

Copy link
Copy Markdown
Contributor Author

@hackcraft Saw your message right after I pushed the commit. I think the original writers may have been watching out for an explicit interface implementation or something of the sort that may have required them to explicitly cast to IDisposable before calling Dispose, but ultimately that shouldn't matter since IEnumerator<T> allows you to call it without casting.

This confused me for a while as well, because there are many questions on Stack Overflow asking about explicit casting to IDisposable, and a lot of other places in the BCL where this happens as well.

@jamesqo

jamesqo commented Jul 12, 2015

Copy link
Copy Markdown
Contributor Author

Hm, it seems we're getting build errors because some of the tests for Enumerable.Average() failed...


Got it. The removal of the null checks has affected the count of numbers in the sequence, for nullable numbers. The branch should build (and be ready) after the next commit.

@jamesqo

jamesqo commented Jul 15, 2015

Copy link
Copy Markdown
Contributor Author

@stephentoub I've undid my changes to Min and Max; I don't think there's anything left to do now.

@jamesqo

jamesqo commented Jul 16, 2015

Copy link
Copy Markdown
Contributor Author

It looks like I'm getting a different error this time- it's coming from Windows builds instead of Linux, and it's something about ThreadPoolBoundHandleTests. Some other PRs have been failing with EtwTests on Linux recently, so I thought I'd post this here.

@stephentoub

Copy link
Copy Markdown
Member

it's something about ThreadPoolBoundHandleTests

Hmm, that's not what I'm seeing (maybe the tests got rerun between now and then). Both the debug and release tests failed for System.Linq.Tests with ~30 failures:

Discovering: System.Linq.Tests
  Discovered:  System.Linq.Tests
  Starting:    System.Linq.Tests
     System.Linq.Tests.SkipWhileTests.Fundamentals [FAIL]
        Assert.True() Failure
        Stack Trace:
           d:\j\workspace\dotnet_corefx_windows_release_prtest\src\System.Linq\tests\SkipWhileTests.cs(12,0): at System.Linq.Tests.SkipWhileTests.Fundamentals()
     System.Linq.Tests.RangeTests.Range_ToArray_ProduceCorrectResult [FAIL]
        Assert.Equal() Failure
        Expected: 0
        Actual:   100
        Stack Trace:
           d:\j\workspace\dotnet_corefx_windows_release_prtest\src\System.Linq\tests\RangeTests.cs(30,0): at System.Linq.Tests.RangeTests.Range_ToArray_ProduceCorrectResult()
     System.Linq.Tests.MinMaxTests.MinInt32 [FAIL]
        System.InvalidOperationException : Sequence contains no elements
        Stack Trace:
           d:\j\workspace\dotnet_corefx_windows_release_prtest\src\System.Linq\src\System\Linq\Enumerable.cs(1789,0): at System.Linq.Enumerable.Min(IEnumerable`1 source)
           d:\j\workspace\dotnet_corefx_windows_release_prtest\src\System.Linq\tests\MinMaxTests.cs(15,0): at System.Linq.Tests.MinMaxTests.MinInt32()
     System.Linq.Tests.SingleTests.FindSingleMatch(target: ???, range: ???) [FAIL]
        System.InvalidOperationException : The test method expected 2 parameter values, but 0 parameter values were provided.
     System.Linq.Tests.SingleTests.FindSingleMatch(target: ???, range: ???) [FAIL]
        System.InvalidOperationException : The test method expected 2 parameter values, but 0 parameter values were provided.
     System.Linq.Tests.EnumerableTests.All [FAIL]
        Assert.False() Failure
        Stack Trace:
           d:\j\workspace\dotnet_corefx_windows_release_prtest\src\System.Linq\tests\EnumerableTests.cs(93,0): at System.Linq.Tests.EnumerableTests.All()
     System.Linq.Tests.EnumerableTests.Any [FAIL]
        Assert.True() Failure
        Stack Trace:
           d:\j\workspace\dotnet_corefx_windows_release_prtest\src\System.Linq\tests\EnumerableTests.cs(101,0): at System.Linq.Tests.EnumerableTests.Any()
     System.Linq.Tests.EnumerableTests.Count [FAIL]
        Assert.Equal() Failure
        Expected: 100
        Actual:   0
        Stack Trace:
           d:\j\workspace\dotnet_corefx_windows_release_prtest\src\System.Linq\tests\EnumerableTests.cs(107,0): at System.Linq.Tests.EnumerableTests.CountAndValidate[T](IEnumerable`1 enumerable, Int32 expectedCount)
           d:\j\workspace\dotnet_corefx_windows_release_prtest\src\System.Linq\tests\EnumerableTests.cs(118,0): at System.Linq.Tests.EnumerableTests.Count()
     System.Linq.Tests.MinMaxTests.MaxInt32WithSelector [FAIL]
        System.InvalidOperationException : Sequence contains no elements
        Stack Trace:
           d:\j\workspace\dotnet_corefx_windows_release_prtest\src\System.Linq\src\System\Linq\Enumerable.cs(2048,0): at System.Linq.Enumerable.Max(IEnumerable`1 source)
           d:\j\workspace\dotnet_corefx_windows_release_prtest\src\System.Linq\tests\MinMaxTests.cs(436,0): at System.Linq.Tests.MinMaxTests.MaxInt32WithSelector()
     System.Linq.Tests.EnumerableTests.OrderByExtremeComparer [FAIL]
        Assert.Equal() Failure
        Expected: <ReverseIterator>d__9c<Int32> []
        Actual:   OrderedEnumerable<Int32, Int32> [9, 8, 7, 6, 5, ...]
        Stack Trace:
           d:\j\workspace\dotnet_corefx_windows_release_prtest\src\System.Linq\tests\EnumerableTests.cs(193,0): at System.Linq.Tests.EnumerableTests.OrderByExtremeComparer()
     System.Linq.Tests.ToArrayTests.ToArray_ProduceCorrectArray [FAIL]
        Assert.Equal() Failure
        Expected: 7
        Actual:   0
        Stack Trace:
           d:\j\workspace\dotnet_corefx_windows_release_prtest\src\System.Linq\tests\ToArrayTests.cs(90,0): at System.Linq.Tests.ToArrayTests.<>c__DisplayClass8.<ToArray_ProduceCorrectArray>b__6(Int32[] resultArray)
           d:\j\workspace\dotnet_corefx_windows_release_prtest\src\System.Linq\tests\ToArrayTests.cs(66,0): at System.Linq.Tests.ToArrayTests.RunToArrayOnAllCollectionTypes[T](T[] items, Action`1 validation)
           d:\j\workspace\dotnet_corefx_windows_release_prtest\src\System.Linq\tests\ToArrayTests.cs(87,0): at System.Linq.Tests.ToArrayTests.ToArray_ProduceCorrectArray()
     System.Linq.Tests.ToArrayTests.ToArray_ArrayWhereSelect(sourceIntegers: ???, convertedStrings: ???) [FAIL]
        System.InvalidOperationException : The test method expected 2 parameter values, but 0 parameter values were provided.
     System.Linq.Tests.ToArrayTests.ToArray_ArrayWhereSelect(sourceIntegers: ???, convertedStrings: ???) [FAIL]
        System.InvalidOperationException : The test method expected 2 parameter values, but 0 parameter values were provided.
     System.Linq.Tests.ToArrayTests.ToArray_ArrayWhereSelect(sourceIntegers: ???, convertedStrings: ???) [FAIL]
        System.InvalidOperationException : The test method expected 2 parameter values, but 0 parameter values were provided.
     System.Linq.Tests.ToArrayTests.ToArray_ListWhereSelect(sourceIntegers: ???, convertedStrings: ???) [FAIL]
        System.InvalidOperationException : The test method expected 2 parameter values, but 0 parameter values were provided.
     System.Linq.Tests.ToArrayTests.ToArray_ListWhereSelect(sourceIntegers: ???, convertedStrings: ???) [FAIL]
        System.InvalidOperationException : The test method expected 2 parameter values, but 0 parameter values were provided.
     System.Linq.Tests.ToArrayTests.ToArray_ListWhereSelect(sourceIntegers: ???, convertedStrings: ???) [FAIL]
        System.InvalidOperationException : The test method expected 2 parameter values, but 0 parameter values were provided.
     System.Linq.Tests.ReverseTests.ReverseMatches(input: ???) [FAIL]
        System.InvalidOperationException : The test method expected 1 parameter value, but 0 parameter values were provided.
     System.Linq.Tests.ReverseTests.ReverseMatches(input: ???) [FAIL]
        System.InvalidOperationException : The test method expected 1 parameter value, but 0 parameter values were provided.
     System.Linq.Tests.ReverseTests.ReverseMatches(input: ???) [FAIL]
        System.InvalidOperationException : The test method expected 1 parameter value, but 0 parameter values were provided.
     System.Linq.Tests.ReverseTests.ReverseMatches(input: ???) [FAIL]
        System.InvalidOperationException : The test method expected 1 parameter value, but 0 parameter values were provided.
     System.Linq.Tests.RepeatTests.Repeat_ToArray_ProduceCorrectResult [FAIL]
        Assert.Equal() Failure
        Expected: 0
        Actual:   100
        Stack Trace:
           d:\j\workspace\dotnet_corefx_windows_release_prtest\src\System.Linq\tests\RepeatTests.cs(30,0): at System.Linq.Tests.RepeatTests.Repeat_ToArray_ProduceCorrectResult()
     System.Linq.Tests.RepeatTests.Repeat_ProduceSameObject [FAIL]
        Assert.Equal() Failure
        Expected: 0
        Actual:   100
        Stack Trace:
           d:\j\workspace\dotnet_corefx_windows_release_prtest\src\System.Linq\tests\RepeatTests.cs(40,0): at System.Linq.Tests.RepeatTests.Repeat_ProduceSameObject()
     System.Linq.Tests.RepeatTests.Repeat_WorkWithNullElement [FAIL]
        Assert.Equal() Failure
        Expected: 0
        Actual:   100
        Stack Trace:
           d:\j\workspace\dotnet_corefx_windows_release_prtest\src\System.Linq\tests\RepeatTests.cs(50,0): at System.Linq.Tests.RepeatTests.Repeat_WorkWithNullElement()
     System.Linq.Tests.MinMaxTests.NaNFirstSingleWithSelector [FAIL]
        System.InvalidOperationException : Sequence contains no elements
        Stack Trace:
           d:\j\workspace\dotnet_corefx_windows_release_prtest\src\System.Linq\src\System\Linq\Enumerable.cs(1860,0): at System.Linq.Enumerable.Min(IEnumerable`1 source)
           d:\j\workspace\dotnet_corefx_windows_release_prtest\src\System.Linq\tests\MinMaxTests.cs(722,0): at System.Linq.Tests.MinMaxTests.NaNFirstSingleWithSelector()
     System.Linq.Tests.MinMaxTests.NaNFirstDoubleWithSelector [FAIL]
        System.InvalidOperationException : Sequence contains no elements
        Stack Trace:
           d:\j\workspace\dotnet_corefx_windows_release_prtest\src\System.Linq\src\System\Linq\Enumerable.cs(1893,0): at System.Linq.Enumerable.Min(IEnumerable`1 source)
           d:\j\workspace\dotnet_corefx_windows_release_prtest\src\System.Linq\tests\MinMaxTests.cs(739,0): at System.Linq.Tests.MinMaxTests.NaNFirstDoubleWithSelector()
     System.Linq.Tests.MinMaxTests.MinInt32WithSelector [FAIL]
        System.InvalidOperationException : Sequence contains no elements
        Stack Trace:
           d:\j\workspace\dotnet_corefx_windows_release_prtest\src\System.Linq\src\System\Linq\Enumerable.cs(1789,0): at System.Linq.Enumerable.Min(IEnumerable`1 source)
           d:\j\workspace\dotnet_corefx_windows_release_prtest\src\System.Linq\tests\MinMaxTests.cs(422,0): at System.Linq.Tests.MinMaxTests.MinInt32WithSelector()
     System.Linq.Tests.MinMaxTests.MaxInt32 [FAIL]
        System.InvalidOperationException : Sequence contains no elements
        Stack Trace:
           d:\j\workspace\dotnet_corefx_windows_release_prtest\src\System.Linq\src\System\Linq\Enumerable.cs(2048,0): at System.Linq.Enumerable.Max(IEnumerable`1 source)
           d:\j\workspace\dotnet_corefx_windows_release_prtest\src\System.Linq\tests\MinMaxTests.cs(29,0): at System.Linq.Tests.MinMaxTests.MaxInt32()
     System.Linq.Tests.MinMaxTests.NaNFirstSingle [FAIL]
        System.InvalidOperationException : Sequence contains no elements
        Stack Trace:
           d:\j\workspace\dotnet_corefx_windows_release_prtest\src\System.Linq\src\System\Linq\Enumerable.cs(1860,0): at System.Linq.Enumerable.Min(IEnumerable`1 source)
           d:\j\workspace\dotnet_corefx_windows_release_prtest\src\System.Linq\tests\MinMaxTests.cs(315,0): at System.Linq.Tests.MinMaxTests.NaNFirstSingle()
     System.Linq.Tests.MinMaxTests.NaNFirstDouble [FAIL]
        System.InvalidOperationException : Sequence contains no elements
        Stack Trace:
           d:\j\workspace\dotnet_corefx_windows_release_prtest\src\System.Linq\src\System\Linq\Enumerable.cs(1893,0): at System.Linq.Enumerable.Min(IEnumerable`1 source)
           d:\j\workspace\dotnet_corefx_windows_release_prtest\src\System.Linq\tests\MinMaxTests.cs(331,0): at System.Linq.Tests.MinMaxTests.NaNFirstDouble()
  Finished:    System.Linq.Tests

  === TEST EXECUTION SUMMARY ===
     System.Linq.Tests  Total: 289, Errors: 0, Failed: 30, Skipped: 0, Time: 0.469s

@stephentoub

Copy link
Copy Markdown
Member

@dotnet-bot test this please

@jamesqo

jamesqo commented Aug 6, 2015

Copy link
Copy Markdown
Contributor Author

Splitting this up into a few new PRs.

@jamesqo jamesqo closed this Aug 6, 2015
@jamesqo
jamesqo deleted the patch-2 branch August 8, 2015 00:52
pgavlin added a commit to pgavlin/corefx that referenced this pull request Oct 20, 2015
This change also re-enables a disabled NameResolution PAL test.

Fixes dotnet#2318.
@pgavlin pgavlin self-assigned this Oct 20, 2015
@karelz karelz modified the milestone: 1.0.0-rtm Dec 3, 2016
picenka21 pushed a commit to picenka21/runtime that referenced this pull request Feb 18, 2022
This change also re-enables a disabled NameResolution PAL test.

Fixes dotnet/corefx#2318.


Commit migrated from dotnet/corefx@cf92d01
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants