Fix explicit layout validator memory alloc problems#56757
Merged
davidwrighton merged 4 commits intodotnet:mainfrom Aug 4, 2021
Merged
Conversation
Member
Author
|
@dotnet/crossgen-contrib |
Member
|
TIL. I suggested using a byte array when this was introduced because CoreCLR uses that and it was an easy to follow algorithm. Didn't think we allow such big classes. [StructLayout(LayoutKind.Explicit, Size = Int32.MaxValue)]
struct Boo { }
[StructLayout(LayoutKind.Explicit)]
class Program
{
[FieldOffset(0)]
Boo b;
static void Main() { }
}This crashes 5.0 CoreCLR on x86 big time (somewhere in native-land that I didn't debug). Fun. |
trylek
approved these changes
Aug 3, 2021
Member
trylek
left a comment
There was a problem hiding this comment.
LGTM, thanks for making this more robust!
src/coreclr/tools/Common/TypeSystem/Common/ExplicitLayoutValidator.cs
Outdated
Show resolved
Hide resolved
src/coreclr/tools/Common/TypeSystem/Common/ExplicitLayoutValidator.cs
Outdated
Show resolved
Hide resolved
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The ExplicitLayoutValidator currently used an array of bytes one to one with the size of the layout of the class. However, we have some test cases for outrageously large explicit layout structures and classes which cause this approach to OOM in crossgen2.
The fix is to move to a data structure defined in terms of intervals. This is probably a bit slower, but it requires vastly less memory for pathological situations.
Fixes #55164
Fixes #53559