System.IO.FileSystem.Watcher.Tests Fails with Open File Descriptors Limit Reached when TestReadyToRun is Enabled

[ivdiazsa]

Description

When building and running System.IO.FileSystem.Watcher.Tests with TestReadyToRun=true, the test run will crash and fail with an error message about the limit of open file descriptors allowed.

Reproduction Steps

1. Clone the repo and build the clr and libs subsets. Make sure you specify TestReadyToRun, so that some necessary pretest components get built.

git clone https://github.com/dotnet/runtime
cd runtime
./build.sh --subset clr+libs --configuration Release -p:TestReadyToRun=true

2. If PR Add ActiveIssue Annotations to Failing TestReadyToRun Tests and Reenable the Executables #96721 has been merged when investigating this issue, then first reenable the test by removing its ActiveIssue annotation in src/libraries/System.IO.FileSystem.Watcher/tests/FileSystemWatcher.File.Move.cs:

// Comment this one out --> [ActiveIssue("https://github.com/dotnet/runtime/issues/96728", typeof(PlatformDetection), nameof(PlatformDetection.IsReadyToRunCompiled))]
private void FileMove_FromWatchedToUnwatched(WatcherChangeTypes eventType)
{
    string dir_watched = CreateTestDirectory(TestDirectory, "dir_watched");
    string dir_unwatched = CreateTestDirectory(TestDirectory, "dir_unwatched");
    string testFile = CreateTestFile(dir_watched, "file");
    using (var watcher = new FileSystemWatcher(dir_watched, "*"))
    {
        string sourcePath = testFile; // watched
        string targetPath = Path.Combine(dir_unwatched, "file");

        Action action = () => File.Move(sourcePath, targetPath);
        Action cleanup = () => File.Move(targetPath, sourcePath);

        ExpectEvent(watcher, eventType, action, cleanup, sourcePath);
    }
}

3. Build and run the test

cd src/libraries/System.IO.FileSystem.Watcher/tests
/path/to/runtime/dotnet.sh build -c Release -t:Test -p:TestReadyToRun=true

Expected behavior

All tests pass, except the skipped ones.

Actual behavior

Test fails with the following error, making the whole run fail:

========================= Begin custom configuration settings ==============================
export __TestArchitecture=x64
export __IsXUnitLogCheckerSupported=1
========================== End custom configuration settings ===============================
----- start Tue Jan 9 03:36:03 PM PST 2024 =============== To repro directly: =====================================================
pushd /path/to/runtime/artifacts/bin/System.IO.FileSystem.Watcher.Tests/Release/net9.0-linux/publish
chmod +rwx System.IO.FileSystem.Watcher.Tests ^&^& ./System.IO.FileSystem.Watcher.Tests -notrait category=OuterLoop -notrait category=failing -xml testResults.xml 
popd
===========================================================================================================
/path/to/runtime/artifacts/bin/System.IO.FileSystem.Watcher.Tests/Release/net9.0-linux/publish /path/to/runtime/src/libraries/System.IO.FileSystem.Watcher/tests
Running assembly:System.IO.FileSystem.Watcher.Tests, Version=9.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51
[FAIL] System.IO.Tests.File_NotifyFilter_Tests.FileSystemWatcher_File_NotifyFilter_LastWriteTime(filter: LastWrite)
System.IO.IOException : The configured user limit (128) on the number of inotify instances has been reached, or the per-process limit on the number of open file descriptors has been reached.
   at System.IO.FileSystemWatcher.StartRaisingEvents()
   at System.IO.Tests.FileSystemWatcherTest.ExecuteAndVerifyEvents(FileSystemWatcher watcher, WatcherChangeTypes expectedEvents, Action action, Boolean assertExpected, String[] expectedPaths, Int32 timeout) in /path/to/runtime/src/libraries/System.IO.FileSystem.Watcher/tests/Utility/FileSystemWatcherTest.cs:line 308
   at System.IO.Tests.FileSystemWatcherTest.ExpectEvent(FileSystemWatcher watcher, WatcherChangeTypes expectedEvents, Action action, Action cleanup, String[] expectedPaths, Int32 attempts, Int32 timeout) in /path/to/runtime/src/libraries/System.IO.FileSystem.Watcher/tests/Utility/FileSystemWatcherTest.cs:line 198
   at System.IO.Tests.File_NotifyFilter_Tests.FileSystemWatcher_File_NotifyFilter_LastWriteTime(NotifyFilters filter) in /path/to/runtime/src/libraries/System.IO.FileSystem.Watcher/tests/FileSystemWatcher.File.NotifyFilter.cs:line 136
   at InvokeStub_File_NotifyFilter_Tests.FileSystemWatcher_File_NotifyFilter_LastWriteTime(Object, Span`1)
   at System.Reflection.MethodBaseInvoker.InvokeWithOneArg(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
[FAIL] System.IO.Tests.Directory_NotifyFilter_Tests.FileSystemWatcher_Directory_NotifyFilter_Attributes(filter: Size)
System.IO.IOException : The configured user limit (128) on the number of inotify instances has been reached, or the per-process limit on the number of open file descriptors has been reached.
   at System.IO.FileSystemWatcher.StartRaisingEvents()
   at System.IO.Tests.FileSystemWatcherTest.ExecuteAndVerifyEvents(FileSystemWatcher watcher, WatcherChangeTypes expectedEvents, Action action, Boolean assertExpected, String[] expectedPaths, Int32 timeout) in /path/to/runtime/src/libraries/System.IO.FileSystem.Watcher/tests/Utility/FileSystemWatcherTest.cs:line 308
   at System.IO.Tests.FileSystemWatcherTest.ExpectEvent(FileSystemWatcher watcher, WatcherChangeTypes expectedEvents, Action action, Action cleanup, String[] expectedPaths, Int32 attempts, Int32 timeout) in /path/to/runtime/src/libraries/System.IO.FileSystem.Watcher/tests/Utility/FileSystemWatcherTest.cs:line 198
   at System.IO.Tests.Directory_NotifyFilter_Tests.<>c__DisplayClass4_0.<FileSystemWatcher_Directory_NotifyFilter_Attributes>b__0() in /path/to/runtime/src/libraries/System.IO.FileSystem.Watcher/tests/FileSystemWatcher.Directory.NotifyFilter.cs:line 46
   at System.IO.Tests.FileSystemWatcherTest.Execute(Action test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName) in /path/to/runtime/src/libraries/System.IO.FileSystem.Watcher/tests/Utility/FileSystemWatcherTest.cs:line 238
   at InvokeStub_Directory_NotifyFilter_Tests.FileSystemWatcher_Directory_NotifyFilter_Attributes(Object, Span`1)
   at System.Reflection.MethodBaseInvoker.InvokeWithOneArg(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
[FAIL] System.IO.Tests.WaitForChangedTests.NonZeroTimeout_NoEvents_TimesOut(enabledBeforeWait: True, useTimeSpan: True)
System.IO.IOException : The configured user limit (128) on the number of inotify instances has been reached, or the per-process limit on the number of open file descriptors has been reached.
   at System.IO.FileSystemWatcher.StartRaisingEvents()
   at System.IO.Tests.WaitForChangedTests.NonZeroTimeout_NoEvents_TimesOut(Boolean enabledBeforeWait, Boolean useTimeSpan) in /path/to/runtime/src/libraries/System.IO.FileSystem.Watcher/tests/FileSystemWatcher.WaitForChanged.cs:line 113
   at InvokeStub_WaitForChangedTests.NonZeroTimeout_NoEvents_TimesOut(Object, Span`1)
   at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
[FAIL] System.IO.Tests.File_Move_Tests.Linux_File_Move_To_Different_Watched_Directory
System.IO.IOException : The configured user limit (128) on the number of inotify instances has been reached, or the per-process limit on the number of open file descriptors has been reached.
   at System.IO.FileSystemWatcher.StartRaisingEvents()
   at System.IO.Tests.FileSystemWatcherTest.ExecuteAndVerifyEvents(FileSystemWatcher watcher, WatcherChangeTypes expectedEvents, Action action, Boolean assertExpected, String[] expectedPaths, Int32 timeout) in /path/to/runtime/src/libraries/System.IO.FileSystem.Watcher/tests/Utility/FileSystemWatcherTest.cs:line 308
   at System.IO.Tests.FileSystemWatcherTest.ExpectEvent(FileSystemWatcher watcher, WatcherChangeTypes expectedEvents, Action action, Action cleanup, String[] expectedPaths, Int32 attempts, Int32 timeout) in /path/to/runtime/src/libraries/System.IO.FileSystem.Watcher/tests/Utility/FileSystemWatcherTest.cs:line 198
   at System.IO.Tests.File_Move_Tests.FileMove_DifferentWatchedDirectory(WatcherChangeTypes eventType) in /path/to/runtime/src/libraries/System.IO.FileSystem.Watcher/tests/FileSystemWatcher.File.Move.cs:line 184
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
Finished System.IO.FileSystem.Watcher.Tests, Version=9.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51

Regression?

No

Known Workarounds

Keep the test disabled so that the other tests in the run can be executed unaffected.

Tagging subscribers to this area: @dotnet/area-system-io
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

When building and running System.IO.FileSystem.Watcher.Tests with TestReadyToRun=true, the test run will crash and fail with an error message about the limit of open file descriptors allowed.

Reproduction Steps

1. Clone the repo and build the clr and libs subsets. Make sure you specify TestReadyToRun, so that some necessary pretest components get built.

git clone https://github.com/dotnet/runtime
cd runtime
./build.sh --subset clr+libs --configuration Release -p:TestReadyToRun=true

2. If PR Add ActiveIssue Annotations to Failing TestReadyToRun Tests and Reenable the Executables #96721 has been merged when investigating this issue, then first reenable the test by removing its ActiveIssue annotation in src/libraries/System.IO.FileSystem.Watcher/tests/FileSystemWatcher.File.Move.cs:

// Comment this one out --> [ActiveIssue("Issue", typeof(PlatformDetection), nameof(PlatformDetection.IsReadyToRunCompiled))]
private void FileMove_FromWatchedToUnwatched(WatcherChangeTypes eventType)
{
    string dir_watched = CreateTestDirectory(TestDirectory, "dir_watched");
    string dir_unwatched = CreateTestDirectory(TestDirectory, "dir_unwatched");
    string testFile = CreateTestFile(dir_watched, "file");
    using (var watcher = new FileSystemWatcher(dir_watched, "*"))
    {
        string sourcePath = testFile; // watched
        string targetPath = Path.Combine(dir_unwatched, "file");

        Action action = () => File.Move(sourcePath, targetPath);
        Action cleanup = () => File.Move(targetPath, sourcePath);

        ExpectEvent(watcher, eventType, action, cleanup, sourcePath);
    }
}

3. Build and run the test

cd src/libraries/System.IO.FileSystem.Watcher/tests
/path/to/runtime/dotnet.sh build -c Release -t:Test -p:TestReadyToRun=true

Expected behavior

All tests pass, except the skipped ones.

Actual behavior

Test fails with the following error, making the whole run fail:

========================= Begin custom configuration settings ==============================
export __TestArchitecture=x64
export __IsXUnitLogCheckerSupported=1
========================== End custom configuration settings ===============================
----- start Tue Jan 9 03:36:03 PM PST 2024 =============== To repro directly: =====================================================
pushd /path/to/runtime/artifacts/bin/System.IO.FileSystem.Watcher.Tests/Release/net9.0-linux/publish
chmod +rwx System.IO.FileSystem.Watcher.Tests ^&^& ./System.IO.FileSystem.Watcher.Tests -notrait category=OuterLoop -notrait category=failing -xml testResults.xml 
popd
===========================================================================================================
/path/to/runtime/artifacts/bin/System.IO.FileSystem.Watcher.Tests/Release/net9.0-linux/publish /path/to/runtime/src/libraries/System.IO.FileSystem.Watcher/tests
Running assembly:System.IO.FileSystem.Watcher.Tests, Version=9.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51
[FAIL] System.IO.Tests.File_NotifyFilter_Tests.FileSystemWatcher_File_NotifyFilter_LastWriteTime(filter: LastWrite)
System.IO.IOException : The configured user limit (128) on the number of inotify instances has been reached, or the per-process limit on the number of open file descriptors has been reached.
   at System.IO.FileSystemWatcher.StartRaisingEvents()
   at System.IO.Tests.FileSystemWatcherTest.ExecuteAndVerifyEvents(FileSystemWatcher watcher, WatcherChangeTypes expectedEvents, Action action, Boolean assertExpected, String[] expectedPaths, Int32 timeout) in /path/to/runtime/src/libraries/System.IO.FileSystem.Watcher/tests/Utility/FileSystemWatcherTest.cs:line 308
   at System.IO.Tests.FileSystemWatcherTest.ExpectEvent(FileSystemWatcher watcher, WatcherChangeTypes expectedEvents, Action action, Action cleanup, String[] expectedPaths, Int32 attempts, Int32 timeout) in /path/to/runtime/src/libraries/System.IO.FileSystem.Watcher/tests/Utility/FileSystemWatcherTest.cs:line 198
   at System.IO.Tests.File_NotifyFilter_Tests.FileSystemWatcher_File_NotifyFilter_LastWriteTime(NotifyFilters filter) in /path/to/runtime/src/libraries/System.IO.FileSystem.Watcher/tests/FileSystemWatcher.File.NotifyFilter.cs:line 136
   at InvokeStub_File_NotifyFilter_Tests.FileSystemWatcher_File_NotifyFilter_LastWriteTime(Object, Span`1)
   at System.Reflection.MethodBaseInvoker.InvokeWithOneArg(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
[FAIL] System.IO.Tests.Directory_NotifyFilter_Tests.FileSystemWatcher_Directory_NotifyFilter_Attributes(filter: Size)
System.IO.IOException : The configured user limit (128) on the number of inotify instances has been reached, or the per-process limit on the number of open file descriptors has been reached.
   at System.IO.FileSystemWatcher.StartRaisingEvents()
   at System.IO.Tests.FileSystemWatcherTest.ExecuteAndVerifyEvents(FileSystemWatcher watcher, WatcherChangeTypes expectedEvents, Action action, Boolean assertExpected, String[] expectedPaths, Int32 timeout) in /path/to/runtime/src/libraries/System.IO.FileSystem.Watcher/tests/Utility/FileSystemWatcherTest.cs:line 308
   at System.IO.Tests.FileSystemWatcherTest.ExpectEvent(FileSystemWatcher watcher, WatcherChangeTypes expectedEvents, Action action, Action cleanup, String[] expectedPaths, Int32 attempts, Int32 timeout) in /path/to/runtime/src/libraries/System.IO.FileSystem.Watcher/tests/Utility/FileSystemWatcherTest.cs:line 198
   at System.IO.Tests.Directory_NotifyFilter_Tests.<>c__DisplayClass4_0.<FileSystemWatcher_Directory_NotifyFilter_Attributes>b__0() in /path/to/runtime/src/libraries/System.IO.FileSystem.Watcher/tests/FileSystemWatcher.Directory.NotifyFilter.cs:line 46
   at System.IO.Tests.FileSystemWatcherTest.Execute(Action test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName) in /path/to/runtime/src/libraries/System.IO.FileSystem.Watcher/tests/Utility/FileSystemWatcherTest.cs:line 238
   at InvokeStub_Directory_NotifyFilter_Tests.FileSystemWatcher_Directory_NotifyFilter_Attributes(Object, Span`1)
   at System.Reflection.MethodBaseInvoker.InvokeWithOneArg(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
[FAIL] System.IO.Tests.WaitForChangedTests.NonZeroTimeout_NoEvents_TimesOut(enabledBeforeWait: True, useTimeSpan: True)
System.IO.IOException : The configured user limit (128) on the number of inotify instances has been reached, or the per-process limit on the number of open file descriptors has been reached.
   at System.IO.FileSystemWatcher.StartRaisingEvents()
   at System.IO.Tests.WaitForChangedTests.NonZeroTimeout_NoEvents_TimesOut(Boolean enabledBeforeWait, Boolean useTimeSpan) in /path/to/runtime/src/libraries/System.IO.FileSystem.Watcher/tests/FileSystemWatcher.WaitForChanged.cs:line 113
   at InvokeStub_WaitForChangedTests.NonZeroTimeout_NoEvents_TimesOut(Object, Span`1)
   at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
[FAIL] System.IO.Tests.File_Move_Tests.Linux_File_Move_To_Different_Watched_Directory
System.IO.IOException : The configured user limit (128) on the number of inotify instances has been reached, or the per-process limit on the number of open file descriptors has been reached.
   at System.IO.FileSystemWatcher.StartRaisingEvents()
   at System.IO.Tests.FileSystemWatcherTest.ExecuteAndVerifyEvents(FileSystemWatcher watcher, WatcherChangeTypes expectedEvents, Action action, Boolean assertExpected, String[] expectedPaths, Int32 timeout) in /path/to/runtime/src/libraries/System.IO.FileSystem.Watcher/tests/Utility/FileSystemWatcherTest.cs:line 308
   at System.IO.Tests.FileSystemWatcherTest.ExpectEvent(FileSystemWatcher watcher, WatcherChangeTypes expectedEvents, Action action, Action cleanup, String[] expectedPaths, Int32 attempts, Int32 timeout) in /path/to/runtime/src/libraries/System.IO.FileSystem.Watcher/tests/Utility/FileSystemWatcherTest.cs:line 198
   at System.IO.Tests.File_Move_Tests.FileMove_DifferentWatchedDirectory(WatcherChangeTypes eventType) in /path/to/runtime/src/libraries/System.IO.FileSystem.Watcher/tests/FileSystemWatcher.File.Move.cs:line 184
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
Finished System.IO.FileSystem.Watcher.Tests, Version=9.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51

Regression?

No

Known Workarounds

Keep the test disabled so that the other tests in the run can be executed unaffected.

Author:	ivdiazsa
Assignees:	-
Labels:	area-System.IO, disabled-test, test-failure
Milestone:	Future

[tmds]

#117148 will help reduce contention on inotify instances. It will definitely impact the amount of inotify instances used while running the tests.

[jkotas]

Tests are still disabled against this issue

runtime/src/libraries/System.IO.FileSystem.Watcher/tests/FileSystemWatcher.File.Move.cs

Line 189 in 240be2a

[ActiveIssue("https://github.com/dotnet/runtime/issues/96728", typeof(PlatformDetection), nameof(PlatformDetection.IsReadyToRunCompiled))]