Skip to content

Fix more flaky tests #38532

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
AndriySvyryd merged 6 commits into from
Jul 7, 2026
Merged

Fix more flaky tests #38532

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
8819fcf
Skip more flakey in-memory tests
AndriySvyryd Jun 30, 2026
45a5cfd
Fix reporting in Helix
AndriySvyryd Jun 30, 2026
0259fad
Also skip Save_changed_optional_one_to_one_with_alternate_key_in_store
AndriySvyryd Jun 30, 2026
d17d1d8
Merge branch 'main' into Unflaking
AndriySvyryd Jul 1, 2026
40afd09
Handle stale cache in CosmosClient
AndriySvyryd Jul 1, 2026
43e6256
Potential fix for pull request finding
AndriySvyryd Jul 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion azure-pipelines-internal-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,7 @@ extends:
Test__Cosmos__DefaultConnection: $(_CosmosConnectionUrl)
Test__SqlServer__DefaultConnection: "Data Source="
displayName: Build
- script: .dotnet\dotnet test test\EFCore.ApiBaseline.Tests -configuration $(_BuildConfig)
- script: .dotnet\dotnet test test\EFCore.ApiBaseline.Tests -p:Configuration=$(_BuildConfig)
displayName: Test API Baselines
condition: succeeded()
- powershell: |
Expand Down
2 changes: 1 addition & 1 deletion azure-pipelines-public.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ stages:
Test__Cosmos__DefaultConnection: $(_CosmosConnectionUrl)
Test__SqlServer__DefaultConnection: "Data Source="
name: Build
- script: .dotnet\dotnet test test\EFCore.ApiBaseline.Tests -configuration $(_BuildConfig)
- script: .dotnet\dotnet test test\EFCore.ApiBaseline.Tests -p:Configuration=$(_BuildConfig)
displayName: Test API Baselines
condition: succeeded()
- powershell: |
Expand Down
7 changes: 5 additions & 2 deletions eng/helix.proj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,15 +125,18 @@
such as [ConditionalClass]/[ConditionalAssembly] add `category=failing` to tests whose
condition evaluates to false; this filter is what actually skips them at the runner.

The 'report-trx' argument is required so each Helix work item produces a TRX file that Arcade's
Helix reporter uploads to Azure DevOps.

This must run AFTER any per-queue ItemGroup that re-Includes MTPProject items.
-->
<ItemGroup>
<MTPProject Update="@(MTPProject)">
<Arguments>$(XUnitV3Arguments) --filter-not-trait category=failing --ignore-exit-code 8</Arguments>
<Arguments>--report-trx --filter-not-trait category=failing --ignore-exit-code 8</Arguments>
</MTPProject>
Comment thread
AndriySvyryd marked this conversation as resolved.
Comment thread
AndriySvyryd marked this conversation as resolved.
</ItemGroup>

<PropertyGroup>
<XUnitWorkItemTimeout Condition = "'$(XUnitWorkItemTimeout)' == ''">03:00:00</XUnitWorkItemTimeout>
<MTPWorkItemTimeout Condition = "'$(MTPWorkItemTimeout)' == ''">03:00:00</MTPWorkItemTimeout>
</PropertyGroup>
</Project>
72 changes: 72 additions & 0 deletions test/EFCore.Cosmos.FunctionalTests/TestUtilities/CosmosTestStore.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -317,6 +317,10 @@ public override Task CleanAsync(DbContext context, bool createTables = true)
private async Task CleanAsyncImpl(DbContext context, bool createTables)
{
var created = await EnsureCreatedAsync(context).ConfigureAwait(false);

// Containers are deleted and recreated below only when the database already existed. A freshly-created
// database has brand-new containers whose metadata cannot be stale.
var containersRecreated = !created;
try
{
if (!created)
Expand All @@ -334,12 +338,23 @@ private async Task CleanAsyncImpl(DbContext context, bool createTables)
created = await context.Database.EnsureCreatedAsync().ConfigureAwait(false);
if (!created)
{
if (containersRecreated)
{
await RefreshContainerMetadataAsync(context).ConfigureAwait(false);
}

await SeedAsync(context).ConfigureAwait(false);
}
}
else
{
await CreateContainersAsync(context).ConfigureAwait(false);

if (containersRecreated)
{
await RefreshContainerMetadataAsync(context).ConfigureAwait(false);
}

await SeedAsync(context).ConfigureAwait(false);
Comment thread
AndriySvyryd marked this conversation as resolved.
}
}
Expand All @@ -357,6 +372,63 @@ private async Task CleanAsyncImpl(DbContext context, bool createTables)
}
}

// Deleting and recreating a container gives it a new resource id (_rid), but the shared CosmosClient caches each
// container's _rid (and the partition key ranges keyed by it) by container name. The first operation on the
// recreated container - whether it is the reseed below or the next test's first query - can then fail with
// "NotFound (404) ... GetTargetPartitionKeyRanges ... failed due to stale cache", and the query pipeline does not
// transparently refresh and retry. Prime the client's caches here, right after recreating the containers and before
// any test touches them. Each attempt re-reads the container by name (refreshing the collection cache with the new
// _rid) and then runs a query - the exact path that otherwise fails. Because the recreate can take a moment to
// become consistent on the emulator gateway, retry with a short delay until a query succeeds. This is the single
// place that handles the stale-metadata problem; it is fully best-effort and must never fail the clean, so all
// errors are ultimately swallowed.
private async Task RefreshContainerMetadataAsync(DbContext context)
{
const int maxAttempts = 10;

try
{
var cosmosClient = context.Database.GetCosmosClient();
var database = cosmosClient.GetDatabase(Name);
var model = context.GetService<IDesignTimeModel>().Model;
var countQuery = new QueryDefinition("SELECT VALUE COUNT(1) FROM c");

foreach (var containerProperties in GetContainersToCreate(model))
{
var container = database.GetContainer(containerProperties.Id);

for (var attempt = 1; attempt <= maxAttempts; attempt++)
{
try
{
await container.ReadContainerAsync().ConfigureAwait(false);

using var iterator = container.GetItemQueryIterator<int>(countQuery);
while (iterator.HasMoreResults)
{
await iterator.ReadNextAsync().ConfigureAwait(false);
}

break;
}
catch when (attempt < maxAttempts)
{
// The recreate has not become consistent yet; wait for the gateway to catch up and retry.
await Task.Delay(200).ConfigureAwait(false);
}
catch
{
// Best-effort priming; never let it fail the clean.
}
}
}
}
catch
{
// Never let cache priming fail the clean.
}
}

private async Task CreateContainersAsync(DbContext context)
{
var databaseAccount = await GetDBAccount().ConfigureAwait(false);
Expand Down
37 changes: 29 additions & 8 deletions test/EFCore.InMemory.FunctionalTests/GraphUpdates/ProxyGraphUpdatesInMemoryTest.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,10 @@ public abstract class ProxyGraphUpdatesInMemoryTestBase<TFixture>(TFixture fixtu
where TFixture : ProxyGraphUpdatesInMemoryTestBase<TFixture>.ProxyGraphUpdatesInMemoryFixtureBase, new()
{
// FK constraint checking.
[Fact]
public override Task Optional_one_to_one_relationships_are_one_to_one()
=> Assert.ThrowsAnyAsync<Exception>(() => base.Optional_one_to_one_relationships_are_one_to_one());

// FK constraint checking.
[Fact]
public override Task Optional_one_to_one_with_AK_relationships_are_one_to_one()
=> Assert.ThrowsAnyAsync<Exception>(() => base.Optional_one_to_one_with_AK_relationships_are_one_to_one());

Expand All @@ -32,6 +30,12 @@ public override Task Optional_one_to_one_are_orphaned(
CascadeTiming deleteOrphansTiming)
=> Task.CompletedTask;

// Cascade delete.
public override Task Optional_one_to_one_with_alternate_key_are_orphaned(
CascadeTiming cascadeDeleteTiming,
CascadeTiming deleteOrphansTiming)
=> Task.CompletedTask;

// Cascade delete.
public override Task Optional_many_to_one_dependents_are_orphaned_in_store(
CascadeTiming cascadeDeleteTiming,
Expand Down Expand Up @@ -94,12 +98,10 @@ public override Task Required_many_to_one_dependents_with_alternate_key_are_casc
=> Task.CompletedTask;

// FK constraint checking.
[Fact]
public override Task Required_one_to_one_relationships_are_one_to_one()
=> Assert.ThrowsAnyAsync<Exception>(() => base.Required_one_to_one_relationships_are_one_to_one());

// FK constraint checking.
[Fact]
public override Task Required_one_to_one_with_AK_relationships_are_one_to_one()
=> Assert.ThrowsAnyAsync<Exception>(() => base.Required_one_to_one_with_AK_relationships_are_one_to_one());

Expand All @@ -115,6 +117,14 @@ public override Task Required_many_to_one_dependents_with_alternate_key_are_casc
public override Task Can_attach_full_optional_graph_of_duplicates()
=> Task.CompletedTask;

// Graph fixup ordering is non-deterministic on InMemory.
public override Task Can_attach_full_required_AK_graph_of_duplicates()
=> Task.CompletedTask;

// Graph fixup ordering is non-deterministic on InMemory.
public override Task No_fixup_to_Deleted_entities()
=> Task.CompletedTask;

// Cascade delete.
public override Task Required_non_PK_one_to_one_with_alternate_key_are_cascade_deleted(
CascadeTiming cascadeDeleteTiming,
Expand All @@ -127,6 +137,10 @@ public override Task Reparent_required_non_PK_one_to_one_with_alternate_key(
bool useExistingRoot)
=> Task.CompletedTask;

// Cascade delete.
public override Task Save_changed_optional_one_to_one_with_alternate_key_in_store()
=> Task.CompletedTask;

// Cascade delete.
public override Task Sever_required_one_to_one(ChangeMechanism changeMechanism)
=> Task.CompletedTask;
Expand Down Expand Up @@ -157,10 +171,17 @@ protected override async Task ExecuteWithStrategyInTransactionAsync(
Func<DbContext, Task> nestedTestOperation2 = null,
Func<DbContext, Task> nestedTestOperation3 = null)
{
await base.ExecuteWithStrategyInTransactionAsync(
testOperation, nestedTestOperation1, nestedTestOperation2, nestedTestOperation3);

await Fixture.ReseedAsync();
// InMemory has no real transactions, so the shared store is mutated directly by each test and must be
// reseeded afterwards.
try
{
await base.ExecuteWithStrategyInTransactionAsync(
testOperation, nestedTestOperation1, nestedTestOperation2, nestedTestOperation3);
}
finally
{
await Fixture.ReseedAsync();
}
}

public abstract class ProxyGraphUpdatesInMemoryFixtureBase : ProxyGraphUpdatesFixtureBase
Expand Down
83 changes: 63 additions & 20 deletions test/EFCore.Relational.Specification.Tests/TransactionTestBase.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -336,41 +336,84 @@ public virtual async Task SaveChanges_uses_ambient_transaction_with_connectionSt
return;
}

DbConnection connection;
using (var context = CreateContextWithConnectionString())
DbConnection connection = null;

await RetryOnDistributedTransactionNotSupportedAsync(async () =>
{
using (TestUtilities.TestStore.CreateTransactionScope())
using (var context = CreateContextWithConnectionString())
{
context.Database.AutoTransactionBehavior = autoTransactionBehavior;
using (TestUtilities.TestStore.CreateTransactionScope())
{
context.Database.AutoTransactionBehavior = autoTransactionBehavior;

connection = context.Database.GetDbConnection();
Assert.Equal(ConnectionState.Closed, connection.State);
connection = context.Database.GetDbConnection();
Assert.Equal(ConnectionState.Closed, connection.State);

await context.AddAsync(
new TransactionCustomer { Id = 77, Name = "Bobble" });
await context.AddAsync(
new TransactionCustomer { Id = 77, Name = "Bobble" });

context.Entry(context.Set<TransactionCustomer>().OrderBy(c => c.Id).Last()).State = EntityState.Added;
context.Entry(context.Set<TransactionCustomer>().OrderBy(c => c.Id).Last()).State = EntityState.Added;

if (async)
{
await Assert.ThrowsAsync<DbUpdateException>(() => context.SaveChangesAsync());
}
else
{
Assert.Throws<DbUpdateException>(() => context.SaveChanges());
}
// SaveChanges is expected to fail with DbUpdateException (duplicate key).
try
{
if (async)
{
await context.SaveChangesAsync();
}
else
{
context.SaveChanges();
}

Assert.Fail("Expected a DbUpdateException to be thrown.");
}
catch (PlatformNotSupportedException)
{
// When the ambient transaction is promoted to a distributed one and throws PlatformNotSupportedException, let it
// propagate so the whole operation is retried.
throw;
}
catch (DbUpdateException)
{
}
Comment thread
Copilot marked this conversation as resolved.

Assert.Equal(ConnectionState.Closed, connection.State);
Assert.Equal(ConnectionState.Closed, connection.State);

context.Database.AutoTransactionBehavior = AutoTransactionBehavior.WhenNeeded;
context.Database.AutoTransactionBehavior = AutoTransactionBehavior.WhenNeeded;
}
}
}
});

Assert.Equal(ConnectionState.Closed, connection.State);

AssertStoreInitialState();
}

// Sometimes an ambient/enlisted transaction ends up spanning more than one physical connection due to concurrency issues
// and is promoted to a distributed transaction, which throws PlatformNotSupportedException.
// Retry the operation a few times (with exponential back-off) so a run that keeps to a single connection (and
// therefore reaches the behavior under test) is used.
private static async Task RetryOnDistributedTransactionNotSupportedAsync(Func<Task> test)
{
const int maxAttempts = 3;
var delay = TimeSpan.FromSeconds(1);

for (var attempt = 1; ; attempt++)
{
try
{
await test();
return;
}
catch (PlatformNotSupportedException) when (attempt < maxAttempts)
{
await Task.Delay(delay);
delay *= 2;
}
}
Comment thread
AndriySvyryd marked this conversation as resolved.
}

[Theory, InlineData(true), InlineData(false)]
public virtual void SaveChanges_throws_for_suppressed_ambient_transactions(bool connectionString)
{
Expand Down
6 changes: 4 additions & 2 deletions test/EFCore.Specification.Tests/Query/NorthwindAggregateOperatorsQueryTestBase.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -249,14 +249,16 @@ public virtual Task Average_over_nested_subquery(bool async)
=> AssertAverage(
async,
ss => ss.Set<Customer>().OrderBy(c => c.CustomerID).Take(3),
selector: c => (decimal)c.Orders.Average(double (o) => 5 + o.OrderDetails.Average(int (od) => od.ProductID)));
selector: c => (decimal)c.Orders.Average(double (o) => 5 + o.OrderDetails.Average(int (od) => od.ProductID)),
asserter: (e, a) => Assert.Equal(e, a, 10));

[Theory, MemberData(nameof(IsAsyncData))]
public virtual Task Average_over_max_subquery(bool async)
=> AssertAverage(
async,
ss => ss.Set<Customer>().OrderBy(c => c.CustomerID).Take(3),
selector: c => (decimal)c.Orders.Average(int (o) => 5 + o.OrderDetails.Max(int (od) => od.ProductID)));
selector: c => (decimal)c.Orders.Average(int (o) => 5 + o.OrderDetails.Max(int (od) => od.ProductID)),
asserter: (e, a) => Assert.Equal(e, a, 10));

[Theory, MemberData(nameof(IsAsyncData))]
public virtual Task Average_on_float_column(bool async)
Expand Down
Loading