[Internal] Refactoring live-tests to reduce flakiness by FabianMeiswinkel · Pull Request #45216 · Azure/azure-sdk-for-java

FabianMeiswinkel · 2025-05-06T14:00:17Z

Description

This PR is an attempt to significantly reduce flakiness of our live-tests. In the past it has been often necessary to re-run live tests - sometimes multiple times due to test issues.

What has been done in this PR:

Pushed faul-injection tests into its own pipeline - tehse tests take quite a bit of time and by definition have slightly higher flakiness
Refactored the fault-injection tests to isolate different test groups into their own file (easier to debug)
Fixed issues in ClientMetricsTests due to corrupted test-state by pushing various class-level fields into its own type that has a clearly defined lifecycle
Implemented three RetryAnalyzers (one specifically for splits, one for sometimes flaky tests and one allowing significantly more retries)
Marked tests regularly failing due to flakiness with appropriate RetryAnalyzers

Some additional work left:

Monitoring individual pipelines and split those taking >60 minutes regularly (especially Circuit breaker live tests are taking > 2 hours still)
Marking any missed flaki tests with appropriate retryAnalyzer annotation

All SDK Contribution checklist:

The pull request does not introduce [breaking changes]
CHANGELOG is updated for new features, bug fixes or other significant changes.
I have read the contribution guidelines.

General Guidelines and Best Practices

Title of the pull request is clear and informative.
There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

Pull request includes test coverage for the included changes.

FabianMeiswinkel · 2025-05-06T14:01:06Z

/azp run java - cosmos - tests

azure-pipelines · 2025-05-06T14:02:00Z

Azure Pipelines successfully started running 1 pipeline(s).

FabianMeiswinkel · 2025-05-06T17:58:11Z

/azp run java - cosmos - tests

FabianMeiswinkel · 2025-05-06T17:58:18Z

/azp run java - cosmos - kafka

FabianMeiswinkel · 2025-05-06T17:58:24Z

/azp run java - cosmos - spark

azure-pipelines · 2025-05-06T17:58:34Z

Azure Pipelines successfully started running 1 pipeline(s).

azure-pipelines · 2025-05-06T17:58:35Z

Azure Pipelines successfully started running 1 pipeline(s).

azure-pipelines · 2025-05-06T17:58:51Z

Azure Pipelines successfully started running 1 pipeline(s).

FabianMeiswinkel · 2025-05-06T22:00:12Z

/azp run java - cosmos - tests

FabianMeiswinkel · 2025-05-06T22:00:20Z

/azp run java - cosmos - kafka

FabianMeiswinkel · 2025-05-06T22:00:27Z

/azp run java - cosmos - spark

azure-pipelines · 2025-05-06T22:00:47Z

Azure Pipelines successfully started running 1 pipeline(s).

azure-pipelines · 2025-05-06T22:00:56Z

Azure Pipelines successfully started running 1 pipeline(s).

azure-pipelines · 2025-05-06T22:01:23Z

Azure Pipelines successfully started running 1 pipeline(s).

FabianMeiswinkel · 2025-05-07T00:13:28Z

/azp run java - cosmos - tests

FabianMeiswinkel · 2025-05-07T00:13:35Z

/azp run java - cosmos - kafka

FabianMeiswinkel · 2025-05-07T00:13:43Z

/azp run java - cosmos - spark

azure-pipelines · 2025-05-07T00:13:48Z

Azure Pipelines successfully started running 1 pipeline(s).

azure-pipelines · 2025-05-07T00:13:51Z

Azure Pipelines successfully started running 1 pipeline(s).

azure-pipelines · 2025-05-07T00:14:02Z

Azure Pipelines successfully started running 1 pipeline(s).

azure-pipelines · 2025-05-09T21:48:25Z

Azure Pipelines successfully started running 1 pipeline(s).

FabianMeiswinkel · 2025-05-09T23:51:17Z

/azp run java - cosmos - tests

FabianMeiswinkel · 2025-05-09T23:51:24Z

/azp run java - cosmos - kafka

FabianMeiswinkel · 2025-05-09T23:51:31Z

/azp run java - cosmos - spring

azure-pipelines · 2025-05-09T23:51:37Z

Azure Pipelines successfully started running 1 pipeline(s).

azure-pipelines · 2025-05-09T23:51:37Z

No pipelines are associated with this pull request.

FabianMeiswinkel · 2025-05-09T23:51:38Z

/azp run java - cosmos - spark

azure-pipelines · 2025-05-09T23:51:47Z

Azure Pipelines successfully started running 1 pipeline(s).

azure-pipelines · 2025-05-09T23:51:48Z

Azure Pipelines successfully started running 1 pipeline(s).

…to users/fabianm/LiveTestsImprovements

FabianMeiswinkel · 2025-05-10T11:06:23Z

/azp run java - cosmos - tests

FabianMeiswinkel · 2025-05-10T11:06:29Z

/azp run java - cosmos - kafka

FabianMeiswinkel · 2025-05-10T11:06:36Z

/azp run java - cosmos - spark

azure-pipelines · 2025-05-10T11:06:42Z

Azure Pipelines successfully started running 1 pipeline(s).

azure-pipelines · 2025-05-10T11:06:45Z

Azure Pipelines successfully started running 1 pipeline(s).

azure-pipelines · 2025-05-10T11:06:55Z

Azure Pipelines successfully started running 1 pipeline(s).

FabianMeiswinkel · 2025-05-10T13:30:14Z

/azp run java - cosmos - tests

FabianMeiswinkel · 2025-05-10T13:30:21Z

/azp run java - cosmos - kafka

FabianMeiswinkel · 2025-05-10T13:30:27Z

/azp run java - cosmos - spark

azure-pipelines · 2025-05-10T13:30:36Z

Azure Pipelines successfully started running 1 pipeline(s).

azure-pipelines · 2025-05-10T13:30:39Z

Azure Pipelines successfully started running 1 pipeline(s).

azure-pipelines · 2025-05-10T13:30:53Z

Azure Pipelines successfully started running 1 pipeline(s).

These change-feed tests are known-flaky (see Azure#48064, Azure#45216); running on shared fixed accounts surfaces transient 408/429/503 more often during forced partition splits and change-feed setup. - Broaden SplitTestsRetryAnalyzer to retry not only SplitTimeoutException but also a transient CosmosException (408/429/503) anywhere in the cause chain, so verifyLeasesOnRestart_AfterSplit tolerates the 503s a split legitimately produces. - Add FlakyTestRetryAnalyzer to readFeedDocumentsStartFromCustomDateForMultiWrite_test (pkversion + epkversion) and ChangeFeedTest.changesFromPartitionKey_AfterInsertingNewDocuments, matching the precedent set for their sibling CFP tests in Azure#48064. No production SDK code or default client behavior is changed. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Refactoring FaultInjection tests

4124657

FabianMeiswinkel requested review from a team and kirankumarkolli as code owners May 6, 2025 14:00

github-actions Bot added the Cosmos label May 6, 2025

Update IncrementalChangeFeedProcessorTest.java

07c77cd

FabianMeiswinkel added 2 commits May 6, 2025 21:47

Update IncrementalChangeFeedProcessorTest.java

8656977

Update IncrementalChangeFeedProcessorTest.java

e7e9735

FabianMeiswinkel added 2 commits May 6, 2025 23:50

Fixing flaky test

9415075

Fixing flaky tests

1393251

Update ClientMetricsTest.java

fb4e3fb

jeet1995 approved these changes May 9, 2025

View reviewed changes

Test fixes

c677e4e

jeet1995 approved these changes May 9, 2025

View reviewed changes

FabianMeiswinkel added 2 commits May 10, 2025 11:03

Update FaultInjectionWithAvailabilityStrategyTestsBase.java

501e57f

Merge branch 'main' of https://github.com/Azure/azure-sdk-for-java in…

db48713

…to users/fabianm/LiveTestsImprovements

Fixing flaky tests

8c96799

kushagraThapar approved these changes May 10, 2025

View reviewed changes

FabianMeiswinkel merged commit 2551f4c into Azure:main May 11, 2025

Uh oh!

Conversation

FabianMeiswinkel commented May 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

All SDK Contribution checklist:

General Guidelines and Best Practices

Testing Guidelines

Uh oh!

FabianMeiswinkel commented May 6, 2025

Uh oh!

azure-pipelines Bot commented May 6, 2025

Uh oh!

FabianMeiswinkel commented May 6, 2025

Uh oh!

FabianMeiswinkel commented May 6, 2025

Uh oh!

FabianMeiswinkel commented May 6, 2025

Uh oh!

azure-pipelines Bot commented May 6, 2025

Uh oh!

azure-pipelines Bot commented May 6, 2025

Uh oh!

azure-pipelines Bot commented May 6, 2025

Uh oh!

FabianMeiswinkel commented May 6, 2025

Uh oh!

FabianMeiswinkel commented May 6, 2025

Uh oh!

FabianMeiswinkel commented May 6, 2025

Uh oh!

azure-pipelines Bot commented May 6, 2025

Uh oh!

azure-pipelines Bot commented May 6, 2025

Uh oh!

azure-pipelines Bot commented May 6, 2025

Uh oh!

FabianMeiswinkel commented May 7, 2025

Uh oh!

FabianMeiswinkel commented May 7, 2025

Uh oh!

FabianMeiswinkel commented May 7, 2025

Uh oh!

azure-pipelines Bot commented May 7, 2025

Uh oh!

azure-pipelines Bot commented May 7, 2025

Uh oh!

azure-pipelines Bot commented May 7, 2025

Uh oh!

azure-pipelines Bot commented May 9, 2025

Uh oh!

FabianMeiswinkel commented May 9, 2025

Uh oh!

FabianMeiswinkel commented May 9, 2025

Uh oh!

FabianMeiswinkel commented May 9, 2025

Uh oh!

azure-pipelines Bot commented May 9, 2025

Uh oh!

azure-pipelines Bot commented May 9, 2025

Uh oh!

FabianMeiswinkel commented May 9, 2025

Uh oh!

azure-pipelines Bot commented May 9, 2025

Uh oh!

azure-pipelines Bot commented May 9, 2025

Uh oh!

FabianMeiswinkel commented May 10, 2025

Uh oh!

FabianMeiswinkel commented May 10, 2025

Uh oh!

FabianMeiswinkel commented May 10, 2025

Uh oh!

azure-pipelines Bot commented May 10, 2025

Uh oh!

azure-pipelines Bot commented May 10, 2025

Uh oh!

azure-pipelines Bot commented May 10, 2025

Uh oh!

FabianMeiswinkel commented May 10, 2025

FabianMeiswinkel commented May 6, 2025 •

edited

Loading