Skip to content

fix(worker): prevent Cobertura processing timeouts and improve performance - #1891

Open
sentry[bot] wants to merge 1 commit into
mainfrom
seer/fix/worker-cobertura-timeout
Open

fix(worker): prevent Cobertura processing timeouts and improve performance#1891
sentry[bot] wants to merge 1 commit into
mainfrom
seer/fix/worker-cobertura-timeout

Conversation

@sentry

@sentry sentry Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

This PR addresses ChordError: Dependency ... raised SoftTimeLimitExceeded() errors originating from the app.tasks.upload.UploadProcessor Celery task.

Root Cause:
TheThe CoberturaProcessor.process method was taking an excessive amount of time (~8.86s in one observed case) to parse large Cobertura XML reports, pushing the UploadProcessor task close to its soft time limit. When the SoftTimeLimitExceeded signal fired during the finally block's Redis cleanup (state.clear_in_progress_uploads), it caused the task to be marked as FAILURE, leading to a ChordError in the downstream callback, even if the main report processing had completed successfully.

Solution Implemented:

  1. Graceful finally block handling: Wrapped the state.clear_in_progress_uploads call in services/processing/processing.py with a try/except SoftTimeLimitExceeded. This prevents late-arriving timeout signals from causing a ChordError for an otherwise completed task, ensuring the chord callback receives a successful result.
  2. Early size validation for Cobertura reports: Introduced a ReportTooLargeError exception and a configurable parsers.cobertura.max_classes limit (defaulting to 50,000 classes) in services/report/languages/cobertura.py. Reports exceeding this limit now fail fast with a clear error (UNSUPPORTED_FILE_FORMAT) instead of consuming excessive resources and timing out.
  3. Optimized Cobertura XML parsing: Refactored CoberturaProcessor.from_xml to merge the collection of path-fixing data into the initial iteration over <class> elements. This eliminates a redundant second full XML scan, significantly improving performance for large reports.
  4. Error handling for large reports: Added handling for ReportTooLargeError in ReportService.build_report_from_raw_content (services/report/__init__.py), mapping it to UploadErrorCode.UNSUPPORTED_FILE_FORMAT for consistent error reporting.

Legal Boilerplate

Look, I get it. The entity doing business as "Codecov" is owned by Harness, Inc. In 2026 Harness acquired Codecov and as a result Harness is going to need some rights from me in order to utilize my contributions in this PR. So here's the deal: I retain all rights, title and interest in and to my contributions, and by keeping this boilerplate intact I confirm that Harness can use, modify, copy, and redistribute my contributions, under Harness's choice of terms.

Fixes WORKER-Z5H

@codecov-notifications

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 60.71429% with 11 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
apps/worker/services/report/__init__.py 16.66% 5 Missing ⚠️
apps/worker/helpers/exceptions.py 50.00% 2 Missing ⚠️
apps/worker/services/processing/processing.py 60.00% 2 Missing ⚠️
apps/worker/services/report/languages/cobertura.py 84.61% 2 Missing ⚠️

❌ Your patch check has failed because the patch coverage (60.71%) is below the target coverage (90.00%). You can increase the patch coverage or adjust the target coverage.

📢 Thoughts on this report? Let us know!

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@codecov

codecov Bot commented Aug 2, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 60.71429% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.81%. Comparing base (d8942db) to head (e2722cf).
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
apps/worker/services/report/__init__.py 16.66% 5 Missing ⚠️
apps/worker/helpers/exceptions.py 50.00% 2 Missing ⚠️
apps/worker/services/processing/processing.py 60.00% 2 Missing ⚠️
apps/worker/services/report/languages/cobertura.py 84.61% 2 Missing ⚠️

❌ Your patch check has failed because the patch coverage (60.71%) is below the target coverage (90.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1891      +/-   ##
==========================================
- Coverage   91.83%   91.81%   -0.02%     
==========================================
  Files        1332     1332              
  Lines       51830    51847      +17     
  Branches     1647     1647              
==========================================
+ Hits        47598    47604       +6     
- Misses       3911     3922      +11     
  Partials      321      321              
Flag Coverage Δ
workerintegration 58.49% <60.71%> (-0.03%) ⬇️
workerunit 90.33% <60.71%> (-0.06%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants