feat(scorecard): add new GitHub metrics for issues, opened/closed prs and GitHub actions - #3472
feat(scorecard): add new GitHub metrics for issues, opened/closed prs and GitHub actions#3472fullsend-ai-coder[bot] wants to merge 7 commits into
Conversation
Missing ChangesetsThe following package(s) are changed by this PR but do not have a changeset:
See CONTRIBUTING.md for more information about how to add changesets. Changed Packages
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #3472 +/- ##
==========================================
+ Coverage 57.93% 57.97% +0.04%
==========================================
Files 2396 2405 +9
Lines 96104 96507 +403
Branches 26803 26824 +21
==========================================
+ Hits 55675 55951 +276
- Misses 40233 40360 +127
Partials 196 196
*This pull request uses carry forward flags. Click here to find out more. Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
|
🤖 Finished Review · ✅ Success · Started 11:49 AM UTC · Completed 12:02 PM UTC |
ReviewFindingsMedium
Low
Previous runReviewFindingsMedium
Low
Info
Previous run (2)ReviewFindingsMedium
Low
Info
Previous run (3)ReviewFindingsHigh
Medium
Low
|
![fullsend-ai-review[bot]](https://avatars.githubusercontent.com/in/3478501?s=60&v=4){kind=small}
christoph-jerolimov
left a comment
christoph-jerolimov
left a comment
There was a problem hiding this comment.
Address the following issues:
GitHubActionsProviderneeds to be splitted because it includes metrics that counts actions and then metrics that creates a ratio. But the MetricProvider API only supports oneThresholdConfigfor all metrics, at the moment. So splitting this into two providers would allow to define the rightThresholdConfigfor these metrics.GithubCIPassRateProvidershould be renamed toGitHubPRPassRateProviderand the metrics should be named something like "GitHub PR CI first time pass rate (7d)". The description should also include that this is about FTPR (first time pass rate).- Add unit tests for GitHubClosedIssuesProvider.ts, GitHubClosedPRsProvider.ts, GitHubOpenedIssuesProvider.ts and GitHubOpenedPRsProvider.ts
|
/fs-fix |
|
🤖 Finished Fix · ✅ Success · Started 12:51 PM UTC · Completed 1:10 PM UTC |
- Split GithubActionsProvider into GithubActionsCountProvider (with COUNT_THRESHOLDS) and GithubActionsRatioProvider (with RATIO_THRESHOLDS) so each provider gets the correct ThresholdConfig - Rename GithubCIPassRateProvider to GithubPRPassRateProvider with updated metric IDs (github.pr_ci_first_time_pass_rate_*) and titles mentioning FTPR (first time pass rate) - Add individual unit test files for GithubClosedIssuesProvider, GithubClosedPRsProvider, GithubOpenedIssuesProvider, and GithubOpenedPRsProvider - Apply encodeURIComponent to owner/repo in REST URL construction Addresses review feedback on #3472
🔧 Fix agent — iteration 1 (human-triggered)Addressed all 3 human review requests: (1) split GithubActionsProvider into count and ratio providers with correct thresholds, (2) renamed GithubCIPassRateProvider to GithubPRPassRateProvider with FTPR naming, (3) added individual unit tests for 4 search-count providers. Also fixed SSRF vulnerability with encodeURIComponent. Disagreed with 4 bot-only findings as out of scope or unnecessary. Fixed (4):
Disagreed (4):
Tests: passed Updated by fullsend fix agent |
|
🤖 Finished Review · ✅ Success · Started 1:13 PM UTC · Completed 1:26 PM UTC |
fullsend-ai-review
Bot
added
the
requires-manual-review
|
/fs-fix inspect and fix e2e playwright tests |
|
🤖 Fix · ❌ Terminated · Started 6:50 AM UTC · Ended 8:25 AM UTC |
- Split GithubActionsProvider into GithubActionsCountProvider (with COUNT_THRESHOLDS) and GithubActionsRatioProvider (with RATIO_THRESHOLDS) so each provider gets the correct ThresholdConfig - Rename GithubCIPassRateProvider to GithubPRPassRateProvider with updated metric IDs (github.pr_ci_first_time_pass_rate_*) and titles mentioning FTPR (first time pass rate) - Add individual unit test files for GithubClosedIssuesProvider, GithubClosedPRsProvider, GithubOpenedIssuesProvider, and GithubOpenedPRsProvider - Apply encodeURIComponent to owner/repo in REST URL construction Addresses review feedback on #3472
006906c to
5f9d7ac
Compare
|
🤖 Finished Fix · ❌ Failure · Started 6:50 AM UTC · Completed 8:25 AM UTC |
|
🤖 Finished Review · ✅ Success · Started 8:27 AM UTC · Completed 8:40 AM UTC |
| } | ||
| } | ||
| } | ||
| `; |
There was a problem hiding this comment.
[medium] edge-case
The getPullRequestsWithReviews and getPullRequestsWithCommitStatuses GraphQL queries use search(first: 100) which limits results to 100 PRs. For active repositories, this silently truncates the dataset, producing inaccurate metrics. No pagination is implemented unlike getWorkflowRuns.
Suggested fix: Implement GraphQL cursor-based pagination or document the 100-item limit and add a log warning when the result count equals 100.
| }>(query, { | ||
| q: fullQuery, | ||
| }); | ||
|
|
There was a problem hiding this comment.
[medium] edge-case
getWorkflowRuns pages through all workflow runs using a while loop with no upper bound on page count or total results. High-activity repos could cause unbounded HTTP requests, memory exhaustion, and rate limit depletion.
Suggested fix: Add a maximum page count (e.g., 10 pages = 1000 runs) to cap the loop.
Rebase on main and migrate all new metric providers to the updated MetricProvider interface from #3560. Remove deprecated single-metric methods (getMetricType, getMetric, getMetricThresholds, calculateMetric, getMetricIds) and use only getMetrics() with thresholds on each Metric object and calculateMetrics() returning a Map. All 10 test suites (37 tests) pass. Addresses review feedback on #3472
🔧 Fix agent — iteration 7 (human-triggered)Rebased PR branch onto latest origin/main per human instruction. Resolved one merge conflict in GithubClient.ts (combined graphql import from main with type imports from branch). All tests and type checks pass. Fixed (1):
Tests: passed Updated by fullsend fix agent |
Align all new metric provider IDs with the lowerCamelCase naming convention established in the codebase. The providers were using snake_case IDs which is inconsistent with the existing pattern (e.g. github.openPRs). Renames: - github.open_issues -> github.openIssues - github.opened_issues_7d -> github.openedIssues7d - github.closed_issues_7d -> github.closedIssues7d - github.opened_prs_7d -> github.openedPRs7d - github.closed_prs_7d -> github.closedPRs7d - github.time_to_review -> github.timeToReview - github.time_to_approve -> github.timeToApprove - github.time_to_merge -> github.timeToMerge - github.actions_started_7d -> github.actionsStarted7d - github.actions_successful_7d -> github.actionsSuccessful7d - github.actions_failed_7d -> github.actionsFailed7d - github.actions_success_ratio_7d -> github.actionsSuccessRatio7d - github.actions_success_ratio_24h -> github.actionsSuccessRatio24h - github.pr_ci_first_time_pass_rate_7d -> github.prCiFirstTimePassRate7d - github.pr_ci_first_time_pass_rate_24h -> github.prCiFirstTimePassRate24h Addresses review feedback on #3472
- Fix RATIO_THRESHOLDS gap in GithubActionsRatioProvider and GithubPRPassRateProvider: change warning from '50-79' to '>=50' to prevent backend crash from uncovered threshold range - Add cursor-based pagination to getPullRequestsWithReviews and getPullRequestsWithCommitStatuses GraphQL queries (max 10 pages) - Cap getWorkflowRuns REST pagination at 10 pages to prevent unbounded API calls on active repos - Update getProviderId() in all new providers to return a unique identifier matching the filename without the Github prefix - Decouple metric IDs from provider IDs in single-metric providers by introducing METRIC_ID constants Addresses review feedback on #3472
Add explicit type annotations to `response` variables in getPullRequestsWithReviews and getPullRequestsWithCommitStatuses. The `cursor` variable created a circular type dependency (response depends on cursor as argument, cursor is reassigned from response), causing TS7022 "implicitly has type 'any'". Moving the type from a generic parameter to a variable annotation breaks the cycle. Addresses review feedback on #3472
c29495f to
1f85327
Compare
1. Remove all const METRIC_IDS / METRIC_ID and inline metric IDs directly into getMetrics() and calculateMetrics() calls across all new providers. 2. Split GithubActionsCountProvider thresholds: started and successful metrics use ACTIVITY_THRESHOLDS (Success >=0, Warning ==0), while failed metric keeps FAILED_COUNT_THRESHOLDS (<10/10-50/>50). 3. Switch GithubClosedIssuesProvider, GithubClosedPRsProvider, GithubOpenedIssuesProvider, and GithubOpenedPRsProvider to new INFORMATIONAL_NUMBER_THRESHOLD (info status for any value >=0) added to scorecard-common threshold.ts. 4. Change GithubPRLifecycleProvider duration from hours to days, update DURATION_THRESHOLDS to Success <7, Warning 7-14, Error >14, and prefix all metric titles with "GitHub PR". Addresses review feedback on #3472
|
/fs-fix Change the new provider ids to be prefixed with "github." And then start with a lower case character. snakeCase |
|
/fs-fix Change the new provider ids to be prefixed with "github." And then start with a lower case character. snakeCase |
|
🤖 Finished Fix · ✅ Success · Started 6:38 PM UTC · Completed 6:54 PM UTC |
Convert all new metric IDs from camelCase to snake_case as requested: - github.openIssues -> github.open_issues - github.openedIssues7d -> github.opened_issues_7d - github.openedPRs7d -> github.opened_prs_7d - github.closedIssues7d -> github.closed_issues_7d - github.closedPRs7d -> github.closed_prs_7d - github.actionsStarted7d -> github.actions_started_7d - github.actionsSuccessful7d -> github.actions_successful_7d - github.actionsFailed7d -> github.actions_failed_7d - github.actionsSuccessRatio7d -> github.actions_success_ratio_7d - github.actionsSuccessRatio24h -> github.actions_success_ratio_24h - github.prCiFirstTimePassRate7d -> github.pr_ci_first_time_pass_rate_7d - github.prCiFirstTimePassRate24h -> github.pr_ci_first_time_pass_rate_24h - github.timeToReview -> github.time_to_review - github.timeToApprove -> github.time_to_approve - github.timeToMerge -> github.time_to_merge Addresses human review feedback on #3472
🔧 Fix agent — iteration 7 (human-triggered)Renamed all 15 new GitHub metric IDs from camelCase to snake_case as requested by human reviewer. All IDs now use the format github.<snake_case_name>. Updated 19 files (10 provider source files and 9 test files). All 47 tests pass across 13 test suites. Fixed (1):
Tests: passed Updated by fullsend fix agent |
|
/fs-fix rebase |
|
🤖 Finished Fix · ✅ Success · Started 9:25 PM UTC · Completed 9:38 PM UTC |
Extend the scorecard GitHub module with 13 new metrics across four domains: Issue/PR counts (5 metrics): - Currently open issues - Opened issues in last 7 days - Opened PRs in last 7 days - Closed issues in last 7 days - Closed PRs in last 7 days PR lifecycle timing (3 metrics, batch provider): - Time to first review (average hours) - Time to first approval (average hours) - Time to merge (average hours) All computed from PRs updated in the last 7 days. GitHub Actions (5 metrics, batch provider): - Started workflow runs in last 7 days - Successfully completed runs in last 7 days - Failed runs in last 7 days - Success ratio for 7 days (percentage) - Success ratio for 24 hours (percentage) Non-terminal runs (pending/running/cancelled) are excluded from success/failure counts and ratio calculations. CI pass rate (2 metrics, batch provider): - First-time CI pass rate for 7 days (percentage) - First-time CI pass rate for 24 hours (percentage) Checks CI status on the last commit of the first push to each PR. PRs without CI checks are excluded. New GithubClient methods use GraphQL for issue/PR queries and REST API (fetch) for workflow runs. All providers follow the existing MetricProvider pattern and are registered in the module init. Batch providers use getMetrics/calculateMetrics for efficient multi-metric computation. Closes #3468
- Split GithubActionsProvider into GithubActionsCountProvider (with COUNT_THRESHOLDS) and GithubActionsRatioProvider (with RATIO_THRESHOLDS) so each provider gets the correct ThresholdConfig - Rename GithubCIPassRateProvider to GithubPRPassRateProvider with updated metric IDs (github.pr_ci_first_time_pass_rate_*) and titles mentioning FTPR (first time pass rate) - Add individual unit test files for GithubClosedIssuesProvider, GithubClosedPRsProvider, GithubOpenedIssuesProvider, and GithubOpenedPRsProvider - Apply encodeURIComponent to owner/repo in REST URL construction Addresses review feedback on #3472
Rebase on main and migrate all new metric providers to the updated MetricProvider interface from #3560. Remove deprecated single-metric methods (getMetricType, getMetric, getMetricThresholds, calculateMetric, getMetricIds) and use only getMetrics() with thresholds on each Metric object and calculateMetrics() returning a Map. All 10 test suites (37 tests) pass. Addresses review feedback on #3472
Align all new metric provider IDs with the lowerCamelCase naming convention established in the codebase. The providers were using snake_case IDs which is inconsistent with the existing pattern (e.g. github.openPRs). Renames: - github.open_issues -> github.openIssues - github.opened_issues_7d -> github.openedIssues7d - github.closed_issues_7d -> github.closedIssues7d - github.opened_prs_7d -> github.openedPRs7d - github.closed_prs_7d -> github.closedPRs7d - github.time_to_review -> github.timeToReview - github.time_to_approve -> github.timeToApprove - github.time_to_merge -> github.timeToMerge - github.actions_started_7d -> github.actionsStarted7d - github.actions_successful_7d -> github.actionsSuccessful7d - github.actions_failed_7d -> github.actionsFailed7d - github.actions_success_ratio_7d -> github.actionsSuccessRatio7d - github.actions_success_ratio_24h -> github.actionsSuccessRatio24h - github.pr_ci_first_time_pass_rate_7d -> github.prCiFirstTimePassRate7d - github.pr_ci_first_time_pass_rate_24h -> github.prCiFirstTimePassRate24h Addresses review feedback on #3472
- Fix RATIO_THRESHOLDS gap in GithubActionsRatioProvider and GithubPRPassRateProvider: change warning from '50-79' to '>=50' to prevent backend crash from uncovered threshold range - Add cursor-based pagination to getPullRequestsWithReviews and getPullRequestsWithCommitStatuses GraphQL queries (max 10 pages) - Cap getWorkflowRuns REST pagination at 10 pages to prevent unbounded API calls on active repos - Update getProviderId() in all new providers to return a unique identifier matching the filename without the Github prefix - Decouple metric IDs from provider IDs in single-metric providers by introducing METRIC_ID constants Addresses review feedback on #3472
Add explicit type annotations to `response` variables in getPullRequestsWithReviews and getPullRequestsWithCommitStatuses. The `cursor` variable created a circular type dependency (response depends on cursor as argument, cursor is reassigned from response), causing TS7022 "implicitly has type 'any'". Moving the type from a generic parameter to a variable annotation breaks the cycle. Addresses review feedback on #3472
1. Remove all const METRIC_IDS / METRIC_ID and inline metric IDs directly into getMetrics() and calculateMetrics() calls across all new providers. 2. Split GithubActionsCountProvider thresholds: started and successful metrics use ACTIVITY_THRESHOLDS (Success >=0, Warning ==0), while failed metric keeps FAILED_COUNT_THRESHOLDS (<10/10-50/>50). 3. Switch GithubClosedIssuesProvider, GithubClosedPRsProvider, GithubOpenedIssuesProvider, and GithubOpenedPRsProvider to new INFORMATIONAL_NUMBER_THRESHOLD (info status for any value >=0) added to scorecard-common threshold.ts. 4. Change GithubPRLifecycleProvider duration from hours to days, update DURATION_THRESHOLDS to Success <7, Warning 7-14, Error >14, and prefix all metric titles with "GitHub PR". Addresses review feedback on #3472
🔧 Fix agent — iteration 8 (human-triggered)Rebased branch onto latest origin/main per human instruction. No conflicts. All tests pass. Fixed (1):
Tests: passed Updated by fullsend fix agent |
47c4936 to
85e383a
Compare
Convert all new metric IDs from camelCase to snake_case as requested: - github.openIssues -> github.open_issues - github.openedIssues7d -> github.opened_issues_7d - github.openedPRs7d -> github.opened_prs_7d - github.closedIssues7d -> github.closed_issues_7d - github.closedPRs7d -> github.closed_prs_7d - github.actionsStarted7d -> github.actions_started_7d - github.actionsSuccessful7d -> github.actions_successful_7d - github.actionsFailed7d -> github.actions_failed_7d - github.actionsSuccessRatio7d -> github.actions_success_ratio_7d - github.actionsSuccessRatio24h -> github.actions_success_ratio_24h - github.prCiFirstTimePassRate7d -> github.pr_ci_first_time_pass_rate_7d - github.prCiFirstTimePassRate24h -> github.pr_ci_first_time_pass_rate_24h - github.timeToReview -> github.time_to_review - github.timeToApprove -> github.time_to_approve - github.timeToMerge -> github.time_to_merge Addresses human review feedback on #3472
|
/fs-fix Drop the last commit and force push the changes to use csmelCase instead of snake_case |
|
🤖 Finished Fix · ✅ Success · Started 4:17 AM UTC · Completed 4:31 AM UTC |
🔧 Fix agent — iteration 8 (human-triggered)Dropped the snake_case rename commit per human instruction. All metric IDs are now lowerCamelCase as per workspace conventions. 47 tests pass. The post-script will force push the branch to exclude the dropped commit. Fixed (1):
Tests: passed Updated by fullsend fix agent |
85e383a to
8e2436f
Compare
|
4 participants
Extend the scorecard GitHub module with 13 new metrics across four domains:
Issue/PR counts (5 metrics):
PR lifecycle timing (3 metrics, batch provider):
All computed from PRs updated in the last 7 days.
GitHub Actions (5 metrics, batch provider):
Non-terminal runs (pending/running/cancelled) are excluded from success/failure counts and ratio calculations.
CI pass rate (2 metrics, batch provider):
Checks CI status on the last commit of the first push to each PR. PRs without CI checks are excluded.
New GithubClient methods use GraphQL for issue/PR queries and REST API (fetch) for workflow runs. All providers follow the existing MetricProvider pattern and are registered in the module init. Batch providers use getMetrics/calculateMetrics for efficient multi-metric computation.
Closes #3468
Post-script verification
agent/3468-extend-github-metrics)7ccaff17753df64c7ab288cdcba34cee5a657254..HEAD)