feat(bulk-import): repository list includes only "Left Overs"

[dom-aug]

Hey, I just made a Pull Request!

This Pull Request fixes RHIDP-11693

The bulk import backend plugin returns repositories that have not been imported yet.

• Before, the bulk import list showed all repositories from GitHub / GitLab and the already imported ones were marked accordingly with the row being disabled.
• Now, only the not yet imported repositories are returned, so no "Imported" repository is being returned.
• Fixes sorting of repositories across pagination.
• Fixes calculation of the total number of repositories to be imported.

✔️ Checklist

• A changeset describing the change and affected packages. (more info)
• Added or Updated documentation
• Tests for new functionality and regression tests for bug fixes
• Screenshots attached (for UI changes)

[rhdh-gh-app]

Important

This PR includes changes that affect public-facing API. Please ensure you are adding/updating documentation for new features or behavior.

Missing Changesets

The following package(s) are changed by this PR but do not have a changeset:

• @red-hat-developer-hub/backstage-plugin-bulk-import

See CONTRIBUTING.md for more information about how to add changesets.

Changed Packages

Package Name	Package Path	Changeset Bump	Current Version
@red-hat-developer-hub/backstage-plugin-bulk-import-backend	workspaces/bulk-import/plugins/bulk-import-backend	minor	v7.2.1
@red-hat-developer-hub/backstage-plugin-bulk-import	workspaces/bulk-import/plugins/bulk-import	none	v7.2.1

[rhdh-qodo-merge]

Review Summary by Qodo

Filter bulk-import repository list to show only not-yet-imported repositories

✨ Enhancement 🐞 Bug fix

Walkthroughs

Description

• Filter bulk-import repository list to show only not-yet-imported repositories
• Refactor GitHub/GitLab API pagination to fetch all repositories across pages
• Remove pagination parameters from service layer, handle client-side filtering
• Add comprehensive test coverage for repository filtering scenarios

Diagram

flowchart LR
  A["Repository APIs<br/>GitHub/GitLab"] -->|"Fetch all pages"| B["listAllRepositories<br/>Functions"]
  B -->|"All repositories"| C["findAllRepositories<br/>Handler"]
  D["Catalog API<br/>Locations"] -->|"Already imported"| C
  C -->|"Filter & slice"| E["Not-yet-imported<br/>repositories"]
  E -->|"Return paginated"| F["Frontend UI"]

Loading

File Changes

1. workspaces/bulk-import/plugins/bulk-import-backend/src/github/utils/utils.ts ✨ Enhancement +122/-0

Add pagination helper functions for repository listing

workspaces/bulk-import/plugins/bulk-import-backend/src/github/utils/utils.ts

---

2. workspaces/bulk-import/plugins/bulk-import-backend/src/github/utils/repoUtils.ts ✨ Enhancement +46/-109

Simplify repository fetching using new pagination helpers

workspaces/bulk-import/plugins/bulk-import-backend/src/github/utils/repoUtils.ts

---

3. workspaces/bulk-import/plugins/bulk-import-backend/src/github/githubApiService.ts ✨ Enhancement +0/-7

Remove pagination parameters from service methods

workspaces/bulk-import/plugins/bulk-import-backend/src/github/githubApiService.ts

---

View more (10)

4. workspaces/bulk-import/plugins/bulk-import-backend/src/github/types.ts ✨ Enhancement +14/-0

Add type definitions for API responses

workspaces/bulk-import/plugins/bulk-import-backend/src/github/types.ts

---

5. workspaces/bulk-import/plugins/bulk-import-backend/src/gitlab/utils/utils.ts ✨ Enhancement +33/-1

Add pagination helper for GitLab repository listing

workspaces/bulk-import/plugins/bulk-import-backend/src/gitlab/utils/utils.ts

---

6. workspaces/bulk-import/plugins/bulk-import-backend/src/gitlab/utils/repoUtils.ts ✨ Enhancement +26/-59

Simplify GitLab repository fetching using new helper

workspaces/bulk-import/plugins/bulk-import-backend/src/gitlab/utils/repoUtils.ts

---

7. workspaces/bulk-import/plugins/bulk-import-backend/src/gitlab/gitlabApiService.ts ✨ Enhancement +0/-3

Remove pagination parameters from GitLab service

workspaces/bulk-import/plugins/bulk-import-backend/src/gitlab/gitlabApiService.ts

---

8. workspaces/bulk-import/plugins/bulk-import-backend/src/service/handlers/repository/repositories.ts 🐞 Bug fix +47/-10

Filter repositories against catalog locations and paginate client-side

workspaces/bulk-import/plugins/bulk-import-backend/src/service/handlers/repository/repositories.ts

---

9. workspaces/bulk-import/plugins/bulk-import-backend/src/service/handlers/repository/repositories.test.ts 🧪 Tests +199/-1

Add comprehensive filtering test cases for repositories

workspaces/bulk-import/plugins/bulk-import-backend/src/service/handlers/repository/repositories.test.ts

---

10. workspaces/bulk-import/plugins/bulk-import-backend/src/service/handlers/repository/repositories-gitlab.test.ts 🧪 Tests +195/-1

Add comprehensive filtering test cases for GitLab

workspaces/bulk-import/plugins/bulk-import-backend/src/service/handlers/repository/repositories-gitlab.test.ts

---

11. workspaces/bulk-import/plugins/bulk-import-backend/src/github/githubApiService.test.ts 🧪 Tests +52/-30

Update mocks to match new API response structure

workspaces/bulk-import/plugins/bulk-import-backend/src/github/githubApiService.test.ts

---

12. workspaces/bulk-import/plugins/bulk-import-backend/__fixtures__/handlers.ts 🧪 Tests +7/-0

Add mock handler for catalog locations endpoint

workspaces/bulk-import/plugins/bulk-import-backend/fixtures/handlers.ts

---

13. workspaces/bulk-import/.changeset/long-schools-learn.md 📝 Documentation +5/-0

Document breaking change in repository filtering behavior

workspaces/bulk-import/.changeset/long-schools-learn.md

---

[rhdh-qodo-merge]

Code Review by Qodo

🐞 Bugs (1) 📘 Rule violations (0) 📎 Requirement gaps (1)

1. Wrong imported-repo detection ✗ 🐞 Bug ≡ Correctness

Description

findAllRepositories filters out a repository as “already imported” if any catalog location target
starts with the repo’s html_url, which also matches non-root catalog files (e.g., monorepo subpaths)
and can hide repositories that should still be importable. This conflicts with existing
import-candidate logic that explicitly ignores non-root catalog-info.yaml locations.

Code

workspaces/bulk-import/plugins/bulk-import-backend/src/service/handlers/repository/repositories.ts[R61-81]

+  const [alreadyImportedRepositories, allRepositoriesResponse] =
+    await Promise.all([
+      deps.catalogHttpClient.listCatalogUrlLocations(),
+      deps.gitApiService.getRepositoriesFromIntegrations(search, pageSize),
+    ]);

-  return repos;
+  const alreadyImportedRepositoriesLocationTargets = Array.from(
+    new Set(alreadyImportedRepositories.uniqueCatalogUrlLocations.keys()),
+  );
+
+  const { repositories: allRepositories, errors } = allRepositoriesResponse;
+
+  const notImportedYetRepositories = allRepositories.filter(repo => {
+    const html_urlWithSlash = repo.html_url.concat('/');
+
+    const alreadyImported = alreadyImportedRepositoriesLocationTargets.some(
+      target => target.startsWith(html_urlWithSlash),
+    );
+
+    return !alreadyImported;
+  });

Evidence

The new filtering logic marks a repo as imported if any catalog Location target starts with the repo
URL prefix, which includes catalog files not at the repository root. Elsewhere, the import pipeline
explicitly treats only root-level ${repoUrl}/blob/${defaultBranch}/${catalogFilename} as an import
Location, meaning the repository list can now hide repos that the import pipeline would still
consider eligible.

workspaces/bulk-import/plugins/bulk-import-backend/src/service/handlers/repository/repositories.ts[61-81]
workspaces/bulk-import/plugins/bulk-import-backend/src/service/handlers/import/bulkImports.ts[315-347]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`findAllRepositories` currently considers a repository imported if **any** catalog Location target starts with `repo.html_url + '/'`. This is too broad: it will treat repos as imported even when only a **non-root** catalog file is registered (e.g., monorepo subpaths), which contradicts existing logic that only considers the **root** catalog-info file an import candidate.

### Issue Context
There is already established logic (`findImportCandidates`) that defines when a Location represents an import candidate: the Location must equal `${repoUrl}/blob/${defaultBranch}/${catalogFilename}` (root of repo). The repository listing should use the same definition when deciding whether a repo is "already imported".

### Fix Focus Areas
- workspaces/bulk-import/plugins/bulk-import-backend/src/service/handlers/repository/repositories.ts[61-88]
- workspaces/bulk-import/plugins/bulk-import-backend/src/service/handlers/import/bulkImports.ts[315-347]

### Suggested fix approach
1. Determine the expected catalog filename via `getCatalogFilename(deps.config)`.
2. Build a set of imported repo URLs by parsing catalog Location targets **and only keeping those that represent root catalog files**, e.g.:
  - Extract `repoUrl` via split on `/blob/` (same as `repoUrlFromLocation`).
  - Keep only targets that match `^${repoUrl}/blob/[^/]+/${catalogFilename}$` (root-level file only).
3. Mark a repo as imported if its `repo.html_url` (normalized for trailing slash/case) matches one of those extracted `repoUrl`s.
4. Avoid `startsWith(repoUrl + '/')` for import detection.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Pagination drops pages ☑ 🐞 Bug ≡ Correctness

Description

retrieveAllPagesData returns only the first page when it cannot find a rel="last" link header,
which can silently omit repositories even when additional pages exist. This can make the bulk-import
“leftovers” list incomplete.

Code

workspaces/bulk-import/plugins/bulk-import-backend/src/github/utils/utils.ts[R444-466]

+async function retrieveAllPagesData<ResponseType>(
+  deps: {
+    logger: LoggerService;
+  },
+  ghApiName: string,
+  fetchPageFn: (pageNumber: number) => Promise<OctokitResponse<ResponseType>>,
+): Promise<OctokitResponse<ResponseType>['data'][]> {
+  const PAGE_NUMBER_REGEX_MATCH_INDEX = 1;
+  const SECOND_PAGE_NUMBER = 2;
+
+  const firstPageResponse = await fetchPageFn(1);
+
+  const lastPageNumberString = firstPageResponse?.headers?.link
+    ?.split(',')
+    ?.find(s => s.includes('rel="last"'))
+    ?.match(/page=(\d+)/)?.[PAGE_NUMBER_REGEX_MATCH_INDEX];
+
+  if (!lastPageNumberString) {
+    deps.logger.debug(
+      `Unable to extract page number from rel='last' link found in response headers from '${ghApiName}' GH endpoint => returning current page size`,
+    );
+    return [firstPageResponse.data];
+  }

Evidence

The helper extracts the last page number only from rel="last"; if not found, it logs and returns
just the first page’s data, without attempting to follow rel="next" or otherwise detect additional
pages.

workspaces/bulk-import/plugins/bulk-import-backend/src/github/utils/utils.ts[444-466]
workspaces/bulk-import/plugins/bulk-import-backend/src/github/utils/utils.ts[370-404]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`retrieveAllPagesData` stops pagination unless it can extract a page number from a `rel="last"` link. If `rel="last"` is missing (even if `rel="next"` exists), only the first page is returned, silently dropping results.

### Issue Context
This helper is used to build the full repository list for token/app access, and correctness depends on fetching all pages.

### Fix Focus Areas
- workspaces/bulk-import/plugins/bulk-import-backend/src/github/utils/utils.ts[370-404]
- workspaces/bulk-import/plugins/bulk-import-backend/src/github/utils/utils.ts[444-483]

### Suggested fix approach
- Prefer `octokit.paginate` instead of manually parsing Link headers.
 - For `repos.listForAuthenticatedUser`, `octokit.paginate(octokit.rest.repos.listForAuthenticatedUser, {...})` returns a flat list.
 - For `apps.listReposAccessibleToInstallation`, use `octokit.paginate` with a map function that returns `response.data.repositories`.
- If keeping manual pagination, add a fallback loop that follows `rel="next"` until it disappears (or until an empty page is returned), rather than returning early when `rel="last"` is absent.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. Unbounded page fetch fanout ☑ 🐞 Bug ➹ Performance

Description

retrieveAllPagesData fires requests for every remaining page at once using Promise.all, which can
create very high concurrency for large accounts/installations. This can trigger rate limiting or
cause resource spikes in the backend.

Code

workspaces/bulk-import/plugins/bulk-import-backend/src/github/utils/utils.ts[R474-483]

+  const pagePromises = [];
+  for (let i = SECOND_PAGE_NUMBER; i <= lastPageNumber; i++) {
+    pagePromises.push(fetchPageFn(i));
+  }
+  const remainingPages = await Promise.all(pagePromises);
+
+  const allPages = [firstPageResponse, ...remainingPages];
+
+  return allPages.map(page => page.data);
+}

Evidence

The implementation constructs an array of promises for all pages 2..N and awaits them in parallel,
with no concurrency limiting or backoff strategy.

workspaces/bulk-import/plugins/bulk-import-backend/src/github/utils/utils.ts[474-483]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`retrieveAllPagesData` spawns one request per page and awaits all of them concurrently. For large `lastPageNumber`, this creates a burst of requests that can exceed GitHub rate limits and stress the backend.

### Issue Context
This is especially likely now that the repository endpoint intends to list *all* repos to support consistent sorting and post-filtering.

### Fix Focus Areas
- workspaces/bulk-import/plugins/bulk-import-backend/src/github/utils/utils.ts[444-483]

### Suggested fix approach
- Replace the Promise.all fanout with one of:
 1) `octokit.paginate` (preferred; handles paging efficiently), or
 2) a bounded-concurrency approach (e.g., fetch sequentially, or batch pages with a small concurrency limit such as 5–10).
- Consider adding basic retry/backoff for 429/secondary-rate-limit responses if the code is expected to handle large repo counts.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

4. No docs for Bulk Import change 📎 Requirement gap ⚙ Maintainability

Description

This PR changes Bulk Import behavior to return only not-yet-imported repositories, but it does not
include any upstream documentation updates describing the new behavior. Missing documentation can
cause user confusion and violates the requirement to update upstream docs for this behavior change.

Code

workspaces/bulk-import/.changeset/long-schools-learn.md[R1-5]

+---
+'@red-hat-developer-hub/backstage-plugin-bulk-import-backend': minor
+---
+
+**BREAKING** Changes the behavior of the bulk-import backend plugin to return only repositories that are yet to be imported by filtering out the already imported ones. Therefore, the frontend will not display already imported repositories with status displayed as "Imported" anymore.

Evidence

The changeset explicitly documents a breaking behavior change (only not-yet-imported repos
returned), but the compliance checklist requires upstream documentation updates reflecting this
behavior; no such documentation updates are present in the PR diff.

Upstream documentation updates completed for Bulk Import change
workspaces/bulk-import/.changeset/long-schools-learn.md[1-5]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The PR changes Bulk Import behavior (only not-yet-imported repositories are returned), but upstream documentation has not been updated to reflect this new behavior.

## Issue Context
The changeset marks this as a breaking behavior change; the compliance requirement expects upstream docs (design docs/release notes/other upstream documentation) to be updated accordingly.

## Fix Focus Areas
- workspaces/bulk-import/.changeset/long-schools-learn.md[1-5]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

5. Catalog call has no fallback 🐞 Bug ☼ Reliability

Description

findAllRepositories now makes catalog location listing a required dependency via Promise.all, so any
error/latency in the catalog locations lookup can fail repository listing entirely rather than
returning a best-effort repo list. The catalog lookup performs multiple network/API calls (including
a high-limit entity fetch), increasing the chance and impact of failures.

Code

workspaces/bulk-import/plugins/bulk-import-backend/src/service/handlers/repository/repositories.ts[R61-65]

+  const [alreadyImportedRepositories, allRepositoriesResponse] =
+    await Promise.all([
+      deps.catalogHttpClient.listCatalogUrlLocations(),
+      deps.gitApiService.getRepositoriesFromIntegrations(search, pageSize),
+    ]);

Evidence

The handler now unconditionally awaits catalogHttpClient.listCatalogUrlLocations() alongside the
SCM fetch; if it throws, the whole request fails. The catalog client’s implementation performs
multiple calls (config locations + /catalog/locations fetch + catalog API entity listing), which
are potential failure points and not guarded here.

workspaces/bulk-import/plugins/bulk-import-backend/src/service/handlers/repository/repositories.ts[61-65]
workspaces/bulk-import/plugins/bulk-import-backend/src/catalog/catalogHttpClient.ts[78-127]
workspaces/bulk-import/plugins/bulk-import-backend/src/catalog/catalogHttpClient.ts[129-165]
workspaces/bulk-import/plugins/bulk-import-backend/src/catalog/catalogHttpClient.ts[191-209]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`findAllRepositories` now depends on `listCatalogUrlLocations()` for filtering, but does not handle failures. A transient catalog outage can make the repository listing endpoint fail entirely.

### Issue Context
`CatalogHttpClient.listCatalogUrlLocations()` performs multiple operations (config scan, HTTP fetch to `/locations`, catalog entity listing). Any of these can throw.

### Fix Focus Areas
- workspaces/bulk-import/plugins/bulk-import-backend/src/service/handlers/repository/repositories.ts[61-98]
- workspaces/bulk-import/plugins/bulk-import-backend/src/catalog/catalogHttpClient.ts[78-127]

### Suggested fix approach
- Wrap the catalog call in try/catch and decide on a deterministic fallback behavior, e.g.:
 - If catalog lookup fails: log error and return the (unfiltered) repo list, or
 - Return `statusCode: 200` with repos but include an `errors` entry indicating filtering could not be applied.
- If filtering must be strict, return a structured 5xx response with an explicit error message rather than throwing an unhandled exception.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

6. Type-only imports not marked ☑ 🐞 Bug ⚙ Maintainability

Description

New code imports type-only symbols as runtime values (e.g., RestEndpointMethodTypes,
OctokitResponse, ProjectSchema), which commonly violates Backstage ESLint/TS rules and increases the
chance of runtime import issues in stricter module setups. The codebase already uses import type
for RestEndpointMethodTypes elsewhere, indicating this is expected style.

Code

workspaces/bulk-import/plugins/bulk-import-backend/src/github/types.ts[R23-24]

+import { RestEndpointMethodTypes } from '@octokit/rest';
+

Evidence

RestEndpointMethodTypes is imported as a value but only used to define exported TypeScript types,
and OctokitResponse/ProjectSchema are similarly used only in type positions. Another file in the
repo imports RestEndpointMethodTypes using import type, suggesting lint/style expectations for
type-only imports.

workspaces/bulk-import/plugins/bulk-import-backend/src/github/types.ts[17-24]
workspaces/bulk-import/plugins/bulk-import-backend/src/github/types.ts[138-148]
workspaces/bulk-import/plugins/bulk-import-backend/src/github/utils/utils.ts[17-30]
workspaces/bulk-import/plugins/bulk-import-backend/src/gitlab/utils/utils.ts[17-28]
workspaces/bulk-import/plugins/bulk-import-backend/src/github/GithubAppManager.test.ts[17-22]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
Several newly added imports bring in symbols that are only used for TypeScript typing, but they are imported as runtime values. In Backstage repos, lint/TS settings commonly enforce consistent type-only imports.

### Issue Context
The codebase already uses `import type { RestEndpointMethodTypes } from '@octokit/rest'` in tests, suggesting this pattern is expected.

### Fix Focus Areas
- workspaces/bulk-import/plugins/bulk-import-backend/src/github/types.ts[17-24]
- workspaces/bulk-import/plugins/bulk-import-backend/src/github/utils/utils.ts[17-30]
- workspaces/bulk-import/plugins/bulk-import-backend/src/gitlab/utils/utils.ts[17-28]

### Suggested fix approach
- Change:
 - `import { RestEndpointMethodTypes } ...` -> `import type { RestEndpointMethodTypes } ...`
 - `import { OctokitResponse } ...` -> `import type { OctokitResponse } ...`
 - `import { ..., ProjectSchema } ...` -> `import { ... }` plus `import type { ProjectSchema } ...` (or `import type` in the same statement if supported by lint config).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

ⓘ The new review experience is currently in Beta. Learn more

[dzemanov]

GitLab auth was not working for me unless I updated Gitlab from:

token: input.credential.token

to

oauthToken: input.credential.token

This file change is not part of this PR and was tested for #2647, so I might have misconfigured something.
@PatAKnight is gitlab working for you?

[dzemanov]

Thank you for the PR @dom-aug, looks great, I have found only minor issues.
I have tested github and gitlab flow with postgres and better-sqlite3.

I would like to point out that repositories are filtered out even when their entities are not yet imported in Catalog, just waiting to be imported (PR with catalog-info.yamlfile was just created and not merged yet - but location is created for the repository). Just checking we are aware of this behavior and is expected. I have confirmed that once the PR is merged, repository eventually appears in Catalog.

Performance is slow even when paginating. I understand it is expected when user has access to a lot of repositories, however, I think we should investigate if it could be improved, at least for paging.

Gitlab:

Screen.Recording.2026-04-13.at.11.36.45.mov

Github:

bulk-import-repos_compressed.mov

[dzemanov]

Thank you @dom-aug for the changes, confirmed that pagination works better:

Screen.Recording.2026-04-20.at.9.47.38.mov

[dzemanov]

@dom-aug can you please regenerate openapi via:

cd plugins/bulk-import-backend
yarn openapi

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud