feat(update): Introduce user-specific config files to prevent merge conflicts

[Fortune-Ndlovu]

Description

This PR introduces a new convention for user-specific config overrides using a structured directory layout inside the existing configs/ folder. These override files are .gitignore'd and will automatically be loaded by the system at runtime.

New directory structure

configs/
├── app-config/
│   ├── app-config.yaml                  # Default (tracked)
│   └── app-config.local.yaml            # User override (git-ignored)
├── dynamic-plugins/
│   ├── dynamic-plugins.yaml             # Default (tracked)
│   └── dynamic-plugins.override.yaml    # User override (git-ignored)
├── extra-files/                         # Optional additional files
│   ├── github-app-credentials.example.yaml
│   ├── github-app-credentials.local.yaml
│   └── (any other user file)

Entrypoint updates

• fixes.sh (install-dynamic-plugins container):

  • Detects and symlinks dynamic-plugins.override.yaml as the active config, falling back to the default otherwise.

• wait-for-plugins.sh (rhdh container):

  • Checks for app-config.local.yaml and appends it to the list of configs passed to the node packages/backend startup command.

Git ignores

Updated .gitignore to:

• Exclude all *.local.yaml and override files
• Exclude all files in extra-files/, except the github-app-credentials.example.yaml

Result

Users can now:

• Add local config overrides without touching version-controlled files.
• Pull upstream changes without merge issues.
• Use *.local.yaml and override files that are auto-loaded at runtime.

Which issue(s) does this PR fix or relate to

• Fixes https://issues.redhat.com/browse/RHIDP-4263

PR acceptance criteria

• Tests
• Documentation

How to test changes / Special notes to the reviewer

[rm3l]

/cc @rm3l @kadel @benwilcock

[rm3l]

This PR introduces a user-friendly approach to managing configuration files in RHDH Local, ensuring users can modify their setup without conflicting with repository updates.

Key Changes:

• Added configs/app-config.local.yaml.sample and configs/dynamic-plugins.yaml.sample as template configuration files.

• Removed configs/app-config.local.yaml and configs/dynamic-plugins.yaml from the repository to avoid merge conflicts.

• Updated .gitignore to ignore user-created config files (app-config.local.yaml, dynamic-plugins.yaml).

• Modified compose.yaml:

  • Ensures missing config files don’t break the setup by falling back to sample files.
  • Simplified volume mounts for install-dynamic-plugins.

• Updated the README:

  • Instructs users to copy sample files before making custom modifications.
  • Clarifies that the system will default to sample configurations if the user skips this step.

As I was concerned that this would not work OOTB without copying the app-config.yaml.sample and dynamic-plugins.yaml.sample files, I was thinking of another approach that could probably help here.

Since we control the entrypoint scripts used by both the install-dynamic-plugins and rhdh containers, and that we are mounting the whole configs folder in both containers, I think maybe we can tweak these scripts such that they allow us to support this "user configs" case.

I mean, with a new configs folder structure like this (making sure to git-ignore any *.local.yaml files):

configs/
├── app-config/
│   ├── app-config.local.yaml (git-ignored)
│   └── app-config.yaml
├── dynamic-plugins/
│   ├── dynamic-plugins.override.yaml (git-ignored)
│   └── dynamic-plugins.yaml
└── extra-files/ (git-ignore everything here except github-app-credentials.example.yaml)
    ├── github-app-credentials.example.yaml
    └── github-app-credentials.local.yaml (git-ignored)
    └── any-other-user-file (git-ignored)

We could do the following:

1. change the fixes.sh script (entrypoint of the install-dynamic-plugins container) to check for the presence of the configs/dynamic-plugins/dynamic-plugins.override.yaml file and create a symlink , like so:

if [ -f "configs/dynamic-plugins/dynamic-plugins.override.yaml" ]; then
    echo "Using dynamic-plugins.override.yaml file"
    ln -sf /opt/app-root/src/configs/dynamic-plugins/dynamic-plugins.override.yaml /opt/app-root/src/dynamic-plugins.yaml
else
    ln -sf /opt/app-root/src/configs/dynamic-plugins/dynamic-plugins.yaml /opt/app-root/src/dynamic-plugins.yaml
fi

1. change the wait-for-plugins.sh script (entrypoint of the rhdh container) to check for the presence of the configs/app-config/app-config.local.yaml file and append it to the node process args, like so:

USER_APP_CONFIG="configs/app-config/app-config.local.yaml"
extraCliArgs=""
if [ -f "$USER_APP_CONFIG" ]; then
    extraCliArgs="--config $USER_APP_CONFIG"
fi

node packages/backend --no-node-snapshot \
    --config app-config.yaml \
    --config app-config.example.yaml \
    --config app-config.example.production.yaml \
    --config $DYNAMIC_PLUGINS_CONFIG \
    --config configs/app-config/app-config.yaml $extraCliArgs

This way, users would just need to create their configs/dynamic-plugins/dynamic-plugins.override.yaml file and/or configs/app-config/app-config.local.yaml and/or any configs/extra-files/* file and it would work automagically.

Just like when working with a local Backstage, this would allow users to append their app-config.local.yaml to the Node process and Backstage would handle the merging of the various app-config files.

Thoughts? @kadel @benwilcock @Fortune-Ndlovu

[Fortune-Ndlovu]

Thanks, Armel for the great suggestion, I’ve now implemented this approach!

Refactored configs/ into a structured layout:

• configs/app-config/app-config.yaml (default)
• configs/app-config/app-config.local.yaml (user override, git-ignored)
• configs/dynamic-plugins/dynamic-plugins.yaml (default)
• configs/dynamic-plugins/dynamic-plugins.override.yaml (user override, git-ignored)
• configs/extra-files/ for any additional user config (like GitHub credentials)

Updated fixes.sh and wait-for-plugins.sh to:

• Automatically load user override configs if present
• Fall back to defaults if not

Removed the need to manually copy .sample files — the system now works out-of-the-box.

Updated .gitignore, compose.yaml, test.yml, and the README.md accordingly.

[Fortune-Ndlovu]

CI unhappy

curl: (7) Failed to connect to localhost port 7007
Max retries reached. Exiting.

The HTTP 200 check step timed out after 30 retries meaning:
The container is running, but Backstage never started
Or the container crashed early
Or something in the new wait-for-plugins.sh or config logic broke app startup 👀

[Fortune-Ndlovu]

/cc @rm3l @kadel @benwilcock for review please!

[Fortune-Ndlovu]

@kadel, do we need example files? I feel like adding additional files may add additional complexity for new users

[rm3l]

do we need example files? I feel like adding additional files may add additional complexity for new users

I agree it could be helpful to have a few example files.
So I'd suggest adding a configs/app-config/app-config.local.example.yaml with a GH integration referencing the configs/extra-files/github-app-credentials.example.yaml file, along with a configs/dynamic-plugins/dynamic-plugins.override.example.yaml file.

[kadel]

@kadel, do we need example files? I feel like adding additional files may add additional complexity for new users

I'm convinced about the opposite. Almost every user will need dynamic-plugins.override.yaml and app-config.local.yaml and without an example that they can easily copy to bootstrap their custom configs we are making it first run a lot more complex.

We are already using the pattern of the example, configs that user's copies with default.env and github-app-credentials.example.yaml. We should do the same with dynamic plugins config and app-config as well.

[Fortune-Ndlovu]

cc/ @kadel @rm3l, final review needed please :)

[rm3l]

LGTM with the review comments addressed. I've also checked that the approach here remains backward-compatible.

/lgtm