Skip to content

[9.0] Improve CI to include legacy adaptor#8248

Merged
chaen merged 2 commits into
DIRACGrid:integrationfrom
Robin-Van-de-Merghel:robin-improve-ci
Aug 4, 2025
Merged

[9.0] Improve CI to include legacy adaptor#8248
chaen merged 2 commits into
DIRACGrid:integrationfrom
Robin-Van-de-Merghel:robin-improve-ci

Conversation

@Robin-Van-de-Merghel

@Robin-Van-de-Merghel Robin-Van-de-Merghel commented Jul 9, 2025

Copy link
Copy Markdown
Contributor

BEGINRELEASENOTES

*Integration Tests
FIX: Add in CI and in integration_test.py legacy adaptors.

ENDRELEASENOTES

@Robin-Van-de-Merghel

Copy link
Copy Markdown
Contributor Author

Todo (here or in another PR): add doc to explain how it works + to show MacOS compatibility (OSX=true ./integration_tests.py ...)

Comment thread integration_tests.py Outdated
Comment thread src/DIRAC/Core/Utilities/Extensions.py
@Robin-Van-de-Merghel Robin-Van-de-Merghel changed the title [V9] Improve CI to include legacy adaptor [9.0] Improve CI to include legacy adaptor Jul 14, 2025
Comment thread tests/Jenkins/dirac_ci.sh Outdated
Comment thread tests/Jenkins/dirac-cfg-setup-diracx.py
Comment thread integration_tests.py Outdated
@Robin-Van-de-Merghel Robin-Van-de-Merghel force-pushed the robin-improve-ci branch 3 times, most recently from 532a6e2 to 1cb40f8 Compare July 17, 2025 19:02
Comment thread .github/workflows/integration.yml Outdated
Comment thread src/DIRAC/Core/Tornado/Client/ClientSelector.py Outdated
Comment thread tests/Jenkins/utilities.sh Outdated
@Robin-Van-de-Merghel Robin-Van-de-Merghel force-pushed the robin-improve-ci branch 3 times, most recently from 8f0d623 to 9eb1d5e Compare July 18, 2025 09:08
Comment thread tests/CI/install_client.sh Outdated
@Robin-Van-de-Merghel Robin-Van-de-Merghel force-pushed the robin-improve-ci branch 5 times, most recently from 7271be7 to e3b1d68 Compare July 23, 2025 06:40
@Robin-Van-de-Merghel

Copy link
Copy Markdown
Contributor Author

Two bugs were found @chrisburr @chaen (discussed F2F with Christophe, told me to pin):

  1. e3b1d68 in dirac-proxy-init.

If we have a group jetkins_users in VO Jenkins, but without VOMS, by doing dirac-proxy-init it will work in DIRAC (work meaning "there won't be an error raised") but our VO will secretly be the DIRAC VO. If we activate DiracX (purpose of this PR), it will lead to a 400 error because DiracX verifies that our VO corresponds to our group, which in this case is False.

In the commit, we can see that I changed a call of getVOMSVOForGroup to getVOForGroup, and this bit that was switched in getVOForGroup:

getVO() or gConfig.getValue(f"{gBaseRegistrySection}/Groups/{group}/VO", "")
# to
gConfig.getValue(f"{gBaseRegistrySection}/Groups/{group}/VO", "") or getVO()
  1. d2f99c0 in setJobAttributes for Status.

In the past, while testing this function we did this:

res = jobStateUpdateClient.setJobStatusBulk(
            jobID,
            {
                str(datetime.datetime.utcnow() + datetime.timedelta(hours=1)): {
                    "Status": JobStatus.WAITING,
                    "MinorStatus": "MinorStatus",
                    "Source": "Unknown",
                },
            }
            False,
)

# ...

res = jobStateUpdateClient.setJobStatus(jobID, JobStatus.RUNNING)

# ...

assert res["Value"]["Status"] == JobStatus.RUNNING

The test is wrong: while doing setJobStatusBulk (at the top), we set a time 1 hour in the future, while in the setJobStatus we set it in the present. As I connected the JobStateUpdateClient to DiracX, this test failed because DiracX follows the right behaviour.

I did not fix setJobStatus because we use now DiracX.

@Robin-Van-de-Merghel Robin-Van-de-Merghel force-pushed the robin-improve-ci branch 5 times, most recently from 0a9dfa9 to c4511f2 Compare July 23, 2025 11:27
Comment thread src/DIRAC/Core/Tornado/Client/ClientSelector.py Outdated
Comment thread tests/CI/docker-compose.yml Outdated
Comment thread src/DIRAC/WorkloadManagementSystem/Client/JobMonitoringClient.py
@Robin-Van-de-Merghel

Copy link
Copy Markdown
Contributor Author

Other fix: Now, if a service is set as "legacy adapted" in the CS but not set in the legacy client (diracxClient = future...), it will raise instantly an error.

(Before, only warn logs were emitted, and by default, a diset client were taken)

[3b20343] fix: Fixed ClientSelector to raise an error if the CS is badly configured
[4b715ef] fix: Fixed doc, and went back on the client selector
[ff736bb] docs: Add documentation to start the integration tests.
[7107442] feat: Add legacy adaptor in CI
[7ae3428] fix: Fixed to support better other branches
[2066814] feat: Add diracx repo choice in the integration test
[7859c6b] fix: Use JobStateUpdateClient instead of deactivating it
[e3b1d68] fix: Resolved dirac-proxy-init issue with multi-VO
[d2f99c0] fix: Resolved bug for setJobAttributes
[fe09e19] fix: Fixed ClientWMS test and reversed TEST_DIRACX flag
[1447a20] fix: Use opensearch db for job parametersDB
[f6116aa] fix: Made mariadb works better
[c4ab3f2] fix: Fixed CI to use sandoxes
[a5a02bc] fix: Activate job endpoints in CI
[a1bc1b1] fix: Applied changes to the ci workflows
[282efcb] refactor: Moved down diracx installation to fix auth
[ac22d9d] feat: Add DiracX tests into DIRAC CI, and use only latest tag.
[0ebf454] fix: Moved up futureServices argument to prevent "Missing mandatory /DiracX/URL configuration"
[d98b705] fix: Fixed small bug and refactored ClientSelector
[8ce234c] fix: Small fixes with docker compose cmd and removed grep
[08d9eb1] fix: Add filters to remove 'Client' from  services name
[bc44891] fix: Remove error when a future client is not available to warn instead
[a1f553d] fix: Changed to make the legacy adaptor automatic.
[3fb8c4d] fix: Fixed destroy command to support MacOSX
[42df58d] fix: Going back, removed CI, and going to add it to DiracX
[28b9789] fix: Small fix (forgot to change to docker**compose)
[61384af] feat: Add in CI new legacy adapted services.
[a3a7f5b] feat: Add legacy adaptor in CI
@chaen chaen merged commit 09fdfae into DIRACGrid:integration Aug 4, 2025
25 checks passed
@DIRACGridBot DIRACGridBot added the sweep:ignore Prevent sweeping from being ran for this PR label Aug 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

sweep:ignore Prevent sweeping from being ran for this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants