Update CI workflows by remia · Pull Request #1770 · AcademySoftwareFoundation/OpenColorIO

remia · 2023-02-19T20:45:33Z

As discussed during TSC meetings, here are some update and new CI workflows.

This PR contains the following changes:

Remove VFX 2019 platform testing from the CI workflow as well as some redondant jobs, there are 3 slots left (out of the 20 for concurrent processing) for the upcoming VFX2023 images.
Consequently remove Python2 from the officially supported Python versions, it is no longer tested and advertised in the package metadata but I didn't actively break it intentionally (ie. compatibility conditional paths are still here).
Extract "latest dependencies" jobs from the nightly analysis into their own workflow, note this workflow is still failing at the moment.
Add a new "platform latest" nightly build using latest runners available on Github, C++20 standard and compiler sanitizers (only address enabled at this time). This detected a few memory out of bounds and leaks issues in apphelper and unit test, nothing in the core library so far.
Update to CMake 3.13 minimum version required to use LINK_OPTIONS and bumped Imath external project build to 3.1.6.

Note some jobs changed their names in the CI workflow, updating the required list is again going to be needed, hopefully this will happen less in the future as I removed some variability from these name like compiler versions which are not always fixed even in ci-20xx docker images (Clang). I also had a recurring issue where the GCC based linux platform latest build was killed by GitHub Action during compilation of OpenEXR with no diagnostic messages, apparently this can happen when some hardware resources limits are reached. This was on my fork but may be different here, this could require further investigations, usually manual relaunch of the failed jobs fix the issue.

Signed-off-by: Rémi Achard <remiachard@gmail.com>

remia · 2023-02-21T16:16:47Z

Chatting with @JeanChristopheMorinPerso, I disabled parallel build on OpenEXR ExternalProject, I'll try to trigger the build a couple of times but on the first try it seemed to have resolved the issue!

Signed-off-by: Rémi Achard <remiachard@gmail.com>

doug-walker

The updated CI workflow looks great!

Few questions about the new workflows ...

doug-walker · 2023-03-04T05:36:30Z

+        run: |
+          share/ci/scripts/multi/install_imath.sh latest $EXT_PATH
+          share/ci/scripts/multi/install_openexr.sh latest $EXT_PATH
+          share/ci/scripts/multi/install_oiio.sh latest $EXT_PATH


It seems we should have some tests with DOCIO_USE_OIIO_FOR_APPS=ON?

Agreed I'll add some, unfortunately I think we will not be able to test until other issues are fixed in that workflow.

Actually I'll not include that change I think and leave that to the upcoming workflow update planned, do you agree @doug-walker?

doug-walker · 2023-03-04T05:38:55Z

+      - name: Install indirect dependencies
+        run: |
+          share/ci/scripts/multi/install_pugixml.sh latest
+      - name: Install fixed ext package versions


I'm worried about these getting out of sync with the CMake script versions. Would it work to install only the Imath/EXR/OIIO/etc. manually and then use DOCIO_INSTALL_EXT_PACKAGES=MISSING for the rest?

I think this is a valid point, though it seems to be a PR in itself, like you said I just copied over the workflow. I'm not confortable making this change without ability to test.

doug-walker · 2023-03-04T05:40:52Z

+        run: |
+          EXT_PATH=/usr/local
+          echo "EXT_PATH=$EXT_PATH" >> $GITHUB_ENV
+      - name: Install indirect dependencies


Not related to your PR, I realize you just copied these over, but I'm curious what these are used for. Same for the other platforms.

These are the dependencies of dependencies we are installing, like boost to name the obvious. In this workflow we are building more things than the CI, mostly OpenImageIO and OSL which have additional dependencies.

doug-walker · 2023-03-04T05:43:04Z

+        shell: bash
+      - name: Install indirect dependencies
+        run: |
+          vcpkg install zlib:x64-windows


Seems like zlib gets installed twice?

Yes, this seem to be introduced by #1725, this might be due to the specific version requirement for the last version of zlib which vcpkg and others didn't have at that time?

doug-walker · 2023-03-04T05:53:00Z

+          # -------------------------------------------------------------------
+          # VFX CY2022, C++17, docs, OpenFX
+          - build: 1
+            build-docs: 'ON'


Is it worth having docs and openfx part of this matrix? It's not clear why Imath/EXR/OIIO/etc. would break those. I think you could leave them off for this workflow.

Same remark as above, not updating this workflow in this PR.

doug-walker · 2023-03-04T06:04:13Z

For anyone else reviewing the new CI settings, I made a spreadsheet showing the current and new matrix:
https://docs.google.com/spreadsheets/d/1Iw5uRw6voB54wFpkSSOGrcAXkciZfx9nlBSVHz6N4P8/edit?usp=sharing

Signed-off-by: Rémi Achard <remiachard@gmail.com>

remia · 2023-03-07T21:00:26Z

Thanks for the review @doug-walker, most of the points are about the old analysis workflow (new dependencies_latest) which I didn't touch and is currently failing in nightly for the past few months. From my understanding @cedrik-fuoco-adsk was about to work on these so I think it's best to push the comments for that upcoming work? I'm not really confortable updating things in there knowing the builds are going to fail early and I won't be able to test.

doug-walker

Agree with all your points Remi. Let's go ahead and merge this and then follow up on the other items in a separate PR, after we get the workflows to start succeeding again.

* Update CI workflows Signed-off-by: Rémi Achard <remiachard@gmail.com> * Test building OpenEXR single threaded Signed-off-by: Rémi Achard <remiachard@gmail.com> * Add comment on disabling parallel build Signed-off-by: Rémi Achard <remiachard@gmail.com> * Update python_requires Signed-off-by: Rémi Achard <remiachard@gmail.com> * Remove comments around CI jobs Signed-off-by: Rémi Achard <remiachard@gmail.com> --------- Signed-off-by: Rémi Achard <remiachard@gmail.com> Signed-off-by: Cédrik Fuoco <cedrik.fuoco@autodesk.com>

remia added 2 commits February 19, 2023 20:31

Update CI workflows

f017534

Signed-off-by: Rémi Achard <remiachard@gmail.com>

Test building OpenEXR single threaded

51bb573

Signed-off-by: Rémi Achard <remiachard@gmail.com>

Add comment on disabling parallel build

c626707

Signed-off-by: Rémi Achard <remiachard@gmail.com>

JeanChristopheMorinPerso reviewed Feb 22, 2023

View reviewed changes

Comment thread setup.cfg Outdated

Update python_requires

a6832fd

Signed-off-by: Rémi Achard <remiachard@gmail.com>

cedrik-fuoco-adsk approved these changes Feb 28, 2023

View reviewed changes

doug-walker reviewed Mar 4, 2023

View reviewed changes

Remove comments around CI jobs

24eb893

Signed-off-by: Rémi Achard <remiachard@gmail.com>

doug-walker approved these changes Mar 8, 2023

View reviewed changes

remia mentioned this pull request Mar 8, 2023

Adsk contrib - Add support for minimum and recommended versions for dependencies #1772

Closed

doug-walker merged commit 3797a92 into AcademySoftwareFoundation:main Mar 9, 2023

remia deleted the ci-workflow-update branch March 12, 2023 21:18

Conversation

remia commented Feb 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

remia commented Feb 21, 2023

Uh oh!

Uh oh!

doug-walker left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

remia Mar 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

doug-walker commented Mar 4, 2023

Uh oh!

remia commented Mar 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

doug-walker left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

remia commented Feb 19, 2023 •

edited

Loading

remia Mar 7, 2023 •

edited

Loading

remia commented Mar 7, 2023 •

edited

Loading