Enabled LTO (Link Time Optimization) except for gui. by openroad-ci · Pull Request #9069 · The-OpenROAD-Project/OpenROAD

openroad-ci · 2025-12-17T03:45:21Z

So far, LTO has been disabled because LTO-on causes a crash due to Qt used in gui library.

We can enable LTO globally if we selectively disable LTO for gui library.

Fixes #9050

- LTO-on causes a crash due to Qt used in gui library. Signed-off-by: Jaehyun Kim <jhkim@precisioninno.com>

Signed-off-by: Jaehyun Kim <jhkim@precisioninno.com>

github-actions · 2025-12-17T03:47:57Z

clang-tidy review says "All clean, LGTM! 👍"

maliberty · 2025-12-17T19:30:33Z

On debian11

lto1: internal compiler error: Segmentation fault

Rocky8 has various undefined references too. @vvbandeira is the compiler version different on these platforms?

jhkim-pii · 2025-12-18T00:38:54Z

3~7% performance gain. But 1.5x clean build and 15x incremental build runtime overhead.

maliberty · 2025-12-18T01:13:38Z

103s for an incremental build is somewhat painful. @QuantamHD @calewis do you have any suggestions for improving that?

vvbandeira · 2025-12-18T12:52:52Z

On debian11
lto1: internal compiler error: Segmentation fault
Rocky8 has various undefined references too. @vvbandeira is the compiler version different on these platforms?

OS          	GCC
ubuntu22.04 	11.4.0
ubuntu24.04 	13.3.0
rockylinux8 	13.3.1
rockylinux9 	11.5.0
debian11    	10.2.1
debian12    	12.2.0

103s for an incremental build is somewhat painful. @QuantamHD @calewis do you have any suggestions for improving that?

Note: data shown is for a 64-core machine, while CI and internal dev VMs are all 32-core, so absolute runtime might be greater than 103s

gadfort · 2025-12-18T13:55:58Z

@maliberty The LTO is also active on all the CPP tests which is noticeably slower to build as well (they probably don't need this on)

calewis · 2025-12-18T14:58:37Z

103s for an incremental build is somewhat painful. @QuantamHD @calewis do you have any suggestions for improving that?

You don't need to build with LTO enabled for development or running tests. Is there a specific thing LTO incremental builds are bottlenecking? There are ways of build profiling to figure out what is taking a long time, but it's been awhile since I used them.

calewis · 2025-12-18T15:16:46Z

+# LTO setting must come after project() for proper compiler detection
+option(LINK_TIME_OPTIMIZATION "Flag to control link time optimization: on by default" ON)
+
+string(TOUPPER "${CMAKE_BUILD_TYPE}" UPPER_BUILD_TYPE)


You don't need to jump through these hoops (at least not with newer cmakes, I didn't check with old ones). You can do:

option(LINK_TIME_OPTIMIZATION "Flag to control link time optimization: on by default" ON) if (LINK_TIME_OPTIMIZATION) include(CheckIPOSupported) # 1. Check if the compiler/linker even supports LTO check_ipo_supported(RESULT lto_supported OUTPUT lto_error) if(lto_supported) message(STATUS "LTO/IPO is supported and will be enabled for Release builds") set(CMAKE_INTERPROCEDURAL_OPTIMIZATION_RELEASE TRUE) set(CMAKE_INTERPROCEDURAL_OPTIMIZATION_RELWITHDEBINFO TRUE) else() message(STATUS "LTO/IPO not supported: ${lto_error}") endif() endif()

instead which avoids any string manipulation. You can set it for specific configs: https://cmake.org/cmake/help/latest/prop_tgt/INTERPROCEDURAL_OPTIMIZATION.html

Applied your code with a slight modification.

Release: LTO-on

RelWithDebInfo: LTO-off

Debug: LTO-off

calewis · 2025-12-18T15:31:25Z

@maliberty if this is an issue for the default build because of testing or something, we should take:

if(NOT CMAKE_BUILD_TYPE)
  set(CMAKE_BUILD_TYPE RELEASE)
endif()

out of the top level cmake file. And let the person building (or the tool) make the decision about which config they want.

maliberty · 2025-12-18T16:19:26Z

@calewis that code just establishes a default if CMAKE_BUILD_TYPE is unset

calewis · 2025-12-18T16:27:17Z

@calewis that code just establishes a default if CMAKE_BUILD_TYPE is unset

I'm aware, but etc/Build.sh doesn't set a config unless you ask for coverage, so using that script will silently give you the release config, which after this PR will give you LTO. If we assume that most people who are building from source are devs and not users then Release config would be the wrong default for development IMO.

maliberty · 2025-12-18T16:31:13Z

That is not a true assumption - many users still build from source.

…ug build types Signed-off-by: Jaehyun Kim <jhkim@precisioninno.com>

github-actions · 2025-12-19T13:48:37Z

clang-tidy review says "All clean, LGTM! 👍"

jhkim-pii · 2025-12-19T14:04:07Z

@maliberty New experiment data.

Release w/ LTO-off (current default) shows the shortest incremental build time.
From an incremental build time perspective, RelWithDebInfo mode is not that attractive.

In this experiment, I redirected stdout to a file. So Release shows a slightly better runtime than the previous data.

jhkim-pii · 2025-12-19T14:11:13Z

@vvbandeira

Regarding the CI fails,

Debian11

Can we upgrade the GNU compiler version?

Rocky-linux8

[2025-12-19T02:51:29.509Z] #9 168.7 /opt/rh/gcc-toolset-13/root/usr/libexec/gcc/x86_64-redhat-linux/13/ld: /usr/local/lib64/libspdlog.a(spdlog.cpp.o): in function <------ Is the static library compiled with old compiler?

spdlog::details::registry::set_tp(std::shared_ptr<spdlog::details::thread_pool>)': [2025-12-19T02:51:29.509Z] #9 168.7 spdlog.cpp:(.text+0x3bff): undefined reference to std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()'
[2025-12-19T02:51:29.509Z] #9 168.7 /opt/rh/gcc-toolset-13/root/usr/libexec/gcc/x86_64-redhat-linux/13/ld: /usr/local/lib64/libspdlog.a(spdlog.cpp.o): in function spdlog::details::registry::shutdown()': [2025-12-19T02:51:29.509Z] #9 168.7 spdlog.cpp:(.text+0x3e16): undefined reference to std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()'
[2025-12-19T02:51:29.509Z] #9 168.7 /opt/rh/gcc-toolset-13/root/usr/libexec/gcc/x86_64-redhat-linux/13/ld: /usr/local/lib64/libspdlog.a(spdlog.cpp.o): in function spdlog::details::registry::~registry()': [2025-12-19T02:51:29.509Z] #9 168.7 spdlog.cpp:(.text+0x3e6a): undefined reference to std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()'
[2025-12-19T02:51:29.509Z] #9 168.7 /opt/rh/gcc-toolset-13/root/usr/libexec/gcc/x86_64-redhat-linux/13/ld: spdlog.cpp:(.text+0x3e97): undefined reference to std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()' [2025-12-19T02:51:29.509Z] #9 168.7 /opt/rh/gcc-toolset-13/root/usr/libexec/gcc/x86_64-redhat-linux/13/ld: /usr/local/lib64/libspdlog.a(spdlog.cpp.o): in function spdlog::details::registry::drop(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)':
[2025-12-19T02:51:29.509Z] #9 168.7 spdlog.cpp:(.text+0x41b2): undefined reference to std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()' [2025-12-19T02:51:29.509Z] #9 168.7 /opt/rh/gcc-toolset-13/root/usr/libexec/gcc/x86_64-redhat-linux/13/ld: /usr/local/lib64/libspdlog.a(spdlog.cpp.o):spdlog.cpp:(.text+0x4268): more undefined references to std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()' follow

calewis · 2025-12-19T15:13:04Z

+    else()
+      message(STATUS "LTO/IPO is disabled")
+    endif()
+    message(STATUS "LTO/IPO is not supported: ${lto_error}")


This is going to print when lto is supported.

Oops. Fixed. Thanks!

Signed-off-by: Jaehyun Kim <jhkim@precisioninno.com>

github-actions · 2025-12-19T15:31:32Z

clang-tidy review says "All clean, LGTM! 👍"

jhkim-pii · 2025-12-22T13:54:42Z

@maliberty The LTO is also active on all the CPP tests which is noticeably slower to build as well (they probably don't need this on)

I tested this, but disabling LTO for non-openroad targets does not improve build time dramatically.
I think it is because various static libraries (lib*.a) still requires long build time to support LTO-on for openroad executable and building non-openroad executable with static libraries takes relatively smaller runtime.

Moreover, disabling LTO for non-openroad executables require non-trivial code changes, which incurs a maintenance issue (whenever a new test executable is created, the LTO-off configuration should be added for the executable).

IMO, disabling LTO for non-openroad executables has more cons than pros.

calewis · 2025-12-22T16:57:43Z

@maliberty The LTO is also active on all the CPP tests which is noticeably slower to build as well (they probably don't need this on)

I tested this, but disabling LTO for non-openroad targets does not improve build time dramatically. I think it is because various static libraries (lib*.a) still requires long build time to support LTO-on for openroad executable and building non-openroad executable with static libraries takes relatively smaller runtime.

Moreover, disabling LTO for non-openroad executables require non-trivial code changes, which incurs a maintenance issue (whenever a new test executable is created, the LTO-off configuration should be added for the executable).

IMO, disabling LTO for non-openroad executables has more cons than pros.

I brought it up before, but I don't see incremental release builds being slow as a big deal. Just my 2 cents. bazel especially makes it easy to build in fast or debug mode and only trigger a release build once you are done with your changes.

maliberty · 2025-12-22T21:19:40Z

Based on your measurements I think we should have release w/LTO, release w/o LTO (for developers), and debug.

jhkim-pii · 2025-12-23T10:14:14Z

I brought it up before, but I don't see incremental release builds being slow as a big deal. Just my 2 cents. bazel especially makes it easy to build in fast or debug mode and only trigger a release build once you are done with your changes.

The incremental build time difference is too big (Release w/o LTO: 7s vs. Release w/ LTO: 106s).
Currently, many developers may use Release build for debugging because it is the default build setting in etc/Build.sh.
Thus, I think enabling the LTO on-by-default in the Release build (CMake) can make many developers uncomfortable.

But no good idea to get performance gain w/ trivial build time loss.
So the current plan is

Enabling LTO on-by-default for Release build because performance gain for users is more important and
Provide a new option (e.g., etc/Build.sh -no-lto) for faster build.

@calewis BTW, LTO is also disabled by default in Bazel build setting. What are your thoughts on enabling LTO (full-lto or thin-lto) by default in Bazel? Is it ok?

QuantamHD · 2025-12-23T10:17:25Z

At Google the default bazel build is fastbuild which is like O2.

In order of devs doing things then it goes -c opt (O3), then --config=release (LTO). I would say it's an anti-pattern to enable LTO by default.

jhkim-pii · 2025-12-23T10:34:24Z

Thank you for sharing your thoughts!

maliberty · 2025-12-23T16:38:03Z

I expect general people building OR will just use whatever the default is. LTO seems like it gives them the best experience as they will likely build rarely and run often. I think working on closed-source software inside a professional organization is different.

Signed-off-by: Jaehyun Kim <jhkim@precisioninno.com>

github-actions · 2026-01-09T06:43:18Z

clang-tidy review says "All clean, LGTM! 👍"

jhkim-pii · 2026-01-09T07:21:36Z

Changed the default build settings as follows. It is to provide the best performance to the users.

Bazel - default build option changes

Use -c opt by default
Use -O3 instead of -O2 (previous default)
Enable Full-LTO by default

Note that clang is the default compiler in Bazel setting.

CMake - default build option changes

Enabled LTO by default

Note that gcc/g++ is the default compiler in CMake and RELEASE with -O3 has been enabled by default already.

Bazel test data

CMake test data

Comment

In my local env, Bazel clean build is slow. I need to debug it.
In the test, incremental build of Bazel w/ clang is faster than CMake incremental build w/ GNU compiler.

calewis · 2026-01-09T16:02:44Z

Changed the default build settings as follows. It is to provide the best performance to the users.

Bazel - default build option changes

Use -c opt by default

Use -O3 instead of -O2 (previous default)

Not that we are perfect now (an I personally don't think it's a hill worth dying on), but -O3 is more likely to cause differences between platforms and toolchains.

Signed-off-by: Jaehyun Kim <jhkim@precisioninno.com>

jhkim-pii · 2026-01-10T04:50:35Z

Changed the default build settings as follows. It is to provide the best performance to the users.

Bazel - default build option changes

Use -c opt by default

Use -O3 instead of -O2 (previous default)

Not that we are perfect now (an I personally don't think it's a hill worth dying on), but -O3 is more likely to cause differences between platforms and toolchains.

I also thought about the possibility of a new issue occurring by -O3, but since CMake has been using -O3 as the default, so I made Bazel consistent as well.

@maliberty What do you think?

Signed-off-by: Jaehyun Kim <jhkim@precisioninno.com>

github-actions · 2026-01-10T05:44:36Z

clang-tidy review says "All clean, LGTM! 👍"

jhkim-pii · 2026-01-10T07:19:11Z

+    set(LTO_UNSUPPORTED_OS TRUE)
+    message(STATUS "Detected Rocky Linux 8: LTO will be disabled due to known build issues")
+  endif()
+endif()


Disabled LTO in CMake for Debian11 & Rocky Linux 8 to avoid the build issues.

maliberty · 2026-01-10T16:39:27Z

@oharboe FYI

maliberty · 2026-01-10T16:39:57Z

Let's see if -O3 is a problem in practice. We can always back it off if needed.

…D-Project-staging/secure-enable-lto-without-gui" This reverts commit d4fd149, reversing changes made to 6875b55.

Revert "Merge pull request #9069 from The-OpenROAD-Project-staging/se…

stefanottili · 2026-01-11T16:19:13Z

Could you comment on why this was reverted ?
thinlto worked on macOS

maliberty · 2026-01-11T16:20:23Z

Failed the ORFS CI completely. Once it is resolved we will reinstate it.

jhkim-pii added 2 commits December 17, 2025 12:09

Enabled LTO (Link Time Optimization) except for gui.

0d97381

- LTO-on causes a crash due to Qt used in gui library. Signed-off-by: Jaehyun Kim <jhkim@precisioninno.com>

Disabled LTO in DEBUG mode

cf965b7

Signed-off-by: Jaehyun Kim <jhkim@precisioninno.com>

openroad-ci assigned jhkim-pii Dec 17, 2025

maliberty mentioned this pull request Dec 17, 2025

LTO-off blocks inlining of dbNetwork::dbToSta() and dbNetwork::staToDb() #9050

Closed

calewis reviewed Dec 18, 2025

View reviewed changes

Enabled LTO for Release only. Disabled LTO for RelWithDebInfo and Deb…

31a5f32

…ug build types Signed-off-by: Jaehyun Kim <jhkim@precisioninno.com>

calewis reviewed Dec 19, 2025

View reviewed changes

Added a missing 'else()'

119b5f8

Signed-off-by: Jaehyun Kim <jhkim@precisioninno.com>

jhkim-pii added 2 commits January 8, 2026 15:43

Enabled -O3 and Full-LTO for the default Bazel build

a0c1449

Signed-off-by: Jaehyun Kim <jhkim@precisioninno.com>

Fixed buildifier issue

7b56bd1

Signed-off-by: Jaehyun Kim <jhkim@precisioninno.com>

build: Disabled LTO for Debian11 and Rocky linux8

f07120d

Signed-off-by: Jaehyun Kim <jhkim@precisioninno.com>

build: Removed a debug message

54a64fc

Signed-off-by: Jaehyun Kim <jhkim@precisioninno.com>

jhkim-pii reviewed Jan 10, 2026

View reviewed changes

jhkim-pii requested a review from maliberty January 10, 2026 07:20

maliberty approved these changes Jan 10, 2026

View reviewed changes

maliberty merged commit d4fd149 into The-OpenROAD-Project:master Jan 10, 2026
13 checks passed

maliberty deleted the secure-enable-lto-without-gui branch January 10, 2026 16:39

maliberty added a commit that referenced this pull request Jan 11, 2026

Merge pull request #9232 from The-OpenROAD-Project-staging/revert

c21cd5f

Revert "Merge pull request #9069 from The-OpenROAD-Project-staging/se…

openroad-ci mentioned this pull request Jan 14, 2026

Enabled LTO (Link Time Optimization) #9254

Merged

openroad-ci mentioned this pull request Jan 23, 2026

build: Enable LTO on-by-default for CMake and LTO off-by-default for Bazel #9326

Merged

Conversation

openroad-ci commented Dec 17, 2025

Uh oh!

github-actions Bot commented Dec 17, 2025

Uh oh!

maliberty commented Dec 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jhkim-pii commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maliberty commented Dec 18, 2025

Uh oh!

vvbandeira commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gadfort commented Dec 18, 2025

Uh oh!

calewis commented Dec 18, 2025

Uh oh!

calewis Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jhkim-pii Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

calewis commented Dec 18, 2025

Uh oh!

maliberty commented Dec 18, 2025

Uh oh!

calewis commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maliberty commented Dec 18, 2025

Uh oh!

github-actions Bot commented Dec 19, 2025

Uh oh!

jhkim-pii commented Dec 19, 2025

Uh oh!

jhkim-pii commented Dec 19, 2025

Debian11

Rocky-linux8

Uh oh!

calewis Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

jhkim-pii Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Dec 19, 2025

Uh oh!

jhkim-pii commented Dec 22, 2025

Uh oh!

calewis commented Dec 22, 2025

Uh oh!

maliberty commented Dec 22, 2025

Uh oh!

jhkim-pii commented Dec 23, 2025

Uh oh!

QuantamHD commented Dec 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jhkim-pii commented Dec 23, 2025

Uh oh!

maliberty commented Dec 23, 2025

Uh oh!

github-actions Bot commented Jan 9, 2026

Uh oh!

jhkim-pii commented Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bazel - default build option changes

CMake - default build option changes

Bazel test data

CMake test data

Comment

Uh oh!

calewis commented Jan 9, 2026

Bazel - default build option changes

Uh oh!

maliberty commented Dec 17, 2025 •

edited

Loading

jhkim-pii commented Dec 18, 2025 •

edited

Loading

vvbandeira commented Dec 18, 2025 •

edited

Loading

calewis Dec 18, 2025 •

edited

Loading

calewis commented Dec 18, 2025 •

edited

Loading

QuantamHD commented Dec 23, 2025 •

edited

Loading

jhkim-pii commented Jan 9, 2026 •

edited

Loading