On `./test/regression` failure it mentions `test/results/diffs` but no such directory exists

[mithro]

Describe the bug

I ran the regression tests and one failed. The output said I should See /ssd/github/The-OpenROAD-Project/OpenROAD/test/results/diffs for differences but this directory doesn't seem to exist.

See the following output;

tansell@tansell-glapstation:~/github/The-OpenROAD-Project/OpenROAD$ ./test/regression flow
------------------------------------------------------
Flow
gcd_nangate45 (tcl) pass
gcd_sky130hd (tcl) pass
gcd_sky130hs (tcl) pass
ibex_sky130hd (tcl) pass
ibex_sky130hs (tcl) pass
aes_nangate45 (tcl) pass
aes_sky130hd (tcl) pass
aes_sky130hs (tcl) pass
tinyRocket_nangate45 (tcl)
 pass
jpeg_sky130hs (tcl) *FAIL* RSZ::max_slew_slack -19% <   0% ; DRT::max_slew_slack -17% < -14%
jpeg_sky130hd (tcl) *FAIL* DRT::max_slew_slack  -1% <  -0%
------------------------------------------------------
Failed 2/11
See /ssd/github/The-OpenROAD-Project/OpenROAD/test/results/diffs for differences
See /ssd/github/The-OpenROAD-Project/OpenROAD/test/results for log files
------------------------------------------------------
Found 1 tool regression failures.

/ssd/github/The-OpenROAD-Project/OpenROAD/test/results does exist.

Expected Behavior

The directory /ssd/github/The-OpenROAD-Project/OpenROAD/test/results/diffs exists?

Environment

Git commit: 15d5c68f995104f38f6a3c8b50932e4c6d44e54b
kernel: Linux 6.3.11-1rodete2-amd64                                                           
os: Debian GNU/Linux rodete                                                                   
cmake version 3.24.2                                                                          
-- The CXX compiler identification is GNU 12.2.0                      
-- Detecting CXX compiler ABI info                                                            
-- Detecting CXX compiler ABI info - done                                                     
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features                                                             
-- Detecting CXX compile features - done                                                      
-- OpenROAD version: v2.0-10417-g15d5c68f9                                                    
-- System name: Linux                                                                         
-- Compiler: GNU 12.2.0                                                                                                                                                                     
-- Build type: RELEASE
-- Install prefix: /usr/local                                                                                                                                                               
-- C++ Standard: 17                                                                           
-- C++ Standard Required: ON
-- C++ Extensions: OFF                                                                        
-- The C compiler identification is GNU 12.2.0                                                
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done                                                       
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Found Python: /usr/bin/python3.11 (found version "3.11.5") found components: Interpreter 
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success 
-- Found Threads: TRUE                                                                        
-- Performing Test C_COMPILER_SUPPORTS__-Wall
-- Performing Test C_COMPILER_SUPPORTS__-Wall - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wall                             
-- Performing Test CXX_COMPILER_SUPPORTS__-Wall - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-array-bounds
-- Performing Test C_COMPILER_SUPPORTS__-Wno-array-bounds - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-array-bounds                              
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-array-bounds - Success                     
-- Performing Test C_COMPILER_SUPPORTS__-Wno-nonnull                                      
-- Performing Test C_COMPILER_SUPPORTS__-Wno-nonnull - Success                                                                                                                              
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-nonnull                                       
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-nonnull - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-maybe-uninitialized
-- Performing Test C_COMPILER_SUPPORTS__-Wno-maybe-uninitialized - Success                                                                                                                  
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-maybe-uninitialized
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-maybe-uninitialized - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-format-overflow          
-- Performing Test C_COMPILER_SUPPORTS__-Wno-format-overflow - Success                                                                                                                      
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-format-overflow
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-format-overflow - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-variable                                                                                                                                
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-variable - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-variable
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-variable - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-function
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-function - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-write-strings
-- Performing Test C_COMPILER_SUPPORTS__-Wno-write-strings - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-write-strings
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-write-strings - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-sign-compare
-- Performing Test C_COMPILER_SUPPORTS__-Wno-sign-compare - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-sign-compare
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-sign-compare - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-deprecated
-- Performing Test C_COMPILER_SUPPORTS__-Wno-deprecated - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-deprecated
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-deprecated - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-c++11-narrowing
-- Performing Test C_COMPILER_SUPPORTS__-Wno-c++11-narrowing - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-c++11-narrowing
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-c++11-narrowing - Failed
-- Performing Test C_COMPILER_SUPPORTS__-Wno-register
-- Performing Test C_COMPILER_SUPPORTS__-Wno-register - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-register
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-register - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-format
-- Performing Test C_COMPILER_SUPPORTS__-Wno-format - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-format
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-format - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-reserved-user-defined-literal
-- Performing Test C_COMPILER_SUPPORTS__-Wno-reserved-user-defined-literal - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-reserved-user-defined-literal
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-reserved-user-defined-literal - Failed
-- Performing Test C_COMPILER_SUPPORTS__-fpermissive
-- Performing Test C_COMPILER_SUPPORTS__-fpermissive - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__-fpermissive
-- Performing Test CXX_COMPILER_SUPPORTS__-fpermissive - Success
-- Performing Test C_COMPILER_SUPPORTS__-x
-- Performing Test C_COMPILER_SUPPORTS__-x - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__-x
-- Performing Test CXX_COMPILER_SUPPORTS__-x - Failed
-- Performing Test C_COMPILER_SUPPORTS__c++
-- Performing Test C_COMPILER_SUPPORTS__c++ - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__c++
-- Performing Test CXX_COMPILER_SUPPORTS__c++ - Failed
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-but-set-variable
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-but-set-variable - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-but-set-variable
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-but-set-variable - Success
-- TCL library: /usr/lib/x86_64-linux-gnu/libtcl.so
-- TCL header: /usr/include/tcl/tcl.h
-- TCL readline library: /usr/lib/x86_64-linux-gnu/libtclreadline.so
-- TCL readline header: /usr/include/x86_64-linux-gnu
-- Found SWIG: /usr/bin/swig4.0 (found suitable version "4.1.0", minimum required is "3.0")  
-- Using SWIG >= 4.1.0 -flatstaticmethod flag for python
-- Found Boost: /home/tansell/.local/lib/cmake/Boost-1.80.0/BoostConfig.cmake (found version "1.80.0")  
-- boost: 1.80.0
-- Found Python3: /usr/include/python3.11 (found version "3.11.5") found components: Development Development.Module Development.Embed 
-- Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.so (found version "1.2.13") 
-- spdlog: 1.8.1
-- Found BISON: /usr/bin/bison (found version "3.8.2") 
-- Found Doxygen: /usr/bin/doxygen (found version "1.9.4") found components: doxygen dot 
-- STA version: 2.4.0
-- STA git sha: 68600a2921a47679e1c3a159bdb852df191a0a62
-- System name: Linux
-- Compiler: GNU 12.2.0
-- Build type: RELEASE
-- Build CXX_FLAGS: -O3 -DNDEBUG
-- Install prefix: /usr/local
-- Found FLEX: /usr/bin/flex (found version "2.6.4") 
-- TCL library: /usr/lib/x86_64-linux-gnu/libtcl.so
-- TCL header: /usr/include/tcl/tcl.h
-- SSTA: 0
-- STA executable: /ssd/github/The-OpenROAD-Project/OpenROAD/src/sta/app/sta
-- GPU is not enabled
-- TCL library: /usr/lib/x86_64-linux-gnu/libtcl.so
-- TCL header: /usr/include/tcl/tcl.h
-- Found re2: /home/tansell/.local/lib/cmake/re2/re2Config.cmake (found version "9.0.0") 
-- Found Clp: /home/tansell/.local/lib/cmake/Clp/ClpConfig.cmake (found version "1.17.7") 
-- Found Cbc: /home/tansell/.local/lib/cmake/Cbc/CbcConfig.cmake (found version "2.10.7") 
-- Found Eigen3: /home/tansell/.local/share/eigen3/cmake/Eigen3Config.cmake (found version "3.4.0") 
-- Found SCIP: /home/tansell/.local/lib/cmake/scip/scip-config.cmake (found version "8.0.1") 
-- GUI is enabled
-- Charts widget is not enabled
-- Found Boost: /home/tansell/.local/lib/cmake/Boost-1.80.0/BoostConfig.cmake (found version "1.80.0") found components: serialization 
-- Found OpenMP_CXX: -fopenmp (found version "4.5") 
-- Found OpenMP: TRUE (found version "4.5")  
-- Could NOT find VTune (missing: VTune_LIBRARIES VTune_INCLUDE_DIRS) 
-- Found Boost: /home/tansell/.local/lib/cmake/Boost-1.80.0/BoostConfig.cmake (found suitable version "1.80.0", minimum required is "1.78")  
-- TCL library: /usr/lib/x86_64-linux-gnu/libtcl.so
-- TCL header: /usr/include/tcl/tcl.h
-- Found Boost: /home/tansell/.local/lib/cmake/Boost-1.80.0/BoostConfig.cmake (found version "1.80.0") found components: serialization system thread 
-- TCL readline enabled
-- Tcl Extended disabled
-- Python3 enabled
-- Configuring done
-- Generating done
-- Build files have been written to: /tmp/tmp.ljVSa4llSF

To Reproduce

Follow instructions at https://openroad.readthedocs.io/en/latest/main/README.html#regression-tests

Relevant log output

No response

Screenshots

No response

Additional Context

No response

[vvbandeira]

@mithro,
The regression gets fail/pass status by two main checks:

1. If their log output is the same
2. If the flow metrics are within a safety margin

If the logs differ, then the diff file is created.
If the metrics fail (in your case), the failing metrics are printed in the console, for example:

jpeg_sky130hs (tcl) *FAIL* RSZ::max_slew_slack -19% <   0% ; DRT::max_slew_slack -17% < -14%

The failing metrics are two:

• RSZ::max_slew_slack -19% < 0%
• DRT::max_slew_slack -17% < -14%

Currently, we do not keep track of different failure modes (1. vs 2., above).
So we print the message to see the diff file whenever there's an error.