My run is stuck at fastroute stage for almost a week.

[sakundu]

Describe the bug

I am trying to run place and route of MemPool Group design on Nangate45 enablement using OpenROAD, but my run is stuck at global route stage for more than five days. I have added cts def, sdc, verilog, memory lef and lib files here.

Here is the log file of fastroute/global route.

Expected Behavior

I don't want it to be stuck in the same place for days. I expect global route to fail or return an error indicating insufficient routing resources (If insufficient routing resources are causing the FastRoute to get stuck). Faster turn around will allow me to adjust the utilization and other settings to successfully run place and route of MemPool Group on Nangate45 using OpenROAD.

Environment

[WARNING] Your current OpenROAD version is outdated.
It is recommened to pull the latest changes.
If problem persists, file a github issue with the re-producible test case.
kernel: Linux 4.18.0-408.el8.x86_64
os: CentOS Stream 8
cmake version 3.16.2
-- The CXX compiler identification is GNU 12.2.0
-- Check for working CXX compiler: /home/tool/gcc/gcc-12.2.0/bin/c++
-- Check for working CXX compiler: /home/tool/gcc/gcc-12.2.0/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- OpenROAD version: v2.0-7480-g8a4065b
-- System name: Linux
-- Compiler: GNU 12.2.0
-- Build type: RELEASE
-- Install prefix: /usr/local
-- C++ Standard: 17
-- C++ Standard Required: ON
-- C++ Extensions: OFF
-- Could NOT find GTest (missing: GTEST_LIBRARY GTEST_INCLUDE_DIR GTEST_MAIN_LIBRARY) 
-- The C compiler identification is GNU 12.2.0
-- Check for working C compiler: /home/tool/gcc/gcc-12.2.0/bin/gcc
-- Check for working C compiler: /home/tool/gcc/gcc-12.2.0/bin/gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Found Python: /usr/bin/python3.6 (found version "3.6.8") found components: Interpreter 
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
-- Performing Test C_COMPILER_SUPPORTS__-Wall
-- Performing Test C_COMPILER_SUPPORTS__-Wall - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wall
-- Performing Test CXX_COMPILER_SUPPORTS__-Wall - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-array-bounds
-- Performing Test C_COMPILER_SUPPORTS__-Wno-array-bounds - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-array-bounds
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-array-bounds - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-nonnull
-- Performing Test C_COMPILER_SUPPORTS__-Wno-nonnull - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-nonnull
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-nonnull - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-maybe-uninitialized
-- Performing Test C_COMPILER_SUPPORTS__-Wno-maybe-uninitialized - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-maybe-uninitialized
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-maybe-uninitialized - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-format-overflow
-- Performing Test C_COMPILER_SUPPORTS__-Wno-format-overflow - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-format-overflow
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-format-overflow - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-variable
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-variable - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-variable
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-variable - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-function
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-function - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-function
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-function - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-write-strings
-- Performing Test C_COMPILER_SUPPORTS__-Wno-write-strings - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-write-strings
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-write-strings - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-sign-compare
-- Performing Test C_COMPILER_SUPPORTS__-Wno-sign-compare - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-sign-compare
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-sign-compare - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-deprecated
-- Performing Test C_COMPILER_SUPPORTS__-Wno-deprecated - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-deprecated
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-deprecated - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-c++11-narrowing
-- Performing Test C_COMPILER_SUPPORTS__-Wno-c++11-narrowing - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-c++11-narrowing
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-c++11-narrowing - Failed
-- Performing Test C_COMPILER_SUPPORTS__-Wno-register
-- Performing Test C_COMPILER_SUPPORTS__-Wno-register - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-register
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-register - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-format
-- Performing Test C_COMPILER_SUPPORTS__-Wno-format - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-format
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-format - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-reserved-user-defined-literal
-- Performing Test C_COMPILER_SUPPORTS__-Wno-reserved-user-defined-literal - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-reserved-user-defined-literal
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-reserved-user-defined-literal - Failed
-- Performing Test C_COMPILER_SUPPORTS__-fpermissive
-- Performing Test C_COMPILER_SUPPORTS__-fpermissive - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-fpermissive
-- Performing Test CXX_COMPILER_SUPPORTS__-fpermissive - Success
-- Performing Test C_COMPILER_SUPPORTS__-x
-- Performing Test C_COMPILER_SUPPORTS__-x - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__-x
-- Performing Test CXX_COMPILER_SUPPORTS__-x - Failed
-- Performing Test C_COMPILER_SUPPORTS__c++
-- Performing Test C_COMPILER_SUPPORTS__c++ - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__c++
-- Performing Test CXX_COMPILER_SUPPORTS__c++ - Failed
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-but-set-variable
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-but-set-variable - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-but-set-variable
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-but-set-variable - Success
-- TCL library: /usr/lib64/libtcl.so
-- TCL header: /usr/include/tcl.h
-- Found SWIG: /home/tool/swig/install/4.0.0/bin/swig (found suitable version "4.0.0", minimum required is "3.0") 
-- Found Boost: /home/tool/boost/boost_1_78_0/lib/cmake/Boost-1.78.0/BoostConfig.cmake (found version "1.78.0")  
-- boost: 1.78.0
-- Found Python3: /usr/lib64/libpython3.6m.so (found version "3.6") found components: Development 
-- Found ZLIB: /usr/lib64/libz.so (found version "1.2.11") 
-- spdlog: 1.8.1
-- Found BISON: /home/tool/bison/install/3.0.5/bin/bison (found version "3.0.5") 
-- Found Doxygen: /home/tool/doxygen/install/1.8.15/bin/doxygen (found version "1.8.15") found components: doxygen missing components: dot
-- STA version: 2.4.0
-- STA git sha: 882681a674e7089e5f02de4ba2050c6be32b6cd1
-- System name: Linux
-- Compiler: GNU 12.2.0
-- Build type: RELEASE
-- Build CXX_FLAGS: -O3 -DNDEBUG
-- Install prefix: /usr/local
-- Found FLEX: /home/tool/flex/2.6.4/bin/flex (found version "2.6.4") 
-- TCL library: /usr/lib64/libtcl.so
-- TCL header: /usr/include/tcl.h
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- SSTA: 0
-- STA executable: /home/fetzfs_projects/MacroPlacement/flow3/master/OpenROAD-flow-scripts/tools/OpenROAD/src/sta/app/sta
-- GPU is not enabled
-- Found re2: /home/tool/ortools/install/or-tools_cpp_CentOSStream-8-64bit_v9.4.1874/lib64/cmake/re2/re2Config.cmake (found version "9.0.0") 
-- Found Clp: /home/tool/ortools/install/or-tools_cpp_CentOSStream-8-64bit_v9.4.1874/lib64/cmake/Clp/ClpConfig.cmake (found version "1.17.7") 
-- Found Cbc: /home/tool/ortools/install/or-tools_cpp_CentOSStream-8-64bit_v9.4.1874/lib64/cmake/Cbc/CbcConfig.cmake (found version "2.10.7") 
-- Found Eigen3: /home/tool/ortools/install/or-tools_cpp_CentOSStream-8-64bit_v9.4.1874/share/eigen3/cmake/Eigen3Config.cmake (found version "3.4.0") 
-- Found SCIP: /home/tool/ortools/install/or-tools_cpp_CentOSStream-8-64bit_v9.4.1874/lib/cmake/scip/scip-config.cmake (found version "8.0.0") 
-- GUI is enabled
-- Found Boost: /home/tool/boost/boost_1_78_0/lib/cmake/Boost-1.78.0/BoostConfig.cmake (found version "1.78.0") found components: serialization 
-- Found OpenMP_CXX: -fopenmp (found version "4.5") 
-- Found OpenMP: TRUE (found version "4.5")  
-- Found VTune: /home/tool/intel/vtune_amplifier/lib64/libittnotify.a  
-- Found Boost: /home/tool/boost/boost_1_78_0/lib/cmake/Boost-1.78.0/BoostConfig.cmake (found suitable version "1.78.0", minimum required is "1.78")  
-- Found Boost: /home/tool/boost/boost_1_78_0/lib/cmake/Boost-1.78.0/BoostConfig.cmake (found version "1.78.0") found components: serialization system thread 
-- TCL readline disabled
-- Tcl Extended disabled
-- Python3 enabled
-- Configuring done
-- Generating done
-- Build files have been written to: /tmp/tmp.QNrR8Dlgqv

To Reproduce

I have uploaded the input def, sdc files here. I am using the default OpenROAD-flow-scripts to run global route.

Relevant log output

No response

Screenshots

No response

Additional Context

No response

[maliberty]

Which DEF are you using?

(neither sounds like a flow step in ORFS)

[maliberty]

Please give precise instructions on how to reproduce your issue.

[sakundu]

Hi @maliberty
My apologies for the confusion earlier. I mistakenly provided an incorrect link. You can find the verilog, def, sdc, and memory lef and lib files at this updated link. Here I converted the 4_cts.odb file into verilog and def formats. I have used the NG45 default settings to run fastroute/global route.

If necessary, I can provide the input floorplan def, verilog file, and sdc file for the ORFS flow. However, please note that completing the floorplan, placement, and cts will likely take around 2-3 days.

[sakundu]

Hi @maliberty,

I have uploaded all the necessary files here for running the MemPool Group on NG45 using ORFS.

To being, please download the mempool_group_orfs_config.tar.gz. Then follow the steps below to reproduce the above error:

tar -xvf mempool_gropu_orfs_config.tar.gz -C ./OpenROAD-flow-scripts/flow/designs/nangate45
cd ./OpenROAD-flow-scripts/flow
make route DESIGN_CONFIG=./designs/nangate45/mempool_group/config.mk FLOW_VARIANT=run1

Thanks,
Sayak

[maliberty]

I ran your design with -congestion_iterations 5 and looked at the congestion markers:

The congestion map doesn't show much in the way of low density areas to push the routes into:

Your design is very congested. I imagine that it is spending a lot of time trying fruitlessly to fix this. I suspect you'll need to adjust your density or macro placement to make this feasible.

You might experiment with the layer derates but be aware that may just move the problem to detailed routing.

[maliberty]

Side note - I assume this is the same design from https://arxiv.org/pdf/2012.02973.pdf where they said "Due to the size and the regularity of MemPool’s design, we used a hierarchical implementation flow". Is there a reason not to do the same here? Is this a stress test?

[sakundu]

In the MacroPlacement repository, we utilized the flat implementation of the MemPool Group. As part of Flow-3, we aim to implement the same using OpenROAD.

Thank you for the visualization and your feedback. I have already lowered the utilization to 40%. I will experiment with even smaller utilization values.

Thanks again,
Sayak

[maliberty]

Is there anything further or should we close this?

[vijayank88]

@sakundu
Are you able to complete the flow?