What would you like to happen?
Beam has a codepath to append heap information into sdk status reports:
worker_status.py, enabled by the --experiments=enable_heap_dump option, see description
We can expand on the info we are currently collecting to allow more introspection into memory allocated by the process so that we can reason about whether memory is growing on the native heap vs Python allocations; we could also try to print glibc stats to provide insights into possible memory fragmentation, see: https://bloomberg.github.io/memray/memory.html#memory-can-be-fragmented for ideas.
Issue Priority
Priority: 2 (default / most feature requests should be filed as P2)
Issue Components
What would you like to happen?
Beam has a codepath to append heap information into sdk status reports:
worker_status.py, enabled by the --experiments=enable_heap_dump option, see description
We can expand on the info we are currently collecting to allow more introspection into memory allocated by the process so that we can reason about whether memory is growing on the native heap vs Python allocations; we could also try to print glibc stats to provide insights into possible memory fragmentation, see: https://bloomberg.github.io/memray/memory.html#memory-can-be-fragmented for ideas.
Issue Priority
Priority: 2 (default / most feature requests should be filed as P2)
Issue Components