[None][chore] memory observability probes for DSv4 mem-opts validation#14308
Conversation
f43cb31 to
88b8b00
Compare
Adds a small leaf helper ``tensorrt_llm/_torch/pyexecutor/trace_log_utils.py``
that any pyexecutor / module file can import without circular dependencies.
It exposes two probes, both no-op unless ``TLLM_LOG_MEM_PROFILE=1``:
1. ``log_mem_snapshot(tag)`` — prints the canonical GPU memory fields
(torch_alloc / torch_reserved + their peaks / free / total). Used at
four coarse warmup boundaries in ``model_engine.py``:
warmup/before_warmup
warmup/after_autotuner
warmup/after_cuda_graph_capture
warmup/after_memory_pool_prepop
2. ``log_tensor_size(tag, tensor, **extra)`` — one-line single-tensor
footprint (shape / dtype / bytes). Wired into
``attention_backend/interface.py`` as ``[mem-profile/rope/new_table]``,
firing once per unique ``RopeParams`` cache miss to report the cos/sin
table size. For models using YaRN scaling at default HF
``max_position_embeddings`` this can be multiple GiB which is
otherwise invisible.
Default OFF is byte-identical to upstream (zero overhead, no log volume).
Single env var switches everything on.
Signed-off-by: Lance Liao <108499334+lancelly@users.noreply.github.com>
88b8b00 to
52096ff
Compare
|
/bot run --disable-fail-fast |
|
PR_Github #49389 [ run ] triggered by Bot. Commit: |
pengbowang-nv
left a comment
pengbowang-nv
left a comment
There was a problem hiding this comment.
Attention change LGTM
…interface.py Signed-off-by: Lance Liao <108499334+lancelly@users.noreply.github.com>
|
PR_Github #49389 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #49533 [ run ] triggered by Bot. Commit: |
|
PR_Github #49533 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #49630 [ run ] triggered by Bot. Commit: |
|
PR_Github #49630 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #49724 [ run ] triggered by Bot. Commit: |
|
PR_Github #49724 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #49807 [ run ] triggered by Bot. Commit: |
|
PR_Github #49807 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #49916 [ run ] triggered by Bot. Commit: |
|
PR_Github #49916 [ run ] completed with state |
NVIDIA#14308) Signed-off-by: Lance Liao <108499334+lancelly@users.noreply.github.com>
NVIDIA#14308) Signed-off-by: Lance Liao <108499334+lancelly@users.noreply.github.com>
NVIDIA#14308) Signed-off-by: Lance Liao <108499334+lancelly@users.noreply.github.com>
NVIDIA#14308) Source-Commit: 2de7acb Signed-off-by: Jiagan Cheng <jiaganc@nvidia.com>
NVIDIA#14308) Source-Commit: 2de7acb Signed-off-by: Jiagan Cheng <jiaganc@nvidia.com>
4 participants
Add some memory profile log.
Moe related workspace and activation tensor should be traced in a seperate PR, as well as attention active tensor.