Skip to content

Stream logs unified improvement - #6127

Merged
zhaoqizqwang merged 2 commits into
aws:master-nova-follow-upsfrom
amazeAmazing:stream-logs-unified-improvement
Jul 30, 2026
Merged

Stream logs unified improvement#6127
zhaoqizqwang merged 2 commits into
aws:master-nova-follow-upsfrom
amazeAmazing:stream-logs-unified-improvement

Conversation

@amazeAmazing

Copy link
Copy Markdown
Contributor

Issue #, if available:

Description of changes:

fix(stream_logs): unify log streaming for MTRL, evaluators, and HyperPod

Summary

  • MTRL stream_logs() now uses correct log group (/aws/sagemaker/Job/AgentRFT) and polls via Job API — previously hung forever
  • SMTJ _stream_logs_smtj() refactored to use LogStreamer + stream_log_loop — same timeout/feedback behavior as MTRL and evaluators
  • Evaluators get stream_logs() on BaseEvaluator (pipeline and HyperPod dispatch)
  • HyperPod _stream_logs_smhp() provides user feedback instead of silent failure
  • Notebook examples added for all evaluator types + MTRL trainer

What changed

File Change
common_utils/log_streamer.py New. LogStreamer (poll-once CW utility) + stream_log_loop() shared helper. Warns after ~30s of no events, exits after ~5min.
agent_rft_job.py Add stream_logs()
multi_turn_rl_trainer.py Override stream_logs() with correct log group
base_trainer.py Refactor _stream_logs_smtj() to use LogStreamer + stream_log_loop. Patch _stream_logs_smhp() with ClientError handling + feedback.
evaluate/base_evaluator.py Add _latest_execution (set by _start_execution and _submit_hyperpod_eval_job) + stream_logs() with HP/pipeline dispatch
Notebooks (5 files) Add # evaluator.stream_logs() / # job.stream_logs() examples

Testing

Unit tests (35):

  • test_log_streamer.py: LogStreamer stream/filter mode, dedup, error handling, stream_log_loop early-exit/terminal/feedback/retry/timeout
  • test_stream_logs.py: AgentRFTJob, MultiTurnRLTrainer, BaseEvaluator, ARN resolution

Integration tests (8, no new jobs launched):

  • test_stream_logs_trainer.py (us-west-2 / 729646638167): MTRL via AgentRFTJob.get(), serverful SMTJ, input validation
  • test_stream_logs_evaluator.py (us-west-2 / 729646638167): BenchMarkEvaluator, CustomScorerEvaluator, LLMAsJudgeEvaluator — each with real completed pipeline execution ARNs

Open questions

  • AgentRFTJob.stream_logs() exists for the attach-to-existing-job use case (AgentRFTJob.get("name").stream_logs()). SMTJ trainers (SFT/DPO/RLVR) have no equivalent attach-and-stream pattern — users must construct a trainer and stuff _latest_training_job. Should we add a similar wrapper or a standalone helper for SMTJ jobs?

How to verify

# MTRL (was: hung forever)
job = AgentRFTJob.get("my-job")
job.stream_logs()

# Evaluator (was: AttributeError)
execution = evaluator.evaluate()
evaluator.stream_logs()

# HyperPod (was: silent)
trainer.stream_logs()  # now shows "Waiting for logs..." feedback

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@amazeAmazing
amazeAmazing force-pushed the stream-logs-unified-improvement branch from 54c2b39 to 28af702 Compare July 29, 2026 22:06
@amazeAmazing
amazeAmazing marked this pull request as ready for review July 29, 2026 22:07
Fixes three stream_logs() bugs identified in bug bash testing:

1. MTRL trainer stream_logs() now uses the correct log group
   (/aws/sagemaker/Job/AgentRFT) and polls status via Job API instead
   of TrainingJob API — previously hung forever showing nothing.

2. Adds stream_logs() to BaseEvaluator and EvaluationPipelineExecution
   with support for pipeline, MTRL eval, and HyperPod backends.

3. Patches _stream_logs_smhp() to provide user feedback instead of
   silently swallowing ResourceNotFoundException and empty events.

Introduces LogStreamer utility (poll-once pattern) and stream_log_loop()
shared helper to eliminate code duplication across all callers.
… jobs

Integration tests for evaluator.stream_logs() against completed pipeline
executions in us-west-2/729646638167. Covers BenchMarkEvaluator,
CustomScorerEvaluator, and LLMAsJudgeEvaluator. No new jobs launched.
@amazeAmazing
amazeAmazing force-pushed the stream-logs-unified-improvement branch from 28af702 to 6e5a200 Compare July 29, 2026 22:08
@zhaoqizqwang
zhaoqizqwang merged commit c001081 into aws:master-nova-follow-ups Jul 30, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants