[GAD] Question regarding Reward Score fluctuations and downstream evaluation

[ZhangChi315]

Thank you so much for sharing your work on GAD! It’s an impressive project, and we’ve been very interested in exploring its implementation.

While reproducing the results, we encountered a few points where we’d love to get your expert guidance:

1. Observations on Reward Score Behavior In our experiments, we noticed that the reward scores exhibit quite significant fluctuations and tend to hover around 0. This seems a bit different from the typical upward trend and eventual stabilization we often see in other RL frameworks.

We were wondering if this behavior is expected for GAD?

From your experience, what are the best indicators or metrics to look for to determine if the model has reached sufficient convergence?

2. Downstream Benchmarks We are also curious if you have had the chance to test GAD on more common downstream evaluation sets (such as IFEval). If so, we would be very interested to hear your thoughts on those results!

Thank you again for your contribution to the community. We look forward to hearing from you whenever you have a moment.