Docs update (#47)

Address #43 and
#45. Thank you to @xinyu-dev
for the raises!

---------

Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Abhibha Gupta <abhibhag@nvidia.com>

Author: bxyu-nvidia

CONTRIBUTING.md

@@ -1,5 +1,34 @@
 # Contributing To NeMo-Gym
 
+## Quality control
+A checklist for all verifier data to be submitted to Nemo Gym. Please follow this pipeline before submitting a merge request.
+
+1. Necessary information to be included in the merge request:
+   1. Corresponding dataset on the spreadsheet.
+   2. Description of the prompt. What is the source of the prompt, which domain is it covering?
+   3. Description of the environment, if there is any.
+   4. Description of the verifier. How is it verified and whether we have checked the correctness of the verifier.
+   5. Legal approval status? If synthetically generated by ourselves with open models, please note there so that we know we don’t need legal approval. 
+2. Simple correctness check: After finishing implementing your own resources_servers (and/or your own customized code for more complicated tasks), please follow the guideline here to run the server, query OpenAI gpt-4o model (or any other model you like) and get 5 example rollouts and corresponding rewards there. Please include in your PR merge request:
+   1. The command you used to run the server for the uploaded data
+   2. The resulting rollout and judges (include 5 examples here for people to understand better the data samples, and to ensure reward here is correct.)
+   3. Other additional notes for running the server properly with the new PR.
+3. Test: Please follow the guideline here to implement your own test and run test for your environment. Tests are strongly encouraged and you must have at least one test for every server you make. Test coverage is not explicitly required which means that YOU ARE RESPONSIBLE FOR YOUR OWN SERVER CORRECTNESS AND FUNCTION.
+4. Reward Profiling: Please run inference on your prompts and environments (a ~500 small subset is OK) on two models:
+   1. Qwen 3 30B A3B
+   2. Qwen 3 235B Instruct (if that’s for agent / agentic coding / instruction following / game environments) or Qwen 3 235B Thinking (if math / competition coding)
+   3. Generate 16 responses for each prompt, and report the reward distribution (percent of all correct, all incorrect, and mixture of correct & incorrect prompts there).
+   4. [If using tool calling] Please also provide metrics around the number of tool calls issued on average per prompt in the environment, and the correlation of the reward with the number of tool calls. 
+5. [After Nemo Gym + Nemo RL integration is done] Training-based correctness check: Please train on the following models with GRPO and include both training accuracy curve and test benchmark accuracy curve:
+   1. Qwen 30B A3B Instruct
+   2. [With more compute available] Qwen 235B Instruct
+6. [PR Check and Review] Please assign another person in your team for reproducing and reviewing the PRs once it’s ready. The person for review needs to
+   1. Verify the content for all the above 1-5 steps 
+   2. Check the correctness of the 5 examples
+   3. Re-run the procedure provided in README to ensure one can generate the same dataset
+   4. After the person confirms reproduction and gives greenlight on the PR, please ping @banghuaz-nvidia @bxyu-nvidia.
+
+
 ## Signing Your Work
 
 * We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license.

README.md

@@ -578,6 +578,12 @@ ng_collect_rollouts +agent_name=multineedle_simple_agent \
     +num_samples_in_parallel=null
 ```
 
+The supported parameters include:
+- `limit`: Limits how many examples from the input JSONL file to process
+- `num_repeats`: Repeats each input example multiple times to collect multiple rollouts per example
+- `num_samples_in_parallel`: Controls how many rollout collection requests run concurrently
+
+
 View the rollouts just collected!
 ```
 ng_viewer +jsonl_fpath=results/multineedle_rollout_collection.jsonl

resources_servers/simple_weather/create_examples.py

@@ -14,7 +14,7 @@
 import json
 
 from nemo_gym.openai_utils import (
-    NeMoGymEasyInputMessageParam,
+    NeMoGymEasyInputMessage,
     NeMoGymResponseCreateParamsNonStreaming,
 )
 
@@ -57,9 +57,7 @@
 example_strs = []
 for query in queries:
     example = base_response_create_params.model_copy(
-        update={
-            "input": base_response_create_params.input + [NeMoGymEasyInputMessageParam(role="user", content=query)]
-        }
+        update={"input": base_response_create_params.input + [NeMoGymEasyInputMessage(role="user", content=query)]}
     )
     example_strs.append(json.dumps({"responses_create_params": example.model_dump(exclude_unset=True)}) + "\n")