Skip to content

fix(sampling): preserve remainder samples across shards - #9859

Open
RerankerGuo wants to merge 1 commit into
modelscope:mainfrom
RerankerGuo:fix/sampling-data-range-remainder
Open

fix(sampling): preserve remainder samples across shards#9859
RerankerGuo wants to merge 1 commit into
modelscope:mainfrom
RerankerGuo:fix/sampling-data-range-remainder

Conversation

@RerankerGuo

Copy link
Copy Markdown
Contributor

PR type

  • Bug Fix
  • New Feature
  • Document Updates
  • More Models or Datasets Support

PR information

SwiftSampling._get_dataset calculated one fixed shard size with integer
division. When the dataset size was not divisible by the number of
data_range shards, every shard used that truncated size and the remainder
samples were never processed.

For example, 10 samples split across 3 processes produced shard sizes
[3, 3, 3], silently dropping sample 9.

This PR distributes the remainder across the first shards. The resulting
shards remain contiguous and ordered, differ in size by at most one, and cover
each input exactly once. Datasets smaller than the shard count are also
handled, with empty shards only after all samples have been assigned.

The change is limited to dataset partitioning for data_range; sampling,
resume, and output behavior are unchanged.

Experiment results

Before the fix:

10 samples / 3 shards -> [3, 3, 3], sample 9 missing
FAILED (Runs=1, success=0, failures=1)

After the fix:

10 samples / 3 shards -> [4, 3, 3], complete ordered coverage
 2 samples / 3 shards -> [1, 1, 0], complete ordered coverage

.venv/bin/python tests/run.py --test_dir tests/sample --pattern test_sampling.py
SUCCESS (Runs=1, success=1)

.venv/bin/pre-commit run --all-files
All hooks passed

Distribute dataset remainders across data_range shards so parallel sampling covers every input exactly once, including datasets smaller than the shard count.

Test: .venv/bin/python tests/run.py --test_dir tests/sample --pattern test_sampling.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant