Any plans to create official train/dev/test splits

[budhadityadeb]

Hi,
I am aware that the dataset has been intended for zero shot learning of unseen tasks.
However, it might make sense to create official train/dev/test splits for the datasets, as the usage might expand beyond zero shot.
I would probably create these splits anyway, but its better to get these from the repo itself.

Thanks
Budha

[danyaljj]

Hi there —

Thanks for bringing up this important issue.
We're definitely thinking about having official train/dev/test splits for the tasks AND instances (CC @swarooprm ).
This is something that we're hoping to address at some point when we have a beta version of the dataset released.
If you have any other thoughts or suggestions, I am all ears!

Daniel

[budhadityadeb]

Hi Daniel,
Thanks for your reply. As I said I will probably create the splits for my own work, and will let you know if I find anything specific that might be important here. For example if there is any need for special k-way splits for few shot scenarios.
Budha