Re-assigning the category labels

[danyaljj]

Starting this thread for any discussions related to the category labels.
Let's use this channel to discuss anything related to this annotation effort.

So far, we have @yeganehkordi and @ghlai9665 helping with this task.

• @ghlai9665 has already started relabeling the category labels (task 1-30 here: Further update task domains & categories; Add Utility Scripts #436) and hopefully, he will have the bandwidth for more (now that he's getting quite fluent in terms of category labels). 
• To make sure that there won't be any conflicts, I think @yeganehkordi can start from task-200 onwards. 
• @swarooprm is interested in revising some of the categories.

[garyhlai]

yup @yeganehkordi starting with task-200 onwards sounds good to me

[yeganehkordi]

Sounds good! Can we address the human evaluation feedbacks at the same time with re-assigning the category labels?

[danyaljj]

Sounds good! Can we address the human evaluation feedbacks at the same time with re-assigning the category labels?

I think that should be fine.

[yeganehkordi]

I mean the same person does the both tasks. I think it will increase our speed.

[danyaljj]

Yeah, that's alright.

[garyhlai]

Guidelines / FAQs:

• Massaging the task hierarchy categories is somewhat encouraged.
• If a subcategory (e.g. Question Answering -> Contextual Question Answering -> Extractive) is mentioned, don't mention its parent category (Question Answering -> Contextual Question Answering)
• src/auto_add_domain.py is helpful for adding domains for all tasks in any particular dataset

[danyaljj]

It might make sense to add these important points to docs/task_hierarchy.md.

[garyhlai]

ok I will add them

[yeganehkordi]

I think we need an environmental science domain. I couldn't find anything relating to the environment in the domains. Also, I guess we don't have a domain for animals.

[swarooprm]

Feel free to add.

[gkaramanolakis]

Hi, I can also help here. Shall I start from task 600 onwards?

[swarooprm]

Please start from 400 onwards.

[gkaramanolakis]

I just updated task categories and domains for tasks 400-452 in PR 475. I will continue from task 452 onwards soon.

[yeganehkordi]

Hi there,
I think it's better to consider different pull requests for the task-hierarchy.md file and merge it frequently so that everyone uses the last version. For example, while reviewing #474, I recognized that @ghlai9665 had added the "Commonsense -> Stories" category, which is appropriate for some tasks that I've picked. I had added "Stories" instead.

[yeganehkordi]

Hi,
Please make sure the task-hierarchy file is alphabetically sorted while adding new items.

[yeganehkordi]

@ghlai9665 @gkaramanolakis I wonder why "captions" is a sub-domain of the "Commonsense" domain. Shouldn't we take it out of commonsense scope?

[garyhlai]

good point. will fix.