diff --git a/CLA.md b/CLA.md new file mode 100644 index 00000000..50565f60 --- /dev/null +++ b/CLA.md @@ -0,0 +1,89 @@ +# The Contributor License Agreement + +The [Cloud Native Computing Foundation](https://www.cncf.io) (CNCF) defines +the legal status of the contributed code in two different types of _Contributor License Agreements_ +(CLAs), [individual contributors](https://github.com/cncf/cla/blob/master/individual-cla.pdf) and [corporations](https://github.com/cncf/cla/blob/master/corporate-cla.pdf). + +FlagAI can only accept original source code from CLA signatories. + + +It is important to read and understand this legal agreement. + +## How do I sign? + +After creating your first Pull Request the linux-foundation-easycla bot will respond with information regarding your CLA status along with a link to sign the CLA. + +EasyCLA bot + +#### 1. Authorize EasyCLA to read some of your GitHub information + +GitHub EasyCLA Authorization + +Click on the "Please click here to be authorized" link to navigate to the GitHub Authorize Linux Foundation: EasyCLA page. Then click Authorize LF-Engineering to give the Linux Foundation read-only access to list the email addresses associated with your GitHub account. + +#### 2. Select from the two types of contributor + +EasyCLA + + +After authorizing EasyCLA, you will be redirected to a page to identify which type of contributor you are. +Select the most appropriate option: + * Individual Contributor: You are contributing as yourself, and not as part of another organization. + * Corporate Contributor: You are contributing on behalf of your employer or other organization. + +#### 3. Sign the CLA + +Once you select the type of contributor, proceed to Sign the CLA and follow the instructions to complete the signing process through DocuSign. + +**Ensure your GitHub e-mail address matches e-mail address used to sign CLA** + +After you have filled out the information, Click "Finish" and you will be redirected back to your Pull Request. + +#### 4. Look for an email indicating successful signup. + +> Hello, +> +> This is a notification email from EasyCLA regarding the project Cloud Native Computing > Foundation (CNCF). +> +> The CLA has now been signed. You can download the signed CLA as a PDF here. +> +> If you need help or have questions about EasyCLA, you can read the documentation or reach out to us for support. +> +> Thanks, +> EasyCLA Support Team + + + +#### 5. Validate your CLA + +Once you are redirected back to your GitHub Pull Request, reply with a comment `/easycla` to update the CLA status of your PR. + + +## Changing your Affiliation + +If you've changed employers and still contribute to Kubernetes, your affiliation +needs to be updated. The Cloud Native Computing Foundation uses [gitdm](https://github.com/cncf/gitdm) +to track who is contributing and from where. Create a pull request on the [gitdm](https://github.com/cncf/gitdm) +repository with a change to the corresponding developer affiliation text file. +Your entry should look similar to this: + +``` +Jorge O. Castro*: jorge!heptio.com, jorge!ubuntu.com, jorge.castro!gmail.com +Heptio +Canonical until 2017-03-31 +``` + +## Troubleshooting + +If you encounter any problems signing the CLA and need further assistance, log a ticket by clicking on the link [please submit a support request ticket](https://jira.linuxfoundation.org/plugins/servlet/theme/portal/4) from the EasyCLA bot's response. Someone from the CNCF will respond to your ticket to help. + +Should you have any issues using the LF Support Site, send a message to the +backup e-mail support address + +## Setting up the CNCF CLA check + +If you are a Kubernetes GitHub organization or repo owner and would like to setup +the Linux Foundation CNCF CLA check for your repositories, [read the docs on setting up the CNCF CLA check](/github-management/setting-up-cla-check.md) + + +[Linux Foundation Support Site]: https://support.linuxfoundation.org/ \ No newline at end of file diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 944c124e..476f4d92 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -8,6 +8,9 @@ side, please stick to the following process: 3. If we decide your concern needs code changes, we would be happy to accept a pull request. Please consider the commit guidelines below. +## Sign the CLA + +Before you can contribute to FlagAI, you will need to sign the [Contributor License Agreement](CLA.md). ## Git Commit Guidelines @@ -34,17 +37,13 @@ pip install -r requirements.txt ``` ### tests - - - -To run all basic tests execute: -```bash -python test.py +Install `pytest` for testing ``` - -To check the test results in +pip install pytest ``` -tests/test_report +To run all basic tests execute: +```bash +pytest ``` ### code formatting diff --git a/README.md b/README.md index 520d8fb6..c3172981 100644 --- a/README.md +++ b/README.md @@ -8,16 +8,16 @@ FlagAI (Fast LArge-scale General AI models) is an fast, easy-to-use and extensible toolkit for large-scale model. Our goal is to support training, fine-tuning, and deployment of large-scale models on various downstream tasks with multi-modality. Currently, we are focusing on NLP models and tasks. In near futher, we will support for other modalities. -* Now it supports GLM, BERT, RoBERTa, GPT2, T5, and models from Huggingface Transformers. +* Now it supports **WuDao GLM** with a maximum of 10 billion parameters (see [Introduction to GLM](/docs/GLM.md)). It also supports **BERT**, **RoBERTa**, **GPT2**, **T5**, and models from Huggingface Transformers. -* It provides APIs to quickly download and use those pre-trained models on a given text, fine-tune them on your own datasets, and then share them with the community on our model hub. +* It provides APIs to quickly download and use those pre-trained models on a given text, fine-tune them on widely-used datasets collected from [SuperGLUE](https://super.gluebenchmark.com/) and [CLUE](https://github.com/CLUEbenchmark/CLUE) benchmarks, and then share them with the community on our model hub. It also provides [prompt-learning](/docs/TUTORIAL_7_PROMPT_LERANING.md) toolkit for few shot tasks. * These models can be applied to (Chinese/English) Text, for tasks like text classification, information extraction, question answering, summarization, and text generation. -* FlagAI is backed by the three most popular data/model parallel libraries — PyTorch/Deepspeed/Megatron-LM — with seamless integration between them. Users can parallel their training/testing process with less than ten lines of code. +* FlagAI is backed by the three most popular data/model parallel libraries — [PyTorch](https://pytorch.org/)/[Deepspeed](https://www.deepspeed.ai/)/[Megatron-LM](https://github.com/NVIDIA/Megatron-LM) — with seamless integration between them. Users can parallel their training/testing process with less than ten lines of code. -The code is partially based on [Transformers](https://github.com/huggingface/transformers) and [DeepSpeedExamples](https://github.com/microsoft/DeepSpeedExamples). +The code is partially based on [GLM](https://github.com/THUDM/GLM), [Transformers](https://github.com/huggingface/transformers) and [DeepSpeedExamples](https://github.com/microsoft/DeepSpeedExamples). @@ -114,13 +114,17 @@ for text in test_data: ``` ## Pretrained Models and examples -* [Poetry generation with GLM-large-ch](docs/TUTORIAL_9_GLM_EXAMPLE_PEOTRY_GENERATION.md) -* [Title Generation with RoBerta-WWM ](/docs/TUTORIAL_10_BERT_EXAMPLE_TITLE_GENERATION.md) -* [Semantic Matching with RoBerta-WWM](/docs/TUTORIAL_11_BERT_EXAMPLE_SEMANTIC_MATCHING.md) -* [NER with RoBerta-WWM](/docs/TUTORIAL_14_BERT_EXAMPLE_NER.md) -* [Writing with GPT-2](/docs/TUTORIAL_15_GPT2_WRITING.md) -* [Title generation with T5](/docs/TUTORIAL_16_T5_EXAMPLE_TITLE_GENERATION.md) -* [Supported tasks](/docs/AllSupportedTasks.md) + +* [Blank_Filling_QA with GLM ](/docs/TUTORIAL_11_GLM_BLANK_FILLING_QA.md) +* [Title Generation with GLM ](/docs/TUTORIAL_12_GLM_EXAMPLE_TITLE_GENERATION.md) +* [Poetry generation with GLM-large-ch](docs/TUTORIAL_13_GLM_EXAMPLE_PEOTRY_GENERATION.md) +* [Using huggingface's t5-11b & tricks ](docs/TUTORIAL_14_HUGGINGFACE_T5.md) +* [Title Generation with RoBerta-WWM](/docs/TUTORIAL_15_BERT_EXAMPLE_TITLE_GENERATION.md) +* [Semantic Matching with RoBerta-WWM](/docs/TUTORIAL_16_BERT_EXAMPLE_SEMANTIC_MATCHING.md) +* [NER with RoBerta-WWM](/docs/TUTORIAL_17_BERT_EXAMPLE_NER.md) +* [Writing with GPT-2](/docs/TUTORIAL_18_GPT2_WRITING.md) +* [Title generation with T5](/docs/TUTORIAL_19_T5_EXAMPLE_TITLE_GENERATION.md) +* [Supported tasks](/docs/TUTORIAL_20_SUPPORTED_TASKS.md) This session explains how the base NLP classes work, how you can load pre-trained models to tag your @@ -131,22 +135,16 @@ language models, sequence labeling models, and text classification models. Let u ## Tutorials We provide a set of quick tutorials to get you started with the library: - -* [Tutorial 1: Basics](docs/TUTORIAL_1_BASICS.md) -* [Tutorial 2: Project structure](docs/TUTORIAL_2_PROJECT_STRUCTURE.md) -* [Tutorial 3: Supported tokenizers](docs/TUTORIAL_3_TOKENIZER.md) -* [Tutorial 4: Supported datasets](docs/TUTORIAL_4_DATASET.md) -* [Tutorial 5: Supported models](https://model.baai.ac.cn/models) -* [Tutorial 6: Training a model](docs/TUTORIAL_8_TRAINING.md) -* [Tutorial 7: AutoLoader](docs/TUTORIAL_12_INSTRUCTIONS_FOR_AutoLoader.md) -* [Tutorial 8: Predictor](docs/TUTORIAL_13_INSTRUCTIONS_FOR_PREDICTOR.md) - -## Learn More About FlagAI -* [Datasets: supported datasets & PET integration.](docs/APPENDIX_TASK.md) -* [Setup enviroments for data/model parallel](docs/EnvironmentSetup.md) -* [Three types of generation](docs/Seq2seqMethod.md) -* [Using huggingface's t5-3b & tricks ](docs/Huggingface_t5.md) -* [Transform a model into Megatron-LM version](docs/ChangeToMegatron.md) +* [Tutorial 1: How to construct and use Tokenizer](/docs/TUTORIAL_1_TOKENIZER.md) +* [Tutorial 2: Dataset Preprocessing Pipeline](/docs/TUTORIAL_2_DATASET.md) +* [Tutorial 3: Major Function of Model Module](/docs/TUTORIAL_3_MODEL.md) +* [Tutorial 4: Customize trainer for model and data-parallel training](/docs/TUTORIAL_4_TRAINER.md) +* [Tutorial 5: Simplify model and tokenizer Initialization by Using Autoloader](/docs/TUTORIAL_5_INSTRUCTIONS_FOR_AutoLoader.md) +* [Tutorial 6: Use off-the-shelf inference Algorithms with Predictor](/docs/TUTORIAL_6_INSTRUCTIONS_FOR_PREDICTOR.md) +* [Tutorial 7: Use FlagAI prompt-learning tool-kit to improve performance on SuperGLUE](/docs/TUTORIAL_7_PROMPT_LERANING.md) +* [Tutorial 8: Setup environment for training models with multi-machine](/docs/TUTORIAL_8_ENVIRONMENT_SETUP.md) +* [Tutorial 9: Text generation with encoder/decoder/encoder-decoder models](/docs/TUTORIAL_9_SEQ2SEQ_METHOD.md) +* [Tutorial 10: How to transform a customized model into a megatron-LM-style parallel model](/docs/TUTORIAL_10_MEGATRON.md) ## Contributing diff --git a/README_zh.md b/README_zh.md index bbfe118c..af4f1fdd 100644 --- a/README_zh.md +++ b/README_zh.md @@ -5,19 +5,19 @@ -------------------------------------------------------------------------------- -FlagAI 是一个快速、易于使用和可扩展的大型模型工具包。 我们的目标是支持在多模态的各种下游任务上训练、微调和部署大规模模型。 目前，我们专注于 NLP 模型和任务。 在不久的将来，我们将支持其他模态。 +FlagAI 是一个快速、易于使用和可扩展的大模型工具包。 我们的目标是支持在多模态的各种下游任务上训练、微调和部署大规模模型。 目前，我们专注于 NLP 模型和任务。 在不久的将来，我们将支持其他模态。

-* 现在它支持 GLM、BERT、RoBERTa、GPT2、T5 模型和 Huggingface Transformers 的模型。 +* 现在它支持最高百亿参数的**WUDAO GLM**(详见[GLM介绍](/doc_zh/GLM.md))。它同时也支持**BERT**、**RoBERTa**、**GPT2**、**T5** 模型和 Huggingface Transformers 的模型。 -* 它提供 API 以快速下载并在给定（中/英文）文本上使用这些预训练模型，在您自己的数据集上对其进行微调，然后在我们的模型中心与社区共享它们。 +* 它提供 API 以快速下载并在给定（中/英文）文本上使用这些预训练模型，在您自己的数据集上对其进行微调(fine-tuning)或者应用[提示学习(prompt-tuning)](/doc_zh/TUTORIAL_7_PROMPT_LERANING.md)，然后在我们的模型中心与社区共享它们。 * 这些模型可以应用于文本，用于文本分类、信息提取、问答、摘要、文本生成等任务，尤其是中文。 -* FlagAI 由三个最流行的数据/模型并行库（PyTorch/Deepspeed/Megatron-LM）提供支持，它们之间实现了无缝集成。 你可以用不到十行代码来并行你的训练/测试过程。 +* FlagAI 由三个最流行的数据/模型并行库（[PyTorch](https://pytorch.org/)/[Deepspeed](https://www.deepspeed.ai/)/[Megatron-LM](https://github.com/NVIDIA/Megatron-LM)）提供支持，它们之间实现了无缝集成。 你可以用不到十行代码来并行你的训练/测试过程。 -本项目的部分代码基于[Transformers](https://github.com/huggingface/transformers) 和 [DeepSpeedExamples](https://github.com/microsoft/DeepSpeedExamples). +本项目的部分代码基于[GLM](https://github.com/THUDM/GLM),[Transformers](https://github.com/huggingface/transformers) 和 [DeepSpeedExamples](https://github.com/microsoft/DeepSpeedExamples). @@ -181,13 +181,16 @@ for text_pair in test_data: ``` # 预训练模型以及样例 -* [RoBERTa-base-ch用于标题生成](doc_zh/TUTORIAL_10_BERT_EXAMPLE_TITLE_GENERATION.md) -* [RoBERTa-base-ch用于语义相似度匹配](doc_zh/TUTORIAL_11_BERT_EXAMPLE_SEMANTIC_MATCHING.md) -* [GLM-large-ch用于诗歌生成](doc_zh/TUTORIAL_9_GLM_EXAMPLE_PEOTRY_GENERATION.md) -* [RoBERTa-base-ch用于命名实体识别](/docs/TUTORIAL_14_BERT_EXAMPLE_NER.md) -* [GPT-2用于文本续写](/docs/TUTORIAL_15_GPT2_WRITING.md) -* [T5用于标题生成](/docs/TUTORIAL_16_T5_EXAMPLE_TITLE_GENERATION.md) -* [所有支持的任务](docs/AllSupportedTasks.md) +* [GLM-large-ch用户完形填空问答](/doc_zh/TUTORIAL_11_GLM_BLANK_FILLING_QA.md) +* [GLM-large-ch用于诗歌生成](doc_zh/TUTORIAL_13_GLM_EXAMPLE_PEOTRY_GENERATION.md) +* [GLM-large-ch用于标题生成](doc_zh/TUTORIAL_12_GLM_EXAMPLE_TITLE_GENERATION.md) +* [对 huggingface t5-11b 模型的支持 以及加速的tricks](doc_zh/TUTORIAL_14_HUGGINGFACE_T5.md) +* [RoBERTa-base-ch用于标题生成](doc_zh/TUTORIAL_15_BERT_EXAMPLE_TITLE_GENERATION.md) +* [RoBERTa-base-ch用于语义相似度匹配](doc_zh/TUTORIAL_16_BERT_EXAMPLE_SEMANTIC_MATCHING.md) +* [RoBERTa-base-ch用于命名实体识别](/doc_zh/TUTORIAL_17_BERT_EXAMPLE_NER.md) +* [GPT-2用于文本续写](/doc_zh/TUTORIAL_18_GPT2_WRITING.md) +* [T5用于标题生成](/doc_zh/TUTORIAL_19_T5_EXAMPLE_TITLE_GENERATION.md) +* [所有支持的任务](doc_zh/TUTORIAL_20_SUPPORTED_TASKS.md) 本节解释了本项目中基础NLP类是如何工作的，如何加载预先训练的模型来标记您的文本，如何使用不同的词或文档嵌入来得到表示，以及如何训练自己的语言模型、序列标注模型和文本分类模型。 @@ -195,22 +198,17 @@ for text_pair in test_data: # 教程 我们提供了一组教程来帮助您快速上手使用本库： -* [教程 1: 基础知识](doc_zh/TUTORIAL_1_BASICS.md) -* [教程 2: 项目结构](doc_zh/TUTORIAL_2_PROJECT_STRUCTURE.md) -* [教程 3: 项目支持的分词器](doc_zh/TUTORIAL_3_TOKENIZER.md) -* [教程 4: 项目支持的数据集](doc_zh/TUTORIAL_4_DATASET.md) -* [教程 5: 项目支持的模型](https://model.baai.ac.cn/models) -* [教程 6: 训练一个模型](doc_zh/TUTORIAL_8_TRAINING.md) -* [教程 7: AutoLoader工具](doc_zh/TUTORIAL_12_INSTRUCTIONS_FOR_AutoLoader.md) -* [教程 8: Predictor工具](doc_zh/TUTORIAL_13_INSTRUCTIONS_FOR_PREDICTOR.md) - - -# 了解更多关于FlagAI -* [数据集：支持的数据集和 `PET` 集成](doc_zh/APPENDIX_TASK.md) -* [数据/模型并行的环境设置](doc_zh/EnvironmentSetup.md) -* [三种不同的生成方式](doc_zh/Seq2seqMethod.md) -* [对 huggingface t5-3b 模型的支持 以及加速的tricks](doc_zh/Huggingface_t5.md) -* [转化一个模型为Megatron-LM的模型并行版本](doc_zh/ChangeToMegatron.md) +* [Tutorial 1: 构建和应用分词器](/doc_zh/TUTORIAL_1_TOKENIZER.md) +* [Tutorial 2: 数据集预处理流程](/doc_zh/TUTORIAL_2_DATASET.md) +* [Tutorial 3: 模型的主要功能及相关结构](/doc_zh/TUTORIAL_3_MODEL.md) +* [Tutorial 4: 模型训练(支持并行化)](/doc_zh/TUTORIAL_4_TRAINER.md) +* [Tutorial 5: 使用AutoLoader工具快速构建模型](/doc_zh/TUTORIAL_5_INSTRUCTIONS_FOR_AutoLoader.md) +* [Tutorial 6: 使用Predictor工具进行预测](/doc_zh/TUTORIAL_6_INSTRUCTIONS_FOR_PREDICTOR.md) +* [Tutorial 7: FlagAI提示学习功能](/doc_zh/TUTORIAL_7_PROMPT_LERANING.md) +* [Tutorial 8: 数据/模型并行的环境设置](/doc_zh/TUTORIAL_8_ENVIRONMENT_SETUP.md) +* [Tutorial 9: 使用**编码器/解码器/编解码**器模型进行文本生成](/doc_zh/TUTORIAL_9_SEQ2SEQ_METHOD.md) +* [Tutorial 10: 转化一个模型为Megatron-LM的模型并行版本](/doc_zh/TUTORIAL_10_MEGATRON.md) + # 贡献代码 感谢您对贡献的兴趣！ 参与的方式有很多； 从我们的[贡献者指南](CONTRIBUTING.md) 开始，然后检查这些[未解决的问题](https://github.com/BAAI-WuDao/Sailing/issues)以执行特定任务。 diff --git a/doc_zh/Advanced_Usage.md b/doc_zh/Advanced_Usage.md new file mode 100644 index 00000000..785ac663 --- /dev/null +++ b/doc_zh/Advanced_Usage.md @@ -0,0 +1,123 @@ +## 高级用法 + +### 自定义模型 +如果想在FlagAI框架中自定义模型或改进模型，比如遵循以下几条规则： + +1. 继承BaseModel，BaseModel 支持以from_pretrain的方式加载参数，以init_from_json的方式构建不同参数的模型。 +2. 自定义模型的```__init__()```函数第一个参数必须为config，其中每一个key为``config.json``文件中的参数信息。 +3. 自定义模型中必须有``load_weights()``函数，输入为预训练参数地址，为自定义模型加载预训练参数。 +4. 自定义模型中``forward()``函数必须返回字典，其中必须包含``logits``，如果传入labels参数，则需要额外返回``loss``数据。 + +以GLM完成序列标注任务为例： + +```python +from flagai.model.base_model import BaseModel +from flagai.model.glm_model import GLMModel +import torch + +class GLMForSequenceClassification(BaseModel): + def __init__(self, config, hidden_dropout=0.1, pool_token='cls', **kwargs): + super().__init__(config, **kwargs) + self.config = config + self.pool_token = pool_token + self.model = GLMModel(config) + self.model.output_predict = False + self.num_class = config['class_num'] + # Multi-choice head. + hidden_size = self.model.hidden_size + self.pool_layer = torch.nn.Linear(hidden_size, hidden_size) + self.multichoice_dropout = torch.nn.Dropout(hidden_dropout) + self.multichoice_head = torch.nn.Linear(hidden_size, self.num_class) + + def forward(self, + input_ids=None, + position_ids=None, + attention_mask=None, + **kwargs): + num_choices = None + if len(input_ids.shape) == 3: + assert self.num_class == 1 + batch_size, num_choices = input_ids.shape[:2] + input_ids = input_ids.reshape(-1, input_ids.size(-1)) + attention_mask = attention_mask.reshape(-1, + *attention_mask.size()[2:]) + position_ids = position_ids.reshape(-1, *position_ids.size()[2:]) + model_out = self.model(input_ids, position_ids, attention_mask) + outputs, mems = model_out['logits'], model_out['hidden_states'] + if self.pool_token == 'start': + output = outputs[torch.arange(outputs.size(0), + dtype=attention_mask.dtype, + device=attention_mask.device), + attention_mask] + elif self.pool_token == 'pad': + output = outputs[torch.arange(outputs.size(0), + dtype=attention_mask.dtype, + device=attention_mask.device), + attention_mask - 1] + elif self.pool_token == 'cls': + output = outputs[:, 0] + else: + raise NotImplementedError + output = torch.tanh(self.pool_layer(output)) + multichoice_output = self.multichoice_dropout(output) + logits = self.multichoice_head(multichoice_output) + if num_choices is not None: + logits = logits.view(-1, num_choices) + if 'labels' not in kwargs: + return {'logits': logits, 'hidden_states': mems} + else: + labels = kwargs['labels'] + if logits.size(1) == 1: + logits = logits.squeeze(1) + loss = F.binary_cross_entropy_with_logits( + logits.contiguous().float(), labels.float()) + else: + loss = F.cross_entropy(logits.contiguous().float(), + labels.long()) + return {"loss": loss, 'logits': logits, 'hidden_states': mems} + + def compute_loss(self, + input_ids=None, + position_ids=None, + attention_mask=None, + labels=None, + **kwargs): + model_output = self.forward(input_ids=input_ids, + position_ids=position_ids, + attention_mask=attention_mask) + assert labels is not None, "labels must not None!" + logits = model_output['logits'] + loss = F.cross_entropy(logits.contiguous().float(), labels.long()) + return { + "loss": loss, + 'logits': model_output['logits'], + 'hidden_states': model_output['hidden_states'] + } + + def load_weights(self, checkpoint_path): + checkpoints = self.model.load_weights_glm(checkpoint_path) + self.load_state_dict(checkpoints, strict=False) +``` + +其中``__init__()``函数中除了config参数必填，其他参数针对不同任务灵活定义，例如如下代码中的 ``hidden_dropout`` 与 ``pool_token``，这些参数在使用``from_pretrain()``函数构建模型的时候，可以灵活进行传入，例如： + +```python +from flagai.model.glm_model import GLMForSequenceClassification +model_dir = "./state_dict/GLM_sequence_classification/" ## this dir is the position for model and vocab and config files. +model = GLMForSequenceClassification.from_pretrain(model_dir, + hidden_dropout=0.1, + pool_token="cls") +``` + +构建好自定义模型，保证预训练参数加载正确，便可以直接在框架中进行使用。 + + +### 选择合适的 Tokenizer + +自定义好模型之后，需要选择一个合适的Tokenizer配合模型训练，FlagAI中支持多种原生Tokenizer，例如： + +1. BertTokenizer: ```from flagai.data.tokenizer.bert_tokenizer import BertTokenizer``` BertTokenizer支持多种模型，包括中文模型，英文模型；Bert模型，RoBERTa模型，中文GPT2模型等等。 +2. GLMLargeChTokenizer: ```from flagai.data.tokenizer.glm_large_ch_tokenizer import GLMLargeChTokenizer``` GLMLargeChTokenizer 支持 ``GLM-large-ch`` 模型。 +3. GLMLargeEnTokenizer: ```from flagai.data.tokenizer.glm_large_en_tokenizer import GLMLargeEnTokenizer``` GLMLargeEnTokenizer 支持 ``GLM-large-en`` 模型。 +3. T5BPETokenizer: ```from flagai.data.tokenizer.t5_tokenizer import T5BPETokenizer``` T5BPETokenizer 支持英文T5模型：``T5-base-en``。 +4. T5PegasusTokenizer: ```from flagai.data.tokenizer.t5_pegasus_tokenizer import T5PegasusTokenizer``` T5PegasusTokenizer 支持中文T5模型：``T5-base-ch``。 \ No newline at end of file diff --git a/doc_zh/DATASET_EXAMPLE.md b/doc_zh/DATASET_EXAMPLE.md new file mode 100644 index 00000000..527c38ef --- /dev/null +++ b/doc_zh/DATASET_EXAMPLE.md @@ -0,0 +1,143 @@ +# Examples in datasets + +## SuperGLUE: BoolQ +| 键值 | 含义 | +|------------------------------|----------| +| passage | 背景信息 | +| question | 根据背景信息提出的是/否问题 | +| label | 取值范围：true/false, 分别代表答案为是/否 | +示例 +```json +{"question": "is barq's root beer a pepsi product", + "passage": "Barq's -- Barq's is an American soft drink. Its brand of root beer is notable for having caffeine. Barq's, created by Edward Barq and bottled since the turn of the 20th century, is owned by the Barq family but bottled by the Coca-Cola Company. It was known as Barq's Famous Olde Tyme Root Beer until 2012.", + "idx": 5, "label": false} +``` + +## SuperGLUE: CB +| 键值 | 含义 | +|------------------------------|----------| +| premise | 前提文本 | +| hypothesis | 假设文本 | +| label |取值范围：entailment/contradiction/neutral 分别代表前提与假设的关系为1.前提能推导出假设 2.前提与假设矛盾 3. 没有足够信息得到两者关联性
| +示例 +```json +{"premise": "Mary is a high school student.", + "hypothesis": "Mary is a student", + "label": "entailment", "idx": 10} +``` + +## SuperGLUE: Copa +| 键值 | 含义 | +|------------------------------|----------| +| premise | 前提文本 | +| choice1 | 选项1 | +| choice2 |选项2| +| question | 取值范围： cause/effect, 分别代表问题是1.两个选项里哪个是前提的原因2.两个选项里哪个是前提的结果 | +| label | 取值范围：0/1, 分别代表选项1为正确答案和选项2为正确答案 | +示例 +```json +{"premise": "My eyes became red and puffy.", + "choice1": "I was sobbing.", "choice2": "I was laughing.", + "question": "cause", "label": 0, "idx": 7} +``` + +## SuperGLUE: MultiRC +| 键值 | 含义 | +|------------------------------|----------| +| text | 背景信息 | +| questions | 包含了一系列针对背景信息提出的问题，其中每个问题下有五个回答，而每个回答后面有一个标签，标签为0代表此回答正确，标签为1代表此回答错误 | +示例 +```json +{"idx": 4, "version": 1.1, + "passage": {"text": "...The two companies numbered about 225 men, and were commanded by General John E. Ross, a veteran Indian fighter... ", + "questions": [{"question": "When the narrator arrived at the headquarters, approximately how many men were present?", + "answers": [{"text": "225 men", "idx": 178, "label": 1}, + {"text": "225", "idx": 179, "label": 1}, + {"text": "525 men", "idx": 180, "label": 0}, + {"text": "235", "idx": 181, "label": 0}, + {"text": "255 men", "idx": 182, "label": 0}], "idx": 41}]}} +``` + + +## SuperGLUE: RTE +| 键值 | 含义 | +|------------------------------|----------| +| premise | 前提文本 | +| hypothesis | 假设文本 | +| label |取值范围：entailment/not entailment 分别代表前提与假设的关系为1.前提能推导出假设 2.前提不能推导出假设| +示例 +```json +{"premise": "Security forces were on high alert after an election campaign in which more than 1,000 people, including seven election candidates, have been killed.", +"hypothesis": "Security forces were on high alert after a campaign marred by violence.", +"label": "entailment", "idx": 4} +``` + + +## SuperGLUE: Wic +| 键值 | 含义 | +|------------------------------|----------| +| sentence1 | 背景文本1 | +| sentence2 | 背景文本2 | +| word | 一个在sentence1和sentence2中都会出现的单词 | +| label | 取值范围：true/false, 分别代表选项1.word在sentence1和sentence2中有着相同的含义 2.在两段话里word的含义不同 | +示例 +```json +{"word": "class", "sentence1": "An emerging professional class.", +"sentence2": "Apologizing for losing your temper, even though you were badly provoked, showed real class.", + "idx": 0, "label": false, "start1": 25, "start2": 85, "end1": 30, "end2": 90, "version": 1.1} +``` + +## SuperGLUE: WSC +| 键值 | 含义 | +|------------------------------|----------------------------------------------------------------------------| +| text | 背景文本 | +| span2_text | 背景文本中出现过的某个代词 | +| span1_text | 背景文本中出现过的某个片段 | +| label | 取值范围：true/false, 分别代表了1.span1_text这个代词指向了span2_text中的内容 2.没有指向到span2_text。 | +示例 +```json +{"word": "class", "sentence1": "An emerging professional class.", +"sentence2": "Apologizing for losing your temper, even though you were badly provoked, showed real class.", + "idx": 0, "label": false, "start1": 25, "start2": 85, "end1": 30, "end2": 90, "version": 1.1} +``` + +## CLUE: AFQMC +| 键值 | 含义 | +|------------------------------|----------| +| sentence1 | 背景文本1 | +| sentence2 | 背景文本2 | +| label | 取值范围：0/1, 分别代表选项1.两句话的含义不同 2.两句话的含义相似 | +示例 +```json +{"sentence1": "双十一花呗提额在哪", "sentence2": "里可以提花呗额度", "label": "0"} +``` + +## CLUE: TNEWS +| 键值 | 含义 | +|------------------------------|----------| +| sentence | 背景文本 | +| label | 背景文本的标签，包含十五种类别 | +| label_des | 标签对应的真正含义 | +示例 +```json +{"label": "102", "label_des": "news_entertainment", "sentence": "江疏影甜甜圈自拍，迷之角度竟这么好看，美吸引一切事物"} +``` + + +## CLUE: CMRC2018 +| 键值 | 含义 | +|------------------------------|----------| +| context | 背景信息 | +| question | 根据背景信息提出来的问题，一段背景信息可以对应多个问题 | +| answers | 对于某个问题，answers包含了三个人工给出来的回答 | +示例 +``` +{'paragraphs': + [{'id': 'TRAIN_186', + 'context': '工商协进会报告...所谓的“傻钱”策略，其实就是买入并持有美国股票这样的普通组合。...', + 'qas':{'question': '消费者信心指数由什么机构发布？', 'id': 'TRAIN_186_QUERY_4', + 'answers': [{'text': "工商协进会", 'answer_start': 759}, + {'text': "工商协进会", 'answer_start': 759}, + {'text': "工商协进会", 'answer_start': 759}]}]}], + 'id': 'TRAIN_186', 'title': '范廷颂'} +``` \ No newline at end of file diff --git a/doc_zh/GLM.md b/doc_zh/GLM.md new file mode 100644 index 00000000..1458d6e6 --- /dev/null +++ b/doc_zh/GLM.md @@ -0,0 +1,60 @@ +# GLM 介绍 + +## 模型简介 + +目前，存在几种不同的预训练模型架构：仅实现编码器架构的自动编码模型（例如BERT），仅实现解码器的自回归模型（例如GPT），以及同时实现编码器和解码器的编码器-解码器模型（例如T5）。 + +[**GLM模型**](https://arxiv.org/abs/2103.10360)与这些模型略有不同。它采用了一种自回归的空白填充方法， 并且在NLP领域三种主要的任务(自然语言理解，无条件生成，有条件生成)上都取得了不错的结果。 + +| Framwork | NLU | Cond.Gen. | Uncond.Gen | +|-----------------|-----|-----------|------------| +| Augoregressive | - | - | ✅ | +| Autoencoding | ✅ | × | × | +| Encoder-Decoder | - | ✅ | - | +| GLM | ✅ | ✅ | ✅ | +GLM的主要功能包括： + +- 任务一：文本的一些区间会被屏蔽（参照自动编码的做法）。 这些区间将被随机重新排列，并以自动回归方式进行预测。屏蔽的区间覆盖原始文本的15%。 +- 任务二：与第一个任务类似，但是区间会覆盖原始文本的50%-100%。 +- 剩下GLM相对于BERT的改动 + - [Pre-LN](http://proceedings.mlr.press/v119/xiong20b.html) + - 2D 位置编码：每个token都有两个位置编码：句子中的全局位置和屏蔽区间内的局部位置。 + - 前馈网络被线性层取代 + +## GLM的表现 + + +### [SuperGLUE](https://super.gluebenchmark.com) +单模型单任务微调在`dev`集上的效果，更多结果在[这里](https://github.com/THUDM/GLM) + +| Model | COPA | WSC | RTE | WiC | CB | MultiRC | BoolQ | ReCoRD | +| ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | +| GLM-10b-ch | 98.0 | 95.2 | 93.1 | 75.7 | 98.7/98.2 | 88.1/63.3 | 88.7 | 94.4/94.0 | +| [RoBERTa-Large](https://github.com/pytorch/fairseq/tree/master/examples/roberta) | 94.0 | 91.3 | 86.6 | 75.6 | 98.2/- | 85.7/- | 86.9 |89.5/89.0| +| [DeBERTa-XXLarge-v2](https://github.com/microsoft/DeBERTa/tree/master/experiments/superglue) | 97.0 | - | 93.5 | - | - | 87.8/63.6 | 88.3 | 94.1/93.7 | + +### [CLUE](https://www.cluebenchmarks.com) +单模型单任务微调在CLUE数据集上的结果（测试还在进行中，这里列出了部分任务）。如果想要使用`GLM-10b-ch`请点[这里](https://model.baai.ac.cn/model-detail/100001)。 + +| 模型 | AFQMC | TNEWS1.0 | IFLYTEK | OCNLI_50K | CSL | CMRC2018 | CHID1.0 | C3 1.0 | +|:--------------:|:------:|:--------:|:-------:|:---------:|:------:|:--------:|:-------:|:------:| +| RoBERTa XLarge | 75.835 | 68.75 | 62.654 | 82.333 | 83.433 | 80.5 | 86.57 | 77.03 | +| GLM-10b-ch | 75.42 | 69.94 | 62.15 | 85 | 86.17 | 70 | 87.009 | 88.335 | + +## FlagAI支持的GLM预训练模型 +参考 [Tutorial 5: 使用AutoLoader工具快速构建模型](/doc_zh/TUTORIAL_5_INSTRUCTIONS_FOR_AutoLoader.md)。 + +## Step-by-step procedure of GLM +1) 如下图所示，原文包含6个token，两个区间被屏蔽：第一个区间包含第3个token，第二个区间包含第5个和第6个token。 + +
+ +2) 将输入分成两个部分： A 部分 (将遮挡区间遮盖掉后的文本)和B部分(被遮挡的区间). 注意所有被遮挡区间的顺序会被重新打乱。 + +
+ +3) GLM 的输入和输出，输入包含输入编码和两组位置编码：第一组是每个token的位置，其中每个遮挡区间中的token共享相同的位置 ID。第二组记录token在遮挡区间内的相对位置。 +
+ +4) 下图里的自注意力机制既通过遮挡文本实现了自编码， 也在预测遮挡区间内文本的过程里实现了自回归。 +
\ No newline at end of file diff --git a/doc_zh/ChangeToMegatron.md b/doc_zh/TUTORIAL_10_METATRON.md similarity index 99% rename from doc_zh/ChangeToMegatron.md rename to doc_zh/TUTORIAL_10_METATRON.md index d4c961d9..55c89bfa 100644 --- a/doc_zh/ChangeToMegatron.md +++ b/doc_zh/TUTORIAL_10_METATRON.md @@ -211,7 +211,7 @@ class RowParallelLinear(torch.nn.Module): ``` ## 2.将`self-Attention` 中的两个Linear layer分别转化为`column/rowParallel` 版本 代码位置：`flagai/model/layers/attentions_mpu.py` -### 2.1 将`x` proj 到 `q, k, v` 的`linear `层，转化为 `collumnParalleLinear` +### 2.1 将`x` proj 到 `q, k, v` 的`linear `层，转化为 `columnParalleLinear` ### 2.2 将输出的`dense`层转化为`rowparallelLinear` 如下： ```python diff --git a/doc_zh/TUTORIAL_11_GLM_BLANK_FILLING_QA.md b/doc_zh/TUTORIAL_11_GLM_BLANK_FILLING_QA.md new file mode 100644 index 00000000..d73f2629 --- /dev/null +++ b/doc_zh/TUTORIAL_11_GLM_BLANK_FILLING_QA.md @@ -0,0 +1,104 @@ + +# GLM 完形填空式生成 + +![img.png](./img/glm_blank_filling.png) +GLM 自然地处理可变长度的空白填充，这对于许多下游任务至关重要。 +基于自回归填空，GLM随机从输入文本中删除连续的标记范围， +遵循自动编码的思想，训练模型重建跨度，遵循自回归的想法预训练。GLM 学习预训练期间的自回归生成。 + +GLM 对下游任务进行微调，并将它们重新定义为空白填充生成。每个任务都与一个人为的完形填空问题相关联，模型预测答案 +到完形填空。 +例如，情感分类任务被重新表述为填充“[SENTENCE] 中的空白。 +"它真的 "。 "好"或"坏"的预测表明 +情绪是积极的还是消极的。 + +有了这样的方式，GLM 受益于预训练和训练之间的一致性 +微调，因为预训练和微调都涉及训练模型以生成给定上下文的文本。 + +为了让预训练更适合文本生成任务的方法，GLM 还研究 +多任务预训练设置，其中模型是联合的训练以重建蒙面跨度并生成更长的文本。 + + +在 GLM 中，共有三种 MASK 方法，分别对应三种预测格式。 +1. ```[MASK]```, 令牌级别。这种屏蔽方法只会屏蔽一个句子中的随机标记， + 在三种遮罩方法中，它的掩码部分最少。所以，生成的内容也有限。例如： ``[CLS]北京故宫是中国[MASK]非物质文化遗产。<|endoftext|><|startofpiece|>现存最大的古代宫殿建筑, 也是``. +2. ```[sMASK]```: 实体级。我们限制遮罩 + spans必须是完整的句子。多个跨度 + （句子）被抽样覆盖 15% + 原始令牌。该目标旨在 + seq2seq 任务，其预测通常是 + 完整的句子或段落。 + ```[sMASK]``` 预测结果比 ```[MASK]```更长. 例如: `` [CLS]人工智能是一个以计算机科学为基础,由计算机、数学、哲学等多学科交叉融合的交叉学科,[sMASK],具有非常巨大的前景。<|endoftext|><|startofpiece|>它涉及的信息量不仅非常巨大,而且也是人工智能发展的一个关键,其研究内容包括人的感觉、知觉和思维,以及如何理解各种现象,以及解释现象的本性和原因等,通过计算机来进行系统的分析推理、建立数学模型,并模拟人类意识。`` +3. ```[gMASK]```: 文档级别。我们对单个跨度进行采样 + 其长度是从原始长度的 50%–100% 上的均匀分布中采样的。 + 该目标旨在生成长文本。 For example: ``[CLS]问题:啤酒伤胃吗?回答:[gMASK]<|startofpiece|>谢邀。 我是啤酒爱好者,但是我不喝酒。 我以前也说过,喝酒伤身,啤酒伤胃,伤肠道。 现在我也知道了啤酒伤人的很多细节,我就不瞎几把的说,大家看图片就知道了。 其实啤酒伤身这个说法只是表面而已。 啤酒中含有少量的碳酸和酒精,碳酸和酒精是成酸性物质,而乙醇是脂溶性的,酒精在胃里能够被分解,生成乙醇和二氧化碳,在体内是水和二氧化碳,两种物质会迅速发生中和反应,结果导致人体出现头痛、呕吐、胸痛、浑身发热等现象,这就是所谓喝大了,喝多了。 啤酒的含糖量在15%左右,喝多了也是伤身的,啤酒含糖量较高的主要成分是水分,而水分的体积比酒精大,所以酒精进入人体,与水相遇,就会产生大量气体,二氧化碳、水、一氧化碳等刺激人体,造成人体大量出汗,使体内温度升高,`` + + +例如，我们用`GLM-large-ch`以自回归空白的形式完成问题任务——填充任务，如果想要使用`GLM-10b-ch`请点[这里](https://model.baai.ac.cn/model-detail/100001): +```python +import torch +from flagai.model.glm_model import GLMModel +from flagai.data.tokenizer import GLMLargeChTokenizer +from flagai.model.predictor.predictor import Predictor +if __name__ == "__main__": + """Main training program.""" + print('Generate Samples') + tokenizer = GLMLargeChTokenizer(vocab_path='./checkpoints/glm-large-ch/cog-pretrain.model', + add_block_symbols=True, + add_task_mask=True, + add_decoder_mask=False, + fix_command_token=False) + model = GLMModel.from_pretrain(model_name='glm-large-ch', only_download_config=False) + model.cuda(torch.cuda.current_device()) + predictor = Predictor(model, tokenizer) + # question-answering + text = '问题：啤酒伤胃吗？回答：[gMASK]' + output=predictor.predict_generate_randomsample(text) + print(text,'\n',output) +``` + +与 BERT 类似，GLM 可以将掩码标记预测为: +```python +import torch +from flagai.model.glm_model import GLMModel +from flagai.data.tokenizer import GLMLargeChTokenizer +from flagai.model.predictor.predictor import Predictor +if __name__ == "__main__": + """Main training program.""" + print('Generate Samples') + tokenizer = GLMLargeChTokenizer(vocab_path='./checkpoints/glm-large-ch/cog-pretrain.model', + add_block_symbols=True, + add_task_mask=True, + add_decoder_mask=False, + fix_command_token=False) + model = GLMModel.from_pretrain(model_name='glm-large-ch', only_download_config=False) + model.cuda(torch.cuda.current_device()) + predictor = Predictor(model, tokenizer) + # question-answering + text = '北京故宫是中国[MASK]非物质文化遗产。' + output=predictor.predict_generate_randomsample(text) + print(text,'\n',output) +``` +并将遮罩句子预测为: + +```python +import torch +from flagai.model.glm_model import GLMModel +from flagai.data.tokenizer import GLMLargeChTokenizer +from flagai.model.predictor.predictor import Predictor +if __name__ == "__main__": + """Main training program.""" + print('Generate Samples') + tokenizer = GLMLargeChTokenizer(vocab_path='./checkpoints/glm-large-ch/cog-pretrain.model', + add_block_symbols=True, + add_task_mask=True, + add_decoder_mask=False, + fix_command_token=False) + model = GLMModel.from_pretrain(model_name='glm-large-ch', only_download_config=False) + model.cuda(torch.cuda.current_device()) + predictor = Predictor(model, tokenizer) + # question-answering + text = '人工智能是一个以计算机科学为基础，由计算机、数学、哲学等多学科交叉融合的交叉学科，[sMASK]，具有非常巨大的前景。' + output=predictor.predict_generate_randomsample(text) + print(text,'\n',output) +``` \ No newline at end of file diff --git a/doc_zh/TUTORIAL_12_GLM_EXAMPLE_TITLE_GENERATION.md b/doc_zh/TUTORIAL_12_GLM_EXAMPLE_TITLE_GENERATION.md new file mode 100644 index 00000000..99d2ba82 --- /dev/null +++ b/doc_zh/TUTORIAL_12_GLM_EXAMPLE_TITLE_GENERATION.md @@ -0,0 +1,186 @@ +# GLM 例子：标题生成 + +## 背景 +标题生成任务需要输入一段文本，模型根据输入文本输出对应的标题。这里使用`GLM-large-ch`作为样例,如果想要使用`GLM-10b-ch`请点[这里](https://model.baai.ac.cn/model-detail/100001)。 + +![](./img/bert_title_generation_model.png) + +## 结果展示 +#### 输入 +``` +"本文总结了十个可穿戴产品的设计原则而这些原则同样也是笔者认为是这个行业最吸引人的地方1为人们解决重复性问题2从人开始而不是从机器开始3要引起注意但不要刻意4提升用户能力而不是取代人", +"2007年乔布斯向人们展示iPhone并宣称它将会改变世界还有人认为他在夸大其词然而在8年后以iPhone为代表的触屏智能手机已经席卷全球各个角落未来智能手机将会成为真正的个人电脑为人类发展做出更大的贡献", +"雅虎发布2014年第四季度财报并推出了免税方式剥离其持有的阿里巴巴集团15％股权的计划打算将这一价值约400亿美元的宝贵投资分配给股东截止发稿前雅虎股价上涨了大约7％至5145美元" +``` + +#### 输出 +``` +可 穿 戴 产 品 设 计 原 则 十 大 原 则 +乔 布 斯 宣 布 iphone 8 年 后 将 成 为 个 人 电 脑 +雅 虎 拟 剥 离 阿 里 巴 巴 15 ％ 股 权 +``` + +## 模型训练（train.py） + +运行前修改训练数据路径src_dir, tgt_dir, 模型路径model_dir。在命令行运行此命令： +```commandline +cd ./examples/glm_title_generation +python ./train.py +``` + +### 1.数据加载 +样例数据在 /examples/bert_title_generation/data/ + +1）定义加载过程 +```python +def read_file(): + src = [] + tgt = [] + + ## read data file to load src and tgt, for example: + ## src = ["article_1", "article_2", "article_3" ......] + ## tgt = ["title_1", "title_2", "title_3" ......] + ## no matter what data you use, you need to construct the right src and tgt. + with open(src_dir, 'r', encoding='utf-8') as f: + lines = f.readlines() + for line in lines: + src.append(line.strip('\n').lower()) + + with open(tgt_dir, 'r', encoding='utf-8') as f: + lines = f.readlines() + for line in lines: + tgt.append(line.strip('\n').lower()) + return src,tgt +``` + +2）定义数据集处理过程（Dataset）： +```python +class GLMSeq2seqDataset(Dataset): + + def __init__(self, sents_src, sents_tgt): + super(GLMSeq2seqDataset, self).__init__() + self.sents_src = sents_src + self.sents_tgt = sents_tgt + + def __getitem__(self, i): + source_text = self.sents_src[i] + target_text = self.sents_tgt[i] + data = tokenizer.encode_plus(source_text, target_text=target_text) + return data + + def __len__(self): + return len(self.sents_src) +``` + +3）定义数据迭代器（DataLoader）中的批处理函数（collate_fn），用于将一批（batch）数据填充（padding）成统一大小 +```python +class GLMSeq2seqDynamicCollateFN(): + def __init__(self, pad_id): + self.pad_id = pad_id + + def pad_token(self, tokens, max_length): + pad_len = max_length-len(tokens) + tokens += [self.pad_id]*pad_len + return tokens + + def pad_position_ids(self, position_ids, max_length): + pad_len = max_length-len(position_ids[0]) + position_ids[0] += [len(position_ids[0])+x for x in range(pad_len)] + position_ids[1] += [1] * pad_len + return position_ids + + def pad_loss_mask(self, loss_mask, max_length): + pad_len = max_length-len(loss_mask) + loss_mask += [0] * pad_len + return loss_mask + + def __call__(self, batch): + input_ids = [data["input_ids"] for data in batch] + target_ids = [data["target_ids"] for data in batch] + position_ids = [data["position_ids"] for data in batch] + attention_mask = [data['attention_mask'] for data in batch] + loss_mask = [data['loss_mask'] for data in batch] + + max_length = max([len(t) for t in input_ids]) + for i in range(len(input_ids)): + input_ids[i] = self.pad_token(input_ids[i], max_length) + target_ids[i] = self.pad_token(target_ids[i], max_length) + position_ids[i] = self.pad_position_ids(position_ids[i], max_length) + loss_mask[i] = self.pad_loss_mask(loss_mask[i], max_length) + return { + 'input_ids': torch.LongTensor(input_ids), + 'target_ids': torch.LongTensor(target_ids), + 'position_ids': torch.LongTensor(position_ids), + 'attention_mask': torch.LongTensor(attention_mask), + 'loss_mask': torch.LongTensor(loss_mask) + } +``` +4）获取训练数据 +```python +train_src, train_tgt = read_file() +print('-----------train data length:', len(train_src)) +my_collate_fn = GLMSeq2seqDynamicCollateFN(pad_id=tokenizer.get_command('pad').Id) +train_dataset = GLMSeq2seqDataset(train_src, + train_tgt) +``` +### 2.加载模型和分词器 + +```python +from flagai.auto_model.auto_loader import AutoLoader + +# the model dir, which contains the 1.config.json, 2.pytorch_model.bin, 3.vocab.txt, +# or we will download these files from the model hub to this dir. +model_dir = "./state_dict/glm/" +# Autoloader can build the model and tokenizer automatically. +# 'seq2seq' is the task_name. +AutoLoader("seq2seq",model_name="GLM-large-ch",model_dir=model_dir) +model = auto_loader.get_model() +tokenizer = auto_loader.get_tokenizer() +``` + +### 3. 训练 + +实例化训练器Trainer, 设置训练参数。 + +```python +import torch.cuda +from flagai.trainer import Trainer + +device = torch.device("cuda" if torch.cuda.is_available() else "cpu") +trainer = Trainer( + env_type="pytorch", + experiment_name="roberta_seq2seq", + batch_size=1, + gradient_accumulation_steps=1, + lr=2e-4, + weight_decay=1e-3, + epochs=10, + log_interval=10, + eval_interval=10000, + load_dir=None, + pytorch_device=device, + save_dir="checkpoints", + save_epoch=1, + num_checkpoints=1, + +) +``` +将模型、数据、批处理函数输入训练器开始训练： +```python +trainer.train(model, + train_dataset=train_dataset, + collate_fn=my_collate_fn) +``` + +## 生成（generate.py） +运行前修改模型配置路径model_dir，训练好的模型路径model_save_path。在命令行运行此命令： +```commandline +cd ./examples/glm_title_generation +python ./generate.py +``` +可选择基于概率筛选的随机抽样（random sample）或集束搜索（beamsearch）两种生成方式： +```python +print(predictor.predict_generate_randomsample(text,out_max_length=66, top_k=10, top_p=.1, + repetition_penalty=4.0, temperature=1.2)) +print(predictor.predict_generate_beamsearch(text, out_max_length=66, beam_size=10)) +``` \ No newline at end of file diff --git a/doc_zh/TUTORIAL_12_INSTRUCTIONS_FOR_AutoLoader.md b/doc_zh/TUTORIAL_12_INSTRUCTIONS_FOR_AutoLoader.md deleted file mode 100644 index 661c560f..00000000 --- a/doc_zh/TUTORIAL_12_INSTRUCTIONS_FOR_AutoLoader.md +++ /dev/null @@ -1,44 +0,0 @@ -## AutoLoader - -### 使用Autoloader快速建立模型以及分词器 -Autoloader会根据**model_name**从ModelHub中自动搜索预训练模型和Tokenizer，并将它们下载到**model_dir** - - - -以语义匹配任务为例: -```python -## target包含所有目标分类 -## 0 代表两句话的意思相似 -## 1 代表两句话有着不同的意思 -target = [0, 1] -auto_loader = AutoLoader(task_name="classification", ## 任务名 - model_name="RoBERTa-base-ch", ## 模型名字 - model_dir=model_dir, ## 模型下载的目录 - load_pretrain_params=True, ## 是否要加载已有的预训练模型参数. - target_size=len(target) ## 最终输出的维度，用来进行分类任务. - ) -``` - -### 所有支持的任务 -**task_name**参数可以为如下值: -1. task_name="classification": 支持不同的分类任务，例如文本分类， 语义匹配， 情感分析... -2. task_name="seq2seq": 支持序列到序列的模型, 例如标题自动生成, 对联自动生成, 自动对话... -3. task_name="sequence_labeling": 支持序列标注任务， 比如实体检测，词性标注，中文分词任务... -4. task_name="sequence_labeling_crf": 为序列标注模型添加条件随机场层. -5. task_name="sequence_labeling_gp": 为序列标注模型添加全局指针层. - -### 所有支持的模型 -所有支持的模型都可以在 **model hub** 中找到。 -不同的模型适应不同的任务。 - -#### Transfomrer编码器: - -例如 model_name="BERT-base-ch" or "RoBERTa-base-ch"时， 这些模型支持上一节中提到的所有任务 - -#### Transformer解码器: - -例如 model_name="GPT2-base-ch"时, 模型支持 "seq2seq" 任务. - -#### Transformer 编码器+解码器: - -例如 model_name="T5-base-ch"时, 模型支持"seq2seq" task. diff --git a/doc_zh/TUTORIAL_9_GLM_EXAMPLE_PEOTRY_GENERATION.md b/doc_zh/TUTORIAL_13_GLM_EXAMPLE_PEOTRY_GENERATION.md similarity index 97% rename from doc_zh/TUTORIAL_9_GLM_EXAMPLE_PEOTRY_GENERATION.md rename to doc_zh/TUTORIAL_13_GLM_EXAMPLE_PEOTRY_GENERATION.md index a1f4535e..8d9c7fa4 100644 --- a/doc_zh/TUTORIAL_9_GLM_EXAMPLE_PEOTRY_GENERATION.md +++ b/doc_zh/TUTORIAL_13_GLM_EXAMPLE_PEOTRY_GENERATION.md @@ -25,6 +25,7 @@ cd ./examples/glm_poetry_generation python ./train.py ``` +这里使用`GLM-large-ch`作为样例,如果想要使用`GLM-10b-ch`请点[这里](https://model.baai.ac.cn/model-detail/100001)。 ### 1.准备训练数据 1）定义文件读取函数，从文件中读取数据，得到src和tgt列表： ```python diff --git a/doc_zh/TUTORIAL_13_INSTRUCTIONS_FOR_PREDICTOR.md b/doc_zh/TUTORIAL_13_INSTRUCTIONS_FOR_PREDICTOR.md deleted file mode 100644 index 5117c885..00000000 --- a/doc_zh/TUTORIAL_13_INSTRUCTIONS_FOR_PREDICTOR.md +++ /dev/null @@ -1,37 +0,0 @@ -## Predictor - -### 通过Predictor快速得到预测结果 -通过Predictor，只需要输入一个**文本**，就可以直接得到对应任务的输出。 - -以gpt2编写任务为例： -```python -from flagai.auto_model.auto_loader import AutoLoader -from flagai.model.predictor.predictor import Predictor -if __name__ == '__main__': - ## 通过autoloader加载模型和tokenizer - loader = AutoLoader("seq2seq", "gpt2_base_chinese", model_dir="./state_dict/") - model = loader.get_model() - tokenizer = loader.get_tokenizer() - ## 定义预测器 - predictor = Predictor(model, tokenizer) - ## 定义输入预计的开头，作为输入 - text = "今天天气不错，" - ## 预测器能通过自动分析模型种类来调取不同方法 - out = predictor.predict_generate_randomsample(text, ## 输入 - input_max_length=512, ## 最大出入长度 - out_max_length=100, ## 最大输出长度 - repetition_penalty=1.5, ## 避免重复输出. (https://arxiv.org/pdf/1909.05858.pdf) - top_k=20, ## 只保留概率最大的k个token. - top_p=0.8) ## 保留累计概率大于等于top_p的token.(http://arxiv.org/abs/1904.09751) - - print(f"out is {out}") - ### out is 到这里来看了一下，很是兴奋，就和朋友一起来这里来了。我们是周五晚上去的，人不多，所以没有排队，而且这里的环境真的很好，在这里享受美食真的很舒服，我们点了一个套餐，两个人吃刚刚好，味道很好。 -``` - -### 所有支持的方法 -1. predict_cls_classifier: 输入文本或文本对得到多分类结果，支持bert、roberta等transformer编码器模型。 -2. predict_masklm: 输入带有[MASK]标记的文本得到原文结果，支持bert、roberta等transformer编码器模型 -3. predict_ner: 输入文本得到ner结果，支持bert、roberta等transformer编码器模型。 -4. predict_generate_beamsearch: 输入文本得到输出文本，属于seq2seq任务。支持bert、roberta、gpt2、t5和glm模型。 -5. predict_generate_randomsample: 输入文本得到输出文本，属于seq2seq任务。支持bert、roberta、gpt2、t5和glm模型。. - diff --git a/doc_zh/Huggingface_t5.md b/doc_zh/TUTORIAL_14_HUGGINGFACE_T5.md similarity index 93% rename from doc_zh/Huggingface_t5.md rename to doc_zh/TUTORIAL_14_HUGGINGFACE_T5.md index dcb0b18a..e0dad603 100644 --- a/doc_zh/Huggingface_t5.md +++ b/doc_zh/TUTORIAL_14_HUGGINGFACE_T5.md @@ -33,21 +33,21 @@ trainer = MyTrainer( batch_size=4, eval_interval=10, log_interval=10, - experiment_name='t5-3b', + experiment_name='t5-11b', pytorch_device='cuda:0', load_dir=None, lr=1e-4, fp16=False) # using huggingface transformers to get tokenizer and models -model_name = 't5-3b' +model_name = 't5-11b' tokenizer = T5Tokenizer.from_pretrained(model_name) model = T5ForConditionalGeneration.from_pretrained(model_name) print("loading model & tokenizer is done!") src_dir = 'train_inputs.txt' tgt_dir = 'train_targets.txt' -model_dir = "./t5-3b" # 模型位置 +model_dir = "./t5-11b" # 模型位置 maxlen = 1024 @@ -139,7 +139,7 @@ trainer.train(model, ``` ## 加速训练的技巧 -我们可能不会在V100 32G上运行t5-3b。所以，我们需要一些技巧来减少GPU内存的使用。 +我们可能不会在V100 32G上运行t5-11b。所以，我们需要一些技巧来减少GPU内存的使用。 ### 第一步：fp16 把模型参数变为 `fp16` ```python @@ -149,15 +149,15 @@ trainer = MyTrainer( batch_size=1, eval_interval=10, log_interval=10, - experiment_name='t5-3b', + experiment_name='t5-11b', pytorch_device='cuda:0', load_dir=None, lr=1e-4, fp16=True) # change to `True` ``` ### 第二步：梯度重计算（checkpoint） -在forward阶段不将中间结果保存。我们可以运行`batch size`=1的t5-3b。 -现在，我们可以用 `gradient_accumulation_steps` train/finetune 一个 t5-3b。 +在forward阶段不将中间结果保存。我们可以运行`batch size`=1的t5-11b。 +现在，我们可以用 `gradient_accumulation_steps` train/finetune 一个 t5-11b。 ```python trainer = MyTrainer( env_type='pytorch', @@ -165,7 +165,7 @@ trainer = MyTrainer( batch_size=1, eval_interval=10, log_interval=10, - experiment_name='t5-3b', + experiment_name='t5-11b', pytorch_device='cuda:0', load_dir=None, lr=1e-4, @@ -181,7 +181,7 @@ trainer = Trainer( batch_size=1, eval_interval=10, log_interval=10, - experiment_name='t5-3b', + experiment_name='t5-11b', load_dir=None, lr=1e-4, fp16=True @@ -205,7 +205,7 @@ trainer = Trainer( batch_size=1, eval_interval=10, log_interval=10, - experiment_name='t5-3b', + experiment_name='t5-11b', load_dir=None, lr=1e-4, fp16=True @@ -230,7 +230,7 @@ trainer = Trainer( batch_size=1, eval_interval=10, log_interval=10, - experiment_name='t5-3b', + experiment_name='t5-11b', load_dir=None, lr=1e-4, fp16=True diff --git a/doc_zh/TUTORIAL_10_BERT_EXAMPLE_TITLE_GENERATION.md b/doc_zh/TUTORIAL_15_BERT_EXAMPLE_TITLE_GENERATION.md similarity index 100% rename from doc_zh/TUTORIAL_10_BERT_EXAMPLE_TITLE_GENERATION.md rename to doc_zh/TUTORIAL_15_BERT_EXAMPLE_TITLE_GENERATION.md diff --git a/doc_zh/TUTORIAL_11_BERT_EXAMPLE_SEMANTIC_MATCHING.md b/doc_zh/TUTORIAL_16_BERT_EXAMPLE_SEMANTIC_MATCHING.md similarity index 100% rename from doc_zh/TUTORIAL_11_BERT_EXAMPLE_SEMANTIC_MATCHING.md rename to doc_zh/TUTORIAL_16_BERT_EXAMPLE_SEMANTIC_MATCHING.md diff --git a/doc_zh/TUTORIAL_14_BERT_EXAMPLE_NER.md b/doc_zh/TUTORIAL_17_BERT_EXAMPLE_NER.md similarity index 100% rename from doc_zh/TUTORIAL_14_BERT_EXAMPLE_NER.md rename to doc_zh/TUTORIAL_17_BERT_EXAMPLE_NER.md diff --git a/doc_zh/TUTORIAL_15_GPT2_WRITING.md b/doc_zh/TUTORIAL_18_GPT2_WRITING.md similarity index 100% rename from doc_zh/TUTORIAL_15_GPT2_WRITING.md rename to doc_zh/TUTORIAL_18_GPT2_WRITING.md diff --git a/doc_zh/TUTORIAL_16_T5_EXAMPLE_TITLE_GENERATION.md b/doc_zh/TUTORIAL_19_T5_EXAMPLE_TITLE_GENERATION.md similarity index 100% rename from doc_zh/TUTORIAL_16_T5_EXAMPLE_TITLE_GENERATION.md rename to doc_zh/TUTORIAL_19_T5_EXAMPLE_TITLE_GENERATION.md diff --git a/doc_zh/TUTORIAL_1_BASICS.md b/doc_zh/TUTORIAL_1_BASICS.md deleted file mode 100644 index a01707de..00000000 --- a/doc_zh/TUTORIAL_1_BASICS.md +++ /dev/null @@ -1,42 +0,0 @@ -## Transformer basics -简单来说，[Attention Is All You Need](https://arxiv.org/abs/1706.03762) 中首次提出的Transformers -可以看作是编码器模块和解码器模块的集成。随着注意力机制的使用， -Transformer可以从输入中提取更好的特征，这点使得基于Transformer的模型当前在大多数语言任务中有着了十分优越的表现。 -Transformer的另一个特点是它与并行计算的兼容性，这是其与RNN等时序模型相比起来的优点。 - -Transformer的结构如下图所示 -
- -在编码器步骤中，首先将输入编码与位置编码相加， -然后将相加的结果传递给多头注意力机制，该机制能够考虑不同位置token之间的相关性信息。 -注意力机制计算的输出将加到原始输入里，然后添加层归一化结构。 -编码器剩下的部分是有层规范化的前馈层，以及第二个相加操作。 - -解码器结构与编码器类似，但有以下区别： - -1. 在开始时，我们需要为解码器屏蔽未来的信息，这是通过将矩阵的上三角设置为0来完成的。 -2. 有一个中间层，其中包含来自编码器的查询(Q)和键(K)，以及来自解码器的值(V)。 -3. 解码器在输出端有一个线性层和softmax层，以确定词汇表中每个标记的输出概率 - -N个编码器和解码器将堆叠在一起形成Transformer，其中N通常为12或24。 - -Transformer的详细结构信息可以参考 [这篇文章](https://towardsdatascience.com/illustrated-guide-to-transformers-step-by-step-explanation-f74876522bc0). - -## Transformer Applications - -目前，存在几种不同的预训练模型体系结构：仅实现编码器体系结构的自编码模型（例如BERT）， -仅实现解码器的自回归模型（例如GPT），以及同时实现编码器和解码器的编码器-解码器模型（例如T5）。 - -[All NLP Tasks Are Generation Tasks: A General Pretraining Framework](https://arxiv.org/abs/2103.10360) 这篇论文里提出的**GLM模型**, 声称使用新的预训练方法，从而使其在分类， -无条件生成和条件生成任务中具有良好的性能。 - -GLM的主要功能包括： - -- 任务一：文本的一些区间会被屏蔽（参照自动编码的做法）。 -这些区间将被随机重新排列，并以自动回归方式进行预测。屏蔽的区间覆盖原始文本的15%。 -- 任务二：与第一个任务类似，但是区间会覆盖原始文本的50%-100% -- 剩下GLM相对于BERT的改动 - - [Pre-LN](http://proceedings.mlr.press/v119/xiong20b.html) - - 2D 位置编码：每个token都有两个位置编码：句子中的全局位置和屏蔽区间内的局部位置。. - - 前馈网络被线性层取代 -关于GLM的自回归以及自编码操作可以查看 [这里](APPENDIX_GLM_IO.md)的示例. diff --git a/doc_zh/TUTORIAL_1_TOKENIZER.md b/doc_zh/TUTORIAL_1_TOKENIZER.md new file mode 100644 index 00000000..1ec6448f --- /dev/null +++ b/doc_zh/TUTORIAL_1_TOKENIZER.md @@ -0,0 +1,81 @@ +# 分词器 + +## 什么是分词？ +分词是 NLP 项目预处理阶段的一个基本步骤，其目的是将散乱的原始文本转换为规整的数字序列， 以便机器学习系统使用。 + +在分词过程中，分词器首先将自然语言文本拆分为语义单元, 这些语义单元被称为子词(token)。随后分词器通过一个词表文件查找子词的序号，这样原始文本就被转换为了一个数字的序列。 + +
+ +值得注意的是，不同的分词器可以有不同的文本分割方式，并且有不同的词表文件, 相关算法的介绍可以在 [这里](tokenization.md) 查看。 + +目前我们支持下列六个分词器: + +| 分词器 | 语言 | +|-----------------------------|-----| +| GLMLargeEnWordPieceTokenizer | 英文 | +| GLMLargeChTokenizer | 中文 | +| GLM10bENBPETokenizer | 英文 | +| T5BPETokenizer | 中文 | +| ROBERTATokenizer | 中文 | +| BertWordPieceTokenizer | 中文 | + + +## 加载分词器 +```python +from flagai.data.tokenizer import GLMLargeEnWordPieceTokenizer +tokenizer = GLMLargeEnWordPieceTokenizer() +``` +在这一步里，Modelhub中的词表文件将被自动下载到`cache_dir`参数中指定的路径。默认设置为分词器文件下的 ./vocab 目录。 + +## 应用分词器 +让我们使用一个分词器将原始文本编码成数字序列，然后将数字序列恢复成原始文本： +```python +text = "Jack is walking a dog." # 输入文本 +encoded_ids = tokenizer.EncodeAsIds(text) # 数字序列 +# 现在 encoded_ids = [2990, 2003, 3788, 1037, 3899, 1012] +recoverd_text = tokenizer.DecodeIds(encoded_ids) # 将数字序列恢复为文本 +# recovered_text此时与text相同 +``` + +## 创建新的分词器 +想创建新的分词器的时候, 需要这么做: +### 1. 在`/flagai/tokenizer`目录下建立一个新的目录 + +### 2. 使用Hugging Face transformer支持的分词器来帮助创建 +FlagAI支持借用Hugging Face里的分词器来创建新的分词器，以T5Tokenizer为例 +```python +from transformers import T5Tokenizer +from ..tokenizer import Tokenizer +class T5BPETokenizer(Tokenizer): + def __init__(self, model_type_or_path="t5-base", cache_dir=None): + self.text_tokenizer = T5Tokenizer.from_pretrained(model_type_or_path, + cache_dir=cache_dir) + self.text_tokenizer.max_len = int(1e12) +``` + +### 3. 自定义分词器的接口 +如果Hugging Face里的分词器不能满足您的需求，那么需要先准备好一份词表，然后手动实现下列函数的功能： + +```python +def EncodeAsIds(self, text: str, process_fn=None): + """输入文本 => 一个token序号列表""" + +def EncodeAsTokens(self, text: str, process_fn=None): + """输入文本 => 一个token列表""" + +def IdToToken(self, Id: int): + """Token序号 => token""" + +def TokenToId(self, token: str): + """Token => token序号""" + return self.text_tokenizer._convert_token_to_id(token) + +def DecodeIds(self, Ids: list[int]): + """一个token序号列表 => 对应的文本""" + return self.DecodeTokens([self.IdToToken(id) for id in Ids]) + +def DecodeTokens(self, tokens: list[str]): + """一个token列表 => 对应的文本""" + return self.text_tokenizer.convert_tokens_to_string(tokens) +``` \ No newline at end of file diff --git a/doc_zh/AllSupportedTasks.md b/doc_zh/TUTORIAL_20_SUPPORTED_TASKS.md similarity index 100% rename from doc_zh/AllSupportedTasks.md rename to doc_zh/TUTORIAL_20_SUPPORTED_TASKS.md diff --git a/doc_zh/TUTORIAL_2_DATASET.md b/doc_zh/TUTORIAL_2_DATASET.md new file mode 100644 index 00000000..2e5d742c --- /dev/null +++ b/doc_zh/TUTORIAL_2_DATASET.md @@ -0,0 +1,360 @@ +# 数据集处理流程 +构建数据集的过程就是NLP的数据预处理过程，其主要目的是将原始的散乱的文件数据重新整理成统一结构的数据，以便语言模型能够直接使用。构建数据集样例的主要流程如下所示（以`CommitmentBank`数据集为例）： + +
+ +目前项目里存在三种数据预处理的情况，即对分类任务的微调，预训练, 以及对生成任务的微调。我们在接下来会分别展开。 + +## 数据处理：分类任务微调([prompt-learning模式](TUTORIAL_7_PROMPT_LEARNING.md)) +### 应用代码 +```python +import torch.utils.data +from flagai.data.dataset import SuperGlueDataset +from flagai.data.tokenizer import GLMLargeEnWordPieceTokenizer +from tests.test_dataset_new_superglue import CollateArguments +from flagai.data.dataset import ConstructSuperglueStrategy + +# 得到默认参数 +cl_args = CollateArguments() + +# 创建分词器 +tokenizer = GLMLargeEnWordPieceTokenizer() + +# 初步读取并处理数据集 +dataset = SuperGlueDataset(task_name='cb', + data_dir='./datasets/', + dataset_type='train', + tokenizer=tokenizer) + +# 构建collate function +collate_fn = ConstructSuperglueStrategy(cl_args, tokenizer, task_name="rte") + +# 创建加载器 +loader = torch.utils.data.DataLoader(dataset, + batch_size=1, + shuffle=False, + num_workers=1, + drop_last=False, + pin_memory=False, + collate_fn=collate_fn) +``` + +### 初步读取并处理数据集 +对应的代码模块如下所示，其包含了两个步骤：自动加载数据集，以及统一所有数据集的结构 +```python +dataset = SuperGlueDataset(task_name='cb', + data_dir='./datasets/', + dataset_type='train', + tokenizer=tokenizer) +``` +#### 1.加载数据集 + +将`task_name`设置成数据集对应的简称后，运行以上代码，相关数据会在后台自动下载。FlagAI目前支持自动加载下列分类数据集： + + +| 数据集名称 | 数据集简称 | 语言 | 所属评测基准 | +|----------------------------------------------|----------|------|----------| +| Broadcoverage Diagnostics | boolq | 英文 | SuperGLUE | +| CommitmentBank | cb | 英文 | SuperGLUE | +| Choice of Plausible Alternatives | copa | 英文 | SuperGLUE | +| Multi-Sentence Reading Comprehension | muiltirc | 英文 | SuperGLUE | +| Recognizing Textual Entailment | rte | 英文 | SuperGLUE | +| Words in Context | wic | 英文 | SuperGLUE | +| The Winograd Schema Challenge | wsc | 英文 | SuperGLUE | +| Ant Financial Question Matching Corpus | afqmc | 中文 | CLUE | +| Short Text Classificaiton for News | tnews | 中文 | CLUE | + +数据集会被自动下载到`data_dir`对应的地址，默认为项目的`./dataset`目录。 + +下载好的数据集目录下会包含三个文件，对应训练集数据，验证集数据，以及测试集数据, 以`CommitmentBank`数据集为例，目录下的`train.jsonl`对应训练集，`val.jsonl`对应验证集，`test.jsonl`对应测试集。一般来说训练集和测试集会包含标签信息，而测试集则没有。这些数据文件在接下来的流程中会分开处理。 + +不同的数据集可能会有不同的文件格式，以及不同的结构。以`CommitmentBank`数据集为例，下面是其中的一个样例 + +
+ +可以看到，其包含了四个部分,如下所示： + +| 键值 | 含义 | 值 | +|-----------|-------------------------------------------------------|--------------------------------| +| premise | 前提文本 | Mary is a high school student. | +| hypothesis | 假设文本 | Mary is a student | +| label | 代表前提和假设关系的标签
包含entailment,neutral和contradiction三种 | entailment | +| idx | 样例在数据集里的序号 | 10 | + +目前所有FlagAI支持数据集的具体结构可以在[这里](DATASET_EXAMPLE.md)查看。 + +#### 2. 统一数据集结构 +在这一步里，我们会统一不同数据集的数据结构，以方便接下来处理。此结构的细节如下： + + +| 键值 | 含义 | 数据格式 | +|--------|-------------------------------------------------------|------| +| guid | a unique textual identifier | str | +| text_a | the sequence of text | str | +| text_b | an optional, second sequence of text | str | +| label | an optional label | str | +| logits | an optional list of per-class logits | list | +| meta | an optional dictionary to store arbitrary meta information | dict | +| ids | an optional numeric index | int | + +例如上一步`CommitmentBank`的样例会被处理成如下的形式 + +
+ +需要注意的是如果因为数据结构太复杂，导致text_a和text_b无法塞下背景文本信息的话，可以把剩下的信息放在`meta`里。 + +当数据集被构造好以后，可以直接在代码里通过索引的方式查看其中某个样例： + +```python +example = dataset[3] # 数据集里第3个样例 +``` + +### 将数据整理成模型的输入 + +对应的功能在下面的函数里实现，其包含了两个步骤：构造模板，分词并构造输入样例。 +```python +collate_fn = ConstructSuperglueStrategy(cl_args, + tokenizer, + task_name=task_name) +``` + +#### 1.构建完形填空模板 + +一个完形填空模板包含了背景文本，空位，以及在提供给空位的选项。模型需要找到正确的选项，并填进空位里。 + +对于每个不同的任务，我们都需要构建不同构造的完型填空问题来让模型回答，以CommitmentBank数据集为例， 其考量的是能否由前提推导出假设, 而有且仅有三种结果：contradiction/neutral/entailment。那么我们可以构建如下的完型填空问题， 其中contradiction/neutral/entailment分别对应true/false/neither。 + +
+ +可以看到，大体上可以分成两步：第一步是组合已有的文本，使其看上去符合完形填空格式；第二步是将原始的标签文本转化为新的标签，作为可能会填入空位的选项。 + +#### 2.分词并构造输入样例 +接下来，我们需要构造模型的输入，第一步是分词，而接下来则需要分成两种情况： + +第一种情况下，数据集包含的标签类别是有限的，比如`CommitmentBank`数据集里只会存在`entailment/contradiction/neutral`三种标签文本。第二类情况里每一段完型填空都会给出不同的选项(一般是一段长文本)。比如在一些阅读理解数据集里，每一个选项都是一段对于文本的不同理解。这两类情况的处理方法如下所示： + +**a)单个token的完形填空** + + +| 键值 | 维度 | 含义 | 构造方法 | +|---------------------------------------------|--------------------------------------|------------|-----------------------------------------------| +| input_ids | torch.Size([seq_length1]) | 输入矩阵 | 由上一步的完形填空文本，加上一些特殊字符2组成 | +| labels | labels: torch.Size([1]) | 标签 | 对应的数字标签，比如0,1,2... | +| position_ids | torch.Size([2， seq_length]) | 位置编码 | 参考[GLM流程](GLM.md)，第一行代表token的绝对位置，第二行代表遮挡部分的相对位置 | +| attention_mask | torch.Size([1]) | 分隔符位置 | | +| target_ids | torch.Size([num_labels3]) | 全量标签列表 | 将所有标签文本分别对应单token的标签，然后将这些标签的序号依次放入target_ids | +| logit_mask | torch.Size([seq_length]) | 对应的文本是否为回答 | 对于每个token, 如果是回答，则对应的地方为1，否则为0 | + + +1: seq_length代表规定的每个输入向量的最大长度 + +2: 特殊字符添加流程：在句首添加[CLS]符号，句尾添加[EOS]符号，直到长度达到seq_length, 如果完形填空输出的文本有两段，则在中间添加[SEP]符号 +
+ +3: num_labels代表完形填空问题中选项的个数 + +**b)多个token的完形填空** + +| 键值 | 维度 | 含义 | 区别 | +|---------------------------------------------|-----------------------------------------|------------|----------------------------------------------------| +| input_ids | torch.Size([num_labels, seq_length]) | 输入矩阵 | 将对应的文本拷贝num_labels份 | +| labels | labels: torch.Size([1]) | 标签 | | +| position_ids | torch.Size([num_labels, 2， seq_length]) | 位置编码 | 将原本的位置编码拷贝num_labels份 | +| attention_mask | torch.Size([num_labels]) | 分隔符位置 | 拷贝num_labels份 | +| target_ids | torch.Size([num_labels, seq_length]) | 每个选项信息 | 矩阵的每一列代表每一个选项，每一行代表当前选项对应的文本 | +| logit_mask | torch.Size([num_labels, seq_length]) | 对应的文本是否为回答 | 拷贝num_labels份 | + + + +### 创建加载器 + +最后将数据放入[PyTorch加载器](https://pytorch.org/docs/stable/data.html?highlight=dataloader#torch.utils.data.DataLoader)即可。 +```python +import torch +loader = torch.utils.data.DataLoader(dataset, + batch_size=1, + shuffle=False, + num_workers=1, + drop_last=False, + pin_memory=False, + collate_fn=collate_fn) +``` +Dataloader里的数据可以通过如下方法查看 + +```python +for data_iterator in loader: + for key, value in data_iterator.items(): + print(key, value) + # break +``` +加载器构造好之后即可用于接下来的训练和预测过程。


+ + +## 数据处理：GLM预训练任务 +预训练任务数据格式样例： +```text +{ + "RECORDS": [ + { + "id": 0, + "uniqueKey": "", + "titleUkey": "", + "dataType": "", + "title": "", + "content": "平安信用卡上门激活失败是为啥？平安信用卡上门激 + 活失败的主要原因有这些：申卡人是在网上申请的新用户，上门激活时 + 携带资料不足，需要申卡人携带身份证、信用卡、预留号码等去平安银 + 行网点验证身份才能激活；申卡人在平安银行预留的手机号码有误；申卡 + 人在激活时输入的相关信息不正确；申卡人在回答上门激活工作人员的问 + 题是，跟之前提交的资料有一些出入，有违规嫌疑。平安信用卡上门激活失 + 败的主要原因有这些：申卡人是在网上申请的新用户，上门激活时携带资料不 + 足，需要申卡人携带身份证、信用卡、预留号码等去平安银行网点验证身份才能 + 激活；申卡人在平安银行预留的手机号码有误；申卡人在激活时输入的相关信息不 + 正确；申卡人在回答上门激活工作人员的问题是，跟之前提交的资料有一 + 些出入，有违规嫌疑。" + }, + ] +} +``` +预训练的任务处理实例代码: +```python +tokenizer = GLMLargeChTokenizer(add_block_symbols=True, + add_task_mask=True, + add_decoder_mask=False, + fix_command_token=True) + +ds_args = DatasetArguments() + +tokenizer = GLMLargeChTokenizer(fix_command_token=True, + add_block_symbols=True, + add_task_mask=True, + add_decoder_mask=False) + +ds_args = add_args(ds_args, tokenizer) + +def create_dataset(tokenizer, should_split): + dataset = get_dataset_lazy("./examples/glm_pretrain/data", # 懒加载 + tokenizer=tokenizer, + pre_tokenize=True, + num_processes=10, + no_lazy_loader=True) + if should_split: + datasets = split_ds(dataset, split=[.8, .2, .0], shuffle=True) # 手动切分 + else: + datasets = [dataset] + + datasets = [ + BlockDataset(ds, + tokenizer, + max_seq_len=512, + sample_across_doc=True, + non_sentence_start=0.0) if ds is not None else None + for ds in datasets + ] + return datasets + +datasets = create_dataset(tokenizer, should_split=True) +``` +预训练的数据处理也遵循着同样的流程，不过有如下的区别 +1. 预训练数据集默认是没有切分成训练集，验证集，以及测试集的，所以还需要手动进行切分 +2. 由于预训练数据集一般来说较为庞大，所以使用了懒加载(lazy loading)，懒加载只在真正用到对象的时候才使其实例化，是一种较为节省资源的操作。 +3. 预训练的时候，collate function会随机按照三种不同的模式处理数据: bert模式（遮挡随机区间），sentence模式(按照完整句子进行遮挡)以及gpt模式（只遮挡一段长区间）。模型的输入相比一般生成任务也会多一个`mode`键。 +4. 预训练不用添加模板，直接按下表构建模型输入即可 + +| 键值 | 维度 | 含义 | 构造方法 | +|----------------|--------------------------------------|------------|--------------------------------------------------| +| input_ids | torch.Size([seq_length1]) | 输入矩阵 | 由上一步的模板文本，加上一些特殊字符组成 | +| position_ids | torch.Size([2， seq_length]) | 位置编码 | 参考[GLM流程](GLM.md)，第一行代表token的绝对位置，第二行代表遮挡部分的相对位置 | +| attention_mask | torch.Size([1]) | 分隔符位置 | 对于生成类的模式，得到源文本结束的位置；否则得到输入文本结束的位置 | +| target_ids | torch.Size([num_labels3]) | 全量标签列表 | 被遮挡住的文本 | +| logit_mask | torch.Size([seq_length]) | 通过遮挡使得模型只会处理目标文本部分的loss | 对于每个token, 如果是回答，则对应的地方为1，否则为0 | +| mode | str | 数据处理模式 | | +


+ +## 数据处理：生成任务微调 +代码实现如下所示： +```python +import torch.utils.data +from flagai.data.dataset import Seq2SeqDataset +from flagai.data.tokenizer import GLMLargeEnWordPieceTokenizer +from tests.test_dataset_new_superglue import Seq2SeqCollateArguments +from flagai.data.dataset import ConstructSeq2seqStrategy + +# 得到默认参数 +cl_args = Seq2SeqCollateArguments() + +# 创建分词器 +tokenizer = GLMLargeChTokenizer(add_block_symbols=True, + TUTORIAL_4_DATASET.md add_task_mask=False, + add_decoder_mask=False, + fix_command_token=False) + +# 初步读取并处理数据集 +dataset = Seq2SeqDataset(task_name='cmrc', + data_dir='./datasets/', + dataset_type='train', + tokenizer=tokenizer) + +# 构建collate function +collate_fn = ConstructSeq2seqStrategy(cl_args, tokenizer, task_name="rte") + +# 创建加载器 +loader = torch.utils.data.DataLoader(dataset, + batch_size=1, + shuffle=False, + num_workers=1, + drop_last=False, + pin_memory=False, + collate_fn=collate_fn) +``` + +### 初步读取并处理数据集 + +目前支持[CMRC2018](https://www.clue.ai/introduce.html)任务，CMRC是一个阅读理解类的任务，需要根据背景文本来回答一系列提问，其数据结构示例如下： + +```text +{'paragraphs': + [{'id': 'TRAIN_186', + 'context': '范廷颂枢机（，），圣名保禄·若瑟（），是越南罗马天主教枢机。1963年被任 + 为主教；1990年被擢升为天主教河内总教区宗座署理；1994年被擢升为总主教，同年年底被擢 + 升为枢机；2009年2月离世。范廷颂于1919年6月15日在越南宁平省天主教发艳教区出生；童年 + 时接受良好教育后，被一位越南神父带到河内继续其学业。范廷颂于1940年在河内大修道院完 + 成神学学业。范廷颂于1949年6月6日在河内的主教座堂晋铎；及后被派到圣女小德兰孤儿院服 + 务。1950年代，范廷颂在河内堂区创建移民接待中心以收容到河内避战的难民。1954年，法越 + 战争结束，越南民主共和国建都河内，当时很多天主教神职人员逃至越南的南方，但范廷颂仍然 + 留在河内。翌年管理圣若望小修院；惟在1960年因捍卫修院的自由、自治及拒绝政府在修院设 + 政治课的要求而被捕。1963年4月5日，教宗任命范廷颂为天主教北宁教区主教，同年8月15日就 + 任；其牧铭为「我信天主的爱」。由于范廷颂被越南政府软禁差不多30年，因此他无法到所属堂 + 区进行牧灵工作而专注研读等工作。范廷颂除了面对战争、贫困、被当局迫害天主教会等问题外 + ，也秘密恢复修院、创建女修会团体等。1990年，教宗若望保禄二世在同年6月18日擢升范廷颂 + 为天主教河内总教区宗座署理以填补该教区总主教的空缺。1994年3月23日，范廷颂被教宗若望 + 保禄二...', + 'qas':{'question': '范廷颂是何时去世的？', 'id': 'TRAIN_186_QUERY_4', + 'answers': [{'text': '范廷颂于2009年2月22日清晨在河内离世', 'answer_start': 759}]}]}], + 'id': 'TRAIN_186', 'title': '范廷颂'} +``` +使用的时候，我们将`task_name`参数改为`cmrc`即可。实现的过程类似分类任务的微调，最终也会将数据集初步处理成一样的结构，对应代码如下： + +```python +dataset = Seq2SeqDataset(task_name='cmrc', data_dir='./datasets/', + dataset_type='train', tokenizer=tokenizer) +``` + +### 将数据整理成模型的输入 + +代码如下所示。与生成任务相比，同样是构造模板以及模型输入，区别在于构造的方式不一样 +```python +collate_fn = ConstructSeq2seqStrategy(cl_args, + tokenizer, + task_name=task_name) +``` + +#### 1.构建填空模板 +由于是阅读理解任务，所以在模板里需要体现出是针对指定的阅读理解问题进行回答的，参考如下构建方式 +
+ + + +#### 2.分词并构造输入样例 +与预训练类似，区别在于没有mode这个键值。 + diff --git a/doc_zh/TUTORIAL_2_PROJECT_STRUCTURE.md b/doc_zh/TUTORIAL_2_PROJECT_STRUCTURE.md deleted file mode 100644 index 46f529b0..00000000 --- a/doc_zh/TUTORIAL_2_PROJECT_STRUCTURE.md +++ /dev/null @@ -1,25 +0,0 @@ -# 项目结构 - -*flagai*的结构如下所示 -``` -Sailing/ - |--flagai/ - | |--data/ # contains datasets and tokenizers - | |--model/ # contains models,blocks and layers (layers->block->model) - | |--fp16/ # contains fp16 tools - | |--mpu/ # fork from megatron-lm - | |--docs/ # documentations - | |--trainer.py/ # trainer for pytorch, deepspeed+mpu - | |--logging.py # - | |--metrics.py/ # contains frequently-used metrics, e.g., accuracy - | |--optimizers.py - | |--schedulers.py.py - | |--test_utils.py # contains tools for testing - | |--utils.py - |--setup.py - |--test.py # for excuting all tests - |--README.md - |--requirements.txt - |--logo.jpg - -``` \ No newline at end of file diff --git a/doc_zh/TUTORIAL_3_MODEL.md b/doc_zh/TUTORIAL_3_MODEL.md new file mode 100644 index 00000000..d29cc6ee --- /dev/null +++ b/doc_zh/TUTORIAL_3_MODEL.md @@ -0,0 +1,93 @@ +# 基类 + +基类 BaseModel 实现了从本地文件或目录或从库提供的预训练模型配置（从BAAI modelhub 的 金山 S3 存储库下载）加载/保存模型的常用方法。 +现在所有支持的模型，对三种最常见的模型类型【encoder，decoder和encoder-decoder】进行了支持。现在GLM模型可以加载所有GLM系列的模型，详见 https://github.com/THUDM/GLM + +## From_pretrain + +同一个模型结构的模型可以用同一个class进行加载，比如BERT-base 和Roberta-base模型都能用BertModel这个class进行加载。From_pretrain为了数据/模型并行的模型加载进行了特定优化，避免重复下载导致的资源浪费。 +通过调用ClassName.from_pretrian()来进行加载，现在我们的model hub中对以下的模型进行了支持，可以直接下载模型配置文件【config.json】，模型权重[pytorch_model.bin]，以及字典文件[vocab.txt]。例子： +```python +from flagai.model.glm_model import GLMForSingleTokenCloze +model = GLMForSingleTokenCloze.from_pretrain(download_path="./state_dict", model_name="GLM-large-ch") +``` +如果是从本地加载模型权重，也可以通过 ClassName.from_pretrain()进行加载。例子： +从`./state_dict/GLM-large-ch`目录中加载模型文件 `pytorch_model.bin` +```python +from flagai.model.glm_model import GLMForSingleTokenCloze +model = GLMForSingleTokenCloze.from_pretrain(download_path="./state_dict", + model_name="GLM-large-ch") +``` +## 所有支持模型 + +| ClassName | ModelName | Language | Model Type | +|-----------------------------------|-----------------|----------|------------| +| flagai.model.glm_model.GLMModel | GLM-10b-ch | chinese | encoder | +| flagai.model.glm_model.GLMModel | GLM-large-ch | chinese | encoder | +| flagai.model.bert_model.BertModel | RoBERTa-base-ch | chinese | encoder | +| flagai.model.gpt2_model.GPT2Model | GPT2_base_ch | chinese | decoder | +| flagai.model.t5_model.T5Model | T5-base-ch | chinese | enc2dec | +| flagai.model.t5_model.T5Model | T5-base-en | chinese | enc2dec | +| flagai.model.bert_model.BertModel | BERT-base-en | english | encoder | +| flagai.model.glm_model.GLMModel | GLM-large-en | english | encoder | + +## 支持的模型+任务 + +同时，我们对在任务上finetune好的模型进行了支持，如下表所示，可以通过ClassName.from_pretrain()来加载模型权重，例如，我们自动下载并加载一个在title-generation任务上训练好的GLM-large-ch模型： +```python +from flagai.model.glm_model import GLMForSeq2Seq +model = GLMForSeq2Seq.from_pretrain(model_name='GLM-large-ch') +``` +我们也提供了AutoLoader类来帮助加载模型，比如GLM-large-ch模型用于seq2seq任务，这里我们采用了任务和模型独立的设计，理论上任务和模型可以自由更换。 +```python +from flagai.auto_model.auto_loader import AutoLoader +auto_loader = AutoLoader("seq2seq", + model_name="GLM-large-ch", + model_dir= "./state_dict") +model = auto_loader.get_model() +``` +| ClassName | Model Name | language | Task | +|-------------------------------------------------|-----------------|----------|-------------------| +| flagai.model.glm_model.GLMForSeq2Seq | GLM-large-ch | chinese | title generation | +| flagai.model.glm_model.GLMForSeq2Seq | GLM-large-ch | chinese | poetry generation | +| flagai.model.bert_model.BertForSequenceLabeling | RoBERTa-base-ch | chinese | title generation | +| flagai.model.bert_model.BertForSequenceLabeling | RoBERTa-base-ch | chinese | NER | +| flagai.model.bert_model.BertForSequenceLabeling | RoBERTa-base-ch | chinese | semantic matching | +| flagai.model.t5_model.T5Model | T5-base-ch | chinese | title generation | +| flagai.model.bert_model.BertForSequenceLabeling | BERT-base-en | english | title gneration | + +## 模型设计 +模型主要的构建逻辑`layer->block>model` +`flagai.model.layer`: 包括mlp，layernorm, activation，attention等各种layer层 + +`flagai.model.block`:通过组装各种layer来构建transformer block，比如BERT block等 + +`flagai.model`: 通过embedding层和stacked blocks 来构建model + +## forward 函数 +Model 的forward函数： +输入是 keyword arguments：包括 input_ids, position_ids, attention_mask等，对冗余的参数会自动忽略 +比如GLM的forward 函数： +```python +def forward(self, + input_ids=None, + position_ids=None, + attention_mask=None, + mems=None, + return_memory=False, + detach_memory=True, + prompt_pos=None, + **kwargs) +``` +输出是 dictionary，包括 logits 和hidden states，这两个是必须的，例如GLM forword函数的返回： +```python +return {'loss': loss, 'logits': logits, 'hidden_states': mems} +``` +## init_from_json +Model 的init_from json函数： +输入是一个dictionary， 输出是一个初始化的model +例如GLMModel的调用如下： +```python +GLMModel.init_from_json(config_file = "./config.json", **kwargs) +``` +**kwargs是预留参数，为了兼容一些模型新增的初始化参数 \ No newline at end of file diff --git a/doc_zh/TUTORIAL_3_TOKENIZER.md b/doc_zh/TUTORIAL_3_TOKENIZER.md deleted file mode 100644 index 8482e109..00000000 --- a/doc_zh/TUTORIAL_3_TOKENIZER.md +++ /dev/null @@ -1,82 +0,0 @@ -# 分词器 - -## 支持的分词器列表 -| 分词器 | 语言 | 相关模型(參考 [ModelHub](https://model.baai.ac.cn/models)) | -|-----------------------------|-----|------------------------------------------------------| -| GLMLargeEnWordPieceTokenizer | 英文 | GLM-large-en | -| GLMLargeChTokenizer | 中文 | GLM-large-ch | -| GLM10bENBPETokenizer | 英文 | glm_10b_en | -| T5BPETokenizer | 中文 | t5_base | -| ROBERTATokenizer | 中文 | RoBERT-base-ch | -| BertWordPieceTokenizer | 中文 | | - -## 介绍 - -自然语言通常以一连串符号的形式表示，我们将每个符号称为字符。 - -在自然语言文本里，每个语义单元(我们称之为token)都由一个或几个连续的字符组成。例如，“狐狸”， -作为一种杂食性的犬科动物，表示为“狐”和“狸”。 -将文本拆分为一连串语义单元的过程被定义为分词(Tokenization)，下面展示了一个例子： - - -原始语句: 小明在北京上学。 - -分词后的语句: [小明, 在, 北京, 上学, 。] - -需要注意的是切割的方法可以自由选择。语义单元可以是单字，也可以是更长的文本，只要其包含适量的语义信息就好。 - -另一个关键的预处理步骤是矢量化，它将原始token序列转换为数字矢量或矩阵，以便可以直接输入到我们的语言模型中。通常会使用一个字典文件将每个token映射到其相应的序号。 - -在我们的项目中，有一些进行分词的类，我们称为分词器。其中每个分词器都能以不同的方式对原始文本进行分词和矢量化。 - -## 加载分词器 -```python -from flagai.data.tokenizer import GLMLargeEnWordPieceTokenizer -tokenizer = GLMLargeEnWordPieceTokenizer() -``` - - -## 创建分词器 -想创建新的分词器的时候, 需要这么做: -### 1. 在`/flagai/tokenizer`目录下建立一个新的目录 - -### 2. 在目录下新建一个python文件，在里面添加自定义的分词器 - -让我们以T5为例，实现一下自定义的分词器 - -```python -from transformers import T5Tokenizer -from ..tokenizer import Tokenizer -class T5BPETokenizer(Tokenizer): - def __init__(self, model_type_or_path="t5-base", cache_dir=None): - self.text_tokenizer = T5Tokenizer.from_pretrained(model_type_or_path, - cache_dir=cache_dir) - self.text_tokenizer.max_len = int(1e12) -``` -如果`model_type_or_path`这项参数的值已经指向了一个huggingface transformers里的分词器，那现在已经成功自定义分词器了！ - -否则，您需要自己在此类下方实现以下功能。 - - -```python -def EncodeAsIds(self, text: str, process_fn=None): - """输入文本 => 一个token序号列表""" - -def EncodeAsTokens(self, text: str, process_fn=None): - """输入文本 => 一个token列表""" - -def IdToToken(self, Id: int): - """Token序号 => token""" - -def TokenToId(self, token: str): - """Token => token序号""" - return self.text_tokenizer._convert_token_to_id(token) - -def DecodeIds(self, Ids: list[int]): - """一个token序号列表 => 对应的文本""" - return self.DecodeTokens([self.IdToToken(id) for id in Ids]) - -def DecodeTokens(self, tokens: list[str]): - """一个token列表 => 对应的文本""" - return self.text_tokenizer.convert_tokens_to_string(tokens) -``` \ No newline at end of file diff --git a/doc_zh/TUTORIAL_4_DATASET.md b/doc_zh/TUTORIAL_4_DATASET.md deleted file mode 100644 index 97b46276..00000000 --- a/doc_zh/TUTORIAL_4_DATASET.md +++ /dev/null @@ -1,139 +0,0 @@ -# Datasets - -## 支持的下游数据集列表 - -| 数据集名称 | 数据集简称 | 语言 | 所属评测基准 | -|----------------------------------------------|----------|-----|----------| -| Broadcoverage Diagnostics | BoolQ | 英文 | SuperGLUE | -| CommitmentBank | CB | 英文 | SuperGLUE | -| Choice of Plausible Alternatives | COPA | 英文 | SuperGLUE | -| Multi-Sentence Reading Comprehension | MultiRC | 英文 | SuperGLUE | -| Recognizing Textual Entailment | RTE | 英文 | SuperGLUE | -| Words in Context | WiC | 英文 | SuperGLUE | -| The Winograd Schema Challenge | WSC | 英文 | SuperGLUE | -| Ant Financial Question Matching Corpus | AFQMC | 中文 | CLUE | -| Short Text Classificaiton for News | TNEWS | 中文 | CLUE | -| Reading Comprehension for Simplified Chinese | CMRC2018 | 中文 | CLUE | - - -## Load datasets - -Let's load a SuperGlue Dataset as following: -让我们用如下所示的方法来加载CLUE测评基准里的AFQMC任务 - -```python -import torch.utils.data -from flagai.data.dataset import SuperGlueDataset -from flagai.data.tokenizer import GLMLargeChTokenizer -from tests.test_dataset_new_superglue import CollateArguments -from flagai.data.dataset import ConstructSuperglueStrategy - -# 得到默认参数 -cl_args = CollateArguments() - -# 创建GLM中文的tokenizer -tokenizer = GLMLargeChTokenizer(add_block_symbols=True, add_task_mask=False, - add_decoder_mask=False, fix_command_token=True) - -# 建立AFQMC的数据集 -dataset = SuperGlueDataset(task_name='afqmc', data_dir='./datasets/', dataset_type='train', - tokenizer=tokenizer) - -# 创建数据整理的函数 -collate_fn = ConstructSuperglueStrategy(cl_args, tokenizer, task_name="afqmc") - -# 创建加载器 -loader = torch.utils.data.DataLoader(dataset, - batch_size=1, - shuffle=False, - num_workers=1, - drop_last=False, - pin_memory=False, - collate_fn=collate_fn) - -# 对加载器进行迭代 -it = iter(loader) -next(it) -batch = next(it) - -# 打印结果信息 -print(batch['input_ids'].tolist()) -print(tokenizer.DecodeIds(batch['input_ids'].tolist()[0])) -print(tokenizer.DecodeIds(batch['target_ids'].tolist()[0])) -``` - -## 创建数据集 - -在flagai/data/dataset/superglue/control.py文件里, 如下所示创建自定义的processor和pvp函数， 然后把他们与自定义dataset的名字的映射关系添加到control.py里的PROCESSOR_DICT和PVPS两个字典里. -```python -class ExampleProcessor(DataProcessor): - - def get_train_examples(self, data_dir): - # Assign the filename of train set - return self._create_examples(os.path.join(data_dir, "train.tsv"), "train") - - def get_dev_examples(self, data_dir, for_train=False): - # Assign the filename of dev set - return self._create_examples(os.path.join(data_dir, "dev.tsv"), "dev") - - def get_test_examples(self, data_dir) -> List[InputExample]: - # Assign the filename of test set - return self._create_examples(os.path.join(data_dir, "test.tsv"), "test") - - def get_labels(self): - # Return all label categories - return ["0", "1"] - - @staticmethod - def _create_examples(path: str, set_type: str) -> List[InputExample]: - """ - InputExample包含下列信息 - text_a (str, required): 文本1 - text_b (str, optional): 文本2 - label (str, required): 标签 - guid (str, required): 每一个InputExample的唯一序号 - """ - examples = [] - df = read_tsv(path) - - for idx, row in df.iterrows(): - guid = f"{set_type}-{idx}" - text_a = punctuation_standardization(row['sentence']) - label = row.get('label', None) - example = InputExample(guid=guid, text_a=text_a, label=label) - examples.append(example) - return examples - - -class ExamplePVP(PVP): - # 把标签映射到对应的含义文本上 - VERBALIZER = {"0": ["中立"], - "1": ["利好"], - "2": ["利空"]} - - @staticmethod - def available_patterns(): - # 输出所有可用的模板 - return [0] - - @property - def is_multi_token(self): - # 如果标签包含大于一个token，就输出True - return True - - def get_parts(self, example: InputExample) -> FilledPattern: - # 把InputExample里面的元素通过设计组合成一个完形填空模板 - text_a= self.shortenable(example.text_a) - if self.pattern_id == 0: - return ["标题：", text_a, "类别：", [self.mask]], [] - else: - raise NotImplementedError("No pattern implemented for id {}".format(self.pattern_id)) - - def verbalize(self, label) -> List[str]: - if self.pattern_id == 0: - return WankePVP.VERBALIZER_A[label] - else: - raise NotImplementedError -``` -关于为什么要建立processor和pvp， 以及具体怎么设计样例可以参考 [这里](APPENDIX_TASK.md). - diff --git a/doc_zh/TUTORIAL_4_TRAINER.md b/doc_zh/TUTORIAL_4_TRAINER.md new file mode 100644 index 00000000..cefacefb --- /dev/null +++ b/doc_zh/TUTORIAL_4_TRAINER.md @@ -0,0 +1,291 @@ +# Trainer + +Trainer 类提供了API用于多种并行框架的训练。API 支持在多个 GPU上使用Pytorch DDP/Deepspeed进行分布式训练，同时支持Megatron-LM+Deepspeed的混合并行分布式训练，同时也通过 NVIDIA Apex 实现混合精度。 +## 入门 +Trainer 包含支持上述功能的基本训练循环。Trainer使用分成两个步骤，初始化和调用. 参考 `examples/glm_superglue` 目录下的代码. + +## 自定义Trainer +要自定义Trainer来快速支持自定义模型，您可以对它们进行子类化并覆盖forward_step方法： +forward_step方法的返回是一个dict +```python +from flagai.trainer import Trainer +class MyTrainer(Trainer): + + def forward_step(self, data, model, mems): + + model_outputs = model(**data) + output = {} + output['loss'] = model_outputs.loss + output['logits'] = model_outputs.logits + output['hidden_states'] = model_outputs.decoder_hidden_states + return output +``` + +Trainer 中有一个控制是否分布式训练的参数 env_type， +```shell +The enviroment type for training. Will default to 'pytorch'. +env_type: `pytorch`, `pytorchDDP`, `deepspeed`, `deepspeed+mpu` + pytorch: single node cpu/gpu + pytorchDDP: single-/multi- node gpu + deepspeed: single-/multi- node gpu + deepspeed+mpu: single-/multi- node gpu +``` + +## 单节点cpu/gpu +```python +trainer = MyTrainer( + env_type='pytorch', + epochs=1, + batch_size=4, + eval_interval=100000, + log_interval=10, + experiment_name='t5-11b', + pytorch_device='cpu', + load_dir=None, + lr=1e-4) +``` +指定显卡/cpu的设置为pytorch_device, 'cpu', 'cuda:0'等 +## fp16 +```python +Model parameters turned to fp16 +trainer = MyTrainer( + env_type='pytorch', + epochs=1, + batch_size=1, + eval_interval=10, + log_interval=10, + experiment_name='t5-11b', + pytorch_device='cuda:0', + load_dir=None, + lr=1e-4, + fp16=True) # change to `True` +``` +## 梯度重计算 +在前向过程中不保存中间变量，节约GPU显存. Paper: [Training Deep Nets with Sublinear Memory Cost](https://arxiv.org/abs/1604.06174v2) + +```python +from transformers import T5ForConditionalGeneration, T5Tokenizer +tokenizer = T5Tokenizer.from_pretrained('t5-11b') +model = T5ForConditionalGeneration.from_pretrained('t5-11b') +model.gradient_checkpointing = True +``` +## 支持huggingface model +FlagAI 项目的example目录位置：examples/t5_huggingface +t5-11b paper: [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://arxiv.org/pdf/1910.10683.pdf) + +The example train t5-11b on a toy dataset. You need a device > 30G memory to run the example. If your have multiple devices, please check out the following session for speedup. +```python +import sys +from flagai.trainer import Trainer +from transformers import T5ForConditionalGeneration, T5Tokenizer +from torch.utils.data import Dataset +import torch + +## Inheriant the Trainer +## overload the forward_step function +class MyTrainer(Trainer): + + def forward_step(self, data, model, mems): + """ + Args: + data: a dict contains a batch of inputs + return: + output: a dict contains `loss` + """ + model_outputs = model(**data) + output = {} + output['loss'] = model_outputs.loss + output['logits'] = model_outputs.logits + output['hidden_states'] = model_outputs.decoder_hidden_states + return output + +# get a customized trainer instance +trainer = MyTrainer( + env_type='pytorch', + epochs=1, + batch_size=4, + eval_interval=10, + log_interval=10, + experiment_name='t5-11b', + pytorch_device='cuda:0', + load_dir=None, + lr=1e-4, + fp16=False) + +# using huggingface transformers to get tokenizer and models +model_name = 't5-11b' +tokenizer = T5Tokenizer.from_pretrained(model_name) +model = T5ForConditionalGeneration.from_pretrained(model_name) + +print("loading model & tokenizer is done!") +src_dir = 'train_inputs.txt' +tgt_dir = 'train_targets.txt' +model_dir = "./t5-11b" # 模型位置 +maxlen = 1024 + + +def read_file(): + src = [] + tgt = [] + + with open(src_dir, 'r', encoding='utf-8') as f: + lines = f.readlines() + for line in lines: + src.append(line.strip('\n').lower()) + + with open(tgt_dir, 'r', encoding='utf-8') as f: + lines = f.readlines() + for line in lines: + tgt.append(line.strip('\n').lower()) + return src, tgt + + +class BertSeq2seqDataset(Dataset): + + def __init__(self, sents_src, sents_tgt, tokenizer, maxlen=512): + super(BertSeq2seqDataset, self).__init__() + self.sents_src = sents_src + self.sents_tgt = sents_tgt + self.tokenizer = tokenizer + self.maxlen = maxlen + + def __getitem__(self, i): + src = self.sents_src[i] + tgt = self.sents_tgt[i] + inputs = tokenizer(src) + with tokenizer.as_target_tokenizer(): + labels = tokenizer(tgt) + output = {} + output['input_ids'] = inputs.input_ids + output['labels'] = labels.input_ids + return output + + def __len__(self): + return len(self.sents_src) + + +def seq2seq_collate_fn(batch): + + def padding(indice, max_length, pad_idx=0): + + pad_indice = [ + item + [pad_idx] * max(0, max_length - len(item)) + for item in indice + ] + return torch.tensor(pad_indice) + + token_ids = [data["input_ids"] for data in batch] + max_length_tk = max([len(t) for t in token_ids]) + labels = [data["labels"] for data in batch] + max_length_lb = max([len(t) for t in labels]) + + token_ids_padded = padding(token_ids, max_length_tk) + labels_padded = padding(labels, max_length_lb) + + data = {"input_ids": token_ids_padded, "labels": labels_padded} + + return data + + +sents_src, sents_tgt = read_file() +data_len = len(sents_tgt) +train_size = int(data_len * 0.8) +train_src = sents_src[:train_size] +train_tgt = sents_tgt[:train_size] + +val_src = sents_src[train_size:] +val_tgt = sents_tgt[train_size:] + +train_dataset = BertSeq2seqDataset(train_src, + train_tgt, + tokenizer=tokenizer, + maxlen=maxlen) +val_dataset = BertSeq2seqDataset(val_src, + val_tgt, + tokenizer=tokenizer, + maxlen=maxlen) +## Training +trainer.train(model, + train_dataset=train_dataset, + collate_fn=seq2seq_collate_fn) +``` + +## 分布式训练 +To accelerate your training/finetuning process, FlagAI integrate three popular parallel frameworks for deep neural networks. +### pytorchDDP +DistributedDataParallel (DDP) link +我们可以通过修改几个参数就能使用分布式的训练，内部直接封装了launch的调用，直接python script_name.py [script_name:是训练脚本的名字] 就可以直接运行。 +```python +trainer = MyTrainer( + env_type='pytorchDDP', + epochs=1, + batch_size=4, + eval_interval=10, + log_interval=10, + experiment_name='t5-11b', + load_dir=None, + lr=1e-4 + # parameters for pytorchDDP + master_ip='127.0.0.1', + master_port=17750, + num_nodes=1, + num_gpus=2, + hostfile='./hostfile', + training_script=__file__, +) +``` +### deepspeed +deepspeed相对于pytorch主要在优化器方面做了优化。 +其提供了cpu-offload的optimizer，可以极大的降低gpu显存的占用。 +对deepspeed优化器的相关设置，需要提供deepspeed.json的配置文件 +```python +trainer = MyTrainer( + env_type='pytorchDDP', + epochs=1, + batch_size=4, + eval_interval=10, + log_interval=10, + experiment_name='t5-11b', + load_dir=None, + lr=1e-4 + # parameters for pytorchDDP + master_ip='127.0.0.1', + master_port=17750, + num_nodes=1, + num_gpus=2, + hostfile='./hostfile', + training_script=__file__, + # deepspeed + deepspeed_config='deepspeed.json' +) +``` +Deepspeed config文件参考 [examples/t5_huggingface/deepspeed.json] +其中，主要的参数为 stage 和 cpu offload +### deepspeed + megatron-lm +现在百亿级模型GLM采用了Megatron-LM的模型并行技术。在模型参数scaling到10b以上级别，单卡的显存就很难将单个模型以及训练时的中间变量全部加载进来。为此，Megatron-LM提供了Tensor的切分方法，主要思想是将矩阵按照行/列进行切分。 FlagAI link，将model转化为Megatron-LM版本。 +如下，FlagAI内部模型（GLM，T5，BERT【包括RoBERTa】，GPT2）支持了Megatron-LM，只要在配置文件中将环境变量修改为deepspeed+mpu，就能启动模型并行的功能。 +对于huggingface版本的模型，暂时没有提供模型并行的支持。 +```python +trainer = MyTrainer( + env_type="deepspeed+mpu", # env_type + epochs=1, + batch_size=8, + eval_interval=10, + log_interval=10, + experiment_name='t5-11b', + load_dir=None, + lr=1e-4, + # parallel settings + master_ip='127.0.0.1', + master_port=17750, + num_nodes=1, + num_gpus=4, + hostfile='hostfile', + training_script=__file__, + # deepspeed + deepspeed_config='deepspeed.json', + # megatron-lm + model_paralle_size = 2 +) +``` + diff --git a/doc_zh/TUTORIAL_5_INSTRUCTIONS_FOR_AutoLoader.md b/doc_zh/TUTORIAL_5_INSTRUCTIONS_FOR_AutoLoader.md new file mode 100644 index 00000000..8c5cb5a8 --- /dev/null +++ b/doc_zh/TUTORIAL_5_INSTRUCTIONS_FOR_AutoLoader.md @@ -0,0 +1,86 @@ +## AutoLoader + +### 通过 AutoLoader 快速构建支持不同下游任务的大型模型。 +![](./img/autoloader_map.png) + +AutoLoader 可以快速找到对应的预训练模型和分词器，只需输入 task_name 和 model_name 即可。 + +以标题生成任务为例： + +```python +from flagai.auto_model.auto_loader import AutoLoader +auto_loader = AutoLoader(task_name="title-generation", ## The task name + model_name="RoBERTa-base-ch", ## The model name. + ) +model = auto_loader.get_model() +tokenizer = auto_loader.get_tokenizer() +``` +```task_name``` 你要想做的任务名字, 除了 ```title-generation```, 你也可以选择其他的细分任务, 例如 ```semantic-matching```、```ner``` and so on. +```python +target = ["O", "B-LOC", "I-LOC", "B-ORG", "I-ORG", "B-PER", "I-PER"] +from flagai.auto_model.auto_loader import AutoLoader +auto_loader = AutoLoader(task_name="ner", + model_name="RoBERTa-base-ch", + class_num=len(target)) +model = auto_loader.get_model() +tokenizer = auto_loader.get_tokenizer() +``` +在构建分类相关任务的模型时，还需要输入一个“class_num”参数来告诉模型要分类多少类。 + +然后 AutoLoader 将从模型中心下载 roberta 预训练模型、配置和词表。 +并且下载的预训练模型、配置和词表将被放入“./state_dict/RoBERTa-base-ch”目录中。 + +另外，通过输入不同的model_name，也可以直接调用已经训练好的下游任务模型，如``Roberta-base-ch-ner``、``Roberta-base-ch-title-generation``等。 +```python +target = ["O", "B-LOC", "I-LOC", "B-ORG", "I-ORG", "B-PER", "I-PER"] +from flagai.auto_model.auto_loader import AutoLoader +auto_loader = AutoLoader(task_name="ner", + model_name="RoBERTa-base-ch-ner", + class_num=len(target)) + +model = auto_loader.get_model() +tokenizer = auto_loader.get_tokenizer() +``` +查看 https://github.com/BAAI-Open/FlagAI/blob/master/quickstart/ner_ch.py 了解更多. +## All supported tasks and models + +![model_and_task_table](../docs/img/model_task_table.png) + **task_name** 参数支持: +### 分类任务 +1. classification +2. semantic-matching +3. emotion-analysis (todo) +4. text-classification (todo) +5. ... + +### 生成任务 +1. seq2seq +2. title-generation +3. writing +4. poetry-generation +5. couplets-generation (todo) +6. ... +### 序列标注任务 +1. sequence-labeling +2. ner +3. ner-crf +4. ner-gp +5. part-speech-tagging (todo) +6. chinese-word-segmentation (todo) +7. ... + +## 所有支持的模型 +所有支持的模型都可以在 **model hub** 中找到。 +不同的模型适应不同的任务。 + +#### Transfomrer encoder: + +例如，model_name="GLM-large-ch" 或 "RoBERTa-base-ch" 这些模型支持上一节中提到的所有任务。 + +#### Transformer decoder: + +例如，model_name="GPT2-base-ch"，模型支持“seq2seq”相关任务。 + +#### Transformer encoder + decoder: + +例如model_name="t5-base-ch"，模型支持“seq2seq”相关任务。 \ No newline at end of file diff --git a/doc_zh/TUTORIAL_6_INSTRUCTIONS_FOR_PREDICTOR.md b/doc_zh/TUTORIAL_6_INSTRUCTIONS_FOR_PREDICTOR.md new file mode 100644 index 00000000..46b11b4b --- /dev/null +++ b/doc_zh/TUTORIAL_6_INSTRUCTIONS_FOR_PREDICTOR.md @@ -0,0 +1,99 @@ +## Predictor + +### 通过Predictor调用一行代码得到理想的预测结果 + +对于不同任务，例如标题生成，命名实体识别，文本分类等，预测的方式各不相同。同时，对于不同的模型，例如encoder、decoder、encoder-decoder模型，预测方式也各不相同。 + +Predictor中集成了不同任务，不同模型的预测代码，通过Pipline的方式传入文本，Predictor会快速解析模型类型，调用不同的预测代码，得到对应模型的预测结果。 + +![](./img/predictor_map.png) + +以gpt2文章续写任务为例，采用随机采样的生成方式： +```python +from flagai.auto_model.auto_loader import AutoLoader +from flagai.model.predictor.predictor import Predictor +import torch +if __name__ == '__main__': + device = torch.device("cuda" if torch.cuda.is_available() else "cpu") + + ## 通过autoloader加载模型和tokenizer + loader = AutoLoader(task_name="writing", + model_name="GPT2-base-ch") + model = loader.get_model() + tokenizer = loader.get_tokenizer() + model.to(device) + ## 定义预测器 + predictor = Predictor(model, tokenizer) + ## 定义文章的开头，作为输入 + text = "今天天气不错，" + ## 预测器能通过自动分析模型种类来调取不同方法 + out = predictor.predict_generate_randomsample(text, ## 输入 + input_max_length=512, ## 最大出入长度 + out_max_length=100, ## 最大输出长度 + repetition_penalty=1.5, ## 避免重复输出. (https://arxiv.org/pdf/1909.05858.pdf) + top_k=20, ## 只保留概率最大的k个token. + top_p=0.8) ## 保留累计概率大于等于top_p的token.(http://arxiv.org/abs/1904.09751) + + print(f"out is {out}") + ### out is 到这里来看了一下，很是兴奋，就和朋友一起来这里来了。我们是周五晚上去的，人不多，所以没有排队，而且这里的环境真的很好，在这里享受美食真的很舒服，我们点了一个套餐，两个人吃刚刚好，味道很好。 +``` +Predictor可以自动分析出模型类型为GPT2，并且自动调用GPT2模型对应的生成方法，得到预测结果。 + +除了```writing```任务之外，Predictor还支持```title-generation```, ``ner``, ``semantic-matching``等任务的预测。例如ner任务： + +```python +import torch +from flagai.auto_model.auto_loader import AutoLoader +from flagai.model.predictor.predictor import Predictor +device = torch.device("cuda" if torch.cuda.is_available() else "cpu") + +target = ["O", "B-LOC", "I-LOC", "B-ORG", "I-ORG", "B-PER", "I-PER"] +auto_loader = AutoLoader(task_name="ner", + model_name="RoBERTa-base-ch-ner", # Load the checkpoints from Modelhub(model.baai.ac.cn/models) + class_num=len(target)) + +model = auto_loader.get_model() +tokenizer = auto_loader.get_tokenizer() +model.to(device) + +predictor = Predictor(model, tokenizer) + +test_data = [ + "6月15日，河南省文物考古研究所曹操高陵文物队公开发表声明承认：“从来没有说过出土的珠子是墓主人的", + "4月8日，北京冬奥会、冬残奥会总结表彰大会在人民大会堂隆重举行。习近平总书记出席大会并发表重要讲话。在讲话中，总书记充分肯定了北京冬奥会、冬残奥会取得的优异成绩，全面回顾了7年筹办备赛的不凡历程，深入总结了筹备举办北京冬奥会、冬残奥会的宝贵经验，深刻阐释了北京冬奥精神，对运用好冬奥遗产推动高质量发展提出明确要求。", + "当地时间8日，欧盟委员会表示，欧盟各成员国政府现已冻结共计约300亿欧元与俄罗斯寡头及其他被制裁的俄方人员有关的资产。", + "这一盘口状态下英国必发公司亚洲盘交易数据显示博洛尼亚热。而从欧赔投注看，也是主队热。巴勒莫两连败，", +] + +for t in test_data: + entities = predictor.predict_ner(t, target, maxlen=256) + result = {} + for e in entities: + if e[2] not in result: + result[e[2]] = [t[e[0]:e[1] + 1]] + else: + result[e[2]].append(t[e[0]:e[1] + 1]) + print(f"result is {result}") +``` +通过传入一行文本，快速得到对应的命名实体识别任务结果，并且``predict_ner`` 接口适配所有支持ner任务的模型，例如BERT, Roberta, BERT-CRF, BERT-GlobalPointer, Roberta-CRF, Roberta-GlobalPointer 等等。 + +![predictor-table](../docs/img/predictor_table.png) + + +### Predictor所有支持的方法 +#### 文本表征 +1. predict_embedding: 输入一个文本来获取嵌入表征，支持bert、roberta等模型。 +#### 文本分类，语义匹配 +1. predict_cls_classifier: 输入文本或文本对得到多分类结果，支持bert、roberta等transformer编码器模型。 +#### Mask语言模型 +1. predict_masklm: 输入带有[MASK]标记的文本得到原文结果，支持bert、roberta等transformer编码器模型 +#### 命名实体识别 +1. predict_ner: 输入文本得到ner结果，支持bert、roberta等transformer编码器模型。 +#### 生成 +1. predict_generate_beamsearch: 输入文本得到输出文本，属于seq2seq任务。支持bert、roberta、gpt2、t5和glm模型。 +2. predict_generate_randomsample: 输入文本得到输出文本，属于seq2seq任务。支持bert、roberta、gpt2、t5和glm模型。 + +### 方法调用说明 +只要所使用的模型支持Predictor的对应方法，便可以直接预测，例如：模型为GLM、T5、GPT2，因为便可以直接调用不同生成方法，但是无法调用其他例如文本分类，命名实体识别预测方法。 + +Bert、Roberta模型支持的方法较多，可以调用上部分所展示的所有方法。 \ No newline at end of file diff --git a/doc_zh/TUTORIAL_7_PROMPT_LEARNING.md b/doc_zh/TUTORIAL_7_PROMPT_LEARNING.md new file mode 100644 index 00000000..ada0e194 --- /dev/null +++ b/doc_zh/TUTORIAL_7_PROMPT_LEARNING.md @@ -0,0 +1,61 @@ +# 提示学习介绍 + +## 背景 +提示学习(prompt-learning)是一种自然语言处理 (NLP) 领域广泛使用的范式，尤其是在数据有限的情况下表现良好。它遵循预训练-微调范式的思想：即预训练好的语言模型将被适配到各种下游任务上。不过与微调不同的是，提示学习会将原本的任务转化为一个特定的模板，这个过程被称为提示工程。 + +## 提示工程 +提示工程中包含两种主要方法：完形填空提示和前缀提示。 完形填空提示更适合使用遮挡类语言模型的下游任务，而前缀提示通常用于文本生成任务。 + +### 完形填空提示 +通常来说，输入文本和标签可以从任务数据集中获得。这些数据将被重新排列成一个完形填空式的短语(包括一个带有空格的段落以及空格处可以填充的选项)，这包括两个步骤： +1. 构建一个由输入部分、答案部分和一些自然语言组成的模板。 +2. 构建一个从原始的标签到可填选项的对应关系。 + +下面是一个完形填空提示的示例 + +数据集包含两个输入文本：前提和假设。标签表示前提和假设之间的关系，标签有三种可能的文本字符串：entailment（一致）、contradiction（矛盾）和neutral(无偏向)。 + +
+ +我们可以设计一个如下的模板： + +
+ +相关的代码实现如下： + +```python +def get_parts(self, example: InputExample): + '''A simplified version of the actual code''' + # Construct the template + cloze_ids = [ + example.text_a,' question: is it true, false or neither that', + example.text_b, ' answer:', [self.mask]] + return cloze_ids +``` + +各种数据集中的标签可以有多种类型，包括布尔值、整数和字符串。我们需要实现一个语言器，将标签映射到完形填空问题中的选项。例如，这里的“entailment”、“contradiction”和“neutral”标签可以映射到“true”、“false”和“neither”，并且每次要填的答案总是在这些选项中选择。 + +语言器本质上是一种映射关系, 所以我们可以用一个Python字典来定义它： +```python +VERBALIZER = { + "contradiction": [" false"], + "entailment": [" true"], + "neutral": [" neither"] +} +def verbalize(self, label) -> List[str]: + return CbPVP.VERBALIZER[label] +``` + +### 前缀提示 +前缀提示并没有用真正的自然语言设计模板，而是直接修改嵌入空间。如下图所示，它在Transformer里插入一个前缀向量，并冻结语言模型的其余参数。与完形填空提示对比，这么做相当于从离散的tokens变成了连续的向量，避免了人工设计模板的好坏对于模型性能的影响过大的问题。 +
+ + +## Reference +[Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://arxiv.org/abs/2107.13586) + +[OpenPrompt: An Open-source Framework for Prompt-learning](https://arxiv.org/abs/2111.01998) + +[Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://arxiv.org/abs/2001.07676) + +[Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://arxiv.org/abs/2101.00190) \ No newline at end of file diff --git a/doc_zh/EnvironmentSetup.md b/doc_zh/TUTORIAL_8_ENVIRONMENT_SETUP.md similarity index 100% rename from doc_zh/EnvironmentSetup.md rename to doc_zh/TUTORIAL_8_ENVIRONMENT_SETUP.md diff --git a/doc_zh/TUTORIAL_8_TRAINING.md b/doc_zh/TUTORIAL_8_TRAINING.md deleted file mode 100644 index 2acad3e0..00000000 --- a/doc_zh/TUTORIAL_8_TRAINING.md +++ /dev/null @@ -1,138 +0,0 @@ - -## 在GPU上微调模型 - -让我们按照如下所示的步骤来对CLUE上的AFQMC以及TNEWS数据集进行微调 - -```python -from easybigmodel.model.glm_model import GLMModel,GLMForSequenceClassification -from easybigmodel.data.tokenizer import GLMLargeEnWordPieceTokenizer, GLMLargeChTokenizer -from easybigmodel.metrics import * - -from easybigmodel.data.dataset import SuperGlueDataset -from easybigmodel.test_utils import CollateArguments -from easybigmodel.data.dataset.superglue.control import DEFAULT_METRICS, MULTI_TOKEN_TASKS, CH_TASKS -import unittest -from easybigmodel.data.dataset import ConstructSuperglueStrategy -for task_name in ['afqmc', 'tnews']: - trainer = Trainer(env_type='pytorch', - epochs=1, - batch_size=4, - eval_interval=100, - log_interval=50, - experiment_name='glm_large', - pytorch_device='cuda', - load_dir=None, - lr=1e-4, - save_epoch=10 - ) - print("downloading...") - - cl_args = CollateArguments() - cl_args.multi_token = task_name in MULTI_TOKEN_TASKS - - model_name='GLM-large-ch' - tokenizer = GLMLargeChTokenizer(add_block_symbols=True, add_task_mask=False, - add_decoder_mask=False, fix_command_token=True) - - - if cl_args.multi_token: - model = GLMForMultiTokenCloze.from_pretrain(model_name=model_name) - else: - model = GLMForSingleTokenCloze.from_pretrain(model_name=model_name) - - train_dataset = SuperGlueDataset(task_name=task_name, data_dir='./datasets/', dataset_type='train', - tokenizer=tokenizer) - train_dataset.example_list = train_dataset.example_list[:2] - - collate_fn = ConstructSuperglueStrategy(cl_args, tokenizer, task_name=task_name) - - valid_dataset = SuperGlueDataset(task_name=task_name, data_dir='/mnt/datasets/yan/', dataset_type='dev', - tokenizer=tokenizer) - valid_dataset.example_list = valid_dataset.example_list[:2] - - metric_methods = DEFAULT_METRICS[task_name] - trainer.train(model, collate_fn=collate_fn, - train_dataset=train_dataset, valid_dataset=valid_dataset, - metric_methods=metric_methods) -``` - -```shell ->>> python train.py -``` - - -## 在deepspeed上进行微调 - -变更到deepspeed只需要在trainer里修改一些参数即可 - -```python - trainer = Trainer(env_type='deepspeed', # change env_type - epochs=1, - batch_size=4, - eval_interval=100, - log_interval=50, - experiment_name='glm_large', - load_dir=None - - # parallel settings - master_ip='127.0.0.1', - master_port=17750, - num_nodes=1, - num_gpus=2, - hostfile='hostfile', - training_script=__file__, - # deepspeed - deepspeed_config='deepspeed.json') -``` -```shell ->>> python train.py -``` -hostfile: 所有节点的设置ip -```shell -127.0.0.1 slots=2 # slots 代表可用的GPU数量 -``` -deepspeed_config: deepspeed的参数设置放在这儿 -```json -{ - "train_micro_batch_size_per_gpu": 4, - "gradient_accumulation_steps": 1, - "steps_per_print": 50, - "gradient_clipping": 1.0, - "zero_optimization": { - "stage": 1, # important - "contiguous_gradients": false, - "overlap_comm": true, - "reduce_scatter": true, - "reduce_bucket_size": 5e7, - "allgather_bucket_size": 5e7, - "cpu_offload": true # important - }, - "zero_allow_untested_optimizer": true, - "fp16": { - "enabled": true, - "loss_scale": 0, - "loss_scale_window": 1000, - "hysteresis": 2, - "min_loss_scale": 1 - }, - "optimizer": { - "type": "Adam", - "params": { - "lr": 5e-6, - "betas": [ - 0.9, - 0.95 - ], - "eps": 1e-8, - "weight_decay": 1e-2 - } - }, - "activation_checkpointing": { - "partition_activations": true, - "contiguous_memory_optimization": false - }, - "wall_clock_breakdown": false, -} - - -``` diff --git a/doc_zh/Seq2seqMethod.md b/doc_zh/TUTORIAL_9_SEQ2SEQ_METHOD.md similarity index 100% rename from doc_zh/Seq2seqMethod.md rename to doc_zh/TUTORIAL_9_SEQ2SEQ_METHOD.md diff --git a/doc_zh/Tutorial_3_TASK.md b/doc_zh/Tutorial_3_TASK.md deleted file mode 100644 index f8c74ce0..00000000 --- a/doc_zh/Tutorial_3_TASK.md +++ /dev/null @@ -1,55 +0,0 @@ -# 下游任务 -高质量的预训练模型可以极大地增强各种NLP任务。 -起初，研究人员只需添加一个分类层，就可以将预训练模型与下游任务的标签相匹配。 - -## 概要 - -## Pattern-Exploiting Training（PET） - -正如 [Exploiting Cloze Questions for Few Shot Text Classification and Natural -Language Inference](https://arxiv.org/abs/2001.07676) 所提出， -PET可以将输入文本重建为完形填空式短语，这使得预先训练的模型更容易理解给定的任务。 - -由于不同任务之间输入文本和标签的设置不同，我们需要指定适当的完形填空风格模式。 -本文介绍了一个情绪分类示例，如下所示。 - -给定的情绪分析任务是：给定输入上下文（例如'Best pizza ever!'）和 -有限类别（这里有两个类别，1和-1分别表示积极和消极）。 -我们的模型需要根据上下文自动预测每个情绪类别标签的概率。 - -![result1](../docs/img/pet_example.png) - -如图所示，首先，正标签和负标签分别表示为“好”和“坏”。 -然后，该模式被设计为： 上下文 + "它是" + mask情绪标签+ "."。 -将“好”和“坏”填充到mask位置的概率将由finetune模型返回。 - -这里我们可以看到，它要求用户为每个给定任务设计一个pattern-verbalizer（PVP）。 -在我们的项目中，这部分放在 easybigmodel/data/dataset/superglue/pvp中。 -在给定任务的类下，通常存在两个名为“verbalize”和“get_parts”的函数， -它们代表verbalizer和模式设计步骤。 -下面是一个例子,请注意，对于一个任务，可以有多种模式。 -``` - def get_parts(self, example: InputExample) -> FilledPattern: - - text_a = self.shortenable(example.text_a) - text_b = self.shortenable(example.text_b) - - if self.pattern_id == 0: - return [[self.mask], ':', text_a, text_b], [] - elif self.pattern_id == 1: - return [[self.mask], ' Question:', text_a, text_b], [] - elif self.pattern_id == 2: - return [text_a, '(', [self.mask], ')', text_b], [] - elif self.pattern_id == 3: - return [text_a, text_b, '(', [self.mask], ')'], [] - elif self.pattern_id == 4: - return ['[ Category:', [self.mask], ']', text_a, text_b], [] - elif self.pattern_id == 5: - return [[self.mask], '-', text_a, text_b], [] - else: - raise ValueError("No pattern implemented for id {}".format( - self.pattern_id)) - - def verbalize(self, label) -> List[str]: - return YahooPVP.VERBALIZER[label] -``` \ No newline at end of file diff --git a/doc_zh/img/TUTORIAL_7_PROMPT_LEARNING.md b/doc_zh/img/TUTORIAL_7_PROMPT_LEARNING.md new file mode 100644 index 00000000..0ada57ed --- /dev/null +++ b/doc_zh/img/TUTORIAL_7_PROMPT_LEARNING.md @@ -0,0 +1,61 @@ +# Introduction to Prompt Learning + +## Background +提示学习(prompt-learning)是一种自然语言处理 (NLP) 领域广泛使用的范式，尤其是在数据有限的情况下表现良好。它遵循预训练-微调范式的思想：即预训练好的语言模型将被适配到各种下游任务上。不过与微调不同的是，提示学习会将原本的任务转化为一个特定的模板，这个过程被称为提示工程。 + +## 提示工程 +提示工程中包含两种主要方法：完形填空提示和前缀提示。 完形填空提示更适合使用遮挡类语言模型的下游任务，而前缀提示通常用于文本生成任务。 + +### Cloze Prompts +通常来说，输入文本和标签可以从任务数据集中获得。这些数据将被重新排列成一个完形填空式的短语(包括一个带有空格的段落以及空格处可以填充的选项)，这包括两个步骤： +1. 构建一个由输入部分、答案部分和一些自然语言组成的模板。 +2. 构建一个从原始的标签到可填选项的对应关系。 + +下面是一个完形填空提示的示例 + +数据集包含两个输入文本：前提和假设。标签表示前提和假设之间的关系，标签有三种可能的文本字符串：entailment（一致）、contradiction（矛盾）和neutral(无偏向)。 + +
+ +我们可以设计一个如下的模板： + +
+ +相关的代码实现如下： + +```python +def get_parts(self, example: InputExample): + '''A simplified version of the actual code''' + # Construct the template + cloze_ids = [ + example.text_a,' question: is it true, false or neither that', + example.text_b, ' answer:', [self.mask]] + return cloze_ids +``` + +各种数据集中的标签可以有多种类型，包括布尔值、整数和字符串。我们需要实现一个语言器，将标签映射到完形填空问题中的选项。例如，这里的“entailment”、“contradiction”和“neutral”标签可以映射到“true”、“false”和“neither”，并且每次要填的答案总是在这些选项中选择。 + +语言器本质上是一种映射关系, 所以我们可以用一个Python字典来定义它： +```python +VERBALIZER = { + "contradiction": [" false"], + "entailment": [" true"], + "neutral": [" neither"] +} +def verbalize(self, label) -> List[str]: + return CbPVP.VERBALIZER[label] +``` + +### 前缀提示 +前缀提示并没有用真正的自然语言设计模板，而是直接修改嵌入空间。如下图所示，它在Transformer里插入一个前缀向量，并冻结语言模型的其余参数。与完形填空提示对比，这么做相当于从离散的tokens变成了连续的向量，避免了人工设计模板的好坏对于模型性能的影响过大的问题。 +
+ + +## Reference +[Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://arxiv.org/abs/2107.13586) + +[OpenPrompt: An Open-source Framework for Prompt-learning](https://arxiv.org/abs/2111.01998) + +[Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://arxiv.org/abs/2001.07676) + +[Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://arxiv.org/abs/2101.00190) \ No newline at end of file diff --git a/doc_zh/img/autoloader_en_map.png b/doc_zh/img/autoloader_en_map.png new file mode 100644 index 00000000..2c39c403 Binary files /dev/null and b/doc_zh/img/autoloader_en_map.png differ diff --git a/doc_zh/img/autoloader_map.png b/doc_zh/img/autoloader_map.png new file mode 100644 index 00000000..91b495da Binary files /dev/null and b/doc_zh/img/autoloader_map.png differ diff --git a/doc_zh/img/dataset_example_1.png b/doc_zh/img/dataset_example_1.png new file mode 100644 index 00000000..e53e414d Binary files /dev/null and b/doc_zh/img/dataset_example_1.png differ diff --git a/doc_zh/img/dataset_example_2.png b/doc_zh/img/dataset_example_2.png new file mode 100644 index 00000000..2130f4b7 Binary files /dev/null and b/doc_zh/img/dataset_example_2.png differ diff --git a/doc_zh/img/dataset_example_3.png b/doc_zh/img/dataset_example_3.png new file mode 100644 index 00000000..8dd35015 Binary files /dev/null and b/doc_zh/img/dataset_example_3.png differ diff --git a/doc_zh/img/dataset_example_4.png b/doc_zh/img/dataset_example_4.png new file mode 100644 index 00000000..894cc3a6 Binary files /dev/null and b/doc_zh/img/dataset_example_4.png differ diff --git a/doc_zh/img/dataset_example_5.png b/doc_zh/img/dataset_example_5.png new file mode 100644 index 00000000..d92a7bf2 Binary files /dev/null and b/doc_zh/img/dataset_example_5.png differ diff --git a/doc_zh/img/dataset_figure_0.png b/doc_zh/img/dataset_figure_0.png new file mode 100644 index 00000000..e53e414d Binary files /dev/null and b/doc_zh/img/dataset_figure_0.png differ diff --git a/doc_zh/img/dataset_figure_2.png b/doc_zh/img/dataset_figure_2.png new file mode 100644 index 00000000..eb324904 Binary files /dev/null and b/doc_zh/img/dataset_figure_2.png differ diff --git a/doc_zh/img/dataset_figure_3.png b/doc_zh/img/dataset_figure_3.png new file mode 100644 index 00000000..be869428 Binary files /dev/null and b/doc_zh/img/dataset_figure_3.png differ diff --git a/doc_zh/img/dataset_figure_4.png b/doc_zh/img/dataset_figure_4.png new file mode 100644 index 00000000..c6942d29 Binary files /dev/null and b/doc_zh/img/dataset_figure_4.png differ diff --git a/doc_zh/img/dataset_pipeline.png b/doc_zh/img/dataset_pipeline.png new file mode 100644 index 00000000..0d3513f6 Binary files /dev/null and b/doc_zh/img/dataset_pipeline.png differ diff --git a/doc_zh/img/glm_blank_filling.png b/doc_zh/img/glm_blank_filling.png new file mode 100644 index 00000000..59477761 Binary files /dev/null and b/doc_zh/img/glm_blank_filling.png differ diff --git a/doc_zh/img/glm_example_1.png b/doc_zh/img/glm_example_1.png new file mode 100644 index 00000000..7e8c4adf Binary files /dev/null and b/doc_zh/img/glm_example_1.png differ diff --git a/doc_zh/img/glm_example_2.png b/doc_zh/img/glm_example_2.png new file mode 100644 index 00000000..0a66052d Binary files /dev/null and b/doc_zh/img/glm_example_2.png differ diff --git a/doc_zh/img/glm_example_3.png b/doc_zh/img/glm_example_3.png new file mode 100644 index 00000000..92edb4ee Binary files /dev/null and b/doc_zh/img/glm_example_3.png differ diff --git a/doc_zh/img/glm_example_4.png b/doc_zh/img/glm_example_4.png new file mode 100644 index 00000000..4256293e Binary files /dev/null and b/doc_zh/img/glm_example_4.png differ diff --git a/doc_zh/img/glm_example_5.png b/doc_zh/img/glm_example_5.png new file mode 100644 index 00000000..ef1612c1 Binary files /dev/null and b/doc_zh/img/glm_example_5.png differ diff --git a/doc_zh/img/glm_performance.png b/doc_zh/img/glm_performance.png new file mode 100644 index 00000000..054894f9 Binary files /dev/null and b/doc_zh/img/glm_performance.png differ diff --git a/doc_zh/img/glm_results2.png b/doc_zh/img/glm_results2.png new file mode 100644 index 00000000..ec326125 Binary files /dev/null and b/doc_zh/img/glm_results2.png differ diff --git a/doc_zh/img/predictor_map.png b/doc_zh/img/predictor_map.png new file mode 100644 index 00000000..d112c25a Binary files /dev/null and b/doc_zh/img/predictor_map.png differ diff --git a/doc_zh/img/prompt_figure_1.png b/doc_zh/img/prompt_figure_1.png new file mode 100644 index 00000000..28dbd5fd Binary files /dev/null and b/doc_zh/img/prompt_figure_1.png differ diff --git a/doc_zh/img/prompt_figure_2.png b/doc_zh/img/prompt_figure_2.png new file mode 100644 index 00000000..b8cf0ea9 Binary files /dev/null and b/doc_zh/img/prompt_figure_2.png differ diff --git a/doc_zh/img/tokenizer_example_1.png b/doc_zh/img/tokenizer_example_1.png new file mode 100644 index 00000000..a84bb8bc Binary files /dev/null and b/doc_zh/img/tokenizer_example_1.png differ diff --git a/doc_zh/tokenization.md b/doc_zh/tokenization.md new file mode 100644 index 00000000..f22461f7 --- /dev/null +++ b/doc_zh/tokenization.md @@ -0,0 +1,246 @@ +[TOC] + +# sub-word algorithm + +## Tokenization + +​ 在自然语言处理中，大部分的数据都是生语料（文本），但是**机器是无法理解文本的**，我们将句子序列送入模型时，模型仅仅只能看到一串字节，它无法知道一个词从哪里开始，到哪里结束，所以也不知道一个词是由哪些字节组成的。 + +​ Tokenizer 的作用就是将文本拆分为单词或者子词（sub-word），然后通过词表将它们转换为其 id，这样送入模型的就是一个 id 的序列。如果词表已经存在，把单词或者子词转为 id 是非常容易的，所以 tokenization 这一步的难点在于，如何把文本拆分成词语或者子词。 + + + +## word-base tokenization + +​ 把一段文本切分成 token 的方法有非常多，举个例子，看一下下面这个句子`Let's tokenize! Isn't this easy?` + +​ 最简单的方法可以根据空格进行切分，结果如下： + +``` +["Let's", 'tokenize!', "Isn't", 'this', 'easy?'] +``` + +​ 这种方法非常直观，但是是有一个问题，我们可以看到在结果中包含这样的 token: `tokenize!` 和 `easy?`，标点连在单词上，这就会导致一个问题，如果我们不把标点连着词的token放入词表，那么这些在词表中未出现的词就会被当做OOV（out of vocabulary）处理，用``等特殊标记替代，这就损失了一部分信息，这样会非常影响模型的表现，但是如果把出现的所有标点连着词的token全部放入词表，那么可以想象词表会非常庞大（进而导致模型非常庞大），产生巨大的计算开销的同时，模型收敛也会变得困难。同时对于在词表中但是出现次数非常少的token，模型也很难学到其表示。 + +​ 所以标点符号也应该被考虑进到切分的规则中，这样就可以避免词表巨大而词表内部词频稀疏的问题。把这个句子进一步切分结果如下： + +``` +["Let", "'", "s", "tokenize", "!", "Isn", "'", "t", "this", "easy", "?"] +``` + +​ 比直接用空格分割好一点，但是还是有一些瑕疵，比如`Isn't`代表的是`Is not`，所以这个单词最好被切分为`["Is", "n't"]`。这就使得分词变得复杂了，这也是在NLP领域不同的模型使用了各种不同 Tokenizer 的原因之一。这也加大了研究者在复现这些工作的时候的难度。一个句子，使用不同的规则，将有许多种不同的分词结果。我们在使用预训练模型做下游任务的时候，只有使用和它训练时使用的同一种 tokenize 的方法处理数据，才能取得相应的效果。 + +​ 为了解决诸如`Isn't`这类词的分词问题，也出现了一些基于规则的分词方法，比如常用的基于规则的分词器 Spacy，这个句子通过这种分词方法切分的结果如下： + +``` +["Let", "'s", "tokenize", "!", "Is", "n't", "this", "easy", "?"] +``` + +​ 三种不同的切分方法效果对比如下： + +![3 different tokenization scheme: on rules, on punctuaction, on white spaces](tokenization.assets/tokenize.png) + +​ 上面说的三种分词方法，都是词级别的分词方法（word tokenization），简单来说，就是以词为单位对文本进行切分。尽管这是将文本切分为更小的单位的最直观的方式，但是这种方式在处理大量语料的时候会产生一个非常严重的问题——生成一个巨大的词表（因为大量的语料中会存在大量的只出现一次/少次的token）。 + +> ​ 比如 Transformer-XL 这个模型，就用的是基于空格和标点的分词方式，词表大小是夸张的267,735。这么大的词表会导致模型在输入端需要一个巨大的 embedding 矩阵（token id - embedding），在输出端需要一个巨大的 generator 矩阵（hidden value - token id），这会使得模型参数量剧增，导致大量的内存占用以及预测的时间复杂度。**总之，这一类模型的词表大小基本很少会超过50,000，特别是单语模型**。 + +​ 那么既然词级别的分词方法不理想，能不能基于字符级别来分词呢？ + + + +## character-base tokenization + +​ 基于字符级别的分词方法非常简单，简单来说，就是将每个字符看作一个词，还是上面那个句子，切分效果如下： + +``` +['L', 'e', 't', "'", 's', ' ', 't', 'o', 'k', 'e', 'n', 'i', 'z', 'e', '!', ' ', 'I', 's', 'n', "'", 't', ' ', 't', 'h', 'i', 's', ' ', 'e', 'a', 's', 'y', '?'] +``` + +​ 尽管这字符级别的分词方式非常简单并且能够大大缩小词表的大小（进而缩小模型的大小），并且不用担心未知词汇。但是这种表示方式，对于模型来说要学习出有意义的词表示难度大大增加了（字母本身就没有任何的内在含义）。 + +>举个例子，从字母“t”中学习到有意义的词表示要比从“today”中学习到有意的词表示难得多。 + +计算复杂度提升（字母的数目远大于token的数目），输出序列的长度将变大，对于Bert、CNN等限制最大长度的模型将很容易达到最大值。 + +​ 因此，字符级别的分词方式一般很难产生效果好的模型，所以为了兼顾到词级别和字符级别分词方式的好处，产生了一种折中的处理方式，称为子词切分方法（subword tokenization）。 + + + +## subword tokenization + +​ 上面提到的两种分词方式各有各的不足。 + +​ 词级别的分词：1、词表巨大；2、容易出现大量词表外的token（OOV问题）；3、对于相似的词语失去相似度信息（比如dog和dogs在词表中是两个完全不同的词） + +​ 字符级别的分词：1、文本转为 id 序列之后长度非常长；2、字母本身没有内在含义，从单一的字母中很难学习到有意的词表示。 + +​ 为了改进分词方法，在``数目和词表示含义丰富性之间达到平衡，于是产生了一个折中的算法：子词切分（subword tokenization），子词切分的方法也分好几种，但是它们都遵从两个原则： + +​ 1、常用的词语不应该被切分为更小的片段 + +​ 2、不常用的词语应该被分解为有意义的子词 + +> ​ 举个例子，`tokenization`这个词，可以被分解为`token`和`ization`，`token`和`ization`作为单独的子词出现的概率更高（比如`token`、`tokens`、`tokenizing`、`tokenization`，或者`tokenization`、`modernization`等等），同时`tokenization`的意思被作为`token`和`ization`的复合意思被保存下来。 + +​ 这种方式在英文上效果不错，在另外一些语言上效果甚至更好，在某些语言中，可以像搭积木一样，通过子词的组合形成几乎任何单词。 + +​ 子词切分方式使得模型能够在保持一个合理的词表大小的情况下，能够学习到更有意义的子词的表示，同时也能让模型拥有更好地处理词表中没有的词语的能力（将词表中没有的词，分解成词表中有的子词）。 + +​ 当前流行的模型中所用到的 subword Tokenizer 主要有三种不同的算法：BPE（Byte-Pair Encoding），WordPiece 和 SentencePiece。不同的模型所使用的 Tokenizer 的类型不同，比如，在 BertTokenizer 中所使用的就是 WordPiece。 + + + +## BPE（Byte-Pair Encoding） + +​ BPE 算法最早是[Neural Machine Translation of Rare Words with Subword Units (Sennrich et al., 2015)](https://arxiv.org/abs/1508.07909)提出的，是最早的一种数据压缩算法之一。BPE 是一个子词分词方法，本身需要依赖于预分词（pre-tokenization），也就是把文本先切分成词语（BPE算法接收的输入是一个词语的列表，而不是一个生语料）。这个预分词可以是简单的基于空格的分词方法，比如 GPT-2 和 RoBERTa；也可以是更复杂的比如基于规则的分词方法，比如 XLM 和 FlauBERT 用的是 Moses；再比如 GPT 用的是 Spacy。 + +​ 在将文本切分成词语之后，就得到了一个词表，以及这个词表中所有词语的词频。接下来，BPE 创建一个基本词表，这个词表中包含语料中所有出现的字符（在英文中就是26个字母和各种符号），然后统计语料中相邻单元对（这里的单元要在基本词表中）的频数，选取频数最高的单元对合成新的子词单元，加入基本词表中，重复进行直到词表大小达到预设值（词表的大小是个超参数，在训练前就设定好了的）。 + +​ 举个例子，假设在预分词之后，我们得到下面这个词表以及它们相应的词频： + +``` +("hug", 10), ("pug", 5), ("pun", 12), ("bun", 4), ("hugs", 5) +``` + +​ 同时，取出语料中所有出现的字符作为基本词表： + +``` +["b", "g", "h", "n", "p", "s", "u"] +``` + +​ 将第一个词表按照基本词表进行切分： + +``` +("h" "u" "g", 10), ("p" "u" "g", 5), ("p" "u" "n", 12), ("b" "u" "n", 4), ("h" "u" "g" "s", 5) +``` + +​ 计算所有相邻单元对的频率，并且选择频率最高的单元对 + +``` +"ug": 10 + 5 + 5 +``` + +​ `ug`被加入词表，并且语料所有的`ug`相连的部分都被组合 + +``` +("h" "ug", 10), ("p" "ug", 5), ("p" "u" "n", 12), ("b" "u" "n", 4), ("h" "ug" "s", 5) +``` + +​ 然后是`un`和`hug`，这时候的词表为： + +``` +["b", "g", "h", "n", "p", "s", "u", "ug", "un", "hug"] +``` + +​ 语料表示为： + +``` +("hug", 10), ("p" "ug", 5), ("p" "un", 12), ("b" "un", 4), ("hug" "s", 5) +``` + +​ 假设此时词表大小已经达到预设值，于是训练停止，这时候模型学到的这个词表（其实就是组合的规则）就会被应用在新遇到的词上，比如这时候遇到一个词`bug`就会被分解为`["b", "ug"]`，但是如果新遇到的词中存在基本词表中都没有的字符，则还是会用``表示，比如`mug`就会被表示为`["", "ug"]`。在实际的训练数据中，每个正常应该用到的字符都会出现至少一次，所以它们都应该包含在基础词表中，但是对于一些非常特殊的符号（比如表情符号），这种情况仍然有可能发生。 + +​ 词表的大小（基础词表的大小 + BPE 算法执行的轮数）是一个超参数，比如 GPT 的词表大小是40478，因为基础词表大小是478，然后算法运行了40000轮。 + +​ 上面的做法其实是 BPE 算法的简化版本，真正的在做 BPE 时还有一些关键的细节需要注意： + +​ 1、在将单词拆分成最小单元时，会在单词序列后加上””(具体实现上可以使用其它符号)来表示中止符。在子词解码时，中止符可以区分单词边界。 + +​ 2、每次合并后词表大小可能出现3种变化： + +- +1，表明加入合并后的新子词，同时原来的2个子词还保留（2个字词分开出现在语料中）。 +- +0，表明加入合并后的新子词，同时原来的2个子词中一个保留，一个被消解（一个子词完全随着另一个子词的出现而紧跟着出现）。 +- -1，表明加入合并后的新子词，同时原来的2个子词都被消解（2个字词同时连续出现）。 + +​ 3、实际上，随着合并的次数增加，词表大小通常先增加后减小 + +​ 4、得到最终词表后，使用时的编码方式（最大长度匹配）： + +- 将词典中的所有子词按照长度由大到小进行排序； +- 对于单词w，依次遍历排好序的词典。查看当前子词是否是该单词的子字符串，如果是，则输出当前子词，并对剩余单词字符串继续匹配。 +- 如果遍历完字典后，仍然有子字符串没有匹配，则将剩余字符串替换为特殊符号输出，如””。 + +​ 5、解码：如果相邻子词间没有中止符，则将两子词直接拼接，否则两子词之间添加分隔符 + +*** + +- **BBPE（Byte-level BPE）** + +​ 在生成基础词表这一步的时候，如果这个基础词表包含了所有可能的字符，那么这个基础词表会非常大，比如用所有的 unicode 字符来当做基础词表。 + +​ 为了能够缩减基础词表的大小，GPT-2 在生成基础词表的时候，用了字节（一个字节有256种不同的组合）来作为基础词表。这是一个非常巧妙的技巧，它把基础词表的大小限制在了256，同时能够确保所有的基本字符都包含在基础词表中（一个字符可以由多个字节组成）。 + +​ GPT2 的分词器能够在不需要 ``的情况下对所有的文本进行分词（还用了一些额外的规则来处理标点符号）。GPT-2 的词表大小是 50257，其中包括256个字节基本词表，一个特殊的文本结束符号，以及算法运行50000轮产生的子词。 + +*** + + + +## WordPiece + +​ WordPiece 算法最早发表于 [Japanese and Korean Voice Search (Schuster et al., 2012)](https://static.googleusercontent.com/media/research.google.com/ja//pubs/archive/37842.pdf)，它和 BPE 算法非常相似。也有一些预训练模型的分词器用的是这个算法，比如 BERT、DistilBERT 和 Electra。与 BPE 算法类似，WordPiece 算法也是初始化一个基础词表，然后每次从词表中选出两个子词合并成新的子词。 + +​ 与 BPE 的主要区别在于，选择两个子词进行合并的规则：BPE 选择频数最高的相邻子词合并，而 WordPiece 使用的是通过语言模型来计算合并两个单词可能造成的影响，然后选择使得似然函数提升最大的字符对。这个提升是通过结合后的字符对减去结合前的字符对之和得到的。 + +​ 假设句子$S=（t1,t2,...,t_n）$由 n 个 token 组成，$t_i$表示第$i$个 token，假设各个 token 之间是相互独立的，则句子$S$的语言模型似然值等于所有子词概率的乘积 +$$ +logP(S) = \sum^n_{i=1}logP(t_i) +$$ +​ 假设把相邻位置的两个 token $x$ 和 $y$ 进行合并，合并后的 token 记为 $z$，此时句子 $S$ 的似然值变化可以表示为： +$$ +logP(t_z)-(logP(t_x)+logP(t_y)) = log(\frac{P(t_z)}{P(t_x)P(T_y)}) +$$ +​ 也就是说，判断`tokenization`相较于`token` + `ization`是否更适合出现。选择能够提升语言模型概率最大的相邻子词加入词表。 + + + +## Unigram + +​ Unigram 算法最早发表于 [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates (Kudo, 2018)](https://arxiv.org/pdf/1804.10959.pdf)。与 WordPiece 一样，Unigram Language Model(ULM) 同样使用语言模型来挑选子词。不同之处在于，BPE 和 WordPiece 算法的词表大小都是从小到大增加的。而 ULM 则是先初始化一个大词表，这个基础词表包含了所有预分词结果中的 token 以及所有常用的子词（也可以通过 BPE 算法初始化），根据评估准则不断丢弃里边的词，直到满足预设条件。ULM 算法考虑了句子的不同分词可能，因而能够输出带概率的多个分词序列。 + +​ 并没有预训练模型单独使用 ULM 作为分词的算法，但是它常与 SentencePiece 一起使用。 + +​ 对于句子$S$，$\mathop{x}\limits ^{\rightarrow} = (x_1,x_2,...,x_m)$为句子的一个分词结果，由 m 个子词组成。所以，当前分词下句子$S$的似然值可以表示为： +$$ +P(\mathop{x}\limits ^{\rightarrow}) = \prod^m_{i=1}P(x_i) +$$ +​ 对于句子$S$，挑选似然值最大的作为分词结果，则可以表示为： +$$ +x^* = \mathop{argmax}\limits_{x \in U(x)}~P(\mathop{x}\limits ^{\rightarrow}) +$$ +​ 这里的$U(x)$包含了句子的所有分词结果。在实际应用中，词表大小有上万个，直接罗列所有的分词组合不具有操作性。针对这个问题，可以通过维特比算法得到 $x^*$来解决。 + +​ ULM 通过 EM 算法来估计每个子词的概率$P(x_i)$，假设当前词表为V，训练数据为|D|，则当前训练数据中所有句子的所有分词组合形成的概率相加，表示为： +$$ +L = \sum^{|D|}_{s=1}log(P(X^{(s)}))=\sum^{|D|}_{s=1}log(\mathop{\sum_{x \in U(X^{(s)})}}P(x)) +$$ +​ 在训练的开始阶段，ULM 初始化一个巨大的基础词表，然后针对当前词表，用上面的算法计算每个子词在训练数据上的概率，然后计算移除每个子词时，语言模型整体的损失上升了多少，记为这个子词的损失值。然后通过排序，按照一定比例移除掉词表中损失值最低的一部分子词（这个比例一般是10%或者20%），这些子词的损失值最低，意味着他们对语言模型的影响最小。每训练一步，都会移除掉部分的子词，直到词表大小达到预设值为止。（ULM 算法训练的过程中会保留基础的字符以保证任何词语都能被处理）。 + +​ 训练完成的 UML 模型对一个句子给出多种的分词可能，比如说一个训练好的 ULM 分词器的词表如下： + +``` +["b", "g", "h", "n", "p", "s", "u", "ug", "un", "hug"] +``` + +​ 那么 `"hugs"` 就有这么几种分词方法： `["hug", "s"]`， `["h", "ug", "s"]` ， `["h", "u", "g", "s"]`。ULM 在训练的过程中，在保存词表的同时会保存每个词表中 token 的概率，在实际使用的时候，一般会选取概率值最高的分词序列，如果需要也能够输出带有概率的不同分词序列。 + +​ + +## SentencePiece + +​ 上面提到的所有分词算法都存在同一个问题：这些分词方式都是建立在原文本通过空格来分隔词语的。但是并不是所有语言词和词之间都是有空格的。还有就是，将分词结果解码到原来的句子中时，会在不同的词之间添加空格，这样解码出来的标点符号就不会和词语连着。 + +> Raw text: Hello world. +> +> Tokenized: [Hello] [world] [.] +> +> Decoded text: Hello world . + +​ 这就是编码解码出现的歧义性，因此需要特别定义规则来实现互逆。还有一个例子是，在解码阶段，欧洲语言词之间要添加空格，而中文等语言则不应添加空格。一种解决方案是，给不同的语言定制不同的预分词算法和解码算法，比如 XLM 就在中文、日文和泰文上做了尝试。 + +​ 为了能够更好地解决这个问题，谷歌在 [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing (Kudo et al., 2018)](https://arxiv.org/pdf/1808.06226.pdf)中提出，把将所有的字符都转化成 Unicode 编码，空格用`_`来代替，然后进行分词操作。这样空格也不需要特别定义规则了，然后在解码结束后恢复即可。然后还是用 BPE 或者 ULM 算法来构建词表。 + +​ SentencePiece 集成了BPE、ULM 算法，除此之外，它也能支持字符和词级别的分词。使用 SentencePiece + ULM 的模型有 ALBERT、XLNet、T5。 + +​ + diff --git a/docs/APPENDIX_TASK.md b/docs/APPENDIX_TASK.md deleted file mode 100644 index b8e67deb..00000000 --- a/docs/APPENDIX_TASK.md +++ /dev/null @@ -1,57 +0,0 @@ -## Introduction to Pattern-Exploiting Training (PET) - -A pre-trained models somehow has the ability to 'understand' natural languages. However, primary pre-trained models can only perform the original -training task (like autoencoding and autoregressive task), which usually does not fit the various tasks we face in reality. -Therefore, we need to come up with a way to adapt the pre-trained models to downstream tasks. - -Initially, an extra task-speficic architecture was added to the final layer of the pretrained-model, which is called finetuning. Nonetheless, the performance of -pretraining-fituning paradigm is not satisfatory especially when the pretraining task deviate from downstream tasks. As a result, [prompt-learning](https://arxiv.org/abs/2111.01998) is -proposed, where the core idea is to design a cloze-style task-specific prompt to combine pretrained-model and downstream tasks. - -[Exploiting Cloze Questions for Few Shot Text Classification and Natural -Language Inference](https://arxiv.org/abs/2001.07676) proposed PET, which can reconstruct the input texts as cloze-style phrases, so that it is easier for pre-trained models to understand given tasks. - -Due to different settings of input texts and labels among different tasks, we need to design appropriate cloze-style patterns. The paper introduced a sentiment classification example as shown below. - -The given sentiment analysis task is: given an input context (eg. 'Best pizza ever!'), and limited categories (here there's 2 categories, -1 and -1 denotes positive and negative sentiment, respectively). Our model needs to automatically predict the probability of each -sentiment category given the context. - -
- -As shown in the picture, firstly the positive and negative labels are verbalized as 'great' and 'bad'. Then the pattern is designed as: - -context + 'It was' + masked sentiment label + '.' - -The probability for filling 'great' and 'bad' into the masked position will be returned by the finetuned model. - -Here we can see that it requires users to design a pattern-verbalizer pair (PVP) for each given task. -In our project, this part is put in flagai/data/dataset/superglue/pvp.py. Under the class for a given task, -normally there exists two functions named 'verbalize' and 'get_parts', which represent the verbalizer and pattern design step. -An example is shown below. Note that for one task, there can be multiple patterns. - -``` - def get_parts(self, example: InputExample) -> FilledPattern: - - text_a = self.shortenable(example.text_a) - text_b = self.shortenable(example.text_b) - - if self.pattern_id == 0: - return [[self.mask], ':', text_a, text_b], [] - elif self.pattern_id == 1: - return [[self.mask], ' Question:', text_a, text_b], [] - elif self.pattern_id == 2: - return [text_a, '(', [self.mask], ')', text_b], [] - elif self.pattern_id == 3: - return [text_a, text_b, '(', [self.mask], ')'], [] - elif self.pattern_id == 4: - return ['[ Category:', [self.mask], ']', text_a, text_b], [] - elif self.pattern_id == 5: - return [[self.mask], '-', text_a, text_b], [] - else: - raise ValueError("No pattern implemented for id {}".format( - self.pattern_id)) - - def verbalize(self, label) -> List[str]: - return YahooPVP.VERBALIZER[label] -``` \ No newline at end of file diff --git a/docs/Advanced_Usage.md b/docs/Advanced_Usage.md new file mode 100644 index 00000000..95824752 --- /dev/null +++ b/docs/Advanced_Usage.md @@ -0,0 +1,122 @@ +## Advanced usage + +### custom model +If you want to customize the model or improve the model in the FlagAI framework, for example, follow the following rules: + +1. Inheriting BaseModel, BaseModel supports loading parameters in the form of from_pretrain and building models with different parameters in the form of init_from_json. +2. The first parameter of the ``__init__()`` function of the custom model must be config, and each key is the parameter information in the ``config.json`` file. +3. There must be a ``load_weights()`` function in the custom model. The input is the address of the pre-training parameters, and the pre-training parameters are loaded for the custom model. +4. The ``forward()`` function in the custom model must return a dictionary, which must contain ``logits``. If the labels parameter is passed in, it needs to additionally return ``loss`` data. + +Take GLM to complete the sequence labeling task as an example: + +```python +from flagai.model.base_model import BaseModel +from flagai.model.glm_model import GLMModel +import torch + +class GLMForSequenceClassification(BaseModel): + def __init__(self, config, hidden_dropout=0.1, pool_token='cls', **kwargs): + super().__init__(config, **kwargs) + self.config = config + self.pool_token = pool_token + self.model = GLMModel(config) + self.model.output_predict = False + self.num_class = config['class_num'] + # Multi-choice head. + hidden_size = self.model.hidden_size + self.pool_layer = torch.nn.Linear(hidden_size, hidden_size) + self.multichoice_dropout = torch.nn.Dropout(hidden_dropout) + self.multichoice_head = torch.nn.Linear(hidden_size, self.num_class) + + def forward(self, + input_ids=None, + position_ids=None, + attention_mask=None, + **kwargs): + num_choices = None + if len(input_ids.shape) == 3: + assert self.num_class == 1 + batch_size, num_choices = input_ids.shape[:2] + input_ids = input_ids.reshape(-1, input_ids.size(-1)) + attention_mask = attention_mask.reshape(-1, + *attention_mask.size()[2:]) + position_ids = position_ids.reshape(-1, *position_ids.size()[2:]) + model_out = self.model(input_ids, position_ids, attention_mask) + outputs, mems = model_out['logits'], model_out['hidden_states'] + if self.pool_token == 'start': + output = outputs[torch.arange(outputs.size(0), + dtype=attention_mask.dtype, + device=attention_mask.device), + attention_mask] + elif self.pool_token == 'pad': + output = outputs[torch.arange(outputs.size(0), + dtype=attention_mask.dtype, + device=attention_mask.device), + attention_mask - 1] + elif self.pool_token == 'cls': + output = outputs[:, 0] + else: + raise NotImplementedError + output = torch.tanh(self.pool_layer(output)) + multichoice_output = self.multichoice_dropout(output) + logits = self.multichoice_head(multichoice_output) + if num_choices is not None: + logits = logits.view(-1, num_choices) + if 'labels' not in kwargs: + return {'logits': logits, 'hidden_states': mems} + else: + labels = kwargs['labels'] + if logits.size(1) == 1: + logits = logits.squeeze(1) + loss = F.binary_cross_entropy_with_logits( + logits.contiguous().float(), labels.float()) + else: + loss = F.cross_entropy(logits.contiguous().float(), + labels.long()) + return {"loss": loss, 'logits': logits, 'hidden_states': mems} + + def compute_loss(self, + input_ids=None, + position_ids=None, + attention_mask=None, + labels=None, + **kwargs): + model_output = self.forward(input_ids=input_ids, + position_ids=position_ids, + attention_mask=attention_mask) + assert labels is not None, "labels must not None!" + logits = model_output['logits'] + loss = F.cross_entropy(logits.contiguous().float(), labels.long()) + return { + "loss": loss, + 'logits': model_output['logits'], + 'hidden_states': model_output['hidden_states'] + } + + def load_weights(self, checkpoint_path): + checkpoints = self.model.load_weights_glm(checkpoint_path) + self.load_state_dict(checkpoints, strict=False) +``` + +Among them, in the ``__init__()`` function, except for the config parameter, the other parameters are flexibly defined for different tasks. +For example, ``hidden_dropout`` and ``pool_token`` in the following code, these parameters can be flexibly passed in when calling ``from_pretrain()``, for example: +```python +from flagai.model.glm_model import GLMForSequenceClassification +model_dir = "./state_dict/GLM_sequence_classification/" ## this dir is the position for model and vocab and config files. +model = GLMForSequenceClassification.from_pretrain(model_dir, + hidden_dropout=0.1, + pool_token="cls") +``` + +After building a custom model and ensuring that the pre-training parameters are loaded correctly, you can use it directly in the framework. + +### Choose the right Tokenizer + +After customizing the model, you need to select a suitable Tokenizer to cooperate with the model training. FlagAI supports a variety of native Tokenizers, such as: + +1. BertTokenizer: ```from flagai.data.tokenizer.bert_tokenizer import BertTokenizer``` BertTokenizer supports a variety of models, including Chinese models, English models; Bert models, RoBERTa models, Chinese GPT2 models, and more. +2. GLMLargeChTokenizer: ```from flagai.data.tokenizer.glm_large_ch_tokenizer import GLMLargeChTokenizer``` GLMLargeChTokenizer supports ``GLM-large-ch`` model。 +3. GLMLargeEnTokenizer: ```from flagai.data.tokenizer.glm_large_en_tokenizer import GLMLargeEnTokenizer``` GLMLargeEnTokenizer supports ``GLM-large-en`` model。 +3. T5BPETokenizer: ```from flagai.data.tokenizer.t5_tokenizer import T5BPETokenizer``` T5BPETokenizer supports englisht T5 model：``T5-base-en``。 +4. T5PegasusTokenizer: ```from flagai.data.tokenizer.t5_pegasus_tokenizer import T5PegasusTokenizer``` T5PegasusTokenizer supports chinese T5 model：``T5-base-ch``。 \ No newline at end of file diff --git a/docs/DATASET_EXAMPLE.md b/docs/DATASET_EXAMPLE.md new file mode 100644 index 00000000..645f4a2b --- /dev/null +++ b/docs/DATASET_EXAMPLE.md @@ -0,0 +1,143 @@ +# Examples in datasets + +## SuperGLUE: BoolQ +| Key | Meaning | +|----------|-----------------------------------------| +| passage | Background information | +| question | true/false question according to passage | +| label | Range：true/false | +Example +```json +{"question": "is barq's root beer a pepsi product", + "passage": "Barq's -- Barq's is an American soft drink. Its brand of root beer is notable for having caffeine. Barq's, created by Edward Barq and bottled since the turn of the 20th century, is owned by the Barq family but bottled by the Coca-Cola Company. It was known as Barq's Famous Olde Tyme Root Beer until 2012.", + "idx": 5, "label": false} +``` + +## SuperGLUE: CB +| Key | Meaning | +|------------|----------------------------------------| +| premise | Premise text | +| hypothesis | Hypothesis text | +| label | Range：entailment/contradiction/neutral | +Example: +```json +{"premise": "Mary is a high school student.", + "hypothesis": "Mary is a student", + "label": "entailment", "idx": 10} +``` + +## SuperGLUE: Copa +| Key | Meaning | +|----------|---------------------------------------------------------------------------------------------------------------------| +| premise | Background information | +| choice1 | The first choice | +| choice2 | The second choice | +| question | Range： cause/effect, which is asking 1.which choice is the cause of premise 2.which choice is the effect of premise | +| label | Range：0/1, which represents the answer is choice1 and choice2 , respectively | +Example: +```json +{"premise": "My eyes became red and puffy.", + "choice1": "I was sobbing.", "choice2": "I was laughing.", + "question": "cause", "label": 0, "idx": 7} +``` + +## SuperGLUE: MultiRC +| Key | Meaning | +|-----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| text | Background information | +| questions | Consists of several questions for text. There are 5 answers under each question, and there is a label for each answer, label is 0 if the answer is correct, otherwise 1. | +Example: +```json +{"idx": 4, "version": 1.1, + "passage": {"text": "...The two companies numbered about 225 men, and were commanded by General John E. Ross, a veteran Indian fighter... ", + "questions": [{"question": "When the narrator arrived at the headquarters, approximately how many men were present?", + "answers": [{"text": "225 men", "idx": 178, "label": 1}, + {"text": "225", "idx": 179, "label": 1}, + {"text": "525 men", "idx": 180, "label": 0}, + {"text": "235", "idx": 181, "label": 0}, + {"text": "255 men", "idx": 182, "label": 0}], "idx": 41}]}} +``` + + +## SuperGLUE: RTE +| Key | Meaning | +|------------|-----------------------------------| +| premise | Premise text | +| hypothesis | Hypothesis text | +| label | Range：entailment/not entailment | +Example: +```json +{"premise": "Security forces were on high alert after an election campaign in which more than 1,000 people, including seven election candidates, have been killed.", +"hypothesis": "Security forces were on high alert after a campaign marred by violence.", +"label": "entailment", "idx": 4} +``` + + +## SuperGLUE: Wic +| Key | Meaning | +|-----------|----------------------------------------------------------------------------------------------------------------------------------------| +| sentence1 | sentence1 | +| sentence2 | sentence2 | +| word | A word that appears in both sentence1 and sentence2. | +| label | Range：true/false, which represents 1. This word has the same meaning in sentence1 and sentence2 2.Different meanings in two sentences. | +Example: +```json +{"word": "class", "sentence1": "An emerging professional class.", +"sentence2": "Apologizing for losing your temper, even though you were badly provoked, showed real class.", + "idx": 0, "label": false, "start1": 25, "start2": 85, "end1": 30, "end2": 90, "version": 1.1} +``` + +## SuperGLUE: WSC +| Key | Meaning | +|------------|-----------------------------------------------------------------------------------| +| text | Background information | +| span2_text | A pronoun in text | +| span1_text | A sentence piece in text | +| label | Range：true/false, which represents 1.span1_text refers to span2_text 2. elsewise. | +Example: +```json +{"word": "class", "sentence1": "An emerging professional class.", +"sentence2": "Apologizing for losing your temper, even though you were badly provoked, showed real class.", + "idx": 0, "label": false, "start1": 25, "start2": 85, "end1": 30, "end2": 90, "version": 1.1} +``` + +## CLUE: AFQMC +| Key | Meaning | +|-----------|---------------------------------------------------------------------------------------------------| +| sentence1 | sentence1 | +| sentence2 | sentence1 | +| label | Range：0/1, Which represents 1.The meanings of two sentences are different 2.Meanings are similar. | +Example: +```json +{"sentence1": "双十一花呗提额在哪", "sentence2": "里可以提花呗额度", "label": "0"} +``` + +## CLUE: TNEWS +| Key | Meaning | +|-----------|---------------------------------| +| sentence | Background information | +| label | labels which have 15 categories | +| label_des | What the label refers to | +Example: +```json +{"label": "102", "label_des": "news_entertainment", "sentence": "江疏影甜甜圈自拍，迷之角度竟这么好看，美吸引一切事物"} +``` + + +## CLUE: CMRC2018 +| Key | Meaning | +|----------|-----------------------------------------------------------------------| +| context | Background information | +| question | Question for context, there could be multiple questions for one text. | +| answers | for a question, there are 3 answers provided by humans | +Example: +``` +{'paragraphs': + [{'id': 'TRAIN_186', + 'context': '工商协进会报告...所谓的“傻钱”策略，其实就是买入并持有美国股票这样的普通组合。...', + 'qas':{'question': '消费者信心指数由什么机构发布？', 'id': 'TRAIN_186_QUERY_4', + 'answers': [{'text': "工商协进会", 'answer_start': 759}, + {'text': "工商协进会", 'answer_start': 759}, + {'text': "工商协进会", 'answer_start': 759}]}]}], + 'id': 'TRAIN_186', 'title': '范廷颂'} +``` \ No newline at end of file diff --git a/docs/GLM.md b/docs/GLM.md new file mode 100644 index 00000000..4c1e7a68 --- /dev/null +++ b/docs/GLM.md @@ -0,0 +1,63 @@ +# Introduction to GLM + +## Model Description + +Currently, there exist several different pre-training model architectures: autoencoding models that only implement encoder architecture (e.g., BERT), +autoregressive models that only implement decoder (e.g., GPT), and encoder-decoder models that implement both encoder and decoder (e.g., T5). + +The **GLM model**, proposed in [GLM: General Language Model Pretraining +with Autoregressive Blank Infilling](https://arxiv.org/abs/2103.10360), is based on a slightly different strategy: autoregressive blank infilling. + +It claims to perform well in the three main categories of NLP taks: classification, unconditional generation, and conditional generation tasks. +| Framwork | NLU | Cond.Gen. | Uncond.Gen | +|-----------------|-----|-----------|------------| +| Augoregressive | - | - | ✅ | +| Autoencoding | ✅ | × | × | +| Encoder-Decoder | - | ✅ | - | +| GLM | ✅ | ✅ | ✅ | +The key features of GLM include: + +- First task: Several spans of the text are masked following the idea of autoencoding. Those spans will be randomly rearranged and be predicted in an autoregressive manner. The masked spans covers 15% original tokens. +- Second task: Similar to the first task, but the span covers 50%-100% original tokens. +- Other model architecture changes compared with BERT + - [Pre-LN](http://proceedings.mlr.press/v119/xiong20b.html) + - 2D positional encoding: Each token has two positional encodings: the global position in the sentence, and the local position inside the masked span. + - Feed-forward network is replaced with a linear layer + +## Performance of GLM + +### [SuperGLUE](https://super.gluebenchmark.com) +Single model, single task finetune on `dev`，more results [here](https://github.com/THUDM/GLM) + +| Model | COPA | WSC | RTE | WiC | CB | MultiRC | BoolQ | ReCoRD | +| ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | +| GLM-10b-ch | 98.0 | 95.2 | 93.1 | 75.7 | 98.7/98.2 | 88.1/63.3 | 88.7 | 94.4/94.0 | +| [RoBERTa-Large](https://github.com/pytorch/fairseq/tree/master/examples/roberta) | 94.0 | 91.3 | 86.6 | 75.6 | 98.2/- | 85.7/- | 86.9 |89.5/89.0| +| [DeBERTa-XXLarge-v2](https://github.com/microsoft/DeBERTa/tree/master/experiments/superglue) | 97.0 | - | 93.5 | - | - | 87.8/63.6 | 88.3 | 94.1/93.7 | + +### [CLUE](https://www.cluebenchmarks.com) +Single model, single task finetune on `dev`, (list part of results on CLU). Learn more about `GLM-10b-ch` click [here](https://model.baai.ac.cn/model-detail/100001)。 + +| 模型 | AFQMC | TNEWS1.0 | IFLYTEK | OCNLI_50K | CSL | CMRC2018 | CHID1.0 | C3 1.0 | +|:--------------:|:------:|:--------:|:-------:|:---------:|:------:|:--------:|:-------:|:------:| +| RoBERTa XLarge | 75.835 | 68.75 | 62.654 | 82.333 | 83.433 | 80.5 | 86.57 | 77.03 | +| GLM-10b-ch | 75.42 | 69.94 | 62.15 | 85 | 86.17 | 70 | 87.009 | 88.335 | + + +## Supported pre-trained GLM models +see [Tutorial 5: Simplify model and tokenizer Initialization by Using Autoloader](/docs/TUTORIAL_5_INSTRUCTIONS_FOR_AutoLoader.md). + +## Step-by-step procedure of GLM +1) Following the example in the paper, the original text contains 6 tokens, and two spans are masked: first span contains the 3rd token and the second span contains the 5th and 6th token. + +
+ +2) The input is divided into 2 parts, part A (corrupted text) and part B (masked spans). Note that the order of spans is shuffled here. +3) +
+ +3) Input and output of GLM, the input contains token embeddings and 2 sets of positional encodings: the first set is the positions of each token, where the tokens in each masked span share the same position ID. The second set records the relative positions inside the masked span. +
+ +4) The self-attention mask that realizes both autoencoding upon corrupted text and autoregressive upon the masked spans. +
diff --git a/docs/ChangeToMegatron.md b/docs/TUTORIAL_10_MEGATRON.md similarity index 100% rename from docs/ChangeToMegatron.md rename to docs/TUTORIAL_10_MEGATRON.md diff --git a/docs/TUTORIAL_11_GLM_BLANK_FILLING_QA.md b/docs/TUTORIAL_11_GLM_BLANK_FILLING_QA.md new file mode 100644 index 00000000..ea2c6643 --- /dev/null +++ b/docs/TUTORIAL_11_GLM_BLANK_FILLING_QA.md @@ -0,0 +1,114 @@ + +# GLM Blank Filling Generation +![img.png](./img/glm_blank_filling.png) + +GLM naturally handles variable-length blank filling which is crucial for many downstream tasks. +Based on autoregressive blank-filling, GLM randomly +blank out continuous spans of tokens from the input text, +following the idea of autoencoding, and train the model to +reconstruct the spans, following the idea of autoregressive +pre-training. +As a result, GLM learns +autoregressive generation during pre-training. + + +GLM finetunes on downstream tasks and reformulate them as blank-filling generation. Each task is associated with a human-crafted cloze question, and the model predicts the answer +to the cloze. For example, a sentiment classification task +is reformulated as filling the blank in “[SENTENCE]. It’s +really ”. The prediction of “good” or “bad” indicates the +sentiment being positive or negative. With such formulation, +GLM benefits from the consistency between pretraining and +finetuning, because both pretraining and finetuning involves +training the model to generate text given context. To make the pre-training +method better suited for text generation tasks, GLM also studys +a multi-task pre-training setup, where the model is jointly +trained to reconstruct masked spans and generate longer +text. + +In GLM, there are three MASK methods, corresponding to three prediction formats respectively. +1. ```[MASK]```, Token-level. This mask method will only mask random tokens in a sentence, + which has the least mask part among the three maks methods. Therefore, + the content generated is also limited. For example: ``[CLS]北京故宫是中国[MASK]非物质文化遗产。<|endoftext|><|startofpiece|>现存最大的古代宫殿建筑, 也是``. +2. ```[sMASK]```: Entity-level. We restrict that the masked + spans must be full sentences. Multiple spans + (sentences) are sampled to cover 15% of + the original tokens. This objective aims for + seq2seq tasks whose predictions are often + complete sentences or paragraphs. + The ```[sMASK]``` is longer than the text predicted by ```[MASK]```. For example: `` [CLS]人工智能是一个以计算机科学为基础,由计算机、数学、哲学等多学科交叉融合的交叉学科,[sMASK],具有非常巨大的前景。<|endoftext|><|startofpiece|>它涉及的信息量不仅非常巨大,而且也是人工智能发展的一个关键,其研究内容包括人的感觉、知觉和思维,以及如何理解各种现象,以及解释现象的本性和原因等,通过计算机来进行系统的分析推理、建立数学模型,并模拟人类意识。`` +3. ```[gMASK]```: Document-level. We sample a single span + whose length is sampled from a uniform distribution over 50%–100% of the original length. + The objective aims for long text generation. For example: ``[CLS]问题:啤酒伤胃吗?回答:[gMASK]<|startofpiece|>谢邀。 我是啤酒爱好者,但是我不喝酒。 我以前也说过,喝酒伤身,啤酒伤胃,伤肠道。 现在我也知道了啤酒伤人的很多细节,我就不瞎几把的说,大家看图片就知道了。 其实啤酒伤身这个说法只是表面而已。 啤酒中含有少量的碳酸和酒精,碳酸和酒精是成酸性物质,而乙醇是脂溶性的,酒精在胃里能够被分解,生成乙醇和二氧化碳,在体内是水和二氧化碳,两种物质会迅速发生中和反应,结果导致人体出现头痛、呕吐、胸痛、浑身发热等现象,这就是所谓喝大了,喝多了。 啤酒的含糖量在15%左右,喝多了也是伤身的,啤酒含糖量较高的主要成分是水分,而水分的体积比酒精大,所以酒精进入人体,与水相遇,就会产生大量气体,二氧化碳、水、一氧化碳等刺激人体,造成人体大量出汗,使体内温度升高,`` + + +As example, GLM finish the question task as an autoregressive blank in- +filling task + +```python +import torch +from flagai.model.glm_model import GLMModel +from flagai.data.tokenizer import GLMLargeChTokenizer +from flagai.model.predictor.predictor import Predictor +if __name__ == "__main__": + """Main training program.""" + print('Generate Samples') + tokenizer = GLMLargeChTokenizer(vocab_path='./checkpoints/glm-large-ch/cog-pretrain.model', + add_block_symbols=True, + add_task_mask=True, + add_decoder_mask=False, + fix_command_token=False) + model = GLMModel.from_pretrain(model_name='glm-large-ch', only_download_config=False) + model.cuda(torch.cuda.current_device()) + predictor = Predictor(model, tokenizer) + # question-answering + text = '问题：啤酒伤胃吗？回答：[gMASK]' + output=predictor.predict_generate_randomsample(text) + print(text,'\n',output) +``` + +Similar to BERT, GLM can predict masked tokens as + +```python +import torch +from flagai.model.glm_model import GLMModel +from flagai.data.tokenizer import GLMLargeChTokenizer +from flagai.model.predictor.predictor import Predictor +if __name__ == "__main__": + """Main training program.""" + print('Generate Samples') + tokenizer = GLMLargeChTokenizer(vocab_path='./checkpoints/glm-large-ch/cog-pretrain.model', + add_block_symbols=True, + add_task_mask=True, + add_decoder_mask=False, + fix_command_token=False) + model = GLMModel.from_pretrain(model_name='glm-large-ch', only_download_config=False) + model.cuda(torch.cuda.current_device()) + predictor = Predictor(model, tokenizer) + # question-answering + text = '北京故宫是中国[MASK]非物质文化遗产。' + output=predictor.predict_generate_randomsample(text) + print(text,'\n',output) +``` +and predict masked sentences as + +```python +import torch +from flagai.model.glm_model import GLMModel +from flagai.data.tokenizer import GLMLargeChTokenizer +from flagai.model.predictor.predictor import Predictor +if __name__ == "__main__": + """Main training program.""" + print('Generate Samples') + tokenizer = GLMLargeChTokenizer(vocab_path='./checkpoints/glm-large-ch/cog-pretrain.model', + add_block_symbols=True, + add_task_mask=True, + add_decoder_mask=False, + fix_command_token=False) + model = GLMModel.from_pretrain(model_name='glm-large-ch', only_download_config=False) + model.cuda(torch.cuda.current_device()) + predictor = Predictor(model, tokenizer) + # question-answering + text = '人工智能是一个以计算机科学为基础，由计算机、数学、哲学等多学科交叉融合的交叉学科，[sMASK]，具有非常巨大的前景。' + output=predictor.predict_generate_randomsample(text) + print(text,'\n',output) +``` \ No newline at end of file diff --git a/docs/TUTORIAL_12_GLM_EXAMPLE_TITLE_GENERATION.md b/docs/TUTORIAL_12_GLM_EXAMPLE_TITLE_GENERATION.md new file mode 100644 index 00000000..782933da --- /dev/null +++ b/docs/TUTORIAL_12_GLM_EXAMPLE_TITLE_GENERATION.md @@ -0,0 +1,187 @@ +# GLM example：title generation + + +## background +The title generation task needs to input a piece of text, and the model outputs the corresponding title according to the input text. +![](./img/bert_title_generation_model.png) + +## Result show +#### Input +``` +"本文总结了十个可穿戴产品的设计原则而这些原则同样也是笔者认为是这个行业最吸引人的地方1为人们解决重复性问题2从人开始而不是从机器开始3要引起注意但不要刻意4提升用户能力而不是取代人", +"2007年乔布斯向人们展示iPhone并宣称它将会改变世界还有人认为他在夸大其词然而在8年后以iPhone为代表的触屏智能手机已经席卷全球各个角落未来智能手机将会成为真正的个人电脑为人类发展做出更大的贡献", +"雅虎发布2014年第四季度财报并推出了免税方式剥离其持有的阿里巴巴集团15％股权的计划打算将这一价值约400亿美元的宝贵投资分配给股东截止发稿前雅虎股价上涨了大约7％至5145美元" +``` + +#### Output +``` +可 穿 戴 产 品 设 计 原 则 十 大 原 则 +乔 布 斯 宣 布 iphone 8 年 后 将 成 为 个 人 电 脑 +雅 虎 拟 剥 离 阿 里 巴 巴 15 ％ 股 权 +``` + +## Model Train（train.py） + +Modify the training data path src_dir, tgt_dir, and model path model_dir before running. Run this command at the command line: +```commandline +cd ./examples/glm_title_generation +python ./train.py +``` + +### 1.Load data +Sample data is at /examples/bert_title_generation/data/ + +1）Define the load function +```python +def read_file(): + src = [] + tgt = [] + + ## read data file to load src and tgt, for example: + ## src = ["article_1", "article_2", "article_3" ......] + ## tgt = ["title_1", "title_2", "title_3" ......] + ## no matter what data you use, you need to construct the right src and tgt. + with open(src_dir, 'r', encoding='utf-8') as f: + lines = f.readlines() + for line in lines: + src.append(line.strip('\n').lower()) + + with open(tgt_dir, 'r', encoding='utf-8') as f: + lines = f.readlines() + for line in lines: + tgt.append(line.strip('\n').lower()) + return src,tgt +``` + +2）Define the data loading process +```python +class GLMSeq2seqDataset(Dataset): + + def __init__(self, sents_src, sents_tgt): + super(GLMSeq2seqDataset, self).__init__() + self.sents_src = sents_src + self.sents_tgt = sents_tgt + + def __getitem__(self, i): + source_text = self.sents_src[i] + target_text = self.sents_tgt[i] + data = tokenizer.encode_plus(source_text, target_text=target_text) + return data + + def __len__(self): + return len(self.sents_src) +``` + +3) Define the batch function (collate_fn) in the data iterator (DataLoader) to pad a batch of data into a uniform size +```python +class GLMSeq2seqDynamicCollateFN(): + def __init__(self, pad_id): + self.pad_id = pad_id + + def pad_token(self, tokens, max_length): + pad_len = max_length-len(tokens) + tokens += [self.pad_id]*pad_len + return tokens + + def pad_position_ids(self, position_ids, max_length): + pad_len = max_length-len(position_ids[0]) + position_ids[0] += [len(position_ids[0])+x for x in range(pad_len)] + position_ids[1] += [1] * pad_len + return position_ids + + def pad_loss_mask(self, loss_mask, max_length): + pad_len = max_length-len(loss_mask) + loss_mask += [0] * pad_len + return loss_mask + + def __call__(self, batch): + input_ids = [data["input_ids"] for data in batch] + target_ids = [data["target_ids"] for data in batch] + position_ids = [data["position_ids"] for data in batch] + attention_mask = [data['attention_mask'] for data in batch] + loss_mask = [data['loss_mask'] for data in batch] + + max_length = max([len(t) for t in input_ids]) + for i in range(len(input_ids)): + input_ids[i] = self.pad_token(input_ids[i], max_length) + target_ids[i] = self.pad_token(target_ids[i], max_length) + position_ids[i] = self.pad_position_ids(position_ids[i], max_length) + loss_mask[i] = self.pad_loss_mask(loss_mask[i], max_length) + return { + 'input_ids': torch.LongTensor(input_ids), + 'target_ids': torch.LongTensor(target_ids), + 'position_ids': torch.LongTensor(position_ids), + 'attention_mask': torch.LongTensor(attention_mask), + 'loss_mask': torch.LongTensor(loss_mask) + } +``` +4）Get the training data +```python +train_src, train_tgt = read_file() +print('-----------train data length:', len(train_src)) +my_collate_fn = GLMSeq2seqDynamicCollateFN(pad_id=tokenizer.get_command('pad').Id) +train_dataset = GLMSeq2seqDataset(train_src, + train_tgt) +``` +### 2.Load model and tokenizer + +```python +from flagai.auto_model.auto_loader import AutoLoader + +# the model dir, which contains the 1.config.json, 2.pytorch_model.bin, 3.vocab.txt, +# or we will download these files from the model hub to this dir. +model_dir = "./state_dict/glm/" +# Autoloader can build the model and tokenizer automatically. +# 'seq2seq' is the task_name. +AutoLoader("seq2seq",model_name="GLM-large-ch",model_dir=model_dir) +model = auto_loader.get_model() +tokenizer = auto_loader.get_tokenizer() +``` + +### 3. Training + +Instantiate the Trainer, and set the training parameters. + +```python +import torch.cuda +from flagai.trainer import Trainer + +device = torch.device("cuda" if torch.cuda.is_available() else "cpu") +trainer = Trainer( + env_type="pytorch", + experiment_name="roberta_seq2seq", + batch_size=1, + gradient_accumulation_steps=1, + lr=2e-4, + weight_decay=1e-3, + epochs=10, + log_interval=10, + eval_interval=10000, + load_dir=None, + pytorch_device=device, + save_dir="checkpoints", + save_epoch=1, + num_checkpoints=1, + +) +``` +Enter the model, data, and batch functions into the trainer to start training: +```python +trainer.train(model, + train_dataset=train_dataset, + collate_fn=my_collate_fn) +``` + +## Generation（generate.py） +Modify the model configuration path model_dir and the trained model path model_save_path before running. Run this command at the command line: + +```commandline +cd ./examples/glm_title_generation +python ./generate.py +``` +You can choose between random sampling based on probability screening (random sample) or beam search (beamsearch) two generation methods: +```python +print(predictor.predict_generate_randomsample(text,out_max_length=66, top_k=10, top_p=.1, + repetition_penalty=4.0, temperature=1.2)) +print(predictor.predict_generate_beamsearch(text, out_max_length=66, beam_size=10)) +``` \ No newline at end of file diff --git a/docs/TUTORIAL_12_INSTRUCTIONS_FOR_AutoLoader.md b/docs/TUTORIAL_12_INSTRUCTIONS_FOR_AutoLoader.md deleted file mode 100644 index 31960b03..00000000 --- a/docs/TUTORIAL_12_INSTRUCTIONS_FOR_AutoLoader.md +++ /dev/null @@ -1,42 +0,0 @@ -## AutoLoader - -### Quickly build models and tokenizers with Autoloader -Autoloader automatically searches for pre-trained models and Tokenizers from the ModelHub based on the **model_name** and downloads them to the **model_dir**. - -Take semantic matching tasks as an example: -```python -## Target contains all target categories -## 0 means that two sentences have the same meaning -## 1 means that two sentences have different meanings -target = [0, 1] -auto_loader = AutoLoader(task_name="classification", ## The task name - model_name="RoBERTa-base-ch", ## The model name. - model_dir=model_dir, ## Model download folder - load_pretrain_params=True, ## Whether to load the pretraining model parameters. If False, only the model will be built and the pretraining parameters will not be downloaded. - target_size=len(target) ## The final output size of model. Use for classification. - ) -``` - -### All supported tasks -The **task_name** parameter supports: -1. task_name="classification": Supports a variety of classified tasks, for example, text classification, semantic matching, emotion analysis... -2. task_name="seq2seq": Supports seq2seq tasks, for example, auto title generation, auto couplet, auto chat... -3. task_name="sequence_labeling": Supports sequence labeling tasks, for example, ner, the part of speech tagging, chinese word segmentation... -4. task_name="sequence_labeling_crf": Add conditional random field layer for sequence labeling model. -5. task_name="sequence_labeling_gp": Add global pointer layer for sequence labeling model. - -### All supported models -All supported models is can be found in **model hub**. -Different models adapt to different tasks. - -#### Transfomrer encoder: - -For example, model_name="BERT-base-ch" or "RoBERTa-base-ch" These models support all of the tasks mentioned in the previous section - -#### Transformer decoder: - -For example, model_name="GPT2-base-ch", the model support "seq2seq" task. - -#### Transformer encoder + decoder: - -For example model_name="T5-base-ch", the model support "seq2seq" task. diff --git a/docs/TUTORIAL_9_GLM_EXAMPLE_PEOTRY_GENERATION.md b/docs/TUTORIAL_13_GLM_EXAMPLE_PEOTRY_GENERATION.md similarity index 100% rename from docs/TUTORIAL_9_GLM_EXAMPLE_PEOTRY_GENERATION.md rename to docs/TUTORIAL_13_GLM_EXAMPLE_PEOTRY_GENERATION.md diff --git a/docs/TUTORIAL_13_INSTRUCTIONS_FOR_PREDICTOR.md b/docs/TUTORIAL_13_INSTRUCTIONS_FOR_PREDICTOR.md deleted file mode 100644 index e39d4561..00000000 --- a/docs/TUTORIAL_13_INSTRUCTIONS_FOR_PREDICTOR.md +++ /dev/null @@ -1,37 +0,0 @@ -## Predictor - -### Get prediction result quickly by the Predictor -By the Predictor, you only need to enter a **text**, and you can get the output of the corresponding task directly. - -Take gpt2 writing tasks as an example: -```python -from flagai.auto_model.auto_loader import AutoLoader -from flagai.model.predictor.predictor import Predictor -if __name__ == '__main__': - ## Load model and tokenizer by the autoloader - loader = AutoLoader("seq2seq", "gpt2_base_chinese", model_dir="./state_dict/") - model = loader.get_model() - tokenizer = loader.get_tokenizer() - ## Define the Predictor - predictor = Predictor(model, tokenizer) - ## Define the start of the text as the input. - text = "今天天气不错，" - ## The predictor can analyse the model automatically to call different method. - out = predictor.predict_generate_randomsample(text, ## input - input_max_length=512, ## input max length - out_max_length=100, ## output max lenght - repetition_penalty=1.5, ## avoid the repetition out. (https://arxiv.org/pdf/1909.05858.pdf) - top_k=20, ## keep only top k tokens with highest probability (top-k filtering). - top_p=0.8) ## keep the top tokens with cumulative probability >= top_p (nucleus filtering).(http://arxiv.org/abs/1904.09751) - - print(f"out is {out}") - ### out is 到这里来看了一下，很是兴奋，就和朋友一起来这里来了。我们是周五晚上去的，人不多，所以没有排队，而且这里的环境真的很好，在这里享受美食真的很舒服，我们点了一个套餐，两个人吃刚刚好，味道很好。 -``` - -### All supported prediction method -1. predict_classification_classifier: Input a text or a text-pair to get the multi-class result, support the bert, roberta and other transformer encoder models. -2. predict_masklm: Input a text with [MASK] tokens to get the result of original text, support the bert, roberta and other transformer encoder models. -3. predict_ner: Input a text to get the ner result, support the bert, roberta and other transformer encoder models. -4. predict_generate_beamsearch: Input a text to get the output text, seq2seq task. support the bert, roberta, gpt2, t5 and glm models. -5. predict_generate_randomsample: Input a text to get the output text, seq2seq task. support the bert, roberta, gpt2, t5 and glm models. - diff --git a/docs/Huggingface_t5.md b/docs/TUTORIAL_14_HUGGINGFACE_T5.md similarity index 93% rename from docs/Huggingface_t5.md rename to docs/TUTORIAL_14_HUGGINGFACE_T5.md index dce3553b..acd18f81 100644 --- a/docs/Huggingface_t5.md +++ b/docs/TUTORIAL_14_HUGGINGFACE_T5.md @@ -33,21 +33,21 @@ trainer = MyTrainer( batch_size=4, eval_interval=10, log_interval=10, - experiment_name='t5-3b', + experiment_name='t5-11b', pytorch_device='cuda:0', load_dir=None, lr=1e-4, fp16=False) # using huggingface transformers to get tokenizer and models -model_name = 't5-3b' +model_name = 't5-11b' tokenizer = T5Tokenizer.from_pretrained(model_name) model = T5ForConditionalGeneration.from_pretrained(model_name) print("loading model & tokenizer is done!") src_dir = 'train_inputs.txt' tgt_dir = 'train_targets.txt' -model_dir = "./t5-3b" # 模型位置 +model_dir = "./t5-11b" # 模型位置 maxlen = 1024 @@ -139,7 +139,7 @@ trainer.train(model, ``` ## Tricks for speedup training -We may not run a t5-3b on a V100 32G. So, we need some tricks to cut down the GPU memory usage. +We may not run a t5-11b on a V100 32G. So, we need some tricks to cut down the GPU memory usage. ### step1.fp16 Model parameters turned to `fp16` ```python @@ -149,15 +149,15 @@ trainer = MyTrainer( batch_size=1, eval_interval=10, log_interval=10, - experiment_name='t5-3b', + experiment_name='t5-11b', pytorch_device='cuda:0', load_dir=None, lr=1e-4, fp16=True) # change to `True` ``` ### step2.gradient recomputation(checkpoints) -Do not save the itermedia results in forward stage. Now you may run t5-3b with `batch size`=1. -Now, we can train/finetune a t5-3b with `gradient_accumulation_steps`. +Do not save the itermedia results in forward stage. Now you may run t5-11b with `batch size`=1. +Now, we can train/finetune a t5-11b with `gradient_accumulation_steps`. ```python trainer = MyTrainer( env_type='pytorch', @@ -165,7 +165,7 @@ trainer = MyTrainer( batch_size=1, eval_interval=10, log_interval=10, - experiment_name='t5-3b', + experiment_name='t5-11b', pytorch_device='cuda:0', load_dir=None, lr=1e-4, @@ -181,7 +181,7 @@ trainer = Trainer( batch_size=1, eval_interval=10, log_interval=10, - experiment_name='t5-3b', + experiment_name='t5-11b', load_dir=None, lr=1e-4, fp16=True @@ -204,7 +204,7 @@ trainer = Trainer( batch_size=1, eval_interval=10, log_interval=10, - experiment_name='t5-3b', + experiment_name='t5-11b', load_dir=None, lr=1e-4, fp16=True @@ -230,7 +230,7 @@ trainer = Trainer( batch_size=1, eval_interval=10, log_interval=10, - experiment_name='t5-3b', + experiment_name='t5-11b', load_dir=None, lr=1e-4, fp16=True diff --git a/docs/TUTORIAL_10_BERT_EXAMPLE_TITLE_GENERATION.md b/docs/TUTORIAL_15_BERT_EXAMPLE_TITLE_GENERATION.md similarity index 100% rename from docs/TUTORIAL_10_BERT_EXAMPLE_TITLE_GENERATION.md rename to docs/TUTORIAL_15_BERT_EXAMPLE_TITLE_GENERATION.md diff --git a/docs/TUTORIAL_11_BERT_EXAMPLE_SEMANTIC_MATCHING.md b/docs/TUTORIAL_16_BERT_EXAMPLE_SEMANTIC_MATCHING.md similarity index 100% rename from docs/TUTORIAL_11_BERT_EXAMPLE_SEMANTIC_MATCHING.md rename to docs/TUTORIAL_16_BERT_EXAMPLE_SEMANTIC_MATCHING.md diff --git a/docs/TUTORIAL_14_BERT_EXAMPLE_NER.md b/docs/TUTORIAL_17_BERT_EXAMPLE_NER.md similarity index 100% rename from docs/TUTORIAL_14_BERT_EXAMPLE_NER.md rename to docs/TUTORIAL_17_BERT_EXAMPLE_NER.md diff --git a/docs/TUTORIAL_15_GPT2_WRITING.md b/docs/TUTORIAL_18_GPT2_WRITING.md similarity index 100% rename from docs/TUTORIAL_15_GPT2_WRITING.md rename to docs/TUTORIAL_18_GPT2_WRITING.md diff --git a/docs/TUTORIAL_16_T5_EXAMPLE_TITLE_GENERATION.md b/docs/TUTORIAL_19_T5_EXAMPLE_TITLE_GENERATION.md similarity index 100% rename from docs/TUTORIAL_16_T5_EXAMPLE_TITLE_GENERATION.md rename to docs/TUTORIAL_19_T5_EXAMPLE_TITLE_GENERATION.md diff --git a/docs/TUTORIAL_1_BASICS.md b/docs/TUTORIAL_1_BASICS.md deleted file mode 100644 index 18e5e613..00000000 --- a/docs/TUTORIAL_1_BASICS.md +++ /dev/null @@ -1,46 +0,0 @@ -## Transformer basics -Briefly, the Transformer, as proposed in [Attention Is All You Need](https://arxiv.org/abs/1706.03762), -can be regarded as an integration of an encoder block and a decoder block. With the use of attention mechanism, -Transformer can extract better features from input, and consequently Transformer-based models have achieved state-of-the-art -performance in most of the language tasks. Another feature of transformer is its compatibility with parallel computing, which is a major -superiority compared with sequential models like RNN. - -The Transformer architecture is shown below - -
- -In the encoder step, firstly the embedding encoded from token positions is added to embeddings encoded from tokens themselves. -Then the input embeddings are passed to a multi-head attention mechanism, which is able to take mutual information among tokens into consideration. -The computed output will add the original input embeddings, followed by a layer normalization. -The rest of the encoder part is a feed-forward layer with addition and a layer normalization. - -The decoder architecture is similar to the encoder, with the following differences: - -1. At the beginning, we need to mask future information for decoders, which is done by setting the upper triangle of the matrix to 0. -2. There is a middle layer that incorporates query and key from encoder, and value from decoder. -3. The decoder has a linear layer and softmax layer at the end to determine output probability for each token in the vocabulary - -N encoders and decoders will be stacked together to form a transformer, where N is usually selected as 12 or 24. - -Step-by-step details of transformers can be viewed in [this article](https://towardsdatascience.com/illustrated-guide-to-transformers-step-by-step-explanation-f74876522bc0). - -## Transformer Applications - -Currently, there exist several different pre-training model architectures: autoencoding models that only implement encoder architecture (e.g., BERT), -autoregressive models that only implement decoder (e.g., GPT), and encoder-decoder models that implement both encoder and decoder (e.g., T5). - -The **GLM model**, proposed in [All NLP Tasks Are Generation Tasks: A General Pretraining Framework](https://arxiv.org/abs/2103.10360), -claims to have good performance in classification, unconditional generation, and conditional generation tasks with the use of a new pre-training method. - -The key features of GLM include: - -- First task: Several spans of the text are masked following the idea of autoencoding. Those spans will be randomly rearranged and be predicted in an autoregressive manner. The masked spans covers 15% original tokens. -- Second task: Similar to the first task, but the span covers 50%-100% original tokens. -- Other model architecture changes compared with BERT - - [Pre-LN](http://proceedings.mlr.press/v119/xiong20b.html) - - 2D positional encoding: Each token has two positional encodings: the global position in the sentence, and the local position inside the masked span. - - Feed-forward network is replaced with a linear layer - -An example showing the autoencoding and autoregressive of GLM can be viewed [here](APPENDIX_GLM_IO.md). - - diff --git a/docs/TUTORIAL_1_TOKENIZER.md b/docs/TUTORIAL_1_TOKENIZER.md new file mode 100644 index 00000000..bc38f365 --- /dev/null +++ b/docs/TUTORIAL_1_TOKENIZER.md @@ -0,0 +1,92 @@ +# How to construct and use Tokenizer + +## What is Tokenization? +**Tokenization** is a fundamental step in the preprocessing stage of NLP projects, +and its purpose is to convert unstructured symbolic texts into numeric matrices, +which are suitable for machine learning systems. + +In the tokenization process, a **tokenizer** is used to split natural language text +into a sequence of semantic units called **tokens**, which are then converted into +ids by looking up the tokens in a vocabulary file. An example of tokenizing +input text `Jack is walking a dog.` is shown below: + +
+ +It is noticeable that different tokenizers can have different ways to split text, +and have different vocabulary files. An introduction to those algorithms can be viewed [here](tokenization.md). + +Our projects currently support six tokenizers +as listed below: + +| Tokenizer | Language | +|------------------------------|----------| +| GLMLargeEnWordPieceTokenizer | English | +| GLMLargeChTokenizer | Chinese | +| GLM10bENBPETokenizer | English | +| T5BPETokenizer | Chinese | +| ROBERTATokenizer | Chinese | +| BertWordPieceTokenizer | Chinese | + + + + +## Loading a tokenizer +```python +from flagai.data.tokenizer import GLMLargeEnWordPieceTokenizer + +tokenizer = GLMLargeEnWordPieceTokenizer() # Load tokenizer +``` +At this step, the vocab files from Modelhub will be automatically downloaded to the path specified in `cache_dir` parameter. It is set to `./vocab` directory under the tokenizer file in default. + +## Applying a tokenizer +The tokenizer can be used to encode text to a list of token IDs, as well as decoding the token IDs to the original text. +```python +text = "Jack is walking a dog." # Input text +encoded_ids = tokenizer.EncodeAsIds(text) # Convert text string to a list of token ids +# Now encoded_ids = [2990, 2003, 3788, 1037, 3899, 1012] +recoverd_text = tokenizer.DecodeIds(encoded_ids) # Recover text string +# recovered_text should be the same as text +``` +## Creating your own tokenizer +Different tokenizers has different vocabulary and different ways to split text. To suit your project, sometimes it is significant to create a new tokenizer, and how to implement that is given below: +### 1. Create a package under `/flagai/tokenizer` + +### 2. Wrap the tokenizer from huggingface + +let's take T5 tokenizer as an example + +```python +from transformers import T5Tokenizer +from ..tokenizer import Tokenizer +class T5BPETokenizer(Tokenizer): + def __init__(self, model_type_or_path="t5-base", cache_dir=None): + self.text_tokenizer = T5Tokenizer.from_pretrained(model_type_or_path, + cache_dir=cache_dir) + self.text_tokenizer.max_len = int(1e12) +``` + +### 3. Define Tokenizer APIs (without huggingface) +If huggingface tokenizers are not used, you need to implement the following class functions by your own. + +```python +def EncodeAsIds(self, text: str, process_fn=None): + """Input text string => a list of token ids""" + +def EncodeAsTokens(self, text: str, process_fn=None): + """Input text string => a list of tokens""" + +def IdToToken(self, Id: int): + """Token id => token""" + +def TokenToId(self, token: str): + """Token => token id""" + return self.text_tokenizer._convert_token_to_id(token) + +def DecodeIds(self, Ids: list[int]): + """A list of token ids => recovered text string""" + return self.DecodeTokens([self.IdToToken(id) for id in Ids]) + +def DecodeTokens(self, tokens: list[str]): + """A list of tokens => recovered text string""" + return self.text_tokenizer.convert_tokens_to_string(tokens) +``` \ No newline at end of file diff --git a/docs/AllSupportedTasks.md b/docs/TUTORIAL_20_SUPPORTED_TASKS.md similarity index 100% rename from docs/AllSupportedTasks.md rename to docs/TUTORIAL_20_SUPPORTED_TASKS.md diff --git a/docs/TUTORIAL_2_DATASET.md b/docs/TUTORIAL_2_DATASET.md new file mode 100644 index 00000000..230c419c --- /dev/null +++ b/docs/TUTORIAL_2_DATASET.md @@ -0,0 +1,364 @@ +# Data set processing flow +The process of constructing a dataset is the data preprocessing process of NLP. Its main purpose is to rearrange the original scattered file data into data of a unified structure so that the language model can directly use it. The main process of constructing a sample dataset is as follows (take the CommitmentBank dataset as an example): + +
+ +At present, there are three kinds of data preprocessing in the project, namely, fine-tuning for classification tasks, pre-training, and fine-tuning for generation tasks. We will expand on them separately in the following. + +## Data processing: fine-tuning for classification tasks ([prompt-learning mode](TUTORIAL_7_PROMPT_LEARNING.md)) +### Application code + +```python +import torch.utils.data +from flagai.data.dataset import SuperGlueDataset +from flagai.data.tokenizer import GLMLargeEnWordPieceTokenizer +from tests.test_dataset_new_superglue import CollateArguments +from flagai.data.dataset import ConstructSuperglueStrategy + +# get default parameters +cl_args = CollateArguments() + +# Create tokenizer +tokenizer = GLMLargeEnWordPieceTokenizer() + +# Initially read and process the dataset +dataset = SuperGlueDataset(task_name='cb', + data_dir='./datasets/', + dataset_type='train', + tokenizer=tokenizer) + +# Construct collate function +collate_fn = ConstructSuperglueStrategy(cl_args, tokenizer, task_name="rte") + +# create loader +loader = torch.utils.data.DataLoader(dataset, + batch_size=1, + shuffle=False, + num_workers=1, + drop_last=False, + pin_memory=False, + collate_fn=collate_fn) +``` + +### Initially read and process the dataset +The corresponding code module is shown below, which consists of two steps: automatically loading the dataset, and unifying the structure of all datasets + +```python +dataset = SuperGlueDataset(task_name='cb', + data_dir='./datasets/', + dataset_type='train', + tokenizer=tokenizer) +``` +#### 1. Load the dataset + +After setting `task_name` to the abbreviation of the task name, the relevant data will be automatically downloaded in the background. FlagAI currently supports automatic loading of the following classification datasets: + +| dataset name | short name| Language | Benchmark | +|----------------------------------------------|----------|------|----------| +| Broadcoverage Diagnostics | boolq | Eng | SuperGLUE | +| CommitmentBank | cb | Eng | SuperGLUE | +| Choice of Plausible Alternatives | copa | Eng | SuperGLUE | +| Multi-Sentence Reading Comprehension | muiltirc | Eng | SuperGLUE | +| Recognizing Textual Entailment | rte | Eng | SuperGLUE | +| Words in Context | wic | Eng | SuperGLUE | +| The Winograd Schema Challenge | wsc | Eng | SuperGLUE | +| Ant Financial Question Matching Corpus | afqmc | Zh | CLUE | +| Short Text Classificaiton for News | tnews | Zh | CLUE | + +The dataset will be automatically downloaded to the address corresponding to `data_dir`, which defaults to the `./dataset` directory of the project. + +The downloaded dataset directory will contain three files, corresponding to the training set data, the validation set data, and the test set data. Take the CommitmentBank dataset as an example, the `train.jsonl` in the directory corresponds to the training set, `val.jsonl `corresponds to the validation set, `test.jsonl` corresponds to the test set. Generally, the training set and test set contain label information, but the test set does not. These data files will be processed separately in the next process. + +Different datasets may have different file formats, as well as different structures. Taking the CommitmentBank dataset as an example, the following is an example of it + +
+ +It contains four parts, as shown below + +| key | meaning | value | +|-----------|-------------------------------------------------------|--------------------------------| +| premise | premise text | Mary is a high school student. | +| hypothesis | hypothetical text | Mary is a student | +| label | The tags representing the relationship between the premise and the hypothesis. Include three kinds of inclusion, neutral and contrast | entailment | +| idx | The sequence number of the sample in the dataset | 10 | + +The specific structure of all FlagAI supported datasets can be viewed [here](DATASET_EXAMPLE.md). + +#### 2. Unified dataset structure +In this step, we will unify the data structures of different datasets to facilitate subsequent processing. The details of this structure are as follows: + + +| key | meaning | format | +|--------|-------------------------------------------------------|------| +| guid | a unique textual identifier | str | +| text_a | the sequence of text | str | +| text_b | an optional, second sequence of text | str | +| label | an optional label | str | +| logits | an optional list of per-class logits | list | +| meta | an optional dictionary to store arbitrary meta information | dict | +| ids | an optional numeric index | int | + +For example, the example of CommitBank in the previous step will be processed into the following form + +
+ +Noted that if text_a and text_b cannot be filled with background text information because the data structure is too complex, you can put the rest of the information in meta. + +When the dataset is constructed, you can view one of the samples directly in the code by indexing: + + +```python +example = dataset[3] # The third example in dataset +``` + +### Organize the data into input to the model + +The corresponding function is implemented in the following function, which consists of two steps: constructing the template, segmenting the word and constructing the input sample. + +```python +collate_fn = ConstructSuperglueStrategy(cl_args, + tokenizer, + task_name=task_name) +``` + +#### 1. Build the cloze template + +A cloze template contains background text, slots, and options provided to the slots. Models need to find the right options and fill in the blanks. + +For each different task, we need to construct cloze questions of different structures for the model to answer. Taking the CommitmentBank dataset as an example, it considers whether the hypothesis can be deduced from the premise, and there are only three results: contradiction/neutral/entailment. Then we can construct the following cloze problem, where contrast/neutral/entailment correspond to true/false/neither respectively. + +
+ +It can be seen that it can be roughly divided into two steps: the first step is to combine the existing text to make it look like a cloze format; the second step is to convert the original label text into a new label, which may be filled in. Option to enter vacancies. + +#### 2. Word segmentation and construct input samples +Next, we need to construct the input of the model. The first step is word segmentation, and then we need to divide it into two cases: +In the first case, the label categories contained in the dataset are limited. For example, in the CommitmentBank dataset, there are only three kinds of label texts—intailment/contradiction/neutral, which are common in classification tasks. In the second case, each cloze will give different options (usually a long text). For example, in some reading comprehension datasets, each option is a different understanding of the text. The two cases are handled as follows: + +**a) Cloze for a single token** + +| key | dimension | meaning | Construction method | +|-----------------|--------------------------------------|------------|-----------------------------------------------| +| input_ids | torch.Size([seq_length1]) | input matrix | composed of the cloze text from the previous step, plus some special characters 2 | +| labels | labels: torch.Size([1]) | labels | corresponding numeric labels, such as 0,1,2... | +| position_ids | torch.Size([2， seq_length]) | position encoding | refer to [GLM process] (GLM.md), the first line represents the absolute position of the token, the second line represents the relative position of the occluded part | +| attention_mask | torch.Size([1]) | separator position | | +| target_ids | torch.Size([num_labels3]) | full label list | All label texts correspond to the labels of a single token, and then put the serial numbers of these labels into target_ids | +| logit_mask | torch.Size([seq_length]) | Whether the corresponding text is an answer | For each token, if it is an answer, the corresponding place is 1, otherwise it is 0 | + +1: seq_length represents the specified maximum length of each input vector + +2: The process of adding special characters: add the [CLS] symbol at the beginning of the sentence and the [EOS] symbol at the end of the sentence until the length reaches seq_length. If the text output by cloze has two paragraphs, it will be in the middle Add [SEP] symbol +
+ +3: num_labels represents the number of options in the cloze problem + +**b) Cloze for multiple tokens** + + + +| key | dimension | meaning | Construction method | +|-----------------|------------------|-----------------|-------------------------------------| +| input_ids | torch.Size([num_labels, seq_length]) | input matrix | copy the corresponding text num_labels copies | +| labels | labels: torch.Size([1]) | Label | | +| position_ids | torch.Size([num_labels, 2， seq_length]) | position encoding | Copy the original position code num_labels copies | +| attention_mask | torch.Size([num_labels]) | separator position | copy num_labels copies | +| target_ids | torch.Size([num_labels, seq_length]) | Information about each option | Each column of the matrix represents each option, and each row represents the text corresponding to the current option | +| logit_mask | torch.Size([num_labels, seq_length]) | Whether the corresponding text is an answer | copy num_labels copies | + + + +### Create loader + +Finally, put the data into the [PyTorch Loader](https://pytorch.org/docs/stable/data.html?highlight=dataloader#torch.utils.data.DataLoader). + +```python +loader = torch.utils.data.DataLoader(dataset, + batch_size=1, + shuffle=False, + num_workers=1, + drop_last=False, + pin_memory=False, + collate_fn=collate_fn) +``` +The data in the Dataloader can be viewed by the following methods + +```python +for data_iterator in train_loader: + for key, value in data_iterator.items(): + print(key, value) + # break +``` +Once the loader is constructed, it can be used for the subsequent training and prediction process.


+ +## Data processing: GLM pre-training task + +Sample pre-training task data format: +```text +{ + "RECORDS": [ + { + "id": 0, + "uniqueKey": "", + "titleUkey": "", + "dataType": "", + "title": "", + "content": "平安信用卡上门激活失败是为啥？平安信用卡上门激 + 活失败的主要原因有这些：申卡人是在网上申请的新用户，上门激活时 + 携带资料不足，需要申卡人携带身份证、信用卡、预留号码等去平安银 + 行网点验证身份才能激活；申卡人在平安银行预留的手机号码有误；申卡 + 人在激活时输入的相关信息不正确；申卡人在回答上门激活工作人员的问 + 题是，跟之前提交的资料有一些出入，有违规嫌疑。平安信用卡上门激活失 + 败的主要原因有这些：申卡人是在网上申请的新用户，上门激活时携带资料不 + 足，需要申卡人携带身份证、信用卡、预留号码等去平安银行网点验证身份才能 + 激活；申卡人在平安银行预留的手机号码有误；申卡人在激活时输入的相关信息不 + 正确；申卡人在回答上门激活工作人员的问题是，跟之前提交的资料有一 + 些出入，有违规嫌疑。" + }, + ] +} +``` +Pre-trained task processing example code: + +```python +tokenizer = GLMLargeChTokenizer(add_block_symbols=True, + add_task_mask=True, + add_decoder_mask=False, + fix_command_token=True) + +ds_args = DatasetArguments() + +tokenizer = GLMLargeChTokenizer(fix_command_token=True, + add_block_symbols=True, + add_task_mask=True, + add_decoder_mask=False) + +ds_args = add_args(ds_args, tokenizer) + +def create_dataset(tokenizer, should_split): + dataset = get_dataset_lazy("./examples/glm_pretrain/data", # load + tokenizer=tokenizer, + pre_tokenize=True, + num_processes=10, + no_lazy_loader=True) + if should_split: + datasets = split_ds(dataset, split=[.8, .2, .0], shuffle=True) # Manual segmentation + else: + datasets = [dataset] + + datasets = [ + BlockDataset(ds, + tokenizer, + max_seq_len=512, + sample_across_doc=True, + non_sentence_start=0.0) if ds is not None else None + for ds in datasets + ] + return datasets + +datasets = create_dataset(tokenizer, should_split=True) +``` + +Pre-training data processing also follows the same process, with the following differences +1. The pre-training data set is not divided into training set, validation set, and test set by default, so it needs to be divided manually +2. Since the pre-training data set is generally relatively large, lazy loading is used. Lazy loading only instantiates the object when it is actually used, which is a relatively resource-saving operation. +3. During pre-training, the collate function will randomly process data according to three different modes: bert mode (occlude random intervals), sentence mode (occlude according to complete sentences) and gpt mode (occlude only a long section). The input of the model will also have one more `mode` key than the general generation task. +4. There is no need to add templates for pre-training, just follow the table below to build the model input + + +| key | dimension | meaning | Construction method | +|----------------|--------------------------------------|------------|--------------------------------------------------| +| input_ids | torch.Size([seq_length1]) | input matrix | consists of the template text from the previous step, plus some special characters | +| position_ids | torch.Size([2， seq_length]) | position encoding | refer to [GLM process](GLM.md), the first line represents the absolute position of the token, the second line represents the relative position of the occluded part | +| attention_mask | torch.Size([1]) | delimiter position | for the pattern of the generated class, get the position where the source text ends; otherwise get the position where the input text ends | +| target_ids | torch.Size([num_labels3]) | full label list | occluded text | +| logit_mask | torch.Size([seq_length]) | By occlusion, the model will only process the loss of the target text part | For each token, if it is an answer, the corresponding place is 1, otherwise it is 0 | +| mode | str | Data processing mode | + +## Data processing: Generating task fine-tuning +The code implementation is as follows: + +```python +import torch.utils.data +from flagai.data.dataset import Seq2SeqDataset +from flagai.data.tokenizer import GLMLargeEnWordPieceTokenizer +from tests.test_dataset_new_superglue import Seq2SeqCollateArguments +from flagai.data.dataset import ConstructSeq2seqStrategy + +# get default parameters +cl_args = Seq2SeqCollateArguments() + +# create tokenizer +tokenizer = GLMLargeChTokenizer(add_block_symbols=True, + TUTORIAL_4_DATASET.md add_task_mask=False, + add_decoder_mask=False, + fix_command_token=False) + +# Initially read and process the dataset +dataset = Seq2SeqDataset(task_name='cmrc', + data_dir='./datasets/', + dataset_type='train', + tokenizer=tokenizer) + +# build collate function +collate_fn = ConstructSeq2seqStrategy(cl_args, tokenizer, task_name="rte") + +# Create a loader +loader = torch.utils.data.DataLoader(dataset, + batch_size=1, + shuffle=False, + num_workers=1, + drop_last=False, + pin_memory=False, + collate_fn=collate_fn) +``` + +### Initially read and process the dataset + +Currently, [CMRC2018](https://www.clue.ai/introduce.html) task is supported. CMRC is a reading comprehension task that needs to answer a series of questions based on the background text. An example of its data structure is as follows: + +```text +{'paragraphs': + [{'id': 'TRAIN_186', + 'context': '范廷颂枢机（，），圣名保禄·若瑟（），是越南罗马天主教枢机。1963年被任 + 为主教；1990年被擢升为天主教河内总教区宗座署理；1994年被擢升为总主教，同年年底被擢 + 升为枢机；2009年2月离世。范廷颂于1919年6月15日在越南宁平省天主教发艳教区出生；童年 + 时接受良好教育后，被一位越南神父带到河内继续其学业。范廷颂于1940年在河内大修道院完 + 成神学学业。范廷颂于1949年6月6日在河内的主教座堂晋铎；及后被派到圣女小德兰孤儿院服 + 务。1950年代，范廷颂在河内堂区创建移民接待中心以收容到河内避战的难民。1954年，法越 + 战争结束，越南民主共和国建都河内，当时很多天主教神职人员逃至越南的南方，但范廷颂仍然 + 留在河内。翌年管理圣若望小修院；惟在1960年因捍卫修院的自由、自治及拒绝政府在修院设 + 政治课的要求而被捕。1963年4月5日，教宗任命范廷颂为天主教北宁教区主教，同年8月15日就 + 任；其牧铭为「我信天主的爱」。由于范廷颂被越南政府软禁差不多30年，因此他无法到所属堂 + 区进行牧灵工作而专注研读等工作。范廷颂除了面对战争、贫困、被当局迫害天主教会等问题外 + ，也秘密恢复修院、创建女修会团体等。1990年，教宗若望保禄二世在同年6月18日擢升范廷颂 + 为天主教河内总教区宗座署理以填补该教区总主教的空缺。1994年3月23日，范廷颂被教宗若望 + 保禄二...', + 'qas':{'question': '范廷颂是何时去世的？', 'id': 'TRAIN_186_QUERY_4', + 'answers': [{'text': '范廷颂于2009年2月22日清晨在河内离世', 'answer_start': 759}]}]}], + 'id': 'TRAIN_186', 'title': '范廷颂'} +``` +When using it, we can change the `task_name` parameter to `cmrc`. The implementation process is similar to the fine-tuning of the classification task, and the data set will be initially processed into the same structure in the end. The corresponding code is as follows: + +```python +dataset = Seq2SeqDataset(task_name='cmrc', data_dir='./datasets/', + dataset_type='train', tokenizer=tokenizer) +``` + +### Organize the data into input to the model + +The code is shown below. Compared with the generation task, it is also the construction template and model input, the difference is that the construction method is different + +```python +collate_fn = ConstructSeq2seqStrategy(cl_args, + tokenizer, + task_name=task_name) +``` + +#### 1. Build a fill-in-the-blank template +Since it is a reading comprehension task, it needs to be reflected in the template to answer the specified reading comprehension question, refer to the following construction method +
+ + +#### 2. Word segmentation and construct input samples +Similar to pre-training, the difference is that there is no mode key. diff --git a/docs/TUTORIAL_2_PROJECT_STRUCTURE.md b/docs/TUTORIAL_2_PROJECT_STRUCTURE.md deleted file mode 100644 index 881c918d..00000000 --- a/docs/TUTORIAL_2_PROJECT_STRUCTURE.md +++ /dev/null @@ -1,25 +0,0 @@ -# Project Structure - -The structure of *flagai* -``` -Sailing/ - |--flagai/ - | |--data/ # contains datasets and tokenizers - | |--model/ # contains models,blocks and layers (layers->block->model) - | |--fp16/ # contains fp16 tools - | |--mpu/ # fork from megatron-lm - | |--docs/ # documentations - | |--trainer.py/ # trainer for pytorch, deepspeed+mpu - | |--logging.py # - | |--metrics.py/ # contains frequently-used metrics, e.g., accuracy - | |--optimizers.py - | |--schedulers.py.py - | |--test_utils.py # contains tools for testing - | |--utils.py - |--setup.py - |--test.py # for excuting all tests - |--README.md - |--requirements.txt - |--logo.jpg - -``` \ No newline at end of file diff --git a/docs/TUTORIAL_3_MODEL.md b/docs/TUTORIAL_3_MODEL.md new file mode 100644 index 00000000..90f71fdb --- /dev/null +++ b/docs/TUTORIAL_3_MODEL.md @@ -0,0 +1,109 @@ +# Base class + +The base class BaseModel implements common methods for loading/saving models from a local file or directory or from a library-provided pretrained model configuration (downloaded from BAAI modelhub's Kingsoft S3 repository). +All supported models now support the three most common model types [encoder, decoder and encoder-decoder]. GLM models can now load all GLM series models, see https://github.com/THUDM/GLM + +## From_pretrain + +Models with the same model structure can be loaded with the same class. For example, BERT-base and Roberta-base models can be loaded with the BertModel class. From_pretrain is optimized for data/model parallel model loading to avoid resource waste caused by repeated downloads. +By calling ClassName.from_pretrian() to load, now our model hub supports the following models, you can directly download the model configuration file [config.json], model weights [pytorch_model.bin], and dictionary files [vocab .txt]. example: + +````python +from flagai.model.glm_model import GLMForSingleTokenCloze +model = GLMForSingleTokenCloze.from_pretrain(download_path="./state_dict", model_name="GLM-large-ch") +```` + +If the model weights are loaded locally, they can also be loaded through ClassName.from_pretrain(). example: +Load the model file `pytorch_model.bin` from the `./state_dict/GLM-large-ch` directory + +````python +from flagai.model.glm_model import GLMForSingleTokenCloze +model = GLMForSingleTokenCloze.from_pretrain(download_path="./state_dict", + model_name="GLM-large-ch") +```` + +## All supported models + +| ClassName | ModelName | Language | Model Type | +|-----------------------------------|------------- ----|----------|------------| +| flagai.model.glm_model.GLMModel | GLM-10b-ch | chinese | encoder | +| flagai.model.glm_model.GLMModel | GLM-large-ch | chinese | encoder | +| flagai.model.bert_model.BertModel | RoBERTa-base-ch | chinese | encoder | +| flagai.model.gpt2_model.GPT2Model | GPT2_base_ch | chinese | decoder | +| flagai.model.t5_model.T5Model | T5-base-ch | chinese | enc2dec | +| flagai.model.t5_model.T5Model | T5-base-en | chinese | enc2dec | +| flagai.model.bert_model.BertModel | BERT-base-en | english | encoder | +| flagai.model.glm_model.GLMModel | GLM-large-en | english | encoder | + +## Supported models + tasks + +At the same time, we support the finetuned model on the task, as shown in the table below, the model weights can be loaded through ClassName.from_pretrain(), for example, we automatically download and load a GLM trained on the title-generation task -large-ch model: + +````python +from flagai.model.glm_model import GLMForSeq2Seq +model = GLMForSeq2Seq.from_pretrain(model_name='GLM-large-ch') +```` + +We also provide the AutoLoader class to help load models. For example, the GLM-large-ch model is used for seq2seq tasks. Here we adopt a task- and model-independent design. In theory, tasks and models can be freely replaced. + +````python +from flagai.auto_model.auto_loader import AutoLoader +auto_loader = AutoLoader("seq2seq", + model_name="GLM-large-ch", + model_dir= "./state_dict") +model = auto_loader.get_model() +```` + +| ClassName | Model Name | language | Task | +|------------------------------------------------- |-----------------|----------|-------------------| +| flagai.model.glm_model.GLMForSeq2Seq | GLM-large-ch | chinese | title generation | +| flagai.model.glm_model.GLMForSeq2Seq | GLM-large-ch | chinese | poetry generation | +| flagai.model.bert_model.BertForSequenceLabeling | RoBERTa-base-ch | chinese | title generation | +| flagai.model.bert_model.BertForSequenceLabeling | RoBERTa-base-ch | chinese | NER | +| flagai.model.bert_model.BertForSequenceLabeling | RoBERTa-base-ch | chinese | semantic matching | +| flagai.model.t5_model.T5Model | T5-base-ch | chinese | title generation | +| flagai.model.bert_model.BertForSequenceLabeling | BERT-base-en | english | title gneration | + +## Model design + +The main construction logic of the model `layer->block>model` +`flagai.model.layer`: including mlp, layernorm, activation, attention and other layers + +`flagai.model.block`: Build a transformer block by assembling various layers, such as BERT block, etc. + +`flagai.model`: build the model by embedding layers and stacked blocks + +## forward function + +Model's forward function: +Input is keyword arguments: including input_ids, position_ids, attention_mask, etc., redundant parameters will be automatically ignored +For example, GLM's forward function: + +````python +def forward(self, + input_ids=None, + position_ids=None, + attention_mask=None, + mems=None, + return_memory=False, + detach_memory=True, + prompt_pos=None, + **kwargs) +```` + +The output is a dictionary, including logits and hidden states, which are required, such as the return of the GLM forword function: + +````python +return {'loss': loss, 'logits': logits, 'hidden_states': mems} +```` + +## init_from_json +Model's init_from json function: +The input is a dictionary, the output is an initialized model +For example, the invocation of GLMModel is as follows: + +````python +GLMModel.init_from_json(config_file = "./config.json", **kwargs) +```` + +**kwargs are reserved parameters, in order to be compatible with new initialization parameters of some models diff --git a/docs/TUTORIAL_3_TOKENIZER.md b/docs/TUTORIAL_3_TOKENIZER.md deleted file mode 100644 index 88664db1..00000000 --- a/docs/TUTORIAL_3_TOKENIZER.md +++ /dev/null @@ -1,85 +0,0 @@ -# Tokenizer - -## Supported tokenizers -| Tokenizer | Language | Related model (see [ModelHub](https://model.baai.ac.cn/models)) | -|------------------------------|----------|-----------------------------------------------------------------| -| GLMLargeEnWordPieceTokenizer | English | GLM-large-en | -| GLMLargeChTokenizer | Chinese | GLM-large-ch | -| GLM10bENBPETokenizer | English | glm_10b_en | -| T5BPETokenizer | Chinese | t5_base | -| ROBERTATokenizer | Chinese | RoBERT-base-ch | -| BertWordPieceTokenizer | Chinese | | - -## Introduction - -Natural language is usually expressed in the form of sequential symbols, -where we call each of those symbols a **character**. - -In any text, each semantic unit is composed of several consecutive characters, -and we call each semantic unit a **token**. For example, 'cat', which refers to a -kind of small domesticated mammal, is expressed as 'c', 'a' and 't'. The process -of splitting texts into a sequence of tokens is defined as **tokenization**, -and an example is shown below: - - -Original sentence: Jack is walking a dog. - -Tokenized sentence: [Jack, is, walking, a, dog, .] - -It is noticeable that token is not equivalent to word. It can also be character, sub-word, sentence piece and so on as long as it holds appropriate amount of semantic information. - -Another critical preprocessing step is vectorization, which turns the raw symbolic sequences into a numeric vector or matrix so that it can be directly fed into our language model. Usually there is achieved by using a vocabulary file to map each token to its corresponding id. - -In our project, there are a bunch of tokenizer classes, where each of them can tokenize and vectorize raw texts in different ways, and there are also other important functions. - -## Loading tokenizer -Load an existing tokenizer: -```python -from flagai.data.tokenizer import GLMLargeEnWordPieceTokenizer -tokenizer = GLMLargeEnWordPieceTokenizer() -``` - -## Creating tokenizer -To create a new tokenizer, you need to: -### 1. Create a package under `/flagai/tokenizer` - -### 2. create a python file to define the tokenizer - -Initialize the tokenizer as below (let's take T5 tokenizer as an example) - -```python -from transformers import T5Tokenizer -from ..tokenizer import Tokenizer -class T5BPETokenizer(Tokenizer): - def __init__(self, model_type_or_path="t5-base", cache_dir=None): - self.text_tokenizer = T5Tokenizer.from_pretrained(model_type_or_path, - cache_dir=cache_dir) - self.text_tokenizer.max_len = int(1e12) -``` -If tokenizer model imported from transformers is used as the text tokenizer, it is all done! - -Otherwise, you need to implement the following class functions by your own. - - -```python -def EncodeAsIds(self, text: str, process_fn=None): - """Input text string => a list of token ids""" - -def EncodeAsTokens(self, text: str, process_fn=None): - """Input text string => a list of tokens""" - -def IdToToken(self, Id: int): - """Token id => token""" - -def TokenToId(self, token: str): - """Token => token id""" - return self.text_tokenizer._convert_token_to_id(token) - -def DecodeIds(self, Ids: list[int]): - """A list of token ids => recovered text string""" - return self.DecodeTokens([self.IdToToken(id) for id in Ids]) - -def DecodeTokens(self, tokens: list[str]): - """A list of tokens => recovered text string""" - return self.text_tokenizer.convert_tokens_to_string(tokens) -``` \ No newline at end of file diff --git a/docs/TUTORIAL_4_DATASET.md b/docs/TUTORIAL_4_DATASET.md deleted file mode 100644 index 8efb7f75..00000000 --- a/docs/TUTORIAL_4_DATASET.md +++ /dev/null @@ -1,137 +0,0 @@ -# Datasets - -## Supported downstream datasets -Our project now supports 12 datasets as listed below: - -| Dataset Name | Identifier | Language | Source Benchmark | -|------------------------|------------|----------|------------------| -| Broadcoverage Diagnostics | BoolQ | English | SuperGLUE | -| CommitmentBank | CB | English | SuperGLUE | -| Choice of Plausible Alternatives | COPA | English | SuperGLUE | -| Multi-Sentence Reading Comprehension | MultiRC | English | SuperGLUE | -| Recognizing Textual Entailment | RTE | English | SuperGLUE | -| Words in Context | WiC | English | SuperGLUE | -| The Winograd Schema Challenge | WSC | English | SuperGLUE | -| Ant Financial Question Matching Corpus | AFQMC | Chinese | CLUE | -| Short Text Classificaiton for News | TNEWS | Chinese | CLUE | -| Reading Comprehension for Simplified Chinese | CMRC2018 | Chinese | CLUE | - - -## Load datasets - -Let's load a SuperGlue Dataset as following: - -```python -import torch.utils.data -from flagai.data.dataset import SuperGlueDataset -from flagai.data.tokenizer import GLMLargeEnWordPieceTokenizer -from tests.test_dataset_new_superglue import CollateArguments -from flagai.data.dataset import ConstructSuperglueStrategy - -# Construct the optional arguments -cl_args = CollateArguments() - -# Build the English large tokenizer for GLM -tokenizer = GLMLargeEnWordPieceTokenizer() - -# Build the rte dataset from SuperGLUE RTE task -dataset = SuperGlueDataset(task_name='rte', data_dir='/mnt/datasets/yan/', dataset_type='train', - tokenizer=tokenizer) - -# Build the collate function -collate_fn = ConstructSuperglueStrategy(cl_args, tokenizer, task_name="rte") - -# Construct the train loader -loader = torch.utils.data.DataLoader(dataset, - batch_size=1, - shuffle=False, - num_workers=1, - drop_last=False, - pin_memory=False, - collate_fn=collate_fn) - -# Iterating loader -it = iter(loader) -next(it) -batch = next(it) - -# Print results -print(batch['input_ids'].tolist()) -print(tokenizer.DecodeIds(batch['input_ids'].tolist()[0])) -print(tokenizer.DecodeIds(batch['target_ids'].tolist()[0])) -``` - -## Create datasets -Under `flagai/data/dataset/superglue/control.py`, create your own processor and pvp as shown in examples below, and add them to the corresponding dictionary. -```python -class ExampleProcessor(DataProcessor): - - def get_train_examples(self, data_dir): - # Assign the filename of train set - return self._create_examples(os.path.join(data_dir, "train.tsv"), "train") - - def get_dev_examples(self, data_dir, for_train=False): - # Assign the filename of dev set - return self._create_examples(os.path.join(data_dir, "dev.tsv"), "dev") - - def get_test_examples(self, data_dir) -> List[InputExample]: - # Assign the filename of test set - return self._create_examples(os.path.join(data_dir, "test.tsv"), "test") - - def get_labels(self): - # Return all label categories - return ["0", "1"] - - @staticmethod - def _create_examples(path: str, set_type: str) -> List[InputExample]: - """ - Construct the Input example, which contains the following keys - text_a (str, required): The content text - text_b (str, optional): Usually the - label (str, required): the labels - guid (str, required): A unique id to one InputExample element - """ - examples = [] - df = read_tsv(path) - - for idx, row in df.iterrows(): - guid = f"{set_type}-{idx}" - text_a = punctuation_standardization(row['sentence']) - label = row.get('label', None) - example = InputExample(guid=guid, text_a=text_a, label=label) - examples.append(example) - return examples - - -class ExamplePVP(PVP): - # Map the actual token (in original file) to the actual meaning of it - VERBALIZER = {"0": ["中立"], - "1": ["利好"], - "2": ["利空"]} - - @staticmethod - def available_patterns(): - # Return ids of all available patterns - return [0] - - @property - def is_multi_token(self): - # If the label can contain more than 1 token, return True - return True - - def get_parts(self, example: InputExample) -> FilledPattern: - # Organize the elements in InputExample into a designed pattern - text_a= self.shortenable(example.text_a) - if self.pattern_id == 0: - return ["标题：", text_a, "类别：", [self.mask]], [] - else: - raise NotImplementedError("No pattern implemented for id {}".format(self.pattern_id)) - - def verbalize(self, label) -> List[str]: - if self.pattern_id == 0: - return WankePVP.VERBALIZER_A[label] - else: - raise NotImplementedError -``` -More information about why and how to create our own dataset can be viewed [here](APPENDIX_TASK.md). - diff --git a/docs/TUTORIAL_4_TRAINER.md b/docs/TUTORIAL_4_TRAINER.md new file mode 100644 index 00000000..d12d0415 --- /dev/null +++ b/docs/TUTORIAL_4_TRAINER.md @@ -0,0 +1,302 @@ +# Trainer + +The Trainer class provides APIs for training with multiple parallel frameworks. The API supports distributed training with Pytorch DDP/Deepspeed on multiple GPUs, as well as mixed parallel distributed training with Megatron-LM+Deepspeed, and mixed precision via NVIDIA Apex. + +## Getting Started +Trainer includes basic training loops that support the above features. To customize the behavior, you can subclass them and override the forward_step method: +The return of the forward_step method is a dict + +```python +from flagai.trainer import Trainer + +class MyTrainer(Trainer): + + def forward_step(self, data, model, mems): + + model_outputs = model(**data) + output = {} + output['loss'] = model_outputs.loss + output['logits'] = model_outputs.logits + output['hidden_states'] = model_outputs.decoder_hidden_states + return output +``` + +There is a parameter env_type in Trainer that controls whether the training is distributed or not. + +```shell +The enviroment type for training. Will default to 'pytorch'. +env_type: `pytorch`, `pytorchDDP`, `deepspeed`, `deepspeed+mpu` + pytorch: single node cpu/gpu + pytorchDDP: single-/multi- node gpu + deepspeed: single-/multi- node gpu + deepspeed+mpu: single-/multi- node gpu +``` + +## Single node cpu/gpu + +```python +trainer = MyTrainer( + env_type='pytorch', + epochs=1, + batch_size=4, + eval_interval=100000, + log_interval=10, + experiment_name='t5-11b', + pytorch_device='cpu', + load_dir=None, + lr=1e-4) +``` +Specify the graphics card/cpu settings as pytorch_device, 'cpu', 'cuda:0', etc. +## fp16 + +```python +Model parameters turned to fp16 +trainer = MyTrainer( + env_type='pytorch', + epochs=1, + batch_size=1, + eval_interval=10, + log_interval=10, + experiment_name='t5-11b', + pytorch_device='cuda:0', + load_dir=None, + lr=1e-4, + fp16=True) # change to `True` +``` +## Gradient recomputation +Do not save the Intermediate results in the forward stage. Paper: [Training Deep Nets with Sublinear Memory Cost](https://arxiv.org/abs/1604.06174v2) + +Now, we can train/finetune a t5-11b with gradient_accumulation_steps. + +```python +from transformers import T5ForConditionalGeneration, T5Tokenizer +tokenizer = T5Tokenizer.from_pretrained('t5-11b') +model = T5ForConditionalGeneration.from_pretrained('t5-11b') +model.gradient_checkpointing = True +``` +## Support huggingface model +The example directory location of the FlagAI project：examples/t5_huggingface +t5-11b paper: [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://arxiv.org/pdf/1910.10683.pdf) + +The example train t5-11b on a toy dataset. You need a device > 30G memory to run the example. If your have multiple devices, please check out the following session for speedup. + +```python +import sys +from flagai.trainer import Trainer +from transformers import T5ForConditionalGeneration, T5Tokenizer +from torch.utils.data import Dataset +import torch + +## Inheriant the Trainer +## overload the forward_step function +class MyTrainer(Trainer): + + def forward_step(self, data, model, mems): + """ + Args: + data: a dict contains a batch of inputs + return: + output: a dict contains `loss` + """ + model_outputs = model(**data) + output = {} + output['loss'] = model_outputs.loss + output['logits'] = model_outputs.logits + output['hidden_states'] = model_outputs.decoder_hidden_states + return output + +# get a customized trainer instance +trainer = MyTrainer( + env_type='pytorch', + epochs=1, + batch_size=4, + eval_interval=10, + log_interval=10, + experiment_name='t5-11b', + pytorch_device='cuda:0', + load_dir=None, + lr=1e-4, + fp16=False) + +# using huggingface transformers to get tokenizer and models +model_name = 't5-11b' +tokenizer = T5Tokenizer.from_pretrained(model_name) +model = T5ForConditionalGeneration.from_pretrained(model_name) + +print("loading model & tokenizer is done!") +src_dir = 'train_inputs.txt' +tgt_dir = 'train_targets.txt' +model_dir = "./t5-11b" # 模型位置 +maxlen = 1024 + + +def read_file(): + src = [] + tgt = [] + + with open(src_dir, 'r', encoding='utf-8') as f: + lines = f.readlines() + for line in lines: + src.append(line.strip('\n').lower()) + + with open(tgt_dir, 'r', encoding='utf-8') as f: + lines = f.readlines() + for line in lines: + tgt.append(line.strip('\n').lower()) + return src, tgt + + +class BertSeq2seqDataset(Dataset): + + def __init__(self, sents_src, sents_tgt, tokenizer, maxlen=512): + super(BertSeq2seqDataset, self).__init__() + self.sents_src = sents_src + self.sents_tgt = sents_tgt + self.tokenizer = tokenizer + self.maxlen = maxlen + + def __getitem__(self, i): + src = self.sents_src[i] + tgt = self.sents_tgt[i] + inputs = tokenizer(src) + with tokenizer.as_target_tokenizer(): + labels = tokenizer(tgt) + output = {} + output['input_ids'] = inputs.input_ids + output['labels'] = labels.input_ids + return output + + def __len__(self): + return len(self.sents_src) + + +def seq2seq_collate_fn(batch): + + def padding(indice, max_length, pad_idx=0): + + pad_indice = [ + item + [pad_idx] * max(0, max_length - len(item)) + for item in indice + ] + return torch.tensor(pad_indice) + + token_ids = [data["input_ids"] for data in batch] + max_length_tk = max([len(t) for t in token_ids]) + labels = [data["labels"] for data in batch] + max_length_lb = max([len(t) for t in labels]) + + token_ids_padded = padding(token_ids, max_length_tk) + labels_padded = padding(labels, max_length_lb) + + data = {"input_ids": token_ids_padded, "labels": labels_padded} + + return data + + +sents_src, sents_tgt = read_file() +data_len = len(sents_tgt) +train_size = int(data_len * 0.8) +train_src = sents_src[:train_size] +train_tgt = sents_tgt[:train_size] + +val_src = sents_src[train_size:] +val_tgt = sents_tgt[train_size:] + +train_dataset = BertSeq2seqDataset(train_src, + train_tgt, + tokenizer=tokenizer, + maxlen=maxlen) +val_dataset = BertSeq2seqDataset(val_src, + val_tgt, + tokenizer=tokenizer, + maxlen=maxlen) +## Training +trainer.train(model, + train_dataset=train_dataset, + collate_fn=seq2seq_collate_fn) +``` + +## Distributed training +To accelerate your training/finetuning process, FlagAI integrate three popular parallel frameworks for deep neural networks. +### pytorchDDP +DistributedDataParallel (DDP) link +We can use distributed training by modifying a few parameters. The launch call is directly encapsulated inside, and python script_name.py [script_name: is the name of the training script] can be run directly. + +```python +trainer = MyTrainer( + env_type='pytorchDDP', + epochs=1, + batch_size=4, + eval_interval=10, + log_interval=10, + experiment_name='t5-11b', + load_dir=None, + lr=1e-4 + # parameters for pytorchDDP + master_ip='127.0.0.1', + master_port=17750, + num_nodes=1, + num_gpus=2, + hostfile='./hostfile', + training_script=__file__, +) +``` +### deepspeed +Compared with pytorch, deepspeed mainly optimizes the optimizer. +It provides an optimizer of cpu-offload, which can greatly reduce the occupation of gpu video memory. +For the relevant settings of the deepspeed optimizer, the configuration file of deepspeed.json needs to be provided + +```python +trainer = MyTrainer( + env_type='pytorchDDP', + epochs=1, + batch_size=4, + eval_interval=10, + log_interval=10, + experiment_name='t5-11b', + load_dir=None, + lr=1e-4 + # parameters for pytorchDDP + master_ip='127.0.0.1', + master_port=17750, + num_nodes=1, + num_gpus=2, + hostfile='./hostfile', + training_script=__file__, + # deepspeed + deepspeed_config='deepspeed.json' +) +``` + +Deepspeed config file reference [examples/t5_huggingface/deepspeed.json] +Among them, the main parameters are stage and cpu offload + +### deepspeed + megatron-lm + +Now the tens of billions of model GLM adopts the model parallel technology of Megatron-LM. When the model parameter scaling is above 10b, it is difficult to load a single model and all the intermediate variables during training in the video memory of a single card. To this end, Megatron-LM provides a Tensor segmentation method. The main idea is to segment the matrix according to rows/columns. FlagAI link, convert the model to the Megatron-LM version. +As follows, FlagAI's internal models (GLM, T5, BERT [including RoBERTa], GPT2) support Megatron-LM. As long as the environment variable is changed to deepspeed+mpu in the configuration file, the model parallel function can be started. +For the huggingface version of the model, there is no support for model parallelism. + +```python +trainer = MyTrainer( + env_type="deepspeed+mpu", # env_type + epochs=1, + batch_size=8, + eval_interval=10, + log_interval=10, + experiment_name='t5-11b', + load_dir=None, + lr=1e-4, + # parallel settings + master_ip='127.0.0.1', + master_port=17750, + num_nodes=1, + num_gpus=4, + hostfile='hostfile', + training_script=__file__, + # deepspeed + deepspeed_config='deepspeed.json', + # megatron-lm + model_paralle_size = 2 +) +``` diff --git a/docs/TUTORIAL_5_INSTRUCTIONS_FOR_AutoLoader.md b/docs/TUTORIAL_5_INSTRUCTIONS_FOR_AutoLoader.md new file mode 100644 index 00000000..ac2be717 --- /dev/null +++ b/docs/TUTORIAL_5_INSTRUCTIONS_FOR_AutoLoader.md @@ -0,0 +1,87 @@ +## AutoLoader + +### Quickly build large models that support different downstream tasks through AutoLoader. + +![](./img/autoloader_en_map.png) + +AutoLoader can quickly find the corresponding pre-trained model and tokenizer, just enter task_name and model_name. + +Take title generation tasks as an example: +```python +from flagai.auto_model.auto_loader import AutoLoader +auto_loader = AutoLoader(task_name="title-generation", ## The task name + model_name="RoBERTa-base-ch", ## The model name. + ) +model = auto_loader.get_model() +tokenizer = auto_loader.get_tokenizer() +``` +```task_name``` is the name of the task you want to do, in addition to ```title-generation```, you can also choose other subdivided tasks, such as ```semantic-matching```、```ner``` and so on. +```python +target = ["O", "B-LOC", "I-LOC", "B-ORG", "I-ORG", "B-PER", "I-PER"] +from flagai.auto_model.auto_loader import AutoLoader +auto_loader = AutoLoader(task_name="ner", + model_name="RoBERTa-base-ch", + class_num=len(target)) +model = auto_loader.get_model() +tokenizer = auto_loader.get_tokenizer() +``` +When building the model of classification related task, you also need to input a ``class_num`` parameter to tell the model how many classed to classify. + +Then AutoLoader will download the roberta pretrained model, config and vocab from model hub. +And the downloaded pretrained model, config and vocab will be put into the "./state_dict/RoBERTa-base-ch" directory. + +In addition, by entering a different model_name, you can also directly call the already trained downstream task model, such as ``Roberta-base-ch-ner``, ``Roberta-base-ch-title-generation`` and so on. + +```python +target = ["O", "B-LOC", "I-LOC", "B-ORG", "I-ORG", "B-PER", "I-PER"] +from flagai.auto_model.auto_loader import AutoLoader +auto_loader = AutoLoader(task_name="ner", + model_name="RoBERTa-base-ch-ner", + class_num=len(target)) + +model = auto_loader.get_model() +tokenizer = auto_loader.get_tokenizer() +``` +See https://github.com/BAAI-Open/FlagAI/blob/master/quickstart/ner_ch.py to learn more. +## All supported tasks and models + +![model_and_task_table](./img/model_task_table.png) +The **task_name** parameter supports: +### classification +1. classification +2. semantic-matching +3. emotion-analysis (todo) +4. text-classification (todo) +5. ... + +### seq2seq +1. seq2seq +2. title-generation +3. writing +4. poetry-generation +5. couplets-generation (todo) +6. ... +### sequence labeling +1. sequence-labeling +2. ner +3. ner-crf +4. ner-gp +5. part-speech-tagging (todo) +6. chinese-word-segmentation (todo) +7. ... + +## All supported models +All supported models is can be found in **model hub**. +Different models adapt to different tasks. + +#### Transfomrer encoder: + +For example, model_name="GLM-large-ch" or "RoBERTa-base-ch" These models support all of the tasks mentioned in the previous section. + +#### Transformer decoder: + +For example, model_name="GPT2-base-ch", the model support "seq2seq" related task. + +#### Transformer encoder + decoder: + +For example model_name="t5-base-ch", the model support "seq2seq" related task. diff --git a/docs/TUTORIAL_6_INSTRUCTIONS_FOR_PREDICTOR.md b/docs/TUTORIAL_6_INSTRUCTIONS_FOR_PREDICTOR.md new file mode 100644 index 00000000..55ecb098 --- /dev/null +++ b/docs/TUTORIAL_6_INSTRUCTIONS_FOR_PREDICTOR.md @@ -0,0 +1,100 @@ +## Predictor + +### Call a line of code through Predictor to get the ideal prediction result + +For different tasks, such as title generation, named entity recognition, text classification, etc., the way of prediction varies. At the same time, for different models, such as encoder, decoder, and encoder-decoder models, the prediction methods are also different. + +Predictor integrates different tasks and prediction codes of different models. The text is passed in through Pipline. Predictor will quickly parse the model type, call different prediction codes, and get the prediction results of the corresponding model. + +![](./img/predictor_map.png) + +Taking the gpt2 article continuation task as an example, the random sampling generation method is adopted: + +```python +from flagai.auto_model.auto_loader import AutoLoader +from flagai.model.predictor.predictor import Predictor +import torch +if __name__ == '__main__': + device = torch.device("cuda" if torch.cuda.is_available() else "cpu") + + ## load model and tokenizer by the Autoloader + loader = AutoLoader(task_name="writing", + model_name="GPT2-base-ch") + model = loader.get_model() + tokenizer = loader.get_tokenizer() + model.to(device) + ## define predictor + predictor = Predictor(model, tokenizer) + ## define the input + text = "今天天气不错，" + ## analyse the model class and to generate + out = predictor.predict_generate_randomsample(text, ## input + input_max_length=512, ## max in length + out_max_length=100, ## max out length + repetition_penalty=1.5, ## avoid output repeated. (https://arxiv.org/pdf/1909.05858.pdf) + top_k=20, ## top k + top_p=0.8) ## (http://arxiv.org/abs/1904.09751) + + print(f"out is {out}") + ### out is 到这里来看了一下，很是兴奋，就和朋友一起来这里来了。我们是周五晚上去的，人不多，所以没有排队，而且这里的环境真的很好，在这里享受美食真的很舒服，我们点了一个套餐，两个人吃刚刚好，味道很好。 +``` +Predictor can automatically analyze that the model type is GPT2, and automatically call the generation method corresponding to the GPT2 model to obtain the prediction result. + +In addition to the ``writing`` task, Predictor also supports the prediction of ```title-generation```, ``ner``, ``semantic-matching`` and other tasks. For example the ner task: + +```python +import torch +from flagai.auto_model.auto_loader import AutoLoader +from flagai.model.predictor.predictor import Predictor +device = torch.device("cuda" if torch.cuda.is_available() else "cpu") + +target = ["O", "B-LOC", "I-LOC", "B-ORG", "I-ORG", "B-PER", "I-PER"] +auto_loader = AutoLoader(task_name="ner", + model_name="RoBERTa-base-ch-ner", # Load the checkpoints from Modelhub(model.baai.ac.cn/models) + class_num=len(target)) + +model = auto_loader.get_model() +tokenizer = auto_loader.get_tokenizer() +model.to(device) + +predictor = Predictor(model, tokenizer) + +test_data = [ + "6月15日，河南省文物考古研究所曹操高陵文物队公开发表声明承认：“从来没有说过出土的珠子是墓主人的", + "4月8日，北京冬奥会、冬残奥会总结表彰大会在人民大会堂隆重举行。习近平总书记出席大会并发表重要讲话。在讲话中，总书记充分肯定了北京冬奥会、冬残奥会取得的优异成绩，全面回顾了7年筹办备赛的不凡历程，深入总结了筹备举办北京冬奥会、冬残奥会的宝贵经验，深刻阐释了北京冬奥精神，对运用好冬奥遗产推动高质量发展提出明确要求。", + "当地时间8日，欧盟委员会表示，欧盟各成员国政府现已冻结共计约300亿欧元与俄罗斯寡头及其他被制裁的俄方人员有关的资产。", + "这一盘口状态下英国必发公司亚洲盘交易数据显示博洛尼亚热。而从欧赔投注看，也是主队热。巴勒莫两连败，", +] + +for t in test_data: + entities = predictor.predict_ner(t, target, maxlen=256) + result = {} + for e in entities: + if e[2] not in result: + result[e[2]] = [t[e[0]:e[1] + 1]] + else: + result[e[2]].append(t[e[0]:e[1] + 1]) + print(f"result is {result}") +``` +By passing in a line of text, you can quickly get the corresponding named entity recognition task results, and the ``predict_ner`` interface adapts to all models that support ner tasks, such as BERT, Roberta, BERT-CRF, BERT-GlobalPointer, Roberta-CRF, Roberta -GlobalPointer and so on. + +![predictor-table](../docs/img/predictor_table.png) + + +### Perdictor所有支持的方法 +#### Text embedding +1. predict_embedding: Enter a text to get the embedding representation, supports models such as bert, roberta, etc. +#### Text classification, semantic matching +1. predict_cls_classifier: Input text or text pairs to get multi-classification results, and support transformer encoder models such as bert and roberta. +#### Mask LM +1. predict_masklm: Enter the text marked with [MASK] to get the original result, support transformer encoder models such as bert and roberta. +#### NER +1. predict_ner: Input text to get ner results, support transformer encoder models such as bert and roberta. +#### Generation +1. predict_generate_beamsearch: The input text gets the output text, which belongs to the seq2seq task. Supports bert, roberta, gpt2, t5 and glm models. +2. predict_generate_randomsample: The input text gets the output text, which belongs to the seq2seq task. Supports bert, roberta, gpt2, t5 and glm models. + +### Method call description +As long as the model used supports the corresponding method of Predictor, it can be directly predicted, for example: the model is GLM, T5, GPT2, because different generation methods can be called directly, but other prediction methods such as text classification and named entity recognition cannot be called. + +Bert and Roberta models support many methods and can call all the methods shown in the previous section. \ No newline at end of file diff --git a/docs/TUTORIAL_7_PROMPT_LERANING.md b/docs/TUTORIAL_7_PROMPT_LERANING.md new file mode 100644 index 00000000..144b5100 --- /dev/null +++ b/docs/TUTORIAL_7_PROMPT_LERANING.md @@ -0,0 +1,63 @@ +# Introduction to Prompt Learning + +## Background +Prompt-learning has been a widely-used paradigm in the field of Natural Language Processing (NLP), especially when data is limited. It follows the idea of pretraining-finetuning paradigm, where pre-trained language models (PLMs) will be adapted to various downstream tasks. Instead of adding task-specific objectives in finetuning, prompt-learning converts each original task to a specific prompting function. This process is named prompt engineering. + +## Prompt Engineering +Their are two main approaches in prompt engineering: cloze prompts and prefix prompts. Cloze prompts are more suitable in downstream tasks using masked LMs, while prefix prompts are generally used in text generation tasks. + +### Cloze Prompts +Generally, the input texts and labels can be obtained from the task dataset. These data will be rearranged into a cloze-style phrase, which includes two steps: +1. Applying a template which consists of an input slot, an answer slot, and self-defined natural language texts +2. Applying a verbalizer that maps the original labels to answers for the answer slot. + +As shown in the following example, the dataset contains two input texts: premise and hypothesis. The label indicates the relationship between premise and hypothesis, and there are three possible text strings for label: entailment, contradiction and neutral. + +
+ +We can design a meaningful cloze template as below + +
+ +And we can implement the template through the following code: + +```python +def get_parts(self, example: InputExample): + '''A simplified version of the actual code''' + # Construct the template + cloze_ids = [ + example.text_a,' question: is it true, false or neither that', + example.text_b, ' answer:', [self.mask]] + return cloze_ids +``` + +The label in various datasets can have multiple types including boolean, integer and string. We need to implement a verbalizer that maps the labels to the text string of choices in cloze question. For example, here the 'entailment', 'contradiction' and 'neutral' labels can be mapped to 'true' ,'false' and 'neither', and each time the answer to be filled is always selected among those choices. + +Verbalizer is essentially a mapping relationship, and we can define it in a Python dictionary as shown below: + +```python +VERBALIZER = { + "contradiction": [" false"], + "entailment": [" true"], + "neutral": [" neither"] +} +def verbalize(self, label) -> List[str]: + return CbPVP.VERBALIZER[label] +``` + +### Prefix Prompts +Instead of designing a prompt in real natural language, prefix prompts directly modifies the embedding space. It inserts a sequence of task-specific vectors and freezes the LM parameters. An advantage of prefix prompts is that embeddings of templates are no longer limited by parameters in pre-trained LMs. + +Here is an example that compares prefix tuning and fine-tuning in the embedding space of transformers，the grey parts of the transformer are frozen during prefix tuning: + +
+ + +## Reference +[Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://arxiv.org/abs/2107.13586) + +[OpenPrompt: An Open-source Framework for Prompt-learning](https://arxiv.org/abs/2111.01998) + +[Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://arxiv.org/abs/2001.07676) + +[Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://arxiv.org/abs/2101.00190) \ No newline at end of file diff --git a/docs/EnvironmentSetup.md b/docs/TUTORIAL_8_ENVIRONMENT_SETUP.md similarity index 100% rename from docs/EnvironmentSetup.md rename to docs/TUTORIAL_8_ENVIRONMENT_SETUP.md diff --git a/docs/TUTORIAL_8_TRAINING.md b/docs/TUTORIAL_8_TRAINING.md deleted file mode 100644 index 386587ae..00000000 --- a/docs/TUTORIAL_8_TRAINING.md +++ /dev/null @@ -1,126 +0,0 @@ - -## Finetuning model with GPU - -Let's run finetuning over SuperGlue Dataset, as following `train.py`: - -```python -from flagai.trainer import Trainer -from flagai.model.glm_model import GLMModel,GLMForSequenceClassification -from flagai.data.tokenizer import GLMBertWordPieceTokenizer -from flagai.metrics import accuracy_metric -from flagai.data.dataset import SuperGlueDataset -from flagai.test_utils import CollateArguments - - trainer = Trainer(env_type='pytorch', - epochs=1, - batch_size=4, - eval_interval=100, - log_interval=50, - experiment_name='glm_large', - pytorch_device='cuda:0', - load_dir=None - ) -lm_model = GLMModel.from_pretrain() -model = GLMForSequenceClassification(lm_model,hidden_size=1024,hidden_dropout=False,pool_token='cls',num_class=2) -tokenizer = GLMBertWordPieceTokenizer(tokenizer_model_type='bert-large-uncased') - -from flagai.data.dataset import ConstructSuperglueStrategy -cl_args = CollateArguments() -cl_args.cloze_eval=False -task_name = "boolq" -if task_name in ['copa', 'wsc', 'record', 'wanke']: - cl_args.multi_token = True -collate_fn = ConstructSuperglueStrategy(cl_args, tokenizer, task_name=task_name) - -train_dataset = SuperGlueDataset(task_name="boolq", data_dir='/mnt/datasets/yan/', dataset_type='train', - tokenizer=tokenizer) -valid_dataset = SuperGlueDataset(task_name="boolq", data_dir='/mnt/datasets/yan/', dataset_type='dev', - tokenizer=tokenizer) - - -trainer.train(model, collate_fn=collate_fn, - train_dataset=train_dataset, valid_dataset=valid_dataset, - eval_metrics=accuracy_metric) -``` - -```shell ->>> python train.py -``` - -Done! The results may be around 80% - -## Finetunine with deepspeed - -we only change some settings in Trainer & run - -```python - trainer = Trainer(env_type='deepspeed', # change env_type - epochs=1, - batch_size=4, - eval_interval=100, - log_interval=50, - experiment_name='glm_large', - load_dir=None - - # parallel settings - master_ip='127.0.0.1', - master_port=17750, - num_nodes=1, - num_gpus=2, - hostfile='hostfile', - training_script=__file__, - # deepspeed - deepspeed_config='deepspeed.json') -``` -```shell ->>> python train.py -``` -hostfile: setting ip for all nodes -```shell -127.0.0.1 slots=2 # slots is the number of your gpus -``` -deepspeed_config: arguments for deepspeed settings -```json -{ - "train_micro_batch_size_per_gpu": 4, - "gradient_accumulation_steps": 1, - "steps_per_print": 50, - "gradient_clipping": 1.0, - "zero_optimization": { - "stage": 1, # important - "contiguous_gradients": false, - "overlap_comm": true, - "reduce_scatter": true, - "reduce_bucket_size": 5e7, - "allgather_bucket_size": 5e7, - "cpu_offload": true # important - }, - "zero_allow_untested_optimizer": true, - "fp16": { - "enabled": true, - "loss_scale": 0, - "loss_scale_window": 1000, - "hysteresis": 2, - "min_loss_scale": 1 - }, - "optimizer": { - "type": "Adam", - "params": { - "lr": 5e-6, - "betas": [ - 0.9, - 0.95 - ], - "eps": 1e-8, - "weight_decay": 1e-2 - } - }, - "activation_checkpointing": { - "partition_activations": true, - "contiguous_memory_optimization": false - }, - "wall_clock_breakdown": false, -} - - -``` diff --git a/docs/Seq2seqMethod.md b/docs/TUTORIAL_9_SEQ2SEQ_METHOD.md similarity index 100% rename from docs/Seq2seqMethod.md rename to docs/TUTORIAL_9_SEQ2SEQ_METHOD.md diff --git a/docs/docs_zh/TUTORIAL_10_BERT_EXAMPLE_TITLE_GENERATION.md b/docs/docs_zh/TUTORIAL_10_BERT_EXAMPLE_TITLE_GENERATION.md deleted file mode 100644 index 8aef76d8..00000000 --- a/docs/docs_zh/TUTORIAL_10_BERT_EXAMPLE_TITLE_GENERATION.md +++ /dev/null @@ -1,112 +0,0 @@ -# BERT 标题生成例子 - -## 背景 -标题生成任务需要输入一段文本，模型根据输入文本输出对应的标题。 - -![](../img/bert_title_generation_model.png) - -## 结果展示 - -#### 输入 -``` -"本文总结了十个可穿戴产品的设计原则而这些原则同样也是笔者认为是这个行业最吸引人的地方1为人们解决重复性问题2从人开始而不是从机器开始3要引起注意但不要刻意4提升用户能力而不是取代人", -"2007年乔布斯向人们展示iPhone并宣称它将会改变世界还有人认为他在夸大其词然而在8年后以iPhone为代表的触屏智能手机已经席卷全球各个角落未来智能手机将会成为真正的个人电脑为人类发展做出更大的贡献", -"雅虎发布2014年第四季度财报并推出了免税方式剥离其持有的阿里巴巴集团15％股权的计划打算将这一价值约400亿美元的宝贵投资分配给股东截止发稿前雅虎股价上涨了大约7％至5145美元" -``` -#### 输出 -``` -可 穿 戴 产 品 设 计 原 则 十 大 原 则 -乔 布 斯 宣 布 iphone 8 年 后 将 成 为 个 人 电 脑 -雅 虎 拟 剥 离 阿 里 巴 巴 15 ％ 股 权 -``` -## Usage - -### 1.Load data -The sample data is in /examples/bert_title_generation/data/ - -You need to define the data loading process in train.py. For example: -```python -def read_file(): - src = [] - tgt = [] - - ## read data file to load src and tgt, for example: - ## src = ["article_1", "article_2", "article_3" ......] - ## tgt = ["title_1", "title_2", "title_3" ......] - ## no matter what data you use, you need to construct the right src and tgt. - with open(src_dir, 'r', encoding='utf-8') as f: - lines = f.readlines() - for line in lines: - src.append(line.strip('\n').lower()) - - with open(tgt_dir, 'r', encoding='utf-8') as f: - lines = f.readlines() - for line in lines: - tgt.append(line.strip('\n').lower()) - return src,tgt -``` - -### 2.Load model and tokenizer - -```python -from flash_tran.auto_model.auto_loader import AutoLoader - -# the model dir, which contains the 1.config.json, 2.pytorch_model.bin, 3.vocab.txt, -# or we will download these files from the model hub to this dir. -# Autoloader can build the model and tokenizer automatically. -# 'seq2seq' is the task_name. -auto_loader = AutoLoader("seq2seq", - model_dir="./state_dict/", - model_name="RoBERTa-base-ch") -model = auto_loader.get_model() -tokenizer = auto_loader.get_tokenizer() -``` - -### 3. Train -Then input this code in commandline to train: -```commandline -python ./train.py -``` -Modify the training configuration by this code: -```python -from flash_tran.trainer import Trainer -import torch -device = torch.device("cuda" if torch.cuda.is_available() else "cpu") -trainer = Trainer(env_type="pytorch", - experiment_name="roberta_seq2seq", - batch_size=8, gradient_accumulation_steps=1, - lr = 2e-4, - weight_decay=1e-3, - epochs=10, log_interval=10, eval_interval=10000, - load_dir=None, pytorch_device=device, - save_dir="checkpoints", - save_epoch=1 - ) -``` -Divide the training set validation set and create the dataset: -```python -sents_src, sents_tgt = read_file() -data_len = len(sents_tgt) -train_size = int(data_len * 0.8) -train_src = sents_src[: train_size] -train_tgt = sents_tgt[: train_size] - -val_src = sents_src[train_size: ] -val_tgt = sents_tgt[train_size: ] - -train_dataset = BertSeq2seqDataset(train_src, train_tgt, tokenizer=tokenizer, maxlen=maxlen) -val_dataset = BertSeq2seqDataset(val_src, val_tgt, tokenizer=tokenizer, maxlen=maxlen) -``` - -### Generation -If you have already trained a model, in order to see the results more intuitively, rather than the loss of the validation set. -You can run the generation file. -First to modify the path of saved model. -```python -model_save_path = "./checkpoints/1001/mp_rank_00_model_states.pt" ## 1001 is example, you need modify the number. -``` -```commandline -python ./generate.py -``` -Then you can see the generation result. - diff --git a/docs/docs_zh/TUTORIAL_11_BERT_EXAMPLE_SEMANTIC_MATCHING.md b/docs/docs_zh/TUTORIAL_11_BERT_EXAMPLE_SEMANTIC_MATCHING.md deleted file mode 100644 index 9e6acdfa..00000000 --- a/docs/docs_zh/TUTORIAL_11_BERT_EXAMPLE_SEMANTIC_MATCHING.md +++ /dev/null @@ -1,112 +0,0 @@ -# BERT 语义匹配例子 - -## 背景 -语义匹配任务是判断输入的两个句子意思是否一致，需要输入句子对，输出2分类（两句话意思相同或不同）的结果。 - -![semantic_model.png](../img/semantic_matching_model.png) - -## 结果展示 - -#### 输入 -```python -test_data = [["后悔了吗","你有没有后悔"], - ["打开自动横屏","开启移动数据"], - ["我觉得你很聪明","你聪明我是这么觉得"]] -``` -#### 输出 -``` -1 -0 -1 -``` -1表示两个句子意思相同，0表示不相同。 -## 使用 - -### 1.数据加载 -样例数据在 /examples/bert_semantic_matching/data/ - -需要针对数据格式定义数据加载方法，例如： -```python -def read_file(data_path): - src = [] - tgt = [] - ##TODO read data file to load src and tgt, for example: - ## src = [["article_1_1", "article_1_2"], ["article_2_1", "artile_2_2"], ......] - ## tgt = [1, 0, ......] - ## no matter what data you use, you need to construct the right src and tgt. - with open(data_path) as f: - lines = f.readlines() - for line in lines: - line = line.split("\t") - if len(line) == 3: - sents_tgt.append(int(line[2])) - sents_src.append([line[0], line[1]]) - - return src,tgt -``` - -### 2.模型与切词器加载 - -```python -from flash_tran.auto_model.auto_loader import AutoLoader - -# the model dir, which contains the 1.config.json, 2.pytorch_model.bin, 3.vocab.txt, -# or we will download these files from the model hub to this dir. -model_dir = "./state_dict/" -# Autoloader can build the model and tokenizer automatically. -# 'classification' is the task_name. -auto_loader = AutoLoader("classification", - model_dir, - model_name="RoBERTa-base-ch") -model = auto_loader.get_model() -tokenizer = auto_loader.get_tokenizer() -``` - -### 3. 训练 -在命令行中输入： -```commandline -python ./train.py -``` -调整训练参数： -```python -from flash_tran.trainer import Trainer -import torch -device = torch.device("cuda" if torch.cuda.is_available() else "cpu") -trainer = Trainer(env_type="pytorch", - experiment_name="roberta-base-ch-semantic-matching", - batch_size=8, gradient_accumulation_steps=1, - lr = 1e-5, - weight_decay=1e-3, - epochs=10, log_interval=100, eval_interval=500, - load_dir=None, pytorch_device=device, - save_dir="checkpoints_semantic_matching", - save_epoch=1 - ) -``` -切分训练集验证集 -```python -src, tgt = read_file(data_path=train_path) -data_len = len(src) -train_size = int(data_len * 0.9) -train_src = src[: train_size] -train_tgt = tgt[: train_size] - -val_src = src[train_size: ] -val_tgt = tgt[train_size: ] - -train_dataset = BertClsDataset(train_src, train_tgt) -val_dataset = BertClsDataset(val_src, val_tgt) -``` - -### 生成 -如果你已经训练好了一个模型，为了更加直观的看到结果，可以进行测试生成 - -首先调整训练好的模型的路径： -```python -model_save_path = "./checkpoints_semantic_matching/9000/mp_rank_00_model_states.pt" -``` -运行： -```commandline -python ./generate.py -``` -然后可以查看运行结果。 diff --git a/docs/docs_zh/TUTORIAL_14_BERT_EXAMPLE_NER.md b/docs/docs_zh/TUTORIAL_14_BERT_EXAMPLE_NER.md deleted file mode 100644 index 62c99f2a..00000000 --- a/docs/docs_zh/TUTORIAL_14_BERT_EXAMPLE_NER.md +++ /dev/null @@ -1,111 +0,0 @@ -# BERT 命名实体识别例子 - -## 背景 -命名实体识别任务是判断输入的句子里面是否存在不同实体，并进行识别标记。 - -BERT模型共支持三种方式完成NER任务： -1. 序列标注方法 -2. 序列标注+CRF方法 -3. GlobalPointer方法 - -本文以方法1为例。 - -![semantic_model.png](../img/bert_ner_model.png) - -## 结果展示 - -#### 输入 -```python -test_data = [ - "6月15日，河南省文物考古研究所曹操高陵文物队公开发表声明承认：“从来没有说过出土的珠子是墓主人的", - "4月8日，北京冬奥会、冬残奥会总结表彰大会在人民大会堂隆重举行。习近平总书记出席大会并发表重要讲话。在讲话中，总书记充分肯定了北京冬奥会、冬残奥会取得的优异成绩，全面回顾了7年筹办备赛的不凡历程，深入总结了筹备举办北京冬奥会、冬残奥会的宝贵经验，深刻阐释了北京冬奥精神，对运用好冬奥遗产推动高质量发展提出明确要求。", - "当地时间8日，欧盟委员会表示，欧盟各成员国政府现已冻结共计约300亿欧元与俄罗斯寡头及其他被制裁的俄方人员有关的资产。", - "这一盘口状态下英国必发公司亚洲盘交易数据显示博洛尼亚热。而从欧赔投注看，也是主队热。巴勒莫两连败，", -] -``` -#### 输出 -``` -{'ORG': ['河南省文物考古研究所', '曹操高陵文物队']} -{'LOC': ['北京', '人民大会堂', '北京', '北京', '北京'], 'PER': ['习近平']} -{'ORG': ['欧盟委员会', '欧盟'], 'LOC': ['俄罗斯', '俄']} -{'ORG': ['英国必发公司', '博洛尼亚', '巴勒莫']} -``` -## 使用 - -### 1.数据加载 -样例数据在 /examples/bert_ner/data/ - -需要针对数据格式定义数据加载方法，例如： -```python -def load_data(filename): - """ load data - data:：[text, (start, end, label), (start, end, label), ...] - """ - D = [] - with open(filename, encoding='utf-8') as f: - f = f.read() - for l in f.split('\n\n'): - if not l: - continue - d = [''] - for i, c in enumerate(l.split('\n')): - char, flag = c.split(' ') - d[0] += char - if flag[0] == 'B': - d.append([i, i, flag[2:]]) - elif flag[0] == 'I': - d[-1][1] = i - - D.append(d) - return D -``` - -### 2.模型与切词器加载 - -```python -from flash_tran.auto_model.auto_loader import AutoLoader -task_name = "sequence-labeling" -model_dir = "./state_dict/" -target = ["O", "B-LOC", "I-LOC", "B-ORG", "I-ORG", "B-PER", "I-PER"] -auto_loader = AutoLoader(task_name, - model_name="RoBERTa-base-ch", - model_dir=model_dir, - class_num=len(target)) -model = auto_loader.get_model() -tokenizer = auto_loader.get_tokenizer() -``` - -### 3. 训练 -在命令行中输入： -```commandline -python ./train.py -``` -调整训练参数： -```python -from flash_tran.trainer import Trainer -import torch -device = torch.device("cuda" if torch.cuda.is_available() else "cpu") -trainer = Trainer(env_type="pytorch", - experiment_name="roberta_ner", - batch_size=8, gradient_accumulation_steps=1, - lr = 1e-5, - weight_decay=1e-3, - epochs=10, log_interval=100, eval_interval=500, - load_dir=None, pytorch_device=device, - save_dir="checkpoints_ner", - save_epoch=1 - ) -``` - -### 生成 -如果你已经训练好了一个模型，为了更加直观的看到结果，可以进行测试生成 - -首先调整训练好的模型的路径： -```python -model_save_path = "./checkpoints_ner/9000/mp_rank_00_model_states.pt" -``` -运行： -```commandline -python ./generate.py -``` -然后可以查看运行结果。 diff --git a/docs/docs_zh/TUTORIAL_15_GPT2_WRITING.md b/docs/docs_zh/TUTORIAL_15_GPT2_WRITING.md deleted file mode 100644 index 8d628e35..00000000 --- a/docs/docs_zh/TUTORIAL_15_GPT2_WRITING.md +++ /dev/null @@ -1,41 +0,0 @@ -# GPT2 模型生成任务 - -## 背景 -GPT2生成任务是只输入一段文字的开头，模型进行续写 - - - -![gpt2.png](../img/gpt2_writing_model.png) - -## 结果展示 - -#### 输入 -```python -text = "今天天气不错，" -``` -#### 输出 -``` -我 们 三 个 女 孩 子 去 吃 的 ， 因 为 是 中 午 ， 所 以 没 有 等 位 。 店 里 面 环 境 还 不 错 ， 比 较 安 静 ， 我 们 要 的 鸳 鸯 锅 底 ， 味 道 一 般 ， 辣 的 感 觉 有 点 不 太 习 惯 。 服 务 态 度 很 好 ， 可 以 刷 卡 。 - -``` -## 使用 - -### 1.模型与切词器加载 - -```python -from flash_tran.auto_model.auto_loader import AutoLoader -from flash_tran.model.predictor.predictor import Predictor -loader = AutoLoader("seq2seq", - "gpt2_base_chinese", - model_dir="./state_dict/") -model = loader.get_model() -tokenizer = loader.get_tokenizer() -predictor = Predictor(model, tokenizer) -``` - -### 运行 - -```commandline -python ./generate.py -``` -然后可以查看运行结果。 diff --git a/docs/docs_zh/TUTORIAL_16_T5_EXAMPLE_TITLE_GENERATION.md b/docs/docs_zh/TUTORIAL_16_T5_EXAMPLE_TITLE_GENERATION.md deleted file mode 100644 index fa15992b..00000000 --- a/docs/docs_zh/TUTORIAL_16_T5_EXAMPLE_TITLE_GENERATION.md +++ /dev/null @@ -1,121 +0,0 @@ -# T5 标题生成 - -## 背景 -T5标题生成任务，输入一段文本，T5模型提取对应的标题。 - - -![](../img/t5_seq2seq.png) - -## Results show - -#### Input text -``` -"本文总结了十个可穿戴产品的设计原则而这些原则同样也是笔者认为是这个行业最吸引人的地方1为人们解决重复性问题2从人开始而不是从机器开始3要引起注意但不要刻意4提升用户能力而不是取代人", -"2007年乔布斯向人们展示iPhone并宣称它将会改变世界还有人认为他在夸大其词然而在8年后以iPhone为代表的触屏智能手机已经席卷全球各个角落未来智能手机将会成为真正的个人电脑为人类发展做出更大的贡献", -"雅虎发布2014年第四季度财报并推出了免税方式剥离其持有的阿里巴巴集团15％股权的计划打算将这一价值约400亿美元的宝贵投资分配给股东截止发稿前雅虎股价上涨了大约7％至5145美元" -``` -#### Out title -``` -可 穿 戴 产 品 设 计 原 则 十 大 原 则 -乔 布 斯 宣 布 iphone 8 年 后 将 成 为 个 人 电 脑 -雅 虎 拟 剥 离 阿 里 巴 巴 15 ％ 股 权 -``` -## Usage - -### 1.Load data -The sample data is in /examples/bert_title_generation/data/ - -You need to define the data loading process in train.py. For example: -```python -def read_file(): - src = [] - tgt = [] - - ## read data file to load src and tgt, for example: - ## src = ["article_1", "article_2", "article_3" ......] - ## tgt = ["title_1", "title_2", "title_3" ......] - ## no matter what data you use, you need to construct the right src and tgt. - with open(src_dir, 'r', encoding='utf-8') as f: - lines = f.readlines() - for line in lines: - src.append(line.strip('\n').lower()) - - with open(tgt_dir, 'r', encoding='utf-8') as f: - lines = f.readlines() - for line in lines: - tgt.append(line.strip('\n').lower()) - return src,tgt -``` - -### 2.Load model and tokenizer - -```python -from flash_tran.auto_model.auto_loader import AutoLoader - -loader = AutoLoader("seq2seq", "t5_base_ch", model_dir="./state_dict/") -model = loader.get_model() -tokenizer = loader.get_tokenizer() -``` - -### 3. Train -Then input this code in commandline to train: -```commandline -python ./train.py -``` -The configuration support multi-gpus training. -Modify the training configuration by this code: -```python -from flash_tran.trainer import Trainer -import torch -device = torch.device("cuda" if torch.cuda.is_available() else "cpu") -trainer = Trainer( - env_type="deepspeed+mpu", - experiment_name="roberta_seq2seq", - batch_size=1, - gradient_accumulation_steps=1, - lr=2e-4, - weight_decay=1e-3, - epochs=10, - log_interval=10, - eval_interval=10000, - load_dir=None, - pytorch_device=device, - save_dir="checkpoints", - save_epoch=1, - num_checkpoints=1, - master_ip='127.0.0.1', - master_port=17750, - num_nodes=1, - num_gpus=2, - hostfile='/data/liuguang/test_Sailing/Sailing/examples/bert_title_generation/hostfile', - deepspeed_config='/data/liuguang/test_Sailing/Sailing/examples/bert_title_generation/deepspeed.json', - training_script=__file__, -) -``` -Divide the training set validation set and create the dataset: -```python -sents_src, sents_tgt = read_file() -data_len = len(sents_tgt) -train_size = int(data_len * 0.8) -train_src = sents_src[: train_size] -train_tgt = sents_tgt[: train_size] - -val_src = sents_src[train_size: ] -val_tgt = sents_tgt[train_size: ] - -train_dataset = BertSeq2seqDataset(train_src, train_tgt, tokenizer=tokenizer, maxlen=maxlen) -val_dataset = BertSeq2seqDataset(val_src, val_tgt, tokenizer=tokenizer, maxlen=maxlen) -``` - -### Generation -If you have already trained a model, in order to see the results more intuitively, rather than the loss of the validation set. -You can run the generation file. -First to modify the path of saved model. -```python -model_save_path = "./checkpoints/1001/mp_rank_00_model_states.pt" ## 1001 is example, you need modify the number. -``` -```commandline -python ./generate.py -``` -Then you can see the generation result. - diff --git a/docs/img.png b/docs/img.png new file mode 100644 index 00000000..9c25dd27 Binary files /dev/null and b/docs/img.png differ diff --git a/docs/img/TUTORIAL_2_DATASET.md b/docs/img/TUTORIAL_2_DATASET.md new file mode 100644 index 00000000..a51a0dcd --- /dev/null +++ b/docs/img/TUTORIAL_2_DATASET.md @@ -0,0 +1,360 @@ +# 数据集处理流程 +构建数据集的过程就是NLP的数据预处理过程，其主要目的是将原始的散乱的文件数据重新整理成统一结构的数据，以便语言模型能够直接使用。构建数据集样例的主要流程如下所示（以CommitmentBank数据集为例）： + +
+ +目前项目里存在三种数据预处理的情况，即对分类任务的微调，预训练, 以及对生成任务的微调。我们在接下来会分别展开。 + +## 数据处理：分类任务微调([prompt-learning模式](TUTORIAL_7_PROMPT_LEARNING.md)) +### 应用代码 +```python +import torch.utils.data +from flagai.data.dataset import SuperGlueDataset +from flagai.data.tokenizer import GLMLargeEnWordPieceTokenizer +from tests.test_dataset_new_superglue import CollateArguments +from flagai.data.dataset import ConstructSuperglueStrategy + +# 得到默认参数 +cl_args = CollateArguments() + +# 创建分词器 +tokenizer = GLMLargeEnWordPieceTokenizer() + +# 初步读取并处理数据集 +dataset = SuperGlueDataset(task_name='cb', + data_dir='./datasets/', + dataset_type='train', + tokenizer=tokenizer) + +# 构建collate function +collate_fn = ConstructSuperglueStrategy(cl_args, tokenizer, task_name="rte") + +# 创建加载器 +loader = torch.utils.data.DataLoader(dataset, + batch_size=1, + shuffle=False, + num_workers=1, + drop_last=False, + pin_memory=False, + collate_fn=collate_fn) +``` + +### 初步读取并处理数据集 +对应的代码模块如下所示，其包含了两个步骤：自动加载数据集，以及统一所有数据集的结构 +```python +dataset = SuperGlueDataset(task_name='cb', + data_dir='./datasets/', + dataset_type='train', + tokenizer=tokenizer) +``` +#### 1.加载数据集 + +将`task_name`设置成任务名的缩写后，相关数据会在后台自动下载。FlagAI目前支持自动加载下列分类数据集： + + +| 数据集名称 | 数据集简称 | 语言 | 所属评测基准 | +|----------------------------------------------|----------|------|----------| +| Broadcoverage Diagnostics | boolq | 英文 | SuperGLUE | +| CommitmentBank | cb | 英文 | SuperGLUE | +| Choice of Plausible Alternatives | copa | 英文 | SuperGLUE | +| Multi-Sentence Reading Comprehension | muiltirc | 英文 | SuperGLUE | +| Recognizing Textual Entailment | rte | 英文 | SuperGLUE | +| Words in Context | wic | 英文 | SuperGLUE | +| The Winograd Schema Challenge | wsc | 英文 | SuperGLUE | +| Ant Financial Question Matching Corpus | afqmc | 中文 | CLUE | +| Short Text Classificaiton for News | tnews | 中文 | CLUE | + +数据集会被自动下载到`data_dir`对应的地址，默认为项目的`./dataset`目录。 + +下载好的数据集目录下会包含三个文件，对应训练集数据，验证集数据，以及测试集数据, 以CommitmentBank数据集为例，目录下的`train.jsonl`对应训练集，`val.jsonl`对应验证集，`test.jsonl`对应测试集。一般来说训练集和测试集会包含标签信息，而测试集则没有。这些数据文件在接下来的流程中会分开处理。 + +不同的数据集可能会有不同的文件格式，以及不同的结构。以CommitmentBank数据集为例，下面是其中的一个样例 + +
+ +可以看到，其包含了四个部分,如下所示： + +| 键值 | 含义 | 值 | +|-----------|-------------------------------------------------------|--------------------------------| +| premise | 前提文本 | Mary is a high school student. | +| hypothesis | 假设文本 | Mary is a student | +| label | 代表前提和假设关系的标签
包含entailment,neutral和contradiction三种 | entailment | +| idx | 样例在数据集里的序号 | 10 | + +目前所有FlagAI支持数据集的具体结构可以在[这里](DATASET_EXAMPLE.md)查看。 + +#### 2. 统一数据集结构 +在这一步里，我们会统一不同数据集的数据结构，以方便接下来处理。此结构的细节如下： + + +| 键值 | 含义 | 数据格式 | +|--------|-------------------------------------------------------|------| +| guid | a unique textual identifier | str | +| text_a | the sequence of text | str | +| text_b | an optional, second sequence of text | str | +| label | an optional label | str | +| logits | an optional list of per-class logits | list | +| meta | an optional dictionary to store arbitrary meta information | dict | +| ids | an optional numeric index | int | + +例如上一步CommitBank的样例会被处理成如下的形式 + +
+ +需要注意的是如果因为数据结构太复杂，导致text_a和text_b无法塞下背景文本信息的话，可以把剩下的信息放在meta里。 + +当数据集被构造好以后，可以直接在代码里通过索引的方式查看其中某个样例： + +```python +example = dataset[3] # 数据集里第3个样例 +``` + +### 将数据整理成模型的输入 + +对应的功能在下面的函数里实现，其包含了两个步骤：构造模板，分词并构造输入样例。 +```python +collate_fn = ConstructSuperglueStrategy(cl_args, + tokenizer, + task_name=task_name) +``` + +#### 1.构建完形填空模板 + +一个完形填空模板包含了背景文本，空位，以及在提供给空位的选项。模型需要找到正确的选项，并填进空位里。 + +对于每个不同的任务，我们都需要构建不同构造的完型填空问题来让模型回答，以CommitmentBank数据集为例， 其考量的是能否由前提推导出假设, 而有且仅有三种结果：contradiction/neutral/entailment。那么我们可以构建如下的完型填空问题， 其中contradiction/neutral/entailment分别对应true/false/neither。 + + + +
+ +可以看到，大体上可以分成两步：第一步是组合已有的文本，使其看上去符合完形填空格式；第二步是将原始的标签文本转化为新的标签，作为可能会填入空位的选项。 + +#### 2.分词并构造输入样例 +接下来，我们需要构造模型的输入，第一步是分词，而接下来则需要分成两种情况： +第一种情况下，数据集包含的标签类别是有限的，比如CommitmentBank数据集里只会存在entailment/contradiction/neutral三种标签文本，，常见于分类任务。第二类情况里每一段完型填空都会给出不同的选项(一般是一段长文本)。比如在一些阅读理解数据集里，每一个选项都是一段对于文本的不同理解。这两类情况的处理方法如下所示： + +**a)单个token的完形填空** + + +| 键值 | 维度 | 含义 | 构造方法 | +|---------------------------------------------|--------------------------------------|------------|-----------------------------------------------| +| input_ids | torch.Size([seq_length1]) | 输入矩阵 | 由上一步的完形填空文本，加上一些特殊字符2组成 | +| labels | labels: torch.Size([1]) | 标签 | 对应的数字标签，比如0,1,2... | +| position_ids | torch.Size([2， seq_length]) | 位置编码 | 参考[GLM流程](GLM.md)，第一行代表token的绝对位置，第二行代表遮挡部分的相对位置 | +| attention_mask | torch.Size([1]) | 分隔符位置 | | +| target_ids | torch.Size([num_labels3]) | 全量标签列表 | 将所有标签文本分别对应单token的标签，然后将这些标签的序号依次放入target_ids | +| logit_mask | torch.Size([seq_length]) | 对应的文本是否为回答 | 对于每个token, 如果是回答，则对应的地方为1，否则为0 | + + +1: seq_length代表规定的每个输入向量的最大长度 + +2: 特殊字符添加流程：在句首添加[CLS]符号，句尾添加[EOS]符号，直到长度达到seq_length, 如果完形填空输出的文本有两段，则在中间添加[SEP]符号 +
+ +3: num_labels代表完形填空问题中选项的个数 + +**b)多个token的完形填空** + +| 键值 | 维度 | 含义 | 区别 | +|---------------------------------------------|-----------------------------------------|------------|----------------------------------------------------| +| input_ids | torch.Size([num_labels, seq_length]) | 输入矩阵 | 将对应的文本拷贝num_labels份 | +| labels | labels: torch.Size([1]) | 标签 | | +| position_ids | torch.Size([num_labels, 2， seq_length]) | 位置编码 | 将原本的位置编码拷贝num_labels份 | +| attention_mask | torch.Size([num_labels]) | 分隔符位置 | 拷贝num_labels份 | +| target_ids | torch.Size([num_labels, seq_length]) | 每个选项信息 | 矩阵的每一列代表每一个选项，每一行代表当前选项对应的文本 | +| logit_mask | torch.Size([num_labels, seq_length]) | 对应的文本是否为回答 | 拷贝num_labels份 | + + + +### 创建加载器 + +最后将数据放入[PyTorch加载器](https://pytorch.org/docs/stable/data.html?highlight=dataloader#torch.utils.data.DataLoader)即可。 +```python +loader = torch.utils.data.DataLoader(dataset, + batch_size=1, + shuffle=False, + num_workers=1, + drop_last=False, + pin_memory=False, + collate_fn=collate_fn) +``` +Dataloader里的数据可以通过如下方法查看 + +```python +for data_iterator in train_loader: + for key, value in data_iterator.items(): + print(key, value) + # break +``` +加载器构造好之后即可用于接下来的训练和预测过程。


+ + +## 数据处理：GLM预训练任务 +预训练任务数据格式样例： +```text +{ + "RECORDS": [ + { + "id": 0, + "uniqueKey": "", + "titleUkey": "", + "dataType": "", + "title": "", + "content": "平安信用卡上门激活失败是为啥？平安信用卡上门激 + 活失败的主要原因有这些：申卡人是在网上申请的新用户，上门激活时 + 携带资料不足，需要申卡人携带身份证、信用卡、预留号码等去平安银 + 行网点验证身份才能激活；申卡人在平安银行预留的手机号码有误；申卡 + 人在激活时输入的相关信息不正确；申卡人在回答上门激活工作人员的问 + 题是，跟之前提交的资料有一些出入，有违规嫌疑。平安信用卡上门激活失 + 败的主要原因有这些：申卡人是在网上申请的新用户，上门激活时携带资料不 + 足，需要申卡人携带身份证、信用卡、预留号码等去平安银行网点验证身份才能 + 激活；申卡人在平安银行预留的手机号码有误；申卡人在激活时输入的相关信息不 + 正确；申卡人在回答上门激活工作人员的问题是，跟之前提交的资料有一 + 些出入，有违规嫌疑。" + }, + ] +} +``` +预训练的任务处理实例代码: +```python +tokenizer = GLMLargeChTokenizer(add_block_symbols=True, + add_task_mask=True, + add_decoder_mask=False, + fix_command_token=True) + +ds_args = DatasetArguments() + +tokenizer = GLMLargeChTokenizer(fix_command_token=True, + add_block_symbols=True, + add_task_mask=True, + add_decoder_mask=False) + +ds_args = add_args(ds_args, tokenizer) + +def create_dataset(tokenizer, should_split): + dataset = get_dataset_lazy("./examples/glm_pretrain/data", # 懒加载 + tokenizer=tokenizer, + pre_tokenize=True, + num_processes=10, + no_lazy_loader=True) + if should_split: + datasets = split_ds(dataset, split=[.8, .2, .0], shuffle=True) # 手动切分 + else: + datasets = [dataset] + + datasets = [ + BlockDataset(ds, + tokenizer, + max_seq_len=512, + sample_across_doc=True, + non_sentence_start=0.0) if ds is not None else None + for ds in datasets + ] + return datasets + +datasets = create_dataset(tokenizer, should_split=True) +``` +预训练的数据处理也遵循着同样的流程，不过有如下的区别 +1. 预训练数据集默认是没有切分成训练集，验证集，以及测试集的，所以还需要手动进行切分 +2. 由于预训练数据集一般来说较为庞大，所以使用了懒加载(lazy loading)，懒加载只在真正用到对象的时候才使其实例化，是一种较为节省资源的操作。 +3. 预训练的时候，collate function会随机按照三种不同的模式处理数据: bert模式（遮挡随机区间），sentence模式(按照完整句子进行遮挡)以及gpt模式（只遮挡一段长区间）。模型的输入相比一般生成任务也会多一个`mode`键。 +4. 预训练不用添加模板，直接按下表构建模型输入即可 + +| 键值 | 维度 | 含义 | 构造方法 | +|----------------|--------------------------------------|------------|--------------------------------------------------| +| input_ids | torch.Size([seq_length1]) | 输入矩阵 | 由上一步的模板文本，加上一些特殊字符组成 | +| position_ids | torch.Size([2， seq_length]) | 位置编码 | 参考[GLM流程](GLM.md)，第一行代表token的绝对位置，第二行代表遮挡部分的相对位置 | +| attention_mask | torch.Size([1]) | 分隔符位置 | 对于生成类的模式，得到源文本结束的位置；否则得到输入文本结束的位置 | +| target_ids | torch.Size([num_labels3]) | 全量标签列表 | 被遮挡住的文本 | +| logit_mask | torch.Size([seq_length]) | 通过遮挡使得模型只会处理目标文本部分的loss | 对于每个token, 如果是回答，则对应的地方为1，否则为0 | +| mode | str | 数据处理模式 | | +


+ +## 数据处理：生成任务微调 +代码实现如下所示： +```python +import torch.utils.data +from flagai.data.dataset import Seq2SeqDataset +from flagai.data.tokenizer import GLMLargeEnWordPieceTokenizer +from tests.test_dataset_new_superglue import Seq2SeqCollateArguments +from flagai.data.dataset import ConstructSeq2seqStrategy + +# 得到默认参数 +cl_args = Seq2SeqCollateArguments() + +# 创建分词器 +tokenizer = GLMLargeChTokenizer(add_block_symbols=True, + TUTORIAL_4_DATASET.md add_task_mask=False, + add_decoder_mask=False, + fix_command_token=False) + +# 初步读取并处理数据集 +dataset = Seq2SeqDataset(task_name='cmrc', + data_dir='./datasets/', + dataset_type='train', + tokenizer=tokenizer) + +# 构建collate function +collate_fn = ConstructSeq2seqStrategy(cl_args, tokenizer, task_name="rte") + +# 创建加载器 +loader = torch.utils.data.DataLoader(dataset, + batch_size=1, + shuffle=False, + num_workers=1, + drop_last=False, + pin_memory=False, + collate_fn=collate_fn) +``` + +### 初步读取并处理数据集 + +目前支持[CMRC2018](https://www.clue.ai/introduce.html)任务，CMRC是一个阅读理解类的任务，需要根据背景文本来回答一系列提问，其数据结构示例如下： + +```text +{'paragraphs': + [{'id': 'TRAIN_186', + 'context': '范廷颂枢机（，），圣名保禄·若瑟（），是越南罗马天主教枢机。1963年被任 + 为主教；1990年被擢升为天主教河内总教区宗座署理；1994年被擢升为总主教，同年年底被擢 + 升为枢机；2009年2月离世。范廷颂于1919年6月15日在越南宁平省天主教发艳教区出生；童年 + 时接受良好教育后，被一位越南神父带到河内继续其学业。范廷颂于1940年在河内大修道院完 + 成神学学业。范廷颂于1949年6月6日在河内的主教座堂晋铎；及后被派到圣女小德兰孤儿院服 + 务。1950年代，范廷颂在河内堂区创建移民接待中心以收容到河内避战的难民。1954年，法越 + 战争结束，越南民主共和国建都河内，当时很多天主教神职人员逃至越南的南方，但范廷颂仍然 + 留在河内。翌年管理圣若望小修院；惟在1960年因捍卫修院的自由、自治及拒绝政府在修院设 + 政治课的要求而被捕。1963年4月5日，教宗任命范廷颂为天主教北宁教区主教，同年8月15日就 + 任；其牧铭为「我信天主的爱」。由于范廷颂被越南政府软禁差不多30年，因此他无法到所属堂 + 区进行牧灵工作而专注研读等工作。范廷颂除了面对战争、贫困、被当局迫害天主教会等问题外 + ，也秘密恢复修院、创建女修会团体等。1990年，教宗若望保禄二世在同年6月18日擢升范廷颂 + 为天主教河内总教区宗座署理以填补该教区总主教的空缺。1994年3月23日，范廷颂被教宗若望 + 保禄二...', + 'qas':{'question': '范廷颂是何时去世的？', 'id': 'TRAIN_186_QUERY_4', + 'answers': [{'text': '范廷颂于2009年2月22日清晨在河内离世', 'answer_start': 759}]}]}], + 'id': 'TRAIN_186', 'title': '范廷颂'} +``` +使用的时候，我们将`task_name`参数改为`cmrc`即可。实现的过程类似分类任务的微调，最终也会将数据集初步处理成一样的结构，对应代码如下： + +```python +dataset = Seq2SeqDataset(task_name='cmrc', data_dir='./datasets/', + dataset_type='train', tokenizer=tokenizer) +``` + +### 将数据整理成模型的输入 + +代码如下所示。与生成任务相比，同样是构造模板以及模型输入，区别在于构造的方式不一样 +```python +collate_fn = ConstructSeq2seqStrategy(cl_args, + tokenizer, + task_name=task_name) +``` + +#### 1.构建填空模板 +由于是阅读理解任务，所以在模板里需要体现出是针对指定的阅读理解问题进行回答的，参考如下构建方式 +
+ + + +#### 2.分词并构造输入样例 +与预训练类似，区别在于没有mode这个键值。 + diff --git a/docs/img/autoloader_en_map.png b/docs/img/autoloader_en_map.png new file mode 100644 index 00000000..2c39c403 Binary files /dev/null and b/docs/img/autoloader_en_map.png differ diff --git a/docs/img/autoloader_map.png b/docs/img/autoloader_map.png new file mode 100644 index 00000000..91b495da Binary files /dev/null and b/docs/img/autoloader_map.png differ diff --git a/docs/img/dataset_figure_0.png b/docs/img/dataset_figure_0.png new file mode 100644 index 00000000..e53e414d Binary files /dev/null and b/docs/img/dataset_figure_0.png differ diff --git a/docs/img/dataset_figure_2.png b/docs/img/dataset_figure_2.png new file mode 100644 index 00000000..eb324904 Binary files /dev/null and b/docs/img/dataset_figure_2.png differ diff --git a/docs/img/dataset_figure_3.png b/docs/img/dataset_figure_3.png new file mode 100644 index 00000000..be869428 Binary files /dev/null and b/docs/img/dataset_figure_3.png differ diff --git a/docs/img/dataset_figure_4.png b/docs/img/dataset_figure_4.png new file mode 100644 index 00000000..c6942d29 Binary files /dev/null and b/docs/img/dataset_figure_4.png differ diff --git a/docs/img/dataset_figure_5.png b/docs/img/dataset_figure_5.png new file mode 100644 index 00000000..894cc3a6 Binary files /dev/null and b/docs/img/dataset_figure_5.png differ diff --git a/docs/img/dataset_pipeline.png b/docs/img/dataset_pipeline.png new file mode 100644 index 00000000..0d3513f6 Binary files /dev/null and b/docs/img/dataset_pipeline.png differ diff --git a/docs/img/glm_blank_filling.png b/docs/img/glm_blank_filling.png new file mode 100644 index 00000000..59477761 Binary files /dev/null and b/docs/img/glm_blank_filling.png differ diff --git a/docs/img/glm_example_1.png b/docs/img/glm_example_1.png new file mode 100644 index 00000000..7e8c4adf Binary files /dev/null and b/docs/img/glm_example_1.png differ diff --git a/docs/img/glm_example_2.png b/docs/img/glm_example_2.png new file mode 100644 index 00000000..0a66052d Binary files /dev/null and b/docs/img/glm_example_2.png differ diff --git a/docs/img/glm_example_3.png b/docs/img/glm_example_3.png new file mode 100644 index 00000000..92edb4ee Binary files /dev/null and b/docs/img/glm_example_3.png differ diff --git a/docs/img/glm_example_4.png b/docs/img/glm_example_4.png new file mode 100644 index 00000000..4256293e Binary files /dev/null and b/docs/img/glm_example_4.png differ diff --git a/docs/img/glm_example_5.png b/docs/img/glm_example_5.png new file mode 100644 index 00000000..ef1612c1 Binary files /dev/null and b/docs/img/glm_example_5.png differ diff --git a/docs/img/glm_performance.png b/docs/img/glm_performance.png new file mode 100644 index 00000000..054894f9 Binary files /dev/null and b/docs/img/glm_performance.png differ diff --git a/docs/img/glm_results2.png b/docs/img/glm_results2.png new file mode 100644 index 00000000..b780699c Binary files /dev/null and b/docs/img/glm_results2.png differ diff --git a/docs/img/predictor_map.png b/docs/img/predictor_map.png new file mode 100644 index 00000000..d112c25a Binary files /dev/null and b/docs/img/predictor_map.png differ diff --git a/docs/img/prompt_figure_1.png b/docs/img/prompt_figure_1.png new file mode 100644 index 00000000..28dbd5fd Binary files /dev/null and b/docs/img/prompt_figure_1.png differ diff --git a/docs/img/prompt_figure_2.png b/docs/img/prompt_figure_2.png new file mode 100644 index 00000000..b8cf0ea9 Binary files /dev/null and b/docs/img/prompt_figure_2.png differ diff --git a/docs/img/tokenize.png b/docs/img/tokenize.png new file mode 100644 index 00000000..71888a1a Binary files /dev/null and b/docs/img/tokenize.png differ diff --git a/docs/img/tokenizer_example_1.png b/docs/img/tokenizer_example_1.png new file mode 100644 index 00000000..a84bb8bc Binary files /dev/null and b/docs/img/tokenizer_example_1.png differ diff --git a/docs/tokenization.md b/docs/tokenization.md new file mode 100644 index 00000000..f22461f7 --- /dev/null +++ b/docs/tokenization.md @@ -0,0 +1,246 @@ +[TOC] + +# sub-word algorithm + +## Tokenization + +​ 在自然语言处理中，大部分的数据都是生语料（文本），但是**机器是无法理解文本的**，我们将句子序列送入模型时，模型仅仅只能看到一串字节，它无法知道一个词从哪里开始，到哪里结束，所以也不知道一个词是由哪些字节组成的。 + +​ Tokenizer 的作用就是将文本拆分为单词或者子词（sub-word），然后通过词表将它们转换为其 id，这样送入模型的就是一个 id 的序列。如果词表已经存在，把单词或者子词转为 id 是非常容易的，所以 tokenization 这一步的难点在于，如何把文本拆分成词语或者子词。 + + + +## word-base tokenization + +​ 把一段文本切分成 token 的方法有非常多，举个例子，看一下下面这个句子`Let's tokenize! Isn't this easy?` + +​ 最简单的方法可以根据空格进行切分，结果如下： + +``` +["Let's", 'tokenize!', "Isn't", 'this', 'easy?'] +``` + +​ 这种方法非常直观，但是是有一个问题，我们可以看到在结果中包含这样的 token: `tokenize!` 和 `easy?`，标点连在单词上，这就会导致一个问题，如果我们不把标点连着词的token放入词表，那么这些在词表中未出现的词就会被当做OOV（out of vocabulary）处理，用``等特殊标记替代，这就损失了一部分信息，这样会非常影响模型的表现，但是如果把出现的所有标点连着词的token全部放入词表，那么可以想象词表会非常庞大（进而导致模型非常庞大），产生巨大的计算开销的同时，模型收敛也会变得困难。同时对于在词表中但是出现次数非常少的token，模型也很难学到其表示。 + +​ 所以标点符号也应该被考虑进到切分的规则中，这样就可以避免词表巨大而词表内部词频稀疏的问题。把这个句子进一步切分结果如下： + +``` +["Let", "'", "s", "tokenize", "!", "Isn", "'", "t", "this", "easy", "?"] +``` + +​ 比直接用空格分割好一点，但是还是有一些瑕疵，比如`Isn't`代表的是`Is not`，所以这个单词最好被切分为`["Is", "n't"]`。这就使得分词变得复杂了，这也是在NLP领域不同的模型使用了各种不同 Tokenizer 的原因之一。这也加大了研究者在复现这些工作的时候的难度。一个句子，使用不同的规则，将有许多种不同的分词结果。我们在使用预训练模型做下游任务的时候，只有使用和它训练时使用的同一种 tokenize 的方法处理数据，才能取得相应的效果。 + +​ 为了解决诸如`Isn't`这类词的分词问题，也出现了一些基于规则的分词方法，比如常用的基于规则的分词器 Spacy，这个句子通过这种分词方法切分的结果如下： + +``` +["Let", "'s", "tokenize", "!", "Is", "n't", "this", "easy", "?"] +``` + +​ 三种不同的切分方法效果对比如下： + +![3 different tokenization scheme: on rules, on punctuaction, on white spaces](tokenization.assets/tokenize.png) + +​ 上面说的三种分词方法，都是词级别的分词方法（word tokenization），简单来说，就是以词为单位对文本进行切分。尽管这是将文本切分为更小的单位的最直观的方式，但是这种方式在处理大量语料的时候会产生一个非常严重的问题——生成一个巨大的词表（因为大量的语料中会存在大量的只出现一次/少次的token）。 + +> ​ 比如 Transformer-XL 这个模型，就用的是基于空格和标点的分词方式，词表大小是夸张的267,735。这么大的词表会导致模型在输入端需要一个巨大的 embedding 矩阵（token id - embedding），在输出端需要一个巨大的 generator 矩阵（hidden value - token id），这会使得模型参数量剧增，导致大量的内存占用以及预测的时间复杂度。**总之，这一类模型的词表大小基本很少会超过50,000，特别是单语模型**。 + +​ 那么既然词级别的分词方法不理想，能不能基于字符级别来分词呢？ + + + +## character-base tokenization + +​ 基于字符级别的分词方法非常简单，简单来说，就是将每个字符看作一个词，还是上面那个句子，切分效果如下： + +``` +['L', 'e', 't', "'", 's', ' ', 't', 'o', 'k', 'e', 'n', 'i', 'z', 'e', '!', ' ', 'I', 's', 'n', "'", 't', ' ', 't', 'h', 'i', 's', ' ', 'e', 'a', 's', 'y', '?'] +``` + +​ 尽管这字符级别的分词方式非常简单并且能够大大缩小词表的大小（进而缩小模型的大小），并且不用担心未知词汇。但是这种表示方式，对于模型来说要学习出有意义的词表示难度大大增加了（字母本身就没有任何的内在含义）。 + +>举个例子，从字母“t”中学习到有意义的词表示要比从“today”中学习到有意的词表示难得多。 + +计算复杂度提升（字母的数目远大于token的数目），输出序列的长度将变大，对于Bert、CNN等限制最大长度的模型将很容易达到最大值。 + +​ 因此，字符级别的分词方式一般很难产生效果好的模型，所以为了兼顾到词级别和字符级别分词方式的好处，产生了一种折中的处理方式，称为子词切分方法（subword tokenization）。 + + + +## subword tokenization + +​ 上面提到的两种分词方式各有各的不足。 + +​ 词级别的分词：1、词表巨大；2、容易出现大量词表外的token（OOV问题）；3、对于相似的词语失去相似度信息（比如dog和dogs在词表中是两个完全不同的词） + +​ 字符级别的分词：1、文本转为 id 序列之后长度非常长；2、字母本身没有内在含义，从单一的字母中很难学习到有意的词表示。 + +​ 为了改进分词方法，在``数目和词表示含义丰富性之间达到平衡，于是产生了一个折中的算法：子词切分（subword tokenization），子词切分的方法也分好几种，但是它们都遵从两个原则： + +​ 1、常用的词语不应该被切分为更小的片段 + +​ 2、不常用的词语应该被分解为有意义的子词 + +> ​ 举个例子，`tokenization`这个词，可以被分解为`token`和`ization`，`token`和`ization`作为单独的子词出现的概率更高（比如`token`、`tokens`、`tokenizing`、`tokenization`，或者`tokenization`、`modernization`等等），同时`tokenization`的意思被作为`token`和`ization`的复合意思被保存下来。 + +​ 这种方式在英文上效果不错，在另外一些语言上效果甚至更好，在某些语言中，可以像搭积木一样，通过子词的组合形成几乎任何单词。 + +​ 子词切分方式使得模型能够在保持一个合理的词表大小的情况下，能够学习到更有意义的子词的表示，同时也能让模型拥有更好地处理词表中没有的词语的能力（将词表中没有的词，分解成词表中有的子词）。 + +​ 当前流行的模型中所用到的 subword Tokenizer 主要有三种不同的算法：BPE（Byte-Pair Encoding），WordPiece 和 SentencePiece。不同的模型所使用的 Tokenizer 的类型不同，比如，在 BertTokenizer 中所使用的就是 WordPiece。 + + + +## BPE（Byte-Pair Encoding） + +​ BPE 算法最早是[Neural Machine Translation of Rare Words with Subword Units (Sennrich et al., 2015)](https://arxiv.org/abs/1508.07909)提出的，是最早的一种数据压缩算法之一。BPE 是一个子词分词方法，本身需要依赖于预分词（pre-tokenization），也就是把文本先切分成词语（BPE算法接收的输入是一个词语的列表，而不是一个生语料）。这个预分词可以是简单的基于空格的分词方法，比如 GPT-2 和 RoBERTa；也可以是更复杂的比如基于规则的分词方法，比如 XLM 和 FlauBERT 用的是 Moses；再比如 GPT 用的是 Spacy。 + +​ 在将文本切分成词语之后，就得到了一个词表，以及这个词表中所有词语的词频。接下来，BPE 创建一个基本词表，这个词表中包含语料中所有出现的字符（在英文中就是26个字母和各种符号），然后统计语料中相邻单元对（这里的单元要在基本词表中）的频数，选取频数最高的单元对合成新的子词单元，加入基本词表中，重复进行直到词表大小达到预设值（词表的大小是个超参数，在训练前就设定好了的）。 + +​ 举个例子，假设在预分词之后，我们得到下面这个词表以及它们相应的词频： + +``` +("hug", 10), ("pug", 5), ("pun", 12), ("bun", 4), ("hugs", 5) +``` + +​ 同时，取出语料中所有出现的字符作为基本词表： + +``` +["b", "g", "h", "n", "p", "s", "u"] +``` + +​ 将第一个词表按照基本词表进行切分： + +``` +("h" "u" "g", 10), ("p" "u" "g", 5), ("p" "u" "n", 12), ("b" "u" "n", 4), ("h" "u" "g" "s", 5) +``` + +​ 计算所有相邻单元对的频率，并且选择频率最高的单元对 + +``` +"ug": 10 + 5 + 5 +``` + +​ `ug`被加入词表，并且语料所有的`ug`相连的部分都被组合 + +``` +("h" "ug", 10), ("p" "ug", 5), ("p" "u" "n", 12), ("b" "u" "n", 4), ("h" "ug" "s", 5) +``` + +​ 然后是`un`和`hug`，这时候的词表为： + +``` +["b", "g", "h", "n", "p", "s", "u", "ug", "un", "hug"] +``` + +​ 语料表示为： + +``` +("hug", 10), ("p" "ug", 5), ("p" "un", 12), ("b" "un", 4), ("hug" "s", 5) +``` + +​ 假设此时词表大小已经达到预设值，于是训练停止，这时候模型学到的这个词表（其实就是组合的规则）就会被应用在新遇到的词上，比如这时候遇到一个词`bug`就会被分解为`["b", "ug"]`，但是如果新遇到的词中存在基本词表中都没有的字符，则还是会用``表示，比如`mug`就会被表示为`["", "ug"]`。在实际的训练数据中，每个正常应该用到的字符都会出现至少一次，所以它们都应该包含在基础词表中，但是对于一些非常特殊的符号（比如表情符号），这种情况仍然有可能发生。 + +​ 词表的大小（基础词表的大小 + BPE 算法执行的轮数）是一个超参数，比如 GPT 的词表大小是40478，因为基础词表大小是478，然后算法运行了40000轮。 + +​ 上面的做法其实是 BPE 算法的简化版本，真正的在做 BPE 时还有一些关键的细节需要注意： + +​ 1、在将单词拆分成最小单元时，会在单词序列后加上””(具体实现上可以使用其它符号)来表示中止符。在子词解码时，中止符可以区分单词边界。 + +​ 2、每次合并后词表大小可能出现3种变化： + +- +1，表明加入合并后的新子词，同时原来的2个子词还保留（2个字词分开出现在语料中）。 +- +0，表明加入合并后的新子词，同时原来的2个子词中一个保留，一个被消解（一个子词完全随着另一个子词的出现而紧跟着出现）。 +- -1，表明加入合并后的新子词，同时原来的2个子词都被消解（2个字词同时连续出现）。 + +​ 3、实际上，随着合并的次数增加，词表大小通常先增加后减小 + +​ 4、得到最终词表后，使用时的编码方式（最大长度匹配）： + +- 将词典中的所有子词按照长度由大到小进行排序； +- 对于单词w，依次遍历排好序的词典。查看当前子词是否是该单词的子字符串，如果是，则输出当前子词，并对剩余单词字符串继续匹配。 +- 如果遍历完字典后，仍然有子字符串没有匹配，则将剩余字符串替换为特殊符号输出，如””。 + +​ 5、解码：如果相邻子词间没有中止符，则将两子词直接拼接，否则两子词之间添加分隔符 + +*** + +- **BBPE（Byte-level BPE）** + +​ 在生成基础词表这一步的时候，如果这个基础词表包含了所有可能的字符，那么这个基础词表会非常大，比如用所有的 unicode 字符来当做基础词表。 + +​ 为了能够缩减基础词表的大小，GPT-2 在生成基础词表的时候，用了字节（一个字节有256种不同的组合）来作为基础词表。这是一个非常巧妙的技巧，它把基础词表的大小限制在了256，同时能够确保所有的基本字符都包含在基础词表中（一个字符可以由多个字节组成）。 + +​ GPT2 的分词器能够在不需要 ``的情况下对所有的文本进行分词（还用了一些额外的规则来处理标点符号）。GPT-2 的词表大小是 50257，其中包括256个字节基本词表，一个特殊的文本结束符号，以及算法运行50000轮产生的子词。 + +*** + + + +## WordPiece + +​ WordPiece 算法最早发表于 [Japanese and Korean Voice Search (Schuster et al., 2012)](https://static.googleusercontent.com/media/research.google.com/ja//pubs/archive/37842.pdf)，它和 BPE 算法非常相似。也有一些预训练模型的分词器用的是这个算法，比如 BERT、DistilBERT 和 Electra。与 BPE 算法类似，WordPiece 算法也是初始化一个基础词表，然后每次从词表中选出两个子词合并成新的子词。 + +​ 与 BPE 的主要区别在于，选择两个子词进行合并的规则：BPE 选择频数最高的相邻子词合并，而 WordPiece 使用的是通过语言模型来计算合并两个单词可能造成的影响，然后选择使得似然函数提升最大的字符对。这个提升是通过结合后的字符对减去结合前的字符对之和得到的。 + +​ 假设句子$S=（t1,t2,...,t_n）$由 n 个 token 组成，$t_i$表示第$i$个 token，假设各个 token 之间是相互独立的，则句子$S$的语言模型似然值等于所有子词概率的乘积 +$$ +logP(S) = \sum^n_{i=1}logP(t_i) +$$ +​ 假设把相邻位置的两个 token $x$ 和 $y$ 进行合并，合并后的 token 记为 $z$，此时句子 $S$ 的似然值变化可以表示为： +$$ +logP(t_z)-(logP(t_x)+logP(t_y)) = log(\frac{P(t_z)}{P(t_x)P(T_y)}) +$$ +​ 也就是说，判断`tokenization`相较于`token` + `ization`是否更适合出现。选择能够提升语言模型概率最大的相邻子词加入词表。 + + + +## Unigram + +​ Unigram 算法最早发表于 [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates (Kudo, 2018)](https://arxiv.org/pdf/1804.10959.pdf)。与 WordPiece 一样，Unigram Language Model(ULM) 同样使用语言模型来挑选子词。不同之处在于，BPE 和 WordPiece 算法的词表大小都是从小到大增加的。而 ULM 则是先初始化一个大词表，这个基础词表包含了所有预分词结果中的 token 以及所有常用的子词（也可以通过 BPE 算法初始化），根据评估准则不断丢弃里边的词，直到满足预设条件。ULM 算法考虑了句子的不同分词可能，因而能够输出带概率的多个分词序列。 + +​ 并没有预训练模型单独使用 ULM 作为分词的算法，但是它常与 SentencePiece 一起使用。 + +​ 对于句子$S$，$\mathop{x}\limits ^{\rightarrow} = (x_1,x_2,...,x_m)$为句子的一个分词结果，由 m 个子词组成。所以，当前分词下句子$S$的似然值可以表示为： +$$ +P(\mathop{x}\limits ^{\rightarrow}) = \prod^m_{i=1}P(x_i) +$$ +​ 对于句子$S$，挑选似然值最大的作为分词结果，则可以表示为： +$$ +x^* = \mathop{argmax}\limits_{x \in U(x)}~P(\mathop{x}\limits ^{\rightarrow}) +$$ +​ 这里的$U(x)$包含了句子的所有分词结果。在实际应用中，词表大小有上万个，直接罗列所有的分词组合不具有操作性。针对这个问题，可以通过维特比算法得到 $x^*$来解决。 + +​ ULM 通过 EM 算法来估计每个子词的概率$P(x_i)$，假设当前词表为V，训练数据为|D|，则当前训练数据中所有句子的所有分词组合形成的概率相加，表示为： +$$ +L = \sum^{|D|}_{s=1}log(P(X^{(s)}))=\sum^{|D|}_{s=1}log(\mathop{\sum_{x \in U(X^{(s)})}}P(x)) +$$ +​ 在训练的开始阶段，ULM 初始化一个巨大的基础词表，然后针对当前词表，用上面的算法计算每个子词在训练数据上的概率，然后计算移除每个子词时，语言模型整体的损失上升了多少，记为这个子词的损失值。然后通过排序，按照一定比例移除掉词表中损失值最低的一部分子词（这个比例一般是10%或者20%），这些子词的损失值最低，意味着他们对语言模型的影响最小。每训练一步，都会移除掉部分的子词，直到词表大小达到预设值为止。（ULM 算法训练的过程中会保留基础的字符以保证任何词语都能被处理）。 + +​ 训练完成的 UML 模型对一个句子给出多种的分词可能，比如说一个训练好的 ULM 分词器的词表如下： + +``` +["b", "g", "h", "n", "p", "s", "u", "ug", "un", "hug"] +``` + +​ 那么 `"hugs"` 就有这么几种分词方法： `["hug", "s"]`， `["h", "ug", "s"]` ， `["h", "u", "g", "s"]`。ULM 在训练的过程中，在保存词表的同时会保存每个词表中 token 的概率，在实际使用的时候，一般会选取概率值最高的分词序列，如果需要也能够输出带有概率的不同分词序列。 + +​ + +## SentencePiece + +​ 上面提到的所有分词算法都存在同一个问题：这些分词方式都是建立在原文本通过空格来分隔词语的。但是并不是所有语言词和词之间都是有空格的。还有就是，将分词结果解码到原来的句子中时，会在不同的词之间添加空格，这样解码出来的标点符号就不会和词语连着。 + +> Raw text: Hello world. +> +> Tokenized: [Hello] [world] [.] +> +> Decoded text: Hello world . + +​ 这就是编码解码出现的歧义性，因此需要特别定义规则来实现互逆。还有一个例子是，在解码阶段，欧洲语言词之间要添加空格，而中文等语言则不应添加空格。一种解决方案是，给不同的语言定制不同的预分词算法和解码算法，比如 XLM 就在中文、日文和泰文上做了尝试。 + +​ 为了能够更好地解决这个问题，谷歌在 [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing (Kudo et al., 2018)](https://arxiv.org/pdf/1808.06226.pdf)中提出，把将所有的字符都转化成 Unicode 编码，空格用`_`来代替，然后进行分词操作。这样空格也不需要特别定义规则了，然后在解码结束后恢复即可。然后还是用 BPE 或者 ULM 算法来构建词表。 + +​ SentencePiece 集成了BPE、ULM 算法，除此之外，它也能支持字符和词级别的分词。使用 SentencePiece + ULM 的模型有 ALBERT、XLNet、T5。 + +​ + diff --git a/examples/bert_title_generation_english/generate.py b/examples/bert_title_generation_english/generate.py index 38a1aca4..9210c5fc 100755 --- a/examples/bert_title_generation_english/generate.py +++ b/examples/bert_title_generation_english/generate.py @@ -6,6 +6,7 @@ model_dir = "../state_dict/" +# Note "./checkpoints_seq2seq/{}/mp_rank_00_model_states.pt", {} is a directory in the checkpoints_seq2seq. model_save_path = "./checkpoints_seq2seq/7079/mp_rank_00_model_states.pt" maxlen = 512 auto_loader = AutoLoader( diff --git a/examples/glm_blank_filling/README.md b/examples/glm_blank_filling/README.md new file mode 100644 index 00000000..34929678 --- /dev/null +++ b/examples/glm_blank_filling/README.md @@ -0,0 +1,110 @@ +GLM naturally handles variable-length blank filling which is crucial for many downstream tasks. +Based on autoregressive blank-filling, GLM randomly +blank out continuous spans of tokens from the input text, +following the idea of autoencoding, and train the model to +reconstruct the spans, following the idea of autoregressive +pre-training. +As a result, GLM learns +autoregressive generation during pre-training. + + +GLM finetunes on downstream tasks and reformulate them as blank-filling generation. Each task is associated with a human-crafted cloze question, and the model predicts the answer +to the cloze. For example, a sentiment classification task +is reformulated as filling the blank in “[SENTENCE]. It’s +really ”. The prediction of “good” or “bad” indicates the +sentiment being positive or negative. With such formulation, +GLM benefits from the consistency between pretraining and +finetuning, because both pretraining and finetuning involves +training the model to generate text given context. To make the pre-training +method better suited for text generation tasks, GLM also studys +a multi-task pre-training setup, where the model is jointly +trained to reconstruct masked spans and generate longer +text. + +In GLM, there are three MASK methods, corresponding to three prediction formats respectively. +1. ```[MASK]```, Token-level. This mask method will only mask random tokens in a sentence, + which has the least mask part among the three maks methods. Therefore, + the content generated is also limited. For example: ``[CLS]北京故宫是中国[MASK]非物质文化遗产。<|endoftext|><|startofpiece|>现存最大的古代宫殿建筑, 也是``. +2. ```[sMASK]```: Entity-level. We restrict that the masked + spans must be full sentences. Multiple spans + (sentences) are sampled to cover 15% of + the original tokens. This objective aims for + seq2seq tasks whose predictions are often + complete sentences or paragraphs. + The ```[sMASK]``` is longer than the text predicted by ```[MASK]```. For example: `` [CLS]人工智能是一个以计算机科学为基础,由计算机、数学、哲学等多学科交叉融合的交叉学科,[sMASK],具有非常巨大的前景。<|endoftext|><|startofpiece|>它涉及的信息量不仅非常巨大,而且也是人工智能发展的一个关键,其研究内容包括人的感觉、知觉和思维,以及如何理解各种现象,以及解释现象的本性和原因等,通过计算机来进行系统的分析推理、建立数学模型,并模拟人类意识。`` +3. ```[gMASK]```: Document-level. We sample a single span + whose length is sampled from a uniform distribution over 50%–100% of the original length. + The objective aims for long text generation. For example: ``[CLS]问题:啤酒伤胃吗?回答:[gMASK]<|startofpiece|>谢邀。 我是啤酒爱好者,但是我不喝酒。 我以前也说过,喝酒伤身,啤酒伤胃,伤肠道。 现在我也知道了啤酒伤人的很多细节,我就不瞎几把的说,大家看图片就知道了。 其实啤酒伤身这个说法只是表面而已。 啤酒中含有少量的碳酸和酒精,碳酸和酒精是成酸性物质,而乙醇是脂溶性的,酒精在胃里能够被分解,生成乙醇和二氧化碳,在体内是水和二氧化碳,两种物质会迅速发生中和反应,结果导致人体出现头痛、呕吐、胸痛、浑身发热等现象,这就是所谓喝大了,喝多了。 啤酒的含糖量在15%左右,喝多了也是伤身的,啤酒含糖量较高的主要成分是水分,而水分的体积比酒精大,所以酒精进入人体,与水相遇,就会产生大量气体,二氧化碳、水、一氧化碳等刺激人体,造成人体大量出汗,使体内温度升高,`` + + +As example, GLM finish the question task as an autoregressive blank in- +filling task + +```python +import torch +from flagai.model.glm_model import GLMModel +from flagai.data.tokenizer import GLMLargeChTokenizer +from flagai.model.predictor.predictor import Predictor +if __name__ == "__main__": + """Main training program.""" + print('Generate Samples') + tokenizer = GLMLargeChTokenizer(vocab_path='./checkpoints/glm-large-ch/cog-pretrain.model', + add_block_symbols=True, + add_task_mask=True, + add_decoder_mask=False, + fix_command_token=False) + model = GLMModel.from_pretrain(model_name='glm-large-ch', only_download_config=False) + model.cuda(torch.cuda.current_device()) + predictor = Predictor(model, tokenizer) + # question-answering + text = '问题：啤酒伤胃吗？回答：[gMASK]' + output=predictor.predict_generate_randomsample(text) + print(text,'\n',output) +``` + +Similar to BERT, GLM can predict masked tokens as + +```python +import torch +from flagai.model.glm_model import GLMModel +from flagai.data.tokenizer import GLMLargeChTokenizer +from flagai.model.predictor.predictor import Predictor +if __name__ == "__main__": + """Main training program.""" + print('Generate Samples') + tokenizer = GLMLargeChTokenizer(vocab_path='./checkpoints/glm-large-ch/cog-pretrain.model', + add_block_symbols=True, + add_task_mask=True, + add_decoder_mask=False, + fix_command_token=False) + model = GLMModel.from_pretrain(model_name='glm-large-ch', only_download_config=False) + model.cuda(torch.cuda.current_device()) + predictor = Predictor(model, tokenizer) + # question-answering + text = '北京故宫是中国[MASK]非物质文化遗产。' + output=predictor.predict_generate_randomsample(text) + print(text,'\n',output) +``` +and predict masked sentences as + +```python +import torch +from flagai.model.glm_model import GLMModel +from flagai.data.tokenizer import GLMLargeChTokenizer +from flagai.model.predictor.predictor import Predictor +if __name__ == "__main__": + """Main training program.""" + print('Generate Samples') + tokenizer = GLMLargeChTokenizer(vocab_path='./checkpoints/glm-large-ch/cog-pretrain.model', + add_block_symbols=True, + add_task_mask=True, + add_decoder_mask=False, + fix_command_token=False) + model = GLMModel.from_pretrain(model_name='glm-large-ch', only_download_config=False) + model.cuda(torch.cuda.current_device()) + predictor = Predictor(model, tokenizer) + # question-answering + text = '人工智能是一个以计算机科学为基础，由计算机、数学、哲学等多学科交叉融合的交叉学科，[sMASK]，具有非常巨大的前景。' + output=predictor.predict_generate_randomsample(text) + print(text,'\n',output) +``` \ No newline at end of file diff --git a/examples/glm_blank_filling/glm_generate_samples.py b/examples/glm_blank_filling/glm_generate_samples.py new file mode 100755 index 00000000..043928b6 --- /dev/null +++ b/examples/glm_blank_filling/glm_generate_samples.py @@ -0,0 +1,64 @@ +# coding=utf-8 +# Copyright (c) 2019, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Sample Generate GPT2""" +import sys +# sys.path.append('/data/wang/models/GLMgeneration') +# sys.path.append('/data/wang/models/FlagAI') +import os +import torch + +from flagai.model.glm_model import GLMModel +from flagai.data.tokenizer import GLMLargeChTokenizer +from flagai.model.predictor.predictor import Predictor +if __name__ == "__main__": + """Main training program.""" + print('Generate Samples') + # Random seeds for reproducability. + # Model, + model = GLMModel.from_pretrain(model_name='GLM-large-ch', + download_path="./state_dict/") + tokenizer = GLMLargeChTokenizer( + vocab_path='./state_dict/GLM-large-ch/cog-pretrain.model', + add_block_symbols=True, + add_task_mask=True, + add_decoder_mask=False, + fix_command_token=False) + + model.cuda(torch.cuda.current_device()) + + predictor = Predictor(model, tokenizer) + # generate samples + text = [ + '问题：啤酒伤胃吗？回答：[gMASK]', "问题：隔夜菜能吃吗？回答：[gMASK]", "问题：如何评价许嵩？回答：[gMASK]" + ] + for t in text: + output = predictor.predict_generate_randomsample( + t, top_k=50, repetition_penalty=4.0, top_p=1.0) + print(t, '\n', output) + + text = ['北京故宫是中国[MASK]非物质文化遗产。', "上海是中国[MASK]大都市。", "天津大学是[MASK]现代大学。"] + for t in text: + output = predictor.predict_generate_randomsample( + t, top_k=50, repetition_penalty=4.0, top_p=1.0) + print(t, '\n', output) + # + text = [ + "人工智能是一个以计算机科学为基础，由计算机、数学、哲学等多学科交叉融合的交叉学科，[sMASK]，具有非常巨大的前景。", + "最近十多年来，人工神经网络的研究工作不断深入，已经取得了很大的进展，[sMASK]，表现出了良好的智能特性。" + ] + for t in text: + output = predictor.predict_generate_randomsample( + t, top_k=50, repetition_penalty=4.0, top_p=1.0) + print(t, '\n', output) diff --git a/examples/glm_poetry_generation/generate.py b/examples/glm_poetry_generation/generate.py index 6d2f97a0..1c1e86b3 100755 --- a/examples/glm_poetry_generation/generate.py +++ b/examples/glm_poetry_generation/generate.py @@ -3,7 +3,9 @@ from flagai.model.predictor.predictor import Predictor device = torch.device("cuda" if torch.cuda.is_available() else "cpu") -model_save_path = "/mnt/finetune_models/glm_poetry/mp_rank_00_model_states.pt" + +# Note "./checkpoints_poetry/{}/mp_rank_00_model_states.pt", {} is a directory in the checkpoints_poetry. +model_save_path = "./checkpoints_poetry/1/mp_rank_00_model_states.pt" auto_loader = AutoLoader("seq2seq", model_name="GLM-large-ch", diff --git a/examples/glm_poetry_generation/train.py b/examples/glm_poetry_generation/train.py index f7b85c08..101c387e 100644 --- a/examples/glm_poetry_generation/train.py +++ b/examples/glm_poetry_generation/train.py @@ -7,7 +7,7 @@ cur_dir = os.path.dirname(os.path.abspath(__file__)) src_dir = cur_dir + '/data/src.txt' tgt_dir = cur_dir + '/data/tgt.txt' -model_dir = "./state_dict/glm/" # ./state_dict/roberta/ # 模型位置 +model_dir = "./state_dict/" # ./state_dict/roberta/ # 模型位置 def read_file(): diff --git a/examples/glm_pretrain/train.py b/examples/glm_pretrain/train.py index 5a5cad3a..4a92d099 100644 --- a/examples/glm_pretrain/train.py +++ b/examples/glm_pretrain/train.py @@ -35,17 +35,10 @@ def __init__(self): save_epoch=10) model = GLMForSeq2Seq.from_pretrain(model_name='GLM-large-ch') - tokenizer = GLMLargeChTokenizer(add_block_symbols=True, - add_task_mask=False, - add_decoder_mask=False, - fix_command_token=True) ds_args = DatasetArguments() - tokenizer = GLMLargeChTokenizer(fix_command_token=True, - add_block_symbols=True, - add_task_mask=True, - add_decoder_mask=False) + tokenizer = GLMLargeChTokenizer() ds_args = add_args(ds_args, tokenizer) @@ -77,9 +70,35 @@ def create_dataset(tokenizer, should_split): collate_fn = ConstructBlockStrategy( tokenizer, 512, eod_token=tokenizer.get_command('eos').Id) metric_methods = DEFAULT_METRICS['pretrain'] - trainer.train(model, - collate_fn=collate_fn, - train_dataset=datasets[0], - valid_dataset=datasets[1], - metric_methods=metric_methods) - + # trainer.train(model, + # collate_fn=collate_fn, + # train_dataset=datasets[0], + # valid_dataset=datasets[1], + # metric_methods=metric_methods) + import torch + + train_loader = torch.utils.data.DataLoader(datasets[0], + batch_size=16, + shuffle=False, + num_workers=1, + drop_last=False, + pin_memory=False, + collate_fn=collate_fn) + for data_iterator in train_loader: + dct = data_iterator + ''' + input_ids torch.Size([16, 2, 256]) + labels torch.Size([16]) + position_ids torch.Size([16, 2, 2, 256]) + attention_mask torch.Size([16, 2]) + target_ids torch.Size([16, 2, 256]) + logit_mask torch.Size([16, 2, 256]) + loss_mask torch.Size([16, 2]) + + ''' + for key, value in dct.items(): + try: + print(key, value.size()) + except: + print(key, len(value)) + break diff --git a/examples/glm_seq2seq/train.py b/examples/glm_seq2seq/train.py index f29fbbb2..dcd8f06e 100644 --- a/examples/glm_seq2seq/train.py +++ b/examples/glm_seq2seq/train.py @@ -1,12 +1,10 @@ -import sys - -sys.path.append('..') from flagai.trainer import Trainer from flagai.model.glm_model import GLMForSeq2Seq from flagai.data.tokenizer import GLMLargeEnWordPieceTokenizer, GLMLargeChTokenizer from flagai.data.dataset import Seq2SeqDataset +from flagai.test_utils import Seq2SeqCollateArguments from flagai.data.dataset.superglue.control import DEFAULT_METRICS, CH_TASKS - +from flagai.data.dataset import ConstructSeq2seqStrategy import unittest @@ -17,10 +15,11 @@ def test_init_trainer_pytorch(self): # task_name :['cmrc',xxxx] task_name = "cmrc" + cl_args = Seq2SeqCollateArguments() trainer = Trainer(env_type='pytorch', epochs=1, batch_size=4, - eval_interval=100, + eval_interval=5, log_interval=50, experiment_name='glm_large', pytorch_device='cpu', @@ -30,9 +29,9 @@ def test_init_trainer_pytorch(self): if task_name in CH_TASKS: tokenizer = GLMLargeChTokenizer(add_block_symbols=True, - add_task_mask=False, + add_task_mask=True, add_decoder_mask=False, - fix_command_token=False) + fix_command_token=True) model_name = 'GLM-large-ch' else: tokenizer = GLMLargeEnWordPieceTokenizer( @@ -47,13 +46,16 @@ def test_init_trainer_pytorch(self): data_dir='./datasets/', dataset_type='dev', tokenizer=tokenizer) - + collate_fn = ConstructSeq2seqStrategy(cl_args, + tokenizer, + task_name=task_name) train_dataset.example_list = train_dataset.example_list[:20] valid_dataset.example_list = valid_dataset.example_list[:20] model = GLMForSeq2Seq.from_pretrain(model_name=model_name) trainer.train(model, + collate_fn=collate_fn, train_dataset=train_dataset, valid_dataset=valid_dataset, metric_methods=DEFAULT_METRICS[task_name]) diff --git a/examples/glm_superglue/deepspeed.json b/examples/glm_superglue/deepspeed.json new file mode 100644 index 00000000..a0dfee13 --- /dev/null +++ b/examples/glm_superglue/deepspeed.json @@ -0,0 +1,40 @@ +{ + "train_micro_batch_size_per_gpu": 2, + "gradient_accumulation_steps": 1, + "steps_per_print": 100, + "gradient_clipping": 1.0, + "zero_optimization": { + "stage": 3, + "contiguous_gradients": false, + "overlap_comm": true, + "reduce_scatter": true, + "reduce_bucket_size": 5e7, + "allgather_bucket_size": 5e7, + "cpu_offload": true + }, + "zero_allow_untested_optimizer": true, + "fp16": { + "enabled": true, + "loss_scale": 0, + "loss_scale_window": 1000, + "hysteresis": 2, + "min_loss_scale": 1 + }, + "optimizer": { + "type": "Adam", + "params": { + "lr": 0.0004, + "weight_decay": 0.01, + "betas": [ + 0.9, + 0.98 + ], + "eps": 1e-6 + } + }, + "activation_checkpointing": { + "partition_activations": false, + "contiguous_memory_optimization": false + }, + "wall_clock_breakdown": false + } \ No newline at end of file diff --git a/examples/glm_superglue/hostfile b/examples/glm_superglue/hostfile new file mode 100644 index 00000000..4c93a71d --- /dev/null +++ b/examples/glm_superglue/hostfile @@ -0,0 +1 @@ +127.0.0.1 slots=4 \ No newline at end of file diff --git a/examples/glm_superglue/train.py b/examples/glm_superglue/train.py index 296a2ae5..d337d432 100644 --- a/examples/glm_superglue/train.py +++ b/examples/glm_superglue/train.py @@ -9,69 +9,52 @@ from flagai.data.dataset import ConstructSuperglueStrategy -class TrainerTestCase(unittest.TestCase): - - def test_init_trainer_pytorch(self): - # task_name options: ['boolq', 'cb', 'copa', 'multirc', 'rte', 'wic', 'wsc', 'afqmc', 'tnews'] - task_name = "wsc" - - trainer = Trainer(env_type='pytorch', - epochs=10, - batch_size=4, - eval_interval=100, - log_interval=50, - experiment_name='glm_large', - pytorch_device='cuda', - load_dir=None, - lr=1e-4) - print("downloading...") - - cl_args = CollateArguments() - cl_args.multi_token = task_name in MULTI_TOKEN_TASKS - - if task_name in CH_TASKS: - model_name = 'GLM-large-ch' - tokenizer = GLMLargeChTokenizer(add_block_symbols=True, - add_task_mask=False, - add_decoder_mask=False, - fix_command_token=True) - else: - model_name = 'GLM-large-en' - tokenizer = GLMLargeEnWordPieceTokenizer() - - if cl_args.multi_token: - model = GLMForMultiTokenCloze.from_pretrain(model_name=model_name) - else: - model = GLMForSingleTokenCloze.from_pretrain(model_name=model_name) - - train_dataset = SuperGlueDataset(task_name=task_name, - data_dir='./datasets/', - dataset_type='train', - tokenizer=tokenizer) - - collate_fn = ConstructSuperglueStrategy(cl_args, - tokenizer, - task_name=task_name) - - valid_dataset = SuperGlueDataset(task_name=task_name, - data_dir='./datasets/', - dataset_type='dev', - tokenizer=tokenizer) - - metric_methods = DEFAULT_METRICS[task_name] - trainer.train(model, - collate_fn=collate_fn, - train_dataset=train_dataset, - valid_dataset=valid_dataset, - metric_methods=metric_methods) - - -def suite(): - suite = unittest.TestSuite() - suite.addTest(TrainerTestCase('test_init_trainer_pytorch')) - return suite - - -if __name__ == '__main__': - runner = unittest.TextTestRunner() - runner.run(suite()) +# task_name options: ['boolq', 'cb', 'copa', 'multirc', 'rte', 'wic', 'wsc', 'afqmc', 'tnews'] +task_name = "afqmc" + +trainer = Trainer(env_type='pytorch', + epochs=10, + batch_size=4, + eval_interval=100, + log_interval=50, + experiment_name='glm_large', + pytorch_device='cuda', + load_dir=None, + lr=1e-4) +print("downloading...") + +cl_args = CollateArguments() +cl_args.multi_token = task_name in MULTI_TOKEN_TASKS + +if task_name in CH_TASKS: + model_name = 'GLM-large-ch' + tokenizer = GLMLargeChTokenizer() +else: + model_name = 'GLM-large-en' + tokenizer = GLMLargeEnWordPieceTokenizer() + +if cl_args.multi_token: + model = GLMForMultiTokenCloze.from_pretrain(model_name=model_name) +else: + model = GLMForSingleTokenCloze.from_pretrain(model_name=model_name) + +train_dataset = SuperGlueDataset(task_name=task_name, + data_dir='./datasets/', + dataset_type='train', + tokenizer=tokenizer) + +collate_fn = ConstructSuperglueStrategy(cl_args, + tokenizer, + task_name=task_name) + +valid_dataset = SuperGlueDataset(task_name=task_name, + data_dir='./datasets/', + dataset_type='dev', + tokenizer=tokenizer) + +metric_methods = DEFAULT_METRICS[task_name] +trainer.train(model, + collate_fn=collate_fn, + train_dataset=train_dataset, + valid_dataset=valid_dataset, + metric_methods=metric_methods) diff --git a/examples/glm_superglue/train_ch.py b/examples/glm_superglue/train_10b_clue.py similarity index 54% rename from examples/glm_superglue/train_ch.py rename to examples/glm_superglue/train_10b_clue.py index c4d06664..0baa19b4 100644 --- a/examples/glm_superglue/train_ch.py +++ b/examples/glm_superglue/train_10b_clue.py @@ -1,31 +1,34 @@ -import torch.utils.data -from torch.optim import Adam -from flagai.schedulers import AnnealingLR from flagai.trainer import Trainer -from flagai.model.glm_model import GLMModel, GLMForSingleTokenCloze -from flagai.data.tokenizer.glm_large_ch.glm_large_ch_tokenizer import GLMLargeChTokenizer +from flagai.model.glm_model import GLMForSingleTokenCloze +from flagai.data.tokenizer import GLMLargeChTokenizer from flagai.metrics import accuracy_metric from flagai.data.dataset import SuperGlueDataset from flagai.test_utils import CollateArguments -device = torch.device("cuda" if torch.cuda.is_available() else "cpu") -trainer = Trainer(env_type='pytorch', - pytorch_device=device, +task_name = 'tnews' +trainer = Trainer(env_type='deepspeed', epochs=2, - batch_size=8, + batch_size=1, eval_interval=1000, - log_interval=500, - save_dir="./glm_superglue_ch") - -model = GLMForSingleTokenCloze.from_pretrain(download_path="./state_dict", - model_name="GLM-large-ch") - -optimizer = Adam(model.parameters(), lr=1e-5, weight_decay=1e-3) - -tokenizer = GLMLargeChTokenizer() - -task_name = 'afqmc' + checkpoint_activations=False, + fp16=True, + log_interval=1, + save_dir="./glm_superglue_en", + master_ip='127.0.0.1', + master_port=17235, + num_nodes=1, + num_gpus=2, + hostfile='./hostfile', + model_parallel_size=2, + deepspeed_config='./deepspeed.json', + training_script=__file__) + +model = GLMForSingleTokenCloze.from_pretrain(download_path="/mnt/test_10b_models", + model_name="glm-10b-ch") + + +tokenizer = GLMLargeChTokenizer() train_dataset = SuperGlueDataset(task_name=task_name, data_dir='/mnt/datasets/yan/', dataset_type='train', @@ -39,7 +42,6 @@ cl_args = CollateArguments() cl_args.cloze_eval = True - if task_name in ['copa', 'wsc', 'record']: cl_args.multi_token = True @@ -48,15 +50,8 @@ collate_fn = ConstructSuperglueStrategy(cl_args, tokenizer, task_name=task_name) - -lr_scheduler = AnnealingLR(optimizer, - start_lr=1e-5, - warmup_iter=int(0.1 * 2 * len(train_dataset)), - decay_style='linear', - num_iters=2 * len(train_dataset)) trainer.train(model, - optimizer=optimizer, - lr_scheduler=lr_scheduler, + train_dataset=train_dataset, valid_dataset=valid_dataset, collate_fn=collate_fn, diff --git a/examples/glm_superglue/train_en.py b/examples/glm_superglue/train_10b_superglue.py similarity index 57% rename from examples/glm_superglue/train_en.py rename to examples/glm_superglue/train_10b_superglue.py index 08b592df..56efdbc1 100644 --- a/examples/glm_superglue/train_en.py +++ b/examples/glm_superglue/train_10b_superglue.py @@ -1,30 +1,33 @@ -import torch.utils.data -from torch.optim import Adam -from flagai.schedulers import AnnealingLR from flagai.trainer import Trainer -from flagai.model.glm_model import GLMModel, GLMForSingleTokenCloze -from flagai.data.tokenizer.glm_large_en.glm_large_en_tokenizer import GLMLargeEnWordPieceTokenizer +from flagai.model.glm_model import GLMForSingleTokenCloze +from flagai.data.tokenizer import GLMLargeEnWordPieceTokenizer from flagai.metrics import accuracy_metric from flagai.data.dataset import SuperGlueDataset from flagai.test_utils import CollateArguments -device = torch.device("cuda" if torch.cuda.is_available() else "cpu") -trainer = Trainer(env_type='pytorch', - pytorch_device=device, +task_name = 'boolq' +trainer = Trainer(env_type='deepspeed+mpu', epochs=2, - batch_size=8, + batch_size=1, eval_interval=1000, - log_interval=500, - save_dir="./glm_superglue_en") - -model = GLMForSingleTokenCloze.from_pretrain(download_path="./state_dict", - model_name="GLM-large-en") - -optimizer = Adam(model.parameters()) + checkpoint_activations=False, + fp16=True, + log_interval=1, + save_dir="./glm_superglue_en", + master_ip='127.0.0.1', + master_port=17755, + num_nodes=1, + num_gpus=2, + hostfile='./hostfile', + model_parallel_size=2, + deepspeed_config='./deepspeed.json', + training_script=__file__) + +model = GLMForSingleTokenCloze.from_pretrain(download_path="/mnt/test_10b_models", + model_name="GLM-10b-en") tokenizer = GLMLargeEnWordPieceTokenizer() -task_name = 'boolq' train_dataset = SuperGlueDataset(task_name=task_name, data_dir='/mnt/datasets/yan/', dataset_type='train', @@ -47,14 +50,7 @@ collate_fn = ConstructSuperglueStrategy(cl_args, tokenizer, task_name=task_name) -lr_scheduler = AnnealingLR(optimizer, - start_lr=1e-5, - warmup_iter=int(0.1 * 2 * len(train_dataset)), - decay_style='linear', - num_iters=2 * len(train_dataset)) trainer.train(model, - optimizer=optimizer, - lr_scheduler=lr_scheduler, train_dataset=train_dataset, valid_dataset=valid_dataset, collate_fn=collate_fn, diff --git a/examples/glm_superglue/train_prefix.py b/examples/glm_superglue/train_prefix.py new file mode 100644 index 00000000..dd374a89 --- /dev/null +++ b/examples/glm_superglue/train_prefix.py @@ -0,0 +1,63 @@ +from flagai.trainer import Trainer +from flagai.model.glm_model import GLMForSingleTokenCloze, GLMForMultiTokenCloze, GLMForMultiTokenClozeFast, GLMForSequenceClassification +from flagai.data.tokenizer import GLMLargeEnWordPieceTokenizer, GLMLargeChTokenizer + +from flagai.data.dataset import SuperGlueDataset +from flagai.test_utils import CollateArguments +from flagai.data.dataset.superglue.control import DEFAULT_METRICS, MULTI_TOKEN_TASKS, CH_TASKS +import unittest +from flagai.data.dataset import ConstructSuperglueStrategy + + +# task_name options: ['boolq', 'cb', 'copa', 'multirc', 'rte', 'wic', 'wsc', 'afqmc', 'tnews'] +task_name = "cb" + +trainer = Trainer(env_type='pytorch', + epochs=10, + batch_size=4, + eval_interval=100, + log_interval=50, + experiment_name='glm_large', + pytorch_device='cuda', + load_dir=None, + lr=1e-4) +print("downloading...") + +cl_args = CollateArguments() +cl_args.cloze_eval = False +cl_args.multi_token = task_name in MULTI_TOKEN_TASKS + +if task_name in CH_TASKS: + model_name = 'GLM-large-ch' + tokenizer = GLMLargeChTokenizer(add_block_symbols=True, + add_task_mask=False, + add_decoder_mask=False, + fix_command_token=True) +else: + model_name = 'GLM-large-en' + tokenizer = GLMLargeEnWordPieceTokenizer() + +model = GLMForSequenceClassification.from_pretrain(model_name=model_name, spell_length=2, + class_num=3, tune_prefix_layers=1) + +train_dataset = SuperGlueDataset(task_name=task_name, + data_dir='./datasets/', + dataset_type='train', + tokenizer=tokenizer) + +collate_fn = ConstructSuperglueStrategy(cl_args, + tokenizer, + task_name=task_name) + +valid_dataset = SuperGlueDataset(task_name=task_name, + data_dir='./datasets/', + dataset_type='dev', + tokenizer=tokenizer) + +metric_methods = DEFAULT_METRICS[task_name] +trainer.train(model, + collate_fn=collate_fn, + train_dataset=train_dataset, + valid_dataset=valid_dataset, + metric_methods=metric_methods) + diff --git a/examples/glm_title_generation/generate.py b/examples/glm_title_generation/generate.py new file mode 100644 index 00000000..a16044cc --- /dev/null +++ b/examples/glm_title_generation/generate.py @@ -0,0 +1,46 @@ +import torch +from flagai.auto_model.auto_loader import AutoLoader +from flagai.model.predictor.predictor import Predictor + +device = torch.device("cuda" if torch.cuda.is_available() else "cpu") + +# Note "./checkpoints_glm_title_generation/{}/mp_rank_00_model_states.pt", {} is a directory in the checkpoints_glm_title_generation. +model_save_path = "./checkpoints_glm_title_generation/8/mp_rank_00_model_states.pt" + +auto_loader = AutoLoader("seq2seq", + model_name="GLM-large-ch", + model_dir="./state_dict/") +model = auto_loader.get_model() +tokenizer = auto_loader.get_tokenizer() + +model.load_state_dict( + torch.load(model_save_path, map_location=device)["module"]) + +model.to(device) +model.eval() +predictor = Predictor(model, tokenizer) + +test_data = [ + "本文总结了十个可穿戴产品的设计原则，而这些原则同样也是笔者认为是这个行业最吸引人的地方，1为人们解决重复性问题2从人开始而不是从机器开始3要引起注意但不要刻意4提升用户能力而不是取代人。", + "2007年乔布斯向人们展示iPhone并宣称它将会改变世界，还有人认为他在夸大其词，然而在8年后以iPhone为代表的触屏智能手机已经席卷全球各个角落，未来智能手机将会成为真正的个人电脑为人类发展做出更大的贡献。", + "雅虎发布2014年第四季度财报并推出了免税方式剥离其持有的阿里巴巴集团15％股权的计划打算将这一价值约400亿美元的宝贵投资分配给股东，截止发稿前雅虎股价上涨了大约7％至5145美元" +] + +for text in test_data: + print('===============================================\n') + print(text, ":") + for i in range(1): #generate several times + print("--------------sample %d :-------------------" % (i)) + print('-----------random sample: --------------') + print( + predictor.predict_generate_randomsample(text, + out_max_length=66, + top_k=10, + top_p=1.0, + repetition_penalty=4.0, + temperature=1.2)) + print('-----------beam search: --------------') + print( + predictor.predict_generate_beamsearch(text, + out_max_length=66, + beam_size=10)) diff --git a/examples/glm_title_generation/train.py b/examples/glm_title_generation/train.py index a00748cd..35823d2c 100644 --- a/examples/glm_title_generation/train.py +++ b/examples/glm_title_generation/train.py @@ -2,6 +2,7 @@ import numpy as np import torch from torch.utils.data import Dataset +from flagai.auto_model.auto_loader import AutoLoader from flagai.model.glm_model import GLMModel, GLMForSeq2Seq from flagai.data.tokenizer import GLMLargeChTokenizer from flagai.trainer import Trainer @@ -10,37 +11,30 @@ trainer = Trainer( env_type="pytorch", - experiment_name="roberta_seq2seq", + experiment_name="GLM_seq2seq", batch_size=1, gradient_accumulation_steps=1, - lr=2e-4, - weight_decay=1e-3, + lr=1e-5, + weight_decay=1e-5, epochs=10, log_interval=10, eval_interval=10000, load_dir=None, pytorch_device=device, - save_dir="checkpoints", + save_dir="checkpoints_glm_title_generation", save_epoch=1, num_checkpoints=1, - master_ip='127.0.0.1', - master_port=17750, - num_nodes=1, - num_gpus=2, - hostfile='./hostfile', - deepspeed_config='./deepspeed.json', - training_script=__file__, ) -cur_dir = os.path.dirname(os.path.abspath(__file__)) -src_dir = cur_dir + '/data/train.src' -tgt_dir = cur_dir + '/data/train.tgt' -model_dir = "./state_dict/roberta/" # 模型位置 -os.makedirs(model_dir, exist_ok=True) -model_save_path = "./bert_auto_title_model.bin" +src_dir = './data/train.src' +tgt_dir = './data/train.tgt' + maxlen = 256 -model = GLMForSeq2Seq.from_pretrain(model_name='GLM-large-ch') -tokenizer = GLMLargeChTokenizer() +auto_loader = AutoLoader("seq2seq", + model_name="GLM-large-ch", + model_dir="./state_dict/") +model = auto_loader.get_model() +tokenizer = auto_loader.get_tokenizer() def read_file(): @@ -60,7 +54,7 @@ def read_file(): return src, tgt -class BertSeq2seqDataset(Dataset): +class GLMSeq2seqDataset(Dataset): def __init__(self, sents_src, @@ -68,7 +62,7 @@ def __init__(self, tokenizer, max_src_length=300, max_tgt_length=200): - super(BertSeq2seqDataset, self).__init__() + super(GLMSeq2seqDataset, self).__init__() self.sents_src = sents_src self.sents_tgt = sents_tgt self.tokenizer = tokenizer @@ -79,58 +73,63 @@ def __init__(self, def __getitem__(self, i): source_text = self.sents_src[i] target_text = self.sents_tgt[i] - cls_id = self.tokenizer.get_command('ENC').Id - mask_token = 'MASK' - mask_id = self.tokenizer.get_command(mask_token).Id - pad_id = self.tokenizer.get_command('pad').Id - sop_id = self.tokenizer.get_command('sop').Id - eop_id = self.tokenizer.get_command('eop').Id - source_tokens = self.tokenizer.EncodeAsIds(" " + source_text) - prompt = [cls_id, mask_id] + self.tokenizer.EncodeAsIds(" Content:") - if len(source_tokens) > self.max_src_length - len(prompt): - source_tokens = source_tokens[:self.max_src_length - len(prompt)] - source_tokens = prompt + source_tokens - - if len(source_tokens) < self.max_src_length: - source_tokens = source_tokens + [pad_id] * (self.max_src_length - - len(source_tokens)) - sep = len(source_tokens) - position_ids = list(range(len(source_tokens))) - block_position_ids = [0] * len(source_tokens) - mask_pos = source_tokens.index(mask_id) - target_tokens = self.tokenizer.EncodeAsIds(" " + target_text) - target_tokens = target_tokens + [eop_id] - if len(target_tokens) > self.max_tgt_length: - target_tokens = target_tokens[:self.max_tgt_length] - loss_mask = [1] * len(target_tokens) - if len(target_tokens) < self.max_tgt_length: - loss_mask += [0] * (self.max_tgt_length - len(target_tokens)) - target_tokens += [pad_id - ] * (self.max_tgt_length - len(target_tokens)) - tokens = source_tokens + [sop_id] + target_tokens[:-1] - loss_mask = [0] * len(source_tokens) + loss_mask - target_ids = [0] * len(source_tokens) + target_tokens - position_ids += [mask_pos] * len(target_tokens) - if self.no_block_position: - block_position_ids += [1] * len(target_tokens) - else: - block_position_ids += list(range(1, len(target_tokens) + 1)) - position_ids = [position_ids, block_position_ids] - sample = { - 'input_ids': np.array(tokens, dtype=np.int64), - 'target_ids': np.array(target_ids, dtype=np.int64), - 'attention_mask': np.array(sep, dtype=np.int64), - 'loss_mask': np.array(loss_mask, dtype=np.int64), - "position_ids": np.array(position_ids, dtype=np.int64) - } - return sample + data = self.tokenizer.encode_plus(source_text, target_text) + + return data def __len__(self): return len(self.sents_src) +class GLMPoetryDynamicCollateFN(): #padding process in each batch + + def __init__(self, pad_id): + self.pad_id = pad_id + + def pad_token(self, tokens, max_length): + pad_len = max_length - len(tokens) + tokens += [self.pad_id] * pad_len + return tokens + + def pad_position_ids(self, position_ids, max_length): + pad_len = max_length - len(position_ids[0]) + position_ids[0] += [len(position_ids[0]) + x for x in range(pad_len)] + position_ids[1] += [1] * pad_len + return position_ids + + def pad_loss_mask(self, loss_mask, max_length): + pad_len = max_length - len(loss_mask) + loss_mask += [0] * pad_len + return loss_mask + + def __call__(self, batch): + input_ids = [data["input_ids"] for data in batch] + target_ids = [data["target_ids"] for data in batch] + position_ids = [data["position_ids"] for data in batch] + attention_mask = [data['attention_mask'] for data in batch] + loss_mask = [data['loss_mask'] for data in batch] + + max_length = max([len(t) for t in input_ids]) + for i in range(len(input_ids)): + input_ids[i] = self.pad_token(input_ids[i], max_length) + target_ids[i] = self.pad_token(target_ids[i], max_length) + position_ids[i] = self.pad_position_ids(position_ids[i], + max_length) + loss_mask[i] = self.pad_loss_mask(loss_mask[i], max_length) + return { + 'input_ids': torch.LongTensor(input_ids), + 'target_ids': torch.LongTensor(target_ids), + 'position_ids': torch.LongTensor(position_ids), + 'attention_mask': torch.LongTensor(attention_mask), + 'loss_mask': torch.LongTensor(loss_mask) + } + + sents_src, sents_tgt = read_file() +my_collate_fn = GLMPoetryDynamicCollateFN( + pad_id=tokenizer.get_command('pad').Id) + data_len = len(sents_tgt) train_size = int(data_len * 0.8) train_src = sents_src[:train_size][:2000] @@ -139,15 +138,18 @@ def __len__(self): val_src = sents_src[train_size:] val_tgt = sents_tgt[train_size:] -train_dataset = BertSeq2seqDataset(train_src, - train_tgt, - tokenizer=tokenizer, - max_src_length=300, - max_tgt_length=200) -val_dataset = BertSeq2seqDataset(val_src, - val_tgt, - tokenizer=tokenizer, - max_src_length=300, - max_tgt_length=200) - -trainer.train(model, train_dataset=train_dataset, valid_dataset=val_dataset) +train_dataset = GLMSeq2seqDataset(train_src, + train_tgt, + tokenizer=tokenizer, + max_src_length=300, + max_tgt_length=200) +val_dataset = GLMSeq2seqDataset(val_src, + val_tgt, + tokenizer=tokenizer, + max_src_length=300, + max_tgt_length=200) + +trainer.train(model, + train_dataset=train_dataset, + valid_dataset=val_dataset, + collate_fn=my_collate_fn) diff --git a/examples/gpt2_text_writting/generate.py b/examples/gpt2_text_writting/generate.py index 3614617d..544d43fd 100644 --- a/examples/gpt2_text_writting/generate.py +++ b/examples/gpt2_text_writting/generate.py @@ -2,9 +2,7 @@ from flagai.model.predictor.predictor import Predictor if __name__ == '__main__': - loader = AutoLoader("seq2seq", - "GPT2-base-ch", - model_dir="./state_dict/") + loader = AutoLoader("seq2seq", "GPT2-base-ch", model_dir="./state_dict/") model = loader.get_model() tokenizer = loader.get_tokenizer() predictor = Predictor(model, tokenizer) diff --git a/examples/gpt2_title_generation/generate.py b/examples/gpt2_title_generation/generate.py index fdb00b34..1dc8eb2b 100755 --- a/examples/gpt2_title_generation/generate.py +++ b/examples/gpt2_title_generation/generate.py @@ -6,6 +6,7 @@ model_dir = "./state_dict/" +# Note "./checkpoints_seq2seq/{}/mp_rank_00_model_states.pt", {} is a directory in the checkpoints_seq2seq. model_save_path = "./checkpoints_seq2seq/1000/mp_rank_00_model_states.pt" maxlen = 256 auto_loader = AutoLoader( diff --git a/examples/bert_faq/1_construct_data.py b/examples/roberta_faq/1_construct_data.py similarity index 100% rename from examples/bert_faq/1_construct_data.py rename to examples/roberta_faq/1_construct_data.py diff --git a/examples/bert_faq/2_test_bert_faq.py b/examples/roberta_faq/2_test_bert_faq.py similarity index 100% rename from examples/bert_faq/2_test_bert_faq.py rename to examples/roberta_faq/2_test_bert_faq.py diff --git a/examples/bert_faq/README.md b/examples/roberta_faq/README.md similarity index 100% rename from examples/bert_faq/README.md rename to examples/roberta_faq/README.md diff --git a/examples/bert_faq/requirements.txt b/examples/roberta_faq/requirements.txt similarity index 100% rename from examples/bert_faq/requirements.txt rename to examples/roberta_faq/requirements.txt diff --git a/examples/bert_ner/data/china-people-daily-ner-corpus/example.dev b/examples/roberta_ner/data/china-people-daily-ner-corpus/example.dev similarity index 100% rename from examples/bert_ner/data/china-people-daily-ner-corpus/example.dev rename to examples/roberta_ner/data/china-people-daily-ner-corpus/example.dev diff --git a/examples/bert_ner/data/china-people-daily-ner-corpus/example.test b/examples/roberta_ner/data/china-people-daily-ner-corpus/example.test similarity index 100% rename from examples/bert_ner/data/china-people-daily-ner-corpus/example.test rename to examples/roberta_ner/data/china-people-daily-ner-corpus/example.test diff --git a/examples/bert_ner/data/china-people-daily-ner-corpus/example.train b/examples/roberta_ner/data/china-people-daily-ner-corpus/example.train similarity index 100% rename from examples/bert_ner/data/china-people-daily-ner-corpus/example.train rename to examples/roberta_ner/data/china-people-daily-ner-corpus/example.train diff --git a/examples/bert_ner/generate.py b/examples/roberta_ner/generate.py similarity index 95% rename from examples/bert_ner/generate.py rename to examples/roberta_ner/generate.py index fa9559f1..4e436279 100755 --- a/examples/bert_ner/generate.py +++ b/examples/roberta_ner/generate.py @@ -6,6 +6,8 @@ task_name = "sequence-labeling" model_dir = "./state_dict/" + +# Note "./checkpoints_ner/{}/mp_rank_00_model_states.pt", {} is a directory in the checkpoints_ner. model_save_path = "./checkpoints_ner/3913/mp_rank_00_model_states.pt" target = ["O", "B-LOC", "I-LOC", "B-ORG", "I-ORG", "B-PER", "I-PER"] diff --git a/examples/bert_ner/generate_crf.py b/examples/roberta_ner/generate_crf.py similarity index 95% rename from examples/bert_ner/generate_crf.py rename to examples/roberta_ner/generate_crf.py index 02aae943..e65af51b 100755 --- a/examples/bert_ner/generate_crf.py +++ b/examples/roberta_ner/generate_crf.py @@ -6,6 +6,8 @@ task_name = "sequence-labeling-crf" model_dir = "./state_dict/" # download dir + +# Note "./checkpoints_ner_crf/{}/mp_rank_00_model_states.pt", {} is a directory in the checkpoints_ner_crf. model_save_path = "./checkpoints_ner_crf/1304/mp_rank_00_model_states.pt" target = ["O", "B-LOC", "I-LOC", "B-ORG", "I-ORG", "B-PER", "I-PER"] diff --git a/examples/bert_ner/generate_global_pointer.py b/examples/roberta_ner/generate_global_pointer.py similarity index 95% rename from examples/bert_ner/generate_global_pointer.py rename to examples/roberta_ner/generate_global_pointer.py index 2fd97bdc..54c57530 100755 --- a/examples/bert_ner/generate_global_pointer.py +++ b/examples/roberta_ner/generate_global_pointer.py @@ -6,6 +6,8 @@ task_name = "sequence-labeling-gp" model_dir = "./state_dict/" # download dir + +# Note "./checkpoints_ner_gp/{}/mp_rank_00_model_states.pt", {} is a directory in the checkpoints_ner_gp. model_save_path = "./checkpoints_ner_gp/2608/mp_rank_00_model_states.pt" target = ["LOC", "ORG", "PER"] diff --git a/examples/bert_ner/train.py b/examples/roberta_ner/train.py similarity index 100% rename from examples/bert_ner/train.py rename to examples/roberta_ner/train.py index 066de984..bafec717 100644 --- a/examples/bert_ner/train.py +++ b/examples/roberta_ner/train.py @@ -10,7 +10,6 @@ device = torch.device("cuda" if torch.cuda.is_available() else "cpu") - train_path = "./data/china-people-daily-ner-corpus/example.train" valid_path = './data/china-people-daily-ner-corpus/example.dev' test_path = './data/china-people-daily-ner-corpus/example.test' @@ -66,6 +65,7 @@ def load_data(filename): D.append(d) return D + train_data = load_data(train_path) val_data = load_data(valid_path) test_data = load_data(test_path) diff --git a/examples/bert_ner/train_crf.py b/examples/roberta_ner/train_crf.py similarity index 100% rename from examples/bert_ner/train_crf.py rename to examples/roberta_ner/train_crf.py diff --git a/examples/bert_ner/train_global_pointer.py b/examples/roberta_ner/train_global_pointer.py similarity index 100% rename from examples/bert_ner/train_global_pointer.py rename to examples/roberta_ner/train_global_pointer.py diff --git a/examples/bert_semantic_matching/data/train.tsv b/examples/roberta_semantic_matching/data/train.tsv similarity index 100% rename from examples/bert_semantic_matching/data/train.tsv rename to examples/roberta_semantic_matching/data/train.tsv diff --git a/examples/bert_semantic_matching/generate.py b/examples/roberta_semantic_matching/generate.py similarity index 88% rename from examples/bert_semantic_matching/generate.py rename to examples/roberta_semantic_matching/generate.py index 9c1ff96a..6548d55e 100755 --- a/examples/bert_semantic_matching/generate.py +++ b/examples/roberta_semantic_matching/generate.py @@ -4,6 +4,7 @@ device = torch.device("cuda:1" if torch.cuda.is_available() else "cpu") +# Note "./checkpoints_semantic_matching/{}/mp_rank_00_model_states.pt", {} is a directory in the checkpoints_semantic_matching. model_save_path = "./checkpoints_semantic_matching/11250/mp_rank_00_model_states.pt" maxlen = 256 diff --git a/examples/bert_semantic_matching/train.py b/examples/roberta_semantic_matching/train.py similarity index 100% rename from examples/bert_semantic_matching/train.py rename to examples/roberta_semantic_matching/train.py diff --git a/examples/bert_title_generation/README.md b/examples/roberta_title_generation/README.md similarity index 100% rename from examples/bert_title_generation/README.md rename to examples/roberta_title_generation/README.md diff --git a/examples/bert_title_generation/data/train.src b/examples/roberta_title_generation/data/train.src similarity index 100% rename from examples/bert_title_generation/data/train.src rename to examples/roberta_title_generation/data/train.src diff --git a/examples/bert_title_generation/data/train.tgt b/examples/roberta_title_generation/data/train.tgt similarity index 100% rename from examples/bert_title_generation/data/train.tgt rename to examples/roberta_title_generation/data/train.tgt diff --git a/examples/bert_title_generation/deepspeed.json b/examples/roberta_title_generation/deepspeed.json similarity index 100% rename from examples/bert_title_generation/deepspeed.json rename to examples/roberta_title_generation/deepspeed.json diff --git a/examples/bert_title_generation/generate.py b/examples/roberta_title_generation/generate.py similarity index 95% rename from examples/bert_title_generation/generate.py rename to examples/roberta_title_generation/generate.py index a84fe473..0564a515 100755 --- a/examples/bert_title_generation/generate.py +++ b/examples/roberta_title_generation/generate.py @@ -6,6 +6,7 @@ model_dir = "./state_dict/" +# Note "./checkpoints_seq2seq/{}/mp_rank_00_model_states.pt", {} is a directory in the checkpoints_seq2seq. model_save_path = "./checkpoints_seq2seq/500/mp_rank_00_model_states.pt" maxlen = 256 auto_loader = AutoLoader( diff --git a/examples/bert_title_generation/hostfile b/examples/roberta_title_generation/hostfile similarity index 100% rename from examples/bert_title_generation/hostfile rename to examples/roberta_title_generation/hostfile diff --git a/examples/bert_title_generation/train.py b/examples/roberta_title_generation/train.py similarity index 100% rename from examples/bert_title_generation/train.py rename to examples/roberta_title_generation/train.py diff --git a/examples/bert_title_generation/train_multi_gpu.py b/examples/roberta_title_generation/train_multi_gpu.py similarity index 100% rename from examples/bert_title_generation/train_multi_gpu.py rename to examples/roberta_title_generation/train_multi_gpu.py diff --git a/examples/t5_flagai_11b/train_title_with_flagai_t5_11b.py b/examples/t5_flagai_11b/train_title_with_flagai_t5_11b.py new file mode 100644 index 00000000..197a3149 --- /dev/null +++ b/examples/t5_flagai_11b/train_title_with_flagai_t5_11b.py @@ -0,0 +1,148 @@ +import sys + +sys.path.append('/mnt/liuguang/FlagAI') +from flagai.trainer import Trainer +from flagai.model.t5_model import T5ForConditionalGeneration +from transformers import T5Tokenizer +from flagai.model.predictor.predictor import Predictor +from torch.utils.data import Dataset +import os +import torch + +cur_dir = os.path.dirname(os.path.abspath(__file__)) + +# train_path = cur_dir + "/data/news.tsv" +train_path = "/mnt/datasets/pens_dataset/train.tsv" + + +class MyTrainer(Trainer): + + def forward_step(self, data, model, mems): + + model_outputs = model(**data) + output = {} + output['loss'] = model_outputs.loss + output['logits'] = model_outputs.logits + output['hidden_states'] = model_outputs.decoder_hidden_states + return output + + +trainer = MyTrainer(env_type='deepspeed', + epochs=1, + batch_size=1, + eval_interval=100000, + log_interval=1, + experiment_name='t5-11b', + load_dir=None, + lr=1e-4, + fp16=True, + master_ip='127.0.0.1', + master_port=17755, + num_nodes=1, + num_gpus=1, + hostfile='./hostfile', + model_parallel_size=1, + deepspeed_config='./deepspeed.json', + training_script=__file__) + + +def read_file(): + src = [] + tgt = [] + + index = 0 + with open(train_path, 'r', encoding='utf-8') as f: + for line in f: + index += 1 + if index == 1: + continue + line = line.strip('\n').split('\t') + src_list = line[4].split(" ") + if len(src_list) > 510: + continue + + src.append(line[4]) + tgt.append(line[3]) + if index == 100000: + break + + return src, tgt + + +model_name = '/mnt/t5-11b' +tokenizer = T5Tokenizer.from_pretrained('t5-11b') + +model = T5ForConditionalGeneration.from_pretrain(download_path='/mnt', + model_name='t5-11b') + +print("loading model & tokenizer is done!") + +maxlen = 1024 + +predictor = Predictor(model, tokenizer) + + +class T5Seq2seqDataset(Dataset): + + def __init__(self, sents_src, sents_tgt, tokenizer, maxlen=512): + super(T5Seq2seqDataset, self).__init__() + self.sents_src = sents_src + self.sents_tgt = sents_tgt + self.tokenizer = tokenizer + self.maxlen = maxlen + + def __getitem__(self, i): + src = self.sents_src[i] + tgt = self.sents_tgt[i] + inputs = tokenizer(src) + with tokenizer.as_target_tokenizer(): + labels = tokenizer(tgt) + output = {} + output['input_ids'] = inputs.input_ids + output['target_ids'] = labels.input_ids + return output + + def __len__(self): + return len(self.sents_src) + + +# +def t5_seq2seq_collate_fn(batch): + + def padding(indice, max_length, pad_idx=0): + + pad_indice = [ + item + [pad_idx] * max(0, max_length - len(item)) + for item in indice + ] + return torch.tensor(pad_indice) + + token_ids_src = [data["input_ids"] for data in batch] + max_length_src = max([len(t) for t in token_ids_src]) + token_ids_tgt = [data["target_ids"] for data in batch] + max_length_tgt = max([len(t) for t in token_ids_tgt]) + + token_ids_padded = padding(token_ids_src, max_length_src) + target_ids_padded = padding(token_ids_tgt, max_length_tgt) + labels_ids = target_ids_padded.clone() + labels_ids[labels_ids == 0] = -100 + target_ids_padded = target_ids_padded[:, :-1].contiguous() + labels_ids = labels_ids[:, 1:].contiguous() + + return { + "input_ids": token_ids_padded, + "decoder_input_ids": target_ids_padded, + "labels": labels_ids + } + + +train_src, train_tgt = read_file() + +train_dataset = T5Seq2seqDataset(train_src, + train_tgt, + tokenizer=tokenizer, + maxlen=maxlen) + +trainer.train(model, + train_dataset=train_dataset, + collate_fn=t5_seq2seq_collate_fn) diff --git a/examples/t5_huggingface/run_train.sh b/examples/t5_huggingface/run_train.sh deleted file mode 100644 index 08015933..00000000 --- a/examples/t5_huggingface/run_train.sh +++ /dev/null @@ -1,10 +0,0 @@ -python -m launch --launcher distributed_deepspeed \ - --hostfile ./hostfile \ - --master_addr 172.31.255.4\ - --master_port 17885 \ - --gpus_per_node 2 \ - --num_nodes 1 \ - ./train_t5.py \ - --deepspeed_config ./deepspeed.json \ - --model_parallel_size 1 - diff --git a/examples/t5_huggingface/train_t5.py b/examples/t5_huggingface/train_t5_11b.py similarity index 78% rename from examples/t5_huggingface/train_t5.py rename to examples/t5_huggingface/train_t5_11b.py index 4b597be9..59e0479a 100644 --- a/examples/t5_huggingface/train_t5.py +++ b/examples/t5_huggingface/train_t5_11b.py @@ -16,21 +16,21 @@ def forward_step(self, data, model, mems): return output -trainer = MyTrainer( - env_type='pytorch', - epochs=1, - batch_size=4, - eval_interval=100000, - log_interval=10, - experiment_name='t5-3b', - pytorch_device='cpu', - load_dir=None, - lr=1e-4, - fp16=False) - -model_name = 't5-3b' +trainer = MyTrainer(env_type='pytorch', + epochs=1, + batch_size=4, + eval_interval=100000, + log_interval=10, + experiment_name='t5-11b', + pytorch_device='cpu', + load_dir=None, + lr=1e-4, + fp16=False) + +model_name = 't5-11b' tokenizer = T5Tokenizer.from_pretrained(model_name) model = T5ForConditionalGeneration.from_pretrained(model_name) +model.gradient_checkpointing = True print("loading model & tokenizer is done!") src_dir = './data/train.src' @@ -54,10 +54,10 @@ def read_file(): return src, tgt -class BertSeq2seqDataset(Dataset): +class T5Seq2seqDataset(Dataset): def __init__(self, sents_src, sents_tgt, tokenizer, maxlen=512): - super(BertSeq2seqDataset, self).__init__() + super(T5Seq2seqDataset, self).__init__() self.sents_src = sents_src self.sents_tgt = sents_tgt self.tokenizer = tokenizer @@ -110,14 +110,14 @@ def padding(indice, max_length, pad_idx=0): val_src = sents_src[train_size:] val_tgt = sents_tgt[train_size:] -train_dataset = BertSeq2seqDataset(train_src, - train_tgt, - tokenizer=tokenizer, - maxlen=maxlen) -val_dataset = BertSeq2seqDataset(val_src, - val_tgt, +train_dataset = T5Seq2seqDataset(train_src, + train_tgt, tokenizer=tokenizer, maxlen=maxlen) +val_dataset = T5Seq2seqDataset(val_src, + val_tgt, + tokenizer=tokenizer, + maxlen=maxlen) trainer.train(model, train_dataset=train_dataset, diff --git a/examples/t5_title_generation/generate.py b/examples/t5_title_generation/generate.py index 98c5daae..35790b8e 100644 --- a/examples/t5_title_generation/generate.py +++ b/examples/t5_title_generation/generate.py @@ -2,7 +2,7 @@ from flagai.model.predictor.predictor import Predictor if __name__ == '__main__': - loader = AutoLoader("seq2seq", "t5_base_ch", model_dir="./state_dict/") + loader = AutoLoader("seq2seq", "T5-base-ch", model_dir="./state_dict/") model = loader.get_model() tokenizer = loader.get_tokenizer() predictor = Predictor(model, tokenizer) diff --git a/examples/t5_title_generation/train.py b/examples/t5_title_generation/train.py index 0d40187d..b4c5e628 100644 --- a/examples/t5_title_generation/train.py +++ b/examples/t5_title_generation/train.py @@ -55,7 +55,7 @@ def read_file(): return src, tgt -class BertSeq2seqDataset(Dataset): +class T5Seq2seqDataset(Dataset): def __init__(self, sents_src, @@ -63,7 +63,7 @@ def __init__(self, tokenizer, max_src_length=300, max_tgt_length=200): - super(BertSeq2seqDataset, self).__init__() + super(T5Seq2seqDataset, self).__init__() self.sents_src = sents_src self.sents_tgt = sents_tgt self.tokenizer = tokenizer @@ -108,15 +108,15 @@ def __len__(self): val_src = sents_src[train_size:] val_tgt = sents_tgt[train_size:] -train_dataset = BertSeq2seqDataset(train_src, - train_tgt, - tokenizer=tokenizer, - max_src_length=300, - max_tgt_length=200) -val_dataset = BertSeq2seqDataset(val_src, - val_tgt, +train_dataset = T5Seq2seqDataset(train_src, + train_tgt, tokenizer=tokenizer, max_src_length=300, max_tgt_length=200) +val_dataset = T5Seq2seqDataset(val_src, + val_tgt, + tokenizer=tokenizer, + max_src_length=300, + max_tgt_length=200) trainer.train(model, train_dataset=train_dataset, valid_dataset=val_dataset) diff --git a/flagai/auto_model/auto_loader.py b/flagai/auto_model/auto_loader.py index 4d0141fb..089c1199 100644 --- a/flagai/auto_model/auto_loader.py +++ b/flagai/auto_model/auto_loader.py @@ -32,7 +32,8 @@ def __getattr__(self, name): "t5_seq2seq": ["flagai.model.t5_model", "T5Model"], "glm_seq2seq": ["flagai.model.glm_model", "GLMForSeq2Seq"], "glm_poetry": ["flagai.model.glm_model", "GLMForSeq2Seq"], - "glm_classification": ["flagai.model.glm_model", "GLMForSequenceClassification"] + "glm_classification": ["flagai.model.glm_model", "GLMForSequenceClassification"], + "glm_title-generation": ["flagai.model.glm_model", "GLMForSeq2Seq"], } MODEL_DICT = { diff --git a/flagai/data/dataset/data_collator/collate_fn.py b/flagai/data/dataset/data_collator/collate_fn.py index 8d79340b..db90b31e 100644 --- a/flagai/data/dataset/data_collator/collate_fn.py +++ b/flagai/data/dataset/data_collator/collate_fn.py @@ -6,6 +6,7 @@ import math from torch.utils.data.dataloader import default_collate from scipy.stats import poisson +from flagai.data.dataset.data_utils import build_sample from flagai.data.dataset.superglue.control import PVPS, SuperGlueProcessor @@ -110,23 +111,129 @@ class ConstructSeq2seqStrategy: def __init__(self, args, tokenizer, task_name): # pattern_id, seq_length, num_prompt_tokens, multi_token, segment_length, fast_decode, dataset_type, cloze_val=True self.tokenizer = tokenizer - self.cloze_eval = args.cloze_eval - self.processor = SuperGlueProcessor().get_processor(None, - task_name)(False) - - self.pvp = PVPS[task_name](args, - tokenizer, - self.processor.get_labels(), - args.seq_length, - pattern_id=args.pattern_id, - num_prompt_tokens=args.num_prompt_tokens, - is_multi_token=args.multi_token, - max_segment_length=args.segment_length, - fast_decode=args.fast_decode) + self.task_name = task_name + self.tokenizer = tokenizer self.args = args + def encode(self, example): + cls_id = self.tokenizer.get_command('ENC').Id + mask_token = 'sMASK' if self.args.task_mask else 'MASK' + mask_id = self.tokenizer.get_command(mask_token).Id + pad_id = self.tokenizer.get_command('pad').Id + sop_id = self.tokenizer.get_command('sop').Id + eop_id = self.tokenizer.get_command('eop').Id + if self.task_name in [ + "gigaword", "cnn_dm", "cnn_dm_original", "xsum", "lang8_hsk" + ]: + source_text, target_text = example.text_a, example.text_b + source_tokens = self.tokenizer.EncodeAsIds(" " + source_text) + prompt = [cls_id, mask_id + ] + self.tokenizer.EncodeAsIds(" Content:") + if len(source_tokens) > self.args.max_src_length - len(prompt): + source_tokens = source_tokens[:self.args.max_src_length - + len(prompt)] + source_tokens = prompt + source_tokens + elif self.task_name == "squad_generation": + source_text = example.text_a + target_text, answer = example.meta["question"], example.meta[ + "answer"] + source_tokens = self.tokenizer.EncodeAsIds(source_text.rstrip() + + " Question:") + answer_tokens = self.tokenizer.EncodeAsIds(" Answer: " + answer) + if len(source_tokens + ) > self.args.max_src_length - len(answer_tokens) - 2: + max_src_length = self.args.max_src_length - len(answer_tokens) - 2 + answer_pattern = self.tokenizer.EncodeAsIds(" " + answer) + + def sub_finder(mylist, pattern): + matches = [] + for i in range(len(mylist)): + if mylist[i] == pattern[0] and mylist[ + i:i + len(pattern)] == pattern: + matches.append(i) + return matches + + answer_indices = sub_finder(source_tokens, answer_pattern) + if len(answer_indices) == 0: + print(f"Answer {answer} not exists in the source text") + source_tokens = source_tokens[:max_src_length] + else: + start_index = max(answer_indices[0] - max_src_length // 2, + 0) + source_tokens = source_tokens[start_index:start_index + + max_src_length] + source_tokens = [cls_id] + source_tokens + [mask_id + ] + answer_tokens + elif self.task_name in ["cmrc"]: + mask_id = self.tokenizer.get_command('MASK').Id + source_text = example.text_a + target_text = example.meta["answer"].strip() + question = example.meta["question"].strip() + source_tokens = self.tokenizer.EncodeAsIds(source_text.rstrip()) + question_tokens = self.tokenizer.EncodeAsIds("问题：" + question + + "答案：") + max_src_length = self.args.max_src_length - len(question_tokens) - 2 + if max_src_length <= 0: + question_tokens = question_tokens[self.args.max_src_length // 4] + source_tokens = [cls_id] + question_tokens + [ + mask_id + ] + source_tokens[:max_src_length] + elif self.task_name in ["wsc"]: + mask_id = self.tokenizer.get_command('MASK').Id + source_text = example.text_a + target_text = example.meta["answer"].strip() + question = example.meta["question"].strip() + source_tokens = self.tokenizer.EncodeAsIds(source_text.rstrip()) + question_tokens = self.tokenizer.EncodeAsIds("what does " + + question + "mean: ") + max_src_length = self.args.max_src_length - len(question_tokens) - 2 + if max_src_length <= 0: + print(question) + question_tokens = question_tokens[self.args.max_src_length // 4] + source_tokens = [cls_id] + question_tokens + [ + mask_id + ] + source_tokens[:max_src_length] + else: + raise NotImplementedError + if len(source_tokens) < self.args.max_src_length: + source_tokens = source_tokens + [pad_id] * (self.args.max_src_length - + len(source_tokens)) + sep = len(source_tokens) + position_ids = list(range(len(source_tokens))) + block_position_ids = [0] * len(source_tokens) + mask_pos = source_tokens.index(mask_id) + + target_tokens = self.tokenizer.EncodeAsIds(" " + target_text) + target_tokens = target_tokens + [eop_id] + if len(target_tokens) > self.args.max_tgt_length: + target_tokens = target_tokens[:self.args.max_tgt_length] + loss_mask = [1] * len(target_tokens) + if len(target_tokens) < self.args.max_tgt_length: + loss_mask += [0] * (self.args.max_tgt_length - len(target_tokens)) + target_tokens += [pad_id] * (self.args.max_tgt_length - + len(target_tokens)) + tokens = source_tokens + [sop_id] + target_tokens[:-1] + loss_mask = [0] * len(source_tokens) + loss_mask + target_ids = [0] * len(source_tokens) + target_tokens + position_ids += [mask_pos] * len(target_tokens) + if self.args.no_block_position: + block_position_ids += [1] * len(target_tokens) + else: + block_position_ids += list(range(1, len(target_tokens) + 1)) + position_ids = [position_ids, block_position_ids] + sample = build_sample(ids=tokens, + positions=position_ids, + target=target_ids, + masks=sep, + loss_mask=loss_mask, + unique_id=example.guid) + return sample + def __call__(self, examples): samples = [] + for example in examples: + sample = self.encode(example) + samples.append(sample) return my_collate(samples) diff --git a/flagai/data/dataset/data_utils.py b/flagai/data/dataset/data_utils.py old mode 100755 new mode 100644 index 3b7fb125..d165dfc6 --- a/flagai/data/dataset/data_utils.py +++ b/flagai/data/dataset/data_utils.py @@ -278,6 +278,7 @@ def build_sample(ids, target=None, logit_mask=None, segment_ids=None, + loss_mask=None, prompt_ids=None, meta=None): """Convert to numpy and return a sample consumed by the batch producer.""" @@ -305,14 +306,17 @@ def build_sample(ids, if logit_mask is not None: logit_mask_np = np.array(logit_mask, dtype=np.int64) sample['logit_mask'] = logit_mask_np + if loss_mask is not None: + loss_mask_np = np.array(loss_mask, dtype=np.int64) + sample['loss_mask'] = loss_mask_np if segment_ids is not None: segment_ids = np.array(segment_ids, dtype=np.int64) sample['segment_id'] = segment_ids if meta is not None: sample['meta'] = meta - #if prompt_ids is not None: - # prompt_ids = np.array(prompt_ids, dtype=np.int64) - # sample['prompt_pos'] = prompt_ids + if prompt_ids and prompt_ids is not None: + prompt_ids = np.array(prompt_ids, dtype=np.int64) + sample['prompt_pos'] = prompt_ids if unique_id is not None: sample['uid'] = unique_id return sample diff --git a/flagai/data/dataset/language_model/dataset.py b/flagai/data/dataset/language_model/dataset.py old mode 100755 new mode 100644 diff --git a/flagai/data/dataset/seq2seq/dataset.py b/flagai/data/dataset/seq2seq/dataset.py old mode 100755 new mode 100644 index f1bca0c6..66eb3370 --- a/flagai/data/dataset/seq2seq/dataset.py +++ b/flagai/data/dataset/seq2seq/dataset.py @@ -415,131 +415,132 @@ def __len__(self): def __getitem__(self, idx): example = self.example_list[idx] - cls_id = self.tokenizer.get_command('ENC').Id - mask_token = 'sMASK' if self.task_mask else 'MASK' - mask_id = self.tokenizer.get_command(mask_token).Id - pad_id = self.tokenizer.get_command('pad').Id - sop_id = self.tokenizer.get_command('sop').Id - eop_id = self.tokenizer.get_command('eop').Id - if self.task_name in [ - "gigaword", "cnn_dm", "cnn_dm_original", "xsum", "lang8_hsk" - ]: - source_text, target_text = example.text_a, example.text_b - source_tokens = self.tokenizer.EncodeAsIds(" " + source_text) - prompt = [cls_id, mask_id - ] + self.tokenizer.EncodeAsIds(" Content:") - if len(source_tokens) > self.max_src_length - len(prompt): - source_tokens = source_tokens[:self.max_src_length - - len(prompt)] - source_tokens = prompt + source_tokens - elif self.task_name == "squad_generation": - source_text = example.text_a - target_text, answer = example.meta["question"], example.meta[ - "answer"] - source_tokens = self.tokenizer.EncodeAsIds(source_text.rstrip() + - " Question:") - answer_tokens = self.tokenizer.EncodeAsIds(" Answer: " + answer) - if len(source_tokens - ) > self.max_src_length - len(answer_tokens) - 2: - max_src_length = self.max_src_length - len(answer_tokens) - 2 - answer_pattern = self.tokenizer.EncodeAsIds(" " + answer) - - def sub_finder(mylist, pattern): - matches = [] - for i in range(len(mylist)): - if mylist[i] == pattern[0] and mylist[ - i:i + len(pattern)] == pattern: - matches.append(i) - return matches - - answer_indices = sub_finder(source_tokens, answer_pattern) - if len(answer_indices) == 0: - print(f"Answer {answer} not exists in the source text") - source_tokens = source_tokens[:max_src_length] - else: - start_index = max(answer_indices[0] - max_src_length // 2, - 0) - source_tokens = source_tokens[start_index:start_index + - max_src_length] - source_tokens = [cls_id] + source_tokens + [mask_id - ] + answer_tokens - elif self.task_name in ["cmrc"]: - mask_id = self.tokenizer.get_command('MASK').Id - source_text = example.text_a - target_text = example.meta["answer"].strip() - question = example.meta["question"].strip() - source_tokens = self.tokenizer.EncodeAsIds(source_text.rstrip()) - question_tokens = self.tokenizer.EncodeAsIds("问题：" + question + - "答案：") - max_src_length = self.max_src_length - len(question_tokens) - 2 - if max_src_length <= 0: - question_tokens = question_tokens[self.max_src_length // 4] - source_tokens = [cls_id] + question_tokens + [ - mask_id - ] + source_tokens[:max_src_length] - elif self.task_name in ["wsc"]: - mask_id = self.tokenizer.get_command('MASK').Id - source_text = example.text_a - target_text = example.meta["answer"].strip() - question = example.meta["question"].strip() - source_tokens = self.tokenizer.EncodeAsIds(source_text.rstrip()) - question_tokens = self.tokenizer.EncodeAsIds("what does " + - question + "mean: ") - max_src_length = self.max_src_length - len(question_tokens) - 2 - if max_src_length <= 0: - print(question) - question_tokens = question_tokens[self.max_src_length // 4] - source_tokens = [cls_id] + question_tokens + [ - mask_id - ] + source_tokens[:max_src_length] - else: - raise NotImplementedError - if len(source_tokens) < self.max_src_length: - source_tokens = source_tokens + [pad_id] * (self.max_src_length - - len(source_tokens)) - sep = len(source_tokens) - position_ids = list(range(len(source_tokens))) - block_position_ids = [0] * len(source_tokens) - mask_pos = source_tokens.index(mask_id) - if self.dataset_type == 'train' or self.dataset_type == "dev": - target_tokens = self.tokenizer.EncodeAsIds(" " + target_text) - target_tokens = target_tokens + [eop_id] - if len(target_tokens) > self.max_tgt_length: - target_tokens = target_tokens[:self.max_tgt_length] - loss_mask = [1] * len(target_tokens) - if len(target_tokens) < self.max_tgt_length: - loss_mask += [0] * (self.max_tgt_length - len(target_tokens)) - target_tokens += [pad_id] * (self.max_tgt_length - - len(target_tokens)) - tokens = source_tokens + [sop_id] + target_tokens[:-1] - loss_mask = [0] * len(source_tokens) + loss_mask - target_ids = [0] * len(source_tokens) + target_tokens - position_ids += [mask_pos] * len(target_tokens) - if self.no_block_position: - block_position_ids += [1] * len(target_tokens) - else: - block_position_ids += list(range(1, len(target_tokens) + 1)) - position_ids = [position_ids, block_position_ids] - sample = { - 'input_ids': np.array(tokens, dtype=np.int64), - 'target_ids': np.array(target_ids, dtype=np.int64), - 'attention_mask': np.array(sep, dtype=np.int64), - 'loss_mask': np.array(loss_mask, dtype=np.int64), - "position_ids": np.array(position_ids, dtype=np.int64), - "uid": example.guid - } - else: - tokens = source_tokens + [sop_id] - position_ids = position_ids + [mask_pos] - block_position_ids = block_position_ids + [1] - position_ids = [position_ids, block_position_ids] - sample = { - 'input_ids': np.array(tokens, dtype=np.int64), - 'attention_mask': np.array(sep, dtype=np.int64), - "position_ids": np.array(position_ids, dtype=np.int64), - "uid": example.guid - } - return sample + return example + # cls_id = self.tokenizer.get_command('ENC').Id + # mask_token = 'sMASK' if self.task_mask else 'MASK' + # mask_id = self.tokenizer.get_command(mask_token).Id + # pad_id = self.tokenizer.get_command('pad').Id + # sop_id = self.tokenizer.get_command('sop').Id + # eop_id = self.tokenizer.get_command('eop').Id + # if self.task_name in [ + # "gigaword", "cnn_dm", "cnn_dm_original", "xsum", "lang8_hsk" + # ]: + # source_text, target_text = example.text_a, example.text_b + # source_tokens = self.tokenizer.EncodeAsIds(" " + source_text) + # prompt = [cls_id, mask_id + # ] + self.tokenizer.EncodeAsIds(" Content:") + # if len(source_tokens) > self.max_src_length - len(prompt): + # source_tokens = source_tokens[:self.max_src_length - + # len(prompt)] + # source_tokens = prompt + source_tokens + # elif self.task_name == "squad_generation": + # source_text = example.text_a + # target_text, answer = example.meta["question"], example.meta[ + # "answer"] + # source_tokens = self.tokenizer.EncodeAsIds(source_text.rstrip() + + # " Question:") + # answer_tokens = self.tokenizer.EncodeAsIds(" Answer: " + answer) + # if len(source_tokens + # ) > self.max_src_length - len(answer_tokens) - 2: + # max_src_length = self.max_src_length - len(answer_tokens) - 2 + # answer_pattern = self.tokenizer.EncodeAsIds(" " + answer) + # + # def sub_finder(mylist, pattern): + # matches = [] + # for i in range(len(mylist)): + # if mylist[i] == pattern[0] and mylist[ + # i:i + len(pattern)] == pattern: + # matches.append(i) + # return matches + # + # answer_indices = sub_finder(source_tokens, answer_pattern) + # if len(answer_indices) == 0: + # print(f"Answer {answer} not exists in the source text") + # source_tokens = source_tokens[:max_src_length] + # else: + # start_index = max(answer_indices[0] - max_src_length // 2, + # 0) + # source_tokens = source_tokens[start_index:start_index + + # max_src_length] + # source_tokens = [cls_id] + source_tokens + [mask_id + # ] + answer_tokens + # elif self.task_name in ["cmrc"]: + # mask_id = self.tokenizer.get_command('MASK').Id + # source_text = example.text_a + # target_text = example.meta["answer"].strip() + # question = example.meta["question"].strip() + # source_tokens = self.tokenizer.EncodeAsIds(source_text.rstrip()) + # question_tokens = self.tokenizer.EncodeAsIds("问题：" + question + + # "答案：") + # max_src_length = self.max_src_length - len(question_tokens) - 2 + # if max_src_length <= 0: + # question_tokens = question_tokens[self.max_src_length // 4] + # source_tokens = [cls_id] + question_tokens + [ + # mask_id + # ] + source_tokens[:max_src_length] + # elif self.task_name in ["wsc"]: + # mask_id = self.tokenizer.get_command('MASK').Id + # source_text = example.text_a + # target_text = example.meta["answer"].strip() + # question = example.meta["question"].strip() + # source_tokens = self.tokenizer.EncodeAsIds(source_text.rstrip()) + # question_tokens = self.tokenizer.EncodeAsIds("what does " + + # question + "mean: ") + # max_src_length = self.max_src_length - len(question_tokens) - 2 + # if max_src_length <= 0: + # print(question) + # question_tokens = question_tokens[self.max_src_length // 4] + # source_tokens = [cls_id] + question_tokens + [ + # mask_id + # ] + source_tokens[:max_src_length] + # else: + # raise NotImplementedError + # if len(source_tokens) < self.max_src_length: + # source_tokens = source_tokens + [pad_id] * (self.max_src_length - + # len(source_tokens)) + # sep = len(source_tokens) + # position_ids = list(range(len(source_tokens))) + # block_position_ids = [0] * len(source_tokens) + # mask_pos = source_tokens.index(mask_id) + # if self.dataset_type == 'train' or self.dataset_type == "dev": + # target_tokens = self.tokenizer.EncodeAsIds(" " + target_text) + # target_tokens = target_tokens + [eop_id] + # if len(target_tokens) > self.max_tgt_length: + # target_tokens = target_tokens[:self.max_tgt_length] + # loss_mask = [1] * len(target_tokens) + # if len(target_tokens) < self.max_tgt_length: + # loss_mask += [0] * (self.max_tgt_length - len(target_tokens)) + # target_tokens += [pad_id] * (self.max_tgt_length - + # len(target_tokens)) + # tokens = source_tokens + [sop_id] + target_tokens[:-1] + # loss_mask = [0] * len(source_tokens) + loss_mask + # target_ids = [0] * len(source_tokens) + target_tokens + # position_ids += [mask_pos] * len(target_tokens) + # if self.no_block_position: + # block_position_ids += [1] * len(target_tokens) + # else: + # block_position_ids += list(range(1, len(target_tokens) + 1)) + # position_ids = [position_ids, block_position_ids] + # sample = { + # 'input_ids': np.array(tokens, dtype=np.int64), + # 'target_ids': np.array(target_ids, dtype=np.int64), + # 'attention_mask': np.array(sep, dtype=np.int64), + # 'loss_mask': np.array(loss_mask, dtype=np.int64), + # "position_ids": np.array(position_ids, dtype=np.int64), + # "uid": example.guid + # } + # else: + # tokens = source_tokens + [sop_id] + # position_ids = position_ids + [mask_pos] + # block_position_ids = block_position_ids + [1] + # position_ids = [position_ids, block_position_ids] + # sample = { + # 'input_ids': np.array(tokens, dtype=np.int64), + # 'attention_mask': np.array(sep, dtype=np.int64), + # "position_ids": np.array(position_ids, dtype=np.int64), + # "uid": example.guid + # } + # return sample class ExtractionDataset(torch.utils.data.Dataset): diff --git a/flagai/data/dataset/superglue/pvp.py b/flagai/data/dataset/superglue/pvp.py index 26fc3b3e..a5b0a04c 100644 --- a/flagai/data/dataset/superglue/pvp.py +++ b/flagai/data/dataset/superglue/pvp.py @@ -335,10 +335,6 @@ def encode_input(raw_parts): segment_ids=segment_id_list, prompt_ids=prompt_list, meta={}) - # print(sample['input_ids'].shape, sample['target_ids'].shape, sample['attention_mask'].shape, - # sample['logit_mask'].shape) - # print(sample['input_ids'], sample['target_ids'], sample['attention_mask'], - # sample['logit_mask']) return sample else: this_parts_a, this_parts_b = copy.deepcopy( diff --git a/flagai/data/tokenizer/glm_large_ch/glm_large_ch_tokenizer.py b/flagai/data/tokenizer/glm_large_ch/glm_large_ch_tokenizer.py index cde22a39..998fd20d 100644 --- a/flagai/data/tokenizer/glm_large_ch/glm_large_ch_tokenizer.py +++ b/flagai/data/tokenizer/glm_large_ch/glm_large_ch_tokenizer.py @@ -27,8 +27,8 @@ def __init__(self, vocab_path=None, add_block_symbols=True, add_task_mask=True, - add_decoder_mask=True, - fix_command_token=False): + add_decoder_mask=False, + fix_command_token=True): """ Args: add_block_symbols: (str): diff --git a/flagai/fp16/fp16.py b/flagai/fp16/fp16.py old mode 100755 new mode 100644 diff --git a/flagai/fp16/loss_scaler.py b/flagai/fp16/loss_scaler.py old mode 100755 new mode 100644 diff --git a/flagai/model/base_model.py b/flagai/model/base_model.py index 57eab7e6..56fe3ea7 100644 --- a/flagai/model/base_model.py +++ b/flagai/model/base_model.py @@ -1,3 +1,4 @@ +from sklearn.linear_model import HuberRegressor from torch.nn import Module import torch import json @@ -31,6 +32,8 @@ def init_from_json(cls, config_file='./config.json', **kwargs): args = json.load(js) for k in kwargs: args[k] = kwargs[k] + if 'checkpoint_activations' not in args: + args['checkpoint_activations'] = False return cls(args, **kwargs) @classmethod @@ -39,28 +42,51 @@ def from_pretrain(cls, model_name='RoBERT-base-ch', only_download_config=False, **kwargs): - model_id = _get_model_id(model_name) + model_id = None + try: + model_id = _get_model_id(model_name) + except: + print("Model hub is not reachable!") config_path = None download_path = os.path.join(download_path, model_name) checkpoint_path = os.path.join(download_path, "pytorch_model.bin") # prepare the download path # downloading the files model: Union[Module, None] - if model_id != "null": - _get_vocab_path(download_path, "vocab.txt", model_id) - if not only_download_config: + if model_id and model_id != "null": + if not os.path.exists(os.path.join(download_path, 'vocab.txt')): + _get_vocab_path(download_path, "vocab.txt", model_id) + if not only_download_config and not os.path.exists(os.path.join(download_path, 'config.json')): checkpoint_path = _get_checkpoint_path(download_path, 'pytorch_model.bin', model_id) config_path = os.path.join(download_path, "config.json") - if not os.path.exists(config_path) and model_id != "null": + if model_id and not os.path.exists(config_path) and model_id != "null": config_path = _get_config_path(download_path, 'config.json', model_id) if os.path.exists(config_path): model = cls.init_from_json(config_path, **kwargs) - if os.path.exists(checkpoint_path): - model.load_weights(checkpoint_path) + if os.getenv('ENV_TYPE')!='deepspeed+mpu': + if os.path.exists(checkpoint_path): + model.load_weights(checkpoint_path) + elif os.getenv('ENV_TYPE')=='deepspeed+mpu': + model_parallel_size = int(os.getenv("MODEL_PARALLEL_SIZE")) + if torch.distributed.is_initialized() and torch.distributed.get_rank() == 0: + # change the mp_size in rank 0 + print("preparing the model weights for model parallel size = {:02d}".format(model_parallel_size)) + from flagai.mp_tools import change_pytorch_model_mp_from_1_to_n, check_pytorch_model_mp_size + if model_parallel_size>1 and not check_pytorch_model_mp_size(download_path, model_parallel_size): + change_pytorch_model_mp_from_1_to_n(download_path, model_parallel_size) + if model_parallel_size>1: + from flagai.mpu import get_model_parallel_rank + model_parallel_rank = get_model_parallel_rank() + checkpoint_path = os.path.join(download_path, + "pytorch_model_{:02d}.bin".format(model_parallel_rank)) + if os.path.exists(checkpoint_path): + model.load_weights(checkpoint_path) + else: + model.load_weights(checkpoint_path) else: model = None return model diff --git a/flagai/model/bert_model.py b/flagai/model/bert_model.py old mode 100755 new mode 100644 index c8a08df3..3ab2974c --- a/flagai/model/bert_model.py +++ b/flagai/model/bert_model.py @@ -24,7 +24,13 @@ from flagai.model.layers.crf import CRFLayer from torch import nn from typing import List - +import os +if os.getenv('ENV_TYPE') == 'deepspeed+mpu': + from flagai.mpu.random import checkpoint +elif os.getenv('ENV_TYPE') == 'deepspeed': + from deepspeed.runtime.activation_checkpointing.checkpointing import checkpoint +else: + from torch.utils.checkpoint import checkpoint def init_bert_weights(module): """ Initialize the weights. @@ -41,11 +47,12 @@ def init_bert_weights(module): class BertStack(torch.nn.Module): - def __init__(self, num_hidden_layers, hidden_size, num_attention_heads, + def __init__(self, config,num_hidden_layers, hidden_size, num_attention_heads, attention_probs_dropout_prob, initializer_range, layernorm_epsilon, hidden_dropout_prob, intermediate_size, hidden_act): super(BertStack, self).__init__() + self.config = config self.layer = torch.nn.ModuleList([ BertBlock(hidden_size, num_attention_heads, attention_probs_dropout_prob, initializer_range, @@ -57,21 +64,20 @@ def __init__(self, num_hidden_layers, hidden_size, num_attention_heads, def forward(self, hidden_states, attention_mask, - output_all_encoded_layers=True, - checkpoint_fn=None): + output_all_encoded_layers=True + ): all_encoder_layers = [] for i, layer_module in enumerate(self.layer): - if checkpoint_fn is not None: - + if self.config['checkpoint_activations']: def create_custom_forward(module): def custom_forward(*inputs): return module(*inputs) return custom_forward - hidden_states = checkpoint_fn( + hidden_states = checkpoint( create_custom_forward(layer_module), hidden_states, attention_mask) else: @@ -80,8 +86,6 @@ def custom_forward(*inputs): if output_all_encoded_layers: all_encoder_layers.append(hidden_states) - if not output_all_encoded_layers or checkpoint_fn is not None: - all_encoder_layers.append(hidden_states) return all_encoder_layers @@ -89,7 +93,6 @@ class BertModel(BaseModel): def __init__(self, config, **kwargs): super(BertModel, self).__init__(config, **kwargs) - hidden_size = config["hidden_size"] intermediate_size = 4 * hidden_size self.hidden_size = hidden_size @@ -112,7 +115,7 @@ def __init__(self, config, **kwargs): self.type_vocab_size, self.layernorm_epsilon, self.hidden_dropout_prob) - self.encoder = BertStack( + self.encoder = BertStack(config, self.num_hidden_layers, hidden_size, self.num_attention_heads, self.attention_probs_dropout_prob, self.initializer_range, self.layernorm_epsilon, self.hidden_dropout_prob, @@ -126,7 +129,7 @@ def forward(self, token_type_ids=None, attention_mask=None, output_all_encoded_layers=True, - checkpoint_fn=None): + **kwargs): # if attention_mask is None: # attention_mask = torch.ones_like(input_ids) @@ -156,8 +159,7 @@ def forward(self, encoded_layers = self.encoder( embedding_output, extended_attention_mask, - output_all_encoded_layers=output_all_encoded_layers, - checkpoint_fn=checkpoint_fn) + output_all_encoded_layers=output_all_encoded_layers) sequence_representation = encoded_layers[-1] for p in self.pooler.parameters(): if p is None: @@ -338,7 +340,6 @@ def forward(self, **data): input_ids = data["input_ids"] token_type_ids = data["segment_ids"] labels = data.get("labels", None) - checkpoint_fn = data.get('checkpoint_fn', None) return_data = {} input_shape = input_ids.shape @@ -349,7 +350,7 @@ def forward(self, **data): token_type_ids, a_mask, True, - checkpoint_fn=checkpoint_fn) + ) sequence_out, decoder_out = self.cls(encoder_out[-1]) return_data["logits"] = decoder_out @@ -383,13 +384,12 @@ def forward(self, **data): labels = data.get("labels", None) attention_mask = data.get("attention_mask", None) - checkpoint_fn = data.get('checkpoint_fn', None) return_data = {} encoder_out, pooler_out = self.model(input_ids, token_type_ids, attention_mask, True, - checkpoint_fn=checkpoint_fn) + ) sequence_out, decoder_out = self.cls(encoder_out[-1]) return_data["logits"] = decoder_out @@ -429,13 +429,12 @@ def forward(self, **data): token_type_ids = data["segment_ids"] labels = data.get("labels", None) - checkpoint_fn = data.get('checkpoint_fn', None) return_data = {} sequence_out, pooler_out = self.model(input_ids, token_type_ids, None, - True, - checkpoint_fn=checkpoint_fn) + True + ) out = self.out_layer(pooler_out) return_data["logits"] = out return_data["hidden_states"] = sequence_out @@ -483,13 +482,12 @@ def forward(self, **data): labels = data.get("labels", None) padding_mask = (input_ids > 0).float() - checkpoint_fn = data.get('checkpoint_fn', None) return_data = {} sequence_out, pooler_out = self.model(input_ids, token_type_ids, None, - True, - checkpoint_fn=checkpoint_fn) + True + ) sequence_out = sequence_out[-1] logits = self.final_dense(sequence_out) @@ -533,12 +531,11 @@ def forward(self, **data): padding_mask = (input_ids > 0).float() return_data = {} - checkpoint_fn = data.get('checkpoint_fn', None) sequence_out, pooler_out = self.model(input_ids, token_type_ids, None, - True, - checkpoint_fn=checkpoint_fn) + True + ) sequence_out = sequence_out[-1] logits = self.final_dense(sequence_out) @@ -578,12 +575,11 @@ def forward(self, **data): token_type_ids = data.get("segment_ids", None) padding_mask = (input_ids > 0).float() labels = data.get("labels", None) - checkpoint_fn = data.get('checkpoint_fn', None) sequence_out, pooler_out = self.model(input_ids, token_type_ids, None, - True, - checkpoint_fn=checkpoint_fn) + True + ) sequence_out = sequence_out[-1] gp_out = self.gp(sequence_out, padding_mask) @@ -612,12 +608,11 @@ def forward(self, **data): token_type_ids = data["segment_ids"] return_data = {} - checkpoint_fn = data.get('checkpoint_fn', None) sequence_out, pooler_out = self.model(input_ids, token_type_ids, None, - True, - checkpoint_fn=checkpoint_fn) + True + ) out = sequence_out[-1] return_data["logits"] = out return_data["hidden_states"] = sequence_out diff --git a/flagai/model/glm_model.py b/flagai/model/glm_model.py old mode 100755 new mode 100644 index 9884fb81..4e2749e2 --- a/flagai/model/glm_model.py +++ b/flagai/model/glm_model.py @@ -22,13 +22,18 @@ from flagai.model.layers.embeddings import VocabParallelEmbedding from flagai.model.base_model import BaseModel from flagai.model.layers.embeddings import PositionalEmbedding -from flagai.mpu.random import checkpoint +from flagai.model.prompt import PromptSpell from flagai.model.utils import normal_init_method from torch.nn import LayerNorm print_rank_0 = print if os.getenv('ENV_TYPE') == 'deepspeed+mpu': from flagai.mpu import copy_to_model_parallel_region + from flagai.mpu.random import checkpoint +elif os.getenv('ENV_TYPE') == 'deepspeed': + from deepspeed.runtime.activation_checkpointing.checkpointing import checkpoint +else: + from torch.utils.checkpoint import checkpoint class GLMStack(torch.nn.Module): @@ -67,6 +72,7 @@ class GLMStack(torch.nn.Module): """ def __init__( self, + config, num_layers, hidden_size, num_attention_heads, @@ -87,6 +93,7 @@ def __init__( attention_scale=1.0, ): super(GLMStack, self).__init__() + self.config = config self.hidden_size = hidden_size # Store activation checkpoiting flag. self.checkpoint_activations = checkpoint_activations @@ -160,15 +167,6 @@ def get_layer(): # Final layer norm before output. self.final_layernorm = LayerNorm(hidden_size, eps=layernorm_epsilon) - # - try: - import deepspeed - if deepspeed.checkpointing.is_configured(): - global get_cuda_rng_tracker, checkpoint - get_cuda_rng_tracker = deepspeed.checkpointing.get_cuda_rng_tracker - checkpoint = deepspeed.checkpointing.checkpoint - except: - from torch.utils.checkpoint import checkpoint def forward(self, hidden_states, @@ -177,11 +175,10 @@ def forward(self, memory_states=None, encoder_states=None, return_memory=False, - detach_memory=True, - checkpoint=None): + detach_memory=True): batch_size, query_length = hidden_states.size()[:2] - memory_length = 0 - memory_states[0].size(1) if memory_states and memory_states[0] else 0 + memory_length = memory_states[0].size(1) if memory_states else 0 + key_length = query_length + memory_length # attention mask is the beginning postion of B region, \in [0, query_len) is_scalar = torch.numel(attention_mask) == 1 @@ -251,34 +248,48 @@ def check_detach(_hidden_states): else: mem_layers = [] - for i, layer in enumerate(self.layers): - args = [hidden_states, attention_mask] if not self.use_decoder_layer else \ - [hidden_states, encoder_states, attention_mask] - - def create_custom_forward(module): - def custom_forward(*inputs): - # None for past_key_value - return module(*inputs) - - return custom_forward - - if self.relative_encoding: - args += [position_embeddings, self.r_w_bias, self.r_r_bias] - mem_i = memory_states[i] if (memory_states - and len(memory_states) > i) else None - - if self.checkpoint_activations and checkpoint is not None: - - hidden_states = checkpoint( - create_custom_forward(layer), - hidden_states, - mem=mem_i, - ) - - else: + def custom(start, end): + def custom_forward(*inputs): + layers_ = self.layers[start:end] + x_, inputs = inputs[0], inputs[1:] + if self.relative_encoding: + inputs, mems_ = inputs[:4], inputs[4:] + else: + inputs, mems_ = inputs[:1], inputs[1:] + for i, layer in enumerate(layers_): + mem_i_ = mems_[i] if mems_ else None + x_ = layer(x_, *inputs, mem=mem_i_) + if self.max_memory_length > 0 or return_memory: + mem_layers.append(check_detach(x_)) + return x_ + + return custom_forward + + if self.config['checkpoint_activations']: + l = 0 + num_layers = len(self.layers) + chunk_length = self.checkpoint_num_layers + while l < num_layers: + args = [hidden_states, attention_mask] if not self.use_decoder_layer else [hidden_states, + encoder_states, + attention_mask] + if self.relative_encoding: + args += [position_embeddings, self.r_w_bias, self.r_r_bias] + if memory_states: + args += memory_states[l: l + chunk_length] + hidden_states = checkpoint(custom(l, l + chunk_length), *args) + l += chunk_length + else: + for i, layer in enumerate(self.layers): + args = [hidden_states, attention_mask] if not self.use_decoder_layer else [hidden_states, + encoder_states, + attention_mask] + if self.relative_encoding: + args += [position_embeddings, self.r_w_bias, self.r_r_bias] + mem_i = memory_states[i] if memory_states else None hidden_states = layer(*args, mem=mem_i) - if self.max_memory_length > 0 or return_memory: - mem_layers.append(check_detach(hidden_states)) + if self.max_memory_length > 0 or return_memory: + mem_layers.append(check_detach(hidden_states)) # Final layer norm. output = self.final_layernorm(hidden_states) @@ -325,6 +336,7 @@ def __init__(self, config, **kwargs): attention_dropout_prob = config["attention_dropout_prob"] output_dropout_prob = config["output_dropout_prob"] max_sequence_length = config["max_sequence_length"] + # max_sequence_length = 512 max_memory_length = config["max_memory_length"] checkpoint_activations = config["checkpoint_activations"] checkpoint_num_layers = config["checkpoint_num_layers"] @@ -332,7 +344,10 @@ def __init__(self, config, **kwargs): relative_encoding = config["relative_encoding"] block_position_encoding = config["block_position_encoding"] output_predict = config["output_predict"] + spell_length = config.get("spell_length", None) + spell_func = config["spell_func"] attention_scale = config["attention_scale"] + tune_prefix_layers = config.get("tune_prefix_layers", None) self.parallel_output = parallel_output self.output_predict = output_predict @@ -346,6 +361,7 @@ def __init__(self, config, **kwargs): # Transformer self.transformer = GLMStack( + config, num_layers, hidden_size, num_attention_heads, @@ -360,6 +376,11 @@ def __init__(self, config, **kwargs): relative_encoding=relative_encoding, block_position_encoding=block_position_encoding) + if spell_length is not None: + self.prompt_spell = PromptSpell(spell_length, self.hidden_size, spell_func) + if tune_prefix_layers != None: + self.freeze_transformer(tune_prefix_layers=tune_prefix_layers) + def freeze_transformer(self, tune_prefix_layers=None): log_str = "Freeze transformer" self.word_embeddings.requires_grad_(False) @@ -902,7 +923,6 @@ def forward(self, attention_mask, prompt_pos=prompt_pos) outputs, mems = model_out['logits'], model_out['hidden_states'] - vocab_size = outputs.size()[-1] target_ids = target_ids.view(-1) loss_mask = loss_mask.view(-1).float() diff --git a/flagai/model/gpt2_model.py b/flagai/model/gpt2_model.py index 9d2825e7..3bb3bd83 100644 --- a/flagai/model/gpt2_model.py +++ b/flagai/model/gpt2_model.py @@ -11,7 +11,13 @@ from flagai.mpu import gather_from_model_parallel_region from flagai.mpu import get_cuda_rng_tracker from flagai.mpu.utils import divide - +if os.getenv('ENV_TYPE') == 'deepspeed+mpu': + from flagai.mpu import copy_to_model_parallel_region + from flagai.mpu.random import checkpoint +elif os.getenv('ENV_TYPE') == 'deepspeed': + from deepspeed.runtime.activation_checkpointing.checkpointing import checkpoint +else: + from torch.utils.checkpoint import checkpoint class GPT2Stack(nn.Module): def __init__(self, config): @@ -83,17 +89,31 @@ def forward( all_self_attentions = () if output_attentions else None all_hidden_states = () if output_hidden_states else None for i, block in enumerate(self.h): - if output_hidden_states: all_hidden_states = all_hidden_states + (hidden_states, ) + if self.config['checkpoint_activations']: + def create_custom_forward(module): + def custom_forward(*inputs): + return module(*inputs) + return custom_forward + + outputs = checkpoint( + create_custom_forward(block), + hidden_states, + attention_mask, + None, + use_cache, + output_attentions, + ) + else: - outputs = block( - hidden_states, - attention_mask=attention_mask, - head_mask=None, - use_cache=use_cache, - output_attentions=output_attentions, - ) + outputs = block( + hidden_states, + attention_mask=attention_mask, + head_mask=None, + use_cache=use_cache, + output_attentions=output_attentions, + ) hidden_states = outputs[0] if use_cache is True: diff --git a/flagai/model/layers/attentions.py b/flagai/model/layers/attentions.py old mode 100755 new mode 100644 diff --git a/flagai/model/layers/embeddings.py b/flagai/model/layers/embeddings.py old mode 100755 new mode 100644 diff --git a/flagai/model/layers/layer_norm.py b/flagai/model/layers/layer_norm.py old mode 100755 new mode 100644 diff --git a/flagai/model/predictor/predictor.py b/flagai/model/predictor/predictor.py index 8ca1c826..8f4d2151 100644 --- a/flagai/model/predictor/predictor.py +++ b/flagai/model/predictor/predictor.py @@ -32,6 +32,12 @@ def __init__(self, """ self.tokenizer = tokenizer + if getattr(self.tokenizer, "token_end_id", None) is None: + setattr(self.tokenizer, "token_end_id", 1) + + if getattr(self.tokenizer, "token_start_id", None) is None: + setattr(self.tokenizer, "token_start_id", 0) + self.model = model self.model.eval() self.class_name = type(model).__name__ diff --git a/flagai/model/predictor/utils.py b/flagai/model/predictor/utils.py index 3da1e506..6f9cddae 100644 --- a/flagai/model/predictor/utils.py +++ b/flagai/model/predictor/utils.py @@ -4,7 +4,7 @@ import torch import numpy as np import torch.nn.functional as F - +import time join = os.path.join @@ -568,57 +568,65 @@ def gpt_random_sample(model, tokenizer, text, input_max_length, out_max_length, def glm_random_sample(model, tokenizer, text, out_max_length, top_k, top_p, repetition_penalty, temperature, device): - #tokenizer:GLMChineseSPTokenizer - #model input: - data = tokenizer.encode_plus(text) - input_ids = torch.tensor([data['input_ids']], - device=device, - dtype=torch.long) - position_ids = torch.tensor([data['position_ids']], - device=device, - dtype=torch.long) - attention_mask = torch.tensor(data['attention_mask'], - device=device, - dtype=torch.long) - #choosabel processor to overlay: - lp = [ - TemperatureLogitsProcessor(temperature=temperature), - TopPLogitsProcessor(top_p=top_p), - RepetitionPenaltyLogitsProcessor(penalty=repetition_penalty), - TopKLogitsProcessor(top_k=top_k) - ] - list_processor = ListProcessor(lp) - - output_ids = [] - end_id = tokenizer.command_name_map['eop'].Id - with torch.no_grad(): - for step in range(out_max_length): - scores = model( - input_ids=input_ids, #GLMModel - position_ids=position_ids, - attention_mask=attention_mask)[ - "logits"] #[1,max_src_length,vocab_size] - logit_score = torch.log_softmax(scores[:, -1], - dim=-1) #[1,vocab_size] - logit_score[:, - tokenizer.CommandTokenIds( - exception=["eop", "gMASK"])] = -float('Inf') - - filtered_logits = list_processor(input_ids, logit_score) - prob = F.softmax(filtered_logits, dim=-1) - next_token = torch.multinomial(prob, num_samples=1) - if end_id == next_token.item(): - break - output_ids.append(next_token.item()) - input_ids = torch.cat((input_ids, next_token.long()), dim=1) - new_position_ids = torch.tensor([[[position_ids.size()[2]], [0]]], - device=device, - dtype=torch.long) - position_ids = torch.cat((position_ids, new_position_ids), - dim=2) #shape:[1,2,inputsize] - if not output_ids: - print("None") - return tokenizer.DecodeIds(output_ids) + if 'MASK]' in text: + return glm_generate_sample( model, tokenizer, text, + out_seq_length=out_max_length, top_k=top_k, + temperature=temperature) + + + + else: + #tokenizer:GLMChineseSPTokenizer + #model input: + data = tokenizer.encode_plus(text) + input_ids = torch.tensor([data['input_ids']], + device=device, + dtype=torch.long) + position_ids = torch.tensor([data['position_ids']], + device=device, + dtype=torch.long) + attention_mask = torch.tensor(data['attention_mask'], + device=device, + dtype=torch.long) + #choosabel processor to overlay: + lp = [ + TemperatureLogitsProcessor(temperature=temperature), + TopPLogitsProcessor(top_p=top_p), + RepetitionPenaltyLogitsProcessor(penalty=repetition_penalty), + TopKLogitsProcessor(top_k=top_k) + ] + list_processor = ListProcessor(lp) + + output_ids = [] + end_id = tokenizer.command_name_map['eop'].Id + with torch.no_grad(): + for step in range(out_max_length): + scores = model( + input_ids=input_ids, #GLMModel + position_ids=position_ids, + attention_mask=attention_mask)[ + "logits"] #[1,max_src_length,vocab_size] + logit_score = torch.log_softmax(scores[:, -1], + dim=-1) #[1,vocab_size] + logit_score[:, + tokenizer.CommandTokenIds( + exception=["eop", "gMASK"])] = -float('Inf') + + filtered_logits = list_processor(input_ids, logit_score) + prob = F.softmax(filtered_logits, dim=-1) + next_token = torch.multinomial(prob, num_samples=1) + if end_id == next_token.item(): + break + output_ids.append(next_token.item()) + input_ids = torch.cat((input_ids, next_token.long()), dim=1) + new_position_ids = torch.tensor([[[position_ids.size()[2]], [0]]], + device=device, + dtype=torch.long) + position_ids = torch.cat((position_ids, new_position_ids), + dim=2) #shape:[1,2,inputsize] + if not output_ids: + print("None") + return tokenizer.DecodeIds(output_ids) def glm_beamsearch(model, tokenizer, text, out_max_length, beam_size): # @@ -697,9 +705,9 @@ def t5_predict_generate(model, with torch.no_grad(): device = next(model.parameters()).device - decoder_input_ids = torch.tensor(decoder_input_ids, device=device) + decoder_input_ids = torch.tensor(decoder_input_ids, device=device, dtype=torch.long) if input_ids is not None: - input_ids = torch.tensor(input_ids, device=device) + input_ids = torch.tensor(input_ids, device=device, dtype=torch.long) if input_ids.ndim == 1: input_ids = input_ids.view(1, -1) @@ -788,6 +796,92 @@ def t5_beam_search(model, return output_ids[output_scores.argmax()] +def glm_sample_sequence(model, tokenizer, context_tokens, context_length, + mems=None, end_tokens=None,out_seq_length=512,temperature=0.9, + top_k=40): + tokens = context_tokens.new_full((1, 1), tokenizer.get_command('sop').Id) + counter = 0 + if mems is None: + mems = [] + + last_beam_num = 1 + + while counter < out_seq_length: + position_ids = context_tokens.new_ones(last_beam_num, 2, 1) + position_ids[:, 0] = context_length + position_ids[:, 1] = counter + 1 + attention_mask = context_tokens.new_zeros([1], device=context_tokens.device, dtype=torch.long) + last_token = tokens[:, -1:] + output_ = model(last_token, position_ids, attention_mask,mems=mems, return_memory=True) + mems = output_['hidden_states'] + next_token_logits = output_['logits'] + next_token_logits = next_token_logits[:, -1] + next_token_logits /= temperature + indices_to_remove = next_token_logits < torch.topk(next_token_logits, top_k)[0][..., -1, None] + next_token_logits[indices_to_remove] = -float('Inf') + log_probs = F.softmax(next_token_logits, dim=-1) + prev = torch.multinomial(log_probs, num_samples=1)[0] + is_end = prev.item() in end_tokens + if is_end: + break + prev = prev.view(1, 1) + tokens = prev if tokens is None else torch.cat((tokens, prev), dim=1) + counter += 1 + return torch.cat((context_tokens, tokens), dim=1), mems + + + +def glm_generate_sample(model, tokenizer,text, top_k=40,seq_length=512,out_seq_length=512, + eod_token=50000,temperature=0.9, ): + device=torch.cuda.current_device() + model.eval() + + generation_mask = '[gMASK]' + if 'MASK]' not in text: + text += ' ' + generation_mask + context_tokens = tokenizer.EncodeAsIds(text) + context_tokens = [tokenizer.get_command('ENC').Id] + context_tokens + if not text.endswith('[gMASK]'): + context_tokens = context_tokens + [tokenizer.get_command('eos').Id] + context_length = len(context_tokens) + context_length_tensor = torch.cuda.LongTensor([context_length]) + context_length = context_length_tensor[0].item() + context_tokens_tensor = torch.cuda.LongTensor(context_tokens) + text = tokenizer.DecodeIds(context_tokens_tensor.tolist()) + + start_time = time.time() + mems = [] + tokens = context_tokens_tensor + tokens = tokens.view(1, -1).contiguous() + tokens = tokens.to(device) + attention_mask = torch.tensor([tokens.size(1)], device=device, dtype=torch.long) + position_ids = torch.arange(tokens.size(1), device=device, dtype=torch.long) + block_position_ids = torch.zeros(tokens.size(1), device=device, dtype=torch.long) + position_ids = torch.stack((position_ids, block_position_ids), dim=0) + position_ids = position_ids.unsqueeze(0) + mask_tokens = ['MASK', 'sMASK', 'gMASK'] + mask_tokens = [tokenizer.get_command(token).Id for token in mask_tokens] + end_tokens = [tokenizer.get_command('eop').Id, eod_token] + mask_positions = [] + for token in mask_tokens: + mask_positions += (context_tokens_tensor == token).nonzero(as_tuple=True)[0].tolist() + mask_positions.sort() + output_ = model(tokens, position_ids, attention_mask, return_memory=True) + mems=output_['hidden_states'] + for mask_position in mask_positions: + position = mask_position + tokens, mems = glm_sample_sequence(model, tokenizer, tokens, position, + mems=mems, end_tokens=end_tokens, + out_seq_length=out_seq_length,temperature=temperature, top_k=top_k) + output_tokens_list = tokens.view(-1).contiguous() + + decode_tokens = tokenizer.DecodeIds(output_tokens_list.tolist()) + + + return decode_tokens + + + def gpt_beam_search(model, token_ids, tokenizer, diff --git a/flagai/model/prompt.py b/flagai/model/prompt.py new file mode 100644 index 00000000..893432f3 --- /dev/null +++ b/flagai/model/prompt.py @@ -0,0 +1,50 @@ +import random +import torch + + +class PromptSpell(torch.nn.Module): + def __init__(self, spell_length, hidden_size, spell_func): + super(PromptSpell, self).__init__() + self.spell_length = spell_length + self.hidden_size = hidden_size + self.spell_embeddings = torch.nn.Embedding(self.spell_length, self.hidden_size) + self.spell_func = spell_func + if self.spell_func == "lstm": + self.lstm_head = torch.nn.LSTM(input_size=self.hidden_size, + hidden_size=self.hidden_size, + num_layers=2, + # dropout=self.lstm_dropout, + bidirectional=True, + batch_first=True) # .to(torch.device("cuda")) + self.mlp_head = torch.nn.Sequential(torch.nn.Linear(2 * self.hidden_size, self.hidden_size), + torch.nn.ReLU(), + torch.nn.Linear(self.hidden_size, self.hidden_size)) + elif self.spell_func == "mlp": + self.mlp_head = torch.nn.Sequential(torch.nn.Linear(self.hidden_size, self.hidden_size), + torch.nn.ReLU(), + torch.nn.Linear(self.hidden_size, self.hidden_size)) + elif self.spell_func != "none": + raise NotImplementedError("Prompt function " + self.spell_func) + + def init_embedding(self, word_embeddings=None, task_tokens=None): + num_words = 5000 + with torch.no_grad(): + for i in range(self.spell_length): + rand_token = random.randrange(num_words) + if task_tokens is None: + target_embedding = word_embeddings[rand_token] + else: + word_embedding = word_embeddings[rand_token] + task_token = random.choice(task_tokens) + task_embedding = word_embeddings[task_token] + ratio = random.random() + target_embedding = word_embedding * ratio + task_embedding * (1 - ratio) + self.spell_embeddings.weight.data[i] = target_embedding + + def forward(self): + prompt_embeds = self.spell_embeddings.weight.unsqueeze(0) + if self.spell_func == "lstm": + prompt_embeds = self.lstm_head(prompt_embeds)[0] + if self.spell_func == "lstm" or self.spell_func == "mlp": + prompt_embeds = self.mlp_head(prompt_embeds) + return prompt_embeds diff --git a/flagai/model/t5_model.py b/flagai/model/t5_model.py index d4a18949..f6f9e237 100644 --- a/flagai/model/t5_model.py +++ b/flagai/model/t5_model.py @@ -17,6 +17,7 @@ import copy import warnings import torch +import os from torch import nn from torch.nn import CrossEntropyLoss from torch import Tensor, device @@ -27,7 +28,13 @@ from flagai.model.layers.attentions import T5Attention from flagai.model.layers.feedforward import T5DenseReluDense, T5DenseGatedGeluDense from flagai.data.tokenizer.t5.t5_tokenizer import T5JiebaTokenizer - +if os.getenv('ENV_TYPE') == 'deepspeed+mpu': + from flagai.mpu import copy_to_model_parallel_region + from flagai.mpu.random import checkpoint +elif os.getenv('ENV_TYPE') == 'deepspeed': + from deepspeed.runtime.activation_checkpointing.checkpointing import checkpoint +else: + from torch.utils.checkpoint import checkpoint # Warning message for FutureWarning: head_mask was separated into two input args - head_mask, decoder_head_mask __HEAD_MASK_WARNING_MSG = """ The input argument `head_mask` was split into two arguments `head_mask` and `decoder_head_mask`. Currently, @@ -43,7 +50,8 @@ class T5PreTrainedModel(BaseModel): models. """ def __init__(self, config, **kwargs): - super(T5PreTrainedModel, self).__init__(config, **kwargs) + super().__init__(config, **kwargs) + # super(T5PreTrainedModel, self).__init__(config, **kwargs) def init_weights(self): for module in self.modules(): @@ -51,7 +59,7 @@ def init_weights(self): def _init_weights(self, module): """ Initialize the weights """ - factor = self.config.initializer_factor # Used for testing weights initialization + factor = self.config['initializer_factor'] # Used for testing weights initialization if isinstance(module, T5LayerNorm): module.weight.data.fill_(factor * 1.0) @@ -65,36 +73,36 @@ def _init_weights(self, module): # and https://github.com/tensorflow/mesh/blob/fa19d69eafc9a482aff0b59ddd96b025c0cb207d/mesh_tensorflow/layers.py#L89 module.wi.weight.data.normal_(mean=0.0, std=factor * - ((self.config.d_model)**-0.5)) + ((self.config['d_model'])**-0.5)) if hasattr(module.wi, "bias") and module.wi.bias is not None: module.wi.bias.data.zero_() module.wo.weight.data.normal_(mean=0.0, std=factor * - ((self.config.d_ff)**-0.5)) + ((self.config['d_ff'])**-0.5)) if hasattr(module.wo, "bias") and module.wo.bias is not None: module.wo.bias.data.zero_() elif isinstance(module, T5DenseGatedGeluDense): module.wi_0.weight.data.normal_(mean=0.0, std=factor * - ((self.config.d_model)**-0.5)) + ((self.config['d_model'])**-0.5)) if hasattr(module.wi_0, "bias") and module.wi_0.bias is not None: module.wi_0.bias.data.zero_() module.wi_1.weight.data.normal_(mean=0.0, std=factor * - ((self.config.d_model)**-0.5)) + ((self.config['d_model'])**-0.5)) if hasattr(module.wi_1, "bias") and module.wi_1.bias is not None: module.wi_1.bias.data.zero_() module.wo.weight.data.normal_(mean=0.0, std=factor * - ((self.config.d_ff)**-0.5)) + ((self.config['d_ff'])**-0.5)) if hasattr(module.wo, "bias") and module.wo.bias is not None: module.wo.bias.data.zero_() elif isinstance(module, T5Attention): # Mesh TensorFlow attention initialization to avoid scaling before softmax # See https://github.com/tensorflow/mesh/blob/fa19d69eafc9a482aff0b59ddd96b025c0cb207d/mesh_tensorflow/transformer/attention.py#L136 - d_model = self.config.d_model - key_value_proj_dim = self.config.d_kv - n_heads = self.config.num_heads + d_model = self.config['d_model'] + key_value_proj_dim = self.config['d_kv'] + n_heads = self.config['num_heads'] module.q.weight.data.normal_( mean=0.0, std=factor * ((d_model * key_value_proj_dim)**-0.5)) module.k.weight.data.normal_(mean=0.0, @@ -108,8 +116,8 @@ def _init_weights(self, module): mean=0.0, std=factor * ((d_model)**-0.5)) def _shift_right(self, input_ids): - decoder_start_token_id = self.config.decoder_start_token_id - pad_token_id = self.config.pad_token_id + decoder_start_token_id = self.config['decoder_start_token_id'] + pad_token_id = self.config['pad_token_id'] assert ( decoder_start_token_id is not None @@ -169,7 +177,6 @@ def forward( output_attentions=None, output_hidden_states=None, return_dict=True, - checkpoint=None, ): if input_ids is not None and inputs_embeds is not None: @@ -248,7 +255,7 @@ def forward( if output_hidden_states: all_hidden_states = all_hidden_states + (hidden_states, ) - if checkpoint is not None: + if self.config['checkpoint_activations']: def create_custom_forward(module): def custom_forward(*inputs): @@ -470,7 +477,7 @@ def invert_attention_mask(self, encoder_attention_mask: Tensor) -> Tensor: return encoder_extended_attention_mask -class T5Model(BaseModel): +class T5Model(T5PreTrainedModel): _keys_to_ignore_on_load_missing = [ r"encoder\.embed_tokens\.weight", r"decoder\.embed_tokens\.weight", @@ -721,26 +728,27 @@ class T5ForConditionalGeneration(T5PreTrainedModel): r"decoder\.block\.0\.layer\.1\.EncDecAttention\.relative_attention_bias\.weight", ] - def __init__(self, config): - super().__init__(config) + def __init__(self, config, **kwargs): + super().__init__(config, **kwargs) self.config = config - self.model_dim = config.d_model + self.model_dim = config['d_model'] - self.shared = nn.Embedding(config.vocab_size, config.d_model) + self.shared = nn.Embedding(config['vocab_size'], config['d_model']) encoder_config = copy.deepcopy(config) - encoder_config.is_decoder = False - encoder_config.use_cache = False - encoder_config.is_encoder_decoder = False + encoder_config['is_decoder'] = False + encoder_config['use_cache'] = False + encoder_config['is_encoder_decoder'] = False self.encoder = T5Stack(encoder_config, self.shared) decoder_config = copy.deepcopy(config) - decoder_config.is_decoder = True - decoder_config.is_encoder_decoder = False - decoder_config.num_layers = config.num_decoder_layers + decoder_config['is_decoder'] = True + decoder_config['is_encoder_decoder'] = False + if 'number_decoder_layers' in config: + decoder_config['num_layers'] = config['num_decoder_layers'] self.decoder = T5Stack(decoder_config, self.shared) - self.lm_head = nn.Linear(config.d_model, config.vocab_size, bias=False) + self.lm_head = nn.Linear(config['d_model'], config['vocab_size'], bias=False) self.init_weights() # Model parallel @@ -748,7 +756,7 @@ def __init__(self, config): self.device_map = None def get_input_embeddings(self): - return self.shared + return self.shared.weight.data def set_input_embeddings(self, new_embeddings): self.shared = new_embeddings @@ -756,10 +764,10 @@ def set_input_embeddings(self, new_embeddings): self.decoder.set_input_embeddings(new_embeddings) def set_output_embeddings(self, new_embeddings): - self.lm_head = new_embeddings + self.lm_head.weight.data = new_embeddings def get_output_embeddings(self): - return self.lm_head + return self.lm_head.weight def get_encoder(self): return self.encoder @@ -768,23 +776,23 @@ def get_decoder(self): return self.decoder def forward( - self, - input_ids=None, - attention_mask=None, - decoder_input_ids=None, - decoder_attention_mask=None, - head_mask=None, - decoder_head_mask=None, - encoder_outputs=None, - past_key_values=None, - inputs_embeds=None, - decoder_inputs_embeds=None, - labels=None, - use_cache=None, - output_attentions=None, - output_hidden_states=None, - return_dict=None, - mems=None, + self, + input_ids=None, + attention_mask=None, + decoder_input_ids=None, + decoder_attention_mask=None, + head_mask=None, + decoder_head_mask=None, + encoder_outputs=None, + past_key_values=None, + inputs_embeds=None, + decoder_inputs_embeds=None, + labels=None, + use_cache=True, + output_attentions=None, + output_hidden_states=None, + return_dict=True, + mems=None, ): r""" labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`): @@ -804,12 +812,10 @@ def forward( >>> input_ids = tokenizer("summarize: studies have shown that owning a dog is good for you ", return_tensors="pt").input_ids # Batch size 1 >>> outputs = model.generate(input_ids) """ - use_cache = use_cache if use_cache is not None else self.config.use_cache - return_dict = return_dict if return_dict is not None else self.config.use_return_dict # FutureWarning: head_mask was separated into two input args - head_mask, decoder_head_mask if head_mask is not None and decoder_head_mask is None: - if self.config.num_layers == self.config.num_decoder_layers: + if self.config['num_layers'] == self.config['num_decoder_layers']: warnings.warn(__HEAD_MASK_WARNING_MSG, FutureWarning) decoder_head_mask = head_mask @@ -828,11 +834,11 @@ def forward( elif return_dict and not isinstance(encoder_outputs, dict): encoder_outputs = { "last_hidden_state": - encoder_outputs[0], + encoder_outputs[0], "hidden_states": - encoder_outputs[1] if len(encoder_outputs) > 1 else None, + encoder_outputs[1] if len(encoder_outputs) > 1 else None, "attentions": - encoder_outputs[2] if len(encoder_outputs) > 2 else None, + encoder_outputs[2] if len(encoder_outputs) > 2 else None, } hidden_states = encoder_outputs['last_hidden_state'] @@ -890,10 +896,6 @@ def forward( self.lm_head = self.lm_head.to(self.encoder.first_device) sequence_output = sequence_output.to(self.lm_head.weight.device) - if self.config.tie_word_embeddings: - # Rescale output before projecting on vocab - # See https://github.com/tensorflow/mesh/blob/fa19d69eafc9a482aff0b59ddd96b025c0cb207d/mesh_tensorflow/transformer/transformer.py#L586 - sequence_output = sequence_output * (self.model_dim**-0.5) # lm_logits = self.lm_head(sequence_output) @@ -907,17 +909,11 @@ def forward( if not return_dict: output = (lm_logits, ) + decoder_outputs[1:] + encoder_outputs return ((loss, ) + output) if loss is not None else output - if self.config.return_logist_only: - if labels is not None: - return {"loss": loss, "logits": lm_logits} - - else: - return {"logits": lm_logits} return { "loss": loss, "logits": lm_logits, - "past_key_values": decoder_outputs.past_key_values, + "past_key_values": decoder_outputs["past_key_values"], "decoder_hidden_states": decoder_outputs["hidden_states"], "decoder_attentions": decoder_outputs["attentions"], "cross_attentions": decoder_outputs["cross_attentions"], @@ -976,6 +972,26 @@ def _reorder_cache(self, past, beam_idx): reordered_layer_past_states, ) return reordered_decoder_past + def load_weights(self, checkpoint_path): + + _keys_to_ignore_on_load_missing = [ + r"encoder\.embed_tokens\.weight", + r"decoder\.embed_tokens\.weight", + r"lm_head\.weight", + ] + _keys_to_ignore_on_load_unexpected = [ + r"decoder\.block\.0\.layer\.1\.EncDecAttention\.relative_attention_bias\.weight", + ] + + checkpoint = torch.load(checkpoint_path, + map_location=torch.device("cpu")) + if "module" in checkpoint: + # ddp + checkpoint = checkpoint["module"] + self.load_state_dict(checkpoint, strict=False) + self.set_output_embeddings(nn.Parameter(self.get_input_embeddings())) + + return checkpoint class T5EncoderModel(T5PreTrainedModel): authorized_missing_keys = [ @@ -985,11 +1001,11 @@ class T5EncoderModel(T5PreTrainedModel): def __init__(self, config): super().__init__() self.config = config - self.shared = nn.Embedding(config.vocab_size, config.d_model) + self.shared = nn.Embedding(config['vocab_size'], config['d_model']) encoder_config = copy.deepcopy(config) - encoder_config.use_cache = False - encoder_config.is_encoder_decoder = False + encoder_config['use_cache'] = False + encoder_config['is_encoder_decoder'] = False self.encoder = T5Stack(encoder_config, self.shared) self.init_weights() @@ -1028,7 +1044,7 @@ def forward( >>> outputs = model(input_ids=input_ids) >>> last_hidden_states = outputs.last_hidden_state """ - return_dict = return_dict if return_dict is not None else self.config.use_return_dict + return_dict = return_dict if return_dict is not None else self.config['use_return_dict'] encoder_outputs = self.encoder( input_ids=input_ids, diff --git a/flagai/mp_tools.py b/flagai/mp_tools.py new file mode 100644 index 00000000..742f4af9 --- /dev/null +++ b/flagai/mp_tools.py @@ -0,0 +1,124 @@ +import sys +import os +import torch +import copy + +def check_pytorch_model_mp_size(checkpoint:str, target_mp:int): + """ + check the checkpoints contains the weights for mp_size = target_mp + """ + assert os.path.isdir(checkpoint) + filenames = os.listdir(checkpoint) + filenames = [ + filename for filename in filenames if filename.startswith("pytorch_model") + ] + if 'pytorch_model.bin' in filenames and target_mp==1: + return True + else: + filenames.remove('pytorch_model.bin') + print("check the weight files in {}, the number of mp_size({}) {} num_of_files({})".format( + checkpoint, target_mp, "=" if target_mp == len(filenames) else "!=",len(filenames) + )) + return target_mp == len(filenames) + +def change_pytorch_model_mp_from_1_to_n(checkpoint:str, target_mp:int): + if check_pytorch_model_mp_size(checkpoint, target_mp): + return + assert os.path.isdir(checkpoint) + filenames = os.listdir(checkpoint) + filenames = [ + filename for filename in filenames if filename.startswith("pytorch_model") + ] + if 'pytorch_model.bin' in filenames and target_mp>1: + filenames= ['pytorch_model.bin'] + filenames = [os.path.join(checkpoint, x) for x in filenames] + + if target_mp == len(filenames): + print("MP size keeps the same.") + exit(0) + + if checkpoint[-1] == '/': + new_checkpoint = checkpoint[:-1] + else: + new_checkpoint = checkpoint + preserve_keys = [ + "lr_scheduler", + "skipped_steps", + "global_steps", + "global_samples", + "dp_world_size", + "iteration", + "client_lr_scheduler", + "np_rng_state", + "random_rng_state", + "torch_rng_state", + "cuda_rng_state", + "rng_tracker_states", + ] + + if target_mp > len(filenames): + print("Increase MP size.") + assert target_mp % len(filenames) == 0 + ratio = target_mp // len(filenames) + for i in range(len(filenames)): + start = ratio * i + end = ratio * (i + 1) + d = torch.load(filenames[i], map_location='cpu') + for j in range(start, end): + d_new = {} + shift = j - start + for k, v in d.items(): + if k != 'module': + if k in preserve_keys: + d_new[k] = copy.deepcopy(d[k]) + elif k == "mp_world_size": + d_new[k] = target_mp + else: + d_new[k] = None + d_new['module'] = {} + with torch.no_grad(): + for k, v in d['module'].items(): + assert len(v.shape) < 3 + if len(v.shape) == 2 and 'position' not in k: + if 'query' in k: + part = v.shape[0] // ratio // 3 + d_new['module'][k] = torch.cat([ + v[shift * part:(shift + 1) * part, :].clone(), + v[(shift + ratio) * part:(shift + 1 + ratio) * + part, :].clone(), + v[(shift + 2 * ratio) * + part:(shift + 1 + 2 * ratio) * + part, :].clone() + ], 0) + elif 'word' in k or 'h_to_4h' in k or 'relative' in k or "r_w_bias" in k or "r_r_bias" in k: + part = v.shape[0] // ratio + d_new['module'][k] = v[shift * part:(shift + 1) * + part, :].clone() + else: + part = v.shape[1] // ratio + d_new['module'][k] = v[:, + shift * part:(shift + 1) * + part].clone() + elif len(v.shape) == 1 and ('dense_h_to_4h' in k + or "attention.relative" in k): + part = v.shape[0] // ratio + d_new['module'][k] = v[shift * part:(shift + 1) * + part].clone() + elif len(v.shape) == 1 and 'query_key_value' in k: + part = v.shape[0] // ratio // 3 + d_new['module'][k] = torch.cat([ + v[shift * part:(shift + 1) * part].clone(), + v[(shift + ratio) * part:(shift + 1 + ratio) * + part].clone(), + v[(shift + 2 * ratio) * + part:(shift + 1 + 2 * ratio) * part].clone() + ], 0) + else: + d_new['module'][k] = v.clone() + print("saving mp_size = {:02d} ".format(j)) + filename = os.path.join(new_checkpoint, + "pytorch_model_{:02d}.bin".format(j)) + torch.save(d_new, filename) + +if __name__ == "__main__": + change_pytorch_model_mp_from_1_to_n('/mnt/test_10b_models/state_dict/GLM-10b-en',2) \ No newline at end of file diff --git a/flagai/mpu/__init__.py b/flagai/mpu/__init__.py old mode 100755 new mode 100644 diff --git a/flagai/mpu/data.py b/flagai/mpu/data.py old mode 100755 new mode 100644 diff --git a/flagai/mpu/func_utils.py b/flagai/mpu/func_utils.py old mode 100755 new mode 100644 diff --git a/flagai/mpu/grads.py b/flagai/mpu/grads.py old mode 100755 new mode 100644 diff --git a/flagai/mpu/initialize.py b/flagai/mpu/initialize.py old mode 100755 new mode 100644 diff --git a/flagai/mpu/local_attention_function.py b/flagai/mpu/local_attention_function.py old mode 100755 new mode 100644 diff --git a/flagai/mpu/mappings.py b/flagai/mpu/mappings.py old mode 100755 new mode 100644 diff --git a/flagai/mpu/random.py b/flagai/mpu/random.py old mode 100755 new mode 100644 diff --git a/flagai/mpu/utils.py b/flagai/mpu/utils.py old mode 100755 new mode 100644 diff --git a/flagai/optimizers.py b/flagai/optimizers.py old mode 100755 new mode 100644 diff --git a/flagai/schedulers.py b/flagai/schedulers.py old mode 100755 new mode 100644 diff --git a/flagai/test_utils.py b/flagai/test_utils.py index 2bdc495a..f77343f7 100644 --- a/flagai/test_utils.py +++ b/flagai/test_utils.py @@ -120,3 +120,45 @@ def __init__(self): self.few_superglue = False self.pattern_text = False + + +class Seq2SeqCollateArguments: + def __init__(self): + self.cloze_eval = True + self.pretrained_bert = False + + # these values could be wrong + self.task_mask = True + self.continuous_prompt = False + self.prefix_prompt = 0 + + self.max_src_length = 464 + self.max_tgt_length = 48 + self.min_tgt_length = 0 + self.no_block_position = True + + # self.sentinel_token = False + # self.max_position_embeddings = 1024 + # self.no_block_position = False + # self.masked_lm = False + # self.pattern_id = 0 + # self.seq_length = 256 + # self.num_prompt_tokens = 0 + # self.multi_token = False + # self.segment_length = 0 + # self.fast_decode = False + # + # self.few_superglue = False + # self.pattern_text = False +class DatasetArguments: + + def __init__(self): + self.task_mask = True # Distinguished the generation and gap-sentence mask + self.block_mask_prob = 0.1 + self.block_lm = True # Whether do masking + self.masked_lm = False # Whether do simple masking (same symbol among masks) + + self.pre_tokenize = True + self.no_lazy_loader = True + self.half_lazy_loader = False + self.sentinel_token = False \ No newline at end of file diff --git a/flagai/trainer.py b/flagai/trainer.py index 7ccbac8f..3305661a 100644 --- a/flagai/trainer.py +++ b/flagai/trainer.py @@ -16,12 +16,11 @@ import numpy as np import torch.distributed as dist from flagai.logger import log_dist -from torch.utils.checkpoint import checkpoint as torch_checkpoint from torch.utils.tensorboard import SummaryWriter from flagai.utils import load_checkpoint, save_checkpoint from flagai.schedulers import AnnealingLR from flagai.optimizers import get_optimizer, get_optimizer_param_groups -from flagai.fp16 import convert_network +from flagai.fp16 import FP16_Module from flagai.utils import Timers from flagai.launch import launch_dist from torch.nn.parallel import DistributedDataParallel as DDP @@ -264,6 +263,7 @@ def initialize_distributed(self): init_method=init_method) # Set the model-parallel / data-parallel communicators. if self.env_type == 'deepspeed+mpu': + os.environ["MODEL_PARALLEL_SIZE"] = str(self.model_parallel_size) try: mpu.initialize_model_parallel(self.model_parallel_size) if 'deepspeed' in self.env_type and self.deepspeed_activation_checkpointing: @@ -361,6 +361,8 @@ def train(self, else: valid_dataloader = valid_dataset + if self.fp16: + model.half() if self.env_type == 'pytorchDDP': model.to(torch.device('cuda', self.local_rank)) model = DDP(model, @@ -370,24 +372,25 @@ def train(self, elif self.env_type == 'pytorch': model.to(self.pytorch_device) else: - model.to(torch.device('cuda', self.local_rank)) + model.cuda(torch.device('cuda', self.local_rank)) """Train the model.""" # Turn on training mode which enables dropout. - model.train() if self.fp16: - model = convert_network(model, torch.half) + model = FP16_Module(model) + + model.train() param_groups = get_optimizer_param_groups(model) if hasattr(param_groups[0], 'params'): # for T5 Model param_groups = param_groups[0]['params'] - if optimizer is None: + if optimizer is None and 'deepspeed' not in self.env_type: optimizer = get_optimizer(param_groups=param_groups, lr=self.lr, weight_decay=self.weight_decay, cpu_optimizer=True, - cpu_torch_adam=True, + cpu_torch_adam=False, fp16=self.fp16) # if lr_scheduler == None: @@ -400,7 +403,7 @@ def train(self, if 'deepspeed' in self.env_type: # initialize the deepspeed - model, _, __, ___ = deepspeed.initialize( + model, optimizer, __, ___ = deepspeed.initialize( model=model, # if huggingface t5: param_groups[0]['params'] model_parameters=param_groups, @@ -447,7 +450,6 @@ def train(self, for x in batch if x not in ['uid', 'meta', 'mode'] } elif 'pytorch' == self.env_type: - batch = { x: batch[x].to(torch.device(self.pytorch_device)) for x in batch if x not in ['uid', 'meta', 'mode'] @@ -628,16 +630,6 @@ def train_step(self, def forward_step(self, data, model, mems): """Simple forward step. """ data['mems'] = mems - data['checkpoint_fn'] = None - if self.env_type == 'pytorchDDP' and self.checkpoint_activations: - raise Exception( - "Not support yet! `This option will cause RuntimeError: Expected to mark a variable ready only once.` " - ) - if self.env_type == 'pytorch' and self.checkpoint_activations: - data['checkpoint_fn'] = torch_checkpoint - if self.env_type == 'deepspeed' and self.checkpoint_activations: - data['checkpoint_fn'] = deepspeed.checkpointing.checkpoint - model_output = model(**data) logits = model_output['logits'] loss = model_output['loss'] diff --git a/logo.png b/logo.png index 3d1c4559..0136a04e 100644 Binary files a/logo.png and b/logo.png differ diff --git a/quickstart/glm_blank_filling_QA_ch.py b/quickstart/glm_blank_filling_QA_ch.py new file mode 100644 index 00000000..d3fe4f83 --- /dev/null +++ b/quickstart/glm_blank_filling_QA_ch.py @@ -0,0 +1,60 @@ +# coding=utf-8 +# Copyright (c) 2019, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Sample Generate GPT2""" +import sys +# sys.path.append('/data/wang/models/GLMgeneration') +# sys.path.append('/data/wang/models/FlagAI') +import os +import torch + +from flagai.model.glm_model import GLMModel +from flagai.data.tokenizer import GLMLargeChTokenizer +from flagai.model.predictor.predictor import Predictor +if __name__ == "__main__": + """Main training program.""" + print('Generate Samples') + # Random seeds for reproducability. + # Model, + model = GLMModel.from_pretrain(model_name='GLM-large-ch', download_path="./state_dict/") + tokenizer = GLMLargeChTokenizer(vocab_path='./state_dict/GLM-large-ch/cog-pretrain.model', + add_block_symbols=True, + add_task_mask=True, + add_decoder_mask=False, + fix_command_token=False) + + model.cuda(torch.cuda.current_device()) + + predictor = Predictor(model, tokenizer) + # generate samples + text = ['问题：啤酒伤胃吗？回答：[gMASK]', + "问题：隔夜菜能吃吗？回答：[gMASK]", + "问题：如何评价许嵩？回答：[gMASK]"] + for t in text: + output=predictor.predict_generate_randomsample(t, top_k=50, repetition_penalty=4.0, top_p=1.0) + print(t,'\n',output) + + text = ['北京故宫是中国[MASK]非物质文化遗产。', + "上海是中国[MASK]大都市。", + "天津大学是[MASK]现代大学。"] + for t in text: + output = predictor.predict_generate_randomsample(t, top_k=50, repetition_penalty=4.0, top_p=1.0) + print(t, '\n', output) + # + text = ["人工智能是一个以计算机科学为基础，由计算机、数学、哲学等多学科交叉融合的交叉学科，[sMASK]，具有非常巨大的前景。", + "最近十多年来，人工神经网络的研究工作不断深入，已经取得了很大的进展，[sMASK]，表现出了良好的智能特性。"] + for t in text: + output = predictor.predict_generate_randomsample(t, top_k=50, repetition_penalty=4.0, top_p=1.0) + print(t, '\n', output) diff --git a/quickstart/glm_title_ch.py b/quickstart/glm_title_ch.py new file mode 100644 index 00000000..7c260ee0 --- /dev/null +++ b/quickstart/glm_title_ch.py @@ -0,0 +1,40 @@ +import torch +from flagai.auto_model.auto_loader import AutoLoader +from flagai.model.predictor.predictor import Predictor + +device = torch.device("cuda" if torch.cuda.is_available() else "cpu") + +auto_loader = AutoLoader("title-generation", + model_name="GLM-large-ch", + model_dir="./state_dict/") +model = auto_loader.get_model() +tokenizer = auto_loader.get_tokenizer() + +model.to(device) +model.eval() +predictor = Predictor(model, tokenizer) + +test_data = [ + "本文总结了十个可穿戴产品的设计原则，而这些原则同样也是笔者认为是这个行业最吸引人的地方，1为人们解决重复性问题2从人开始而不是从机器开始3要引起注意但不要刻意4提升用户能力而不是取代人。", + "2007年乔布斯向人们展示iPhone并宣称它将会改变世界，还有人认为他在夸大其词，然而在8年后以iPhone为代表的触屏智能手机已经席卷全球各个角落，未来智能手机将会成为真正的个人电脑为人类发展做出更大的贡献。", + "雅虎发布2014年第四季度财报并推出了免税方式剥离其持有的阿里巴巴集团15％股权的计划打算将这一价值约400亿美元的宝贵投资分配给股东，截止发稿前雅虎股价上涨了大约7％至5145美元" +] + +for text in test_data: + print('===============================================\n') + print(text, ":") + for i in range(1): #generate several times + print("--------------sample %d :-------------------" % (i)) + print('-----------random sample: --------------') + print( + predictor.predict_generate_randomsample(text, + out_max_length=66, + top_k=10, + top_p=1.0, + repetition_penalty=4.0, + temperature=1.2)) + print('-----------beam search: --------------') + print( + predictor.predict_generate_beamsearch(text, + out_max_length=66, + beam_size=10)) diff --git a/quickstart/sm_ch.py b/quickstart/sm_ch.py deleted file mode 100644 index 52cc9e95..00000000 --- a/quickstart/sm_ch.py +++ /dev/null @@ -1,20 +0,0 @@ -import torch -from flagai.auto_model.auto_loader import AutoLoader -from flagai.model.predictor.predictor import Predictor - -device = torch.device("cuda" if torch.cuda.is_available() else "cpu") -maxlen = 256 - -auto_loader = AutoLoader("semantic-matching", - model_name="RoBERTa-base-ch", # Load the checkpoints from Modelhub(model.baai.ac.cn/models) - class_num=2) -model = auto_loader.get_model() -tokenizer = auto_loader.get_tokenizer() - -predictor = Predictor(model, tokenizer) - -test_data = [["后悔了吗", "你有没有后悔"], ["打开自动横屏", "开启移动数据"], - ["我觉得你很聪明", "你聪明我是这么觉得"]] - -for text_pair in test_data: - print(text_pair, "相似" if predictor.predict_cls_classifier(text_pair) == 1 else "不相似")