Skip to content

llm pretrain#256

Merged
ftgreat merged 145 commits into
masterfrom
gpm_dev
Mar 21, 2023
Merged

llm pretrain#256
ftgreat merged 145 commits into
masterfrom
gpm_dev

Conversation

@ftgreat

@ftgreat ftgreat commented Mar 10, 2023 •
edited
Loading

Copy link
Copy Markdown
Collaborator

name: Pull Request
title: '[PR]'
assignees: 'BAAI-OpenPlatform,ftgreat'

Description

Please describe here what the PR does.

Checklist

  • bug fixed
  • new feature provided
  • documentation updated
  • tests added

shunxing1234 and others added 30 commits January 4, 2023 14:47
@shunxing1234
Merge pull request #5 from shunxing1234/dev_bmtrain
60fc013 
Dev bmtrain
add gpt2-xl
e442128 
Signed-off-by: ftgreat <ftgreat@github.com>
update bmtrain trainer
42d027f 
Signed-off-by: ftgreat <ftgreat@github.com>
up env_trainer
905c96a 
Signed-off-by: ftgreat <ftgreat@github.com>
fix env_trainer
c9c973d 
Signed-off-by: ftgreat <ftgreat@github.com>
keep env_trainer according to trainer
e2ccb70 
Signed-off-by: ftgreat <ftgreat@github.com>
add gpt2_xl_pretrain
2c6a3d4 
Signed-off-by: ftgreat <ftgreat@github.com>
@ftgreat
add opt_pretrain
11fa6a4 
Signed-off-by: ftgreat <ftgreat@163.com>
@ftgreat
add cpu offloading
52dbaa6 
Signed-off-by: ftgreat <ftgreat@163.com>
@ftgreat
add train_env
04c6fda 
Signed-off-by: ftgreat <ftgreat@163.com>
@ftgreat
add train_env
e400c26 
Signed-off-by: ftgreat <ftgreat@163.com>
@ftgreat
add gpt3_pretrain
e031e0a 
Signed-off-by: ftgreat <ftgreat@163.com>
@ftgreat
add run_train.sh for gpt3_pretrain
113ba4d 
Signed-off-by: ftgreat <ftgreat@163.com>
add gpt2_title_generation debug.txt
c703201 
Signed-off-by: ftgreat <ftgreat@github.com>
add gpt2_title_generation dev.txt
5ba5b11 
Signed-off-by: ftgreat <ftgreat@github.com>
@ftgreat
add train_env_xl_bmtrain.py
8180318 
Signed-off-by: ftgreat <ftgreat@163.com>
@ftgreat
Merge branch 'gpm_dev' of github.com:ftgreat/FlagAI into gpm_dev
2fc47d9
@ftgreat
add tokens_stat.py
98c9e6f 
Signed-off-by: ftgreat <ftgreat@163.com>
up logger
fad93fa 
Signed-off-by: ftgreat <ftgreat@github.com>
@ftgreat
up run_train.sh
705a1b6 
Signed-off-by: ftgreat <ftgreat@163.com>
@ftgreat
fix logger for bmt
3de75ce 
Signed-off-by: ftgreat <ftgreat@163.com>
@ftgreat
add run_train.sh
50f2fbb 
Signed-off-by: ftgreat <ftgreat@163.com>
@ftgreat
up logger.py
8df26d4 
Signed-off-by: ftgreat <ftgreat@163.com>
@ftgreat
fix logger.py for bmtrain
b524d53 
Signed-off-by: ftgreat <ftgreat@163.com>
@ftgreat
fix logger.py for bmtrain heto
86f9f82 
Signed-off-by: ftgreat <ftgreat@163.com>
@ftgreat
add gradient_accumulation for bmtrain
60dd252 
Signed-off-by: ftgreat <ftgreat@163.com>
@ftgreat
fix avg_loss when gradient_accumulation gt 1
26c45a2 
Signed-off-by: ftgreat <ftgreat@163.com>
@ftgreat
update logger.py
1b1ef1e 
Signed-off-by: ftgreat <ftgreat@163.com>
env_trainer trainer align comments
64a794d 
Signed-off-by: ftgreat <ftgreat@github.com>
@ftgreat
add env_trainer_v1 trainer_v1
52d30d1 
Signed-off-by: ftgreat <ftgreat@163.com>
marscrazy
marscrazy reviewed Mar 10, 2023
View reviewed changes
Comment thread examples/gpt2_xl_pretrain/train_env_xl_bmtrain_debug.py
trainer = EnvTrainer(env_args)

'''
trainer = Trainer(

@marscrazy marscrazy Mar 10, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove the unused codes

marscrazy
marscrazy reviewed Mar 10, 2023
View reviewed changes
Comment thread examples/gpt3_pretrain/train_env_xl_bmtrain_debug_rank_split_v2.py
src = []
tgt = []

if enable_debug:

@marscrazy marscrazy Mar 10, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove debug code？

marscrazy
marscrazy reviewed Mar 10, 2023
View reviewed changes
Comment thread examples/gpt3_pretrain/train_env_xl_bmtrain_galactica_6.7b.py
sys.exit(0)

## TODO
### 需要根据配置情况填写路径

@marscrazy marscrazy Mar 10, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

translation

marscrazy
marscrazy reviewed Mar 10, 2023
View reviewed changes
Comment thread examples/gpt3_pretrain/train_env_xl_bmtrain_gpm_13b.py
model = GPT2Model.init_from_json(config_file=config_file)
print('*'*20, "model", model)
trainer.pre_train(model)

@marscrazy marscrazy Mar 10, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

translation

marscrazy
marscrazy reviewed Mar 10, 2023
View reviewed changes
Comment thread examples/gpt3_pretrain/train_env_xl_bmtrain_gpm_13b.py
### merged_text_document_valid_indexmap_41313229ns_1024sl_2023s_doc_idx.npy

### gpm
data_prefix = '/share/project/ldwang/data/indexed_dataset/gpm/merged_text_document'

@marscrazy marscrazy Mar 10, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

data path

marscrazy
marscrazy reviewed Mar 10, 2023
View reviewed changes
Comment thread examples/gpt3_pretrain/train_env_xl_bmtrain_gpm_xl.py
trainer.pre_train(model)

# TODO
data_path = '/share/project/ldwang/data/pile'

@marscrazy marscrazy Mar 10, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the data path

marscrazy
marscrazy reviewed Mar 10, 2023
View reviewed changes
Comment thread examples/gpt3_pretrain/train_env_xl_bmtrain_gpt2_base.py
trainer.pre_train(model)

# TODO
### 需要根据数据集情况填写

@marscrazy marscrazy Mar 10, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

translation

ftgreat and others added 21 commits March 13, 2023 17:19
@ftgreat
remove eos in gpt predictor
5ccb7ff 
Signed-off-by: ftgreat <ftgreat@163.com>
@ftgreat
Merge pull request #260 from ftgreat/gpm_dev
20372a8 
remove eos in gpt predictor
@ftgreat
remove eos tokens for bream search
d0af929 
Signed-off-by: ftgreat <ftgreat@163.com>
@ftgreat
update utils add iteration_in_epoch
e341dcb 
Signed-off-by: ftgreat <ftgreat@163.com>
@ftgreat
fix ommit first token in outputs
432a677 
Signed-off-by: ftgreat <ftgreat@163.com>
@ftgreat
Merge pull request #268 from ftgreat/gpm_dev
08d55f3 
Gpm dev
@ftgreat
add gpt2_new_1w
cc9b6c8 
Signed-off-by: ftgreat <ftgreat@163.com>
@ftgreat
Merge pull request #274 from ftgreat/gpm_dev
39b8bd0 
add gpt2_new_1w
@ftgreat
mv gpt2_new_1w to gpt2_new_100k
3246e4d 
Signed-off-by: ftgreat <ftgreat@163.com>
@ftgreat
weighted_datasets demo
e0804fe 
Signed-off-by: ftgreat <ftgreat@163.com>
@ftgreat
Merge pull request #275 from ftgreat/gpm_dev
89946c7 
mv gpt2_new_1w to gpt2_new_100k
@ftgreat
Merge pull request #276 from ftgreat/gpm_dev
f53c321 
weighted_datasets demo
@ftgreat
up weighted_datasets demo
411ae9f 
Signed-off-by: ftgreat <ftgreat@163.com>
@ftgreat
up weighted_datasets demo2
8886787 
Signed-off-by: ftgreat <ftgreat@163.com>
@ftgreat
Merge pull request #277 from ftgreat/gpm_dev
e40c43e 
Gpm dev
@ftgreat
up weighted_datasets demo
337e89c 
Signed-off-by: ftgreat <ftgreat@163.com>
@ftgreat
Merge pull request #278 from ftgreat/gpm_dev
d765457 
up weighted_datasets demo
@ftgreat
add build_weighted_datasets.py
760deaa 
Signed-off-by: ftgreat <ftgreat@163.com>
@ftgreat
Merge pull request #279 from ftgreat/gpm_dev
acea42d 
add build_weighted_datasets.py
@ftgreat
fix data_loader sampler for ds&mpu
6022bce 
Signed-off-by: ftgreat <ftgreat@163.com>
@ftgreat
Merge pull request #280 from ftgreat/gpm_dev
1aef572 
fix data_loader sampler for ds&mpu
@ftgreat ftgreat merged commit 590d178 into master Mar 21, 2023
@ftgreat ftgreat mentioned this pull request Mar 21, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@BAAI-OpenPlatform BAAI-OpenPlatform Awaiting requested review from BAAI-OpenPlatform
1 more reviewer
@marscrazy marscrazy marscrazy left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@ftgreat @marscrazy @shunxing1234 @isuco