Bugfix: missing hparam type_vocab_size#32
Conversation
|
Here is some of the model upon bert DO NOT use
|
|
Hi, This seems like a good change. But surely bert.cpp:358 also needs to be changed? Where the hparams are read from the model file |
sure, I will add it. |
|
|
||
| model.word_embeddings = ggml_new_tensor_2d(ctx, wtype, n_embd, n_vocab); | ||
| model.token_type_embeddings = ggml_new_tensor_2d(ctx, wtype, n_embd, 2); | ||
| model.token_type_embeddings = ggml_new_tensor_2d(ctx, wtype, n_embd, n_vocab_size); |
There was a problem hiding this comment.
@skeskinen
here is a change since the tensor is related to n_vocab_size.
for many case, it is 2, but not a const.
There was a problem hiding this comment.
| int32_t n_intermediate = 1536; | ||
| int32_t n_head = 12; | ||
| int32_t n_layer = 6; | ||
| int32_t n_vocab_size = 2; |
There was a problem hiding this comment.
Here is a break change since new field.
I believe some others will happen in future.
So we may try to shift into GGUF.
https://github.com/philpax/ggml/blob/gguf-spec/docs/gguf.md
| fin.read((char *)&hparams.n_intermediate, sizeof(hparams.n_intermediate)); | ||
| fin.read((char *)&hparams.n_head, sizeof(hparams.n_head)); | ||
| fin.read((char *)&hparams.n_layer, sizeof(hparams.n_layer)); | ||
| fin.read((char *)&hparams.n_vocab_size, sizeof(hparams.n_vocab_size)); |
2 participants
type_vocab_sizeis also a hparam (can not use const as 2).