LLaMA FastTokenizer does not add `eos_token_id` at the end.

[osainz59]

System Info

• transformers version: 4.29.0.dev0
• Platform: Linux-4.18.0-305.19.1.el8_4.x86_64-x86_64-with-glibc2.28
• Python version: 3.9.7
• Huggingface_hub version: 0.13.3
• Safetensors version: 0.3.0
• PyTorch version (GPU?): 2.1.0.dev20230411+cu117 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?: No
• Using distributed or parallel set-up in script?: No

Who can help?

@ArthurZucker

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

As mentioned on the title, the LLaMA tokenizer does not add the eos_token at the end of the inputs. This only happens on the fast version (use_fast=True).

Steps to reproduce the behaviour:

1. Load the LLaMA tokenizer

tokenizer = AutoTokenizer.from_pretrained(LLAMA_PATH, add_eos_token=True, use_fast=True)

2. Tokenize something

simple_sentence = "This is a sentence to test if the tokenizer adds eos token."
simple_sentence_ids = tokenizer(
    simple_sentence, add_special_tokens=True
).input_ids

3. Print the input_ids to check if the eos_token_id (2) is added at the end.

print(simple_sentence_ids)

4. Output:

[1, 910, 338, 263, 10541, 304, 1243, 565, 278, 5993, 3950, 12778, 321, 359, 5993, 29889]

Expected behavior

Expected output

[1, 910, 338, 263, 10541, 304, 1243, 565, 278, 5993, 3950, 12778, 321, 359, 5993, 29889, 2]

[ArthurZucker]

Yes! Quick fix, use the slow tokenizer. Otherwise I'll open a PR to add template processing!
Thanks for reporting!

[gorjanradevski]

But it shouldn't add an eos token right? The LM is not trained to generate a token after the eos I believe.

[osainz59]

But it shouldn't add an eos token right? The LM is not trained to generate a token after the eos I believe.

By default, but if specified with add_eos_token=True it should. You can always fine-tune the model to make the model learn when to stop.

[Elfsong]

I guess they would set the pad_token_id using the eos_token_id?
model.config.pad_token_id = model.config.eos_token_i

[ndvbd]

Same here, doing add_eos_token=True doesn't do anything

[ArthurZucker]

This should have been fixed by #22959

[jonathangomesselman]

I guess they would set the pad_token_id using the eos_token_id? model.config.pad_token_id = model.config.eos_token_i

I believe if you just set the pad_token = eos_token the model still is not learning to predict the eos_token because the corresponding attn_mask does not include the token and the labels ignores that token - i.e. no loss is computed for it. Not 100% sure about this, but that was what it seemed like from some self exploration.

[avacaondata]

The same is happening with Falcon...

[ArthurZucker]

When you say the same, what do you mean?

[avacaondata]

That it doesn't generate <|endoftext|> (token id 11) when calling generate, therefore it never stops generating. I have tried by setting eos_token_id to 193, which corresponds to \n, but I don't think that's a clean fix. I have noticed that when tokenizing the inputs with the Falcon-40b tokenizer, it's not adding eos_token_id at the end of input ids.

[ArthurZucker]

Few things here.
Llama has no official model so make sure the one you are using is up to date and has the same eos token id for the model.config / generation config and the tokenizer.

For falcon, code is on the hub, but latest code of transformers adds the eos if you set “add_eos=True”. In the doc for llama you can find that initializing a model with “add_eos=True” will make it add the eos when tokenizing.

[avacaondata]

Actually I was talking about Falcon, not llama, because I'm facing an issue similar to the ones people are reporting with Llama. In fact I upgraded my transformers version to the last version in main branch, and the problem persists... The model never generates a EOS token, so it never stops generating...
I have tried to explicitly add a string "<|endoftext|>" at the end of the inputs for fine-tuning, but still doesn't work.

What can I do to make falcon generate a eos token ?