convert : restore compat with old Falcon models

[cebtenzzre]

This restores the ability to convert models like WizardLM-Uncensored-Falcon-40b that still use the old format.

[cmp-nct]

I wanted to give llama.cpp a try with falcon and I have not been able to either use any of the most recent gguf (invalid character) models from HF nor being able to convert any falcon 40B model using the latest git.
"KeyError: 65029" is the current error I get (on latest HF openassist releases, like falcon-40b-sft-mix-1226)

[cebtenzzre]

"KeyError: 65029" is the current error I get (on latest HF openassist releases, like falcon-40b-sft-mix-1226)

@goerch I originally noted this here: #3252 (comment)
We need to re-add the code that inserted the padding tokens, it shouldn't have been removed.

[cmp-nct]

Bugfix: in convert-falcon-hf-to-gguf.py you can use this modification:

    if i in reverse_vocab:
        tokens.append(reverse_vocab[i])
        scores.append(0.0) # dummy
        toktypes.append(gguf.TokenType.NORMAL)
    else:
        padding_token = f"[PAD{i}]".encode("utf8")
        text = bytearray(padding_token)
        tokens.append(text)
        scores.append(0.0) # dummy
        toktypes.append(gguf.TokenType.NORMAL)

This will add padding tokens as needed, the model will work from there on.
However, there is a warning message about a token mismatch, maybe it's nothing.