Using LLamaPro and LORA gives error: KeyError: 'train.num_layer_trainable'

[MarlNox]

Reminder

• I have read the README and searched the existing issues.

System Info

Latest version, Ubuntu 24.04

Reproduction

Run Llama Pro and Lora together for finetuning on a model with expanded blocks.

Expected behavior

It should run normally. and start training the model.

Others

No response

[MarlNox]

Doing the expansion with the llama pro script in the library fixed the issue. I was initially doing the expansion manually.

[MarlNox]

@hiyouga Apparently the error persists. I just check the lora finetune i did, i hadn't checked the llamapro box. Checkign it, makes the error return.

[MarlNox]

To create a public link, set share=True in launch().
Traceback (most recent call last):
File "/home/user/anaconda3/envs/llama-factory/lib/python3.10/site-packages/gradio/queueing.py", line 541, in process_events
response = await route_utils.call_process_api(
File "/home/user/anaconda3/envs/llama-factory/lib/python3.10/site-packages/gradio/route_utils.py", line 276, in call_process_api
output = await app.get_blocks().process_api(
File "/home/user/anaconda3/envs/llama-factory/lib/python3.10/site-packages/gradio/blocks.py", line 1928, in process_api
result = await self.call_function(
File "/home/user/anaconda3/envs/llama-factory/lib/python3.10/site-packages/gradio/blocks.py", line 1526, in call_function
prediction = await utils.async_iteration(iterator)
File "/home/user/anaconda3/envs/llama-factory/lib/python3.10/site-packages/gradio/utils.py", line 657, in async_iteration
return await iterator.anext()
File "/home/user/anaconda3/envs/llama-factory/lib/python3.10/site-packages/gradio/utils.py", line 650, in anext
return await anyio.to_thread.run_sync(
File "/home/user/anaconda3/envs/llama-factory/lib/python3.10/site-packages/anyio/to_thread.py", line 56, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
File "/home/user/anaconda3/envs/llama-factory/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 2144, in run_sync_in_worker_thread
return await future
File "/home/user/anaconda3/envs/llama-factory/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 851, in run
result = context.run(func, *args)
File "/home/user/anaconda3/envs/llama-factory/lib/python3.10/site-packages/gradio/utils.py", line 633, in run_sync_iterator_async
return next(iterator)
File "/home/user/anaconda3/envs/llama-factory/lib/python3.10/site-packages/gradio/utils.py", line 816, in gen_wrapper
response = next(iterator)
File "/home/user/Desktop/LLaMA-Factory/src/llamafactory/webui/runner.py", line 324, in run_train
yield from self._launch(data, do_train=True)
File "/home/user/Desktop/LLaMA-Factory/src/llamafactory/webui/runner.py", line 294, in _launch
args = self._parse_train_args(data) if do_train else self._parse_eval_args(data)
File "/home/user/Desktop/LLaMA-Factory/src/llamafactory/webui/runner.py", line 182, in _parse_train_args
args["num_layer_trainable"] = get("train.num_layer_trainable")
File "/home/user/Desktop/LLaMA-Factory/src/llamafactory/webui/runner.py", line 103, in
get = lambda elem_id: data[self.manager.get_elem_by_id(elem_id)]
File "/home/user/Desktop/LLaMA-Factory/src/llamafactory/webui/manager.py", line 55, in get_elem_by_id
return self._id_to_elem[elem_id]
KeyError: 'train.num_layer_trainable'

[hiyouga]

fixed

[MarlNox]

Worked. Thanks a lot mate. You're a lifesaver. Working on my thesis with LlamaFactory.

Could you by any chance check this one out aswell?

Traceback (most recent call last):
File "/home/user/anaconda3/envs/llama-factory/bin/llamafactory-cli", line 8, in
sys.exit(main())
File "/home/user/Music/LLaMA-Factory/src/llamafactory/cli.py", line 111, in main
run_exp()
File "/home/user/Music/LLaMA-Factory/src/llamafactory/train/tuner.py", line 50, in run_exp
run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks)
File "/home/user/Music/LLaMA-Factory/src/llamafactory/train/sft/workflow.py", line 47, in run_sft
model = load_model(tokenizer, model_args, finetuning_args, training_args.do_train)
File "/home/user/Music/LLaMA-Factory/src/llamafactory/model/loader.py", line 140, in load_model
model = load_unsloth_pretrained_model(config, model_args)
File "/home/user/Music/LLaMA-Factory/src/llamafactory/model/model_utils/unsloth.py", line 53, in load_unsloth_pretrained_model
from unsloth import FastLanguageModel
File "/home/user/anaconda3/envs/llama-factory/lib/python3.10/site-packages/unsloth/init.py", line 26, in
raise ImportError(f"Unsloth: Please import Unsloth before {module}.")
ImportError: Unsloth: Please import Unsloth before bitsandbytes.

[hiyouga]

try reinstall unsloth

[MarlNox]

Reinstalling unsloth, as per the guidelines in the unsloth library. Provided the error below, which didnt fix even after removing and reinstalling both bitsandbytes and unsloth.

Traceback (most recent call last):
  File "/home/user/anaconda3/envs/llama-factory/bin/llamafactory-cli", line 8, in <module>
    sys.exit(main())
  File "/home/user/Music/LLaMA-Factory/src/llamafactory/cli.py", line 111, in main
    run_exp()
  File "/home/user/Music/LLaMA-Factory/src/llamafactory/train/tuner.py", line 50, in run_exp
    run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks)
  File "/home/user/Music/LLaMA-Factory/src/llamafactory/train/sft/workflow.py", line 47, in run_sft
    model = load_model(tokenizer, model_args, finetuning_args, training_args.do_train)
  File "/home/user/Music/LLaMA-Factory/src/llamafactory/model/loader.py", line 162, in load_model
    model = init_adapter(config, model, model_args, finetuning_args, is_trainable)
  File "/home/user/Music/LLaMA-Factory/src/llamafactory/model/adapter.py", line 310, in init_adapter
    model = _setup_lora_tuning(
  File "/home/user/Music/LLaMA-Factory/src/llamafactory/model/adapter.py", line 246, in _setup_lora_tuning
    model = get_unsloth_peft_model(model, model_args, peft_kwargs)
  File "/home/user/Music/LLaMA-Factory/src/llamafactory/model/model_utils/unsloth.py", line 79, in get_unsloth_peft_model
    return FastLanguageModel.get_peft_model(**peft_kwargs, **unsloth_peft_kwargs)
  File "/home/user/anaconda3/envs/llama-factory/lib/python3.10/site-packages/unsloth/models/llama.py", line 1658, in get_peft_model
    assert(module in accepted_modules)
AssertionError

[hiyouga]

i thought unsloth may be incompatible with llama pro