Skip to content

[LLM] support fix seq len and cmd run service #10004

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ZHUI merged 3 commits into from
Mar 6, 2025
Merged

[LLM] support fix seq len and cmd run service #10004

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
3871129
[LLM] support fix seq len
ltd0924 Mar 6, 2025
766f639
[LLM] support cmd run service
ltd0924 Mar 6, 2025
fd2bdfe
[LLM] support cmd run service
ltd0924 Mar 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions llm/server/dockerfiles/Dockerfile_serving_cuda118_cudnn8
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,12 @@ RUN mkdir -p /opt/source/ && cd /opt/source/ \
&& python3 -m pip install --no-cache-dir -r PaddleNLP/requirements.txt \
&& python3 -m pip install --no-cache-dir -r PaddleNLP/llm/server/server/requirements.txt

RUN cp /opt/source/PaddleNLP/llm/server/server/scripts/start_server.sh /opt/output/Serving/ \
&& cp /opt/source/PaddleNLP/llm/server/server/scripts/stop_server.sh /opt/output/Serving/
RUN cp /opt/source/PaddleNLP/llm/server/server/scripts/start_server.sh start_server \
&& chmod +x start_server \
&& mv start_server /usr/local/bin/ \
&& cp /opt/source/PaddleNLP/llm/server/server/scripts/stop_server.sh start_server \
&& chmod +x stop_server \
&& mv stop_server /usr/local/bin/

ENV PYTHONPATH="/opt/source/PaddleNLP/llm/server/server:/opt/source/PaddleNLP"

Expand Down
8 changes: 6 additions & 2 deletions llm/server/dockerfiles/Dockerfile_serving_cuda124_cudnn9
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,12 @@ RUN mkdir -p /opt/source/ && cd /opt/source/ \
&& python3 -m pip install --no-cache-dir -r PaddleNLP/requirements.txt \
&& python3 -m pip install --no-cache-dir -r PaddleNLP/llm/server/server/requirements.txt

RUN cp /opt/source/PaddleNLP/llm/server/server/scripts/start_server.sh /opt/output/Serving/ \
&& cp /opt/source/PaddleNLP/llm/server/server/scripts/stop_server.sh /opt/output/Serving/
RUN cp /opt/source/PaddleNLP/llm/server/server/scripts/start_server.sh start_server \
&& chmod +x start_server \
&& mv start_server /usr/local/bin/ \
&& cp /opt/source/PaddleNLP/llm/server/server/scripts/stop_server.sh stop_server \
&& chmod +x stop_server \
&& mv stop_server /usr/local/bin/

ENV PYTHONPATH="/opt/source/PaddleNLP/llm/server/server:/opt/source/PaddleNLP"

Expand Down
12 changes: 6 additions & 6 deletions llm/server/server/server/engine/config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -338,13 +338,13 @@ def reset_value(self, value_name, key, config):
reset_value(self, "block_size", "infer_model_block_size", config)
reset_value(self, "max_seq_len", "infer_model_max_seq_len", config)
reset_value(self, "return_full_hidden_states", "return_full_hidden_states", config)
if self.seq_len_limit > self.max_seq_len:
self.seq_len_limit = self.max_seq_len
logger.warning(f"The loading model requires len(input_ids) <= {self.max_seq_len}, now reset MAX_SEQ_LEN.")

assert (
self.seq_len_limit <= self.max_seq_len
), f"The loading model requires len(input_ids) <= {self.max_seq_len}, but now the setting MAX_SEQ_LEN={self.seq_len_limit}."
assert (
self.dec_len_limit <= self.max_seq_len
), f"The loading model requires MAX_DEC_LEN <= {self.max_seq_len}, but now the setting MAX_DEC_LEN={self.dec_len_limit}."
if self.dec_len_limit > self.max_seq_len:
self.dec_len_limit = self.max_seq_len
logger.warning(f"The loading model requires MAX_DEC_LEN <= {self.max_seq_len}, now reset MAX_DEC_LEN.")

def get_unique_name(self, name):
"""
Expand Down
Loading