[QST] How to get the prefill latency and TPOT resepectly when using C++ runtime

[gujiewen]

Recently trt_llm v0.16 has supported qwen2vl. However, it use c++ runtime, and c++ runtime seems deal with the whole inference. I mean c++ runtime get the input and give me the final output sequence.

        multi_responses = self.session.await_responses(request_ids)
        responses = [
            response for responses in multi_responses for response in responses
        ]

        return self._fill_output(responses, output_ids, end_id, return_dict,
                                 output_sequence_lengths, output_log_probs,
                                 output_cum_log_probs, batch_input_ids, [],
                                 streaming, request_ids, False, max_new_tokens,
                                 sampling_config, is_draft_target_model)

and in _fill_output

        for response in responses:
            if response.has_error():
                raise RuntimeError(response.error_msg)

            result = response.result
            batch_idx = request_ids.index(response.request_id)
            if is_beam_search:
                for beam, output_tokens in enumerate(result.output_token_ids):
                    fill_output_ids(output_tokens, batch_idx, beam)
            else:
                fill_output_ids(result.output_token_ids[0], batch_idx,
                                result.sequence_index)

        if output_sequence_lengths:
            sequence_lengths = [[len(token_ids) for token_ids in beams]
                                for beams in output_ids]

So how could I get the accurate TTFT and TPOT respectly?

[hello-11]

@gujiewen if you want to get the prefill latency, you can set the out_length=1.

[gujiewen]

@gujiewen if you want to get the prefill latency, you can set the out_length=1.

How about the TPOT

[karljang]

I'm not sure if this issue is still relevant to you, but there’s a related PR (#5785) that implemented a few metrics. That said, I’m not certain whether it fully addresses your requirements.
At the same time, I'm closing this issue due to prolonged inactivity, if the problem persists in the latest release, please feel free to open a new one.
Thank you!