Return log probabilities for tokens

[yunfeng-scale]

Hi, we'd like to have log probabilities for tokens returned from the model, in addition to the token ids. Can you help with this feature request?

[byshiue]

Here is a flag to output log probs of generated tokens https://github.com/NVIDIA/TensorRT-LLM/blob/main/tensorrt_llm/runtime/generation.py#L280. The log probs of input tokens are not supported now, it will be supported in near future.

[yunfeng-scale]

@byshiue thanks. Possible to include in the C++ implementation too? https://github.com/NVIDIA/TensorRT-LLM/blob/main/cpp/include/tensorrt_llm/batch_manager/callbacks.h#L32 currently inflight batching is supported through C++ runtime