How to infer multiple prompts(bs>1) at the same time?

[vicwer]

No description provided.

[KerfuffleV2]

Possibly with the server example: https://github.com/ggerganov/llama.cpp/tree/master/examples/server

You would need a script or something to manage the queries and collect the result.

[liuxiaohao-xn]

Possibly with the server example: https://github.com/ggerganov/llama.cpp/tree/master/examples/server

You would need a script or something to manage the queries and collect the result.

i have the same question.

I have multiple prompts, I want to feed them all at once to the model to generate the outputs，can you tell me how to achieve it ?