Batch mode in GPU

[uasan]

We like minimalist and productive inference, thank you!
But our use cases, and those of many users, require batch processing.
Batch mode isn't productive on CPUs. Do you have plans to implement inference execution on GPUs (TensorRT, CUDA)?

If so, you'll become the top solution for ETL processing that requires generative models.

[jan-wassenberg]

:) I agree batch inference is useful and interesting, but it's not clear to me why it should not work well on CPU.
We do not have any plans to support GPUs here.
Might vLLM be a good solution for your use case?

[uasan]

but it's not clear to me why it should not work well on CPU.

Because the CPU has much fewer cores compared to the GPU.

Might vLLM be a good solution for your use case?

We're doing it this way now, but vLLM has very complex dependencies; it pulls almost the entire Python stack and breaks when library versions change.

[jan-wassenberg]

CUDA 'cores' are misleadingly named and not comparable to cores. One could instead look at FLOPs, I believe Turin CPUs are similar there to some GPUs, at least without the tensor cores. But that also matters less than utilization and end to end perf :)

I hear you about dependencies, that does not sound ideal.