No, basically, the requests are processed in batches, together, and the order they're listed in matters for the results, as the grid (tiles) that the GPU is ultimately processing, are different depending on what order they entered at.
So if you want batching + determinism, you need the same batch with the same order which obviously don't work when there are N+1 clients instead of just one.
chrisjj|1 month ago
embedding-shape|1 month ago
So if you want batching + determinism, you need the same batch with the same order which obviously don't work when there are N+1 clients instead of just one.