top | item 43127156

(no title)

pritambaral | 1 year ago

> ... making sure that objects are placed into exactly 1 batch. curious if you ran into any bottlenecks there?

A single application-layer thread doing batches of batch creation (heh). Not instant, but fast enough. I did have to add 'batchmaker is done' onto the 'no batch left' condition for worker exit.

> ... that pushes the locking from selecting queue entries to ...

To selecting batches. A batch is immutable once created. If work has to be restarted to handle new/updated objects, all batches are wiped and the batchmaker (and workers, anyway) start over.

discuss

order

No comments yet.