[ivalm]

You can assign multiarm bandit trials on a lazy per user basis.

So first time user touches feature A they are assigned to some trial arm T_A and then all subsequent interactions keep them in that trial arm until the trial finishes.

[kridsdale1]

The systems I’ve use pre-allocate users effectively randomly an arm by hashing their user id or equivalent.

[ivalm]

To make sure user id U doesn’t always end up in eg control group it’s useful to concatenate the id with experiment uuid.

[ryan-duve]

How do you handle different users having different numbers of trials when calculating the "click through rate" described in the article?

[s1mplicissimus]

careful when doing that though! i've seen some big eyes when people assumed IDs to be uniform randomly distributed and suddenly their "test group" was 15% instead of the intended 1%. better generate a truely random value using your languages favorite crypto functions and be able to work with it without fear of busting production

[hinkley]

Just make sure you do the hash right so you don’t end up with cursed user IDs like EverQuest.