(no title)
jaschasd | 2 years ago
For experiments 1 through 4, N was 38, 389, 396, and 389. The subjects were not undergrad psych students.
The article linked in the parent comment does not correspond to any experiment in the blog post or the Nature Comms paper.
godelski|2 years ago
> __Experiment 1__ included 38 participants with normal or corrected vision. Participants gave informed consent and were awarded reasonable compensation for their time and effort. Participants were recruited from our institute but were not involved in any projects with the research team. __Experiment 1 control__ (i.e., Experiment SI-5) included 50 participants recruited from an online rating platform. For __Experiments 2–5__, we performed psychophysics experiments using an online rating platform. In each experimental condition, approximately 100 participants were recruited to participate in the task (see Supplementary Table 18 for the exact number). No statistical method was used to predetermine the number of participants, but the sample size was decided to be comparable to that used in previous similar studies. Participants received compensation in the range of $8–$15 per hour based on the expected difficulty of the task. No sex or age information was gathered from the participants for all our studies. Our participants were all located in North America and were financially compensated for their participation. __We excluded participants if__ they were not engaged in the task, as assessed using randomly placed catch trials with an unambiguous answer (e.g., pairing an unperturbed dog image with a cat image and asking which image is more cat-like). If a participant failed one catch trial for Experiments 2, 3, and 5, or two catch trials for Experiment 4, the task automatically terminated and their data was not analyzed.
Parent's numbers are specifically drawn from Figure 3 caption. (Some text may not format correctly. Apologies if I didn't catch)
> a Participants are shown two perturbations of the same image, of true class T, and are asked to select the image which is more like an instance of some adversarial class A. The image pair remains visible until a choice is made. b One of the two choices is an adversarial perturbation that increases the probability of classifying the image as A, denoted A↑. Experiment 2: T = A; the second image is perturbed to be less A-like, denoted A↓. Experiment 3: T ≠ A; the second image is formed by adding a right-left flipped version of the adversarial perturbation, which controls for the magnitude of the perturbation while removing the image-to-perturbation correspondence. Experiment 4: T ≠ A; the second image is an adversarial perturbation toward a third class , denoted . c We show examples of adversarial images which empirically yielded human responses consistent with those of the ANN (indicated by the red box) for ϵ = 2 and 16, corresponding to the lowest and largest perturbation magnitudes used in these experiments. Example images in (a–c) are obtained from the Microsoft COCO dataset62 and OpenImages dataset63; images in (a, b, and c) left are used for illustration outside of our stimulus set due to license limitations. d Box plots (same convention as Fig. 2c) quantifying participant bias toward A↑ (where A = T for Experiment 2 and A ≠ T for Experiments 3 and 4), as a function of ϵ for four different conditions (each a different adversarial class A) collected from n=389 participants for Experiment 2 (cat n = 100, dog n = 100, bird n = 90, bottle n = 99), n = 396 participants for Experiment 3 (cat n = 96, dog n = 100, bird n = 101, bottle n = 99) and n = 389 independent participants for Experiment 4 (sheep vs chair n = 97, dog vs bottle n = 99, cat vs truck n = 98, elephant vs clock n = 94). The red points (with ± 1 SE bars) indicate the mean across conditions. The black dashed line indicates the performance of a random strategy that is insensitive to the adversarial perturbations.