[intoamplitudes]

First impressions:

1. The data in most of the plots (see the appendix) look fake. Real life data does not look that clean.

2. In May of 2022, 6 months before chatGPT put genAI in the spotlight, how does a second-year PhD student manage to convince a large materials lab firm to conduct an experiment with over 1,000 of its employees? What was the model used? It only says GANs+diffusion. Most of the technical details are just high-level general explanations of what these concepts are, nothing specific.

"Following a short pilot program, the lab began a large-scale rollout of the model in May of 2022." Anyone who has worked at a large company knows -- this just does not happen.

[btrettel]

On point 2, the study being apparently impossible to conduct as described was also a problem for Michael LaCour. Seems like an underappreciated fraud-detection heuristic.

https://en.wikipedia.org/wiki/When_Contact_Changes_Minds

https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&d...

> As we examined the study’s data in planning our own studies, two features surprised us: voters’ survey responses exhibit much higher test-retest reliabilities than we have observed in any other panel survey data, and the response and reinterview rates of the panel survey were significantly higher than we expected.

> The firm also denied having the capabilities to perform many aspects of the recruitment procedures described in LaCour and Green (2014).

[raphman]

FWIW, in the q&a after a talk, he claims that it was a GNN (graph neural network), not a GAN.

(In this q&a, the audience does not really question the validity of the research.)

https://doi.org/10.52843/cassyni.n74lq7

[mncharity]

Wayback of the Sloan School seminar page shows him doing one on February 24, 2025. I wonder how that went.

I miss google search's Cache. As with the seminar, several other hits on MIT pages have been removed. I'm reminded of a PBS News Hour story, on free fusion energy from water in your basement (yes, really), which was memory holed shortly after. The next-ish night they seemed rather put out, protesting they had verified the story... with "a scientist".

That cassyni talk link... I've seen a lot of MIT talks (a favorite mind candy), and though Sloan talks were underrepresented, that looked... more than a little odd. MIT Q&A norms are diverse, from the subtle question you won't appreciate if you haven't already spotted the fatal flaw, to bluntness leaving the speaker in tears. I wonder if there's a seminar tape.

[rdtsc]

Oh interesting. I haven't talked to any recent graduates but I would expect an MIT PhD student to be more articulate and not say "like" every other word.

There was a question at the end that made him a little uncomfortable:

[1:00:20]

   Q: Did you use academic labs only or did you use private labs?

   A: (uncomfortable pause) Oh private, yeah, so like all corporate, yeah...

   Q: So, no academic labs?

   A: I think it's a good question (scratches head uncomfortably, seemingly trying to hide), what this would look like in an academic setting, cause like, ... the goals are driven by what product we're going make ... academia is all, like "we're looking around trying to create cool stuff"...

My 8 year-old is more articulated than this person. Perhaps they are just nervous, I'll give them that I guess.

[raphman]

Oh, he also claimed that he got IRB approval from "MIT’s Committee on the Use of Humans as Experimental Subjects under ID E-5842. JEL Codes: O31, O32, O33, J24, L65." before conducting this research, i.e., at a time when he wasn't even a PhD student.

[constantcrying]

A month by month record of scientists time spend on different tasks is on its face absurd. The proposed methodology, automatic textual analysis of scientists written records, giving you a year worth of a near constant time split pre AI is totally unbelievable.

The data quality for that would need to be unimaginably high.

[lumost]

If a paper is difficult to replicate in a high volume field.. will it ever be replicated? The question we should be asking is how many fraudulent papers are there in the field?

I’ve even worked in places where some ML researchers seemingly made up numbers for years on end.

[3s]

I agree with point 1, at least superficially. But re: point 2, there are a lot of companies with close connections to MIT (and other big institutions like Stanford) that are interested in deploying cutting edge research experiments, especially if they already have established ties with the lab/PI

[pixl97]

>The data in most of the plots (see the appendix) look fak

Could a Benford's Law analysis apply here to detect that?

[constantcrying]

How would you apply it, why would it be applicable?

[mzs]

  % gunzip -c arXiv-2412.17866v1.tar.gz | tar xOf - main.tex | grep '\bI have\b'
  To summarize, I have established three facts. First, AI substantially increases the average rate of materials discovery. Second,  it  disproportionately benefits researchers with high initial productivity. Third, this heterogeneity is driven almost entirely  by differences in judgment. To understand the mechanisms behind these results, I investigate the dynamics of human-AI collaboration in science.
          \item Compared to other methods I have used, the AI tool generates potential materials that are more likely to possess desirable properties.
          \item The AI tool generates potential materials with physical structures that are more distinct than those produced by other methods I have used.
  % gunzip -c arXiv-2412.17866v1.tar.gz | tar xOf  - main.tex | grep '\b I \b' | wc
      25    1858   12791
  %

[rafram]

Not sure what you’re trying to say.