adminprof's comments

I think you're missing two fatal problems in this "publish all raw data and code" mindset. I don't think the desire of commercialization is high on the list of fatal problems preventing people from publishing data+software.

1) How do you handle research in domains where the data is about people, so that releasing it harms their privacy? Healthcare, web activity, finances. Sure you can try to anonymize it, anonymization is imperfect, and even fully anonymized data can be joined to other data sources to de-identify people; k-anonymity only works in a closed ecosystem. If we live in a world where search engine companies don't publish their research because of this constraint, that seems worse than the current system.

2) How does one define "re-runnable processing"? Software rots, dependencies disappear, operating systems become incompatible with software, permission models change. Does every researcher now need a docker expert to publish? Who verifies that something is re-runnable, and how are they paid for it?

It seems like it's just that the goals are different. Computer science researchers could also make the opposite case, "The underwhelming impact of software development on producing state-of-the-art computer science discoveries."

I think your comment is pretty spot on. Just three things I'd add to, are that 1) the advisor is not intentionally downgrading your letter to "just good", but it's that they're obligated to write a better than "just good" letter for someone who has been more productive both in terms of research and in being a leader in the community (these two things go hand in hand). Writing the same letter for two students regardless of what they accomplish would be unfair.

And 2) sometimes getting more funding is simply not possible, as in the advisor has basically reached the limit of what they can do. There's a limit to the number of proposals that one can submit and the number of calls that fit their research agenda. So what I think you're missing is that if an advisor has less funding, there's going to be more pressure to finish sooner and less freedom to explore ideas beyond what's written in a previous grant proposal.

3) I've never heard of a tenured professor that concerned about their publication rate. In fact, most of them don't even update their CVs or websites with the last few years of papers. It's always the student who is trying to get more papers.

Students are users of a class, as much as a dentist is a user of a dental software application. Neither were involved in the making of the class or software. Both were delivered the experience as designed by the course staff (instructor and/or TAs), or software team. Both only see the final product after months or years of development, done by specialists trained for many years even before that.

The error you're making is stating that "students are involved in the entire process" which is laughable. Many classes have gone through years of iteration, and even new courses take many months to develop before students set foot in the classroom, not to mention the years of experience and education needed to get the instructor to the point that they can even make a class in several months.

I'd like to see in a forum for dentists a thread where they offer clever suggestions for how to change software development. Clearly, all these dentists have used plenty of software before.

They would have seemingly great ideas like "programmers should get 10% of their pay docked each time I encounter a bug in a program" or "the real solution is to hire someone to test the program from start to finish before releasing it".

I bet programmers on Hacker News would be livid upon hearing these suggestions, but seem to have no problem announcing their clever solutions about other disciplines (not excluding myself).

Simply, because using absolute points would lead to debate over every single decimal of a point. A student with 89.95% would be determined to fight to get 90.05% and then continue to fight to get 90.15% so it only makes the problem worse.

This doesn't work when I've tried it. How many students or times have you implemented this policy? First, it doesn't make sense when the regrade is most objective (like points were calculated wrong, or the grader didn't see something that the student wrote). And if you say that it doesn't apply for straightforward grading mistakes, then you get emails asking you whether something is a grading mistake or has the chance of lowering a grade.

And I've tried this policy before, and got students who wrote in my course evals something like "the professor intentionally tries to scare students from asking for regrades by threatening to lower their grade even more." And then what about when you are still asked for a regrade (which in my experience was not zero, but maybe about a third or half as much as without this policy). In those cases, you end up doing way way more, so the level of effort actually increases.

> Please don't use such circular logic. We're debating whether the research properly qualifies as human subject research; we're not debating about what the IRB actually decided on that question, because they may have gotten it wrong.

Yes that's what we're debating. But you used the word "exemption" which has a specific technical meaning in human subjects research, and I'm saying that it's not an exemption. There are 8 tests for exemption, and I'm pointing out that this is not an IRB exemption.

> The policy itself is certainly the intended subject of the research. But the methods they've chosen mean they are also collecting and analyzing information about the responses of real live humans to their interactions and interventions, and that qualifies this as human subject research irrespective of the naive intentions of the researchers. Having a non-human subject does not preclude also having a human subject.

Do you have a source for this interpretation? It sounds like this is your interpretation, but not the federal one. Following your interpretation, surveys of companies (e.g. emailing [email protected] to ask how many employees they have) would fall under the definition of human subjects.

Thanks for the continued conversation, but I think this is my last comment. Nothing personal, but this is a bit exhausting. It seems like you're debating two other people on this forum about this exact definition, and you might consider that maybe you're just wrong about your interpretation?

Here's one final source, if it helps provide closure:

To meet the definition of human subjects, you must ask “about whom” questions. Questions about your respondents' attitudes, opinions, preferences, behavior, experiences, or characteristics, are all considered “about whom” questions. Questions about an organization, a policy, or a process are “about what” questions.

https://campusirb.duke.edu/resources/guides/defining-researc...

Not at all,

1) this is clearly not an exempt study, which is a category of its own that the IRB reviews and makes a judgment on. The authors would immediately have been able to point out the protocol number of the exempt study if it were exempt. Rather it's not considered human subjects as the authors clearly state on their FAQ.

2) it seems like you're thrown off by the example, because if you ended your sentence as "The bit you've quoted is intended to clarify that "about whom" means the subject is the patient" then we would be in agreement, and it'd be more obvious that the subject is, in fact, the website's policies/procedures. Here's an excerpt from the written text of the common rule,

"“About whom” – a human subject research project requires the data received from the living individual to be about the person."

https://hso.research.uiowa.edu/defining-human-subjects

He seems to have good intentions, but does not seem to have knowledge of IRB which may make this situation worse.

Specifically, he confuses "does not constitute human subjects research" with "exemption" which is a pretty big difference and anyone who works with human subjects should know this.

From his Twitter thread, "Update: They are now saying they have an exemption. They have not made any forms available or explained the lack of informed consent."

Exemptions are protocols that have been reviewed, and deemed exempt based on one of 8 very specific criteria. Studies deemed not constituting human subjects research are returned by the IRB, and not considered reviewed.

Given that the authors actually said "...to the Princeton University Institutional Review Board, which determined that our study does not constitute human subjects research" this is clearly NOT an exemption, and informed consent is not a consideration as far as the IRB is concerned.

This is actually the most technically correct answer on this page. Everyone is going by their own opinions about definitions of what constitutes human subjects research, rather than starting from the primary sources. IRB guidelines are dictated by the federal government "common rule", a common standard adopted by all institutions that receive federal funding.

"about whom" is a key criteria from the federal government to determine whether something fits the definition of human subjects research. Here's a quote from HHS:

"The phrase ‘about whom’ is important. A human subject is the person that the information is about, not necessarily the person providing the information. In the case of biospecimens, the human subject is the person from whom the specimen was taken."

https://www.hhs.gov/ohrp/sites/default/files/OHRP-HHS-Learni...

Reading that, it's clear that the Princeton study does not fit the definition of human subjects research. The complainants may be able to sue for damages to the university, but not because the study was improperly classified as human subjects.

Sure, as someone who has also been on both sides, that's not what I was describing at all though. None of the examples I gave are handled by an editor, except maybe "managing issues dealing with misconduct" depending on the situation, but maybe by the publisher's legal team. But the other issues are not handled by an editor in my experience.

So I think your assumption that academic publishing is "a website hosting up PDFs" is the source of confusion for "what justifies this much money." Just like AirBnB isn't just a website hosting JPEGs, while being worth 129 billion dollars. Hosting PDFs can already be done by arXiv or Google Drive or Github as you said.

Customer support is for peer reviewers who can't log into their account, for managing issues dealing with misconduct, for handling issues with payments, for post-publishing corrections and errata, for passing accounts from editors who become non-responsive to other editors, etc. Not just dealing with readers or subscribers.

This comes up every time there's an article about academic publishing. Yes peer reviewers do the reviewing, but it's the long-term infrastructure and coordination that the journal provides.

AirBnB's content is generated by users, but AirBnB itself requires software development, legal, customer support, HR, program managers, quality control, etc. Same with publishers.

Note that this journal now has a publishing fee for authors to cover these costs, rather than a fee for the reader as before. The 2022 fee for each author is $1,705 according to the FAQ. So moving to open access it not about removing the costs (which many people on Hacker News seems to always assume), but changing who pays for it.

I would like to subscribe to your newsletter.

Agree with all your points. The author has been in academia all his life so after 20 years or so, I can see how he can feel frustrated by it.

I think the general summary of what you're pointing out is that in industry it's easy to do things that the company wants to do (make money, make your boss look good, make life easier for the execs), and hard to do things that you want to do (cool stuff, disseminate ideas, and explore).

When you start at a company, you're optimistic and campaign to do a balance of both, but over time it wears you out so almost every defaults to doing the company work and fades out of public view. And yes, MSR is generally the exception but even then notice how many researchers there now do something related to "optimizing productivity".

Do these status dashboards have any more credibility left? Always showing green when services are down.

I think I see the confusion between us. You don't see the difference between selling data to a company, and selling ad space where the advertiser can choose for what demographics it shows up for.

Let me try to make it more clear. Do you see the difference between "hey Chase Bank, do you want to buy this file containing data about grey-area's interests, age, political stance, credit score, purchasing habits, etc." versus "hey Chase Bank, do you want to put an ad on my website that is only shown to people with credit scores above 600 and are interested in savings accounts"?

If they both seem the same to you, then I don't think your perspective is one that a reasonable person would take. If you do see the difference, then Facebook is doing the latter, but the word "selling data" conjures the former, which you do recognize as a different matter.

I'm going to ignore the nonsensical definition you gave of selling data = selling movies, being the only other definition of selling you could imagine, in hopes that it was just an oversight.

They actually don't, unless you define selling as they allow advertisers to select what demographics/attributes their ads target. But the actual data stays on the Facebook servers. If you're referring to the apps having access to user data, that was not selling at all, but instead a permission originally granted by users by probably forgotten about. Basically, unless you contort the definition of selling to a very different meaning, that's simply not true.

And if you do use that definition of selling, then everyone is selling your data. All the politicians who decry tech companies are selling your data using the same definition. Every advertiser, retail store, bank, basically every large business offers other businesses a way to access a specific subset of their users.

Yet everyone is freaking out and moralizing about nonfinancial data voluntarily given to Facebook. If only the credit bureaus kept our financial and identity data as Facebook kept your list of favorite movies and your selfies.